feat: add devnet 5 support#378
Draft
MegaRedHand wants to merge 8 commits into
Draft
Conversation
…361) ## 🗒️ Description / Motivation Ports the typed two-level multi-signature envelope introduced by contributor commit [`anshalshukla/leanSpec@0ab09dd`](anshalshukla/leanSpec@0ab09dd) ("dummy type 1 and type 2 aggregation with block proofs") to ethlambda: - `TypeOneMultiSignature` — single-message N-signer proof; replaces `AggregatedSignatureProof` on the `SignedAggregatedAttestation` gossip wire. - `TypeTwoMultiSignature` — merged multi-message proof binding every per-attestation Type-1 plus a singleton proposer Type-1 over the block root. - `SignedBlock.signature: BlockSignatures` → `SignedBlock.proof: ByteListMiB` carrying the SSZ-encoded merged Type-2. The upstream commit is WIP (verify functions are explicit stubs, not yet in canonical `leanethereum/leanSpec`). ethlambda leads the wire-shape migration so the type plumbing is in place when canonical absorbs the refactor and real `lean_multisig` bindings land. **Opening as draft until canonical catches up.** ## What Changed Landed as three commits, one per phase. Each phase compiled and passed `make test` independently. ### Phase 1 — `f2d0fb5` — additive type plumbing - `crates/common/types/src/block.rs` — added `TypeOneInfo`, `TypeOneMultiSignature`, `TypeOneInfos` (SSZ-list limit `MAX_ATTESTATIONS_DATA + 1`), `TypeTwoMultiSignature`, and `BytecodeClaim` (typed alias for `H256`, placeholder until `lean_multisig` defines the trusted evaluation). - SSZ round-trip + capacity unit tests. - Pure additive: no consumers yet. ### Phase 2 — `18a60b5` — gossip-layer pipeline - `crates/common/types/src/attestation.rs` — `SignedAggregatedAttestation.proof` → `TypeOneMultiSignature`. - `crates/blockchain/src/aggregation.rs` — `AggregatedGroupOutput.proof`, `aggregate_job`, `resolve_child_pubkeys`, `select_proofs_greedily` all carry/read Type-1. - `crates/storage/src/store.rs` — `PayloadEntry.proofs: Vec<TypeOneMultiSignature>`; subsumption logic reads `info.participants`. - Block-builder helpers (`compact_attestations`, `extend_proofs_greedily`, `build_block`) operate on Type-1 throughout. - Temporary `to_legacy` / `from_legacy` boundary at block assembly + block-body ingestion so `SignedBlock` wire stayed legacy through Phase 2. ### Phase 3 — `fc9ce1f` — block wire + storage - `SignedBlock.signature: BlockSignatures` → `SignedBlock.proof: ByteListMiB`. Legacy `BlockSignatures` / `AttestationSignatures` / `AggregatedSignatureProof` removed. - `crates/blockchain/src/lib.rs::propose_block` wraps the proposer XMSS as a singleton Type-1, calls `aggregate_type_2`, SSZ-encodes the merged proof, and stashes it on `SignedBlock.proof`. - `crates/blockchain/src/store.rs::verify_signatures` rewritten as a structural-only check (mirrors upstream `verify_type_2` stub): decode the merged proof, assert `info.len() == attestations.len() + 1`, validate per-attestation `(message, slot, participants)` alignment and the trailing proposer entry; no per-Type-1 crypto. - `crates/storage/src/store.rs::write_signed_block` / `get_signed_block` now store `ByteListMiB` blobs in the existing `BlockSignatures` column family (renaming deferred to avoid a CF migration). - `aggregate_type_2` is a no-crypto stub today: it preserves the full `TypeOneInfos` metadata list but leaves `proof: ByteListMiB::default()`. Real merging arrives when `lean_multisig` exposes a merged-proof primitive — the existing `aggregate_proofs` only handles single-message merging. - Test fixtures regenerated from canonical leanSpec (`make leanSpec/fixtures`). The regen also cleared three pre-existing forkchoice spec failures on `main` (`AttestationTooFarInFuture` ×2, `AggregateVerificationFailed(InvalidProof)` on `test_valid_gossip_aggregated_attestation`) — they were stale-fixture artifacts. ## Correctness / Behavior Guarantees **Verified at gossip:** `on_gossip_aggregated_attestation` continues to run real `ethlambda_crypto::verify_aggregated_signature` on every `SignedAggregatedAttestation`. Invalid aggregates are rejected at the gossip boundary just like before. **Block-level becomes structural:** Block-level verification no longer crypto-verifies the merged proof. The merged proof bytes can't be split client-side (the type-2 merging primitive doesn't exist in `lean-multisig` yet — the existing `aggregate_proofs` is single-message only). `verify_signatures` enforces: - `info.len() == attestations.len() + 1`, - each `info[i]` matches the corresponding `block.body.attestations[i]` on `participants`, `slot`, and `message`, - the trailing `info[N]` has `message == block_root`, `slot == block.slot`, single-bit `participants` set to `block.proposer_index`, - all participant indices fit within the validator registry. This is the conscious "mirror upstream stubs" trade-off agreed during planning. When `lean_multisig` ships a real `verify_type_2`, the structural stub is swapped for the real call. **Block-body ingestion preserves fork-choice LMD GHOST inputs:** since the merged proof can't be split, `process_new_block` inserts info-only Type-1 entries (real `(message, slot, participants)`, empty proof bytes) into the payload buffer. `extract_latest_known_attestations` works unchanged. Empty-bytes entries never get fed back into `aggregate_proofs` (that path is only hit when multiple proofs share the same `AttestationData`, in which case at least one came from gossip with real bytes). **Storage:** Table name kept (`BlockSignatures`) to avoid a RocksDB CF migration; doc comment updated. Renaming to `Table::BlockProof` is a follow-up. **Skipped tests, all behind `TODO(type1-type2)`:** - `ssz_spectests.rs`: `SignedBlock`, `BlockSignatures`, `AggregatedSignatureProof`, `SignedAggregatedAttestation` — on-disk SSZ bytes still use the legacy schema since canonical leanSpec hasn't absorbed the refactor. - `signature_spectests.rs`: `test_invalid_proposer_signature` — relies on block-level proposer-signature crypto, which is now a structural stub. Attempted to bump `LEAN_SPEC_COMMIT_HASH` to `anshalshukla/leanSpec@0ab09dd` to regenerate fixtures against the new schema. Reverted: the upstream testing harness in that commit (`leanSpec/packages/testing/src/consensus_testing/keys.py`) still imports `AttestationSignatures`, which the same commit removes — `fill` crashes on module load. Documented in a `NOTE(type1-type2)` in the Makefile. ## Tests Added / Run - Added: SSZ round-trip and capacity unit tests for the new Type-1/Type-2 containers in `crates/common/types/src/block.rs`. - Updated: `verify_signatures_rejects_participants_mismatch`, `build_block_caps_attestation_data_entries`, `on_block_rejects_duplicate_attestation_data`, the `compact_attestations` and `extend_proofs_greedily` tests, all `forkchoice_spectests.rs` step builders, `signature_types.rs` fixture converter, and the `rpc::test_get_latest_finalized_block` test — all rebuilt to construct the new merged-proof shape. - Verified locally: - `make fmt` — clean - `cargo clippy --workspace --all-targets -- -D warnings` — clean - `cargo test --workspace --release` — green (84 forkchoice spec tests, 7 signature spec tests with 1 expected skip, all unit tests pass) ## Related Issues / PRs - Upstream commit being ported: [`anshalshukla/leanSpec@0ab09dd`](anshalshukla/leanSpec@0ab09dd) - Follow-ups when canonical absorbs the refactor: - Swap the structural `verify_type_2` stub for the real `lean_multisig` primitive. - Revert `LEAN_SPEC_COMMIT_HASH` skip markers in `ssz_spectests.rs` and `signature_spectests.rs`. - Consider renaming `Table::BlockSignatures` → `Table::BlockProof`. ## ✅ Verification Checklist - [x] Ran `make fmt` — clean - [x] Ran `make lint` (clippy with `-D warnings`) — clean - [x] Ran `cargo test --workspace --release` — all passing --------- Co-authored-by: Tomás Grüner <47506558+MegaRedHand@users.noreply.github.com>
# Conflicts: # crates/blockchain/src/store.rs # crates/common/test-fixtures/src/fork_choice.rs # crates/net/p2p/src/req_resp/handlers.rs # crates/net/rpc/src/lib.rs # crates/storage/src/store.rs
…370) Branched from #378 ## Summary - Bump `lean-multisig` / `leansig_wrapper` to devnet5 HEAD (`0242c909`) and rewrite `ethlambda-crypto` on the new Type-1 / Type-2 API. - Align the on-wire block proof with [leanSpec PR #717](leanEthereum/leanSpec#717) — `SignedBlock.proof` carries the SSZ-encoded `TypeTwoMultiSignature { proof: ByteList512KiB }` container, which collapses to `[4-byte LE offset = 4][type2_wire]` on the wire. - Port leanSpec PR #717's `SyncService._deconstruct_block_into_store` to the actor: imported blocks are SNARK-split per attestation, merged with local partial Type-1s, and (on aggregators) re-published on gossip. - Re-fixture against leanSpec `d9d2e67` (just past PR #717) and migrate the prod_scheme key JSON shape so signature and forkchoice spec tests cover the real cryptographic verifier end-to-end. ## Branch commit list | sha | what | |--|--| | `1cd80dd` | Crate-level integration: new Type-1 / Type-2 wrappers, real merge in `propose_block`, real `verify_type_2` in `verify_block_signatures`. | | `2c9dec0` | First pass at PR #717 envelope: `TypeOneInfo { participants, proof }`, `TypeTwoMultiSignature { info, proof }`, drop `bytecode_claim`. `split_type_2_signature(index)` → `split_type_2_by_message(message)`. | | `5361136` | Strip per-component Type-1 bytes when packing the Type-2 envelope — real Type-1s are ~225 KiB, N+1 copies blow the (old) 1 MiB `ByteListMiB` cap. | | `3199e7d` → `70c7cdb` | Experimental `--crypto-merge-t1-into-t2` flag, **reverted**. The merge runs synchronously on the actor today; moving it off-thread is a follow-up. | | `604ea4c` | Plan B: flatten `TypeOneMultiSignature` to `{ participants, proof }`, delete `TypeOneInfo` / `TypeOneInfos` / Rust `TypeTwoMultiSignature` wrappers, rename `ByteListMiB` → `ByteList512KiB`. | | `cc3df59` | Plan A: `reaggregate_from_block` module + actor hook. Caps `MAX_REAGGREGATIONS_PER_BLOCK = 4`, skips attestations behind the store's justified checkpoint, runs only when the chain is in sync. Aggregator-only republish on gossip. | | `4238a94` | Bump pinned `LEAN_SPEC_COMMIT_HASH` to `d9d2e67`, switch fixture generation to `--fork Lstar --scheme=test/prod`, port fixture parsers to the PR #717 schema (`signedBlock.proof.data` blob, `attestation.proof.proof.data` for gossip aggregates). | | `961aba4` | Restore the thin SSZ container header in front of the merged proof: `SignedBlock::wrap_merged_proof` / `merged_proof_bytes` helpers; lower `MAX_ATTESTATIONS_DATA` from 16 to 8 to match leanSpec PR #717. | ## Crypto crate API | function | wraps | notes | |--|--|--| | `aggregate_signatures(pks, sigs, msg, slot)` | `aggregate_type_1([], raw_xmss, …)` | Type-1 from raw XMSS only | | `aggregate_mixed(children, raw_pks, raw_sigs, msg, slot)` | `aggregate_type_1(children, raw_xmss, …)` | mixed Type-1 children + raw XMSS | | `aggregate_proofs(children, msg, slot)` | `aggregate_type_1(children, [], …)` | recursive Type-1 merge | | `verify_aggregated_signature(proof, pks, msg, slot)` | `verify_type_1` | Type-1 SNARK verify + explicit binding check | | `merge_type_1s_into_type_2(parts)` | `merge_many_type_1` | bundle N Type-1s into a Type-2 | | `verify_type_2_signature(proof_bytes, pks_per_component, expected_bindings)` | `verify_type_2` | Type-2 SNARK verify + per-component binding check; takes `&[u8]` after envelope strip | | `split_type_2_by_message(proof_bytes, pks_per_component, message)` | `split_type_2` (after locating index by message) | disaggregate to one Type-1; mirrors leanSpec `split_by_msg` | Type-1 / Type-2 proof bytes are `compress_without_pubkeys()` form throughout. `verify_type_2_signature` and `split_type_2_by_message` take `&[u8]` so callers feed the raw bytes (post-envelope-strip) directly. ## Wire format ``` TypeOneMultiSignature { participants: AggregationBits, proof: ByteList512KiB } SignedBlock.proof bytes: [4-byte LE offset = 4][raw lean-multisig Type-2 wire] ``` The 4-byte prefix is the SSZ Container-with-one-varlen-field offset header — the spec's `TypeTwoMultiSignature { proof: ByteList512KiB }` container. `SignedBlock::merged_proof_bytes()` / `SignedBlock::wrap_merged_proof()` keep the magic number off the call sites. No Rust struct for `TypeTwoMultiSignature` — per-component participants come from `block.body.attestations[i].aggregation_bits` and `block.proposer_index`, not the envelope. `MAX_ATTESTATIONS_DATA = 8` (down from 16, matching leanSpec PR #717). The merged Type-2 binds `MAX_ATTESTATIONS_DATA + 1 = 9` components, within lean-multisig's `MAX_RECURSIONS = 16`. ## Reaggregate-from-block New `crates/blockchain/src/reaggregate.rs`. After `process_block` succeeds and the chain is in sync, the actor: 1. Selects up to 4 attestations whose target outruns the store's justified checkpoint AND whose participants extend the local coverage. 2. `split_type_2_by_message`-splits each one out of the block's merged Type-2 proof. 3. Merges with locally-held partial Type-1s via `aggregate_proofs`. 4. Writes the combined proof into `latest_new_aggregated_payloads`. Aggregator-role nodes republish on gossip. 5 unit tests cover the candidate-selection rules without paying SNARK cost (target-slot gate, participant subset gate, hard cap, priority ordering). Each `split_type_2_by_message` runs a fresh SNARK; it currently executes synchronously on the actor thread. Moving to an off-thread worker mirroring `aggregation::run_aggregation_worker` is a natural follow-up if profiling shows it bleeding into the slot budget. ## Test coverage | Suite | Result | |---|---| | `signature_spectests` | 13 / 13 | | `forkchoice_spectests` | 84 / 84 | | `stf_spectests` | 35 / 35 | | `ssz_spectests` | 51 / 51 | | `test_driver_e2e` (Hive) | 8 / 8 | | `ethlambda-blockchain` unit | 24 / 24 (incl. 5 new reaggregate-from-block) | | Workspace total | 419 / 0 | Fixture regeneration: `LEAN_SPEC_COMMIT_HASH = d9d2e67`, generated with `make leanSpec/fixtures` (`uv run fill --fork Lstar --scheme=prod -o fixtures`). The prod_scheme key JSON shape upstream still has the pre-#725 flat layout (`attestation_public` / `attestation_secret` at top level) which the post-#725 `keys.py:395` no longer reads; this branch carries a one-shot local migration to the nested shape (`attestation_keypair.public_key` etc.) so the prod-scheme fixture filler runs. A small upstream PR converting the 12 `prod_scheme/*.json` files would let us drop the local step. ## Devnet validation Pending — re-run on a multi-node devnet now that `verify_type_2` actually executes on the import path. Expected to surface latency cliffs that the `--crypto-merge-t1-into-t2` flag previously hid.
MegaRedHand
added a commit
that referenced
this pull request
May 27, 2026
Pinning to 825bec6 unintentionally pulled in leanSpec's devnet5 aggregated proof wire format (PR #717, an unavoidable ancestor of the download fix #745). That format requires the full crypto-stack migration tracked in the devnet5 PRs (#378/#370) and is not yet on main, so the forkchoice spec tests failed with AggregateVerificationFailed(DeserializationFailed). Repoint to f12000b (PR #725), the commit just before devnet5. It carries the key-schema rename that unbreaks fixture generation against the released `latest` test keys, while staying on the old proof format that the current leanMultisig@5eba3b1 still understands. The proofData -> proof field rename is therefore not needed and is reverted. f12000b predates the download fix (#745), whose download_keys reads the still-open download tempfile and intermittently aborts with EOFError. Work around it in both the Makefile and CI by fetching+extracting the prod keys with curl+tar before `fill`, so the buggy code path is skipped. Both blocks are flagged for removal once the pin moves past #745. Verified against freshly regenerated f12000b fixtures: forkchoice 84/0, signatures 11/0, full workspace 421/0.
5 tasks
MegaRedHand
added a commit
that referenced
this pull request
May 27, 2026
…star (#391) ## Summary The pinned `LEAN_SPEC_COMMIT_HASH` (`18fe71f`, 2026-04-29) can no longer generate fixtures: the released `latest` test keys moved to the new key JSON schema (`attestation_keypair.public_key` etc.), so the old code fails with `KeyError: 'attestation_public'` during fixture generation. The fix bumps the pin to **`f12000b`** (leanSpec PR #725, 2026-05-17), which reads the new key schema. This is the commit **just before** leanSpec's devnet5 work, chosen deliberately: - An earlier draft of this PR pinned to `825bec6` (the #745 download fix). That commit unavoidably includes leanSpec PR #717 (`ac5f259`), which switched the aggregated-proof wire format to **devnet5**. The Rust client still runs `leanMultisig@5eba3b1` (pre-devnet5), so the forkchoice spec tests failed with `AggregateVerificationFailed(DeserializationFailed)`. The devnet5 proof format requires the full crypto-stack migration tracked in #378/#370, which is not yet on `main`. - `f12000b` predates #717, so fixtures keep the **old proof format** that the current crypto stack deserializes correctly. No Rust changes are needed. Also rename `--fork devnet` → `--fork Lstar` to match the upstream fork rename (already in effect at `f12000b`). ### Download workaround `f12000b` predates leanSpec PR #745, whose `download_keys` reads the still-open (unflushed) download tempfile, intermittently truncating the gzip tail and aborting with `EOFError` (surfaced as `Aborted!`). Both the Makefile and CI now fetch+extract the prod keys with `curl`+`tar` before `fill`, which fully writes the archive before reading it; `fill` then sees the keys present and skips its own buggy download. Both blocks are commented `Remove once the pin moves past PR #745.` ## Why not 825bec6 / devnet5? The devnet5 aggregated-proof wire format and the matching `ethlambda-crypto` rewrite live in #378 / #370 and are not yet merged to `main`. Bumping fixtures to a devnet5 leanSpec commit here would require duplicating that crypto work. This PR stays scoped to "unbreak fixture generation on `main`" and remains on the pre-devnet5 format until the devnet5 PRs land. ## Test plan - [x] `rm -rf leanSpec && make leanSpec/fixtures` — clean download (via curl+tar) + extract, 554 fixtures generated, no `Aborted!`. - [x] `cargo test -p ethlambda-blockchain --test forkchoice_spectests` — 84/0. - [x] `cargo test -p ethlambda-blockchain --test signature_spectests` — 11/0. - [x] `make test` — full workspace suite, 421/0. - [ ] CI runs green.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
🗒️ Description / Motivation
What Changed
Correctness / Behavior Guarantees
Tests Added / Run
Related Issues / PRs
✅ Verification Checklist
make fmt— cleanmake lint(clippy with-D warnings) — cleancargo test --workspace --release— all passing