Skip to content

Full-duplex n-ary search (emulator side) - #55

Open
SDL-A1 wants to merge 18 commits into
FairgateLabs:devfrom
SDL-A1:full-duplex-search
Open

SDL-A1 wants to merge 18 commits into
FairgateLabs:devfrom
SDL-A1:full-duplex-search

Conversation

@SDL-A1

@SDL-A1 SDL-A1 commented Oct 18, 2026

Copy link
Copy Markdown

What and why

The dissection search locates a boundary between an agreed state and a disagreed
state. Agreement at a position is a symmetric relation, so the selection does not
depend on which party evaluates it. The current protocol spends one message on a
dissection and a second on the selection over it; if either party can select, a
single message can carry both, and the search advances one n-ary level per
message instead of per round trip.

That halves the dispute's transaction count and, because every stage carries a
challenge timelock, its worst-case duration — which is what bounds how long
capital stays locked.

Selected per program by search_mode, defaulting to asymmetric, so nothing
changes for existing program definitions. The next_winternitz call order is
preserved in that mode, so existing setups derive identical key material.

Contents

  • SearchMode, with dissector/selector as the single source of truth for
    stage ownership, so key generation, graph construction and script key lookup
    cannot drift apart.
  • advance_full_duplex: select over the counterparty's dissection, then dissect
    the subinterval just selected. Two thin role wrappers share one implementation.
  • Equivocation at the disputed boundary. A party is bound to its own earlier
    commitment, and to the counterparty's value at the agreed boundary. At the
    disputed boundary the relation inverts: selecting it declared disagreement,
    so claiming that value is the misbehaviour. Punishing inequality there would
    penalise an honest party.
  • Synthetic RISC-V programs for tests, so the checkpoint-backed paths can be
    exercised without the docker toolchain.

A related correction

The trace-hash, halt and program-counter challenges are assertions about the
challenged party's own claim. Under the asymmetric search they can be stated
against the value committed during the search, because where no equivocation has
been established the two coincide. Under full duplex they do not: at the disputed
boundary the committed value may be the counterparty's, which the party is bound
to differ from. They now use the party's re-signed values explicitly —
behaviour-identical in asymmetric mode.

Testing

  • A full-duplex search over a real trace locates the divergence, across several
    divergence points, including the handover to prover_final_trace.
  • Full-duplex and asymmetric searches commit identical digits on the same trace,
    for n ∈ {2,4,8}.
  • The equivocation cases are tested against the script interpreter in both
    directions, including that an honest party differing at the disputed boundary
    is not punished. Mutating the condition to a constant fails exactly that test.
  • 52 unit tests pass, plus 54 in bitcoin-script-riscv. The 35 pre-existing
    failures in decision::challenge::tests are unchanged — they need compiled
    reference programs from the docker build.

Notes for review

The inverted comparison is the part most worth a second pair of eyes: if my
reading of the agreed/disputed boundary semantics is wrong, it is wrong in the
direction that penalises an honest party.

Companion PRs: job type in rust-bitvmx-job-dispatcher, orchestration in
rust-bitvmx-client. This one is self-contained and should land first.

🤖 Generated with Claude Code

Sergio Demian Lerner added 10 commits October 18, 2026 01:58
The dissection search locates a boundary between an agreed state and a
disagreed state. Agreement at a position is a symmetric relation, so the
selection decision does not depend on which party evaluates it. The
asymmetric protocol spends one message on a dissection and a second on a
selection; if either party can select, a single message can carry both,
and the search advances one n-ary level per message instead of per round
trip.

This commit adds the search logic only. The transaction graph is
untouched, so nothing here changes the deployed protocol.

- choose_segment: rename prover_hashes/my_hashes to published_hashes/
  local_hashes and document the symmetry. The arithmetic was already
  role-neutral; only the naming implied otherwise.
- Party, StageMessage, SearchState: model a stage as a selection over the
  counterparty's previous dissection plus a dissection of the subinterval
  just selected. Stage ownership alternates by parity and so is fixed at
  setup, when the one-time keys are generated.
- run_full_duplex_search and run_asymmetric_search: drivers over two trace
  views, for comparison.

Tests cover: both searches locate the same step and commit the same
selection digits; the message count falls from 2R+1 to R+1; the outcome is
independent of which party dissects first; the digits reconstruct the
agreed step; and a party that agrees with every published interior hash
inherits the disagreement already recorded at the upper boundary, which is
the case where the traces agree everywhere the search can look.

The equivalence tests assume both parties follow the protocol against
their own traces. Soundness under adversarial selection is a separate
argument and is not covered by these tests.
Which party dissects at which stage decides which one-time keys each party
must generate, and the dispute transaction graph is built and pre-signed
before any dispute exists. The choice therefore belongs to the program
definition, alongside nary_search, not to anything evaluated at dispute
time.

SearchMode::dissector and ::selector are the single source of truth for
stage ownership, so key generation, graph construction and script key
lookup cannot drift apart.

For the full-duplex search the party dissecting at stage 1 is derived from
the parity of the round count, so that the trailing selection -- the digit
over the final dissection, which has no stage of its own -- always falls to
the prover and rides on the prover's adjudication transaction rather than
costing a further message. This keeps the search at R+1 messages for any
round count.

Defaults to Asymmetric, so existing program definitions are unaffected.
Under the full-duplex search the dissections alternate, so a position's
state hash may have been committed by either party. Which inconsistency
is punishable depends on who committed it.

A party is always bound to its own earlier commitment. It is also bound
to the counterparty's value at the agreed boundary, because selecting
that boundary is what declared the agreement. At the disputed boundary
the relation inverts: selecting it declared disagreement, so the party is
bound to differ, and claiming the counterparty's value is the
misbehaviour. Punishing inequality there would penalise an honest party
for correctly disagreeing.

equivocation_resign_challenge therefore takes counterparty_committed and
asserts equality exactly when that holds and the kind is NextHash, which
is the disputed boundary. Positions are read from whichever party
dissected their round; under the asymmetric search that is always the
prover, so the behaviour is unchanged there.

Three script-interpreter tests cover the cases in both directions,
including that a party which differs at the disputed boundary is not
punished. Mutating the condition to a constant fails exactly the
disputed-boundary test and no other.

Separately, the trace-hash, halt and program-counter challenges are
assertions about the challenged party's own claim. In the asymmetric
search they could be stated against the value committed during the
search, because where no equivocation has been established the two
coincide. Under the full-duplex search they no longer do: at the disputed
boundary the committed value may be the counterparty's, which the party
is bound to differ from, so comparing against it would validate the wrong
party's claim. These now use the party's re-signed values explicitly,
which is behaviour-identical under the asymmetric search.
The asymmetric search splits a narrowing level across two calls, one per
message: the prover dissects, the verifier selects. A full-duplex stage
does both, so the search advances a whole level per message.

advance_full_duplex is that step: select over the counterparty's previous
dissection, then dissect the subinterval just selected. Stage 1 has no
predecessor and only dissects. The two role wrappers differ only in which
checkpoint log they load and which side of it holds whose hashes, so the
decision logic has a single implementation.

The prover now selects as well as dissects, so its log gains the running
bound on the disputed step and the counterparty's dissections. Both are
serde-defaulted, leaving existing checkpoints loadable. The bound is
initialised from the prover's own claim; left at zero it would clamp
every selection to step zero, which two tests now guard.

The decision logic itself is the already-tested choose_segment, and the
pure drivers in nary_search cover the stage sequencing. The checkpoint
glue here is not exercised by tests: it needs compiled RISC-V programs,
which are produced by the docker build.
Completes the emulator's surface for the full-duplex search: a stage can
now be invoked and its outcome reported, with the selection digit absent
at stage 1, which has no previous dissection to select over.

The command declares long-only flags. Deriving short names here would
produce several conflicting `-c` arguments, which makes clap's debug
assertions reject the command outright. Note that the existing
prover-get-hashes-for-round and verifier-choose-segment commands already
have this conflict and panic when invoked from the command line in a
debug build, so these subcommands are reachable only through the library
API. That is left alone here.
The reference programs are compiled by a RISC-V toolchain that runs under
docker, so without it every checkpoint-backed path was untestable:
loading, executing, dissecting a trace. This writes a minimal RISC-V ELF
and a matching program definition into a temporary directory instead,
which exercises the real loader, executor and checkpointing rather than
standing in for them.

On top of it, two tests that could not be written before:

- A full-duplex search over a real trace locates the divergence, for
  several divergence points. The prover's trace hash is corrupted from a
  known step, so the two views first differ exactly there.
- The full-duplex and asymmetric searches commit the same digits on the
  same trace, across nary 2, 4 and 8. The equivalence was previously only
  checked against synthetic hash vectors.

Writing them found a real bug. A party only selects at its own stages, so
it must also apply the digit the counterparty published at the previous
stage; without that its base lagged a level behind and it compared
against the wrong positions. Recording both parties' digits in order is
also what makes the sequence the base-n representation of the disputed
step, which is what the final script reconstructs. advance_full_duplex
now takes that digit, and skips the dissection past the last round, where
only the trailing selection remains.

The 35 pre-existing failures in decision::challenge::tests are unchanged:
they exercise specific compiled programs, which a synthetic one cannot
stand in for.
get_hashes indexes a round's dissection by round number, but a party
under the full-duplex search only dissects every other round, so its own
hash_rounds was dense and shifted: looking up round 2 in a list holding
only round 1 panicked as soon as the search handed over to the final
trace.

Both parties now record one round-indexed vector of published
dissections, filling in the counterparty's round before their own.
Stages run in order, so that keeps the vector aligned with round numbers
for either party. This also removes the need for the helper that
reassembled the published view from two separate lists: in both modes a
round's dissection is simply the entry under its round number.

The trailing selection is reported but no longer committed by the stage
that computes it. It rides on the adjudication message, and
prover_final_trace already applies it; committing it twice advanced the
base a level too far.

The end-to-end test now drives the handoff to prover_final_trace, so the
digit sequence it checks is the one the endgame actually sees.
One-time keys dominate setup, so it matters that making the search
symmetric redistributes them rather than adding any. Each level still
commits n-1 state hashes and one selection digit; only ownership moves.
The asymmetric search gives every hash to the prover and every digit to
the verifier, and the full-duplex search splits both by stage parity.
Hoisting the boundary position lookups out of the branches that use them
turned a never-taken path into an unconditional panic: step 0 is agreed
by construction and so has no position in the step mapping, which
get_hashes already special-cases. Any challenge raised at step 0 aborted
on the unwrap.

Both lookups are optional now, and an absent position counts as the
challenged party's own, which is what it always is under the asymmetric
search.

The same commit claimed that using a party's re-signed hashes in the
trace-hash, halt and program-counter challenges was behaviour-identical
under the asymmetric search. It is not: those challenges can be forced,
and forcing skips the equivocation checks that otherwise guarantee the
committed and re-signed values agree. The choice is now per boundary --
the committed value where the challenged party published it, the
re-signed value only where the counterparty did.

Caught by test_challenge_entrypoint and test_challenge_trace_hash_zero,
which pass on dev and failed here. They were invisible until the
docker-riscv32 submodule was initialised, since without it every test
needing a program fixture fails to load.
The trailing selection has no dissection of its own: it rides on the
adjudication message, which also carries the expanded trace for the
disputed step. Computing them in separate jobs made the implementation
contradict that, and cost a round trip the dispute does not need.

prover_full_duplex_final_trace selects over the final dissection and
produces the trace in one call, reporting both. The digit has to come
back: the party that publishes the adjudication message must put it in
that message's witness, and it cannot do that if the digit only ever
existed inside the emulator.
Sergio Demian Lerner added 8 commits November 13, 2026 02:52
NArySearchDefinition asserted that nary is a power of two, and underneath
it was doing arithmetic in bits: a digit extracted by shifting and
masking, a step rebuilt by shifting back, the trace padded to a power of
two and split into log2(nary)-bit chunks.

Compact BitVMX II dissects 192 ways.  States pack three to a 64-byte
FlatMerkle element, so the admissible branching factors are those with
ceil((b-1)/3) a power of two -- 11,12,13 then 23,24,25 and so on up
through 191,192,193 -- and not one of them is itself a power of two.  A
power-of-two element count and a power-of-two b are different conditions
and they never coincide.  2^30 is not divisible by 192 either, so the
paddings disagree about the trace length as well.

The change is smaller than it looks, because the bit-sliced arithmetic
already WAS mixed-radix arithmetic: a step index is full_rounds digits in
base nary and, where the bits did not divide evenly, one final digit in a
smaller base.  radix_tests pins that equivalence -- five properties
checked against the shift-and-mask implementation before it was touched,
over seven branching factors and six trace lengths -- so the rewrite into
divide-and-modulo is checked rather than asserted.

Behaviour at every power-of-two nary is deliberately unmoved.  new()
keeps the old path verbatim for them, so a 2,000-step trace still runs at
2,048 and not 4,096, and all 87 existing tests pass untouched.  My first
attempt replaced it with "the smallest radix that covers", which is
tidier and changes 939 of the decompositions I sampled; the challenge
tests caught it.

For other b the trace pads to b^rounds and every round carries a full
digit, which is the shape the paper specifies and why it quotes
192^4 = 1,358,954,496 rather than 2^30.

bits_nary is now a ceiling, so a base-192 digit is eight bits plus a
range check -- digit_needs_range_check, false for every power-of-two
radix, so nothing existing grows a check it did not have.  That check is
not yet emitted on chain; this commit is the arithmetic.

98 tests.
A dispute makes two setup-time choices about its search and they are
independent.  SearchMode decides who sends what and when.  The medium
decides how the hashes a stage sends are carried, and the combination
worth having is full duplex with a published dissection.

  publication: committed | op-return | annex

Committed is the default and is what every existing definition means, so
nothing moves: an_existing_definition_is_unchanged pins that a YAML
saying nothing about publication still parses to the asymmetric search
with committed hashes and no dissection size at all.

The choice is what sets the branching factor, which is why it belongs
beside SearchMode rather than inside it.  A committed dissection is READ
by Script, so the verifier for a one-time signature is written out in the
witness and a tapscript may hold a thousand stack elements -- eighty-
eight per twenty-byte digest at the deployed Winternitz parameters, so
twelve hashes whatever the fee.  A published one is SEEN: BIP341 strips
the annex before the stack is formed and an OP_RETURN output is never
executed, so it costs no stack at all.

The medium carries what the graph will need of it: weight_per_byte, one
for a witness and four for an output; relayed_by_default_policy, false
only for the annex, so that a deployment condition is visible in the type
rather than in a comment; and dissection_bytes on the definition, twenty
bytes a hash and nary-1 of them.

At nary = 192 that is 3,820 bytes a stage over four rounds, which is the
shape the paper prices.  This commit is the choice and its arithmetic;
the graph does not read it yet.

107 tests.
This is what replaces last_step.  There, a read carried the number of the
step that last wrote its address, and a challenger who disputed the claim
opened a second search to that step; six branches -- FutureRead, ReadValue,
ReadSection, WriteSection, Uninitialized and Initialized -- exist to police
that pointer.  Here a read carries a Merkle path and the dispute is a hash
check against a root the machine state commits to.

The tree is sparse and binary over four-byte words, SHA256(w)[0..20] at a
leaf and SHA256(l || r)[0..20] above, with the empty subtree precomputed
per level so untouched memory costs nothing.  verify_step checks the reads
against the pre-state root and the single write against the post-state
root from one proof: a write changes exactly one leaf, so the same
siblings carry both roots and no second proof is needed.

The depth is a parameter because the paper's is not the deployed one.
Definition 8.1 puts the tree at depth 18 over 2^20 bytes; BitVMX puts its
register file at 0xF000_0000 and its stack at 0xE000_0000 with 8 MB under
it, so a tree over a contiguous megabyte cannot address the registers at
all.  DEPLOYED_DEPTH is 30, one leaf per word of the 32-bit space, and
PAPER_DEPTH is kept because the published figures are quoted at it.

Twenty-one tests.  Four are vectors from compact2/python/memory.py -- the
zero leaf, the empty root, a populated root and a post-write root agree
byte for byte, which is what makes the two implementations one
specification rather than two constructions of the same shape.  One
measures the deployed depth: 89 siblings and a 92-node working against 52
and 55, which is 740 more proof bytes and, because 181 nodes still pack
into the same 64-element FlatMerkle file, no additional published bytes --
with 11 nodes of margin out of 192.

definitions declared serde's derive feature, which it uses and had been
getting from another crate's unification.

Workspace green: 140 + 21 + 107 and four integration tests.
Program now holds a MemoryTree over the 32-bit word-addressed space and
keeps it current as it runs.  write_mem updates the leaf it wrote, and
register writes go through a new Program::set_register that does the same
-- registers are memory-mapped, so a register write is an ordinary memory
write as far as the tree is concerned.  Thirteen call sites moved from
program.registers.set to it.

The tree is not serialized into a checkpoint.  It is a pure function of
the section data and the register file, so deserialize_from_file rebuilds
it, and a test pins that: a Program round-tripped through JSON comes back
with an empty tree and rebuilding recovers the exact root.

Loading builds it in bulk.  An ELF writes every initialised word of every
section and those words are contiguous, so they share nearly all their
ancestors; bulk_write rebuilds each affected level once instead of walking
a full path per word.

The authoritative contents are the sections plus Registers, not the
registers *section* -- find_section refuses that one, so its data is never
written and would have built a tree of zeros where the register file is.

What guards all of this is memory_tree_is_consistent, which rebuilds from
memory and compares.  the_tree_tracks_execution_step_by_step runs to every
limit from 1 to 23 and checks it at each, so a write path that forgets the
tree shows up there and nowhere else.  A register also opens as an
ordinary leaf: its multiproof verifies against the root.

Also fixed a test helper that swallowed the error it should have failed
on -- an unknown input section makes execute_program return before the
first instruction, which looks exactly like a program that changes no
memory, and did for one draft of these tests.

emulator green: 111 lib tests and every integration suite, the RISC-V
compliance run included.  definitions 24.
The endgame opens a read against a root.  For that to mean anything the
root has to be determined by the state the search agreed on, so a step's
hash now commits to the memory root that step produced:

    h[t+1] = Blake3(h[t] || write || pc || micro || root[t+1])[..20]

TraceStep carries the post-state root as an Option.  Under LastStep it is
None and to_bytes is the thirteen bytes it always was, so every existing
hash is unchanged and the script that checks them still hashes 33.  Under
Merkle it is thirty-three, and the script's message becomes 53 --
(40 + 8 + 8 + 8 + 2 + 40) / 2 in nibbles, which is the next commit.

MemoryModel selects between them, on the Program and in the program
definition's YAML, defaulting to last_step.  An existing definition means
exactly what it meant before; `memory_model: merkle` asks for the compact
endgame's model.  This is the same seam search_mode and publication use,
which is what lets a verification backend be swapped without the
applications above it noticing.

The initial root is deliberately not chained: the first state hash
predates any step, and step zero's state is agreed by construction --
both parties load the same program and the same input, and setup commits
to both.

Five tests.  The ones worth naming: a Merkle step's root equals the
program's root after that step; the two models produce different state
hashes for the same execution, which is the whole point; and the message
grows by exactly one root with the original thirteen bytes untouched.

emulator 116 lib tests and every integration suite, compliance included.
definitions 24.
The chain never checks a whole multiproof.  At the deployed depth one
instruction's proof carries 89 siblings, and recomputing that in Script
would be absurd.  The prover publishes the proof and the working it
drives, and only a node someone actually disputes is checked -- so the
on-chain cost of the memory argument is one of these scripts.

memory_node_challenge asks whether a claimed parent is H(left || right);
memory_leaf_challenge whether a claimed leaf is H(w).  Both are naysayers,
succeeding when the published working is wrong, as trace_hash_challenge
does.  Measured: 71,040 script bytes, peak stack 659.

The hash is blake3, and that is forced rather than chosen.  Bitcoin has no
OP_CAT, so a Merkle path cannot use the native OP_SHA256 -- there is no way
to build a parent's preimage from two children on the stack.  The hash must
be written in Script over nibbles, and then what binds is the
thousand-element stack limit, not weight:

    SHA-256 over 40 bytes   213,758 B script, peak 979
    blake3 -> 20 bytes       69,872 B script, peak 659

SHA-256 reaches 979 before anything else is on the stack.  A node check
must also hold the claimed parent, forty nibbles, so it is 1,019 and the
script aborts with StackSize instead of deciding.  A const assertion
records that.  blake3 leaves 341 spare, emits twenty bytes natively rather
than thirty-two needing truncation, and is already BitVMX's Script hash, so
both share one set of lookup tables.

definitions::memory_tree still hashes with SHA-256, per the paper, so the
two do not yet agree; the_rust_tree_and_this_script_do_not_yet_agree
asserts that gap so it cannot go latent, and fails the moment it closes.

Two tests were green for the wrong reason before this landed.  The first
version used SHA-256 and every honest case "passed" because the script was
aborting on StackSize -- a broken script is indistinguishable from an
upheld node unless the reason is checked.  Runs now return Refuted, Upheld
or Broke(reason), and Upheld must be a decided OP_VERIFY failure.

Workspace green: 151 script, 24 definitions, 119 emulator, every
integration suite.
The garbled RV32M circuit refuted honest signed divisions with a zero
quotient and a negative dividend, because it added a 32-bit remainder to
a 64-bit product and the add truncated.  Script verifies division a
different way, so the question was whether it shared the defect.

It does not.  div(-1, 2), div(-7, -9), div(-5, 3), the exact operands
that broke the circuit, and i32::MIN+1 over i32::MAX all pass.  That is
what fuzz_div_rem was already covering without anybody noticing: random
i32 pairs land on |n| < |d| with n negative about a quarter of the time,
so Script has been exercising the case all along.

Which means the two backends did disagree about a valid instruction, and
only one of them was wrong.  Both sets of cases are now named rather than
reached by chance, on both sides.

The emulator's division tests never reached it either -- they cover
divide by zero, overflow, and four non-zero quotients -- so five cases
are added there too.
So the garbled step circuit can be checked against the same 47 ELFs the
emulator and the Script verifier already are.  13,367 instructions from
47 programs, with the operands, the write, the next pc, the division
witness and the memory-access kinds.

Ignored by default: it feeds protocol-builder's compliance_circuit_test
rather than asserting anything itself.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant