Repository navigation
Conversation
Add docs/plans/2026-10-shared-recovery-tool.md. Its first version checks a machine the operator trusts, runs a best-effort flush, replays the machine to the stop block C, exports its state, and imports it. The queued directs cross the rebuild as an input-index floor F, which replaces the terminal drain. The plan records the owner's decisions of 2026-10-29 and the open ones. Fix the docs the design session found wrong or incomplete: cockroach recovery, scheduler semantics, the application contract and history guide, snapshot lifecycle, the Cartesi Machine facts (Dave epochs, hash-tree checks), the watchdog README, runbook and operator guide, the threat model, the tracks board, and the register (fridge bound, proposed canonical rules). Add two watchdog to-dos: stop calling verify_hash_tree, and tie the watchdog to the validators. Correct two setup rustdoc comments.
The tool is now `segment a b`: from L1 and the scheduler library alone, it folds the scheduler automaton (batch nonce, fridge, frame-clock floor) and emits the candidate L2 stream for the inputs in (a, b], grouped by the emitting input's block, plus the automaton's state at b. It needs no machine, nonces, application state or code. The sequencer runs the stream from a state projected from a canonical machine, and each client from a backup at an accepted checkpoint's exact executed-input count. Record the owner's next-template rules (fridge bound, monotone frame clocks, authorization decides inclusion, direct context, scheduler query), mark the matching scheduler-semantics rules pending, and point the docs that link the plan at its new sections.
Write the tool's interface as ML signatures: one segment per consumer, each from its own base, and the scheduler state at C for the sequencer only, so a client never holds the fridge. Bases become InputBox positions, which every state source already names, so Dave's machine needs no replay to a block end; Dave's machine stays the sequencer's default. Explain what the executed-input count is for, and base each client on a watchdog agreement, the only point that maps a base to its history with a comparison behind it. The watchdog must log every agreement with the sequencer's count, era and generation. Add the client backup rules (every commit writes the count and keeps its rows, a cut discards later snapshots, snapshot anywhere), notes on snapshotting large SQLite databases, and the playbook after an incident. Bound client checkpoints by the agreed head in the current cockroach procedure too.
The tool no longer folds from genesis. It reads three values from the base machine's scheduler query (position, batch nonce, scheduler clock L), reads L1 blocks (L, C] once, and splits them at the machine's position: below it, batches are dropped unjudged and well-formed directs rebuild the fridge; from it, the canonical scheduler runs from the machine's own state. The sequencer executes the candidates, and each client executes its part of the executed list from its own count; the machine must be no newer than any client's base. Phase 1 is exact under three next-template rules the owner settled on 2026-11-02: strict frame clocks, whole-block overflow drains (the bound holds one block of directs), and a scheduler clock that follows every drain. Keep the application's effective block separate from L. Record them in the register, mark the matching gate-d behaviors pending in the scheduler semantics, and update the plans that summarize the tool. The plan now opens with the model and moves caveats and the SQLite notes later. One open decision proposes L' as the direct floor in place of the index F settled on 2026-10-29.
Settle the current plan's direct floor at the block L' (in its own era field; base_safe_block stays C), and link the new candidate from its status. Add docs/plans/2026-11-recovery-from-a-machine.md, a candidate that would replace the shared recovery tool plan after a re-evaluation of every decision in it. The rebuild runs our scheduler with the application from a canonical machine over the fresh data directory's input table, and records the executed list; one base serves every consumer, each client checking its comparison digest against the machine. Because it reads by index, the machine also reports its fridge start, which drops strict frame clocks, whole-block overflow, the backstop raising L and the L' floor. It also drops 'authorization decides inclusion' and the effective block, and notes that the lane must drop malformed directs. A table lists each difference and marks the settled decisions it reopens.
Add docs/plans/2026-11-sequencer-recovery/, a third plan in eight files that would replace the shared recovery tool plan and its candidate. Only `sequencer setup --recovery <machine>` remains. Setup reads the machine's counts and scheduler state, the fridge included, through paged inspect queries; the scheduler alone rebuilds the history from L1 into a new table; the application executes it during setup; and clients realign through the existing feed from their own count. Reading the fridge from the machine drops phase 1, the effective block, strict frame clocks, whole-block overflow and the backstop's clock raise. The plan drafts definitions for the real docs (the L2 transaction sequence, what counts, batch validity), lists what an application must provide, outlines the interfaces in Rust, and lists the changes its review found that implementation must carry through. Point the two earlier plans' status lines at it, and note in the register which next-template rules it would drop or reopen.
Split the plan into three layers: the shared software (scheduler.md states every check and gate for the next template, with the new rules marked; application.md outlines the machine's state export and the trait additions), operations (operations.md: one setup from any base, and the playbooks), and the design (sequencer.md, todo.md). Record the owner's 2026-11-05 decisions: an underfunded op executes and counts, blocks are monotone through a scheduler clock and strict frames, overflow drains whole older blocks, and setup never refuses on intent. Archive the two earlier recovery plans in docs/recovery/history/ and repoint the docs that cited them.
A read-only review (33 agents, 25 findings, 6 survived, 4 distinct, all minor) found: rebuilt direct rows lacked a batch nonce, so a client's row check would reject canonical states; setup on a completed directory was unspecified; the paged state export broke the watchdog's inspect query; and the playbook never cleared the watchdog latch and read every application error as a canonical halt.
A completed or failed setup leaves state nothing can vouch for, so setup refuses it loudly without touching it, and a failed setup's directory is discarded. todo.md notes what this costs: today's retry in place and the idempotent no-op go.
Each choice already lives in the file that owns it, with its reason, and every open item is in todo.md or marked Open in scheduler.md. A dated table of rulings without reasons only invites dogma and drifts from the design.
A check that protects the user is consensus and skips the op; a check that protects the operator is admission and only the lane makes it. scheduler.md draws the matrix, and interfaces.md outlines the scheduler and the application in Rust and C around it: shared code compares the nonce the application stores, and admission is its own call. The scheduler no longer runs application code: a payload cap replaces the direct filter. The machine exports where its fridge starts instead of the fridge, since the fridge is a queue over L1, and setup checks that export against its input table. Strict frames, the backstop's raise and whole-block overflow stay. Only gate f is needed for monotone blocks; the three keep the clock's split of the directs, which setup and the conformance vectors now check. The two reasons given for executing an underfunded op were wrong and are replaced.
The canonical scheduler is now a concrete `Scheduler` in sequencer-core that needs no application: it turns L1 inputs into L2 transactions (`L2Tx`, a direct or a user op), and one shared `execute` runs each of them for the guest, the lane and catch-up. The `Application` trait keeps what the application stores and applies apart from what shared code decides: shared code compares the nonce, and admission is its own call. This is a new template. Frames are strict, a first frame below the clock is rejected, and the fridge has a bound and a payload cap, so a direct over the cap never executes. A user op whose fee cap is below the frame's price, or whose nonce is wrong, is skipped. An underfunded op is charged what its sender has and executes; only the lane refuses it, at POST /tx. The machine answers a `scheduler` inspect query with its 48-byte export. Every crate runs on the new core. The wallet's state loses its clock (84 to 76 bytes), the C application ABI is version 2, and the lane and startup recovery share one storage predicate for the directs that execute. `setup --recovery` and the fold replay are removed; recovery from a machine arrives with `setup --base`. The five golden guest cases of the earlier template's vectors are ignored until this template has its own.
One `setup [--base <MACHINE_DIR>] [--dry-run]` starts every era, from genesis or from a stored canonical machine. It reads the base's export, flushes the submitter's wallet to pin a stop block, syncs L1 through it, checks for an escape, checks the export against the input table, rebuilds the history with the scheduler alone, executes it, and publishes the era in one transaction. docs/recovery/cockroach.md is the procedure's home. Setup now takes the submitter key (not for --dry-run) and refuses any non-empty data directory, so it is no longer idempotent. Every retryable L1 check runs before the first write. Storage gains the `rebuilt_history` table and, on the era, the rebuilt history end and the first queued input. Setup opens no root frame: run's first frame drains from the first queued input. The era baseline is the first comparison checkpoint, for a genesis base too. The feed serves the rebuilt rows, then the live rows, and a client of an older era adopts the new one at its own count (`HistoryInfo::adopt_claim`, the SDK's `adopt_current_era`). `/history`'s baseline gains a required field, `rebuilt_history_end`. The sequencer links the emulator to read a base machine (`sequencer::machine::read_machine_export`). The release workflow therefore builds natively instead of with cross, and released binaries need the Ubuntu 24.04 or Debian 13 C++ runtime, libslirp0 and libgomp1; that workflow has not run on GitHub Actions yet. bindings/c-scheduler adds scheduler.h, a proposal for the C port, with the reference export behind it.
consensus_vectors_v2.json replaces the v1 file for this template. Its byte-level sections are v1's, unchanged. Its guest section has 41 scenarios with the `scheduler` report after every input, and a scheduler-only section covers overflow and resume. A native test, the guest in the machine and a replay through scheduler.h each hold it, and no test is ignored any more. The devnet drills recover from genesis and from a stored machine, on the Rust host and the C host. The stored base holds a queued direct, the new era's root frame drains the queued directs in order, and a fresh watchdog agrees at the baseline and again after the era's first accepted batch. The watchdog's machine tests pass on the new guest and establish which stored machines are a valid base: checkpoints always, the others only when the machine waits between inputs. The base loader's failure arms are tested behind a private seam, with no change to what it does. docs/review/2026-11-10-recovery-measurements.md records the rebuild's rate, the `state` report's 2 MiB limit and the cost of draining a full fridge.
The record was written before its tree had a commit. The harnesses were temporary and are not in that revision; the record gives their method.
`ExecutedInputCount` is `L2TxCount`, `Application::executed_input_count()` is `l2_tx_count()`, and `Executed::Done` and `TxFate::Done` carry `index` where they carried `offset`. Every other Rust name for the quantity follows: `next_l2_tx_count`, `base_l2_tx_count`, and `l2_tx_index` for one transaction's place in the history. Panic and error messages say "L2 tx count", and the docs use the one name. Rename only: no behavior changes. The HTTP fields, the `X-Executed-Input-Count` header, openapi.yaml and the storage columns keep `executed_input_count`, and the Rust structs that serialize to those fields keep it with them; the plan's todo lists them under the egress redesign. The guest's binary holds these names, so both template hashes move.
The register quoted the two template hashes of one local build. Any source change in a guest crate moves them, and the rename just did, so the entry now says where to read them instead.
The projection contract's worked trace becomes a rebuild test. A malformed batch consumes no batch nonce while its backstop executes the overdue directs, and the last frame drains the direct at its own safe block, so the era has seven transactions, next batch nonce 2 and first queued input 8.
Cockroach recovery rebuilds the history itself, and each client realigns through the feed from its own count, so nothing needs the era's raw L1 prefix from the sequencer. This removes what the tracks board listed for deletion once the recovery landed: - `GET /historical-l1-inputs`, its storage read, its response limit and its OpenAPI entry; - the SDK's `historical_l1_inputs` and the `HistoricalL1Input*` types; - the main `historical_bootstrap` test, whose worked trace the rebuild's tests now hold; - the projection contract and the Track 3 integration plan, whose deployment follow-ups move to the register. `/history` is unchanged. The checkpoint compatibility test stays, as integration_tests/recovery_compatibility.rs, with the fixtures it shared with the deleted test.
The first complete CI run of this work passed, with the six recovery scenarios and the divergence drill in its rollups-e2e job. Only the release workflow is still unverified on GitHub Actions, since it runs on a tag or by hand.
…h setup's time `run` logged the latest terminal fault at WARN unless it predated the completed setup, on the idea that a later setup had resolved it. Setup now runs once, in an empty directory, and a failed setup leaves a directory to discard, so every fault in a set-up database was recorded after its setup completed. The comparison could only return "resolved" after a backward clock step, which hid a real fault. This removes `fault_is_news`, its debug arm and `Storage::setup_completed_at_ms`, whose only caller it was.
Setup ran a load-or-insert through a second connection and refused a database that held state without an identity. Setup creates its database in an empty directory, so the identity is never already there and no state can precede it: both branches were unreachable. Setup now inserts the identity on its own handle with `Storage::pin_deployment_identity`, and the row's key refuses a second one. `Storage::has_persisted_deployment_state`, `ensure_deployment_identity` and `IdentityError::OrphanedState` go.
…tained The input reader passed `FrontierMode` and the batch submitter on every append. Both are facts storage already holds. Setup's syncs deferred the frontier because no era existed yet, and `run` populated it because one did, so the flag was "has the era been published". The submitter always equalled the pinned deployment identity's. A deferred append on a published era would have skipped the content-identity check, and the empty frontier would then have made startup flush and cascade over an undetected divergence. That call can no longer be written. `populate_safe_accepted_batches` now returns before the era exists and reads the submitter from the pinned identity, failing loud when it is missing. `FrontierMode`, `InputReader::set_frontier_mode` and the reader's submitter field go. The test append helpers keep their submitter argument as the fixture's identity pin and assert it against an existing one, which exposed six lane tests that pinned one address and appended with another.
The schema guarded 23 columns with `typeof(x) = '…'` checks, so that a mis-bound value could not reach trigger arithmetic. About 80 other columns had none: `frames.safe_block` accepted text, and text in `wallet_nonce_watermark.watermark` would have sorted above every integer in the upsert's `MAX`. Every table is now STRICT, which applies the same column affinity and then requires the storage class, for every column, before any trigger runs. The guards go. The count had four column names. `application_inputs.offset` and `snapshots.executed_input_count` become `l2_tx_count`, `history_state.base_executed_input_count` becomes `base_l2_tx_count`, and `history_generation_cuts.preserved_input_count` becomes `preserved_l2_tx_count`. Wire names are unchanged: every read is positional, so no JSON field or header came from a column name. A STRICT refusal is `SQLITE_CONSTRAINT_DATATYPE`, which the existing classifier already treats as persistent; a test pins that, and another that no table is left non-STRICT. The feed's two corruption tests can no longer store a wrongly typed value, so one deletes the frames and the other strips STRICT from its table first. An operator's `sqlite3` must be 3.37 or later to open the file.
`publish_era` wrote one fact, that this era exists and where it starts, into four places in one transaction: `history_state`, the `batch_tree_anchor` row (seeded 0 by the migration, then updated), `setup_complete`, and the baseline snapshot. `history_state` also held the one mutable field, the recovery generation, which always equalled the greatest key of `history_generation_cuts`. `setup_complete` had no immutability trigger, so deleting its row un-completed a database. The `era` table now holds the id, the base count, the rebuilt history end, the stop block, the first queued input and the anchor batch nonce in one row that no statement may update or delete. A database is set up exactly when that row exists. The recovery generation is the greatest cut: a cut cannot be inserted without an era or out of sequence, and a cascade advances the generation by inserting its cut and nothing else. Rust reads the row through one `Era` (and publishes a `NewEra`) in place of `HistoryState` and `EraRecord`, which named and typed the same record differently. Every reader of the anchor now fails when the era is absent, where the seeded row used to answer 0, and each trigger that compares against the era uses `IS NOT`, so a missing era refuses the write instead of passing it: no batch can be rooted before the era. The migration's genesis era, which only crate tests reach, is compiled under `cfg(test)`, since an era row now means a set-up database. Tests that need another era open one with `Storage::open_with_era`.
Setup stored every transaction the scheduler handed out with no count, then read the rows back, executed them and updated each executed row's count. That second pass is why `rebuilt_history` had a nullable count, a partial unique index, an `(l1_input_index, position)` key and a trigger that let a row gain its count once, in order and without gaps. After setup only the feed reads the table, and only its counted rows. Setup now executes each transaction as the scheduler hands it out and stores the executed ones, keyed by their L2 tx count. A user op the executor skips has no row. The scheduler still never reads the application: only the loop changed, and `scheduler.h` already describes feeding one input and pulling its transactions. The table's key is the count, its rows are immutable and cannot be deleted, each count follows the last, and directs arrive in input order. `record_rebuilt_counts`, `rebuilt_history_after`, `RebuiltKey` and `RebuiltRow` go. The stored rows used to witness that every direct the scheduler drained had executed. They hold only what executed now, so setup checks them against the input table after the loop: the stored directs must be exactly the admitted inputs between the base's first queued input and the scheduler's final one. A direct that skips is unreachable, and an application error names the transaction, which has no row to point at.
`batches` was shaped as a tree, yet the parent of a new batch was always the highest-indexed valid batch, so the pointer was derivable. The schema did not forbid what the shape invited: the nonce trigger never checked that the parent was valid, and the nonce index was not unique, so two valid batches could carry one nonce and every lookup by nonce, the content-identity check included, would then pick a row arbitrarily. `parent_batch_index` goes. A new batch carries the last valid nonce plus one, or the era's anchor when no batch is valid, and a unique partial index allows one valid batch per nonce. With neither a valid batch nor an era the insert aborts. One trigger states a batch row's lifecycle in place of five: it changes only by sealing once with its hash while valid, or by being invalidated once. It also refuses what nothing refused before: sealing an invalidated batch, sealing and invalidating in one statement, and changing `batch_index` or `created_at_ms`. A CHECK ties the payload hash to the seal. Invalidated rows stay, write-once marked, and the cascade is unchanged. Dead branches remain recoverable from creation order and nonces.
"Accepted nonce N means our valid batch N" lived in a side table, `safe_accepted_batches`, joined back to the batch by nonce. A row was only ever inserted after the content-identity check found a valid closed batch at that nonce, so each row was one batch's fact. Two rules were held by Rust alone: an accepted batch is never invalidated, and the accepted batches are a prefix of the valid path. Had a cascade reached an accepted batch, checkpoint selection would have picked the replacement batch's snapshot through the nonce join. The input reader now records an acceptance as `batches.accepted_input`, written once. The lifecycle trigger admits it only on a sealed valid batch and refuses to invalidate an accepted one; a second trigger makes each acceptance follow the greatest accepted nonce, at a later input; a batch cannot be inserted already accepted. The divergence freeze on `batches` now covers acceptance, where the accepted frontier used to rely on its single writer. The scan, `scheduler_accepts`, the content-identity check and the divergence marker are unchanged. Readers find the frontier through the column's unique index, and the cascade pivot keeps its seek by nonce. `first_frame_safe_block` had no reader, and the inclusion block is the accepting input's block, as it always was. The module is renamed `storage/acceptance.rs`.
A snapshot was two rows: `dumps` held the artifact path and a durable `lease_count`, `snapshots` held the boundary. The lease count was the only mutable column, written by every download through a brief writer connection and reset at startup, and releasing it needed a scheduler, a reporter and the database path carried in every `Storage`. `snapshots(snapshot_id, batch_index, l2_tx_count, prefix)` is now the one immutable row per artifact. Leases are per-snapshot counters in a `SnapshotLeases` registry that `run` shares between the egress handlers, the lane's GC and startup hygiene. A lease means something only while this process serves a body, and only `run` serves and collects under the data directory's process lock, so nothing durable records one. The registry follows one protocol. Acquire loads the schema on its read-only connection, locks the registry, selects in one deferred read transaction, increments the counter and unlocks. GC begins its IMMEDIATE transaction, then locks the registry, deletes only unleased candidates and unlocks after the commit returns, so no acquire reads a row GC is deleting. Release is the guard's drop: a decrement, no lock, no I/O. `collect_snapshots` returns the collected prefixes for the caller to delete after the commit (I13). The egress handlers open read-only connections. A database file that is missing at request time is therefore a 500 from the open, as on the history endpoints, instead of a terminal fault from an empty schema. Removed: the release scheduler and reporter, the startup lease reset, the HTTP supervisor that drove releases, `Storage.path`, and the tests of that machinery. HTTP statuses, headers and bodies are unchanged.
…vate `WriteHead` was `Copy` with eight public fields and three public mutators. Only storage advanced it, but any module could build or edit one, and a head that disagreed with the durable Tip was caught only in part: a wrong batch met the Tip triggers, while an older frame, a position past the frame's rows or a fee that differed from the frame row met nothing. The fields are private behind read accessors, the mutators are private to storage, and the only constructor outside tests is the loader. The head carries the open batch's nonce, so the lane's snapshot path reads it from the head instead of its own read transaction, and `close_frame_and_batch_with_snapshot` opens the successor frame at the head's safe block instead of taking it as a parameter. Storage reloads the Tip and compares the head's persisted facts (batch, nonce, frame, both counts, fee, safe block) at every frame rotation and batch close; a disagreement panics with both heads printed. The size target is left out, since a reload samples the policy at reload time. Storage stops importing the lane's types: `append_executed_user_ops_chunk` takes an iterator of a borrowed `ExecutedUserOp` view. `storage` is `pub(crate)`. No other workspace crate named it, and the types that appear in public signatures stay public inside it. `Storage::read` and `Storage::write` are private to storage; tests outside it use `raw_read` and `raw_write` from the test helpers. The compiler now reports storage items with no production caller, so four accessors and two fixtures that only tests use are compiled under `cfg(test)`. No schema, wire or lane-ordering change.
`Storage::open` runs the migrations. In `run` it was called by the lane's handle and three times per reader tick, and the first call of a run happened inside startup recovery's reader sync, after recovery's inspection had already opened the database. `run` now migrates once, immediately before startup recovery. Only the black-box warning, the lifecycle preflight and the identity load open the database earlier, read-only; the preflight must come first because it refuses a missing or half-set-up database with its own error. Recovery, the lane's handle and the reader's cursor read and append open without migrating. `setup` is unchanged. Four sites that opened a writer only to read now open read-only: the reader's storage preflight, recovery's inspection, and the watermark reads of startup recovery's flush and of `flush-mempool`. Each runs under the process lock before any worker starts, so no writer can turn its 50 ms busy timeout into a failure. The reader's per-tick cursor read keeps a writer handle for the 5 s timeout: the lane commits beside it, and a BUSY there would end the reader.
One SQL predicate decides whether a safe input executes as a direct: the sender is not the batch submitter, and the payload is within the application's cap. The submitter came from the pinned `deployment_identity` row; the cap was passed by every caller through 21 signatures, always with the value `A::scheduler_limits().max_direct_payload_bytes`. `deployment_identity` gains `max_direct_payload_bytes`, which setup pins with the rest of the identity (the dry run too, in its scratch database). Each statement that uses the predicate loads both operands from the pinned row through one helper, `DirectOperands`, so neither is bound alone, and a missing identity still fails loud. The parameter is gone from storage, the lane, startup recovery and setup's export check. `run` refuses a binary whose cap differs from the pinned one, through the existing identity mismatch (terminal, exit 30) naming the field. That is the only new refusal: storage reads its own predicate's operands from one row, and this does not compare the binary with the deployed image.
Several storage results were unnamed tuples, and the local `batch_index`
left storage as a bare `u64` beside batch nonces, also `u64`.
`DangerStatus::CanonicalDivergence` carries a nonce while the three
`*InDanger` arms carry a batch index, and the doc of
`DangerStatus::batch_index()` called its value a nonce. No production code
mixed them; the types now keep it that way.
- `Divergence { nonce, safe_input_index }` is the divergence marker.
- `FeeQuote { frame_fee, recommended_fee }` is the `/fee` quote, read from
the Tip's latest frame and one policy read instead of a whole
`WriteHead`. The response is unchanged.
- `SubmitterInput { safe_input_index, block_number }` is what
`first_batch_submitter_input_after_block` returns.
- `BatchId` is the local batch key wherever it leaves storage: the head's
accessor, `PendingBatch`, the `*InDanger` arms, `RecoverTip`, the expected
batch of the aging-Tip recovery and the cascade's invalidated list. It has
no arithmetic and no constructor outside storage and tests, and it prints
as the bare number, so logs and error texts read as before. Inside
storage, helpers that bind SQL keep the raw integer.
`docs/recovery/README.md` names a local batch and a batch nonce apart in the
danger table.
… use a set-up database Storage's production files carried items only tests use, the schema migration carried a hook that one test of the test-only genesis insert needed, and storage's own tests got their database from a create-on-open path production cannot reach: a schema and a genesis era, with no identity and no baseline snapshot. - The test-only reads (`log_gas_price_updated_at_ms`, `current_safe_block_timestamp`, `last_safe_progress_ms`) and the other test-only items (`StoredSafeInput`, `NewEra::EMPTY_GENESIS`, `BatchId::from_raw`/`to_raw`, `SafeInputRange::empty_at`, two `WriteHead` accessors) live in `storage/test_helpers.rs`. - The production migration is plain `M::up(schema)`. The hook parameter is gone; the test-only genesis insert keeps its own hook under `cfg(test)`. The hook's test covered only that insert, so a new test pins the property production has: a failing migration leaves no table and no version. - `set_up_storage` builds a database the way setup does: `initialize_for_command(Setup)`, the identity pin, `publish_era` with its baseline. Storage's tests and the fixtures other modules share use it (75 sites directly, 59 through `open_temp_storage`). Tests that need no identity, no baseline or no era say so and use a lower-level fixture. Tests outside storage that register their own identity or baseline (the lane, startup hygiene, the snapshot endpoints, setup, the reader and the integration tests) stay on create-on-open, so that path and its genesis insert remain, in test builds only. No production behaviour changes.
`append_ingested_safe_inputs_with_timestamp` stamped `l1_safe_head.synced_at_ms` with the wall clock inside storage. The danger check already takes `now_ms` so storage tests need no clock mock, but ten tests rewrote `synced_at_ms` with raw SQL because the append took none. The append takes `now_ms` and stamps it. Its one production caller, the reader's `append_safe_inputs`, reads `clock::unix_now_ms()` inside the same blocking closure immediately before the call, so the stamp is at most the commit time: the staleness arms can only err toward "stale", never mask an outage (I12). The divergence marker's `detected_at_ms` is an audit timestamp no check reads and keeps storage's clock. Every raw `UPDATE l1_safe_head SET synced_at_ms` in the crate's tests is now an append at the wanted time, through a new helper `append_safe_inputs_observed_at`. The raw `block_timestamp` rewrites stay: no append rewinds a block timestamp.
"The wallet-nonce watermark is raised durably before any broadcast at a new nonce" (I14) was a call order in two places: the poster's `submit_batches`, pinned by a test, and the flusher's `flush_and_wait`, pinned by none. A reordering would let a transaction escape the flush's coverage. `Covered` in `l1/watermark.rs` is a witness with a private field that only `cover(sink, highest_wallet_nonce)` mints, after `raise_to` succeeds. The poster's `send_batch_at_wallet_nonce` and the flusher's `submit_noops` take `&Covered` and assert each wallet nonce they send is at or below it, so a send before the raise does not compile and a send above it panics. The flusher sends `[latest, max(pending, watermark + 1))` and raised exactly when that range is non-empty, so it always holds a witness when it sends; its no-op call moves inside that branch. Which nonces are sent, when, and what the flush waits for are unchanged, and no raise is added. New tests: the flusher raises before its first no-op (a failing sink leaves the pending nonce unmoved), and a send above the witness panics on each side.
The standard recovery saves no confirmation, cannot finish before the outage's cause is fixed, and fires hours into an incident that has already paged everyone. It is also the most complex part of the sequencer, and every client has to mirror it. The plan deletes it. The sequencer stops for good once the oldest batch L1 has not accepted can no longer be accepted, which is a proof from the safe head, and cockroach recovery is the remedy. The page gives the scenarios, the operator's and the client's playbooks, what goes and what stays, the order of work, the drills, and what is still to decide. It also moves the staleness window from a crate constant to the application's scheduler limits, and takes the recovery generation off every client interface.
`MAX_WAIT_BLOCKS` was a constant of `sequencer-core`: 1200 blocks for every application. The window is how long an application lets a sequencer hold a direct back, and how long operators have to fix an outage once the sequencer no longer repairs itself, so it is the application's to choose. `SchedulerLimits` gains `max_wait_blocks` beside the fridge bound and the payload cap, and the static `Application::scheduler_limits()` returns all three. `SchedulerConfig`, in Rust and in `scheduler.h`, already took the window as a field; `for_application` now reads it from the application. The crate constant is gone. The wallet keeps 1200, so no rule changes. The C engine's limits record gains the field. Its ABI is version 3, so an engine built against the older header is refused instead of reporting a window of zero. Setup pins the window with the deployment identity, as it pins the payload cap, and `run` refuses a binary whose window differs: it would judge acceptance by another rule than the one the data directory was built under. The schema changes in place, so an older data directory needs a new setup. The pin ties a binary to its data directory, not to the deployed template: a machine's scheduler export carries no constants. The guest crates changed, so both template hashes move. Verified on rebuilt images: 990 workspace tests, the in-machine suite 27/27, the watchdog machine tests 30/30 and the devnet suite 53/53.
Both numbers go to the logs and to `/healthz`; a metrics endpoint on the internal listener is later work. The two TLA+ models are archived, not deleted. Setup's flush stays best effort and does not read the old directory: cockroach recovery is the procedure for when that directory is lost or cannot be trusted. Step 1 is marked done. The wallet's window and the batch's key remain open.
The flusher sat in `recovery/` because startup recovery was its first caller. `setup` and `flush-mempool` call it too, and they keep it when the standard recovery goes. It imports only L1 modules. A move: `recovery/flusher.rs` becomes `l1/flusher.rs`, with its imports.
The standard recovery stopped the sequencer short of the staleness window, flushed the submitter's wallet, cut the history back to the accepted batches and resumed in the same database. It saved no confirmation, could not finish before the outage's cause was fixed, and was the most complex part of the sequencer. This removes it. The only recovery is the cockroach recovery. A history is void once the oldest batch L1 has not accepted can no longer be accepted. With every input through the safe head H judged and that batch's first frame at safe block s, any landing still to come is at H + 1 or later, so H + 1 - s >= max_wait_blocks means the scheduler skips it wherever it lands, and every later batch then fails the nonce gate. It is a proof from observed L1 and it stays true, so it is derived from the database on every read and never stored. `run` checks two terminal facts, a recorded divergence and a void history: before any L1 call, again inside the transaction that opens a batch, while running (the landing watch, every two seconds), and on a clean shutdown. A start exits 30; a running process aborts. Startup is check, sync, open a batch if none is open: nothing is repaired, flushed or replaced. A provider failure of the first sync falls back to the stored view only when a batch is already open. The first `run` of an era exits 20 instead: a batch opened then would take its first frame from a safe head that may be hours old. `/healthz` and a `landing lag` log line report `landing_lag_blocks` (H + 1 - s, void when it reaches `max_wait_blocks`) and `safe_head_age_seconds`, with no threshold in the sequencer. Storage reads the window from the pinned deployment identity, as it reads the payload cap, so `ProtocolTiming` is gone and the off-chain acceptance check is a free function. Removed: the danger detector with its margin, wall-clock estimates and stale-view gate; the flush at startup, the cascade and the open-batch repair; the admission witness; exit code 10, which stays unused; `--preemptive-margin-blocks` and `--l1-read-stale-after-blocks`; `l1_safe_head.synced_at_ms` (the schema changes in place). The two TLA+ models of the mechanism move to the archive, and `docs/recovery/README.md` becomes the page for batch landing and outages. Batch invalidation as data and the recovery generation on the wire remain. Nothing writes either any more, and the next two steps remove them. The devnet suite goes from 53 scenarios to 43: the 13 that exercised the repair are deleted, nine are rewritten, and three are new (batches held back and released inside the window, a foreign batch at our nonce, the first run of an era with L1 down). Every row of the outage page's scenario table has a drill. Verified on rebuilt images (the template hashes move): 908 workspace tests, the in-machine suite 27/27, the watchdog machine tests 30/30 and the devnet suite 43/43.
Nothing invalidates a batch any more: a history that cannot land ends the era, and the operator rebuilds in a new data directory. So inside one database each batch is created once, sealed once, accepted once, and never removed. The schema still carried invalidation as data: a write-once `invalidated_at_ms`, a partial unique index and three views that hid invalid rows, and a predicate on validity in most triggers and queries. `batches` loses `invalidated_at_ms`. Its nonce is plainly unique, and a new batch carries the last nonce + 1, or the era's anchor for the first. A row changes only by sealing once with its hash or by being accepted once while sealed. The `valid_*` views are gone; `closed_batches` and `open_batch` name the two readers that remain. The rest of the list is append-only by construction. No batch, frame, user op or application input is ever deleted, and a frame or user op is never updated, so a sealed batch always re-encodes to the bytes its hash was taken from. Every connection sets `recursive_triggers`: without it an `INSERT OR REPLACE` deletes a conflicting row without firing those triggers. No production statement relied on any of this. An index on `application_inputs` that a unique index already covered is dropped. Every query that filtered on validity loses the filter. The lane's next-nonce read for a sender is a plain seek on `user_ops`, without the join through valid batches. The schema changes in place, so an older data directory needs a new setup. No guest crate, wire format or client interface changes. Verified: 904 workspace tests and the devnet suite 43/43.
A batch carried two numbers: a local `batch_index` and the scheduler's `nonce`. They were distinct only while a nonce could name two batches, an invalidated one and its replacement. Since the batches became an append-only list, the nonce is the era's anchor plus the index, and the poster's record carried both side by side. `batches.nonce` is the primary key and `batch_index` is gone, with the separate unique index on the nonce. The tables that point at a batch name it `batch_nonce`. In Rust `BatchId` is gone: a batch is named by its nonce, a `u64`, as `sequencer-core` already names it. Functions that took both lose one, and the lookups that translated one into the other are deleted. No behaviour changes. Log fields and error texts that printed a batch's local index print its nonce; the two are equal in a genesis era and differ by the anchor in a rebuilt one. The schema changes in place. Verified: 904 workspace tests and the devnet suite 43/43.
The recovery generation was a second identity inside an era. It
advanced each time the automatic recovery cut the history, so a client
had to notice the change, roll back and subscribe again. Nothing
advances it since the sequencer stopped repairing itself: inside one
database the history only appends, and the one event a client handles
is a new era. It leaves every client interface and the database.
- A claim is an era id and a count: `HistoryClaim { era_id, next_input }`.
`HistoryVersion` and `RecoveryGeneration` are gone.
- `/history` answers `history: { era_id, available_from, head }` with no
`compatibility`, and takes only `era_id`.
- The WS subscribe query is `era_id` and `next_input`.
- `ERA_CHANGED` names the server's current era as `era_id`.
`STALE_GENERATION` is gone.
- The snapshot routes send no `X-Recovery-Generation`.
- `history_generation_cuts` and its triggers are gone; the schema changes
in place.
- The SDK follows: `history(expected_era)`, and its snapshot claim is
built from the era and count headers.
Both query types refuse a field they do not know, so a client that
still sends a generation gets a 400 instead of being accepted silently.
An older SDK cannot talk to this server, nor this SDK to an older one;
each fails to decode rather than misread.
Inside one database a page that refuses the claim it continues from can
only mean the history changed under the reader, so the feed treats it
as a storage invariant failure instead of closing the socket.
`sequencer-core` is a guest crate, so both template hashes move.
Verified on rebuilt images: 899 workspace tests, the in-machine suite
27/27, the watchdog machine tests 30/30 and the devnet suite 43/43.
An audit of every living document against the code, after the deletion
of the automatic recovery, found the docs accurate except for these.
Three passages overstated setup's flush. It runs on a new database,
which has no wallet-nonce watermark, so it is best effort: it replaces
what the provider's pool holds, its escape check refuses what it sees,
and a submission of the old era that lands later reads as a foreign
batch. The README, I9 and I14 said it settles every slot.
Two gaps. The register's release notes now give the whole upgrade
path: a new setup, the removed flags (a command line that still passes
one exits 2, outside the exit-code contract), the client interfaces,
the C engine ABI, the runtime and the template hashes. And the
application contract says what choosing the staleness window means: it
is the operators' whole outage budget, and `run` refuses one too short
for its batches.
The rest is wording left from the deleted mechanisms ("valid",
"surviving", "admitted", "version-checked"), a snapshot header whose
baseline value was described as 0 where the server sends the era's stop
block, the lane's reading of the divergence marker, and the watchdog's
latch after a void history, which has nothing to clear.
The wallet's window was 1200 blocks, about 4 hours. Since the sequencer stopped repairing itself, the window is all the time operators have to fix an outage before the era ends, and with the default two-hour batch a quiet wallet deployment had under two hours to land each batch. It is now 7200 blocks, about 24 hours. The scheduler's rules do not change; one constant of the wallet's template does. The consensus vectors follow, by the smallest edit. Four of the 41 guest scenarios pin the window's boundary (`stale_batch`, the two backstop scenarios and `empty_batch_at_any_age`): their ten late landings move by 6000 blocks, the one batch payload whose frame names a late safe block moves with them (`drain_7212_batch`), and two expected clocks follow it. Every outcome, handed-out transaction, output and final state is unchanged, and the fixture's three consumers agree: the native reference, the C scheduler binding and the guest in the machine. `scheduler_only` keeps its own explicit window. Two tests that hard-coded blocks around 1200 while reading the wallet's window derive them from it. The worked-boundary rebuild test gains a second stop block right after its malformed batch: the first could not tell a backstop drain from a frame's, and a rebuild without the backstop now fails the new one. Both template hashes move. Verified on rebuilt images: 900 workspace tests, the in-machine suite 27/27, the watchdog machine tests 30/30 and the devnet suite 43/43.
`rollback_snapshot` and the "rollback-safe" checkpoint were named for the automatic recovery, which rolled the application back to them. Nothing rolls back. The snapshot of the latest accepted batch, or the baseline before any, is still what the comparison endpoints serve, what the collector keeps and what `run` requires before it launches, so it is the accepted snapshot. A rename, with its comments.
Its steps are built: the window is the application's, the sequencer stops for good on a void history, the batches are an append-only list keyed by nonce, a history's identity is its era, and the wallet's window is 24 hours. `docs/recovery/README.md` is the home of the behaviour, so the page that tracked the work goes, with its pointers.
`terminal_faults` keeps the cause of a terminal failure with the data directory, for causes that would otherwise be lost with the logs. A terminal fact, a recorded divergence or a void history, is never lost: the database holds it and every `run` states it in full. Whether one was copied anyway depended on where its check sat. A recorded divergence is refused by the lifecycle preflight, before the bracket that writes rows, and left none. A void history is refused inside the bracket, so every refused start appended another identical row, and the next start warned about the previous refusal. A clean shutdown that found either fact wrote one. The recorder now skips a failure that states a terminal fact (`CommandError::is_terminal_fact`), so none of those paths writes a row. The exit code, the exit line and the refusal are unchanged. The devnet drills' refused starts all leave the black box as it was, so their `BlackBox` expectation is gone. Verified: 901 workspace tests and the devnet suite 43/43. No guest crate changed, so the template hashes stand.
The owner's requirement: libdex uses one scheduler throughout, its own, compiled into its guest and run natively by setup through `scheduler.h`. The reference scheduler is the specification's implementation. The plan called the header's direction open in six places. They now state the requirement and what is not built: a `Scheduler` trait and an adapter over the header, which the to-do lists as build work. The questions for Bart are rewritten after the owner's answers: what is ours to recommend is no longer asked, and each question says what it is about. Also noted: a batch proxy to rotate the submitter's key, and a filter on directs before they join the fridge.
libdex's author answered the first round of questions on 2026-11-12. The dated record keeps the answers and what each means for this tree. They change four things here: - Setup cannot read a libdex base: its state is a 512 MiB range and setup reads only a 2 MiB inspect report. The range read is now required, not a precaution. - The lane must close a batch on its real size. With libdex's largest op as the bound, a batch would close at 2 ops where about 110 fit. - An over-long direct is dropped, deposits included, which loses the deposit's tokens. The register proposes cutting it to the cap. - The nonce rule does not fit libdex, which creates accounts only on a deposit and deletes session wallets with their nonces. The register names the two ways out, and the deferred replay plan no longer says its trigger is absent. The todo's questions shrink to the five follow-ups.
Whether replay protection must also cover direct inputs depends on it.
Two worker tests poll the stored price with a fresh read-only connection while the worker opens and closes its own writer on every refresh. A reader that meets the writer gives up after its 50 ms busy timeout, and the tests unwrapped that error, so a slow runner failed run_forever_retries_a_failed_first_refresh with "database is locked". The poll now takes busy as "not yet" and still panics on any other error.
With the fast confirmation rule on, the tag trails the head by a few blocks and can fall back to the finalized block. Recorded as a check to make before a deployment reads such a node.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
One
setup [--base <MACHINE_DIR>] [--dry-run]starts every era, from genesis or from a stored canonical machine, and rebuilds the sequencer's history from L1. Procedure and code map: cockroach recovery. Plan and what remains: sequencer recovery plan.b691d88): a free-standingSchedulerinsequencer-coreturns L1 inputs into L2 transactions, and one sharedexecuteruns them for the guest, the lane and catch-up. TheApplicationtrait keeps what the application stores apart from what shared code decides.52b242d): setup reads the base's export, flushes the submitter's wallet to pin a stop block, syncs L1, checks for an escape, rebuilds the history and publishes the era in one transaction. The feed serves the rebuilt rows, then the live rows.36337b9):consensus_vectors_v2.jsonis held by a native test, the guest in the machine and a replay throughscheduler.h. Devnet drills recover from genesis and from a stored machine, on the Rust and C hosts, and a fresh watchdog agrees afterwards. Numbers: recovery measurements.e38baab): the executed-input count is the L2 tx count (L2TxCount,l2_tx_count(),Executed::Done { index }). The HTTP fields keep the old name until the egress redesign.676fd0a): with clients realigning through the feed, the sequencer stops serving raw L1 inputs for projection replay.GET /historical-l1-inputs, the SDK'shistorical_l1_inputs, the projection contract and the Track 3 integration plan are gone.6dcda8a..252caed, sixteen commits): the schema states its own rules, and the Rust interface stops exposing what only storage may change. Each commit message carries its reasoning; invariants has the result.snapshotstable; download leases live in the process.l2_tx_count.storageis crate-private, the lane's head is opaque, and results and the local batch id are named types.Coveredwitness makes a broadcast before the wallet-nonce watermark is raised fail to compile.93fa991..67dc5dc): the automatic recovery is deleted, and cockroach recovery is the only one. The sequencer stops for good once its oldest unaccepted batch can no longer land: batch landing and outages.runchecks two terminal facts, a recorded divergence and a void history, at start and while running. Startup repairs nothing.SchedulerLimits::max_wait_blocks), and setup pins it. The wallet's is 7200 blocks, about 24 hours.docs:commits beforeb691d88are the plan's history and may be squashed before merge.Risk and compatibility
--preemptive-margin-blocksand--l1-read-stale-after-blocksare removed, so a deployment that still sets them fails to start.runof an era needs L1. With the provider down it exits 20 and opens no batch; a restart with a batch already open still boots on its stored view.--dry-run) and refuses any non-empty data directory.setup --recoveryand the fold replay are removed.sqlite3must be 3.37 or later to open the file. The register lists the whole upgrade path.runrefuses a changed scheduler limit. A binary whose staleness window or direct-payload cap differs from the one setup pinned stops with the identity mismatch (exit 30), so changing either means a new setup./history's baseline gains a required field,rebuilt_history_end, so an older SDK build fails to decode it. A client of an older era adopts the new one at its own count (adopt_current_era)./historical-l1-inputsis removed, with the SDK'shistorical_l1_inputsand theHistoricalL1Input*types.STALE_GENERATION,from_generation,compatibilityandX-Recovery-Generationare gone, and the subscribe query and/historyanswer400to a field they do not know. An SDK build from before cannot talk to this server./healthzgains three fields:landing_lag_blocks,max_wait_blocksandsafe_head_age_seconds. Its status code and other fields are unchanged.stateinspect report's limit, which is about 27,000 wallet accounts. Reading it through a memory range is a follow-up.Not in this PR
Verification
Local, at
67dc5dc: 901 workspace tests with none ignored, clippy with warnings denied and fmt; devnet suite 43/43. At0db8bb8, on rebuilt guest images (no guest crate changed since): the OpenAPI lint; in-machine suite 27/27, watchdog machine tests 30/30. Each step was reviewed independently before it was committed.CI was last fully green at
4d72b77. Since then therustjob has failed at its package install, before compiling: the runner cannot resolve the Ubuntu mirror.