Repository navigation
dir-sim: the history as a log of verified append-only batches - #1482
Merged
Merged
Conversation
log (pure, no I/O) is what D-DSP-2's storage crate writes and reads. LogState::batch holds only what an image has beyond the log: values, roots and versions since the last save, tags and verdicts that changed. A batch carries any number of logical versions (the history's own clock, never the storage version), and committed() advances the state only after the store write succeeded; a batch is committed once, in order. Every batch starts with a header: number, first version, version count, record count, FNV-1a checksum. replay refuses the whole log at the first batch with a numbering gap, a missing header, records out of position, a count or checksum mismatch, or versions that do not follow. Nothing is repaired; replay_prefix is the separate call that reads up to the damage and names it. The encoding destructures every struct exhaustively, so a new field is a compile error rather than a field dropped from the log. Decoding rebuilds through validating constructors (RemoteMailbox::new, Dn128::new) and refuses trailing bytes. Tests: every Change, Violation and Recipient variant and a full ObservedNode round-trip, every strict prefix refused; two-batch log restores the same reads; one batch holds five versions; four kinds of torn save refused; the explicit prefix; forward-only commits. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01G22yT6htkcdyXsihxxXdrg
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01G22yT6htkcdyXsihxxXdrg
AdaWorldAPI
marked this pull request as ready for review
October 11, 2026 10:32
|
Warning Review limit reachedYour organization has reached its usage spending cap. Adjust your spending cap in the billing tab. Next included review available in 59 minutes. View limit details
Warning Billing warning: we have not been able to collect payment for this subscription for more than 72 hours. Please update the payment method or pay any pending invoices in Billing to avoid service interruption. Comment |
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 251ec01460
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
LogState::batch only compared counts, so an image with the logged number of values and versions but different content was accepted: the sparse loops skipped the changed prefix and could return None, reporting a save while replay kept the old history. LogState now keeps a digest of the logged values, roots (in version order) and versions, and batch refuses an image whose prefix digests differently (NotAContinuation). LogState's Default holds the digest of the empty history, so the first save is checked like the rest. replay_prefix cloned the accumulated image before every batch, making a restore quadratic in the number of saves. A batch is now applied in place and undone on failure by truncating values, roots and versions to their previous lengths; its tags and verdicts collect apart and merge only when the batch is whole. Tests: a_divergent_history_is_not_a_continuation (changed value, version and root each refused; red before: the changed value saved as nothing new); a_batch_failing_after_it_applied_is_undone (a header claiming one version more fails only after the batch's value and tag were applied; the image and state are those of the batches before it). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01G22yT6htkcdyXsihxxXdrg
AdaWorldAPI
pushed a commit
that referenced
this pull request
Oct 11, 2026
D-DSP-2b. A new workspace-excluded crate, lance-graph-dir-sim-lance, so lance-graph-dir-sim itself stays free of lance, arrow and datafusion. It stores the records lance_graph_dir_sim::log builds and reads them back; it decides nothing about their content. - LanceLog::append stores one batch as one Lance commit (Create for the first, then Append) and only then advances the LogState. The writer holds the dataset version it last saw: if the dataset is elsewhere, nothing is written (StaleWriter), because Lance would rebase an append onto a newer version and interleave two writers' batches. A commit that does not land as the next version is reported as Race. - LanceLog::restore reads every record in one scan and hands them to replay; a missing dataset is an empty log. - The logical versions stay in the records; the Lance version is only the physical commit counter. Tests: two saves restore to the same reads and the same log state; a missing dataset is an empty log; a stale writer writes nothing, on create and on append; a batch stored with a record missing is refused at restore; a batch out of order is refused before anything is written. CI: .github/workflows/dir-sim-persist.yml builds and tests the crate (protoc and the protobuf includes installed, which lance's build needs). Board: D-DSP-2a shipped (#1482), D-DSP-4 shipped (#1481), D-DSP-2 in PR. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01G22yT6htkcdyXsihxxXdrg
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Decision 3(b): OGAR's logical version stays independent of Lance's physical commit version. Ordered logical versions persist in atomic, verifiable batches, following the sparse-append / resident-fold pattern. A torn save is detected, and incomplete recovery is refused. This is D-DSP-2a.
The new
logmodule is pure, with no I/O. It is what the D-DSP-2b storage crate will write and read.LogState::batch: holds only what an image has beyond the log: values, roots and versions since the last save, plus tags and verdicts that changed. A batch carries any number of logical versions, counted on the history's own clock and never on the storage version.LogStatekeeps a digest of the logged values, roots and versions, andbatchrefuses (NotAContinuation) an image whose logged prefix differs, not only a shorter one.committed(): advances the state only after the store write has succeeded. Each batch is committed once, in order.replay: refuses the whole log at the first batch with a numbering gap, a missing header, records out of position, a count or checksum mismatch, or versions that do not follow. Nothing is repaired.replay_prefix: a separate, explicit call that reads up to the damage and names it.RemoteMailbox::new,Dn128::new) and refuses trailing bytes.Tests
Change,ViolationandRecipientvariant and a fullObservedNoderound-trip, and every strict prefix of each encoding is refused.replay_prefixreads up to the damage and names it.Disable runs were red as required (7 guards).
Rebased onto #1481:
cargo +1.99.0 fmt --check,clippy --all-targets -D warningsandcargo test(231 passed, 0 failed) onlance-graph-dir-sim. The supersession index is current.Not in this PR
D-DSP-2b, the Lance I/O crate that writes these batches, is queued on the board.
🤖 Generated with Claude Code
https://claude.ai/code/session_01G22yT6htkcdyXsihxxXdrg
Generated by Claude Code