diff --git a/.claude/board/entries/2026-10-10-split-tunnel-sparse-append-resident-fold.md b/.claude/board/entries/2026-10-10-split-tunnel-sparse-append-resident-fold.md new file mode 100644 index 000000000..60374042f --- /dev/null +++ b/.claude/board/entries/2026-10-10-split-tunnel-sparse-append-resident-fold.md @@ -0,0 +1,40 @@ +# 2026-10-10 — split-tunnel persistence measured in a consumer: sparse append on write, resident fold on read + +**Status:** MEASURED (one consumer: HubSPO-rs `hubspo-store::log`, PRs #23–#25) · WORKING-MODEL for other carriers. +**Knowledge doc:** `.claude/knowledge/split-tunnel-sparse-append-resident-fold.md` (the pattern, its rules and how to adopt it). + +First measured instance of two OPEN working models: +- newest-first over append-only, from `2026-09-23-stable-row-ids-hold-tombstones-exact-seal-still-owns-cohort.md`; +- the persistence half of the alpha split tunnel / merge-on-read `SparseDelta`, from `2026-10-05-quack-storage-portability.md`. + +It does not use or change the `Alpha*` types: `AlphaOverlay` carries no payload, so this is the persistence half beside it. + +**Pattern:** +- **Write:** sparse `(version, owner, op, row, lane, present, value)` records, one Lance version per batch. Never updated in place, never compacted. +- **Recovery:** Lance is read once, at replay. +- **Read:** every read is a Quack / mask-risc fold over the same records held resident. Newest wins by append order, with no sort. + +**Measured** (release, 2026-10-10; 1.2M log records, 1,000 versions, 1,001 fragments): + +| read | median | +|---|---| +| history as a resident fold | **0.72 ms** | +| replay, once at startup | 532 ms | +| history by scanning Lance per read | 336 ms | +| history by address index into in-place updates (P-HIST-1) | 46 ms | +| budget | < 20 ms | + +- Sparse appends write 1,000 versions in 9.2–12.4 s, against ~300 s for in-place updates. +- 29 B per resident record. + +**DECISION (2026-10-10):** +- Logs of this shape are never compacted: append order is version order. +- Derived reads (history, diffs) write nothing. +- The only new persistent state is a learned outcome, recorded after as one merged, auditable record. +- BASIS: `2026-09-24-cost-scales-with-dirty-rows-replay-budget-before-materialization.md` (one write ≈ 10⁶ folds). + +**OPEN:** +- one writer per log comes from ownership (one writer per mailbox), not from the version checks. A racing second writer is detected only after its batch is durable (Lance rebases appends; `spog-alpha-channel-v1.md` F6). In-band `(cycle, batch_hash)` reconciliation is not in the reference implementation; +- the same pattern over 512-byte `NodeRow` images or V3 facet payloads; +- the `changed_coordinates` representation (still open, unchanged); +- resident sets larger than memory (a checkpoint becomes the recovery read). diff --git a/.claude/board/entries/README.md b/.claude/board/entries/README.md index 9ceeb67e1..cf044e315 100644 --- a/.claude/board/entries/README.md +++ b/.claude/board/entries/README.md @@ -25,12 +25,13 @@ index row, (3) no duplicate entry id. Checks 1 and 2 are deliberately opposite directions; the stranding this convention prevents shows up in exactly one of them, never both. -298 entries, 2026-08-06 .. 2026-10-10. +299 entries, 2026-08-06 .. 2026-10-10. | date | entry id | finding | file | |---|---|---|---| | 2026-10-10 | `D-XGP-8` | | [2026-10-10-woa-anchors-timesheet.md](2026-10-10-woa-anchors-timesheet.md) | | 2026-10-10 | `D-HPR-0` | | [2026-10-10-v3-mandatory-hotplug-reading.md](2026-10-10-v3-mandatory-hotplug-reading.md) | +| 2026-10-10 | `split-tunnel-sparse-append-resident-fold` | Sparse append on write, resident fold on read: history 0.72 ms vs 336 ms scanning Lance; Lance read once, at replay; never compacted | [2026-10-10-split-tunnel-sparse-append-resident-fold.md](2026-10-10-split-tunnel-sparse-append-resident-fold.md) | | 2026-10-10 | `render-classid-reversal-audit` | | [2026-10-10-render-classid-reversal-audit.md](2026-10-10-render-classid-reversal-audit.md) | | 2026-10-10 | `E-W2-SPACE-AXIS-CUSTOM-SPACE-LEAVES-THE-CLASSID-1` | W2's space axis is decided: fixed spaces 0–3 stay in the classid, custom spaces move to the payload or edge | [2026-10-10-e-w2-space-axis-custom-space-leaves-the-classid-1.md](2026-10-10-e-w2-space-axis-custom-space-leaves-the-classid-1.md) | | 2026-10-10 | `D-XGP-3-BILLABLE` | | [2026-10-10-d-xgp-3-billable-converted.md](2026-10-10-d-xgp-3-billable-converted.md) | diff --git a/.claude/knowledge/split-tunnel-sparse-append-resident-fold.md b/.claude/knowledge/split-tunnel-sparse-append-resident-fold.md new file mode 100644 index 000000000..9e751138e --- /dev/null +++ b/.claude/knowledge/split-tunnel-sparse-append-resident-fold.md @@ -0,0 +1,168 @@ +# Split-tunnel persistence: sparse append on write, resident fold on read + +**READ BY:** any session designing a write path, a history / audit / "what +changed" read, a replay or recovery path, or anything that reads a Lance +dataset on a hot path. Also `v3-kanban-executor-engineer`, +`fold-carrier-scientist`, `query-stage-profiler`, and anyone touching +`contract::alpha` / `alpha_tunnel` who wants the persistence half. + +**Status:** +- MEASURED in one consumer (HubSPO-rs `hubspo-store::log`, PRs #23–#25, 2026-10-10). +- VERIFIED-IN-CODE there. +- WORKING-MODEL for every other consumer and carrier shape (see § Scope). + +**BASIS:** +- `board/entries/2026-09-24-cost-scales-with-dirty-rows-replay-budget-before-materialization.md`: + one fold ≈ 1.7 ns, one write ≈ 1.7 ms floor, so one write buys about 10⁶ folds. +- `board/entries/2026-09-24-three-clocks-never-meet-implicitly.md`: a Lance scan is + recovery-clock cost (334 ms at 1,001 fragments). +- `board/entries/2026-09-23-stable-row-ids-hold-tombstones-exact-seal-still-owns-cohort.md` + § "newest-first over append-only" and + `board/entries/2026-10-05-quack-storage-portability.md` § "OPEN — SparseDelta". This doc + is the first measured instance of both working models. + +**REVISIT WHEN** a consumer's resident records no longer fit memory, or a read +needs data from before the last replay that the resident set no longer holds. + +## The pattern in one picture + +``` + WRITE path (persistence clock, ms) READ path (live clock, ns) + +batch ──► check ──► encode sparse records ──► Lance append (one version per batch) + │ never updated in place + │ never compacted + └──────────► resident records ◄──── fold (Quack / mask-risc) + ▲ history, "what changed", + │ newest-wins merge + Lance ──► replay ────┘ (read ONCE: startup / recovery) +``` + +The **split tunnel** here is the separation of the two paths, which is the +same separation `alpha_tunnel.rs` makes in memory. Reads never go through +storage, and writes never go through the read structures. The writer appends +sparse records to Lance; readers fold over the same records held in memory. +`AlphaOverlay` carries no payload, so it cannot be the persistence half +(`2026-10-05-quack-storage-portability.md` § "OPEN — SparseDelta": "Alpha does not supply that signal (it carries no payload)"). This pattern is that +persistence half. It is **not** a use of the `Alpha*` types. + +## The rules + +1. **Write only what changed, as sparse records.** One record per changed + coordinate: `(version, owner, op, row, lane, present, value)` in the reference + implementation, with `op ∈ {BORN, SET, DELETE}`. An insert writes BORN plus one + SET per lane. An update writes one SET. A delete writes one DELETE. Never a + full-row image of unchanged lanes. +2. **One batch = one Lance version, from exactly one writer.** The single + writer comes from **ownership**: one writer per mailbox (`mailbox_owner`, + write on behalf). The version checks do not enforce it. + - The reference implementation checks the dataset's latest version before + the append and after it (VERIFIED-IN-CODE: `log.rs` `commit`). + - The check before the append refuses a *stale* writer (one that is behind + before it writes). + - It cannot refuse a *racing* writer. Two writers that both pass the check + both append, because Lance rebases an `Append` instead of failing it + (`spog-alpha-channel-v1.md` F6: a read-then-append guard is a TOCTOU and + cannot "refuse, not renumber"). + - The check after the append only *detects* that race, once the second batch + is already durable. Replay then refuses the log, which stays refused until + reconciled. + - The durable fix is in-band idempotency, `(cycle, batch_hash)` in the same + commit and reconciled first (F6). The reference implementation does not + have it yet (OPEN). +3. **Never update in place, never compact.** + - DECISION 2026-10-10, BASIS: append order is version order. + - So "newest wins" is applied by replaying in append order, with no sort. + A log out of order is refused at replay, as a batch whose version is not + the next one. + - Compaction would add Lance versions and break the version ⇔ batch mapping. + - It is also unnecessary: the read path never scans Lance. +4. **Lance is read exactly once per process: at replay.** + - Replay decodes every record, applies them through the same `apply` a live + writer uses, and only on success installs them as the resident set. + - A refused replay leaves the resident set unchanged, so a corrupt record can + never reach a reader. +5. **Every committed batch's records also stay resident.** + - `commit` appends them to the resident set after the Lance append succeeds. + - A writer whose resident set is not at the table's version refuses to + commit; otherwise reads would silently miss earlier batches. +6. **Reads fold over the resident records.** + - Example: the history of `(row, lane)` is one Quack program, + `row = r ∧ ((lane = c ∧ op = SET) ∨ op = DELETE)`, run on mask-risc over + the resident `U32` lanes. Only matches are decoded. + - No index, no column projection, no Lance filter expression (so no + DataFusion), no compaction. +7. **Derived reads write nothing.** History, diffs and "what changed since v" + are recomputed by folding (rule 6), never stored. + - DECISION 2026-10-10: the only new persistent state is a *learned* outcome, + recorded after the fact as one merged, auditable record. + - BASIS: rule-of-thumb 10⁶ folds per write. + +## Measured (reference implementation) + +HubSPO-rs `crates/hubspo-store/examples/probe_history.rs`. +- Run: release, one container, 2026-10-10. +- Data: 100k records, 1,000 versions × 1,000 updates, 1,001 Lance fragments, + 1,199,991 log records. + +| read | median (range) | runs | +|---|---|---| +| history, resident fold (this pattern) | **0.72 ms** (0.64–1.99) | 101 | +| replay from Lance (once, at startup) | 532 ms (447–611) | 5 | +| history by scanning Lance on every read (#24) | 336 ms (320 ms of it the scan) | 7 | +| history by address index into in-place updates (P-HIST-1) | 46 ms best | — | +| budget (P-HIST-1) | < 20 ms | — | + +- **The read** costs about 0.6 ns per resident record, consistent with the + 10⁶-folds-per-write rule. +- **The scan** is the whole cost of a Lance-read history: the program itself + is 3–15 ms of it. +- **Removing the sort** in favour of append order was speed-neutral: run-to-run + noise (up to 46 ms) exceeded every difference between variants. +- **Writes** (P-SPARSE-1, same consumer): 1,000 versions of sparse appends take + 9.2–12.4 s, against ~300 s for in-place updates. The appends leave no deletion + vectors to read. +- **Memory:** 29 B per resident record in the reference shape (1.2M records ≈ 35 MB). + +**Disable runs, all red** (`tests/log.rs`, each guard removed alone): +- resident-version check in `commit`; +- records installed before replay validates; +- `commit` not appending to the resident set; +- the `row`, `op = DELETE` and `op = SET` terms of the history program; +- the I32 sign bit. + +A replayed log gives histories identical to the one that wrote them. + +## Scope — what is and is not shown + +- **Shown:** one consumer, one carrier shape (u32 lanes, I32 as bit pattern), + one writer per log, single-process recovery. +- **Not shown, and not provided by this pattern:** protection against two + concurrent writers (rule 2). Ownership must guarantee one writer. A race is + only detected afterwards, and in-band reconciliation is OPEN. +- **WORKING-MODEL, not shown:** + - 512-byte `NodeRow` images or V3 facet payloads as the record; + - field-bitmap `changed_coordinates` (the open `SparseDelta` representation + question stays open); + - multi-mailbox logs; + - a resident set larger than memory (then a checkpoint, a base written once + plus the tail since it, becomes the recovery read, still never a hot-path + read). +- **Not a claim:** that Lance scans are slow in general. They are recovery + cost, measured here; the pattern just keeps them off the read path. + +## How to adopt it + +1. Define the sparse record for your carrier (rule 1). Keep `version` and + `owner` on every record. +2. Wrap the writer as `check → append one version → apply → extend resident`, + in that order (rules 2, 5). +3. Write `replay` as the only reader of Lance (rule 4). Install the resident + set only after every batch applied. +4. Express every read as a Quack program over the resident lanes (rule 6). + If a read seems to need Lance, it is a recovery read, not a hot-path read. +5. Measure with a probe that times the fold and the replay separately, and + pin the guards with disable runs. + +Reference implementation: HubSPO-rs `crates/hubspo-store/src/log.rs` (`Log::commit`, +`Log::replay`, `Log::history`) and `tests/log.rs`. diff --git a/CLAUDE.md b/CLAUDE.md index 6b98ff70e..de8da0b59 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -1553,6 +1553,7 @@ SIBLING REPOS: .claude/knowledge/ogar-consumer-preflight.md — MANDATORY before any consumer (woa-rs/medcare-rs/smb-office-rs) classid/bridge/codebook work; the consumer-side spellbook (pull via *Port::class_id / contract::ogar_codebook::canonical_concept_id; NEVER construct a *Bridge or copy the codebook). Inverse of OGAR's SURREAL-AST-TRAP-PREFLIGHT; operational mirror of docs/CONSUMER-BRIDGE-DEPRECATION.md (#589/#590). .claude/knowledge/compilation-vs-runtime-substrate.md — MANDATORY before filing a cross-repo issue or writing any seam/lowering design that pairs a consumer repo with SoA/temporal.rs/NARS/RBAC/mailbox vocabulary: contracts compile types; codegen emits into consumer repos; cargo builds them; the running engine's internals never gate a build or a typed-value construction (the door-knocking-compiler test). Born from OGAR #208 (closed hallucinated). Carried by layer-boundary-warden / codegen-flow-cartographer. .claude/knowledge/assembler-vs-storage-substrate.md — MANDATORY before assigning responsibility across the OGAR↔lance-graph boundary or naming any "ingest" API: OGAR = the V3 assembler substrate (IR/mint/codegen); lance-graph = spine (contracts+query); Lance = calcification, never an ingest endpoint. Ownership quick-map + failure signatures from the 2026-07-14 #691/#208 chain. Carried by layer-boundary-warden / codegen-flow-cartographer. +.claude/knowledge/split-tunnel-sparse-append-resident-fold.md — PATTERN (MEASURED in a consumer): write = sparse append, one Lance version per batch, never compacted; read = fold over resident records; Lance read once, at replay. History 0.72 ms vs 336 ms scanning Lance. Read before designing any write path, history/diff read, or a hot-path Lance read .claude/CALIBRATION_STATUS_GROUND_TRUTH.md — OVERRIDE: read BEFORE any SESSION_*.md .claude/PLAN_BF16_DISTANCE_TABLES.md — 5-phase plan for BF16 distance tables .claude/TECHNICAL_DEBT_SIGNED_SESSION.md — 56% useful, 44% bypass (honest review)