diff --git a/.claude/board/FRUITS.md b/.claude/board/FRUITS.md index 4ec2cdb41..cabae0e54 100644 --- a/.claude/board/FRUITS.md +++ b/.claude/board/FRUITS.md @@ -4,7 +4,7 @@ Generated by `.claude/tools/fruits_index.py --write`. Do not hand-edit: add or change a `**Fruit:**` line on the ledger row and regenerate. CI fails when this file is stale. -Open: 34. Closed: 1. +Open: 35. Closed: 1. ## discovery torque @@ -43,6 +43,7 @@ _none open_ | tier | row | status | ledger | |---|---|---|---| | High-signal | ISS-SOURCE-ROOT-AGENT-OR-REPORT — is a source root the observer or one report? (2026-10-31) | OPEN (operator decision). **Basis:** VERIFIED-IN-CODE (`crates/xlog-relcomp-probe/tests/provenance.rs` `root()` hashes the observer and the report's facts; `unrelated_sources_never_collapse_to_one_root` pins that one observer with two different reports gives two roots). | `ISSUES.md` | +| High-signal | ISS-RECIPE-LOCO-VOCABULARY-EXISTS-TWICE — two `impl Vocabulary` claim the same 34 op bytes and disagree (2026-10-26) | OPEN (latent). **Basis:** VERIFIED-IN-CODE (OGAR `crates/ogar-loco/src/nars.rs:133-175`, read at OGAR `68cb29b`; lance-graph `crates/lance-graph-ogar/src/recipe_vocab.rs:134-185`, read at `main` `87b2899b`). | `ISSUES.md` | | High-signal | ISS-SIX-DECISIONS-NOT-EXECUTED-ACTION-GATE — `commit` still fails open where the decision says fail closed (2026-10-23) | OPEN. **Basis:** VERIFIED-IN-CODE (`crates/lance-graph-contract/src/action.rs:266` `pub fn commit`, `:328` `pub fn commit_via`; read at `main` `19c1324a`). | `ISSUES.md` | | High-signal | ISS-SIX-DECISIONS-NOT-EXECUTED-FLOAT-REFS — the fold path still pulls OGAR `main` into the default build (2026-10-23) | OPEN. **Basis:** VERIFIED-IN-CODE (`crates/lance-graph/Cargo.toml:58` unconditional `lance-graph-r2il`; `crates/lance-graph-r2il/Cargo.toml:19-20` OGAR `branch = "main"`; no `fold` feature in any `crates/*/Cargo.toml`, 39 manifests read in full by a Haiku scout, `:51-58` re-read). | `ISSUES.md` | | High-signal | ISS-SIX-DECISIONS-NOT-EXECUTED-WIRE50-ISA-LEGACY — CE64 50..52 presence and the 46..49 legacy reading have no code (2026-10-23) | OPEN. **Basis:** VERIFIED-IN-CODE (`Option` only in `crates/lance-graph-planner/tests/maturation_ladder.rs`, EXPERIMENTAL; no `ISA_LEGACY` / `legacy_opcode` / `LegacyRead` in `crates/`, scout over the named patterns; `causal-edge/src/edge.rs` has `fn activation(self) -> i8`). | `ISSUES.md` | diff --git a/.claude/board/INTEGRATION_PLANS.md b/.claude/board/INTEGRATION_PLANS.md index 1b639bcdc..b8a5d491b 100644 --- a/.claude/board/INTEGRATION_PLANS.md +++ b/.claude/board/INTEGRATION_PLANS.md @@ -1,3 +1,8 @@ +## 2026-11-02 — D-CODEGRAPH-0: a resident Codegraph, Charon → ruff → lance-graph + crai-rs → `.claude/plans/d-codegraph-0-resident-codegraph-v1.md` + +**Status:** PROPOSED; slice A (ruff Charon intake) started. **Board:** [entry](entries/2026-11-02-d-codegraph-0-gap-map.md), [STATUS_BOARD](STATUS_BOARD.md) D-CODEGRAPH-0. Kythe semantic facts + R2IL executable semantics + DT64/DR binding. Gap map G1–G9: no Charon intake in ruff, no Kythe producer in Rust, no `ref/call` edges, no Rust-body → R2IL lowering, no resident or incremental store, closure capped at 64 per tile, body anchor open, ruff sink unmerged. Slices A (intake) → B (resident, incremental) → C (queries, diagnostics, bodies) → D (MCP) → E (token measurement). +**Confidence:** gap map read from `main` of four repos; rust-analyzer oracle measured on the fixture; no intake run yet. + ## 2026-11-01 — D-STAUNEN-5: resident recursive plasticity → `.claude/plans/d-staunen-5-resident-recursive-plasticity-v1.md` **Status:** PROPOSED, plan only. **Board:** [STATUS_BOARD](STATUS_BOARD.md) D-STAUNEN-5. First closed learning loop on the XLOG-on-cube foundation: one binary relation, its transitive closure as the prediction, a declared set of dormant candidate edges. A novel stimulus (an observed pair the closure misses) licenses one activation, under the contract of Atlas's `update_mask` (fixed mask shape, 0/1, atomic between sequences, regrowth clears history, needs committed state) plus a generation guard; Atlas is the contract source, not the executor. The delta stays in the cube as `ndarray::simd` mask words. Two competing policies (H1 parsimony, H2 novelty × eligibility) are scored on held-out pairs from a hidden graph; later evidence consolidates or reverts each update, and a selector learns which policy to use. Also records the xlog-cube coverage order (projection composition, mask filters, resident execution, computed columns, F100, N-ary gated) and that 140/230 measures admission, not checked answers. diff --git a/.claude/board/ISSUES.md b/.claude/board/ISSUES.md index 955a7118b..32adaa089 100644 --- a/.claude/board/ISSUES.md +++ b/.claude/board/ISSUES.md @@ -27,6 +27,25 @@ - **What closes it:** the cause, from rust-analyzer's source, then either a fix upstream or a documented limit on the exporter. +## ISS-RECIPE-LOCO-VOCABULARY-EXISTS-TWICE — two `impl Vocabulary` claim the same 34 op bytes and disagree (2026-10-26) + +**Status:** OPEN (latent). **Basis:** VERIFIED-IN-CODE (OGAR `crates/ogar-loco/src/nars.rs:133-175`, read at OGAR `68cb29b`; lance-graph `crates/lance-graph-ogar/src/recipe_vocab.rs:134-185`, read at `main` `87b2899b`). +**Fruit:** High-signal + +- Both map recipe `id` to `FnIndex(DOMAIN_FLOOR + id - 1)` and both + `impl Vocabulary` over that range. They disagree on two columns: + `nars.rs:142-147` gives `Datapath` recipes arity 2 and `Gate` / `Control` + arity 1, with no value codebook; `recipe_vocab.rs:142-144` gives every recipe + arity 1 (its doc: a second operand "would be a value with no source"), and + `:171-176` declares `CB_ATTENTION_FOCUS`. +- Latent today: `NarsVocabulary` has no consumer outside its own file, and + `lance-graph-ogar` is excluded from the workspace. It pulls `ogar-loco` at + `branch = "main"` (`lance-graph-ogar/Cargo.toml:140`), so both meet the next + time that crate resolves against a newer OGAR. +- **What closes it:** an operator ruling on the arity (revision (f) Q7), then + one implementation delegating to the other. The change spans OGAR and + lance-graph. Ledger: `.claude/research/2026-10-recovery/LEDGER-REVISION-2026-10-26f.md` S1. + ## ISS-FISHERZ-ENCODE-SATURATES-SILENTLY — out-of-family cosines encode to the edge code with no flag (2026-10-23) **Status:** OPEN. **Basis:** VERIFIED-IN-CODE (`crates/bgz-tensor/src/fisher_z.rs:67`, `i8_val.clamp(-128.0, 127.0) as i8`, read at `main` `8f610a85`) + MEASURED. diff --git a/.claude/board/STATUS_BOARD.md b/.claude/board/STATUS_BOARD.md index 27f02c3d3..ed478ed72 100644 --- a/.claude/board/STATUS_BOARD.md +++ b/.claude/board/STATUS_BOARD.md @@ -231,6 +231,18 @@ Plan: `.claude/plans/d-soc-fence-v1.md`. Plan only; nothing runs yet. **Evidence boundaries:** D-SAG-6 #1760 merged docs; Math Foundry #1693/#1694 E1 lab merged; D-NFC #1753 native-first; #1775 D-DRAV draft / AV1–AV12 NOT RUN; MORK oracle with unresolved production licence/allocator; EXO-148 stable durable types unresolved. No complete distributed reasoning implementation or measured AI/data-center scaling claim. +## D-CODEGRAPH-0 — resident Codegraph: Charon → ruff → lance-graph + crai-rs (2026-11-02) + +Plan: [d-codegraph-0-resident-codegraph-v1.md](../plans/d-codegraph-0-resident-codegraph-v1.md). Entry: [gap map](entries/2026-11-02-d-codegraph-0-gap-map.md). + +| D-id | scope | status | gate / falsifier | +|---|---|---|---| +| **D-CODEGRAPH-0-A** | ruff `ruff_charon`: LLBC → Kythe facts with a witness status per call, for one real crate | LANDED on ruff `claude/clone-repositories-71a5sw` (`53c6711a84`): calls byte-identical to `llbc_calls.py` on the fixture and `mask-risc`; receipt conserved (16,149); Kythe `/xrefs` 3/3 call pins | rust-analyzer oracle (`sag-ra-probe GRAPH=1`) and rustc on `fixtures/codegraph`; `POSSIBLE` never counted as a confirmed caller | +| **D-CODEGRAPH-0-B** | resident rows via the ruff sink; per-file digests; incremental invalidation | QUEUED | clean import == incremental import | +| **D-CODEGRAPH-0-C** | callers / callees / impact / diagnostics; tiled closure; body → R2IL design check | QUEUED | each guard disable-verified; unused import proven only without macros, traits, globs, re-exports | +| **D-CODEGRAPH-0-D** | local stdio MCP over C | QUEUED | only after B and C gates | +| **D-CODEGRAPH-0-E** | token measurement with and without the Codegraph | QUEUED | same revision, questions, model; nothing claimed before measured | + ## D-SAG-6 — Rust Codegraph → Mathematics → Logic → Tracert (2026-10-24) **Status: S0 PROPOSAL / DOCS ONLY.** Plan: [d-sag-6-rust-codegraph-math-logic-tracert-v1.md](../plans/d-sag-6-rust-codegraph-math-logic-tracert-v1.md). Agent prompt: [d-sag-6-rust-codegraph-logic-tracert-claude-code.md](../prompts/d-sag-6-rust-codegraph-logic-tracert-claude-code.md). Coordination: [EXO-50](https://linear.app/exo-red/issue/EXO-50/agent-message-bus-gpt-xgrok-ccsession-id). Continuation of D-SAG / D-DRMC / Ascent-BYODS, **not** a revived SoftwareGraph, separate registry, runtime or new proof authority. Parallel #1759 only reconciles existing board status. diff --git a/.claude/board/entries/2026-10-26-dqm-m1-timed-arm.md b/.claude/board/entries/2026-10-26-dqm-m1-timed-arm.md new file mode 100644 index 000000000..25636b525 --- /dev/null +++ b/.claude/board/entries/2026-10-26-dqm-m1-timed-arm.md @@ -0,0 +1,49 @@ +# 2026-10-26 — D-DQM-M1: timed arm in both differential halves; exit gate NOT met + +**Status:** MEASURED (both arms ran); M1 exit gate open. No performance claim. + +Plan: `.claude/plans/duckdb-dr-quack-machine-room-reuse-v1.md` M1. + +## What landed + +- `tests/duckdb/timing.py` (DuckDB arm): reuses `oracle.load` and + `oracle.encode`; parity first, then `t_prepare_ns` (PREPARE alone) and + `t_execute_ns` (EXECUTE + fetchall), median of `--reps`, `SET threads`. + `oracle.py` only gained `load(con)`; `cases.tsv` regenerates byte-identical. +- `duckdb_differential.rs` (Quack arm): `QUACK_DIFF_TIMED_REPS` (default 0, + so the default run is unchanged: METRIC lines diffed identical before and + after). When set, METRIC gains `t_compile_or_bind_ns`, `t_execute_ns`, + `timed_reps`; every rep asserts the same answer as the untimed run. + +## Run (same machine, 4 cores, load 0.2, x86-64-v3 default config) + +``` +QUACK_DIFF_TIMED_REPS=101 cargo test --release -p lance-graph-quack \ + --test duckdb_differential -- --nocapture --test-threads=1 # 39 pass, 1 ignored +python -I crates/lance-graph-quack/tests/duckdb/timing.py --threads 1 --reps 101 + # 38/38 parity=OK, DuckDB 2.0.0 +``` + +- Quack, 34 timed cases: bind 56–1201 ns, execute 440–7899 ns (median of 101). +- DuckDB, 38 cases: prepare 449–1714 µs, execute 437–3251 µs. Reported, not + gated (D-DQM's reference-arm exception). +- 27 cases pair by name; 7 Quack-only (avg split into sum and count, two + k-program variants, the ordered-view join); 11 DuckDB-only (the avg and + having cases go through `run_group_avg` / `run_group_having`, which build + no CaseMetrics and are untimed). + +## Why the exit gate is not met (so no ratio is quoted) + +The gate asks for identical measured boundaries. They are not: + +1. DuckDB's execute includes the Python call and fetchall conversion; Quack's + includes clearing its out-buffer and the per-rep answer check. +2. Quack's bind includes lowering plus `Scratch` allocation; DuckDB's prepare + is parse/bind/plan. +3. `run_scalar`'s bind rep re-runs only `Scratch::for_program`, not the + caller's lowering; `run_group` times separate scratches. +4. No oracle checksum in the METRIC line yet. + +OPEN: a DuckDB execute measured without the Python boundary (C API or a +server-side loop) or a stated FFI floor; time the avg/having helpers; +align the `run_scalar` bind boundary; add the checksum field. diff --git a/.claude/board/entries/2026-10-26-ledger-revision-1727-1784.md b/.claude/board/entries/2026-10-26-ledger-revision-1727-1784.md new file mode 100644 index 000000000..0c75bdd5a --- /dev/null +++ b/.claude/board/entries/2026-10-26-ledger-revision-1727-1784.md @@ -0,0 +1,34 @@ +# 2026-10-26 — Recovery ledger revised against #1727–#1784 and ndarray #368–#376 + +**Status:** WORKING-MODEL (research document; no code changed). + +`.claude/research/2026-10-recovery/LEDGER-REVISION-2026-10-26f.md` maps 61 +first-parent commits (`main` at `18fbe43a`) and 8 ndarray PRs (`master` at +`195844bd`) onto revision (e)'s open list, and folds in the open list a session +carried from 2026-09-18. OGAR had no merges in the window. + +- **Main finding:** the five unexecuted rulings of 2026-10-20 are still + unexecuted; no PR in the window touched them (re-read: `action.rs:266, 517`, + `lance-graph/Cargo.toml:58`, `supervisor.rs:203-212`), while about 22 plan + files were added (K30). +- **Closed (C73–C84):** the D-DRF focus-lease lab arc (P1, P2, G, W1, W3, W4, + W8, churn, TLB); D-DRAV-AV1; `lance-graph-onnx` with ULP-bounded Tanh/Exp + over ndarray; ndarray numerics on every backend (#368–#376); D-NFR-6; + D-MPF-1; D-DRMC L1; thought-git squash; EXO-129 (torque killed); the new + board ledgers; catch-ups through #1758; the bgz-tensor Fisher-Z + decomposition (the clamp itself stays open). +- **From the 2026-09-18 list:** 64k slab, ndarray #317's two corrections, the + semantic DuckDB differential, door 1 (facet register compare) and door 3 + (`pack`) are closed; Mississippi Queen is closed as rejected by + measurement (Tern2 replaced it); the hop is reframed (one hop exists three + ways, multi-hop is the RECURSION row); the duplicate 34-recipe vocabulary is + open and latent; two doc items are excluded (frozen prose, dated entry). +- **Links:** K26 (D-DRF W8's budget and ndarray #376 already implement the + lane-fold capstone's points 10 and 9), K27 (D-DRF-1 is a fifth admission + instance and the first called an exception), K28 (ndarray → ONNX is the one + cross-repo seam wired end to end), K29 (the hop and RECURSION are one item), + K30. + +OPEN: Q1–Q6 of revision (e); Q7 the recipe-op arity (S1); Q8 where the D-DRF-1 +zero-copy exception is written; Q9 which capstone governs the ONNX / +native-method membrane. diff --git a/.claude/board/entries/2026-11-02-d-codegraph-0-gap-map.md b/.claude/board/entries/2026-11-02-d-codegraph-0-gap-map.md new file mode 100644 index 000000000..a517e08ac --- /dev/null +++ b/.claude/board/entries/2026-11-02-d-codegraph-0-gap-map.md @@ -0,0 +1,65 @@ +# D-CODEGRAPH-0: gap map from Charon to a resident Codegraph + +**Date 2026-11-02 · D-id D-CODEGRAPH-0 · Status MEASURED (gap map), slice A started · Plan [d-codegraph-0-resident-codegraph-v1](../../plans/d-codegraph-0-resident-codegraph-v1.md)** + +The representation is Kythe semantic facts + R2IL executable semantics + DT64/DR binding (operator, 2026-10-31); the route is Charon → ruff → lance-graph + crai-rs. Checked against `main` of lance-graph (`527a0c7`), OGAR (`5b82ae0`), ruff (`4c3acd4`) and the last 50 PRs of lance-graph and OGAR, the last 16 of ruff. + +Findings: +- **No Charon intake exists in ruff**, on `main` or any remote branch. Charon has only been run out of process, with Python readers in `sag-ra-probe/tools/`. +- **No Kythe producer exists in Rust.** The only one is `sag-ra-probe/tools/kythe_entries.py`, fed by rust-analyzer facts, and it emits no `ref/call` edges. +- **No Rust-body → R2IL lowering exists.** R2IL is lifted from machine code (`ruff_r2il`) and executed for the fold and arithmetic band (`lance-graph-r2il`). +- `charon-lib` from `AdaWorldAPI/charon` builds on stable with `--no-default-features` and exposes `deserialize_llbc`, so the intake can link it. Producing LLBC needs the nightly Charon driver. +- crai-rs holds review semantics and falsifier matrices only; its fixtures were never run. +- `sag-ra-probe` did not build from a fresh resolution: `ra_ap_hir_def` pins `unicode-ident =1.0.24` (Unicode 17) while `unicode-properties` 0.1.5 is Unicode 18, and `ra-ap-rustc_lexer` asserts they match. Fixed by pinning `unicode-properties = "=0.1.4"`. + +Added in this commit: +- `sag-ra-probe GRAPH=1` (with `NO_MIR=1` and `FILES=`): typed rust-analyzer rows for use as the independent oracle (`file`, `node`, `occ` with `use` / `pubuse` roles, `call`, `impl`, `import`, `macro`, `diag`). +- The oracle fixture `fixtures/codegraph/{rev1,rev2}`. rustc on rev1 reports `unused_imports` at `cg_lib/src/shapes.rs:2` and `dead_code` at `cg_lib/src/util.rs:15`. The probe on rev1: 25 nodes, 104 occurrences, 10 calls, 3 impls, 9 imports, 1 macro marker, 3 diagnostics. +- rust-analyzer's call hierarchy misses `t3(1)` inside `assert_eq!`; its index records the reference. Any unused-import finding in a file with a macro call is therefore `POSSIBLE`, never proven. + +OPEN: the method body anchor (G7), whether R2IL can carry Rust places, drops and panics (G4), merge order of ruff's `claude/r2il-pr2-*` sink branches (G8). + +## Python pipeline on the fixture (2026-11-02, same day) + +The existing `sag-ra-probe` tools, run unchanged on `fixtures/codegraph/rev1` before any new Rust: +- Unit tests: `tools/test_tools.py` 15/15, `tools/test_kythe_entries.py` 9/9. +- **Kythe bridge:** `XREFS=1` → `kythe_entries.py` (457 entries, 0 skipped) → Kythe v0.0.76 `/xrefs`: 22/22 symbols equal rust-analyzer's occurrence sets; 19/19 hand-read pins pass (`fixtures/codegraph/rev1_xrefs.pin.tsv`, positions read with `cat -n`). Disable run: one pin moved by one column fails, and only that pin. +- **Charon** (AdaWorldAPI fork `c69e231`, `nightly-2026-09-17`, built with `--no-default-features --features rustc` because the default `minirust` feature needs a git submodule): `cg_lib` 27 s, 218 KB LLBC, `has_errors False`. +- `llbc_calls.py` → `span_join.py` → `project_rows.py` on `cg_lib`: + - **join:** 11/11 bodies (10 EXACT, 1 ENCLOSED), 0 unjoined; the two `Shape::area` impls are told apart by position; + - **impls:** 5/5 paired, 0 one-sided; + - **rows:** 22 DEFINES, 4 HAS_IMPLS (rust-analyzer); 3 CALLS WITNESSED, 2 MAY_DISPATCH POSSIBLE (`trait:Clause` for the bound, `dynamic` for `dyn`), 4 MAY_PANIC (Charon). +- `cg_app`: `run` → `area_of`, `area_dyn`, `double` and the `Square` impl's `area` as static calls, all cross-crate. + +What the two frontends disagree on, and who is right: +1. **The call inside `assert_eq!`.** Charon witnesses `checked → triple` (through the alias `t3`). rust-analyzer's call hierarchy misses it; its index records only the reference. Charon is right: the call is compiled. +2. **Compiled-out code.** rust-analyzer records a reference to `double` at `cg_lib/src/lib.rs` 261..267, inside `#[cfg(any())]`, and Kythe serves it as a live reference. Charon has no such item. rustc is the authority: the reference does not exist in the build. Any reader of rust-analyzer references must drop those inside its own `inactive_code` diagnostic ranges (here 210..272). + +OPEN from this run: a cross-crate callee is named by Charon as `cg_lib::shapes::Impl::area`, ambiguous by name; the join only covers the local crate, so cross-crate callees still need a span join against the defining crate. + +## The earlier sessions' runs, reproduced here (2026-11-02) + +Before building on D-SAG-6-1/6-2 (#1876, #1878, #1883, #1894), their recorded runs were repeated in this container with the same tools: + +| Run | Recorded | Here | +|---|---|---| +| `ident_chain.py` on `sag_ident` A→B, A→B declared, A→C, A→D | AMBIGUOUS; DECLARED (`sid-6` continues); MOVED via resolve; REBOUND | identical | +| Kythe on `sag_fixture` | 138 rows, 591 entries, 4 `Attribute` skipped, 24/24 symbols, 3 pins fail (all inside `assert_eq!` in `tests/law.rs`), exit 1 | identical | +| Charon + `span_join.py` on `lance-graph-mask-risc` | 459 bodies: EXACT 140, ENCLOSED 186, DERIVED 123, UNJOINED 10; impls paired 175, ra-only 0, charon-only 205 | EXACT **141**, ENCLOSED 186, DERIVED **122**, UNJOINED 10 (1 `const _`, 9 drop glue); impls identical | + +The one difference: the derived `PartialEq::eq` of `Operand` (`ir.rs:6`) and of `Tern2Fold` (`ir.rs:1533`) now join EXACT, because rust-analyzer reports a symbol for the derive-generated method whose name sits on the `#[derive(...)]` line and EXACT is tried before DERIVED. The joined item is the same; only the class changed. `mask-risc`'s source has not changed since #1883, so the cause is a tool or frontend version, not determined from the record. Measured here: whole-workspace rust-analyzer dump 9.2 s load + 134 s index (1,491 files, 1,144,242 occurrences); Charon on `mask-risc` 33 s build, 441 MB LLBC (`--no-dedup-serialized-ast`); `span_join.py` 63 s. + +Not rerun: the contract crate (7,189 bodies). At 441 MB of LLBC for 459 bodies, the Python join loading the file whole is the limit, which is one concrete reason the intake belongs in Rust over `charon_lib`'s typed reader. + +## Slice A: `ruff_charon` (ruff `53c6711a84`, branch `claude/clone-repositories-71a5sw`) + +A ruff crate (excluded from the ruff workspace, like `ruff_r2il`) reading LLBC through `charon-lib` from the AdaWorldAPI fork on stable 1.99. Three outputs: +- `calls`: **byte-identical to `llbc_calls.py`** on `cg_lib` (12 lines), `cg_app` (2) and `lance-graph-mask-risc` (460 lines, 441 MB LLBC, 5.6 s release); +- `rows`: `DEFINES` / `CALLS_RESOLVED` / `MAY_DISPATCH` / `UNRESOLVED_CALL`, grade `COMPILER`, byte spans, residuals, and a receipt `harvested == represented + residual`. `mask-risc`: 16,149 = 16,149 + 0 (459 DEFINES, 15,684 CALLS_RESOLVED, 2 MAY_DISPATCH, 4 UNRESOLVED_CALL); +- `kythe`: `defines`, `ref/call`, `childof` and file nodes. Through Kythe v0.0.76 `/xrefs`, `tools/kythe_calls_check.py` with hand-read pins (`fixtures/codegraph/rev1_calls.pin.tsv`): 3/3, including `t3(1)` inside `assert_eq!` and the bound dispatch `s.area()` pointing at the trait. Without file nodes Kythe returns no references and all 3 fail. + +Closed by this slice: the cross-crate callee ambiguity. Charon's formatter names impls (`cg_lib::shapes::impl_Shape_for_Square::area`), so `cg_app::run`'s callee is the `Square` impl, not a path shared by two items. + +Tests (6 integration, 3 unit) on the fixture's committed LLBC, with `llbc_calls.py`'s output as the reference. Disable runs, each red in exactly its own test: bound dispatch counted as resolved; no file nodes; callee id dropped from the report; impl names by path only. + +Not yet: rust-analyzer as an independent check of the rows (the comparison is still the Python span join), resident storage (slice B), body → R2IL (G4). diff --git a/.claude/board/entries/README.md b/.claude/board/entries/README.md index 211bbc773..98d9b3ffd 100644 --- a/.claude/board/entries/README.md +++ b/.claude/board/entries/README.md @@ -25,10 +25,11 @@ index row, (3) no duplicate entry id. Checks 1 and 2 are deliberately opposite directions; the stranding this convention prevents shows up in exactly one of them, never both. -542 entries, 2026-08-06 .. 2026-11-01. +545 entries, 2026-08-06 .. 2026-11-02. | date | entry id | finding | file | |---|---|---|---| +| 2026-11-02 | `D-CODEGRAPH-0` | | [2026-11-02-d-codegraph-0-gap-map.md](2026-11-02-d-codegraph-0-gap-map.md) | | 2026-11-01 | `D-XLOG-NATIVE-7` | | [2026-11-01-xlog-resident-hypercube-context.md](2026-11-01-xlog-resident-hypercube-context.md) | | 2026-11-01 | `D-XLOG-NATIVE-8` | | [2026-11-01-xlog-behaviour-lens-f100.md](2026-11-01-xlog-behaviour-lens-f100.md) | | 2026-11-01 | `D-DT64-7-5` | | [2026-11-01-maptable-versioned-local-map.md](2026-11-01-maptable-versioned-local-map.md) | @@ -85,12 +86,14 @@ exactly one of them, never both. | 2026-10-28 | `D-DRCA-1` | | [2026-10-28-d-drca-1-hhtl16-dr-projection.md](2026-10-28-d-drca-1-hhtl16-dr-projection.md) | | 2026-10-27 | `harvest-integration-composite-key-and-simd-scope` | | [2026-10-27-harvest-integration-composite-key-and-simd-scope.md](2026-10-27-harvest-integration-composite-key-and-simd-scope.md) | | 2026-10-26 | `D-ONNX-INTAKE-0` | | [2026-10-26-onnx-frontend-intake-bakeoff.md](2026-10-26-onnx-frontend-intake-bakeoff.md) | +| 2026-10-26 | `ledger-revision-1727-1784` | | [2026-10-26-ledger-revision-1727-1784.md](2026-10-26-ledger-revision-1727-1784.md) | | 2026-10-26 | `D-DT64-0` | | [2026-10-26-dt64-holographic-interface.md](2026-10-26-dt64-holographic-interface.md) | | 2026-10-26 | `D-DR64C-A` | | [2026-10-26-dr64-cubed-amendment.md](2026-10-26-dr64-cubed-amendment.md) | | 2026-10-26 | `D-DR256-WHERE` | | [2026-10-26-dr256-where-capstone.md](2026-10-26-dr256-where-capstone.md) | | 2026-10-26 | `D-DR256-OD` | | [2026-10-26-dr256-on-demand-l1.md](2026-10-26-dr256-on-demand-l1.md) | | 2026-10-26 | `D-DR256` | | [2026-10-26-dr256-global-union-lab.md](2026-10-26-dr256-global-union-lab.md) | | 2026-10-26 | `D-DQM` | FAIL | [2026-10-26-dqm-p1-chunk-touched.md](2026-10-26-dqm-p1-chunk-touched.md) | +| 2026-10-26 | `D-DQM-M1` | | [2026-10-26-dqm-m1-timed-arm.md](2026-10-26-dqm-m1-timed-arm.md) | | 2026-10-26 | `D-TFLP-V2` | | [2026-10-26-d-tflp-v2-synergy-revision.md](2026-10-26-d-tflp-v2-synergy-revision.md) | | 2026-10-26 | `D-TFLP-EVIDENCE-1` | | [2026-10-26-d-tflp-tanh-proof-learning-tract-harvest.md](2026-10-26-d-tflp-tanh-proof-learning-tract-harvest.md) | | 2026-10-26 | `D-TFLP` | | [2026-10-26-d-tflp-65-landing.md](2026-10-26-d-tflp-65-landing.md) | diff --git a/.claude/plans/d-codegraph-0-resident-codegraph-v1.md b/.claude/plans/d-codegraph-0-resident-codegraph-v1.md new file mode 100644 index 000000000..637aea111 --- /dev/null +++ b/.claude/plans/d-codegraph-0-resident-codegraph-v1.md @@ -0,0 +1,124 @@ +# D-CODEGRAPH-0 — a resident Codegraph: Charon → ruff → lance-graph + crai-rs + +**Date 2026-11-02 · Status PROPOSED; slice A in progress · Board: [STATUS_BOARD](../board/STATUS_BOARD.md) D-CODEGRAPH-0 · Entry: [gap map](../board/entries/2026-11-02-d-codegraph-0-gap-map.md)** + +**Goal.** Import Rust source once into a persistent, queryable, incrementally +maintained Codegraph, so coding agents query established facts instead of +reconstructing them by text search. Correctness gates come before any agent +interface; MCP is last. + +**Representation (operator, 2026-10-31, restated 2026-11-02).** Kythe +semantic facts + R2IL executable semantics + DT64/DR binding. Quack, +mask-risc and relalg are polyfill underneath R2IL, not the representation. +Route: **Charon → ruff → lance-graph + crai-rs.** + +| Layer | Owner | Carries | +|---|---|---| +| Rust frontend | Charon (`AdaWorldAPI/charon`, `charon-lib` stable with `--no-default-features`) | LLBC: items, spans, bodies, resolved calls, trait-ref kind per call | +| Intake | ruff (`ruff_charon`, new) | LLBC → Kythe facts (nodes, anchors, edges) with a witness status per edge | +| Executable semantics | `ogar-r2il` (vocabulary) + `lance-graph-r2il` (`FoldDialect`, executes) | method bodies as R2IL words; never pre-converted to SPO (OGAR ruling 2026-08-26) | +| Binding | `lance-graph-dt64` (`MethodBinding`, `BindStore`, DR `MapTable`) | method identity, admission, global → local maps | +| Query / execution polyfill | Quack → mask-risc, `lance-graph-xlog` cube | populations, closures, counts | +| Review semantics | crai-rs (CR-H0 review semantics, CR-H1 falsifiers) | finding record, evidence grade, lifecycle | +| Independent oracle | rust-analyzer (`sag-ra-probe GRAPH=1`) + rustc (`cargo check`) | a second frontend to compare against; never the ingestion path | + +## 1. Gap map (verified 2026-11-02 against `main`) + +**Exists:** +- `sag-ra-probe` (#1876, #1878, #1883, #1894, lab, excluded): rust-analyzer + facts; Charon run out of process with call classification in Python + (`tools/llbc_calls.py`); span join of the two frontends; identity that + survives a witnessed move; Kythe entries from rust-analyzer facts with + `/xrefs` parity 24/24. +- `lance-graph-r2il` (#1895, #1897, #1899, #1902–#1904): `FoldDialect` + executes the R2IL fold and arithmetic band over mask-risc; polyfill lens + `0xFFFF::0xF000`. +- `ogar-r2il`: the R2IL arity table (82 ops, `0x90..=0xE1`) plus the fold band. +- `tract-harvest-probe` (#1876): the codegraph row layout (4+12 key per + node / value / use site, one edge row per use, attributes in value columns). +- `dt64-admission-probe` (#1891): consumes DEFINES / CALLS / effect evidence + rows to decide whether a method may execute. +- `lance-graph-xlog` cube (#1907, #1911–#1913): relations, joins, closure. +- crai-rs: review semantics and falsifier matrices (docs only; nothing run). + +**Missing, in pipeline order:** + +| # | Gap | Evidence | Slice | +|---|---|---|---| +| G1 | No Charon intake in ruff (no branch mentions Charon or LLBC) | ruff `origin/main` + every remote branch, 2026-11-02 | A | +| G2 | No Kythe producer in Rust; the only one is `sag-ra-probe/tools/kythe_entries.py`, fed by rust-analyzer | `git grep` over lance-graph, OGAR, ruff | A | +| G3 | No Kythe `ref/call` edges anywhere: rust-analyzer gives references, and a reference is not a call | `sag-ra-probe/KYTHE.md` "Not produced" | A | +| G4 | No Rust-body → R2IL lowering; R2IL arms are machine code (`ruff_r2il`, r2sleigh) and fold/arithmetic (`lance-graph-r2il`) | crate descriptions + sources | C (design check first) | +| G5 | No resident store or incremental update for code facts | no crate holds them | B | +| G6 | Closure in the xlog cube refuses more than 64 positions per tile; frontier masks may not feed another program (`graph_hop` quarantined, ISS-GATHER-HOP-1616) | `cube.rs` `closure`, `graph_hop.rs` header | C | +| G7 | Method body anchor is open: `ogar-loco::Program` is not the body (operator correction 2026-10-31) | `2026-10-31-coresearch-mandatory-codegraph.md` | C | +| G8 | The ruff → lance-graph sink is unmerged (`RefinedTruthSink`, ruff branches `claude/r2il-pr2-*`) | `git ls-remote` ruff | B | +| G9 | `sag-ra-probe` did not build from a fresh resolution (`unicode-ident` 1.0.24 vs `unicode-properties` 0.1.5) | build log 2026-11-02; fixed by pinning `unicode-properties = "=0.1.4"` | done | + +## 2. Slices + +**A — intake for one real crate.** `ruff_charon`: read LLBC with +`charon_lib::deserialize_llbc`, emit Kythe entries (anchors with byte +offsets, `defines/binding`, `ref`, `ref/call`, `childof`, `extends`) and a +witness status per call: `RESOLVED` (`Fun`, `TraitImpl`), `POSSIBLE` +(`Clause`, `Dyn`), `UNKNOWN` (function pointer). Checked against the +rust-analyzer oracle on `sag-ra-probe/fixtures/codegraph` and against rustc. +Then one real crate (`lance-graph-mask-risc`, measured by Charon before: +459/459 bodies). + +**B — resident and incremental.** Rows reach lance-graph through the ruff +sink seam (G8) and are keyed per the `tract-harvest-probe` layout. Per-file +content digests; an update invalidates the changed files plus the files that +reference symbols defined in them; parity test: clean import equals +incremental import as sets. + +**C — queries, diagnostics, bodies.** Callers, callees, references, impact +through the xlog cube and Quack; tiled closure for more than 64 positions +(G6); structural diagnostics with crai-rs's finding record. Body lowering to +R2IL only after its own design check (G4, G7). + +**D — local MCP.** A thin adapter over C's query functions, only after the +B and C gates pass. + +**E — token economics.** Same questions, same model, same revision, with and +without the Codegraph; no savings claimed before this is measured. + +## 3. Oracle fixture + +`crates/sag-ra-probe/fixtures/codegraph/{rev1,rev2}` (three crates): +aliases (`use … as t3`), a public re-export (`pub use … as Sq`), cross-module +and cross-crate calls, trait dispatch through a bound and through `dyn`, a +`#[cfg(any())]` item, a reference used only inside `assert_eq!`, unresolved +names (`cg_broken`), a module cycle (`shapes` ↔ `util`), and in rev2 a deleted +function and a new one. + +rustc on rev1 (`cargo check -p cg_lib -p cg_app`): `unused_imports` at +`cg_lib/src/shapes.rs:2` (`triple`), `dead_code` at `cg_lib/src/util.rs:15` +(`private_unused`). The alias `t3`, used only inside `assert_eq!`, is not +reported unused. + +The rust-analyzer oracle is `sag-ra-probe GRAPH=1 NO_MIR=1`, which emits +`file`, `node`, `occ` (with `use` / `pubuse` roles), `call`, `impl`, +`import`, `macro` and `diag` rows. Measured on rev1: 25 nodes, 104 +occurrences, 10 calls, 3 impls, 9 imports, 1 macro marker, 3 diagnostics, +6.6 s wall time. One known gap of the oracle itself: rust-analyzer's call +hierarchy misses the call `t3(1)` inside `assert_eq!`, while its index does +record the reference. + +## 4. Gates + +- Every resolved edge traces to a frontend fact; nothing is inferred from + names. +- `POSSIBLE` dispatch is never counted as a confirmed caller. +- An import is reported unused only where the file has no macro call with + arguments and the import is not a trait, a glob or a re-export; otherwise + the finding is `POSSIBLE` at most. +- Clean-import and incremental-import results are equal. +- Each guard has a disable run that turns its test red. + +## 5. Open + +- Where the method body revision is anchored (G7). +- Whether R2IL can carry Rust places, drops and panics, or needs a separate + reading (G4). +- Merge order of the ruff sink branches (G8). diff --git a/.claude/research/2026-10-recovery/LEDGER-REVISION-2026-10-26f.md b/.claude/research/2026-10-recovery/LEDGER-REVISION-2026-10-26f.md new file mode 100644 index 000000000..97fd1cc66 --- /dev/null +++ b/.claude/research/2026-10-recovery/LEDGER-REVISION-2026-10-26f.md @@ -0,0 +1,199 @@ +# Ledger revision, 2026-10-26 (f): #1727–#1784, ndarray #368–#376 + +Maps the merges since revision (e) (`LEDGER-REVISION-2026-10-23e.md`, #1635–#1726) +onto the open list, and folds in one more list: the open items a session +carried from 2026-09-18 (the five "doors", the hop into quack, the 64k slab, +the DuckDB differential, Mississippi Queen, the duplicate 34-recipe vocabulary, +four doc corrections). Revision (e) and its predecessors are not edited; this +file supersedes the rows it names. Board entry: +`.claude/board/entries/2026-10-26-ledger-revision-1727-1784.md`. + +**Scope.** lance-graph: 61 first-parent commits, #1727–#1784, `main` at +`18fbe43a`. ndarray: 8 PRs, #368–#376, `master` at `195844bd` (revision (e) +read `37a3b59`). OGAR: none (`68cb29b`, same as revision (e)). Method as in +(e): a script extracted per-PR facts (status, top directory, new files); seven +Haiku runs (five `harvest-scout`, one `quote-harvester`, one re-check scout) +over named search spaces. Every file:line below was read on the main thread +at `18fbe43a` unless marked *navigation only* (a search hit, not read). + +## 1. The 2026-09-18 list, revised + +| id | item (2026-09-18) | read at `18fbe43a` | now | +|---|---|---|---| +| S1 | **Duplicate 34-recipe loco vocabulary**: `ogar-loco::nars::NarsVocabulary` and `lance-graph-ogar::recipe_vocab::RecipeVocabulary` both claim `DOMAIN_FLOOR + id − 1` | both exist and still disagree: `nars.rs:142-147` arity 2 for `Datapath`, 1 for `Gate`/`Control`, no value codebook; `recipe_vocab.rs:142-144` arity 1 for all, `:171-176` `CB_ATTENTION_FOCUS`. `NarsVocabulary`'s only hits are its own definition. `lance-graph-ogar` is an EXCLUDED crate with its own workspace and pulls `ogar-loco` at `branch = "main"` (`Cargo.toml:140`) | OPEN, latent (same as 09-18). Needs a ruling on which arity is right (Q7) | +| S2 | **The hop into quack** (`src_mask → edge lane → dst_mask`), "PR5" | landed three ways. (a) fk semijoin: `Filter::semijoin` lowers to `MaskOp::Gather` (`quack/src/lib.rs:2296, 2390`); mask-risc's own doc says *"`hop` (PR5) LANDED as `MaskOp::Gather` … and `Terminal::ScatterOrU32`"* (`mask-risc/src/lib.rs:24-27`). (b) Edge-table one-hop in `lance-graph-cypher-quack` (`lib.rs:552` `one_hop`, re-anchored on the edge table), #1376. (c) `OutgoingHop` (#1616), quarantined behind the off-by-default feature `quarantined-gather-hop` (`mask-risc/src/lib.rs:132-137`) | **Reframed.** One hop exists. Two or more hops are refused (`cypher-quack/src/lib.rs:583-587`: *"the walk count needs a foreign-value sum"*). What remains is a lawful multi-hop, which is revision (e)'s RECURSION + GATHER-CODE. Merged into those rows | +| S3 | Phase 7: the 64k slab benchmark | D-QCK-6 *"the 64k vertical slice across five arms"* Shipped (`18c1d85`, STATUS_BOARD:1156); `count_probe` 4855 at 64k on four arms, 0 B per execute (STATUS_BOARD:1199); the lane-fold capstone pins the numbers (605 / 626 / 636 ns); freeze@64k measured (#1276) | **CLOSED.** Its remaining edge is the capstone's points 3 and 6 (tile skip, single-tile probe), §3c | +| S4 | Phase 8: DuckDB semantic differential + critical-falsifier report | semantic differential shipped: `quack/tests/duckdb_differential.rs` + `tests/duckdb/{fixture.rs,oracle.py}` (#1256); PostgreSQL K3 / NULL parity (#1668, #1677, #1682) | **Semantic half CLOSED.** OPEN: a timed DuckDB column (capstone point 5); D-CML-8, the DataFusion support differential, Queued (STATUS_BOARD:1185) | +| S5 | Mississippi Queen (W2): the weighted arm | entry `2026-09-25-window-scheduling-and-two-level-ternlog.md`: *"Mississippi-Queen windows lose; a two-level ternlog fold wins 1.7–1.8× for 4–5 planes"*; Tern2 shipped (#1287). `causal-plane-inventory.md:265`: no implementation | **CLOSED, rejected by measurement.** Tern2 is what replaced it | +| S6 | EPIPHANIES entry (16) missing the `−32` correction | `EPIPHANIES.md` is now a 355-line index (split 2026-09-20); the prose is in `EPIPHANIES-ARCHIVE-2026-09-20.md` | **EXCLUDED.** Historical prose is frozen by the 2026-09-20 closeout rule (`CLAUDE.md` § Closeout); the code (`facet.rs`) carries the correction | +| S7 | `three-prefix-fold-carriers.md` "A wins everywhere" | still `:122`; `:129-131` call depth 5 "a tie, not a win". One later reproduction (three runs per arm) had C ahead of A at depth 5 / identical (C 5.22 / 5.22 / 5.22 against A 5.59 / 5.69 / 5.59) | OPEN, low. The claim needs a host and run count beside it; the sign of the depth-5 result is not settled | +| S8 | ndarray #317: G8's `>> 2` justified by `4⁴` (which justifies `>> 1`), and a falsifier that cannot catch a wrong shift | `masking-ops-state.md:225-237` names both factorizations and says *"Do not cite 4⁴ as the justification for `>> 2`; it justifies `>> 1`"*; `:282-287` adds the third arm with an independently planted divergence | **CLOSED** (corrected 2026-09-18) | +| S9 | `LATEST_STATE.md` says `ruff#115` is still open | `:1017` unchanged; ruff#115 was closed 2026-09-18 and its 8 docstrings landed as ruff `1b26e67` | **EXCLUDED.** Dated append-only entry; recorded here instead of edited | +| S10 | The five doors | (1) the facet as one register: `shared_prefix_tiles` is one `xor` + `tzcnt` over `as_u128()` with the canon/custom tile swap (`facet.rs:327-338`); the 16-byte facet match predicate shipped (#1326, `ternary_match_strided16_to_mask`). (2) `TruthU8` in rail slots: no code (*navigation only*: 0 hits in contract, mask-risc, ndarray simd/hpc). (3) `pack`: ndarray `simd_masking_ops.rs:4688` `fn pack`, the one predicate tail (#313). (4) generation-keyed mask tile: no code under that name; D-DRF's facet-keyed focus with eviction is the nearest thing (K26). (5) random-access width: the `u16` address is settled by the lane-fold capstone; gather along the `u16` map exists in the D-DRF lab | (1) and (3) **CLOSED**. (2) OPEN, unbuilt. (4) OPEN, partly met by D-DRF (K26). (5) **CLOSED as a ruling**, its executor is lab-only | +| S11 | Decisions 1 / 1b / 3 (alpha channel first; quack as the spine; belief needs proper wiring) | alpha: `contract::alpha` + `spog_tenants` shipped; D-SPOG-W-0 / D-ALPHA-G-0 (#1371, #1372). Quack: the Cypher → Quack seam exists (#1376), `cypher-mask-lowering-v2` is a PROPOSAL. Belief: `RevisionKind` (`contract/src/revision.rs:199`), `BeliefArena` (`planner/src/nars/belief.rs:124`); `Opcode::Revision` pools a duplicated source (TD-NARS-REVISION-UNGUARDED-CAUSAL-EDGE-ISA-1, #1733) | Alpha **landed**. Quack **landed as a seam**, multi-hop open (S2). Belief **merged into** the TD row: "proper wiring" now has a measured defect to start from | +| S12 | The #1240 post-merge board commit | written 2026-09-18 (`d52cc2086`), never merged; by 2026-10-26 it was about 500 PRs behind `main`, and its LATEST_STATE / PR_ARC entries described a state long superseded | **EXCLUDED.** The branch was restarted from `main` without it. Its one live finding, the duplicate vocabulary, is S1 and has a fresh ISSUES row | + +## 2. New closed ends (C73–C84) + +| id | what closed | evidence | +|---|---|---| +| C73 | **D-DRF lab arc:** immutable DR64 focus lease (P1), focus-vs-direct break-even (P2), focus keyed on its facet (G), staged writeback coalesced per canonical row (W1), ordered same-facet apply (W3), flushed row images handed to the real `CycleWriter` (W4), eviction to a resident budget (W8), MRU vs LRU under churn, gather cost against distinct pages (TLB) | #1762–#1774; entries `2026-10-24-drf-*`, `2026-10-25-drf-*`. TEST-PINNED, lab; the W8 entry says no CI job runs the crate | +| C74 | D-DRAV-AV1: a map-only membership mask over a bound DR64 map | #1776; `2026-10-25-drav-av1-membership.md` | +| C75 | `lance-graph-onnx`: ONNX operator names as aliases of native ndarray functions; `Tanh` / `Exp` as ULP-bounded aliases (`onnx::bounded`) | #1754, #1758; `lance-graph-onnx/src/lib.rs:64-92` re-exports `ndarray::hpc::vml::{vsabs, vssqrt, vsexp, vstanh}` and `ndarray::simd::*_i32_to_mask` | +| C76 | ndarray numerics across every backend: `exp` / `tanh` within 1 ULP (#368, #369), `mul_add` rounds once (#374), saturating narrows down the 512-bit ladder (#372, #373), TFLite / gemmlowp requantization bit-exact (#375), Codec8 phase a (#370), `masked_sum_i32` dense-branch-free / sparse-walk hybrid (#376) | ndarray PR titles; C75 consumes #368/#369 | +| C77 | D-NFR-6: native operations over a `SoaEnvelope` borrow, no table, no runtime frame guard | #1755, `crates/soa-view-probe`, CI job `soa-view-probe.yml` | +| C78 | D-MPF-1: the Stage 1 math polyfill already exists; one expression, oracle, refusals, cost split | #1752, `crates/math-polyfill-probe`, CI job `math-polyfill-probe.yml` | +| C79 | D-DRMC L1: a D-DRMC-5 certificate reused on a fresh frame (X8 partly); the certificate keeps egglog's `ProofStore` + `ProofId`, not its rendering | #1740, #1742 | +| C80 | Thought-git squash: chain-of-origin squash into the most-updated lane, gated on contradiction, blame survives; protected readers of a squash (D-TGA-5D) | #1736, #1739 | +| C81 | EXO-129: a learned 256×256 table does not compose where the codebook is not closed → torque T-2026-10-23-1 KILLED | #1731 | +| C82 | Board ledgers: `BREADCRUMBS.md`, `TORQUE.md`, `GOLDSTAUB.md`, generated `FRUITS.md` (`fruits_index.py`, CI-gated) | #1729 | +| C83 | Board catch-ups through #1758 (arc, INTEGRATION_PLANS, LATEST_STATE, stale In-PR rows) | #1727, #1744, #1759 | +| C84 | bgz-tensor: symmetric signed-i8 probe and Fisher-Z error decomposition (lab, no wire change) | #1757. The silent clamp itself is unchanged (`fisher_z.rs:67`), so ISS-FISHERZ stays open | + +**Plans with no code in the window:** D-NAM-0..5 + PZL (#1778–#1784, 7 PRs), +D-ONNX-RELALG-CAP S0–S5 + convergence (#1744–#1751, 7 PRs, two competing +capstones: #1745 is a HOLD on #1744 vs #1750), D-DTM-1 (#1777), D-SAG-6 +(#1760), D-DRF-1 (#1756), D-NFR / D-NFC (#1753), D-DRAV-1 (#1775). About 22 +plan files against five new code crates. + +## 3. Open items, revised + +Fruit: **L** low, **H** high, **Heavy**. Tier: O = operator decision, S = +Sonnet, H5 = a Haiku card (named). + +### 3a. The six decisions of 2026-10-20: still not executed + +Re-checked at `18fbe43a` by one scout; `action.rs:266, 517`, +`lance-graph/Cargo.toml:55-58` and `supervisor.rs:203-212` re-read on the main +thread. No PR in the window touched any of them. + +| id | status at `18fbe43a` | fruit | tier | +|---|---|---|---| +| ACTION-GATE | `commit` (`action.rs:266`) unchanged; `commit_via_no_required_role_proceeds_regardless_of_rbac` (`:517`) still pins fail-open; no `Public`/`Restricted` declaration in `action.rs` (the `Marking::Restricted` in `property.rs:782` is a data marking, a different concept) | H | S | +| FLOAT-REFS | `lance-graph-r2il` unconditional (`Cargo.toml:58`), which pulls OGAR `branch = "main"` (`lance-graph-r2il/Cargo.toml:19-20`); no `fold` feature in any manifest | L–H | H5 audit, then S | +| WIRE-50 + ISA-LEGACY | `Maturation3` only in `planner/tests/maturation_ladder.rs`; no `ISA_LEGACY` / `legacy_opcode` / `LegacyRead` / `activation_i4` (*navigation only*) | H | S, one PR (K20) | +| O30 / PROVENANCE | the only `struct SourceRef` is `lance-graph-report/src/plan.rs:284` | Heavy | S after K21 | +| ADMISSION code half | GATHER-CODE classification not started | H | H5 quotes, then S | + +### 3b. Carried, unchanged + +SUP-DISPATCH (`supervisor.rs:203-212`), SUP-CAPACITY (`:42`, `:389`), +SUP-DRAIN (`drain.rs:97`); DURABLEREF (still doc comments only; `BaseRef` +stand-in now 29 lines in `thought-git-probe`); D-CGW-4 (`counterfactual_verdict` +callers only in `pearl.rs` tests); ISS-BF16-POLYFILL-PER-ELEMENT-ALLOC +(ndarray `simd_ops.rs:640`, `vec![0.0f32; k]` inside the `j` loop, 256 times +per call); ISS-FISHERZ (`fisher_z.rs:67`); TD-NARS-REVISION-UNGUARDED +(`causal-edge/src/isa.rs:197, 214`); ISS-GATHER-HOP-1616 (quarantined, CI tests +it under its feature: `rust-test.yml:130`); RECURSION; D-CMR-5; CERT-COMPILER; +DRMC-PROBES (X6, X7, X16, X18; X8 partly done, C79); FRAME-SEAL-OWNER; +PAL256-G5; THOUGHT-GIT-END; TUPLE-SITES; everything revision (e) marked +*carried, not re-checked*. + +### 3c. The lane-fold capstone's ten points (`.grok/board/CLAUDE_LANE_FOLD_CAPSTONE.md`, 2026-10-05) + +Not re-read item by item. Navigation over `lance-graph-quack` and +`lance-graph-mask-risc` found no `collapsed_terminals`, no `lane_guard`, no +aperture cap symbol, no `Range` `_under` twin and no timed DuckDB column, and +one `dead_tile` hit in an example (`fold_join_probe.rs`). Read as still open, +in the capstone's own order: collapse rewriter (1), guardrail on the build (2), +tile skip (3), `Pred::Range` as a `u16` span with an `_under` twin (4), timed +DuckDB column (5), single-tile probe (6), K knee as a refusal (8), scattered +extract (9), aperture cap (10). Point 7 is a citation rule, not work. + +### 3d. New open items (from #1727–#1784) + +| id | item | source | fruit | tier | +|---|---|---|---|---| +| **DRF-ZERO-COPY-EXCEPTION** | D-DRF-1 records a DECISION (2026-10-24): the DR64³ foreign-key cache is *"a scoped exception to the zero-copy law"*. `CLAUDE.md` says *"There are no boundaries"* and has no pointer to the exception | `entries/2026-10-24-d-drf-1-focus-lease-foreign-cache.md:6` | H | O (where it is written), then H5 spec-editor | +| **DRF-NO-CI** | `drf-focus-probe` has no CI job; its tests are TEST-PINNED by hand only | W8 entry, Status line | L | S (workflow file) | +| **DRF-EVICTION** | LRU is the worst policy under a cyclic working set one larger than the budget (30 builds in 30 loads); real thought workloads are not measured | W8 entry, OPEN | H | S (probe) | +| **CAPSTONE-COMPETITION** | two ONNX-RELALG capstones (#1744, #1750) with #1745 as a HOLD, plus D-NAM-0 as a third capstone for the same membrane | #1744, #1745, #1750, #1778 | — | O | +| **HOP-DOC** | `mask-risc/src/lib.rs:126` says the hop *"is PR5's and is declared when it lands"*; `:24-27` in the same file says it LANDED | read | L | H5 spec-editor | +| **ONNX-ALIAS-GAP** | ndarray #372, #373, #375 (saturating narrows, TFLite requantization) have no alias in `lance-graph-onnx` yet; D-NAM-2 plans exactly those leaves | C75, C76, `d-nam-2-onnx-tflite-leaf-dto-v1.md` | L–H | S | + +## 4. Connecting the dots + +- **K26 — D-DRF already implements two of the lane-fold capstone's open + points, under different names.** Capstone point 10 asks for a cap on + distinct apertures (*"a million masks at 8 KB are 8 GB"*); D-DRF-W8's + `FocusManager::evict_to_budget` is a resident-bytes cap over gathered, + facet-keyed foci, with disables. Point 9 asks to extract `u16`s only when + the dead-word fraction is low; ndarray #376 makes `masked_sum_i32` sum + dense words branch-free and walk sparse ones, and D-DRF's fold calls + exactly that kernel (P1 entry line 8). Neither document cites the other. + Linking them turns two open points into "adopt W8's budget as the aperture + cap" and "measure the #376 crossover on lane workloads". +- **K27 — admission has a fifth instance.** Revision (e)'s K22 listed four + layers asking "what may be built before execution, under what witness". + D-DRF-1 adds a fifth and is the first to call itself an *exception*: a + gathered column (16 KiB per focus) built under a lease over a sealed map, + with dirty rows written back through `CycleWriter`. That is the strongest + case yet for one admission vocabulary (witness, lease, promotion refused), + because otherwise every lab that needs a cache will name its own exception. +- **K28 — the one cross-repo seam in the window that is wired end to end is + ndarray numerics → `lance-graph-onnx`.** It is pure re-export (an alias + where the contract is identical, an adapter where it is not, a refusal + otherwise), so a numeric fix in ndarray reaches the ONNX membrane with no + code in lance-graph. #375 and #372/#373 are the next aliases (ONNX-ALIAS-GAP). +- **K29 — the hop was never missing; multi-hop is the RECURSION row.** S2's + "missing leg" is now three implementations, one quarantined. The + edge-table re-anchor in `cypher-quack` reads each edge row once and needs + no foreign value. Every refusal beyond one hop names the same missing + thing (a foreign-value sum), which is revision (e)'s RECURSION driver. + Item S2 and RECURSION are therefore one item. +- **K30 — the queue grew while planning continued.** Revision (e)'s K19 said + the six rulings needed execution, not decisions. In the following 61 + commits none was executed, and about 22 plan files were added. Execution, + not planning, is still the cheapest high-signal work in the repository. + +## 5. Fruits + +### Low-hanging (Haiku-led where mechanical; verified by the orchestrator) + +1. **HOP-DOC** — `spec-editor`, one anchored replacement at `mask-risc/src/lib.rs:126`. +2. **S7** — `spec-editor` adds a host / run-count clause after `three-prefix-fold-carriers.md:131`; the wording comes from the orchestrator. +3. **FLOAT-REFS consumer audit** (carried) — `harvest-scout`: every `lance_graph::fold` import in the workspace and sibling repos. +4. **GATHER-CODE quotes** (carried) — `quote-harvester` over `quack/src/lib.rs:2077, 2092, 2296, 2390`, `mask-risc/src/exec.rs` (the `MaskOp::Gather` arm), `reference.rs:437, 822, 922`. +5. **ROWS-MISSING** (carried, grown) — `census-counter`: every D-id in `INTEGRATION_PLANS.md` and `.claude/plans/` with no STATUS row, now including D-NAM-*, D-ONNX-*, D-DRF-*, D-DRAV-*. +6. **DRF-NO-CI** — one workflow file (Sonnet). + +### High-signal (one Sonnet slice each, test first, decision already made) + +1. **ACTION-GATE** — flip the pinning test to expect denial; add the declaration; route or deprecate `commit`. +2. **WIRE-50 + ISA-LEGACY** in one PR (K20). +3. **FLOAT-REFS**, after fruit L3. +4. **SUP-CAPACITY → SUP-DRAIN → SUP-DISPATCH**. +5. **BF16 polyfill** in ndarray: transpose B once outside the loops; 256 allocations per call become zero. +6. **Capstone point 1, the collapse rewriter** with its 1,000-sum test, then point 3 (tile skip). The capstone's own order of work. +7. **K26 link**: adopt W8's budget as capstone point 10's aperture cap. +8. **ONNX-ALIAS-GAP**: alias #375 / #372 / #373 in `lance-graph-onnx`. +9. **D-CGW-4** promotion gate. + +### Heavy (needs a placement or scope decision first) + +1. **K21**: `DurableRef` → `SourceRef` → commit-witness anchor, at D-TGB-7. +2. **RECURSION / multi-hop** under ADMISSION B (K29). +3. **One admission vocabulary** (K22 + K27). +4. **CERT-COMPILER**, **PAL256-G5**, **THOUGHT-GIT-END** (carried). + +## 6. Questions for the operator + +Q1–Q6 of revision (e) are unanswered and carried. New: + +| # | question | why it blocks | +|---|---|---| +| Q7 | S1: which arity is right for a recipe op, 2-for-`Datapath` (`nars.rs`) or 1-for-all (`recipe_vocab.rs`)? | one of the two `impl Vocabulary` must delegate to the other; the change spans OGAR and lance-graph | +| Q8 | Where is the D-DRF-1 zero-copy exception written, beside "There are no boundaries" in `CLAUDE.md` or only in its entry? | DRF-ZERO-COPY-EXCEPTION; K27 | +| Q9 | Which capstone governs the ONNX / native-method membrane: D-ONNX-RELALG-CAP (#1744 or #1750) or D-NAM-0? | CAPSTONE-COMPETITION | + +## 7. Agent ledger (Haiku harvest tier) + +| card | brief | result | checked by the orchestrator | wrong | missed | +|---|---|---|---|---|---| +| harvest-scout | duplicate recipe vocabulary | DONE | `nars.rs:128-175`, `recipe_vocab.rs:134-185` read | 0 | 0 known | +| harvest-scout | quack hop surface | DONE | `mask-risc/src/lib.rs:18-30, 108-150`, `cypher-quack/src/lib.rs:545-600` read | 0 | did not search `lance-graph-sap` / dir-sim (outside its space, stated) | +| quote-harvester | four doc corrections | DONE | quotes used as given; `LATEST_STATE.md` read only at `:1005-1024` (file too large) | 0 | 0 | +| harvest-scout | 64k / differential / Mississippi | PARTIAL | STATUS rows 1156, 1185, 1199 cited from its quotes | 0 | 36 STATUS lines truncated by the tool; plan Status lines not extracted | +| harvest-scout | five doors | DONE | `facet.rs:300-340`, ndarray `simd_masking_ops.rs:4676-4700` read | 0 | zero-hit surfaces reported as zero hits, not read | +| harvest-scout | alpha / belief / Cypher plan | PARTIAL | plan Status lines used as given | 0 | `belief-abi-restoration-v1` Status not found; four ISSUES lines truncated | +| harvest-scout | revision (e) re-check | DONE | `action.rs:262-272, 514-520`, `lance-graph/Cargo.toml:55-59`, `supervisor.rs:203-212` read | 0 | 0 known | diff --git a/.github/workflows/rust-test.yml b/.github/workflows/rust-test.yml index 4ddf76302..37ed8b095 100644 --- a/.github/workflows/rust-test.yml +++ b/.github/workflows/rust-test.yml @@ -142,6 +142,14 @@ jobs: # un-checked LOWERING, not merely an un-run test. - name: Run quack tests run: cargo test --manifest-path crates/lance-graph-quack/Cargo.toml + # lance-graph-dt64 and lance-graph-xlog are workspace members whose tests + # no workflow ran: same reason as the mask-risc and quack lines above. + # dt64: bind_store + dr integration tests; xlog: conformance, cube, lens + # and staunen5. Added after the crates joined, not the day they did. + - name: Run dt64 tests + run: cargo test --manifest-path crates/lance-graph-dt64/Cargo.toml + - name: Run xlog tests + run: cargo test --manifest-path crates/lance-graph-xlog/Cargo.toml # weather-poc is workspace-EXCLUDED. A new workflow introduced by the # same pull request cannot gate that pull request because `pull_request` # workflow selection comes from the base branch. Run both surfaces here diff --git a/crates/lance-graph-quack/tests/duckdb/oracle.py b/crates/lance-graph-quack/tests/duckdb/oracle.py index 605754a00..4a9156ce1 100644 --- a/crates/lance-graph-quack/tests/duckdb/oracle.py +++ b/crates/lance-graph-quack/tests/duckdb/oracle.py @@ -81,8 +81,7 @@ def encode(case_id: str, rows: list[tuple]) -> str: return str(val) -def main() -> None: - con = duckdb.connect(database=":memory:") +def load(con) -> None: con.execute( f"CREATE TABLE partner AS SELECT * FROM read_csv_auto('{DATA / 'partner.csv'}')" ) @@ -93,6 +92,11 @@ def main() -> None: f"CREATE TABLE line AS SELECT * FROM read_csv_auto('{DATA / 'line.csv'}')" ) + +def main() -> None: + con = duckdb.connect(database=":memory:") + load(con) + with open(CASES, newline="") as f: reader = csv.reader(f, delimiter="\t") header = next(reader) diff --git a/crates/lance-graph-quack/tests/duckdb/timing.py b/crates/lance-graph-quack/tests/duckdb/timing.py new file mode 100644 index 000000000..d3cca1067 --- /dev/null +++ b/crates/lance-graph-quack/tests/duckdb/timing.py @@ -0,0 +1,101 @@ +#!/usr/bin/env python3 +"""DuckDB timing arm for the lance-graph-quack differential suite. + +Runs the same cases as `oracle.py` (same fixtures, same `cases.tsv`) and +reports a DuckDB timing column next to each case. Parity comes first: a case +is timed only if DuckDB's encoded result equals the committed `expected`. + +Boundaries, in plain words: + - `t_execute_ns` includes the Python result conversion of `fetchall()`. That + is small for the scalar cases and larger for the `rows_proj` row + projection, which materialises Python tuples. + - `t_prepare_ns` times `PREPARE q AS ` alone; the `DEALLOCATE q` runs + OUTSIDE the timed region (it is cleanup, not part of preparing). + - The numbers are a reference arm: reported, not gated. No timing assertion + exists anywhere in this file. + - DuckDB runs single-threaded by default (`--threads 1`) to match Quack's + single-threaded execution. + +Output goes to stdout only; nothing is written to any file. + +Usage (run OUTSIDE the repo so tests/duckdb/ does not shadow the package): + cd /tmp && /bin/python -I /crates/lance-graph-quack/tests/duckdb/timing.py \ + [--threads N] [--reps R] +""" + +import argparse +import csv +import importlib.util +import pathlib +import statistics +import sys +import time + +import duckdb + +HERE = pathlib.Path(__file__).resolve().parent + +# `-I` removes the script directory from sys.path, so load oracle.py by path. +_spec = importlib.util.spec_from_file_location("lgq_oracle", HERE / "oracle.py") +oracle = importlib.util.module_from_spec(_spec) +_spec.loader.exec_module(oracle) + + +def main() -> int: + ap = argparse.ArgumentParser() + ap.add_argument("--threads", type=int, default=1) + ap.add_argument("--reps", type=int, default=101) + args = ap.parse_args() + + con = duckdb.connect(database=":memory:") + con.execute(f"SET threads TO {int(args.threads)}") + oracle.load(con) + + with open(oracle.CASES, newline="") as f: + reader = csv.reader(f, delimiter="\t") + header = next(reader) + assert header == ["id", "sql", "expected"], f"unexpected header: {header}" + cases = [r for r in reader if r] + + now = time.perf_counter_ns + failed = 0 + for case_id, sql, expected in cases: + got = oracle.encode(case_id, con.execute(sql).fetchall()) + if got != expected: + print(f"DUCKDB case={case_id} parity=FAIL") + failed += 1 + continue + + prep = [] + for _ in range(args.reps): + t0 = now() + con.execute(f"PREPARE q AS {sql}") + t1 = now() + con.execute("DEALLOCATE q") + prep.append(t1 - t0) + + con.execute(f"PREPARE q AS {sql}") + exe = [] + last = None + for _ in range(args.reps): + t0 = now() + last = con.execute("EXECUTE q").fetchall() + exe.append(now() - t0) + con.execute("DEALLOCATE q") + + if oracle.encode(case_id, last) != expected: + print(f"DUCKDB case={case_id} parity=FAIL") + failed += 1 + continue + + print( + f"DUCKDB case={case_id} duckdb={duckdb.__version__} " + f"threads={args.threads} reps={args.reps} " + f"t_prepare_ns={int(statistics.median(prep))} " + f"t_execute_ns={int(statistics.median(exe))} parity=OK" + ) + return 1 if failed else 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/crates/lance-graph-quack/tests/duckdb_differential.rs b/crates/lance-graph-quack/tests/duckdb_differential.rs index d15aaa3bc..41ed82760 100644 --- a/crates/lance-graph-quack/tests/duckdb_differential.rs +++ b/crates/lance-graph-quack/tests/duckdb_differential.rs @@ -77,6 +77,29 @@ fn alloc_delta(before: usize) -> usize { BYTES.load(Ordering::Relaxed) - before } +/// Opt-in timing arm: `QUACK_DIFF_TIMED_REPS=N` runs each timed site N extra +/// times AFTER parity has been asserted. Unset or unparsable means `0`, and +/// the default run is unchanged. +fn timed_reps() -> usize { + std::env::var("QUACK_DIFF_TIMED_REPS") + .ok() + .and_then(|s| s.parse::().ok()) + .unwrap_or(0) +} + +/// Median of `reps` per-run wall times in ns. Callers only call it when +/// `reps > 0`; with `reps == 0` it returns `0` rather than indexing. +fn median_ns(reps: usize, mut f: impl FnMut()) -> u64 { + let mut samples: Vec = Vec::with_capacity(reps); + for _ in 0..reps { + let t = std::time::Instant::now(); + f(); + samples.push(u64::try_from(t.elapsed().as_nanos()).unwrap_or(u64::MAX)); + } + samples.sort_unstable(); + samples.get(reps / 2).copied().unwrap_or(0) +} + // --------------------------------------------------------------------- // cases.tsv — hand-rolled TSV, no serde in this crate's dependency graph. // --------------------------------------------------------------------- @@ -170,15 +193,32 @@ struct CaseMetrics { /// a zero-materialisation proof: the terminal is O(1) WHEN GIVEN that /// view; producing the view is the open question. `0` everywhere else. fixture_view_bytes: usize, + /// Median ns of one BIND rep: the site's lowering call(s) plus its + /// `Scratch::for_program` sizing, repeated. `None` unless + /// `QUACK_DIFF_TIMED_REPS > 0` and the site is timed. + t_compile_or_bind_ns: Option, + /// Median ns of one EXECUTE rep: the site's `execute_into` call(s) only, + /// after binding, with the `Out` buffer reset before each rep. `None` + /// unless `QUACK_DIFF_TIMED_REPS > 0` and the site is timed. + t_execute_ns: Option, + /// How many reps each median was taken over (`0` when not timed). + timed_reps: usize, } fn print_metric(id: &str, m: &CaseMetrics) { let scratch_bytes = m.scratch_words * std::mem::size_of::(); + let timing = match (m.t_compile_or_bind_ns, m.t_execute_ns) { + (Some(b), Some(e)) => format!( + " t_compile_or_bind_ns={b} t_execute_ns={e} timed_reps={}", + m.timed_reps + ), + _ => String::new(), + }; match m.programs { Some(k) => eprintln!( "METRIC case={id} ops={} scratch_words={} tile_words={} scratch_bytes={} \ alloc_bytes_exec={} rows_materialized={} index_vec_len={} out_bytes={} \ - pair_relation_bytes={} population_state_bytes={} fixture_view_bytes={} programs={k}", + pair_relation_bytes={} population_state_bytes={} fixture_view_bytes={} programs={k}{timing}", m.ops, m.scratch_words, m.tile_words, @@ -194,7 +234,7 @@ fn print_metric(id: &str, m: &CaseMetrics) { None => eprintln!( "METRIC case={id} ops={} scratch_words={} tile_words={} scratch_bytes={} \ alloc_bytes_exec={} rows_materialized={} index_vec_len={} out_bytes={} \ - pair_relation_bytes={} population_state_bytes={} fixture_view_bytes={}", + pair_relation_bytes={} population_state_bytes={} fixture_view_bytes={}{timing}", m.ops, m.scratch_words, m.tile_words, @@ -214,7 +254,8 @@ fn print_metric(id: &str, m: &CaseMetrics) { /// encode. The encoding matches `oracle.py`'s exactly — see /// `tests/duckdb/README.txt`. fn run_query(id: &str, planes: &Planes<'_>, filter: Filter, agg: Agg) -> (String, CaseMetrics) { - let program = lower(&Query { filter, agg }).expect("lowers"); + let query = Query { filter, agg }; + let program = lower(&query).expect("lowers"); let mut scratch = Scratch::for_program(&program, planes.n_rows).expect("carves"); let tile_words = scratch.words(); let scratch_words = scratch_words_for(tile_words, scratch.slots()).expect("sized"); @@ -270,6 +311,38 @@ fn run_query(id: &str, planes: &Planes<'_>, filter: Filter, agg: Agg) -> (String } }; let out_bytes = encoded.len(); + + // Opt-in timing arm — runs only after parity inputs are final. + let reps = timed_reps(); + let (t_compile_or_bind_ns, t_execute_ns) = if reps > 0 { + let t_bind = median_ns(reps, || { + let p = lower(&query).expect("lowers"); + let s = Scratch::for_program(&p, planes.n_rows).expect("carves"); + std::hint::black_box((p, s)); + }); + let is_keep = matches!(program.terminal, Terminal::Keep { .. }); + let first_kept = kept_buf.clone(); + let t_exec = median_ns(reps, || { + kept_buf.iter_mut().for_each(|w| *w = 0); + let out = if is_keep { + Out::Mask(&mut kept_buf) + } else { + Out::None + }; + let v = + execute_into(&program, planes, &Foreign::NONE, &mut scratch, out).expect("runs"); + assert_eq!(v, value, "case {id}: timed rep diverged from the first run"); + if is_keep { + assert_eq!( + kept_buf, first_kept, + "case {id}: timed rep kept-mask diverged from the first run" + ); + } + }); + (Some(t_bind), Some(t_exec)) + } else { + (None, None) + }; ( encoded, CaseMetrics { @@ -284,6 +357,9 @@ fn run_query(id: &str, planes: &Planes<'_>, filter: Filter, agg: Agg) -> (String pair_relation_bytes: 0, population_state_bytes: 0, fixture_view_bytes: 0, + t_compile_or_bind_ns, + t_execute_ns, + timed_reps: reps, }, ) } @@ -346,6 +422,7 @@ fn run_group( let mut ops = plan.filter.ops.len(); let mut pairs = Vec::with_capacity(plan.groups.len()); + let mut first_vals: Vec = Vec::with_capacity(plan.groups.len()); for (k, prog) in plan.groups.iter().enumerate() { let mut g_scratch = Scratch::for_program(prog, planes.n_rows).expect("carves"); scratch_words_total += @@ -356,6 +433,7 @@ fn run_group( execute_into(prog, &widened, &Foreign::NONE, &mut g_scratch, Out::None).expect("runs"); alloc_bytes_exec += alloc_delta(before); ops += prog.ops.len(); + first_vals.push(v); let val_str = match v { Value::Count(c) => c.to_string(), @@ -366,6 +444,63 @@ fn run_group( } let encoded = pairs.join(";"); let out_bytes = encoded.len(); + + // Opt-in timing arm — one rep = the whole phase-1 + K-program sequence. + let reps = timed_reps(); + let (t_compile_or_bind_ns, t_execute_ns) = if reps > 0 { + let t_bind = median_ns(reps, || { + let p = lower_group_by(&g, 0).expect("lowers"); + let fs = Scratch::for_program(&p.filter, planes.n_rows).expect("carves"); + let gs: Vec = p + .groups + .iter() + .map(|prog| Scratch::for_program(prog, planes.n_rows).expect("carves")) + .collect(); + std::hint::black_box((p, fs, gs)); + }); + let mut g_scratches: Vec = plan + .groups + .iter() + .map(|prog| Scratch::for_program(prog, planes.n_rows).expect("carves")) + .collect(); + let mut rep_bits = vec![0u64; words_for(planes.n_rows)]; + let t_exec = median_ns(reps, || { + rep_bits.iter_mut().for_each(|w| *w = 0); + let k_val = execute_into( + &plan.filter, + planes, + &Foreign::NONE, + &mut f_scratch, + Out::Mask(&mut rep_bits), + ) + .expect("runs"); + assert_eq!( + k_val, kept, + "case {id}: timed phase-1 diverged from the first run" + ); + assert_eq!( + rep_bits, kept_bits, + "case {id}: timed phase-1 kept-mask diverged from the first run" + ); + let rep_masks: [&[u64]; 1] = [rep_bits.as_slice()]; + let rep_widened = Planes { + n_rows: planes.n_rows, + masks: &rep_masks, + lanes: planes.lanes, + }; + for (k, (prog, gs)) in plan.groups.iter().zip(g_scratches.iter_mut()).enumerate() { + let v = + execute_into(prog, &rep_widened, &Foreign::NONE, gs, Out::None).expect("runs"); + assert_eq!( + v, first_vals[k], + "case {id}: timed group-{k} diverged from the first run" + ); + } + }); + (Some(t_bind), Some(t_exec)) + } else { + (None, None) + }; ( encoded, CaseMetrics { @@ -380,6 +515,9 @@ fn run_group( pair_relation_bytes: 0, population_state_bytes: 0, fixture_view_bytes: 0, + t_compile_or_bind_ns, + t_execute_ns, + timed_reps: reps, }, ) } @@ -397,11 +535,11 @@ fn run_group_reduce( agg: GroupAgg, groups: usize, ) -> (String, CaseMetrics) { - let program = lower(&Query { + let query = Query { filter, agg: Agg::GroupReduce { key, agg }, - }) - .expect("lowers"); + }; + let program = lower(&query).expect("lowers"); let fold = match program.terminal { Terminal::GroupReduce { fold, .. } => fold, other => panic!("case {id}: lowered onto {other:?}, not GroupReduce"), @@ -415,6 +553,8 @@ fn run_group_reduce( execute_into(&program, planes, foreign, &mut scratch, Out::I64(&mut out)).expect("runs"); let alloc_bytes_exec = alloc_delta(before); assert_eq!(value, Value::GroupReduced, "case {id}"); + // The raw sink as the first run left it, kept only for the opt-in timing arm. + let first_raw = (timed_reps() > 0).then(|| out.clone()); // Through the consumer boundary: the raw sink never reaches the encoder. let empty_is_null = !matches!(fold, GroupFold::Count); let present = normalize_group_sink(fold, &mut out); @@ -431,6 +571,30 @@ fn run_group_reduce( .collect::>() .join(";"); let out_bytes = encoded.len(); + + // Opt-in timing arm. + let reps = timed_reps(); + let (t_compile_or_bind_ns, t_execute_ns) = match first_raw { + Some(first_raw) if reps > 0 => { + let t_bind = median_ns(reps, || { + let p = lower(&query).expect("lowers"); + let s = Scratch::for_program(&p, planes.n_rows).expect("carves"); + std::hint::black_box((p, s)); + }); + let t_exec = median_ns(reps, || { + out.fill(0x5a5a_i64); + let v = execute_into(&program, planes, foreign, &mut scratch, Out::I64(&mut out)) + .expect("runs"); + assert_eq!(v, value, "case {id}: timed rep diverged from the first run"); + assert_eq!( + out, first_raw, + "case {id}: timed rep sink diverged from the first run" + ); + }); + (Some(t_bind), Some(t_exec)) + } + _ => (None, None), + }; ( encoded, CaseMetrics { @@ -445,6 +609,9 @@ fn run_group_reduce( pair_relation_bytes: 0, population_state_bytes: 0, fixture_view_bytes: 0, + t_compile_or_bind_ns, + t_execute_ns, + timed_reps: reps, }, ) } @@ -880,14 +1047,14 @@ fn group_sum_cc() { // The one-terminal spelling: ONE program, `Terminal::GroupSumI32`, no // K-program loop. - let program = lower(&Query { + let query = Query { filter: filter.clone(), agg: Agg::GroupSumI32 { key: COST_CENTER, val: AMOUNT, }, - }) - .expect("lowers"); + }; + let program = lower(&query).expect("lowers"); let mut scratch = Scratch::for_program(&program, planes.n_rows).expect("carves"); let tile_words = scratch.words(); let scratch_words = scratch_words_for(tile_words, scratch.slots()).expect("sized"); @@ -910,6 +1077,39 @@ fn group_sum_cc() { .collect::>() .join(";"); let out_bytes = encoded.len(); + + // Opt-in timing arm. + let reps = timed_reps(); + let (t_compile_or_bind_ns, t_execute_ns) = if reps > 0 { + let t_bind = median_ns(reps, || { + let p = lower(&query).expect("lowers"); + let s = Scratch::for_program(&p, planes.n_rows).expect("carves"); + std::hint::black_box((p, s)); + }); + let first_out = out; + let t_exec = median_ns(reps, || { + out = [0i64; 8]; + let v = execute_into( + &program, + &planes, + &Foreign::NONE, + &mut scratch, + Out::I64(&mut out), + ) + .expect("runs"); + assert_eq!( + v, value, + "case group_sum_cc: timed rep diverged from the first run" + ); + assert_eq!( + out, first_out, + "case group_sum_cc: timed rep sink diverged from the first run" + ); + }); + (Some(t_bind), Some(t_exec)) + } else { + (None, None) + }; let m = CaseMetrics { ops: program.ops.len(), scratch_words, @@ -922,6 +1122,9 @@ fn group_sum_cc() { pair_relation_bytes: 0, population_state_bytes: 0, fixture_view_bytes: 0, + t_compile_or_bind_ns, + t_execute_ns, + timed_reps: reps, }; print_metric("group_sum_cc", &m); assert_case(&cases, "group_sum_cc", &encoded); @@ -974,11 +1177,11 @@ fn join_sum_country() { Filter::cmp(STATUS, Cmp::EqU32(1)), Filter::eq_u32_via(PARTNER_ID, ForeignLane(0), 3), ]); - let line_program = lower(&Query { + let query = Query { filter: line_filter, agg: Agg::SumI32(AMOUNT), - }) - .expect("lowers"); + }; + let line_program = lower(&query).expect("lowers"); let mut scratch = Scratch::for_program(&line_program, planes.n_rows).expect("carves"); let tile_words = scratch.words(); let scratch_words = scratch_words_for(tile_words, scratch.slots()).expect("sized"); @@ -992,6 +1195,27 @@ fn join_sum_country() { other => panic!("expected a sum, got {other:?}"), }; let out_bytes = encoded.len(); + + // Opt-in timing arm. + let reps = timed_reps(); + let (t_compile_or_bind_ns, t_execute_ns) = if reps > 0 { + let t_bind = median_ns(reps, || { + let p = lower(&query).expect("lowers"); + let s = Scratch::for_program(&p, planes.n_rows).expect("carves"); + std::hint::black_box((p, s)); + }); + let t_exec = median_ns(reps, || { + let v = execute_into(&line_program, &planes, &foreign, &mut scratch, Out::None) + .expect("runs"); + assert_eq!( + v, value, + "case join_sum_country: timed rep diverged from the first run" + ); + }); + (Some(t_bind), Some(t_exec)) + } else { + (None, None) + }; let m = CaseMetrics { ops: line_program.ops.len(), scratch_words, @@ -1009,6 +1233,9 @@ fn join_sum_country() { pair_relation_bytes: 0, population_state_bytes: 0, fixture_view_bytes: 0, + t_compile_or_bind_ns, + t_execute_ns, + timed_reps: reps, }; print_metric("join_sum_country", &m); assert_case(&cases, "join_sum_country", &encoded); @@ -1044,11 +1271,11 @@ fn join_count_docs_with_posted() { // ── Arm 1: the generated (unordered) layout is REFUSED. ── let lanes = fx.line.lanes(); let planes = lanes.planes(); - let runs = lower(&Query { + let query = Query { filter: Filter::cmp(STATUS, Cmp::EqU32(1)), agg: Agg::CountDistinctOrderedU32 { key: DOC_ID_U32 }, - }) - .expect("lowers"); + }; + let runs = lower(&query).expect("lowers"); assert!(matches!(runs.terminal, Terminal::CountKeyRunsU32 { .. })); let mut r_scratch = Scratch::for_program(&runs, planes.n_rows).expect("carves"); let refused = execute_into(&runs, &planes, &Foreign::NONE, &mut r_scratch, Out::None); @@ -1093,6 +1320,27 @@ fn join_count_docs_with_posted() { Value::Count(c) => c.to_string(), other => panic!("expected a count, got {other:?}"), }; + // Opt-in timing arm — the ordered-view arm only; the refusal arm is not + // timed. + let reps = timed_reps(); + let (t_compile_or_bind_ns, t_execute_ns) = if reps > 0 { + let t_bind = median_ns(reps, || { + let p = lower(&query).expect("lowers"); + let s = Scratch::for_program(&p, c_planes.n_rows).expect("carves"); + std::hint::black_box((p, s)); + }); + let t_exec = median_ns(reps, || { + let v = execute_into(&runs, &c_planes, &Foreign::NONE, &mut c_scratch, Out::None) + .expect("runs"); + assert_eq!( + v, c_value, + "case join_count_docs_with_posted: timed rep diverged from the first run" + ); + }); + (Some(t_bind), Some(t_exec)) + } else { + (None, None) + }; let m = CaseMetrics { ops: runs.ops.len(), scratch_words: c_scratch_words, @@ -1109,6 +1357,9 @@ fn join_count_docs_with_posted() { // NULL flag. Not the terminal's state — and not evidence that a // resident doc-major view exists. fixture_view_bytes: order.len() * (std::mem::size_of::() + 9 * 4 + 1), + t_compile_or_bind_ns, + t_execute_ns, + timed_reps: reps, }; print_metric("join_count_docs_with_posted_ordered_view_given", &m); // A distinct count is permutation-invariant: same DuckDB row, same answer. @@ -1130,15 +1381,15 @@ fn join_group_sum_country() { let lanes = fx.line.lanes(); let planes = lanes.planes(); - let program = lower(&Query { + let query = Query { filter: Filter::cmp(STATUS, Cmp::EqU32(1)), agg: Agg::GroupSumViaI32 { fk: PARTNER_ID, key: ForeignLane(0), val: AMOUNT, }, - }) - .expect("lowers"); + }; + let program = lower(&query).expect("lowers"); let mut scratch = Scratch::for_program(&program, planes.n_rows).expect("carves"); let tile_words = scratch.words(); let scratch_words = scratch_words_for(tile_words, scratch.slots()).expect("sized"); @@ -1168,6 +1419,39 @@ fn join_group_sum_country() { .collect::>() .join(";"); let out_bytes = encoded.len(); + + // Opt-in timing arm. + let reps = timed_reps(); + let (t_compile_or_bind_ns, t_execute_ns) = if reps > 0 { + let t_bind = median_ns(reps, || { + let p = lower(&query).expect("lowers"); + let s = Scratch::for_program(&p, planes.n_rows).expect("carves"); + std::hint::black_box((p, s)); + }); + let first_out = out; + let t_exec = median_ns(reps, || { + out = [0i64; 8]; + let v = execute_into( + &program, + &planes, + &foreign, + &mut scratch, + Out::I64(&mut out), + ) + .expect("runs"); + assert_eq!( + v, value, + "case join_group_sum_country: timed rep diverged from the first run" + ); + assert_eq!( + out, first_out, + "case join_group_sum_country: timed rep sink diverged from the first run" + ); + }); + (Some(t_bind), Some(t_exec)) + } else { + (None, None) + }; let m = CaseMetrics { ops: program.ops.len(), scratch_words, @@ -1184,6 +1468,9 @@ fn join_group_sum_country() { pair_relation_bytes: 0, population_state_bytes: 0, fixture_view_bytes: 0, + t_compile_or_bind_ns, + t_execute_ns, + timed_reps: reps, }; print_metric("join_group_sum_country", &m); assert_case(&cases, "join_group_sum_country", &encoded); @@ -1268,6 +1555,24 @@ fn run_scalar( let value = execute_into(program, planes, foreign, &mut scratch, Out::None) .unwrap_or_else(|e| panic!("case {id}: {e:?}")); let alloc_bytes_exec = alloc_delta(before); + + // Opt-in timing arm. The lowering happens in the caller, so one bind rep + // here is the `Scratch::for_program` sizing only. + let reps = timed_reps(); + let (t_compile_or_bind_ns, t_execute_ns) = if reps > 0 { + let t_bind = median_ns(reps, || { + let s = Scratch::for_program(program, planes.n_rows).expect("carves"); + std::hint::black_box(s); + }); + let t_exec = median_ns(reps, || { + let v = execute_into(program, planes, foreign, &mut scratch, Out::None) + .unwrap_or_else(|e| panic!("case {id}: {e:?}")); + assert_eq!(v, value, "case {id}: timed rep diverged from the first run"); + }); + (Some(t_bind), Some(t_exec)) + } else { + (None, None) + }; ( value, CaseMetrics { @@ -1282,6 +1587,9 @@ fn run_scalar( pair_relation_bytes: 0, population_state_bytes: 0, fixture_view_bytes: 0, + t_compile_or_bind_ns, + t_execute_ns, + timed_reps: reps, }, ) } diff --git a/crates/sag-ra-probe/Cargo.toml b/crates/sag-ra-probe/Cargo.toml index ac1b44dac..9cbdab07f 100644 --- a/crates/sag-ra-probe/Cargo.toml +++ b/crates/sag-ra-probe/Cargo.toml @@ -20,6 +20,13 @@ ra_ap_hir = "0.0.358" ra_ap_ide = "0.0.358" ra_ap_load-cargo = "0.0.358" ra_ap_project_model = "0.0.358" +ra_ap_syntax = "0.0.358" +# Not used directly: a pin. ra-ap-rustc_lexer asserts at compile time that +# unicode-ident and unicode-properties report the same Unicode version. +# ra_ap_hir_def pins unicode-ident =1.0.24 (Unicode 17), while a fresh +# resolution (this crate tracks no lockfile) picks unicode-properties 0.1.5 +# (Unicode 18) and the lexer fails to build (2026-11-02). +unicode-properties = "=0.1.4" [profile.dev] debug = 0 diff --git a/crates/sag-ra-probe/fixtures/codegraph/rev1/Cargo.toml b/crates/sag-ra-probe/fixtures/codegraph/rev1/Cargo.toml new file mode 100644 index 000000000..81e702c77 --- /dev/null +++ b/crates/sag-ra-probe/fixtures/codegraph/rev1/Cargo.toml @@ -0,0 +1,5 @@ +# D-CODEGRAPH-0 oracle fixture, revision 1. Its own workspace; nothing builds it +# except the probe and the rustc oracle (`cargo check`), from this directory. +[workspace] +members = ["cg_lib", "cg_app", "cg_broken"] +resolver = "2" diff --git a/crates/sag-ra-probe/fixtures/codegraph/rev1/cg_app/Cargo.toml b/crates/sag-ra-probe/fixtures/codegraph/rev1/cg_app/Cargo.toml new file mode 100644 index 000000000..8d0249344 --- /dev/null +++ b/crates/sag-ra-probe/fixtures/codegraph/rev1/cg_app/Cargo.toml @@ -0,0 +1,8 @@ +[package] +name = "cg_app" +version = "0.1.0" +edition = "2021" +publish = false + +[dependencies] +cg_lib = { path = "../cg_lib" } diff --git a/crates/sag-ra-probe/fixtures/codegraph/rev1/cg_app/src/lib.rs b/crates/sag-ra-probe/fixtures/codegraph/rev1/cg_app/src/lib.rs new file mode 100644 index 000000000..5330ddfec --- /dev/null +++ b/crates/sag-ra-probe/fixtures/codegraph/rev1/cg_app/src/lib.rs @@ -0,0 +1,8 @@ +use cg_lib::shapes::{Rect, Shape}; +use cg_lib::Sq; + +pub fn run() -> u32 { + let s = Sq(3); + let r = Rect(1, 2); + cg_lib::area_of(&s) + cg_lib::area_dyn(&r) + s.area() + cg_lib::util::double(2) +} diff --git a/crates/sag-ra-probe/fixtures/codegraph/rev1/cg_broken/Cargo.toml b/crates/sag-ra-probe/fixtures/codegraph/rev1/cg_broken/Cargo.toml new file mode 100644 index 000000000..4aeeba141 --- /dev/null +++ b/crates/sag-ra-probe/fixtures/codegraph/rev1/cg_broken/Cargo.toml @@ -0,0 +1,5 @@ +[package] +name = "cg_broken" +version = "0.1.0" +edition = "2021" +publish = false diff --git a/crates/sag-ra-probe/fixtures/codegraph/rev1/cg_broken/src/lib.rs b/crates/sag-ra-probe/fixtures/codegraph/rev1/cg_broken/src/lib.rs new file mode 100644 index 000000000..16e97c4a6 --- /dev/null +++ b/crates/sag-ra-probe/fixtures/codegraph/rev1/cg_broken/src/lib.rs @@ -0,0 +1,5 @@ +use nothing::here; + +pub fn f() -> u32 { + missing_fn(1) +} diff --git a/crates/sag-ra-probe/fixtures/codegraph/rev1/cg_lib/Cargo.toml b/crates/sag-ra-probe/fixtures/codegraph/rev1/cg_lib/Cargo.toml new file mode 100644 index 000000000..f21d6d7ff --- /dev/null +++ b/crates/sag-ra-probe/fixtures/codegraph/rev1/cg_lib/Cargo.toml @@ -0,0 +1,5 @@ +[package] +name = "cg_lib" +version = "0.1.0" +edition = "2021" +publish = false diff --git a/crates/sag-ra-probe/fixtures/codegraph/rev1/cg_lib/src/lib.rs b/crates/sag-ra-probe/fixtures/codegraph/rev1/cg_lib/src/lib.rs new file mode 100644 index 000000000..b31d29a95 --- /dev/null +++ b/crates/sag-ra-probe/fixtures/codegraph/rev1/cg_lib/src/lib.rs @@ -0,0 +1,18 @@ +pub mod macro_use; +pub mod shapes; +pub mod util; + +pub use shapes::Square as Sq; + +pub fn area_of(s: &S) -> u32 { + s.area() +} + +pub fn area_dyn(s: &dyn shapes::Shape) -> u32 { + s.area() +} + +#[cfg(any())] +pub fn disabled() -> u32 { + util::double(1) +} diff --git a/crates/sag-ra-probe/fixtures/codegraph/rev1/cg_lib/src/macro_use.rs b/crates/sag-ra-probe/fixtures/codegraph/rev1/cg_lib/src/macro_use.rs new file mode 100644 index 000000000..dc7dc7293 --- /dev/null +++ b/crates/sag-ra-probe/fixtures/codegraph/rev1/cg_lib/src/macro_use.rs @@ -0,0 +1,6 @@ +use crate::util::triple as t3; + +pub fn checked() -> u32 { + assert_eq!(t3(1), 3); + 1 +} diff --git a/crates/sag-ra-probe/fixtures/codegraph/rev1/cg_lib/src/shapes.rs b/crates/sag-ra-probe/fixtures/codegraph/rev1/cg_lib/src/shapes.rs new file mode 100644 index 000000000..fba756550 --- /dev/null +++ b/crates/sag-ra-probe/fixtures/codegraph/rev1/cg_lib/src/shapes.rs @@ -0,0 +1,28 @@ +use crate::util::double; +use crate::util::triple; + +pub trait Shape { + fn area(&self) -> u32; +} + +pub struct Square(pub u32); + +pub struct Rect(pub u32, pub u32); + +impl Shape for Square { + fn area(&self) -> u32 { + double(self.0) / 2 * self.0 + } +} + +impl Shape for Rect { + fn area(&self) -> u32 { + self.0 * self.1 + } +} + +impl Square { + pub fn side(&self) -> u32 { + self.0 + } +} diff --git a/crates/sag-ra-probe/fixtures/codegraph/rev1/cg_lib/src/util.rs b/crates/sag-ra-probe/fixtures/codegraph/rev1/cg_lib/src/util.rs new file mode 100644 index 000000000..1990a52db --- /dev/null +++ b/crates/sag-ra-probe/fixtures/codegraph/rev1/cg_lib/src/util.rs @@ -0,0 +1,17 @@ +use crate::shapes::Square; + +pub fn double(x: u32) -> u32 { + x * 2 +} + +pub fn triple(x: u32) -> u32 { + x * 3 +} + +pub fn unit() -> Square { + Square(1) +} + +fn private_unused() -> u32 { + 0 +} diff --git a/crates/sag-ra-probe/fixtures/codegraph/rev1_calls.pin.tsv b/crates/sag-ra-probe/fixtures/codegraph/rev1_calls.pin.tsv new file mode 100644 index 000000000..a02085b14 --- /dev/null +++ b/crates/sag-ra-probe/fixtures/codegraph/rev1_calls.pin.tsv @@ -0,0 +1,4 @@ +# callee-signature path (relative to rev1/) line (1-based) call expression -- read off the source by hand (cat -n), not from Charon +cg_lib::util::triple cg_lib/src/macro_use.rs 4 t3(1) +cg_lib::util::double cg_lib/src/shapes.rs 14 double(self.0) +cg_lib::shapes::Shape cg_lib/src/lib.rs 8 s.area() diff --git a/crates/sag-ra-probe/fixtures/codegraph/rev1_xrefs.pin.tsv b/crates/sag-ra-probe/fixtures/codegraph/rev1_xrefs.pin.tsv new file mode 100644 index 000000000..bbbf91171 --- /dev/null +++ b/crates/sag-ra-probe/fixtures/codegraph/rev1_xrefs.pin.tsv @@ -0,0 +1,20 @@ +# moniker role path (relative to rev1/) line (1-based) column (0-based byte) -- read off the fixture source by hand (cat -n), not from rust-analyzer +cg_lib::util/::double(). def cg_lib/src/util.rs 3 7 +cg_lib::util/::double(). ref cg_lib/src/shapes.rs 1 17 +cg_lib::util/::double(). ref cg_lib/src/shapes.rs 14 8 +cg_lib::util/::double(). ref cg_app/src/lib.rs 7 74 +cg_lib::util/::triple(). def cg_lib/src/util.rs 7 7 +cg_lib::util/::triple(). ref cg_lib/src/shapes.rs 2 17 +cg_lib::util/::triple(). ref cg_lib/src/macro_use.rs 1 17 +cg_lib::util/::triple(). ref cg_lib/src/macro_use.rs 4 15 +cg_lib::shapes/::Square# def cg_lib/src/shapes.rs 8 11 +cg_lib::shapes/::Square# ref cg_lib/src/lib.rs 5 16 +cg_lib::shapes/::Square# ref cg_app/src/lib.rs 2 12 +cg_lib::shapes/::Square# ref cg_app/src/lib.rs 5 12 +cg_lib::shapes/::Shape#::area(). def cg_lib/src/shapes.rs 5 7 +cg_lib::shapes/::Shape#::area(). ref cg_lib/src/lib.rs 8 6 +cg_lib::shapes/::Shape#::area(). ref cg_lib/src/lib.rs 12 6 +cg_lib::shapes/::impl#::Square[]::Shape[]::area(). ref cg_app/src/lib.rs 7 51 +cg_lib::area_of(). ref cg_app/src/lib.rs 7 12 +cg_lib::area_dyn(). ref cg_app/src/lib.rs 7 34 +cg_lib::util/::private_unused(). def cg_lib/src/util.rs 15 3 diff --git a/crates/sag-ra-probe/fixtures/codegraph/rev2/Cargo.toml b/crates/sag-ra-probe/fixtures/codegraph/rev2/Cargo.toml new file mode 100644 index 000000000..4202d70fe --- /dev/null +++ b/crates/sag-ra-probe/fixtures/codegraph/rev2/Cargo.toml @@ -0,0 +1,5 @@ +# D-CODEGRAPH-0 oracle fixture, revision 2. Its own workspace; nothing builds it +# except the probe and the rustc oracle (`cargo check`), from this directory. +[workspace] +members = ["cg_lib", "cg_app", "cg_broken"] +resolver = "2" diff --git a/crates/sag-ra-probe/fixtures/codegraph/rev2/cg_app/Cargo.toml b/crates/sag-ra-probe/fixtures/codegraph/rev2/cg_app/Cargo.toml new file mode 100644 index 000000000..8d0249344 --- /dev/null +++ b/crates/sag-ra-probe/fixtures/codegraph/rev2/cg_app/Cargo.toml @@ -0,0 +1,8 @@ +[package] +name = "cg_app" +version = "0.1.0" +edition = "2021" +publish = false + +[dependencies] +cg_lib = { path = "../cg_lib" } diff --git a/crates/sag-ra-probe/fixtures/codegraph/rev2/cg_app/src/lib.rs b/crates/sag-ra-probe/fixtures/codegraph/rev2/cg_app/src/lib.rs new file mode 100644 index 000000000..5330ddfec --- /dev/null +++ b/crates/sag-ra-probe/fixtures/codegraph/rev2/cg_app/src/lib.rs @@ -0,0 +1,8 @@ +use cg_lib::shapes::{Rect, Shape}; +use cg_lib::Sq; + +pub fn run() -> u32 { + let s = Sq(3); + let r = Rect(1, 2); + cg_lib::area_of(&s) + cg_lib::area_dyn(&r) + s.area() + cg_lib::util::double(2) +} diff --git a/crates/sag-ra-probe/fixtures/codegraph/rev2/cg_broken/Cargo.toml b/crates/sag-ra-probe/fixtures/codegraph/rev2/cg_broken/Cargo.toml new file mode 100644 index 000000000..4aeeba141 --- /dev/null +++ b/crates/sag-ra-probe/fixtures/codegraph/rev2/cg_broken/Cargo.toml @@ -0,0 +1,5 @@ +[package] +name = "cg_broken" +version = "0.1.0" +edition = "2021" +publish = false diff --git a/crates/sag-ra-probe/fixtures/codegraph/rev2/cg_broken/src/lib.rs b/crates/sag-ra-probe/fixtures/codegraph/rev2/cg_broken/src/lib.rs new file mode 100644 index 000000000..16e97c4a6 --- /dev/null +++ b/crates/sag-ra-probe/fixtures/codegraph/rev2/cg_broken/src/lib.rs @@ -0,0 +1,5 @@ +use nothing::here; + +pub fn f() -> u32 { + missing_fn(1) +} diff --git a/crates/sag-ra-probe/fixtures/codegraph/rev2/cg_lib/Cargo.toml b/crates/sag-ra-probe/fixtures/codegraph/rev2/cg_lib/Cargo.toml new file mode 100644 index 000000000..f21d6d7ff --- /dev/null +++ b/crates/sag-ra-probe/fixtures/codegraph/rev2/cg_lib/Cargo.toml @@ -0,0 +1,5 @@ +[package] +name = "cg_lib" +version = "0.1.0" +edition = "2021" +publish = false diff --git a/crates/sag-ra-probe/fixtures/codegraph/rev2/cg_lib/src/lib.rs b/crates/sag-ra-probe/fixtures/codegraph/rev2/cg_lib/src/lib.rs new file mode 100644 index 000000000..b31d29a95 --- /dev/null +++ b/crates/sag-ra-probe/fixtures/codegraph/rev2/cg_lib/src/lib.rs @@ -0,0 +1,18 @@ +pub mod macro_use; +pub mod shapes; +pub mod util; + +pub use shapes::Square as Sq; + +pub fn area_of(s: &S) -> u32 { + s.area() +} + +pub fn area_dyn(s: &dyn shapes::Shape) -> u32 { + s.area() +} + +#[cfg(any())] +pub fn disabled() -> u32 { + util::double(1) +} diff --git a/crates/sag-ra-probe/fixtures/codegraph/rev2/cg_lib/src/macro_use.rs b/crates/sag-ra-probe/fixtures/codegraph/rev2/cg_lib/src/macro_use.rs new file mode 100644 index 000000000..37ed36ef0 --- /dev/null +++ b/crates/sag-ra-probe/fixtures/codegraph/rev2/cg_lib/src/macro_use.rs @@ -0,0 +1,6 @@ +use crate::util::twice as t2; + +pub fn checked() -> u32 { + assert_eq!(t2(1), 2); + 1 +} diff --git a/crates/sag-ra-probe/fixtures/codegraph/rev2/cg_lib/src/shapes.rs b/crates/sag-ra-probe/fixtures/codegraph/rev2/cg_lib/src/shapes.rs new file mode 100644 index 000000000..4a0243a40 --- /dev/null +++ b/crates/sag-ra-probe/fixtures/codegraph/rev2/cg_lib/src/shapes.rs @@ -0,0 +1,27 @@ +use crate::util::double; + +pub trait Shape { + fn area(&self) -> u32; +} + +pub struct Square(pub u32); + +pub struct Rect(pub u32, pub u32); + +impl Shape for Square { + fn area(&self) -> u32 { + double(self.0) / 2 * self.0 + } +} + +impl Shape for Rect { + fn area(&self) -> u32 { + self.0 * self.1 + } +} + +impl Square { + pub fn side(&self) -> u32 { + self.0 + } +} diff --git a/crates/sag-ra-probe/fixtures/codegraph/rev2/cg_lib/src/util.rs b/crates/sag-ra-probe/fixtures/codegraph/rev2/cg_lib/src/util.rs new file mode 100644 index 000000000..a3be71922 --- /dev/null +++ b/crates/sag-ra-probe/fixtures/codegraph/rev2/cg_lib/src/util.rs @@ -0,0 +1,17 @@ +use crate::shapes::Square; + +pub fn double(x: u32) -> u32 { + x * 2 +} + +pub fn twice(x: u32) -> u32 { + double(x) +} + +pub fn unit() -> Square { + Square(1) +} + +fn private_unused() -> u32 { + 0 +} diff --git a/crates/sag-ra-probe/src/main.rs b/crates/sag-ra-probe/src/main.rs index ec53cdc0b..4d45f903f 100644 --- a/crates/sag-ra-probe/src/main.rs +++ b/crates/sag-ra-probe/src/main.rs @@ -175,6 +175,10 @@ fn main() -> anyhow::Result<()> { t.elapsed().as_millis() ); + // NO_MIR=1 skips MIR lowering, which the GRAPH rows do not read. + if std::env::var("NO_MIR").is_ok() { + bodies.clear(); + } let t = Instant::now(); let (ok, err, kinds) = ra_ap_hir::attach_db(host.raw_database(), || { let db = host.raw_database(); @@ -391,6 +395,323 @@ fn main() -> anyhow::Result<()> { } } } + // GRAPH=1: the typed rows the codegraph (`crates/lance-graph-codegraph`) + // reads. Every row is a fact rust-analyzer witnessed; nothing is inferred + // here. Offsets are byte offsets into the file, paths are relative to the + // workspace root, and files outside the root are skipped. + // file (always every file) + // node + // occ + // call + // impl + // macro a macro call with arguments: references + // inside it may be missing (KYTHE.md) + // diag + // import + // + // `` is the kind-preserving identity of `identity()`. `use` / `pubuse` + // mark a reference inside a `use` item (private / any visibility). + // FILES=a.rs,b.rs restricts every row except `file` to what an incremental + // update of those files needs: rows located in them, plus `occ` and `call` + // rows whose target is defined in them. + if std::env::var("GRAPH").is_ok() { + use ra_ap_syntax::{AstNode, ast, ast::HasName, ast::HasVisibility}; + use std::collections::{BTreeSet, HashMap}; + let rel = |file_id| { + vfs.file_path(file_id).as_path().and_then(|a| { + std::path::Path::new(a.as_str()) + .strip_prefix(&root) + .ok() + .map(|p| p.display().to_string()) + }) + }; + let only: Option> = std::env::var("FILES").ok().map(|v| { + v.split(',') + .filter(|s| !s.is_empty()) + .map(str::to_owned) + .collect() + }); + let keep = |f: &str| only.as_ref().is_none_or(|o| o.contains(f)); + let fnv = |t: &str| { + t.bytes().fold(0xcbf29ce484222325u64, |h, b| { + (h ^ u64::from(b)).wrapping_mul(0x100000001b3) + }) + }; + // Identity and kind of every definition the index holds, by the + // position of its name, so call targets resolve to the same id. + let mut by_pos: HashMap<(ra_ap_ide::FileId, u32), (String, SymbolInformationKind)> = + HashMap::new(); + let mut def_file: HashMap = HashMap::new(); + for (_, d) in &all { + let (Some(MonikerResult::Moniker(m)), Some(def)) = (&d.moniker, d.definition) else { + continue; + }; + let id = identity(&m.identifier); + by_pos.insert( + (def.file_id, u32::from(def.range.start())), + (id.clone(), d.kind), + ); + if let Some(f) = rel(def.file_id) { + def_file.insert(id, f); + } + } + // The id each reference position resolves to, for `import` rows. + let mut occ_at: HashMap<(ra_ap_ide::FileId, u32), String> = HashMap::new(); + for (_, d) in &all { + let Some(MonikerResult::Moniker(m)) = &d.moniker else { + continue; + }; + let id = identity(&m.identifier); + for r in &d.references { + occ_at.insert( + (r.range.file_id, u32::from(r.range.range.start())), + id.clone(), + ); + } + } + let target_kept = |id: &str| def_file.get(id).is_some_and(|f| keep(f)); + let mut parsed: HashMap = HashMap::new(); + for f in &idx.files { + let Some(path) = rel(f.file_id) else { continue }; + let Ok(text) = analysis.file_text(f.file_id) else { + continue; + }; + println!("file\t{path}\t{:016x}", fnv(&text)); + if !keep(&path) { + continue; + } + let Ok(tree) = analysis.parse(f.file_id) else { + continue; + }; + for mc in tree.syntax().descendants().filter_map(ast::MacroCall::cast) { + let has_args = mc.token_tree().is_some_and(|tt| { + tt.syntax() + .descendants_with_tokens() + .any(|e| e.kind() == ra_ap_syntax::SyntaxKind::IDENT) + }); + if has_args { + let r = mc.syntax().text_range(); + println!( + "macro\t{path}\t{}\t{}", + u32::from(r.start()), + u32::from(r.end()) + ); + } + } + // One `import` row per leaf of every `use` tree: + // import + // + // The target is what rust-analyzer resolved the leaf's last + // segment to (for a glob: the module the glob reads). + for u in tree.syntax().descendants().filter_map(ast::Use::cast) { + let vis = if u.visibility().is_some() { + "pub" + } else { + "priv" + }; + let ur = u.syntax().text_range(); + for t in u.syntax().descendants().filter_map(ast::UseTree::cast) { + if t.use_tree_list().is_some() { + continue; + } + let glob = t.star_token().is_some(); + let Some(seg) = t.path().and_then(|p| p.segment()) else { + continue; + }; + let Some(nr) = seg.name_ref() else { continue }; + let lr = nr.syntax().text_range(); + let target = occ_at + .get(&(f.file_id, u32::from(lr.start()))) + .cloned() + .unwrap_or_else(|| "unresolved".into()); + let alias = t + .rename() + .and_then(|r| r.name()) + .map(|n| n.text().to_string()) + .unwrap_or_else(|| "-".into()); + println!( + "import\t{target}\t{path}\t{}\t{}\t{}\t{}\t{alias}\t{vis}\t{}", + u32::from(ur.start()), + u32::from(ur.end()), + u32::from(lr.start()), + u32::from(lr.end()), + if glob { "glob" } else { "name" } + ); + } + } + parsed.insert(f.file_id, tree); + let cfg = ra_ap_ide::DiagnosticsConfig::test_sample(); + if let Ok(diags) = + analysis.full_diagnostics(&cfg, ra_ap_ide::AssistResolveStrategy::None, f.file_id) + { + for dg in diags { + if dg.range.file_id != f.file_id { + continue; + } + println!( + "diag\t{}\t{path}\t{}\t{}\t{}", + dg.code.as_str(), + u32::from(dg.range.range.start()), + u32::from(dg.range.range.end()), + dg.message.replace(['\t', '\n'], " ") + ); + } + } + } + let role = + |file_id: ra_ap_ide::FileId, at: ra_ap_ide::TextSize, parsed: &mut HashMap<_, _>| { + let tree: &ra_ap_syntax::SourceFile = parsed + .entry(file_id) + .or_insert_with(|| analysis.parse(file_id).expect("parse")); + let tok = tree.syntax().token_at_offset(at).right_biased()?; + let u = tok.parent_ancestors().find_map(ast::Use::cast)?; + Some(if u.visibility().is_some() { + "pubuse" + } else { + "use" + }) + }; + let cfg_calls = ra_ap_ide::CallHierarchyConfig { + exclude_tests: false, + ra_fixture: ra_ap_ide::RaFixtureConfig::default(), + }; + for (_, d) in &all { + let Some(MonikerResult::Moniker(m)) = &d.moniker else { + continue; + }; + let id = identity(&m.identifier); + if let Some(def) = d.definition + && let Some(file) = rel(def.file_id) + && keep(&file) + { + let (bs, be, digest) = match d.definition_body { + Some(b) => { + let text = analysis.file_text(b.file_id).ok(); + let body = text.as_deref().and_then(|t| { + t.get(usize::from(b.range.start())..usize::from(b.range.end())) + }); + ( + u32::from(b.range.start()), + u32::from(b.range.end()), + body.map(|t| format!("{:016x}", fnv(t))) + .unwrap_or("none".into()), + ) + } + None => (0, 0, "none".into()), + }; + println!( + "node\t{id}\t{:?}\t{file}\t{}\t{}\t{bs}\t{be}\t{digest}", + d.kind, + u32::from(def.range.start()), + u32::from(def.range.end()) + ); + } + for r in &d.references { + let Some(file) = rel(r.range.file_id) else { + continue; + }; + if !(keep(&file) || target_kept(&id)) { + continue; + } + let kind = if r.is_definition { + "def" + } else { + role(r.range.file_id, r.range.range.start(), &mut parsed).unwrap_or("ref") + }; + println!( + "occ\t{id}\t{file}\t{}\t{}\t{kind}", + u32::from(r.range.range.start()), + u32::from(r.range.range.end()) + ); + } + // Calls out of every function-like definition in the workspace: + // rust-analyzer's call hierarchy, which resolves each callee the + // way the IDE does. A call through a trait bound or `dyn` + // resolves to the trait's method declaration (kind + // `TraitMethod`), which names a candidate set, not a callee. + let fn_like = matches!( + d.kind, + SymbolInformationKind::Function + | SymbolInformationKind::Method + | SymbolInformationKind::StaticMethod + | SymbolInformationKind::TraitMethod + ); + let Some(def) = d.definition else { continue }; + let Some(caller_file) = rel(def.file_id) else { + continue; + }; + if !fn_like { + continue; + } + let pos = FilePosition { + file_id: def.file_id, + offset: def.range.start(), + }; + let Ok(Some(items)) = analysis.outgoing_calls(&cfg_calls, pos) else { + continue; + }; + for it in items { + let target = it + .target + .focus_range + .and_then(|fr| by_pos.get(&(it.target.file_id, u32::from(fr.start())))); + let (callee, ckind) = match target { + Some((cid, k)) => (cid.clone(), format!("{k:?}")), + None => (format!("unknown:{}", it.target.name), "Unknown".to_string()), + }; + for fr in it.ranges { + let Some(file) = rel(fr.file_id) else { + continue; + }; + if !(keep(&file) || keep(&caller_file) || target_kept(&callee)) { + continue; + } + println!( + "call\t{id}\t{callee}\t{ckind}\t{file}\t{}\t{}", + u32::from(fr.range.start()), + u32::from(fr.range.end()) + ); + } + } + } + // Trait implementations: one row per impl block of a local trait or + // local type, with the ids the rows above use. + let impls = ra_ap_hir::attach_db(host.raw_database(), || { + let db = host.raw_database(); + let mut out = Vec::new(); + for c in Crate::all(db) + .into_iter() + .filter(|c| c.origin(db).is_local()) + { + graph_impls(db, c.root_module(db), &mut out); + } + out + }); + let mut seen = BTreeSet::new(); + for (trait_pos, self_pos, loc) in impls { + let Some((file_id, range)) = loc else { + continue; + }; + let Some(file) = rel(file_id) else { continue }; + let name = |p: Option<(ra_ap_ide::FileId, u32)>, none: &str| { + p.and_then(|p| by_pos.get(&p).map(|(id, _)| id.clone())) + .unwrap_or_else(|| none.into()) + }; + let tr = name(trait_pos, "inherent"); + let st = name(self_pos, "unknown"); + if !(keep(&file) || target_kept(&tr) || target_kept(&st)) { + continue; + } + let row = format!( + "impl\t{tr}\t{st}\t{file}\t{}\t{}", + u32::from(range.start()), + u32::from(range.end()) + ); + if seen.insert(row.clone()) { + println!("{row}"); + } + } + } println!( "static_index_ms\t{}\nfiles\t{files}\noccurrences\t{occ}\ntokens\t{toks}\ntokens_with_definition\t{defs}\nreference_rows\t{refs}", t.elapsed().as_millis() @@ -529,3 +850,63 @@ fn list_impls(db: &dyn ra_ap_hir::db::HirDatabase, m: Module, rows: &mut Vec; + +fn name_pos( + db: &dyn ra_ap_hir::db::HirDatabase, + src: Option>, +) -> NamePos { + use ra_ap_syntax::AstNode; + let src = src?; + let name = src.value.name()?; + let fr = src.with_value(name.syntax()).original_file_range_rooted(db); + Some((fr.file_id.file_id(db), u32::from(fr.range.start()))) +} + +/// `GRAPH`'s impl rows: (trait name, self type name, impl block range) for +/// every impl of a local ADT or trait. Duplicates (an impl of a local trait +/// for a local type is found from both) are removed by the caller. +fn graph_impls( + db: &dyn ra_ap_hir::db::HirDatabase, + m: Module, + out: &mut Vec<( + NamePos, + NamePos, + Option<(ra_ap_ide::FileId, ra_ap_ide::TextRange)>, + )>, +) { + use ra_ap_hir::HasSource; + let mut push = |imp: ra_ap_hir::Impl| { + let tr = imp.trait_(db).and_then(|t| name_pos(db, t.source(db))); + let st = imp.self_ty(db).as_adt().and_then(|a| match a { + ra_ap_hir::Adt::Struct(s) => name_pos(db, s.source(db)), + ra_ap_hir::Adt::Enum(e) => name_pos(db, e.source(db)), + ra_ap_hir::Adt::Union(u) => name_pos(db, u.source(db)), + }); + let loc = imp.source(db).map(|src| { + let fr = src.syntax().original_file_range_rooted(db); + (fr.file_id.file_id(db), fr.range) + }); + out.push((tr, st, loc)); + }; + for d in m.declarations(db) { + match d { + ModuleDef::Adt(a) => { + for imp in ra_ap_hir::Impl::all_for_type(db, a.ty(db)) { + push(imp); + } + } + ModuleDef::Trait(t) => { + for imp in ra_ap_hir::Impl::all_for_trait(db, t) { + push(imp); + } + } + _ => {} + } + } + for c in m.children(db) { + graph_impls(db, c, out) + } +} diff --git a/crates/sag-ra-probe/tools/kythe_calls_check.py b/crates/sag-ra-probe/tools/kythe_calls_check.py new file mode 100644 index 000000000..16de66cc3 --- /dev/null +++ b/crates/sag-ra-probe/tools/kythe_calls_check.py @@ -0,0 +1,88 @@ +#!/usr/bin/env python3 +"""Serve Charon-derived Kythe entries and check that /xrefs returns the pinned calls. + + python3 -I kythe_calls_check.py --tools DIR --entries entries.json --pin calls.pin.tsv [--port 18098] + +entries.json comes from `ruff_charon kythe --strip /` +(one JSON object per line). Each pin line is + + callee-signature path line (1-based) expression + +read off the source by hand: the call's text and where it starts. A pin passes +when Kythe returns, among the callee's references, an anchor in that file +starting at that line whose byte range covers exactly `expression`. The source +is read from --root to locate the expression; Kythe is the one asked whether +the call exists. Exit 1 on any failing pin. +""" +import argparse +import json +import os +import re +import subprocess +import sys +import tempfile +import urllib.parse +import urllib.request + +sys.path.insert(0, os.path.dirname(os.path.abspath(__file__))) +from kythe_xrefs_check import query_xrefs, run, wait_up # noqa: E402 + + +def load_pins(path, root): + pins = [] + for line in open(path): + if line.startswith("#") or not line.strip(): + continue + callee, rel, lineno, expr = line.rstrip("\n").split("\t") + text = open(os.path.join(root, rel), "rb").read() + lines = text.split(b"\n") + n = int(lineno) + start_of_line = sum(len(l) + 1 for l in lines[: n - 1]) + col = lines[n - 1].find(expr.encode()) + if col < 0: + raise SystemExit(f"pin expression not on {rel}:{n}: {expr!r}") + start = start_of_line + col + pins.append((callee, rel, start, start + len(expr.encode()), expr)) + return pins + + +def main(argv=None): + ap = argparse.ArgumentParser(description=__doc__.split("\n")[0]) + ap.add_argument("--tools", required=True) + ap.add_argument("--entries", required=True) + ap.add_argument("--pin", required=True) + ap.add_argument("--root", required=True) + ap.add_argument("--corpus", default="lance-graph") + ap.add_argument("--port", type=int, default=18098) + args = ap.parse_args(argv) + pins = load_pins(args.pin, args.root) + bad = 0 + with tempfile.TemporaryDirectory() as work: + sorted_ents = os.path.join(work, "entries.bin") + with open(args.entries) as fi, open(sorted_ents, "wb") as fo: + run([os.path.join(args.tools, "entrystream"), "--read_format=json", "--sort", + "--write_format=delimited"], stdin=fi, stdout=fo) + tables = os.path.join(work, "tables") + run([os.path.join(args.tools, "write_tables"), "--entries", sorted_ents, "--out", tables], + stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL) + base = f"http://localhost:{args.port}" + srv = subprocess.Popen([os.path.join(args.tools, "http_server"), "--serving_table", tables, + "--listen", f"localhost:{args.port}"], + stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL) + try: + wait_up(base, srv) + for callee, rel, start, end, expr in pins: + _, refs, _ = query_xrefs(base, args.corpus, callee) + hit = (rel, start, end) in refs + bad += not hit + print(f"{'PASS' if hit else 'FAIL'}\tcall {callee} at {rel} {start}..{end} {expr!r}" + + ("" if hit else f"\tkythe has {sorted(k for k in refs if k[0] == rel)}")) + finally: + srv.terminate() + srv.wait(timeout=5) + print(f"call pins: {len(pins) - bad} pass, {bad} fail") + sys.exit(1 if bad else 0) + + +if __name__ == "__main__": + main()