From c32e550e58a4225869f0526dc7790bf41d86417f Mon Sep 17 00:00:00 2001 From: Claude Date: Tue, 27 Oct 2026 08:22:07 +0000 Subject: [PATCH] board: re-scope composite-key issue to destination binding; SIMD and dependency scope notes ISS-REPORT-NO-COMPOSITE-KEY-GROUP-FOLD said Report lacked a composite-key group fold. It has one: mask-risc GroupKey::Pair, Quack GroupAddr::Pair and ndarray masked_group_*_pair all landed on 2026-09-23 (869ba24d, #1275), the day the issue was filed. lance-graph-report/src/exec.rs has named the real gap since #1331: destination binding / coordinate resolution. Report refuses to lower to dense Pair on purpose. The issue and the D-RPT-5 row now say so, with dated supersession notes. The 32x12 / 4M-row / 25-pass / 177 ms measurement is kept verbatim. Also, on existing rows only (no new debt, per H-12): - TD-SIMD-SWEEP-W2: the ndarray_bridge intrinsics compile only without the default ndarray-hpc feature. - TD-NDARRAY-PATCH-0_16: lance-index 13.0.0 still requires crates.io ndarray 0.16.1 (validated against the published crate); fork path conflicts with E-LANCE-IS-UPSTREAM-AUTHORITATIVE-1. The new entry records CP-1..CP-7, the tests still required (wasm mul_add parity, cross-implementation ternlog parity, bound-ordinal pass count) and the operator decisions (EXO-112, EXO-46, ternlog placement). Docs only. No source, manifest, ABI or contract change. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_015yemQWusMpV2ueqrHkMnkB --- .claude/board/ISSUES.md | 67 ++++++++- .claude/board/STATUS_BOARD.md | 2 +- .claude/board/TECH_DEBT.md | 2 + ...ntegration-composite-key-and-simd-scope.md | 128 ++++++++++++++++++ .claude/board/entries/README.md | 3 +- 5 files changed, 198 insertions(+), 4 deletions(-) create mode 100644 .claude/board/entries/2026-10-27-harvest-integration-composite-key-and-simd-scope.md diff --git a/.claude/board/ISSUES.md b/.claude/board/ISSUES.md index 4a8a28f0f..9cf4a5244 100644 --- a/.claude/board/ISSUES.md +++ b/.claude/board/ISSUES.md @@ -334,7 +334,70 @@ compares only `edge.w_slot()` with the mailbox's slot and reads no classid. ## ISS-REPORT-NO-COMPOSITE-KEY-GROUP-FOLD — a 2-D fold costs one pass per partition member (2026-09-23) -**Status:** OPEN. **Basis:** VERIFIED-IN-CODE. mask-risc's `GroupKey` is one +> **⊘ ROOT CAUSE SUPERSEDED (2026-10-27).** The symptom and the measurement +> below stand. The cause and the closing condition do not: a composite-key +> group fold already exists. What Report lacks is **destination binding / +> coordinate resolution**, not a fold primitive. The original text is kept +> below as history. +> +> **Status:** OPEN, re-scoped. **Basis:** VERIFIED-IN-CODE at lance-graph +> `aa569bc5`, ndarray `af4910a1`: +> - mask-risc `GroupKey::Pair { hi, lo, stride }` (`ir.rs:505`), executed for +> Count / MinI32 / MaxI32 / SumSymI32 (`exec.rs:1995-2024`) and power sums +> (`exec.rs:2056`), with a reference oracle (`reference.rs:695, :740`). +> - Quack `GroupAddr::Pair` lowers onto it (`lance-graph-quack/src/lib.rs:2460`); +> `tests/duckdb_differential.rs:783-823` checks it against DuckDB. +> - ndarray exports `masked_group_count_u32_pair` and the min / max / sum / +> sum_sym / power-sum `_pair` siblings (`src/simd.rs:933-957`). +> - `lance-graph-report/src/exec.rs:42-69` names the gap itself: once a compact +> destination ordinal or functional reference is bound, the existing +> `GroupAddr::Local` / `Via` keyed fold is the execution primitive. +> `CoordSpec` has only `Field`, `Bucket` and `MaskSet`, so Report cannot bind +> a functional reference (`user → department`), a composed route, or several +> semantic coordinates to one compact destination ordinal. +> - Report deliberately does not lower to `Pair`: `hi · stride + lo` addresses +> the dense Cartesian product, which is valid only where a consumer asks for a +> dense mixed-radix destination universe. +> +> **History of the stale premise.** `GroupKey::Pair` / `GroupAddr::Pair` landed +> on 2026-09-23 (`869ba24d`, merged in #1275), the day this entry was filed. A +> Report lowering onto `Pair` was tried and reverted before merge in #1331, and +> `9e26e376` (2026-10-05) moved the gap wording in `exec.rs`; the record is +> `entries/2026-10-05-report-pair-key-and-stack-recon.md`. This entry and the +> D-RPT-5 row were not updated then. +> +> **Open question (replaces "what closes it"):** can a source-attested semantic +> destination be bound to a compact, witnessed local coordinate, so that Report +> uses the existing Quack keyed fold (`Local` / `Via`) instead of one pass per +> partition member? Relation to other contracts, none of which provides this +> binding today: +> - **D-DT64-Q0** (`plans/d-dt64-q0-borrowed-quack-binding-seam-v1.md`, #1799) +> lists a "global↔local coordinate witness" among its binding components and +> assigns it to the D-RMAP / EXO-46 preparation membrane. DT64 records and +> validates bindings. It does not issue local ordinals, and its first +> executable proof is NOT RUN. +> - **EXO-46** (counted global→focus binding membrane for production) is OPEN. +> ADMISSION-B (`entries/2026-10-20-six-decisions.md` §1) admits same-table, +> version-witnessed seeks and frozen codebook decoding as representation +> translation. It does not admit a position read from another table's column. +> A `Via` destination must stay inside that ruling. +> - **DR64³** (D-NFC, `plans/d-dr3-holo-onnx-s3a-s3c-v1.md`) is the intended +> native relational/focus representation, but its third axis has no type +> (NFR-OP-2 OPEN). Like Report refusing dense `Pair`, Q0 Phase Q3 says a dense +> cube is not a prerequisite and keeps the direct path when construction cost +> dominates. +> +> **Not measured yet:** whether a bound compact ordinal turns the 25-pass +> 32×12 / 4M-row case into one keyed-fold pass plus binding cost. That needs +> the binding to exist first, and binding cost has to be counted separately +> from fold cost. +> +> **What does NOT close it (unchanged):** a report-side row loop or a +> population-sized composite key lane. Also: automatic lowering to dense +> `Pair`, and a query-local interner presented as the canonical path (it is a +> fallback only). + +**Status (historical, 2026-09-23):** OPEN. **Basis:** VERIFIED-IN-CODE. mask-risc's `GroupKey` is one `U32` lane or a `Via` hop, so `lance-graph-report` folds the widest coordinate as the key and evaluates every other coordinate as a PARTITION. A partition is a member conjunct, run tile-local, one pass per member tuple. @@ -343,7 +406,7 @@ member conjunct, run tile-local, one pass per member tuple. - **Consequence:** a high-cardinality, densely-observed partition side exceeds `PlannerPolicy::pass_budget`, and the plan is REFUSED (`ReportError::PassBudget`). It is never run slowly, and never allocated densely. -- **What closes it:** a multi-lane (composite-key) masked group fold in `ndarray::simd`, +- **What closes it (⊘ superseded 2026-10-27; this fold already existed, see above):** a multi-lane (composite-key) masked group fold in `ndarray::simd`, named in mask-risc as a `GroupKey` variant, so that ONE pass folds (a, b) → a dense or sparse cell. - **What does NOT close it:** a report-side row loop, or a population-sized composite diff --git a/.claude/board/STATUS_BOARD.md b/.claude/board/STATUS_BOARD.md index 25053e566..86c238f33 100644 --- a/.claude/board/STATUS_BOARD.md +++ b/.claude/board/STATUS_BOARD.md @@ -932,7 +932,7 @@ Entry: `entries/2026-09-23-report-plan-zero-copy-pivot-docir-convergence.md`. | **D-RPT-2** | z8run `crates/z8run-lance`: 8 native nodes, handle envelopes, physical-key cache (AdaWorldAPI/z8run#1) | Shipped | z8run-lance `tests/nodes.rs` | | **D-RPT-3** | OGAR `SlotOutcome::Grid` / `grid_of` / typst `emit_grid` (AdaWorldAPI/OGAR#307) | Shipped | ogar-doc-ir resolver grid tests | | **D-RPT-4** | `crates/lance-graph-report-ogar`: ReportSource (DocObjectSource) over any object source; Live/Revision; emit_typst | Shipped | `tests/convergence.rs` | -| **D-RPT-5** | composite-key (multi-lane) group fold primitive in ndarray::simd / mask-risc | Queued | `ISS-REPORT-NO-COMPOSITE-KEY-GROUP-FOLD` | +| **D-RPT-5** | composite-key (multi-lane) group fold primitive in ndarray::simd / mask-risc | ⊘ Superseded as scoped (2026-10-27): the primitive already exists (`GroupKey::Pair`, `GroupAddr::Pair`, ndarray `masked_group_*_pair`, since 2026-09-23). Open remainder: Report destination binding / coordinate resolution, blocked on EXO-46 | `ISS-REPORT-NO-COMPOSITE-KEY-GROUP-FOLD` (re-scoped); `entries/2026-10-27-harvest-integration-composite-key-and-simd-scope.md` | ## D-WFL — CodeRabbit round 3 on `9fd6956d` (2026-09-19): three findings, all valid, all outside the diff diff --git a/.claude/board/TECH_DEBT.md b/.claude/board/TECH_DEBT.md index 893433f99..07c3f9d5b 100644 --- a/.claude/board/TECH_DEBT.md +++ b/.claude/board/TECH_DEBT.md @@ -2278,6 +2278,7 @@ multi-antecedent path for real. - **Status:** Open — **HALF DONE** (regraded 2026-09-07, plan inventory): `types.rs` carries 0 raw-intrinsic lines — `hamming_distance_dispatch` routes through `ndarray::hpc::bitwise::hamming_distance_raw` (`types.rs:462`), migrated on its own path. `ndarray_bridge.rs` is unchanged: 66 raw lines, `_mm512_popcnt_epi64` at `:465` and `:493` where `U64x8::popcnt`/`xor_popcount` (W1a-#5, shipped) should be consumed. The "22 raw ops" for `types.rs` below is history. - **Description:** `crates/lance-graph/src/graph/blasgraph/types.rs` (22 raw ops) + `ndarray_bridge.rs` (60 raw ops, plus 2 AP-SIMD-7 duplicate LUTs) — Hamming + popcount workload over palette CSR/CSC. Migration consumes `U64x8::popcnt`, existing `U8x64::nibble_popcount_lut`, and `ndarray::hpc::bitwise::hamming_distance_raw`. - **Cross-ref:** `crates/lance-graph/src/graph/blasgraph/{types.rs:440-506, ndarray_bridge.rs:149-299}`; TD-NDARRAY-SIMD-POPCOUNT-U64. +- **Scope note (2026-10-27, VERIFIED-IN-CODE at `aa569bc5`):** every raw-intrinsic function in `ndarray_bridge.rs` (`hamming_avx2` `:264-267`, `popcount_avx2` `:313`, the AVX-512BW pair `:359`, `:403`, the VPOPCNTDQ pair `:449`, `:479`; `_mm512_popcnt_epi64` still at `:465`, `:493`) sits under `#[cfg(not(feature = "ndarray-hpc"))]`. `ndarray-hpc` is in `crates/lance-graph/Cargo.toml:107` `default`, so the default build compiles none of them; `dispatch_hamming` routes to `ndarray::hpc::bitwise::hamming_distance_raw` (`:149-152`). The debt is real only for `--no-default-features` builds. Same finding as ndarray's `coresearch-2026-10-11-simd-fleet.md` X16. Whether that fallback should exist at all depends on the contract/ndarray dependency shape (EXO-112). The live path is itself a direct `ndarray::hpc::*` consumer; per H-12 the `hpc::*` consumers are classified before any new debt row is opened. No new row here. --- @@ -4670,6 +4671,7 @@ W6 entropy-ledger reframe of `DEEPNSM-NSM-1`. - **Status:** **Open** (2026-05-28). Until resolved, BLOCKED(D) in the workspace root `Cargo.toml` stays open and the transitive uses crates.io ndarray 0.16.1. - **Introduced-by-PR:** N/A (latent since #423's lance 4→6 bump; the 0.16-vs-0.17 gap was always there but invisible without an explicit patch attempt). - **Payoff-estimate:** small if path 1 is taken (a `0.16-fork` branch + patch line) once the AdaWorldAPI patches' relevance to 0.16 is audited; otherwise gated on upstream. +- **⊘ Premise refresh (2026-10-27, per EXO-113):** "lance-index 6.0.0" is lance-6-era wording; the workspace is on `lance-index = "13.*"` (`Cargo.toml:385`). The substance holds at 13: the published `lance-index-13.0.0` crate (fetched from crates.io, `Cargo.toml:363-365`) declares `ndarray = "0.16.1"` with `matrixmultiply-threading`, and the only source file naming ndarray is `src/vector/bq/builder.rs`. So lance/lancedb reach ndarray only transitively, via lance-index, from crates.io, never the AdaWorldAPI fork. Path 3 (fork `lance-index`) conflicts with `E-LANCE-IS-UPSTREAM-AUTHORITATIVE-1` (lance family consumed from crates.io only) and should not be taken without a new operator ruling. Paths 1 and 2 are unchanged. `crates/lance-graph/Cargo.toml:27-28` still carries a lance-6.0-era comment ("arrow + datafusion bumped to 58 + 53 to align with lance 6.0's transitive versions"); the workspace is arrow 58 / datafusion 54. Correcting that comment is a manifest edit, so it was left out of the docs-only PR that recorded this note. ## TD-DEEPNSM-CLIPPY-195 — 12 pre-existing default-clippy lints in deepnsm (clippy 1.95 bump) diff --git a/.claude/board/entries/2026-10-27-harvest-integration-composite-key-and-simd-scope.md b/.claude/board/entries/2026-10-27-harvest-integration-composite-key-and-simd-scope.md new file mode 100644 index 000000000..d0e7dd3fd --- /dev/null +++ b/.claude/board/entries/2026-10-27-harvest-integration-composite-key-and-simd-scope.md @@ -0,0 +1,128 @@ +# Harvest integration 2026-10-27: composite-key root cause, SIMD scope, dependency docs + +**Status:** FINDING (VERIFIED-IN-CODE), board corrections only. No source, +manifest, ABI or contract change. Nothing merged or deployed. +**Inputs:** evidence harvest session `session_01MUeXRoWwaJE5wTsjp6ujLb` at +lance-graph `aa569bc5c41ca38edf1ce3db945007543b0c82f6` and ndarray +`af4910a1f13c23b2f0cab06f2f3a06a14eb1387f`. Integrated and re-verified by +session `session_015yemQWusMpV2ueqrHkMnkB`. Both revisions were still the tip +of `main` on 2026-10-27. Every harvested claim below was re-read in source +before it was used. None was taken from the harvest alone. + +## CP-1: ADOPT. Composite-key issue root cause corrected + +`ISS-REPORT-NO-COMPOSITE-KEY-GROUP-FOLD` said the composite-key fold was +missing. It exists: `GroupKey::Pair` (mask-risc `ir.rs:505`, executed at +`exec.rs:1995-2085`), Quack `GroupAddr::Pair` (`lib.rs:2460`, DuckDB +differential at `tests/duckdb_differential.rs:783-823`), and ndarray +`masked_group_*_pair` (`src/simd.rs:933-957`). It landed in `869ba24d` (#1275) +on 2026-09-23, the same day the issue was filed. The gap Report actually has is +named in `lance-graph-report/src/exec.rs:42-69` (wording moved in `9e26e376`, +#1331): destination binding / coordinate resolution. Report refuses to lower to +dense `Pair` on purpose. + +Edited in place, history kept: the ISSUES entry (supersession block, original +text marked historical, the 32×12 / 4M rows / 25 passes / 177 ms measurement +unchanged) and the D-RPT-5 status cell. + +**The remaining question:** can a source-attested semantic destination be +bound to a compact, witnessed local coordinate so that Report uses the existing +`Local` / `Via` keyed fold? D-DT64-Q0 names a "global↔local coordinate witness" +component and assigns it to D-RMAP / EXO-46. DT64 validates bindings and issues +no ordinals. Its executable proof is NOT RUN. EXO-46 is OPEN. ADMISSION-B +limits a `Via` destination to same-table, version-witnessed seeks. DR64³ has +no third-axis type (NFR-OP-2). None of these provides the binding today. + +## CP-2: ADOPT. `ndarray_bridge` intrinsics are a non-default fallback + +All raw-intrinsic Hamming/popcount functions in +`crates/lance-graph/src/graph/blasgraph/ndarray_bridge.rs` are under +`#[cfg(not(feature = "ndarray-hpc"))]`. `ndarray-hpc` is a default feature +(`crates/lance-graph/Cargo.toml:107`). Recorded as a scope note on the existing +`TD-SIMD-SWEEP-W2`. No new row (H-12). + +## CP-3: TEST REQUIRED. wasm `mul_add` and `reduce_sum` are separate facts + +Two independent observations in ndarray, neither explaining the other: + +1. **`mul_add`.** `src/simd_wasm.rs` now rounds once by design. f32 is + computed in f64 with round-to-odd addition (`fma_f32_in_f64`, `:2461`); f64 + uses an emulated Boldo–Melquiond FMA inside a checked operand range with a + lane fallback (`:551-563`). `crates/simd-masking-parity` + `check_mul_add_single_rounding` compares against an exact oracle with + double-rounding tie fixtures. `simd-matrix.yaml:380-383` runs it for wasm + with and without `simd128`. That is sampled evidence (the f32 arm runs 4096 + rounds of 16 lanes; the f64 arm's fixture count was not read), not a proof. ndarray's own `.claude/blackboard.md` (2026-10-11, + "Open: `mul_add` not bit-identical across realizations (wasm)") and + `knowledge/wasm-simd-consumer-guide.md` (table row "`mul_add`: two + roundings") were never reconciled with the change. The guide's row is now + stale for the default build. The `relaxed-simd` caveat (`simd_wasm.rs:37-38`) + is a separate leg that the parity run does not cover. +2. **`reduce_sum`.** wasm uses a balanced tree, the scalar backend a + sequential fold (consumer guide table). That is an ordering difference in a + different operation. It does not bear on `mul_add` and does not retire the + warning. + +**Required:** a parity result that names realization, target features and +fixture set for `mul_add` (with `relaxed-simd` stated as in or out of scope), +and a separate statement of the `reduce_sum` tolerance contract. Both are +ndarray-side edits, out of scope for this lance-graph PR. + +## CP-4: AMEND. The ternlog runtime dispatcher already exists, twice + +- mask-risc `src/ternlog_dispatch.rs`: 256-arm `match`, generated by + `tools/gen_ternlog_dispatch.py`, checked in CI + (`rust-test.yml:131-135`, `--check`). No lance-graph crate outside mask-risc + calls it. +- lance-graph-java `native/lgj-abi/src/kernels.rs:360-361` + `simd_mask_ternlog_assign_dyn`: a separate 16×16 function-pointer table, + behind `lgj_mask_ternlog(…, imm: u8)` (`exports.rs:853`). +- MedCare-rs `spog_masks.rs` uses const-generic immediates only; no runtime + dispatch. + +So there are two runtime consumers with two independent implementations. That +is the measured demand. Before anything moves into ndarray: a test showing both +agree on all 256 immediates over a shared fixture, and a decision on dispatch +form (`match` vs table), which has not been measured. + +## CP-5: ADOPT. EXO-111 locations refreshed, EXO-112 kept as blocker + +`crates/lance-graph-contract/src/mul.rs` at `aa569bc5`: `is_x86_feature_detected!` +at `:974-975`, `use core::arch::x86_64::*` at `:1086`, `#[target_feature]` +functions at `:1098`, `:1124`, `:1140`, `:1201`, `:1242`. Unchanged from the +issue text. EXO-112 still blocks it in Linear. + +## CP-6: DEFER + +The contract's zero-dependency comment (`lance-graph-contract/Cargo.toml:10-17`) +follows EXO-112's unresolved operator decision. Not edited. + +## CP-7: ADOPT after validation. lance-6-era dependency text + +Fetched the published `lance-index-13.0.0` crate: `Cargo.toml:363-365` +`ndarray = "0.16.1"`, used only in `src/vector/bq/builder.rs`. EXO-113 is +exact. A dated premise refresh was added to `TD-NDARRAY-PATCH-0_16`. +Its path 3 (fork lance-index) conflicts with +`E-LANCE-IS-UPSTREAM-AUTHORITATIVE-1`. The stale comment at +`crates/lance-graph/Cargo.toml:27-28` ("arrow + datafusion bumped to 58 + 53 to align with lance 6.0's transitive versions"; +actual: arrow 58, datafusion 54, lance 13) is a manifest edit and is left for a +separate PR. + +## H-12 + +No new debt rows were opened. The `ndarray::hpc::*` consumers (for example +`ndarray_bridge.rs:152`, `types.rs:462`) still need classifying before any are +filed. + +## OPEN + +- Tests: CP-3 `mul_add` parity statement (ndarray); CP-4 cross-implementation + ternlog parity, 256 immediates; Report bound-ordinal pass count vs the 25-pass + baseline (after a binding exists). +- Operator decisions: EXO-112 dependency shape (blocks EXO-111, CP-6); EXO-46 + production binding membrane (blocks the Report question); whether CP-4's + dispatcher moves into ndarray; TD-NDARRAY-PATCH-0_16 path 1 vs 2. +- Context only, no action taken: the read-only workbench at `mcp.exo.red` + (builds on D-SAG-6, #1760). Candidate first queries: EXO-111 blocked by + EXO-112; symbol ownership and facade mapping; Report destination binding; DT64 + identity and ClassId predicate binding; DR focus coordinates and Quack. diff --git a/.claude/board/entries/README.md b/.claude/board/entries/README.md index 96134545d..486739bb2 100644 --- a/.claude/board/entries/README.md +++ b/.claude/board/entries/README.md @@ -25,10 +25,11 @@ index row, (3) no duplicate entry id. Checks 1 and 2 are deliberately opposite directions; the stranding this convention prevents shows up in exactly one of them, never both. -487 entries, 2026-08-06 .. 2026-10-26. +488 entries, 2026-08-06 .. 2026-10-27. | date | entry id | finding | file | |---|---|---|---| +| 2026-10-27 | `harvest-integration-composite-key-and-simd-scope` | | [2026-10-27-harvest-integration-composite-key-and-simd-scope.md](2026-10-27-harvest-integration-composite-key-and-simd-scope.md) | | 2026-10-26 | `D-ONNX-INTAKE-0` | | [2026-10-26-onnx-frontend-intake-bakeoff.md](2026-10-26-onnx-frontend-intake-bakeoff.md) | | 2026-10-26 | `D-DT64-0` | | [2026-10-26-dt64-holographic-interface.md](2026-10-26-dt64-holographic-interface.md) | | 2026-10-26 | `D-DR64C-A` | | [2026-10-26-dr64-cubed-amendment.md](2026-10-26-dr64-cubed-amendment.md) |