Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
67 changes: 65 additions & 2 deletions .claude/board/ISSUES.md
Original file line number Diff line number Diff line change
Expand Up @@ -334,7 +334,70 @@ compares only `edge.w_slot()` with the mailbox's slot and reads no classid.

## ISS-REPORT-NO-COMPOSITE-KEY-GROUP-FOLD — a 2-D fold costs one pass per partition member (2026-09-23)

**Status:** OPEN. **Basis:** VERIFIED-IN-CODE. mask-risc's `GroupKey` is one
> **⊘ ROOT CAUSE SUPERSEDED (2026-10-27).** The symptom and the measurement
> below stand. The cause and the closing condition do not: a composite-key
> group fold already exists. What Report lacks is **destination binding /
> coordinate resolution**, not a fold primitive. The original text is kept
> below as history.
>
> **Status:** OPEN, re-scoped. **Basis:** VERIFIED-IN-CODE at lance-graph
> `aa569bc5`, ndarray `af4910a1`:
> - mask-risc `GroupKey::Pair { hi, lo, stride }` (`ir.rs:505`), executed for
> Count / MinI32 / MaxI32 / SumSymI32 (`exec.rs:1995-2024`) and power sums
> (`exec.rs:2056`), with a reference oracle (`reference.rs:695, :740`).
> - Quack `GroupAddr::Pair` lowers onto it (`lance-graph-quack/src/lib.rs:2460`);
> `tests/duckdb_differential.rs:783-823` checks it against DuckDB.
> - ndarray exports `masked_group_count_u32_pair` and the min / max / sum /
> sum_sym / power-sum `_pair` siblings (`src/simd.rs:933-957`).
> - `lance-graph-report/src/exec.rs:42-69` names the gap itself: once a compact
> destination ordinal or functional reference is bound, the existing
> `GroupAddr::Local` / `Via` keyed fold is the execution primitive.
> `CoordSpec` has only `Field`, `Bucket` and `MaskSet`, so Report cannot bind
> a functional reference (`user → department`), a composed route, or several
> semantic coordinates to one compact destination ordinal.
> - Report deliberately does not lower to `Pair`: `hi · stride + lo` addresses
> the dense Cartesian product, which is valid only where a consumer asks for a
> dense mixed-radix destination universe.
>
> **History of the stale premise.** `GroupKey::Pair` / `GroupAddr::Pair` landed
> on 2026-09-23 (`869ba24d`, merged in #1275), the day this entry was filed. A
> Report lowering onto `Pair` was tried and reverted before merge in #1331, and
> `9e26e376` (2026-10-05) moved the gap wording in `exec.rs`; the record is
> `entries/2026-10-05-report-pair-key-and-stack-recon.md`. This entry and the
> D-RPT-5 row were not updated then.
>
> **Open question (replaces "what closes it"):** can a source-attested semantic
> destination be bound to a compact, witnessed local coordinate, so that Report
> uses the existing Quack keyed fold (`Local` / `Via`) instead of one pass per
> partition member? Relation to other contracts, none of which provides this
> binding today:
> - **D-DT64-Q0** (`plans/d-dt64-q0-borrowed-quack-binding-seam-v1.md`, #1799)
> lists a "global↔local coordinate witness" among its binding components and
> assigns it to the D-RMAP / EXO-46 preparation membrane. DT64 records and
> validates bindings. It does not issue local ordinals, and its first
> executable proof is NOT RUN.
> - **EXO-46** (counted global→focus binding membrane for production) is OPEN.
> ADMISSION-B (`entries/2026-10-20-six-decisions.md` §1) admits same-table,
> version-witnessed seeks and frozen codebook decoding as representation
> translation. It does not admit a position read from another table's column.
> A `Via` destination must stay inside that ruling.
> - **DR64³** (D-NFC, `plans/d-dr3-holo-onnx-s3a-s3c-v1.md`) is the intended
> native relational/focus representation, but its third axis has no type
> (NFR-OP-2 OPEN). Like Report refusing dense `Pair`, Q0 Phase Q3 says a dense
> cube is not a prerequisite and keeps the direct path when construction cost
> dominates.
>
> **Not measured yet:** whether a bound compact ordinal turns the 25-pass
> 32×12 / 4M-row case into one keyed-fold pass plus binding cost. That needs
> the binding to exist first, and binding cost has to be counted separately
> from fold cost.
>
> **What does NOT close it (unchanged):** a report-side row loop or a
> population-sized composite key lane. Also: automatic lowering to dense
> `Pair`, and a query-local interner presented as the canonical path (it is a
> fallback only).

**Status (historical, 2026-09-23):** OPEN. **Basis:** VERIFIED-IN-CODE. mask-risc's `GroupKey` is one
`U32` lane or a `Via` hop, so `lance-graph-report` folds the widest coordinate as
the key and evaluates every other coordinate as a PARTITION. A partition is a
member conjunct, run tile-local, one pass per member tuple.
Expand All @@ -343,7 +406,7 @@ member conjunct, run tile-local, one pass per member tuple.
- **Consequence:** a high-cardinality, densely-observed partition side exceeds
`PlannerPolicy::pass_budget`, and the plan is REFUSED (`ReportError::PassBudget`).
It is never run slowly, and never allocated densely.
- **What closes it:** a multi-lane (composite-key) masked group fold in `ndarray::simd`,
- **What closes it (⊘ superseded 2026-10-27; this fold already existed, see above):** a multi-lane (composite-key) masked group fold in `ndarray::simd`,
named in mask-risc as a `GroupKey` variant, so that ONE pass folds (a, b) → a
dense or sparse cell.
- **What does NOT close it:** a report-side row loop, or a population-sized composite
Expand Down
2 changes: 1 addition & 1 deletion .claude/board/STATUS_BOARD.md
Original file line number Diff line number Diff line change
Expand Up @@ -932,7 +932,7 @@ Entry: `entries/2026-09-23-report-plan-zero-copy-pivot-docir-convergence.md`.
| **D-RPT-2** | z8run `crates/z8run-lance`: 8 native nodes, handle envelopes, physical-key cache (AdaWorldAPI/z8run#1) | Shipped | z8run-lance `tests/nodes.rs` |
| **D-RPT-3** | OGAR `SlotOutcome::Grid` / `grid_of` / typst `emit_grid` (AdaWorldAPI/OGAR#307) | Shipped | ogar-doc-ir resolver grid tests |
| **D-RPT-4** | `crates/lance-graph-report-ogar`: ReportSource (DocObjectSource) over any object source; Live/Revision; emit_typst | Shipped | `tests/convergence.rs` |
| **D-RPT-5** | composite-key (multi-lane) group fold primitive in ndarray::simd / mask-risc | Queued | `ISS-REPORT-NO-COMPOSITE-KEY-GROUP-FOLD` |
| **D-RPT-5** | composite-key (multi-lane) group fold primitive in ndarray::simd / mask-risc | ⊘ Superseded as scoped (2026-10-27): the primitive already exists (`GroupKey::Pair`, `GroupAddr::Pair`, ndarray `masked_group_*_pair`, since 2026-09-23). Open remainder: Report destination binding / coordinate resolution, blocked on EXO-46 | `ISS-REPORT-NO-COMPOSITE-KEY-GROUP-FOLD` (re-scoped); `entries/2026-10-27-harvest-integration-composite-key-and-simd-scope.md` |

## D-WFL — CodeRabbit round 3 on `9fd6956d` (2026-09-19): three findings, all valid, all outside the diff

Expand Down
2 changes: 2 additions & 0 deletions .claude/board/TECH_DEBT.md
Original file line number Diff line number Diff line change
Expand Up @@ -2278,6 +2278,7 @@ multi-antecedent path for real.
- **Status:** Open — **HALF DONE** (regraded 2026-09-07, plan inventory): `types.rs` carries 0 raw-intrinsic lines — `hamming_distance_dispatch` routes through `ndarray::hpc::bitwise::hamming_distance_raw` (`types.rs:462`), migrated on its own path. `ndarray_bridge.rs` is unchanged: 66 raw lines, `_mm512_popcnt_epi64` at `:465` and `:493` where `U64x8::popcnt`/`xor_popcount` (W1a-#5, shipped) should be consumed. The "22 raw ops" for `types.rs` below is history.
- **Description:** `crates/lance-graph/src/graph/blasgraph/types.rs` (22 raw ops) + `ndarray_bridge.rs` (60 raw ops, plus 2 AP-SIMD-7 duplicate LUTs) — Hamming + popcount workload over palette CSR/CSC. Migration consumes `U64x8::popcnt`, existing `U8x64::nibble_popcount_lut`, and `ndarray::hpc::bitwise::hamming_distance_raw`.
- **Cross-ref:** `crates/lance-graph/src/graph/blasgraph/{types.rs:440-506, ndarray_bridge.rs:149-299}`; TD-NDARRAY-SIMD-POPCOUNT-U64.
- **Scope note (2026-10-27, VERIFIED-IN-CODE at `aa569bc5`):** every raw-intrinsic function in `ndarray_bridge.rs` (`hamming_avx2` `:264-267`, `popcount_avx2` `:313`, the AVX-512BW pair `:359`, `:403`, the VPOPCNTDQ pair `:449`, `:479`; `_mm512_popcnt_epi64` still at `:465`, `:493`) sits under `#[cfg(not(feature = "ndarray-hpc"))]`. `ndarray-hpc` is in `crates/lance-graph/Cargo.toml:107` `default`, so the default build compiles none of them; `dispatch_hamming` routes to `ndarray::hpc::bitwise::hamming_distance_raw` (`:149-152`). The debt is real only for `--no-default-features` builds. Same finding as ndarray's `coresearch-2026-10-11-simd-fleet.md` X16. Whether that fallback should exist at all depends on the contract/ndarray dependency shape (EXO-112). The live path is itself a direct `ndarray::hpc::*` consumer; per H-12 the `hpc::*` consumers are classified before any new debt row is opened. No new row here.

---

Expand Down Expand Up @@ -4670,6 +4671,7 @@ W6 entropy-ledger reframe of `DEEPNSM-NSM-1`.
- **Status:** **Open** (2026-05-28). Until resolved, BLOCKED(D) in the workspace root `Cargo.toml` stays open and the transitive uses crates.io ndarray 0.16.1.
- **Introduced-by-PR:** N/A (latent since #423's lance 4→6 bump; the 0.16-vs-0.17 gap was always there but invisible without an explicit patch attempt).
- **Payoff-estimate:** small if path 1 is taken (a `0.16-fork` branch + patch line) once the AdaWorldAPI patches' relevance to 0.16 is audited; otherwise gated on upstream.
- **⊘ Premise refresh (2026-10-27, per EXO-113):** "lance-index 6.0.0" is lance-6-era wording; the workspace is on `lance-index = "13.*"` (`Cargo.toml:385`). The substance holds at 13: the published `lance-index-13.0.0` crate (fetched from crates.io, `Cargo.toml:363-365`) declares `ndarray = "0.16.1"` with `matrixmultiply-threading`, and the only source file naming ndarray is `src/vector/bq/builder.rs`. So lance/lancedb reach ndarray only transitively, via lance-index, from crates.io, never the AdaWorldAPI fork. Path 3 (fork `lance-index`) conflicts with `E-LANCE-IS-UPSTREAM-AUTHORITATIVE-1` (lance family consumed from crates.io only) and should not be taken without a new operator ruling. Paths 1 and 2 are unchanged. `crates/lance-graph/Cargo.toml:27-28` still carries a lance-6.0-era comment ("arrow + datafusion bumped to 58 + 53 to align with lance 6.0's transitive versions"); the workspace is arrow 58 / datafusion 54. Correcting that comment is a manifest edit, so it was left out of the docs-only PR that recorded this note.

## TD-DEEPNSM-CLIPPY-195 — 12 pre-existing default-clippy lints in deepnsm (clippy 1.95 bump)

Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,128 @@
# Harvest integration 2026-10-27: composite-key root cause, SIMD scope, dependency docs

**Status:** FINDING (VERIFIED-IN-CODE), board corrections only. No source,
manifest, ABI or contract change. Nothing merged or deployed.
**Inputs:** evidence harvest session `session_01MUeXRoWwaJE5wTsjp6ujLb` at
lance-graph `aa569bc5c41ca38edf1ce3db945007543b0c82f6` and ndarray
`af4910a1f13c23b2f0cab06f2f3a06a14eb1387f`. Integrated and re-verified by
session `session_015yemQWusMpV2ueqrHkMnkB`. Both revisions were still the tip
of `main` on 2026-10-27. Every harvested claim below was re-read in source
before it was used. None was taken from the harvest alone.

## CP-1: ADOPT. Composite-key issue root cause corrected

`ISS-REPORT-NO-COMPOSITE-KEY-GROUP-FOLD` said the composite-key fold was
missing. It exists: `GroupKey::Pair` (mask-risc `ir.rs:505`, executed at
`exec.rs:1995-2085`), Quack `GroupAddr::Pair` (`lib.rs:2460`, DuckDB
differential at `tests/duckdb_differential.rs:783-823`), and ndarray
`masked_group_*_pair` (`src/simd.rs:933-957`). It landed in `869ba24d` (#1275)
on 2026-09-23, the same day the issue was filed. The gap Report actually has is
named in `lance-graph-report/src/exec.rs:42-69` (wording moved in `9e26e376`,
#1331): destination binding / coordinate resolution. Report refuses to lower to
dense `Pair` on purpose.

Edited in place, history kept: the ISSUES entry (supersession block, original
text marked historical, the 32×12 / 4M rows / 25 passes / 177 ms measurement
unchanged) and the D-RPT-5 status cell.

**The remaining question:** can a source-attested semantic destination be
bound to a compact, witnessed local coordinate so that Report uses the existing
`Local` / `Via` keyed fold? D-DT64-Q0 names a "global↔local coordinate witness"
component and assigns it to D-RMAP / EXO-46. DT64 validates bindings and issues
no ordinals. Its executable proof is NOT RUN. EXO-46 is OPEN. ADMISSION-B
limits a `Via` destination to same-table, version-witnessed seeks. DR64³ has
no third-axis type (NFR-OP-2). None of these provides the binding today.

## CP-2: ADOPT. `ndarray_bridge` intrinsics are a non-default fallback

All raw-intrinsic Hamming/popcount functions in
`crates/lance-graph/src/graph/blasgraph/ndarray_bridge.rs` are under
`#[cfg(not(feature = "ndarray-hpc"))]`. `ndarray-hpc` is a default feature
(`crates/lance-graph/Cargo.toml:107`). Recorded as a scope note on the existing
`TD-SIMD-SWEEP-W2`. No new row (H-12).

## CP-3: TEST REQUIRED. wasm `mul_add` and `reduce_sum` are separate facts

Two independent observations in ndarray, neither explaining the other:

1. **`mul_add`.** `src/simd_wasm.rs` now rounds once by design. f32 is
computed in f64 with round-to-odd addition (`fma_f32_in_f64`, `:2461`); f64
uses an emulated Boldo–Melquiond FMA inside a checked operand range with a
lane fallback (`:551-563`). `crates/simd-masking-parity`
`check_mul_add_single_rounding` compares against an exact oracle with
double-rounding tie fixtures. `simd-matrix.yaml:380-383` runs it for wasm
with and without `simd128`. That is sampled evidence (the f32 arm runs 4096
rounds of 16 lanes; the f64 arm's fixture count was not read), not a proof. ndarray's own `.claude/blackboard.md` (2026-10-11,
"Open: `mul_add` not bit-identical across realizations (wasm)") and
`knowledge/wasm-simd-consumer-guide.md` (table row "`mul_add`: two
roundings") were never reconciled with the change. The guide's row is now
stale for the default build. The `relaxed-simd` caveat (`simd_wasm.rs:37-38`)
is a separate leg that the parity run does not cover.
2. **`reduce_sum`.** wasm uses a balanced tree, the scalar backend a
sequential fold (consumer guide table). That is an ordering difference in a
different operation. It does not bear on `mul_add` and does not retire the
warning.

**Required:** a parity result that names realization, target features and
fixture set for `mul_add` (with `relaxed-simd` stated as in or out of scope),
and a separate statement of the `reduce_sum` tolerance contract. Both are
ndarray-side edits, out of scope for this lance-graph PR.

## CP-4: AMEND. The ternlog runtime dispatcher already exists, twice

- mask-risc `src/ternlog_dispatch.rs`: 256-arm `match`, generated by
`tools/gen_ternlog_dispatch.py`, checked in CI
(`rust-test.yml:131-135`, `--check`). No lance-graph crate outside mask-risc
calls it.
- lance-graph-java `native/lgj-abi/src/kernels.rs:360-361`
`simd_mask_ternlog_assign_dyn`: a separate 16×16 function-pointer table,
behind `lgj_mask_ternlog(…, imm: u8)` (`exports.rs:853`).
- MedCare-rs `spog_masks.rs` uses const-generic immediates only; no runtime
dispatch.

So there are two runtime consumers with two independent implementations. That
is the measured demand. Before anything moves into ndarray: a test showing both
agree on all 256 immediates over a shared fixture, and a decision on dispatch
form (`match` vs table), which has not been measured.

## CP-5: ADOPT. EXO-111 locations refreshed, EXO-112 kept as blocker

`crates/lance-graph-contract/src/mul.rs` at `aa569bc5`: `is_x86_feature_detected!`
at `:974-975`, `use core::arch::x86_64::*` at `:1086`, `#[target_feature]`
functions at `:1098`, `:1124`, `:1140`, `:1201`, `:1242`. Unchanged from the
issue text. EXO-112 still blocks it in Linear.

## CP-6: DEFER

The contract's zero-dependency comment (`lance-graph-contract/Cargo.toml:10-17`)
follows EXO-112's unresolved operator decision. Not edited.

## CP-7: ADOPT after validation. lance-6-era dependency text

Fetched the published `lance-index-13.0.0` crate: `Cargo.toml:363-365`
`ndarray = "0.16.1"`, used only in `src/vector/bq/builder.rs`. EXO-113 is
exact. A dated premise refresh was added to `TD-NDARRAY-PATCH-0_16`.
Its path 3 (fork lance-index) conflicts with
`E-LANCE-IS-UPSTREAM-AUTHORITATIVE-1`. The stale comment at
`crates/lance-graph/Cargo.toml:27-28` ("arrow + datafusion bumped to 58 + 53 to align with lance 6.0's transitive versions";
actual: arrow 58, datafusion 54, lance 13) is a manifest edit and is left for a
separate PR.

## H-12

No new debt rows were opened. The `ndarray::hpc::*` consumers (for example
`ndarray_bridge.rs:152`, `types.rs:462`) still need classifying before any are
filed.

## OPEN

- Tests: CP-3 `mul_add` parity statement (ndarray); CP-4 cross-implementation
ternlog parity, 256 immediates; Report bound-ordinal pass count vs the 25-pass
baseline (after a binding exists).
- Operator decisions: EXO-112 dependency shape (blocks EXO-111, CP-6); EXO-46
production binding membrane (blocks the Report question); whether CP-4's
dispatcher moves into ndarray; TD-NDARRAY-PATCH-0_16 path 1 vs 2.
- Context only, no action taken: the read-only workbench at `mcp.exo.red`
(builds on D-SAG-6, #1760). Candidate first queries: EXO-111 blocked by
EXO-112; symbol ownership and facade mapping; Report destination binding; DT64
identity and ClassId predicate binding; DR focus coordinates and Quack.
3 changes: 2 additions & 1 deletion .claude/board/entries/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -25,10 +25,11 @@ index row, (3) no duplicate entry id. Checks 1 and 2 are deliberately
opposite directions; the stranding this convention prevents shows up in
exactly one of them, never both.

487 entries, 2026-08-06 .. 2026-10-26.
488 entries, 2026-08-06 .. 2026-10-27.

| date | entry id | finding | file |
|---|---|---|---|
| 2026-10-27 | `harvest-integration-composite-key-and-simd-scope` | | [2026-10-27-harvest-integration-composite-key-and-simd-scope.md](2026-10-27-harvest-integration-composite-key-and-simd-scope.md) |
| 2026-10-26 | `D-ONNX-INTAKE-0` | | [2026-10-26-onnx-frontend-intake-bakeoff.md](2026-10-26-onnx-frontend-intake-bakeoff.md) |
| 2026-10-26 | `D-DT64-0` | | [2026-10-26-dt64-holographic-interface.md](2026-10-26-dt64-holographic-interface.md) |
| 2026-10-26 | `D-DR64C-A` | | [2026-10-26-dr64-cubed-amendment.md](2026-10-26-dr64-cubed-amendment.md) |
Expand Down
Loading