Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 4 additions & 3 deletions .claude/board/STATUS_BOARD.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,9 +4,10 @@ Plan: `.claude/plans/sql-polyfill-r2il-v1.md`.

| D-id | Deliverable | Status | Gate |
|---|---|---|---|
| **D-SQLP-0** | Plan + board rows | In PR | — |
| **D-SQLP-1** | `FoldDialect` moved into a `lance-graph-r2il` crate | In PR (workspace-excluded, own CI step; probe 40/40 unchanged, crate 6 tests) | existing `row_bridge.rs` tests unchanged and green; one arity table |
| **D-SQLP-2** | First procedures (`filter_lt`, `sum`, `group_sum`, `margin`) as `PROC_DEF` bodies; `FoldDialect` wires `IntOr` and adds a fused mode (mask-risc `fuse`) beside the in-place mode | Queued | scalar-reference parity; disable runs red; both modes keep the same rows, fused holds ternlogs |
| **D-SQLP-0** | Plan + board rows | Shipped (#1476, #1478) | — |
| **D-SQLP-1** | `FoldDialect` moved into a `lance-graph-r2il` crate | Shipped (#1479; workspace-excluded, own CI step; probe 40/40 unchanged, crate 7 tests) | existing `row_bridge.rs` tests unchanged and green; one arity table |
| **D-TELEPORT-0** | Source-restricted fold probe: scan/join vs gated gather vs extent; range intersection; sorted-lane crossing point | Shipped (probe) | every arm equals the oracle on clustered and scattered layouts |
| **D-SQLP-2** | First procedures (`filter_lt`, `sum`, `group_sum`, `margin`) as inlined R2IL call sequences (`lance-graph-r2il::procs`); `FoldDialect` wires `IntOr` and adds a fused mode (mask-risc `fuse`) beside the in-place mode | In PR (7 tests; five disable runs red; probe suite unchanged) | scalar-reference parity; disable runs red; both modes keep the same rows, fused holds ternlogs |
| **D-SQLP-3** | Benchmark: direct ndarray vs hand-built `Program` vs interpreted procedure | Queued | interpreted ≤ 1.05× hand-built at 65,536 and 1M rows (pre-registered) |
| **D-SQLP-4** | Build-time generator: declared procedure → Rust with const-generic opcodes | Queued, gated on D-SQLP-3 | ≤ 1.05× direct ndarray; no interpreter dispatch in the hot loop |
| **D-SQLP-5** | quack, `plan_eval`, SAP binder lower through the procedures | Queued, gated on D-SQLP-3 | byte-identical `mask_risc::Program` for every pinned case; lance-graph-java first gets a `Program`-level test (today's `lowering_convergence` compares row counts); quack's `sql_null_3vl.rs` cases pass through the procedure path per mode (`MinI32`/`MaxI32` cases after W7 `MIN`/`MAX`) |
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,84 @@
# 2026-10-11 — Source-restricted folds: eliminated work vs accelerated work (D-TELEPORT-0)

**Status:** MEASURED. Probe only, no production primitive.
`crates/lance-graph-quack/examples/teleport_probe.rs`.

```text
CARGO_PROFILE_RELEASE_DEBUG=0 cargo run --release -p lance-graph-quack --example teleport_probe
```

The question: for `R(s,o) = K(s,o) AND B(o)`, does addressing only the rows
that can survive ELIMINATE work, or does each arm do the same work faster?
Every arm runs the shipped mask-risc executor; every count is asserted
against a per-row oracle. Work counters come from the masks and program
shape, never from a timer.

Setup: 1,048,576 edges (`s: i32`, `o: u32` = node row ordinal), 65,536 nodes,
`B` a resident node bitmap, `K = s < t`. Host: 4 shared vCPUs, load ~1.4;
timings vary up to ~1.5× between runs, so only order-of-magnitude
differences are read. No cache-miss counter (`perf` unavailable).

## Arms

- **A** `Pred K`, ungated `Gather`, `And`, `Count` (scan, join, filter).
- **B** `Pred K`, `Gather under K`, `Count` (fused, gated gather).
- **C** clustered layout only: `hi = partition_point(s < t)`, then `Gather`,
`Count` over `execute_extent(0..hi)`. The search is timed with the arm.

## Results (sel_B 0.5; sel_B 0.01 matches within noise)

| sel_K | A words / fk-reads / ns | B words / fk-reads / ns | C words / fk-reads / ns |
|---|---|---|---|
| 0.5 | 16384 / 1048576 / ~1.8 ms | 16384 / 523683 / ~1.1 ms | 8183 / 523683 / ~0.8 ms |
| 0.01 | 16384 / 1048576 / ~1.7 ms | 16384 / 10459 / ~0.19 ms | 164 / 10459 / ~14.5 µs |
| 0.0001 | 16384 / 1048576 / ~1.8 ms | 16384 / 126 / ~0.17 ms | 2 / 126 / ~0.48 µs |

Scattered layout: A ~1.8 ms flat; B falls to a floor of ~0.17–0.29 ms; C
does not exist.

## Reading

- **B eliminates foreign reads, not source reads.** Its fk-reads track
`|K|`, but it still reads every `s` and writes every tile. Its floor
(~0.17 ms) is the K predicate scan.
- **C eliminates source reads.** Words visited and fk-reads both track `|K|`;
the `s` lane is read ~20 times (the binary search). At 0.01% C is ~370×
faster than B, search included. At 50% it is ~1.4×: little is left to eliminate.
- **The scattered control behaves as predicted.** With no interval, the
restriction collapses to B, so C's gain is the restriction itself, not a
faster kernel.

## Schnittpunkt (intersection)

- Two row ranges ANDed lower today to `Range`, `Range`, `And`, `Count`:
3 passes, 49,152 scratch words, ~10 µs. The intersection
`[max(lo), min(hi))` computed before execution is one `Range` that hits
`Lowering::Range`: no slot, two words touched, ~28 ns. Same answer at
every width. **quack does not fold two ranges into one today**; the fused
path requires exactly one `Range` op (`ir.rs` `fused_terminal`).
- **What the ~28 ns is, and is not.** It is COUNT's closed form: the fused
`Range` terminal (`Lowering::Range`, `Count`/`Any` only) answers from the
two boundaries, `hi - lo`, and visits no row. A fold that needs the rows
(`SUM`, `GROUP_SUM`, `KEEP`, a gated gather) still visits every survivor of
the intersected range; there the rewrite removes two passes and two slots
but the cost stays proportional to the range. The finding is the
reduction of a three-pass computation to a property of two boundaries,
not the speed ratio.
- A monotone predicate on a sorted lane (`s < t`) scanned: ~150–190 µs.
The crossing point by `partition_point` (~20 reads) plus one fused
`Range`: ~0.1 µs. Only `prefix_facet`/`aperture_facet` under a witness
make this reduction today; no lowering does it for an `i32` compare on a
lane known to be sorted.

## Fact change vs rule change (arm B shape, scattered, sel_K 0.01)

Flip one `B` bit and re-run: ~0.40 ms. New program, compile, scratch, run:
~0.35 ms. Both are the run; neither change costs anything the run does not.

## OPEN

- An intersection rewrite (`Range ∧ Range → Range`) and a sorted-lane
monotone-compare → `Range` lowering are the two measured gains here. Both
need an order witness for any lane that is not a sealed facet lane; that
witness does not exist for `i32` lanes.
- Not measured: multi-range extents, cache misses, 16M rows.
3 changes: 2 additions & 1 deletion .claude/board/entries/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -25,10 +25,11 @@ index row, (3) no duplicate entry id. Checks 1 and 2 are deliberately
opposite directions; the stranding this convention prevents shows up in
exactly one of them, never both.

302 entries, 2026-08-06 .. 2026-10-11.
303 entries, 2026-08-06 .. 2026-10-11.

| date | entry id | finding | file |
|---|---|---|---|
| 2026-10-11 | `D-TELEPORT-0` | | [2026-10-11-teleport-probe-source-restriction.md](2026-10-11-teleport-probe-source-restriction.md) |
| 2026-10-11 | `synergy-ledger-classid-hotplug` | | [2026-10-11-synergy-ledger-classid-hotplug.md](2026-10-11-synergy-ledger-classid-hotplug.md) |
| 2026-10-11 | `coresearch-labels-address-lifetime` | | [2026-10-11-coresearch-labels-address-lifetime.md](2026-10-11-coresearch-labels-address-lifetime.md) |
| 2026-10-11 | `coresearch-instance-authz` | | [2026-10-11-coresearch-instance-authz.md](2026-10-11-coresearch-instance-authz.md) |
Expand Down
15 changes: 14 additions & 1 deletion .claude/plans/sql-polyfill-r2il-v1.md
Original file line number Diff line number Diff line change
Expand Up @@ -151,6 +151,19 @@ It follows the 2026-09-05 lance-graph ruling that planning moves to
`IntAdd` stays scalar-only. NULLs in a foreign value lane stay unmodelled,
as in quack. BASIS for this decision and the previous ones: quack's
`sql_where` doc and `tests/sql_null_3vl.rs`.
- **DECISION (2026-10-11): procedures are inlined until `PROC_*` executes.**
W2 writes each procedure as a Rust function that emits R2IL calls through
`Dialect::call` (`lance-graph-r2il::procs`). It adds no opcode; what reaches
mask-risc is what the calls written out by hand produce. BASIS: `ogar-loco`
routes `PROC_DEF`/`PROC_CALL`/`PROC_ARG` to `Dialect::call`, not the engine;
`PROC_CALL` has no fixed stack arity and no `body_refs`, so a call site
cannot name its target body. REVISIT WHEN: the OPEN item below is settled.
- **OPEN (OGAR): an executable `PROC_DEF`/`PROC_CALL` contract.** It needs
per-call arity, a `body_refs` entry naming the target, call frames with
`PROC_ARG` and returns, a rule for who owns `VAR_GET`/`VAR_SET` across a
call, and a corrected `interpret.rs` doc (it says the engine refuses
`PROC_DEF`; the dialect receives it). Until then a "procedure body" is the
inlined call sequence.
- **OPEN: SQL text parser.** DataFusion is in its grace period, so its parser
is not the route. Candidates: a standalone parser crate, or a declared-query
schema that needs no text parser for the compile-time mode. Decided in W6.
Expand All @@ -163,7 +176,7 @@ It follows the 2026-09-05 lance-graph ruling that planning moves to
|---|---|---|---|---|
| W0 | D-SQLP-0 | lance-graph | This plan + board rows | — |
| W1 | D-SQLP-1 | lance-graph | Move `FoldDialect` and its `Val`/`FoldError` types from `r2il-mask-abi-probe/tests/row_bridge.rs` into a workspace crate `lance-graph-r2il`; the probe's tests depend on it | every existing `row_bridge.rs` test passes unchanged against the crate; still one arity table in the tree (`ogar_r2il`'s); no new opcode; the named refusals still refuse |
| W2 | D-SQLP-2 | lance-graph | First procedures as `PROC_DEF` bodies over R2IL calls: `filter_lt` (`Load` + `IntSLess` + `KEEP`), `sum`, `group_sum`, and `margin` (two `SUM` + `IntSub`). Prerequisites in `FoldDialect`: wire `IntOr` onto `MaskOp::Or` like `IntAnd`; add the fused mode (a body's Boolean calls collected into one `BoolExpr` and lowered through mask-risc `fuse`) beside today's in-place mode | each answer equals a scalar reference on fixtures with ≥ 3 groups and both empty and full selections; no new opcode; disable runs: drop `KEEP` from `filter_lt` → red; swap `IntSub` operands in `margin` → red; an `IntOr` test that is red if it is lowered as `And`; a two-mode test where both keep the same rows, in-place emits per-call ops and fused holds ternlogs |
| W2 | D-SQLP-2 | lance-graph | First procedures as inlined R2IL call sequences (`procs`, see §4; `PROC_DEF` bodies once the OGAR contract exists): `filter_lt` (`Load` + `IntSLess` + `KEEP`), `sum`, `group_sum`, and `margin` (two `SUM` + `IntSub`). Prerequisites in `FoldDialect`: wire `IntOr` onto `MaskOp::Or` like `IntAnd`; add the fused mode (a body's Boolean calls collected into one `BoolExpr` and lowered through mask-risc `fuse`) beside today's in-place mode | each answer equals a scalar reference on fixtures with ≥ 3 groups and both empty and full selections; no new opcode; disable runs: drop `KEEP` from `filter_lt` → red; swap `IntSub` operands in `margin` → red; an `IntOr` test that is red if it is lowered as `And`; a two-mode test where both keep the same rows, in-place emits per-call ops and fused holds ternlogs |
| W3 | D-SQLP-3 | lance-graph | Benchmark `margin` three ways: direct `ndarray::simd` calls, hand-built mask-risc `Program`s, the interpreted procedure. N = 65,536 and 1,048,576; selectivity ~1 % and ~50 % | answers identical before timing; release, quiet machine, median of 101 runs, avx512 and avx2 reported separately. Pre-registered: interpreted ≤ 1.05× hand-built `Program` at both N |
| W4 | D-SQLP-4 | lance-graph | Build-time generator: a declared procedure body → Rust that calls mask-risc / `ndarray::simd` directly, with opcodes as const generics (the `ternlog::<IMM>` pattern) | generated code's answers identical to the interpreted body; generated ≤ 1.05× direct ndarray at both N; disassembly of the hot loop contains no interpreter dispatch |
| W5 | D-SQLP-5 | lance-graph, lance-graph-java | quack, `plan_eval` and the SAP binder lower through the procedures | quack and SAP: for every case their tests pin, the `mask_risc::Program` produced through the procedures is byte-identical to today's, per mode (`lower` against in-place, `lower_fused` against fused; `the_fused_lowering_trades_slots_for_passes` stays green). lance-graph-java: first add a `Program`-level test comparing the procedure lowering against the legacy `plan_lower` output (none exists today; `lowering_convergence` compares row counts only), then require byte-identical `Program`s for its 28 combine vectors. Nullable predicates: run the `sql_null_3vl.rs` cases through the procedure path in both modes, against the same Kleene oracle, byte-identical to `lower(sql_where(f))` / `lower_fused(sql_where(f))` respectively: required cases, nested `NOT`/`AND`/`OR`, gating-before-negation, NULL ≠ 0, random trees, deep chains, and the `Rows` / `Count` / `SumI32` aggregate cases. Its `MinI32` / `MaxI32` cases join once W7's `MIN` / `MAX` items land (explicit prerequisite for those cases only); add falsifiers only for shapes that suite does not cover. No other behaviour change |
Expand Down
Loading
Loading