diff --git a/.claude/board/STATUS_BOARD.md b/.claude/board/STATUS_BOARD.md index fae8c6a75..89f4c9753 100644 --- a/.claude/board/STATUS_BOARD.md +++ b/.claude/board/STATUS_BOARD.md @@ -4,9 +4,10 @@ Plan: `.claude/plans/sql-polyfill-r2il-v1.md`. | D-id | Deliverable | Status | Gate | |---|---|---|---| -| **D-SQLP-0** | Plan + board rows | In PR | — | -| **D-SQLP-1** | `FoldDialect` moved into a `lance-graph-r2il` crate | In PR (workspace-excluded, own CI step; probe 40/40 unchanged, crate 6 tests) | existing `row_bridge.rs` tests unchanged and green; one arity table | -| **D-SQLP-2** | First procedures (`filter_lt`, `sum`, `group_sum`, `margin`) as `PROC_DEF` bodies; `FoldDialect` wires `IntOr` and adds a fused mode (mask-risc `fuse`) beside the in-place mode | Queued | scalar-reference parity; disable runs red; both modes keep the same rows, fused holds ternlogs | +| **D-SQLP-0** | Plan + board rows | Shipped (#1476, #1478) | — | +| **D-SQLP-1** | `FoldDialect` moved into a `lance-graph-r2il` crate | Shipped (#1479; workspace-excluded, own CI step; probe 40/40 unchanged, crate 7 tests) | existing `row_bridge.rs` tests unchanged and green; one arity table | +| **D-TELEPORT-0** | Source-restricted fold probe: scan/join vs gated gather vs extent; range intersection; sorted-lane crossing point | Shipped (probe) | every arm equals the oracle on clustered and scattered layouts | +| **D-SQLP-2** | First procedures (`filter_lt`, `sum`, `group_sum`, `margin`) as inlined R2IL call sequences (`lance-graph-r2il::procs`); `FoldDialect` wires `IntOr` and adds a fused mode (mask-risc `fuse`) beside the in-place mode | In PR (7 tests; five disable runs red; probe suite unchanged) | scalar-reference parity; disable runs red; both modes keep the same rows, fused holds ternlogs | | **D-SQLP-3** | Benchmark: direct ndarray vs hand-built `Program` vs interpreted procedure | Queued | interpreted ≤ 1.05× hand-built at 65,536 and 1M rows (pre-registered) | | **D-SQLP-4** | Build-time generator: declared procedure → Rust with const-generic opcodes | Queued, gated on D-SQLP-3 | ≤ 1.05× direct ndarray; no interpreter dispatch in the hot loop | | **D-SQLP-5** | quack, `plan_eval`, SAP binder lower through the procedures | Queued, gated on D-SQLP-3 | byte-identical `mask_risc::Program` for every pinned case; lance-graph-java first gets a `Program`-level test (today's `lowering_convergence` compares row counts); quack's `sql_null_3vl.rs` cases pass through the procedure path per mode (`MinI32`/`MaxI32` cases after W7 `MIN`/`MAX`) | diff --git a/.claude/board/entries/2026-10-11-teleport-probe-source-restriction.md b/.claude/board/entries/2026-10-11-teleport-probe-source-restriction.md new file mode 100644 index 000000000..dea8edd9b --- /dev/null +++ b/.claude/board/entries/2026-10-11-teleport-probe-source-restriction.md @@ -0,0 +1,84 @@ +# 2026-10-11 — Source-restricted folds: eliminated work vs accelerated work (D-TELEPORT-0) + +**Status:** MEASURED. Probe only, no production primitive. +`crates/lance-graph-quack/examples/teleport_probe.rs`. + +```text +CARGO_PROFILE_RELEASE_DEBUG=0 cargo run --release -p lance-graph-quack --example teleport_probe +``` + +The question: for `R(s,o) = K(s,o) AND B(o)`, does addressing only the rows +that can survive ELIMINATE work, or does each arm do the same work faster? +Every arm runs the shipped mask-risc executor; every count is asserted +against a per-row oracle. Work counters come from the masks and program +shape, never from a timer. + +Setup: 1,048,576 edges (`s: i32`, `o: u32` = node row ordinal), 65,536 nodes, +`B` a resident node bitmap, `K = s < t`. Host: 4 shared vCPUs, load ~1.4; +timings vary up to ~1.5× between runs, so only order-of-magnitude +differences are read. No cache-miss counter (`perf` unavailable). + +## Arms + +- **A** `Pred K`, ungated `Gather`, `And`, `Count` (scan, join, filter). +- **B** `Pred K`, `Gather under K`, `Count` (fused, gated gather). +- **C** clustered layout only: `hi = partition_point(s < t)`, then `Gather`, + `Count` over `execute_extent(0..hi)`. The search is timed with the arm. + +## Results (sel_B 0.5; sel_B 0.01 matches within noise) + +| sel_K | A words / fk-reads / ns | B words / fk-reads / ns | C words / fk-reads / ns | +|---|---|---|---| +| 0.5 | 16384 / 1048576 / ~1.8 ms | 16384 / 523683 / ~1.1 ms | 8183 / 523683 / ~0.8 ms | +| 0.01 | 16384 / 1048576 / ~1.7 ms | 16384 / 10459 / ~0.19 ms | 164 / 10459 / ~14.5 µs | +| 0.0001 | 16384 / 1048576 / ~1.8 ms | 16384 / 126 / ~0.17 ms | 2 / 126 / ~0.48 µs | + +Scattered layout: A ~1.8 ms flat; B falls to a floor of ~0.17–0.29 ms; C +does not exist. + +## Reading + +- **B eliminates foreign reads, not source reads.** Its fk-reads track + `|K|`, but it still reads every `s` and writes every tile. Its floor + (~0.17 ms) is the K predicate scan. +- **C eliminates source reads.** Words visited and fk-reads both track `|K|`; + the `s` lane is read ~20 times (the binary search). At 0.01% C is ~370× + faster than B, search included. At 50% it is ~1.4×: little is left to eliminate. +- **The scattered control behaves as predicted.** With no interval, the + restriction collapses to B, so C's gain is the restriction itself, not a + faster kernel. + +## Schnittpunkt (intersection) + +- Two row ranges ANDed lower today to `Range`, `Range`, `And`, `Count`: + 3 passes, 49,152 scratch words, ~10 µs. The intersection + `[max(lo), min(hi))` computed before execution is one `Range` that hits + `Lowering::Range`: no slot, two words touched, ~28 ns. Same answer at + every width. **quack does not fold two ranges into one today**; the fused + path requires exactly one `Range` op (`ir.rs` `fused_terminal`). +- **What the ~28 ns is, and is not.** It is COUNT's closed form: the fused + `Range` terminal (`Lowering::Range`, `Count`/`Any` only) answers from the + two boundaries, `hi - lo`, and visits no row. A fold that needs the rows + (`SUM`, `GROUP_SUM`, `KEEP`, a gated gather) still visits every survivor of + the intersected range; there the rewrite removes two passes and two slots + but the cost stays proportional to the range. The finding is the + reduction of a three-pass computation to a property of two boundaries, + not the speed ratio. +- A monotone predicate on a sorted lane (`s < t`) scanned: ~150–190 µs. + The crossing point by `partition_point` (~20 reads) plus one fused + `Range`: ~0.1 µs. Only `prefix_facet`/`aperture_facet` under a witness + make this reduction today; no lowering does it for an `i32` compare on a + lane known to be sorted. + +## Fact change vs rule change (arm B shape, scattered, sel_K 0.01) + +Flip one `B` bit and re-run: ~0.40 ms. New program, compile, scratch, run: +~0.35 ms. Both are the run; neither change costs anything the run does not. + +## OPEN + +- An intersection rewrite (`Range ∧ Range → Range`) and a sorted-lane + monotone-compare → `Range` lowering are the two measured gains here. Both + need an order witness for any lane that is not a sealed facet lane; that + witness does not exist for `i32` lanes. +- Not measured: multi-range extents, cache misses, 16M rows. diff --git a/.claude/board/entries/README.md b/.claude/board/entries/README.md index 41d9396a3..897b79d3c 100644 --- a/.claude/board/entries/README.md +++ b/.claude/board/entries/README.md @@ -25,10 +25,11 @@ index row, (3) no duplicate entry id. Checks 1 and 2 are deliberately opposite directions; the stranding this convention prevents shows up in exactly one of them, never both. -302 entries, 2026-08-06 .. 2026-10-11. +303 entries, 2026-08-06 .. 2026-10-11. | date | entry id | finding | file | |---|---|---|---| +| 2026-10-11 | `D-TELEPORT-0` | | [2026-10-11-teleport-probe-source-restriction.md](2026-10-11-teleport-probe-source-restriction.md) | | 2026-10-11 | `synergy-ledger-classid-hotplug` | | [2026-10-11-synergy-ledger-classid-hotplug.md](2026-10-11-synergy-ledger-classid-hotplug.md) | | 2026-10-11 | `coresearch-labels-address-lifetime` | | [2026-10-11-coresearch-labels-address-lifetime.md](2026-10-11-coresearch-labels-address-lifetime.md) | | 2026-10-11 | `coresearch-instance-authz` | | [2026-10-11-coresearch-instance-authz.md](2026-10-11-coresearch-instance-authz.md) | diff --git a/.claude/plans/sql-polyfill-r2il-v1.md b/.claude/plans/sql-polyfill-r2il-v1.md index 1dde4f312..997838016 100644 --- a/.claude/plans/sql-polyfill-r2il-v1.md +++ b/.claude/plans/sql-polyfill-r2il-v1.md @@ -151,6 +151,19 @@ It follows the 2026-09-05 lance-graph ruling that planning moves to `IntAdd` stays scalar-only. NULLs in a foreign value lane stay unmodelled, as in quack. BASIS for this decision and the previous ones: quack's `sql_where` doc and `tests/sql_null_3vl.rs`. +- **DECISION (2026-10-11): procedures are inlined until `PROC_*` executes.** + W2 writes each procedure as a Rust function that emits R2IL calls through + `Dialect::call` (`lance-graph-r2il::procs`). It adds no opcode; what reaches + mask-risc is what the calls written out by hand produce. BASIS: `ogar-loco` + routes `PROC_DEF`/`PROC_CALL`/`PROC_ARG` to `Dialect::call`, not the engine; + `PROC_CALL` has no fixed stack arity and no `body_refs`, so a call site + cannot name its target body. REVISIT WHEN: the OPEN item below is settled. +- **OPEN (OGAR): an executable `PROC_DEF`/`PROC_CALL` contract.** It needs + per-call arity, a `body_refs` entry naming the target, call frames with + `PROC_ARG` and returns, a rule for who owns `VAR_GET`/`VAR_SET` across a + call, and a corrected `interpret.rs` doc (it says the engine refuses + `PROC_DEF`; the dialect receives it). Until then a "procedure body" is the + inlined call sequence. - **OPEN: SQL text parser.** DataFusion is in its grace period, so its parser is not the route. Candidates: a standalone parser crate, or a declared-query schema that needs no text parser for the compile-time mode. Decided in W6. @@ -163,7 +176,7 @@ It follows the 2026-09-05 lance-graph ruling that planning moves to |---|---|---|---|---| | W0 | D-SQLP-0 | lance-graph | This plan + board rows | — | | W1 | D-SQLP-1 | lance-graph | Move `FoldDialect` and its `Val`/`FoldError` types from `r2il-mask-abi-probe/tests/row_bridge.rs` into a workspace crate `lance-graph-r2il`; the probe's tests depend on it | every existing `row_bridge.rs` test passes unchanged against the crate; still one arity table in the tree (`ogar_r2il`'s); no new opcode; the named refusals still refuse | -| W2 | D-SQLP-2 | lance-graph | First procedures as `PROC_DEF` bodies over R2IL calls: `filter_lt` (`Load` + `IntSLess` + `KEEP`), `sum`, `group_sum`, and `margin` (two `SUM` + `IntSub`). Prerequisites in `FoldDialect`: wire `IntOr` onto `MaskOp::Or` like `IntAnd`; add the fused mode (a body's Boolean calls collected into one `BoolExpr` and lowered through mask-risc `fuse`) beside today's in-place mode | each answer equals a scalar reference on fixtures with ≥ 3 groups and both empty and full selections; no new opcode; disable runs: drop `KEEP` from `filter_lt` → red; swap `IntSub` operands in `margin` → red; an `IntOr` test that is red if it is lowered as `And`; a two-mode test where both keep the same rows, in-place emits per-call ops and fused holds ternlogs | +| W2 | D-SQLP-2 | lance-graph | First procedures as inlined R2IL call sequences (`procs`, see §4; `PROC_DEF` bodies once the OGAR contract exists): `filter_lt` (`Load` + `IntSLess` + `KEEP`), `sum`, `group_sum`, and `margin` (two `SUM` + `IntSub`). Prerequisites in `FoldDialect`: wire `IntOr` onto `MaskOp::Or` like `IntAnd`; add the fused mode (a body's Boolean calls collected into one `BoolExpr` and lowered through mask-risc `fuse`) beside today's in-place mode | each answer equals a scalar reference on fixtures with ≥ 3 groups and both empty and full selections; no new opcode; disable runs: drop `KEEP` from `filter_lt` → red; swap `IntSub` operands in `margin` → red; an `IntOr` test that is red if it is lowered as `And`; a two-mode test where both keep the same rows, in-place emits per-call ops and fused holds ternlogs | | W3 | D-SQLP-3 | lance-graph | Benchmark `margin` three ways: direct `ndarray::simd` calls, hand-built mask-risc `Program`s, the interpreted procedure. N = 65,536 and 1,048,576; selectivity ~1 % and ~50 % | answers identical before timing; release, quiet machine, median of 101 runs, avx512 and avx2 reported separately. Pre-registered: interpreted ≤ 1.05× hand-built `Program` at both N | | W4 | D-SQLP-4 | lance-graph | Build-time generator: a declared procedure body → Rust that calls mask-risc / `ndarray::simd` directly, with opcodes as const generics (the `ternlog::` pattern) | generated code's answers identical to the interpreted body; generated ≤ 1.05× direct ndarray at both N; disassembly of the hot loop contains no interpreter dispatch | | W5 | D-SQLP-5 | lance-graph, lance-graph-java | quack, `plan_eval` and the SAP binder lower through the procedures | quack and SAP: for every case their tests pin, the `mask_risc::Program` produced through the procedures is byte-identical to today's, per mode (`lower` against in-place, `lower_fused` against fused; `the_fused_lowering_trades_slots_for_passes` stays green). lance-graph-java: first add a `Program`-level test comparing the procedure lowering against the legacy `plan_lower` output (none exists today; `lowering_convergence` compares row counts only), then require byte-identical `Program`s for its 28 combine vectors. Nullable predicates: run the `sql_null_3vl.rs` cases through the procedure path in both modes, against the same Kleene oracle, byte-identical to `lower(sql_where(f))` / `lower_fused(sql_where(f))` respectively: required cases, nested `NOT`/`AND`/`OR`, gating-before-negation, NULL ≠ 0, random trees, deep chains, and the `Rows` / `Count` / `SumI32` aggregate cases. Its `MinI32` / `MaxI32` cases join once W7's `MIN` / `MAX` items land (explicit prerequisite for those cases only); add falsifiers only for shapes that suite does not cover. No other behaviour change | diff --git a/crates/lance-graph-quack/examples/teleport_probe.rs b/crates/lance-graph-quack/examples/teleport_probe.rs new file mode 100644 index 000000000..6f62cd655 --- /dev/null +++ b/crates/lance-graph-quack/examples/teleport_probe.rs @@ -0,0 +1,588 @@ +//! D-TELEPORT-0: does a fold restricted at the source ELIMINATE work, or only +//! do the same work faster? +//! +//! The question being tested: whether `R(s, o) = K(s, o) AND B(o)` can be +//! answered by addressing only the rows that can survive ("teleportation"), +//! instead of scanning, joining and filtering ("navigation"). A speedup alone +//! does not settle it: AVX-512 or ternlog fusion make a scan faster without +//! skipping any of it. So every arm reports WORK COUNTERS computed from the +//! masks themselves, next to its time. +//! +//! Nothing here is a production primitive. Every arm runs the shipped +//! `lance-graph-mask-risc` executor. The probe adds no op, no lowering and no +//! kernel; the one probe-local loop is the row-at-a-time oracle every count is +//! asserted against. The binary searches that find a range are +//! `slice::partition_point`, the same call `ordered_lane` uses. +//! +//! Edges have an `i32` source lane `s` and a `u32` object lane `o` that holds +//! the node's ROW ORDINAL (the precondition for O(1) address-to-row). `B` is a +//! RESIDENT bitmap over the node table. `K = s < t`. +//! +//! | arm | program | what it is | +//! |---|---|---| +//! | A | `Pred K`, ungated `Gather(o -> B)`, `And`, `Count` | scan, join, filter | +//! | B | `Pred K`, `Gather(o -> B) under K`, `Count` | fused: gather only K's survivors | +//! | C | `Gather(o -> B)`, `Count`, over `execute_extent(0..hi)` | source-restricted: `hi` found by binary search on sorted `s` | +//! +//! C is only defined on the CLUSTERED layout (edges sorted by `s`). On the +//! scattered layout `K` is not an interval, and C does not exist; that is the +//! control. +//! +//! The intersection section ("Schnittpunkt") tests the cheapest restriction of +//! all: two row ranges ANDed together. +//! +//! | arm | program | +//! |---|---| +//! | S1 | `Range r1`, `Range r2`, `And`, `Count`: what a two-range conjunction lowers to today | +//! | S2 | one `Range [max(lo), min(hi))`, `Count`: the intersection computed before execution | +//! | S3 | `Pred s < t` scanned on a sorted lane, `Count` | +//! | S4 | `hi = partition_point(s < t)`, one `Range [0, hi)`, `Count`: the crossing point found by binary search | +//! +//! ```text +//! CARGO_PROFILE_RELEASE_DEBUG=0 cargo run --release -p lance-graph-quack --example teleport_probe +//! ``` +//! +//! Timings are printed, never asserted. Every count is asserted. No cache-miss +//! counter is reported: `perf` is not available in the environment this probe +//! was written in, and a number this probe cannot measure is not printed. + +use std::alloc::{GlobalAlloc, Layout, System}; +use std::sync::atomic::{AtomicUsize, Ordering}; +use std::time::Instant; + +use lance_graph_mask_risc::exec::{execute_extent, execute_into, Scratch}; +use lance_graph_mask_risc::{ + words_for, ExecError, Foreign, ForeignPlane, LaneRef, Lowering, MaskOp, Operand, Out, Planes, + Pred, Program, Terminal, Value, +}; + +// ───────────────────────── allocation counter ───────────────────────── + +struct Counting; +static ALLOCS: AtomicUsize = AtomicUsize::new(0); + +// SAFETY: a pure pass-through to `System`; the counter is the only addition. +unsafe impl GlobalAlloc for Counting { + unsafe fn alloc(&self, layout: Layout) -> *mut u8 { + ALLOCS.fetch_add(1, Ordering::Relaxed); + // SAFETY: same layout, same contract as the caller's. + unsafe { System.alloc(layout) } + } + unsafe fn dealloc(&self, ptr: *mut u8, layout: Layout) { + // SAFETY: `ptr` came from `alloc` above with this `layout`. + unsafe { System.dealloc(ptr, layout) } + } +} + +#[global_allocator] +static GLOBAL: Counting = Counting; + +// ───────────────────────── data ───────────────────────── + +/// SplitMix64. +struct Rng(u64); +impl Rng { + fn next(&mut self) -> u64 { + self.0 = self.0.wrapping_add(0x9E37_79B9_7F4A_7C15); + let mut z = self.0; + z = (z ^ (z >> 30)).wrapping_mul(0xBF58_476D_1CE4_E5B9); + z = (z ^ (z >> 27)).wrapping_mul(0x94D0_49BB_1331_11EB); + z ^ (z >> 31) + } +} + +const S_SCALE: i32 = 1_000_000; +const NODES: usize = 65_536; + +struct Edges { + s: Vec, + o: Vec, +} + +/// `n` edges. `clustered` sorts by `s`, so `K = s < t` is one leading run. +/// The shuffle for the scattered layout keeps the SAME multiset of edges. +fn edges(n: usize, clustered: bool) -> Edges { + let mut r = Rng(0x7E1E_9047); + let mut pairs: Vec<(i32, u32)> = (0..n) + .map(|_| { + ( + (r.next() % S_SCALE as u64) as i32, + (r.next() % NODES as u64) as u32, + ) + }) + .collect(); + if clustered { + pairs.sort_unstable(); + } + Edges { + s: pairs.iter().map(|p| p.0).collect(), + o: pairs.iter().map(|p| p.1).collect(), + } +} + +/// `B`: a resident bitmap over the node table, `sel_b` of the nodes set. +fn node_plane(sel_b: f64) -> Vec { + let mut r = Rng(0xB0B0); + let mut w = vec![0u64; words_for(NODES)]; + for i in 0..NODES { + if (r.next() % 1_000_000) < (sel_b * 1_000_000.0) as u64 { + w[i / 64] |= 1 << (i % 64); + } + } + w +} + +fn bit(w: &[u64], i: usize) -> bool { + w[i / 64] >> (i % 64) & 1 == 1 +} + +fn thresh(sel: f64) -> i32 { + (sel * f64::from(S_SCALE)) as i32 +} + +/// The oracle: per row, no mask, no executor. +fn oracle(e: &Edges, t: i32, b: &[u64]) -> usize { + e.s.iter() + .zip(&e.o) + .filter(|(s, o)| **s < t && bit(b, **o as usize)) + .count() +} + +// ───────────────────────── work counters ───────────────────────── + +/// What an arm touched, derived from the masks and the program shape, never +/// from a timer. `words` = mask words of this table the program visits; +/// `fk_reads` = `o` values read; `gathers` = foreign-plane bits read; +/// `scratch_w` = scratch words written. +#[derive(Clone, Copy)] +struct Work { + words: usize, + lane_reads: usize, + fk_reads: usize, + scratch_w: usize, +} + +fn count(v: Result) -> usize { + match v { + Ok(Value::Count(c)) => c, + other => panic!("expected a Count, got {other:?}"), + } +} + +fn lowering_name(p: &Program) -> &'static str { + match p.compile().lowering() { + Lowering::Range(_) => "Range(fused)", + Lowering::Ternlog(_) => "Ternlog", + Lowering::TernlogKeep(_) => "TernlogKeep", + Lowering::Tern2(_) => "Tern2", + Lowering::Tern3(_) => "Tern3", + Lowering::Tiled => "Tiled", + } +} + +/// Run `p` over `extent` `reps` times; return the count, ns/exec and +/// allocations per exec. The scratch is built before the timer. +fn time( + p: &Program, + planes: &Planes<'_>, + foreign: &Foreign<'_>, + extent: std::ops::Range, + reps: usize, +) -> (usize, f64, f64) { + let mut scratch = Scratch::for_program(p, planes.n_rows).expect("scratch"); + let full = extent == (0..planes.n_rows); + let run = |s: &mut Scratch<'_>| { + if full { + count(execute_into(p, planes, foreign, s, Out::None)) + } else { + count(execute_extent( + p, + planes, + foreign, + s, + Out::None, + extent.clone(), + )) + } + }; + let first = run(&mut scratch); + let a0 = ALLOCS.load(Ordering::Relaxed); + let mut best = f64::MAX; + // Median-of-runs would need a buffer per arm; best-of-batches is enough + // for the question asked and allocates nothing inside the timer. + for _ in 0..5 { + let t = Instant::now(); + for _ in 0..reps { + assert_eq!(run(&mut scratch), first); + } + best = best.min(t.elapsed().as_nanos() as f64 / reps as f64); + } + let allocs = (ALLOCS.load(Ordering::Relaxed) - a0) as f64 / (5 * reps) as f64; + (first, best, allocs) +} + +fn row(name: &str, p: &Program, w: Work, ns: f64, allocs: f64) { + println!( + " {:<3} {:<12} words {:>7} lane-reads {:>8} fk-reads {:>8} scratch-w {:>7} allocs {:.1} {:>10.0} ns", + name, + lowering_name(p), + w.words, + w.lane_reads, + w.fk_reads, + w.scratch_w, + allocs, + ns + ); +} + +// ───────────────────────── arms ───────────────────────── + +const S: u16 = 0; // `s` lane +const O: u16 = 1; // `o` lane + +fn k(t: i32, dst: u16) -> MaskOp { + MaskOp::Pred { + pred: Pred::LtI32 { lane: S, t }, + under: None, + dst, + } +} + +fn gather(under: Option, dst: u16) -> MaskOp { + MaskOp::Gather { + lane: O, + foreign: 0, + under, + dst, + } +} + +fn count_of(slot: u16) -> Terminal { + Terminal::Count { + mask: Operand::Scratch(slot), + } +} + +fn arm_a(t: i32) -> Program { + Program::new( + vec![ + k(t, 0), + gather(None, 1), + MaskOp::And { + a: Operand::Scratch(0), + b: Operand::Scratch(1), + dst: 2, + }, + ], + count_of(2), + ) +} + +fn arm_b(t: i32) -> Program { + Program::new( + vec![k(t, 0), gather(Some(Operand::Scratch(0)), 1)], + count_of(1), + ) +} + +fn arm_c() -> Program { + Program::new(vec![gather(None, 0)], count_of(0)) +} + +// ───────────────────────── sections ───────────────────────── + +fn join_sweep(n: usize, reps: usize) { + println!("\n== K(s,o) AND B(o): COUNT, n = {n} edges, {NODES} nodes =="); + let words = words_for(n); + for clustered in [true, false] { + let e = edges(n, clustered); + let lanes = [LaneRef::I32(&e.s), LaneRef::U32(&e.o)]; + let planes = Planes { + n_rows: n, + masks: &[], + lanes: &lanes, + }; + println!( + "\n layout: {}", + if clustered { + "CLUSTERED (sorted by s)" + } else { + "SCATTERED (same edges, shuffled)" + } + ); + for sel_b in [0.5, 0.01] { + let b = node_plane(sel_b); + let fp = [ForeignPlane { + words: &b, + rows: NODES, + }]; + let foreign = Foreign { + planes: &fp, + lanes: &[], + }; + for sel_k in [0.5, 0.1, 0.01, 0.001, 0.0001] { + let t = thresh(sel_k); + let want = oracle(&e, t, &b); + let k_rows = e.s.iter().filter(|s| **s < t).count(); + println!(" sel_K {sel_k:<7} sel_B {sel_b:<5} |K| {k_rows:>8} |R| {want:>8}"); + + let pa = arm_a(t); + let (ca, ns, al) = time(&pa, &planes, &foreign, 0..n, reps); + assert_eq!(ca, want, "arm A"); + row( + "A", + &pa, + Work { + words, + lane_reads: n, + fk_reads: n, + scratch_w: 3 * words, + }, + ns, + al, + ); + + let pb = arm_b(t); + let (cb, ns, al) = time(&pb, &planes, &foreign, 0..n, reps); + assert_eq!(cb, want, "arm B"); + row( + "B", + &pb, + Work { + words, + lane_reads: n, + fk_reads: k_rows, + scratch_w: 2 * words, + }, + ns, + al, + ); + + if clustered { + // The crossing point of the sorted lane with `t`: every + // row before it is in K, none after. `partition_point` is + // O(log n) and reads ~log2(n) values of `s`. + let hi = e.s.partition_point(|s| *s < t); + assert_eq!(hi, k_rows); + // The search is part of the arm, so it is timed with it. + let tm = Instant::now(); + for _ in 0..reps { + std::hint::black_box( + std::hint::black_box(&e.s).partition_point(|s| *s < t), + ); + } + let search_ns = tm.elapsed().as_nanos() as f64 / reps as f64; + let pc = arm_c(); + let (cc, ns, al) = time(&pc, &planes, &foreign, 0..hi, reps); + let ns = ns + search_ns; + assert_eq!(cc, want, "arm C"); + let ew = words_for(hi); + row( + "C", + &pc, + Work { + words: ew, + lane_reads: (n as f64).log2().ceil() as usize, + fk_reads: hi, + scratch_w: ew, + }, + ns, + al, + ); + } + } + } + } +} + +fn range_op(lo: u32, hi: u32, dst: u16) -> MaskOp { + MaskOp::Pred { + pred: Pred::Range { lo, hi }, + under: None, + dst, + } +} + +fn schnittpunkt(n: usize, reps: usize) { + println!("\n== Schnittpunkt: the intersection of two restrictions, n = {n} =="); + let words = words_for(n); + let e = edges(n, true); + let lanes = [LaneRef::I32(&e.s), LaneRef::U32(&e.o)]; + let planes = Planes { + n_rows: n, + masks: &[], + lanes: &lanes, + }; + let nn = n as u32; + + // Two row ranges, intersection width swept from wide to narrow. + println!(" -- two row ranges ANDed --"); + for frac in [0.5, 0.1, 0.01, 0.0001] { + let width = ((f64::from(nn) * frac) as u32).max(1); + let (l1, h1) = (nn / 4, nn / 4 + width + width / 2); + let (l2, h2) = (nn / 4 + width / 2, nn / 4 + width / 2 + width); + let (lo, hi) = (l1.max(l2), h1.min(h2)); + let want = (hi - lo) as usize; + + let s1 = Program::new( + vec![ + range_op(l1, h1, 0), + range_op(l2, h2, 1), + MaskOp::And { + a: Operand::Scratch(0), + b: Operand::Scratch(1), + dst: 2, + }, + ], + count_of(2), + ); + let (c, ns, al) = time(&s1, &planes, &Foreign::NONE, 0..n, reps); + assert_eq!(c, want, "S1"); + println!(" intersection {:>8} rows", want); + row( + "S1", + &s1, + Work { + words, + lane_reads: 0, + fk_reads: 0, + scratch_w: 3 * words, + }, + ns, + al, + ); + + let s2 = Program::new(vec![range_op(lo, hi, 0)], count_of(0)); + let (c, ns, al) = time(&s2, &planes, &Foreign::NONE, 0..n, reps); + assert_eq!(c, want, "S2"); + row( + "S2", + &s2, + Work { + // The fused range count reads no lane and writes no slot; it + // counts the interval from its two endpoints. + words: 2, + lane_reads: 0, + fk_reads: 0, + scratch_w: 0, + }, + ns, + al, + ); + } + + println!(" -- a monotone predicate on a sorted lane: scan vs crossing point --"); + for sel in [0.5, 0.01, 0.0001] { + let t = thresh(sel); + let want = e.s.iter().filter(|s| **s < t).count(); + + let s3 = Program::new(vec![k(t, 0)], count_of(0)); + let (c, ns, al) = time(&s3, &planes, &Foreign::NONE, 0..n, reps); + assert_eq!(c, want, "S3"); + println!(" sel {sel:<7} |K| {want:>8}"); + row( + "S3", + &s3, + Work { + words, + lane_reads: n, + fk_reads: 0, + scratch_w: words, + }, + ns, + al, + ); + + // The binary search is part of the arm: it is timed with it. + let tm = Instant::now(); + let mut hi = 0; + for _ in 0..reps { + hi = std::hint::black_box(&e.s).partition_point(|s| *s < t); + } + let search_ns = tm.elapsed().as_nanos() as f64 / reps as f64; + let s4 = Program::new(vec![range_op(0, hi as u32, 0)], count_of(0)); + let (c, ns, al) = time(&s4, &planes, &Foreign::NONE, 0..n, reps); + assert_eq!(c, want, "S4"); + row( + "S4", + &s4, + Work { + words: 2, + lane_reads: (n as f64).log2().ceil() as usize, + fk_reads: 0, + scratch_w: 0, + }, + ns + search_ns, + al, + ); + } +} + +/// A fact change flips one bit of `B` and re-runs the SAME program; a rule +/// change builds and compiles a NEW program. Both are timed per change. +fn change_cost(n: usize, reps: usize) { + println!("\n== fact change vs rule change, n = {n}, arm B shape =="); + let e = edges(n, false); + let lanes = [LaneRef::I32(&e.s), LaneRef::U32(&e.o)]; + let planes = Planes { + n_rows: n, + masks: &[], + lanes: &lanes, + }; + let mut b = node_plane(0.5); + let t = thresh(0.01); + let p = arm_b(t); + let mut scratch = Scratch::for_program(&p, n).expect("scratch"); + + let tm = Instant::now(); + let mut got = 0; + for i in 0..reps { + let w = (i * 7) % b.len(); + b[w] ^= 1; + let fp = [ForeignPlane { + words: &b, + rows: NODES, + }]; + let foreign = Foreign { + planes: &fp, + lanes: &[], + }; + got = count(execute_into(&p, &planes, &foreign, &mut scratch, Out::None)); + } + let fact_ns = tm.elapsed().as_nanos() as f64 / reps as f64; + // The oracle runs once, outside the timer, on the final state of `B`. + assert_eq!(got, oracle(&e, t, &b), "fact change"); + + let fp = [ForeignPlane { + words: &b, + rows: NODES, + }]; + let foreign = Foreign { + planes: &fp, + lanes: &[], + }; + let tm = Instant::now(); + for i in 0..reps { + let q = arm_b(t + i as i32); + let _ = std::hint::black_box(q.compile()); + let mut s = Scratch::for_program(&q, n).expect("scratch"); + let _ = count(execute_into(&q, &planes, &foreign, &mut s, Out::None)); + } + let rule_ns = tm.elapsed().as_nanos() as f64 / reps as f64; + println!(" fact change (flip a B bit, re-run): {fact_ns:>10.0} ns"); + println!(" rule change (new program, compile, scratch, run): {rule_ns:>10.0} ns"); +} + +fn main() { + let n: usize = std::env::var("TELEPORT_N") + .ok() + .and_then(|v| v.parse().ok()) + .unwrap_or(1 << 20); + let reps: usize = std::env::var("TELEPORT_REPS") + .ok() + .and_then(|v| v.parse().ok()) + .unwrap_or(20); + join_sweep(n, reps); + schnittpunkt(n, reps); + change_cost(n, reps); +} diff --git a/crates/lance-graph-r2il/src/lib.rs b/crates/lance-graph-r2il/src/lib.rs index 5c27ab642..2081e881f 100644 --- a/crates/lance-graph-r2il/src/lib.rs +++ b/crates/lance-graph-r2il/src/lib.rs @@ -40,12 +40,32 @@ //! //! R2IL band: `Load` (spaces 0 = U32, 1 = I32, 2 = U64, 3 = resident mask //! plane), `IntSub` (scalar/scalar), `IntAnd` (with survivor gating), -//! `IntNot`, `IntEqual`, `IntSLess` and `PopCount` are wired. `IntLess` and +//! `IntOr`, `IntNot`, `IntEqual`, `IntSLess` and `PopCount` are wired. `IntLess` and //! `IntLessEqual` are refused as [`FoldError::NoUnsignedLaneCompare`] //! (mask-risc has no unsigned lane compare); `IntSub` of two addresses as //! [`FoldError::LaneVersusLane`]; an unknown `Load` space as -//! [`FoldError::UnknownLoadSpace`]; every other byte, `IntAdd` and `IntOr` -//! included, as [`FoldError::Unimplemented`]. +//! [`FoldError::UnknownLoadSpace`]; every other byte, `IntAdd` included, as +//! [`FoldError::Unimplemented`]. +//! +//! # Boolean modes +//! +//! [`BoolMode`] decides how `IntAnd`, `IntOr` and `IntNot` on masks lower, +//! mirroring quack's `lower` / `lower_fused`: +//! +//! - [`BoolMode::InPlace`] (default): one mask-risc op per call. `IntAnd` +//! gates a just-emitted predicate under its left operand (survivor gating) +//! instead of spending an `And`. +//! - [`BoolMode::Fused`]: the calls build one [`BoolExpr`]; the first consumer +//! (`PopCount`, `SUM`, `GROUP_SUM`, `KEEP`) lowers it through mask-risc +//! [`fuse()`] into ternlogs. Fewer passes, no survivor gating. +//! +//! Both modes keep the same rows. Predicates are emitted the same way in +//! both; only the Boolean combination differs. +//! +//! # Procedures +//! +//! [`procs`] holds named call sequences (`filter_lt`, `sum`, `group_sum`, +//! `margin`). They are Rust functions over [`Dialect::call`], not opcodes. //! //! # Slot limit //! @@ -72,11 +92,13 @@ //! are pinned against `ogar_r2il::R2ILFn::MNEMONICS` by this crate's tests, //! so the arity table stays in `ogar_r2il` alone. +pub mod procs; + use std::cell::Cell; use lance_graph_mask_risc::{ - execute_into, scratch_words_for, tile_words_for, Foreign, LaneKind, MaskOp, Operand, Out, - Planes, Pred, Program, Scratch, Terminal, Value, + execute_into, fuse, scratch_words_for, tile_words_for, BoolExpr, Foreign, FuseError, LaneKind, + MaskOp, Operand, Out, Planes, Pred, Program, Scratch, Terminal, Value, }; use ogar_loco::{Dialect, FnIndex}; use ogar_r2il::{GROUP_SUM, KEEP, R2IL_BASE, RANGE, SUM, VIA}; @@ -93,6 +115,7 @@ use ogar_r2il::{GROUP_SUM, KEEP, R2IL_BASE, RANGE, SUM, VIA}; pub const LOAD: FnIndex = FnIndex(R2IL_BASE + 1); pub const INT_SUB: FnIndex = FnIndex(R2IL_BASE + 10); pub const INT_AND: FnIndex = FnIndex(R2IL_BASE + 20); +pub const INT_OR: FnIndex = FnIndex(R2IL_BASE + 21); /// `IntNot` = ordinal 23, arity 1 (confirmed against `R2ILFn::MNEMONICS`). /// Over a population it lowers to `MaskOp::Not`. pub const INT_NOT: FnIndex = FnIndex(R2IL_BASE + 23); @@ -191,8 +214,30 @@ pub enum FoldError { BranchOnPopulation, /// A finalized fold's `execute_into` call itself refused the program. Exec(lance_graph_mask_risc::ExecError), + /// The fused mode's Boolean expression could not be lowered by mask-risc + /// `fuse` (it needed a scratch slot past `u16::MAX`). + Fuse(FuseError), +} + +/// How a body's Boolean calls (`IntAnd`, `IntOr`, `IntNot` on masks) are +/// lowered. See the crate doc's § Boolean modes. +#[derive(Debug, Clone, Copy, PartialEq, Eq, Default)] +pub enum BoolMode { + /// One mask-risc op per call, written in place. `IntAnd` gates a + /// just-emitted predicate under its left operand instead of spending an + /// `And` (survivor gating). The default. + #[default] + InPlace, + /// The calls build one [`BoolExpr`]; at the fold it is lowered through + /// mask-risc [`fuse()`] into ternlogs. Fewer passes, no survivor gating. + Fused, } +/// Slot numbers at or above this name a pending fused expression, not a +/// scratch slot. Real slots never reach it: a fold is limited to +/// [`MAX_SLOTS`]. +const VIRTUAL: u16 = 0x8000; + /// Which caller buffer a finalized fold writes into. #[derive(Debug, Clone, Copy, PartialEq, Eq)] enum OutKind { @@ -253,6 +298,14 @@ pub struct FoldDialect<'p> { /// Set by [`Dialect::truthy`] when asked to branch on a non-`Scalar` /// value. `Cell`, not a plain field, because `truthy` takes `&self`. poison: Cell>, + /// [`BoolMode`]; fixed at construction. + mode: BoolMode, + /// Fused mode only: the pending Boolean expressions of the current + /// epoch, named by `VIRTUAL + index`. Cleared at every fold. + exprs: Vec, + /// Fused mode only: where each pending expression was lowered to, once a + /// consumer needed it, so a second consumer reuses the same ops. + lowered: Vec>, } impl<'p> FoldDialect<'p> { @@ -278,6 +331,9 @@ impl<'p> FoldDialect<'p> { facade_passes: 0, programs_run: 0, poison: Cell::new(None), + mode: BoolMode::InPlace, + exprs: Vec::new(), + lowered: Vec::new(), } } @@ -335,6 +391,85 @@ impl<'p> FoldDialect<'p> { self.scratch.len() } + /// This dialect with its Boolean calls lowered in `mode`. + #[must_use] + pub fn with_mode(mut self, mode: BoolMode) -> Self { + self.mode = mode; + self + } + + /// The Boolean mode this dialect was built with. + pub fn mode(&self) -> BoolMode { + self.mode + } + + /// The expression a mask operand stands for, for the fused mode. + fn expr_of(&self, v: Val) -> Result { + match v { + Val::Address(Addr::Plane(p)) => Ok(BoolExpr::Leaf(Operand::Plane(p))), + Val::Slot { slot, epoch } => { + if epoch != self.epoch { + return Err(FoldError::StaleSlot); + } + if slot >= VIRTUAL { + self.exprs + .get(usize::from(slot - VIRTUAL)) + .cloned() + .ok_or(FoldError::StaleSlot) + } else { + Ok(BoolExpr::Leaf(Operand::Scratch(slot))) + } + } + _ => Err(FoldError::WrongOperandKind(INT_AND)), + } + } + + /// Push a pending fused expression as a virtual slot. + fn push_expr(&mut self, e: BoolExpr, stack: &mut Vec) { + let slot = VIRTUAL + u16::try_from(self.exprs.len()).unwrap_or(u16::MAX - VIRTUAL); + self.exprs.push(e); + self.lowered.push(None); + stack.push(Val::Slot { + slot, + epoch: self.epoch, + }); + } + + /// The operand a consumer reads for `slot`. A real slot is itself; a + /// virtual one is lowered through mask-risc `fuse` the first time it is + /// consumed, its ternlogs appended to this fold's ops. + fn operand_of(&mut self, slot: u16) -> Result { + if slot < VIRTUAL { + return Ok(Operand::Scratch(slot)); + } + let i = usize::from(slot - VIRTUAL); + if let Some(Some(op)) = self.lowered.get(i) { + return Ok(*op); + } + let expr = self.exprs.get(i).ok_or(FoldError::StaleSlot)?; + let f = fuse(expr, self.next_slot).map_err(FoldError::Fuse)?; + self.next_slot = f + .next_slot + .ok_or(FoldError::Fuse(FuseError::SlotOverflow))?; + self.ops.extend(f.ops); + self.lowered[i] = Some(f.result); + Ok(f.result) + } + + /// Fused-mode `IntAnd`/`IntOr`: combine two mask operands into one + /// pending expression. + fn fuse_binary( + &mut self, + a: Val, + b: Val, + stack: &mut Vec, + build: fn(Box, Box) -> BoolExpr, + ) -> Result<(), FoldError> { + let e = build(Box::new(self.expr_of(a)?), Box::new(self.expr_of(b)?)); + self.push_expr(e, stack); + Ok(()) + } + /// The next unused scratch slot within the current `ops` accumulation, /// advancing [`Self::next_slot`] by one. fn fresh(&mut self) -> u16 { @@ -398,6 +533,8 @@ impl<'p> FoldDialect<'p> { let ops = std::mem::take(&mut self.ops); self.next_slot = 0; self.epoch += 1; + self.exprs.clear(); + self.lowered.clear(); self.facade_passes += ops.len(); self.programs_run += 1; let program = Program::new(ops, terminal); @@ -429,6 +566,11 @@ impl<'p> FoldDialect<'p> { } } +/// A value that names a mask: a scratch slot or a resident plane. +fn is_mask(v: Val) -> bool { + matches!(v, Val::Slot { .. } | Val::Address(Addr::Plane(_))) +} + impl Dialect for FoldDialect<'_> { type Value = Val; type Error = FoldError; @@ -621,6 +763,9 @@ impl Dialect for FoldDialect<'_> { INT_AND => { let b = stack.pop().ok_or(FoldError::Underflow(f))?; let a = stack.pop().ok_or(FoldError::Underflow(f))?; + if self.mode == BoolMode::Fused && is_mask(a) && is_mask(b) { + return self.fuse_binary(a, b, stack, BoolExpr::And); + } match (a, b) { ( Val::Slot { @@ -658,6 +803,20 @@ impl Dialect for FoldDialect<'_> { }); Ok(()) } + // Two resident planes: nothing to gate, one `And` pass. + (Val::Address(Addr::Plane(pa)), Val::Address(Addr::Plane(pb))) => { + let dst = self.fresh(); + self.ops.push(MaskOp::And { + a: Operand::Plane(pa), + b: Operand::Plane(pb), + dst, + }); + stack.push(Val::Slot { + slot: dst, + epoch: self.epoch, + }); + Ok(()) + } (Val::Scalar(x), Val::Scalar(y)) => { stack.push(Val::Scalar(x & y)); Ok(()) @@ -665,6 +824,36 @@ impl Dialect for FoldDialect<'_> { _ => Err(FoldError::WrongOperandKind(f)), } } + // Like `IntAnd` without the survivor gate: an `Or` cannot gate its + // right operand under its left, so every in-place `IntOr` on masks + // is one `Or` pass. + INT_OR => { + let b = stack.pop().ok_or(FoldError::Underflow(f))?; + let a = stack.pop().ok_or(FoldError::Underflow(f))?; + if let (Val::Scalar(x), Val::Scalar(y)) = (a, b) { + stack.push(Val::Scalar(x | y)); + return Ok(()); + } + if !(is_mask(a) && is_mask(b)) { + return Err(FoldError::WrongOperandKind(f)); + } + if self.mode == BoolMode::Fused { + return self.fuse_binary(a, b, stack, BoolExpr::Or); + } + let operand = |v: Val, epoch_now: u32| match v { + Val::Address(Addr::Plane(p)) => Ok(Operand::Plane(p)), + Val::Slot { slot, epoch } if epoch == epoch_now => Ok(Operand::Scratch(slot)), + _ => Err(FoldError::StaleSlot), + }; + let (a, b) = (operand(a, self.epoch)?, operand(b, self.epoch)?); + let dst = self.fresh(); + self.ops.push(MaskOp::Or { a, b, dst }); + stack.push(Val::Slot { + slot: dst, + epoch: self.epoch, + }); + Ok(()) + } INT_SUB => { let b = stack.pop().ok_or(FoldError::Underflow(f))?; let a = stack.pop().ok_or(FoldError::Underflow(f))?; @@ -685,9 +874,8 @@ impl Dialect for FoldDialect<'_> { if epoch != self.epoch { return Err(FoldError::StaleSlot); } - let n = self.finalize_and_run_scalar(Terminal::Count { - mask: Operand::Scratch(s), - })?; + let mask = self.operand_of(s)?; + let n = self.finalize_and_run_scalar(Terminal::Count { mask })?; stack.push(Val::Scalar(n)); Ok(()) } @@ -705,7 +893,7 @@ impl Dialect for FoldDialect<'_> { if epoch != self.epoch { return Err(FoldError::StaleSlot); } - (Operand::Scratch(s), idx) + (self.operand_of(s)?, idx) } ( Val::Address(Addr::Lane { @@ -717,7 +905,7 @@ impl Dialect for FoldDialect<'_> { if epoch != self.epoch { return Err(FoldError::StaleSlot); } - (Operand::Scratch(s), idx) + (self.operand_of(s)?, idx) } _ => return Err(FoldError::WrongOperandKind(f)), }; @@ -743,22 +931,26 @@ impl Dialect for FoldDialect<'_> { else { return Err(FoldError::WrongOperandKind(f)); }; - let mask = Operand::Scratch(s); - let terminal = match key { + // The key shape is checked first, so a wrong-kind body is + // reported as wrong-kind. + let via = match key { Val::Address(Addr::Lane { - idx: key, + idx, kind: LaneKind::U32, - }) => Terminal::GroupSumI32 { mask, key, val }, - Val::Address(Addr::Via { fk, key }) => { - Terminal::GroupSumViaI32 { mask, fk, key, val } - } + }) => (idx, None), + Val::Address(Addr::Via { fk, key }) => (key, Some(fk)), _ => return Err(FoldError::WrongOperandKind(f)), }; - // Epoch checked AFTER the shape checks so a wrong-kind body is - // reported as wrong-kind, and BEFORE the fold, like SUM's arm. + // Then the epoch, BEFORE a fused expression is lowered and + // before the fold, like SUM's arm. if epoch != self.epoch { return Err(FoldError::StaleSlot); } + let mask = self.operand_of(s)?; + let terminal = match via { + (key, None) => Terminal::GroupSumI32 { mask, key, val }, + (key, Some(fk)) => Terminal::GroupSumViaI32 { mask, fk, key, val }, + }; match self.finalize_and_run(terminal, OutKind::Group)? { Value::GroupSummed => Ok(()), other => { @@ -768,6 +960,11 @@ impl Dialect for FoldDialect<'_> { } INT_NOT => { let a = stack.pop().ok_or(FoldError::Underflow(f))?; + if self.mode == BoolMode::Fused && is_mask(a) { + let e = BoolExpr::Not(Box::new(self.expr_of(a)?)); + self.push_expr(e, stack); + return Ok(()); + } let operand = match a { Val::Scalar(x) => { stack.push(Val::Scalar(!x)); @@ -814,12 +1011,8 @@ impl Dialect for FoldDialect<'_> { if epoch != self.epoch { return Err(FoldError::StaleSlot); } - match self.finalize_and_run( - Terminal::Keep { - mask: Operand::Scratch(slot), - }, - OutKind::Keep, - )? { + let mask = self.operand_of(slot)?; + match self.finalize_and_run(Terminal::Keep { mask }, OutKind::Keep)? { Value::Mask(_) => Ok(()), other => panic!("KEEP terminal shape guarantees Mask, got {other:?}"), } diff --git a/crates/lance-graph-r2il/src/procs.rs b/crates/lance-graph-r2il/src/procs.rs new file mode 100644 index 000000000..c9d1963ab --- /dev/null +++ b/crates/lance-graph-r2il/src/procs.rs @@ -0,0 +1,119 @@ +//! Procedures: named R2IL call sequences over [`FoldDialect`]. +//! +//! A procedure is a Rust function that emits R2IL calls through +//! [`Dialect::call`] and pushes literals onto the loco stack. It adds no +//! opcode and no lowering: what reaches mask-risc is exactly what the same +//! calls written out by hand would produce. This is the inlined form of +//! D-SQLP-2. An executable `PROC_DEF`/`PROC_CALL` contract (per-call arity, +//! call frames, `PROC_ARG`, returns) is an OPEN item in OGAR; these functions +//! are what such a body would contain. +//! +//! Stack conventions: a "mask" is a [`Val::Slot`] or a resident plane +//! address on top of the stack. Lane arguments are indexes into +//! [`Planes::lanes`](lance_graph_mask_risc::Planes::lanes). + +use ogar_loco::{Dialect, FnIndex}; +use ogar_r2il::{GROUP_SUM, KEEP, SUM}; + +use crate::{FoldDialect, FoldError, Val, INT_SUB, INT_S_LESS, LOAD}; + +/// A selection that a procedure re-emits once per fold (see [`margin`]). +pub type Select<'s> = dyn Fn(&mut FoldDialect<'_>, &mut Vec) -> Result<(), FoldError> + 's; + +/// `LOAD` space for a `U32` lane. +const SPACE_U32: u8 = 0; +/// `LOAD` space for an `I32` lane. +const SPACE_I32: u8 = 1; + +fn call( + d: &mut FoldDialect<'_>, + f: FnIndex, + v: [u8; 3], + stack: &mut Vec, +) -> Result<(), FoldError> { + d.call(f, v, stack) +} + +/// Push the address of lane `idx` in `space`. +fn load( + d: &mut FoldDialect<'_>, + stack: &mut Vec, + space: u8, + idx: u16, +) -> Result<(), FoldError> { + stack.push(Val::Scalar(i64::from(idx))); + call(d, LOAD, [space, 0, 0], stack) +} + +/// `lane < t` on an `I32` lane. Pushes the mask. +/// +/// Calls: `LOAD`, `IntSLess`. +pub fn select_lt( + d: &mut FoldDialect<'_>, + stack: &mut Vec, + lane: u16, + t: i32, +) -> Result<(), FoldError> { + load(d, stack, SPACE_I32, lane)?; + stack.push(Val::Scalar(i64::from(t))); + call(d, INT_S_LESS, [0; 3], stack) +} + +/// The rows where `lane < t`, written into the dialect's `KEEP` sink. +/// Pushes nothing. +/// +/// Calls: `LOAD`, `IntSLess`, `KEEP`. +pub fn filter_lt( + d: &mut FoldDialect<'_>, + stack: &mut Vec, + lane: u16, + t: i32, +) -> Result<(), FoldError> { + select_lt(d, stack, lane, t)?; + call(d, KEEP, [0; 3], stack) +} + +/// `SUM(val_lane | mask)` for the mask on top of the stack. Pushes the sum. +/// +/// Calls: `LOAD`, `SUM`. +pub fn sum(d: &mut FoldDialect<'_>, stack: &mut Vec, val_lane: u16) -> Result<(), FoldError> { + load(d, stack, SPACE_I32, val_lane)?; + call(d, SUM, [0; 3], stack) +} + +/// Per-group `SUM(val_lane | mask)` keyed by the `U32` lane `key_lane`, for +/// the mask on top of the stack, into the dialect's group sink. Pushes +/// nothing. +/// +/// Calls: `LOAD`, `LOAD`, `GROUP_SUM`. +pub fn group_sum( + d: &mut FoldDialect<'_>, + stack: &mut Vec, + key_lane: u16, + val_lane: u16, +) -> Result<(), FoldError> { + load(d, stack, SPACE_U32, key_lane)?; + load(d, stack, SPACE_I32, val_lane)?; + call(d, GROUP_SUM, [0; 3], stack) +} + +/// `SUM(revenue | sel) - SUM(cost | sel)`. Pushes the difference. +/// +/// Each `SUM` is its own fold, and a fold ends the epoch of every slot built +/// before it, so `select` is emitted once per `SUM` rather than held across +/// the first one. +/// +/// Calls: `select`, `LOAD`, `SUM`, `select`, `LOAD`, `SUM`, `IntSub`. +pub fn margin( + d: &mut FoldDialect<'_>, + stack: &mut Vec, + select: &Select<'_>, + revenue: u16, + cost: u16, +) -> Result<(), FoldError> { + select(d, stack)?; + sum(d, stack, revenue)?; + select(d, stack)?; + sum(d, stack, cost)?; + call(d, INT_SUB, [0; 3], stack) +} diff --git a/crates/lance-graph-r2il/tests/dialect.rs b/crates/lance-graph-r2il/tests/dialect.rs index 1acfec66c..9b05084e4 100644 --- a/crates/lance-graph-r2il/tests/dialect.rs +++ b/crates/lance-graph-r2il/tests/dialect.rs @@ -9,7 +9,7 @@ use lance_graph_mask_risc::{words_for, Foreign, LaneRef, Planes}; use lance_graph_r2il::{ Addr, FoldDialect, FoldError, Val, INT_AND, INT_EQUAL, INT_LESS, INT_LESS_EQUAL, INT_NOT, - INT_SUB, INT_S_LESS, LOAD, MAX_SLOTS, POP_COUNT, + INT_OR, INT_SUB, INT_S_LESS, LOAD, MAX_SLOTS, POP_COUNT, }; use ogar_loco::{Dialect, FnIndex}; use ogar_r2il::{ @@ -31,6 +31,7 @@ fn local_r2il_ordinals_name_the_ogar_r2il_opcodes() { (LOAD, "Load"), (INT_SUB, "IntSub"), (INT_AND, "IntAnd"), + (INT_OR, "IntOr"), (INT_NOT, "IntNot"), (INT_EQUAL, "IntEqual"), (INT_LESS, "IntLess"), @@ -79,14 +80,11 @@ fn every_named_refusal_is_still_produced() { }; let foreign = Foreign::NONE; - // Fold-band bytes that are not wired, and two R2IL-band ones. + // Fold-band bytes that are not wired, and one R2IL-band one. `IntOr` is + // wired since D-SQLP-2. let int_add = FnIndex(R2IL_BASE + 9); - let int_or = FnIndex(R2IL_BASE + 21); assert_eq!(mnemonic(int_add), "IntAdd"); - assert_eq!(mnemonic(int_or), "IntOr"); - for f in [ - MIN, MAX, KEY_RUNS, ANY, ALL, SCATTER_OR, BLEND, int_add, int_or, - ] { + for f in [MIN, MAX, KEY_RUNS, ANY, ALL, SCATTER_OR, BLEND, int_add] { let mut d = FoldDialect::new(&planes, &foreign); let mut stack = vec![Val::Scalar(1), Val::Scalar(2)]; assert_eq!( diff --git a/crates/lance-graph-r2il/tests/procs.rs b/crates/lance-graph-r2il/tests/procs.rs new file mode 100644 index 000000000..dc39944eb --- /dev/null +++ b/crates/lance-graph-r2il/tests/procs.rs @@ -0,0 +1,341 @@ +//! D-SQLP-2: procedures, `IntOr`, and the two Boolean modes. +//! +//! Every answer is compared against a scalar reference computed row by row +//! from the same lanes. Fixtures have four groups and are run with an empty +//! selection, a partial one and a full one. + +use lance_graph_mask_risc::{words_for, Foreign, LaneRef, MaskOp, Planes}; +use lance_graph_r2il::procs::{filter_lt, group_sum, margin, select_lt, sum}; +use lance_graph_r2il::{ + BoolMode, FoldDialect, FoldError, Val, INT_AND, INT_NOT, INT_OR, POP_COUNT, +}; +use ogar_loco::Dialect; +use ogar_r2il::KEEP; + +const N: usize = 1_000; +const GROUPS: usize = 4; + +// Lane indexes in every fixture below. +const X: u16 = 0; // i32, i % 50 +const Y: u16 = 1; // i32, (i * 7) % 31 +const REVENUE: u16 = 2; // i32 +const COST: u16 = 3; // i32 +const KEY: u16 = 4; // u32, i % GROUPS + +struct Fixture { + x: Vec, + y: Vec, + revenue: Vec, + cost: Vec, + key: Vec, +} + +fn fixture() -> Fixture { + Fixture { + x: (0..N).map(|i| (i % 50) as i32).collect(), + y: (0..N).map(|i| ((i * 7) % 31) as i32).collect(), + revenue: (0..N).map(|i| (i % 13) as i32 + 10).collect(), + cost: (0..N).map(|i| (i % 5) as i32).collect(), + key: (0..N).map(|i| (i % GROUPS) as u32).collect(), + } +} + +fn lanes(f: &Fixture) -> [LaneRef<'_>; 5] { + [ + LaneRef::I32(&f.x), + LaneRef::I32(&f.y), + LaneRef::I32(&f.revenue), + LaneRef::I32(&f.cost), + LaneRef::U32(&f.key), + ] +} + +/// `x < t` thresholds: empty, partial, full. +const THRESHOLDS: [i32; 3] = [0, 17, 50]; + +fn rows(m: &[u64]) -> Vec { + (0..N).filter(|&r| m[r / 64] >> (r % 64) & 1 == 1).collect() +} + +/// FAILS IF: `filter_lt` stops writing exactly the rows `x < t` (dropping its +/// `KEEP` leaves the sink empty and fails the partial and full cases). +#[test] +fn filter_lt_keeps_the_rows_the_reference_keeps() { + let f = fixture(); + let lanes = lanes(&f); + let planes = Planes { + n_rows: N, + masks: &[], + lanes: &lanes, + }; + for t in THRESHOLDS { + let mut sink = vec![0u64; words_for(N)]; + { + let mut d = FoldDialect::with_keep(&planes, &Foreign::NONE, &mut sink); + let mut stack = vec![]; + filter_lt(&mut d, &mut stack, X, t).expect("filter_lt"); + assert!(stack.is_empty()); + } + let want: Vec = (0..N).filter(|&r| f.x[r] < t).collect(); + assert_eq!(rows(&sink), want, "t = {t}"); + } +} + +/// FAILS IF: `sum` stops matching `SUM(revenue | x < t)`. +#[test] +fn sum_matches_the_reference() { + let f = fixture(); + let lanes = lanes(&f); + let planes = Planes { + n_rows: N, + masks: &[], + lanes: &lanes, + }; + for t in THRESHOLDS { + let want: i64 = (0..N) + .filter(|&r| f.x[r] < t) + .map(|r| i64::from(f.revenue[r])) + .sum(); + let mut d = FoldDialect::new(&planes, &Foreign::NONE); + let mut stack = vec![]; + select_lt(&mut d, &mut stack, X, t).expect("select"); + sum(&mut d, &mut stack, REVENUE).expect("sum"); + assert_eq!(stack, vec![Val::Scalar(want)], "t = {t}"); + } +} + +/// FAILS IF: `group_sum` stops matching a per-group reference over four +/// groups. +#[test] +fn group_sum_matches_the_reference() { + let f = fixture(); + let lanes = lanes(&f); + let planes = Planes { + n_rows: N, + masks: &[], + lanes: &lanes, + }; + for t in THRESHOLDS { + let mut want = [0i64; GROUPS]; + for r in (0..N).filter(|&r| f.x[r] < t) { + want[f.key[r] as usize] += i64::from(f.revenue[r]); + } + let mut sink = [0i64; GROUPS]; + { + let mut d = FoldDialect::with_groups(&planes, &Foreign::NONE, &mut sink); + let mut stack = vec![]; + select_lt(&mut d, &mut stack, X, t).expect("select"); + group_sum(&mut d, &mut stack, KEY, REVENUE).expect("group_sum"); + assert!(stack.is_empty()); + } + assert_eq!(sink, want, "t = {t}"); + } +} + +/// FAILS IF: `margin` stops being `SUM(revenue) - SUM(cost)` over the +/// selection. The fixture keeps revenue and cost different everywhere, so +/// swapping the `IntSub` operands flips the sign and fails. +#[test] +fn margin_is_revenue_minus_cost() { + let f = fixture(); + let lanes = lanes(&f); + let planes = Planes { + n_rows: N, + masks: &[], + lanes: &lanes, + }; + for t in THRESHOLDS { + let want: i64 = (0..N) + .filter(|&r| f.x[r] < t) + .map(|r| i64::from(f.revenue[r] - f.cost[r])) + .sum(); + let mut d = FoldDialect::new(&planes, &Foreign::NONE); + let mut stack = vec![]; + margin( + &mut d, + &mut stack, + &|d, s| select_lt(d, s, X, t), + REVENUE, + COST, + ) + .expect("margin"); + assert_eq!(stack, vec![Val::Scalar(want)], "t = {t}"); + assert_eq!(d.programs_run(), 2, "one fold per SUM"); + } + // Non-vacuity: the partial case has a positive margin, so a sign flip + // cannot pass it. + assert!((0..N).any(|r| f.x[r] < 17 && f.revenue[r] != f.cost[r])); +} + +/// `(x < a) OP (y < b)` counted, in `mode`. Returns the count and the ops of +/// the program that ran. +fn count_two( + planes: &Planes<'_>, + mode: BoolMode, + op: ogar_loco::FnIndex, + a: i32, + b: i32, +) -> (i64, Vec) { + let mut d = FoldDialect::new(planes, &Foreign::NONE).with_mode(mode); + let mut stack = vec![]; + select_lt(&mut d, &mut stack, X, a).expect("x < a"); + select_lt(&mut d, &mut stack, Y, b).expect("y < b"); + d.call(op, [0; 3], &mut stack).expect("combine"); + d.call(POP_COUNT, [0; 3], &mut stack).expect("count"); + let Some(Val::Scalar(n)) = stack.pop() else { + panic!("count pushes a scalar"); + }; + (n, d.last_program().expect("ran").ops.clone()) +} + +/// FAILS IF: `IntOr` is lowered as anything but a union. The thresholds are +/// chosen so the union and the intersection differ. +#[test] +fn int_or_is_a_union_in_both_modes() { + let f = fixture(); + let lanes = lanes(&f); + let planes = Planes { + n_rows: N, + masks: &[], + lanes: &lanes, + }; + let union = (0..N).filter(|&r| f.x[r] < 10 || f.y[r] < 5).count() as i64; + let inter = (0..N).filter(|&r| f.x[r] < 10 && f.y[r] < 5).count() as i64; + assert_ne!(union, inter, "the fixture must separate OR from AND"); + for mode in [BoolMode::InPlace, BoolMode::Fused] { + let (n, _) = count_two(&planes, mode, INT_OR, 10, 5); + assert_eq!(n, union, "{mode:?}"); + } +} + +/// FAILS IF: the two modes stop keeping the same rows, the in-place mode +/// stops emitting one op per Boolean call, or the fused mode stops lowering +/// the combination to ternlogs. +#[test] +fn both_modes_keep_the_same_rows_with_different_ops() { + let f = fixture(); + let lanes = lanes(&f); + let planes = Planes { + n_rows: N, + masks: &[], + lanes: &lanes, + }; + // `(x < 20 OR y < 7) AND NOT (x < 3)`, kept. + let body = |d: &mut FoldDialect<'_>, s: &mut Vec| -> Result<(), FoldError> { + select_lt(d, s, X, 20)?; + select_lt(d, s, Y, 7)?; + d.call(INT_OR, [0; 3], s)?; + select_lt(d, s, X, 3)?; + d.call(INT_NOT, [0; 3], s)?; + d.call(INT_AND, [0; 3], s)?; + d.call(KEEP, [0; 3], s) + }; + let want: Vec = (0..N) + .filter(|&r| (f.x[r] < 20 || f.y[r] < 7) && f.x[r] >= 3) + .collect(); + assert!(!want.is_empty() && want.len() < N); + + let mut kept = Vec::new(); + for mode in [BoolMode::InPlace, BoolMode::Fused] { + let mut sink = vec![0u64; words_for(N)]; + let ops = { + let mut d = FoldDialect::with_keep(&planes, &Foreign::NONE, &mut sink).with_mode(mode); + let mut stack = vec![]; + body(&mut d, &mut stack).expect("body"); + d.last_program().expect("ran").ops.clone() + }; + assert_eq!(rows(&sink), want, "{mode:?}"); + let ternlogs = ops + .iter() + .filter(|o| matches!(o, MaskOp::Ternlog { .. })) + .count(); + let per_call = ops + .iter() + .filter(|o| { + matches!( + o, + MaskOp::Or { .. } | MaskOp::Not { .. } | MaskOp::And { .. } + ) + }) + .count(); + match mode { + BoolMode::InPlace => { + assert_eq!(ternlogs, 0, "in-place emits no ternlog: {ops:?}"); + assert!(per_call >= 2, "in-place emits per-call ops: {ops:?}"); + } + BoolMode::Fused => { + assert!(ternlogs >= 1, "fused holds ternlogs: {ops:?}"); + assert_eq!(per_call, 0, "fused emits no per-call op: {ops:?}"); + } + } + kept.push(rows(&sink)); + } + assert_eq!(kept[0], kept[1]); +} + +/// FAILS IF: a fused expression consumed by a stale slot is folded instead +/// of refused. +#[test] +fn a_fused_slot_from_an_earlier_fold_is_refused() { + let f = fixture(); + let lanes = lanes(&f); + let planes = Planes { + n_rows: N, + masks: &[], + lanes: &lanes, + }; + let mut d = FoldDialect::new(&planes, &Foreign::NONE).with_mode(BoolMode::Fused); + let mut stack = vec![]; + select_lt(&mut d, &mut stack, X, 20).unwrap(); + select_lt(&mut d, &mut stack, Y, 7).unwrap(); + d.call(INT_OR, [0; 3], &mut stack).unwrap(); + let stale = stack[0]; + d.call(POP_COUNT, [0; 3], &mut stack).unwrap(); + let mut stack = vec![stale]; + assert_eq!( + d.call(POP_COUNT, [0; 3], &mut stack), + Err(FoldError::StaleSlot) + ); +} + +/// FAILS IF: an `IntAnd` of two resident planes stops being accepted in +/// either mode, or the two modes stop agreeing on it. +#[test] +fn plane_and_plane_is_accepted_in_both_modes() { + let f = fixture(); + let lanes = lanes(&f); + let pa: Vec = (0..words_for(N)) + .map(|w| 0xF0F0_F0F0_F0F0_F0F0 ^ w as u64) + .collect(); + let pb: Vec = (0..words_for(N)) + .map(|w| 0x3333_3333_3333_3333u64.rotate_left(w as u32)) + .collect(); + // mask-risc refuses set bits past `n_rows` in a resident plane. + let tail = |mut w: Vec| { + let last = w.len() - 1; + w[last] &= (1u64 << (N % 64)) - 1; + w + }; + let (pa, pb) = (tail(pa), tail(pb)); + let masks = [pa.as_slice(), pb.as_slice()]; + let planes = Planes { + n_rows: N, + masks: &masks, + lanes: &lanes, + }; + let want = (0..N) + .filter(|&r| pa[r / 64] >> (r % 64) & 1 == 1 && pb[r / 64] >> (r % 64) & 1 == 1) + .count() as i64; + assert!(want > 0 && want < N as i64); + for mode in [BoolMode::InPlace, BoolMode::Fused] { + let mut d = FoldDialect::new(&planes, &Foreign::NONE).with_mode(mode); + let mut stack = vec![ + Val::Address(lance_graph_r2il::Addr::Plane(0)), + Val::Address(lance_graph_r2il::Addr::Plane(1)), + ]; + d.call(INT_AND, [0; 3], &mut stack) + .expect("plane AND plane"); + d.call(POP_COUNT, [0; 3], &mut stack).expect("count"); + assert_eq!(stack, vec![Val::Scalar(want)], "{mode:?}"); + } +}