From 251df396f0a13636c67331d08550b419209bb66a Mon Sep 17 00:00:00 2001 From: "Eric B. Ridge" Date: Fri, 18 Sep 2026 09:58:01 -0400 Subject: [PATCH] docs: describe boldi-vigna --- boldi-vigna/README.md | 135 ++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 135 insertions(+) create mode 100644 boldi-vigna/README.md diff --git a/boldi-vigna/README.md b/boldi-vigna/README.md new file mode 100644 index 0000000..334f2c7 --- /dev/null +++ b/boldi-vigna/README.md @@ -0,0 +1,135 @@ +# boldi-vigna + +`boldi-vigna` is a small, pure-Rust engine for evaluating positional queries +using Boldi-Vigna minimal-interval semantics. + +Given the sorted token positions for each query term in a single document, it +produces the document's minimal matching intervals: inclusive `[start, end]` +spans that witness why the query matched. Instead of answering only "does this +document match?", the crate returns the antichain of minimal witnesses, which +is exactly what makes phrase, proximity, snippet extraction, span frequency, and +proximity-aware scoring compose cleanly. + +This crate is intentionally narrow in scope. It does not tokenize text, scan +postings lists, or rank whole corpora. It just evaluates positional operator +trees over already-decoded term-position streams. + +## What makes it distinctive + +- It implements minimal-interval semantics, not ad-hoc span matching. Results + are sorted and minimal under inclusion. +- It compiles a `SpanQuery` once into a reusable `SpanSolver`, then reuses that + solver across many documents. +- Exact term-only phrases compile to a specialized fast path instead of paying + the full generic minimal-interval evaluator on every document. +- Evaluation is lazy: `intervals()` yields one minimal interval at a time. +- Inputs are borrowed through `TermPositions`, so callers can feed existing + position buffers without copying. +- It is pure Rust and storage-agnostic, so it fits equally well in tests, query + evaluators, and search engines. + +## Query model + +The public API is deliberately small: + +- `SpanQuery`: operator tree describing the positional condition. +- `SpanSolver`: compiled evaluator for a `SpanQuery`. +- `Interval`: one minimal witness span. +- `TermPositions`: trait supplying sorted positions for `SpanQuery::Term(i)`. + +The supported operators are: + +| Operator | Meaning | +|-----------------------------------|------------------------------------------------------------| +| `Term(i)` | Point intervals for term `i` | +| `Ordered([...])` | Children must appear in document order, non-overlapping | +| `Unordered([...])` | Minimal window containing one interval from each child | +| `Or([...])` | Disjunction, with antichain filtering of non-minimal spans | +| `MaxGaps { .. }` | Filter by uncovered positions between sub-intervals | +| `MaxWidth { .. }` | Filter by total interval width | +| `WithinPositions { .. }` | Restrict matches to a token-position window | +| `Containing`, `ContainedBy` | Containment relations between interval streams | +| `NotContaining`, `NotContainedBy` | Negative containment relations | +| `Overlapping`, `NonOverlapping` | Overlap relations | +| `Before`, `After` | Relative order relations | + +There are also a few convenience constructors: + +- `SpanQuery::phrase([..])` for exact phrases +- `SpanQuery::sloppy_phrase([..], max_gaps)` for phrase-with-slop +- `SpanQuery::first_n(inner, n)` for "must occur in the first `n` positions" + +## Example + +```rust +use boldi_vigna::{Interval, SpanQuery, SpanSolver}; + +let query = SpanQuery::phrase([0, 1, 2]); // exact phrase over three terms +let mut solver = SpanSolver::new(&query).unwrap(); + +// Term 0 at positions 0 and 10, term 1 at 1 and 11, term 2 at 2 and 12. +// The term index in SpanQuery::Term(i) corresponds to the slot in this array. +let positions = [vec![0, 10], vec![1, 11], vec![2, 12]]; + +let matches: Vec = solver.intervals(&positions).collect(); +assert_eq!(matches, vec![Interval::new(0, 2), Interval::new(10, 12)]); + +// Reuse the same compiled solver for another document. +let other_doc = [vec![5], vec![6], vec![20]]; +assert_eq!(solver.span_freq(&other_doc), 0); +``` + +`TermPositions` has blanket impls for common borrowed and owned layouts such as +`Vec>`, `Vec<&[u32]>`, `[Vec; N]`, and `[&[u32]; N]`. + +## How it is used in this repository + +This workspace uses `boldi-vigna` as the positional evaluation core: + +- `tinql` lowers user-facing span syntax into `boldi_vigna::SpanQuery`. +- `postgres` performs document-level candidate selection first, then decodes + positions only for surviving documents and runs `SpanSolver` to verify the + positional constraint and compute span frequency / proximity features. + +That split is the sweet spot for this crate: let the index narrow down +candidates cheaply, then let minimal-interval semantics do the precise +document-local positional reasoning. + +## Implementation notes + +- Intervals are inclusive and 0-indexed. +- Outputs are sorted by `(start, end)`. +- Outputs form an antichain: no returned interval strictly contains another. +- `SpanSolver::new()` validates the query tree's arity and term references. +- `TermPositions` is expected to provide sorted ascending positions for each + referenced term. + +## References + +The implementation is based primarily on the Boldi-Vigna work on +minimal-interval semantics and antichains of intervals: + +1. Paolo Boldi and Sebastiano Vigna, _Efficient lazy algorithms for + minimal-interval semantics_. SPIRE 2006. + - DOI: + - AIR entry: + - Historical first paper for the lazy interval algorithms. Vigna's papers + page notes that this version is superseded by the later "optimally lazy" + treatment. + +2. Paolo Boldi and Sebastiano Vigna, _Efficient optimally lazy algorithms for + minimal-interval semantics_. _Theoretical Computer Science_ 648, 2016. + - DOI: + - AIR entry: + - arXiv preprint: + - This is the main algorithmic reference for the core minimal-interval + operators implemented here. + +3. Paolo Boldi and Sebastiano Vigna, _On the lattice of antichains of finite + intervals_. _Order_ 38(1), 2018. + - DOI: + - AIR entry: + - arXiv preprint: + - Useful for the algebraic view of antichains of intervals and for thinking + about interval-set operators as first-class objects rather than one-off + query hacks.