[BubbleCal]

FTS build workers can retain large buffers and emit many partitions. This draft adds a bounded, opt-in background merge pipeline plus a size-based streaming final merge. Larger output partitions exposed query RSS and phrase-tail regressions; eliminating those regressions remains the acceptance goal.

Current head: 8188b1dd6d8ab263d6b34100aa1bfd585873b83c, based on merged main d27fd014d4851ecfc7d1acef68d9bf705bd5453d. This revision adds bounded, request-owned position read-ahead while essential CPU/I/O work is pending. Current plus speculative encoded windows share the original clause budget; no background task, larger cache, larger window or concurrency override is introduced. New-head CI and uninstrumented performance are pending. Parent e6bb passed correctness CI and fixed offset cache placement, but still failed phrase-tail acceptance. Both features remain default-off and this PR remains draft.

Build pipeline

• Indexing workers emit bounded small runs. Queues are bounded by count and KiB-rounded retained bytes, with separate producer, queue, merger-input and working reservations. Merger concurrency is admitted from available memory.
• Final retained-output planning is independent of requested merger count. Bounded fan-in streaming consolidation can combine persisted runs beyond the in-memory working set. Final reads, rewrites, cleanup and dataset commit count toward complete build time.
• Success drains accepted CPU/write/tail work before root metadata and dataset commit. Errors stop admission and join started work. Future cancellation initiates asynchronous drain; same-builder retries use monotonic partition IDs to avoid overwriting in-flight writes.
• Existing persisted grammars and query semantics are preserved. Current mini-block encoding is selected for new position output; legacy readers/writers are not extended.

Enable with LANCE_FTS_PIPELINED_MERGE=1 (default off). LANCE_FTS_PIPELINE_MERGE_WORKERS is maximum concurrency, default1. Optional RUN_MIB, TARGET_MIB, FINAL_TARGET_MIB and QUEUE_RUNS use the same prefix. CPU/8 is not an adopted default. The final disk pass is sequential by merge group, not a complete Lucene background compactor.

Query reader and current metadata fix

The shared reader changes compact partial mini-block output, preserve verified Arrow allocation ownership, release exhausted inputs after the last dependent selection/task, skip unused single-term phrase positions, and score ready inputs within the existing I/O windows using bounded byte-aware CPU batches.

LANCE_FTS_SELECTIVE_POSITIONS=1 separately enables a default-off, standalone exact-phrase reader. WAND retains owned conjunction/top-k/score-floor state across async window reads. CPU closures perform no I/O, locks or waits. Repeated terms share encoded windows; request-scoped I/O handles never enter global caches.

Eligibility: finite top-k, standalone AND, exact phrase, factor1,2–16 independent query positions and supported current shared-position mini-block encoding. Unsupported/grouped/sloppy/legacy/oversized cases select ordinary loading before consuming search state. Complete adjacent codec blocks form deterministic2MiB target windows; one indivisible larger block must fit its clause budget. At most four active partitions/query share32MiB each between their current and speculative encoded windows. These bounds exclude postings, offsets, decoder scratch, cache residency, transient I/O and other queries; they are not process RSS limits.

The current fix:

• BinaryValueLocation contains only a validated scalar file range. The existing file-scoped metadata cache keys locations by physical column and row, and accounts them within its existing budget.
• Every returned prepared handle binds the current scheduler and priority. Cached locations retain no request recorder, I/O handle or payload owner. Failed preparation is not a successful cache entry.
• Immutable position offsets prefer the existing bounded, file-scoped metadata cache, with their backing allocation and Arc header accounted. Payload eviction does not invalidate surviving metadata. Readers without a separate tier use the original index-cache fallback; a disabled metadata cache stays disabled. Weak handles retain no I/O scope or cache backend. Window plans retain shared offsets; dynamic I/O handles remain request-local.
• The2MiB window target,32MiB/partition bound, four active searches and default-off policies are unchanged. No experimental tuning overrides or instrumentation are shipped.

Sophon PE leaves and QN fallback can reach the shared reader if explicitly enabled. Routing, RPC/scorer payloads, QN aggregation, dependency pins and production settings are unchanged. Caches and query overlap remain per worker; no distributed speedup, global RSS bound or Enterprise adoption is established.

Validation

Current8188 Rust CI and Python/Java workflows are running. New cases cover current+next byte admission, consuming a completed speculative read once, avoiding speculative work when essential work is already ready, deferred speculative errors, and dropping/skipping accepted pending I/O with Moka/Quick/disabled caches. These extend existing row/score/phrase/floor/retry/cancellation coverage; full multi-window query qualification remains part of the ongoing draft. Local format/diff checks passed; no local correctness tests/builds were run.

Parente6bb passed all six workflows. Linux logs report192 focused tests,23,358 full merge-tree tests and605 exact-head CPU1/Tokio2 tests passed, including both previously failing eviction cases and both disabled-metadata cases. Strict Clippy, MSRV and cross-platform checks passed. These results do not accept8188.

The before-fix90b Linux job ran192 focused tests:190 passed, and exactly the filtered/unfiltered payload-eviction cases failed at small immutable position offsets must use the file metadata budget. The fix preserves both assertions: after payload eviction, metadata offset loader counts must stay unchanged and hit counts must grow, while row identities/scores remain equal. Separate disabled-metadata cases require zero resident bytes and the same semantic output. Before-fix strict Clippy/MSRV checks passed; new-head results are pending.

Prior 825a4e297: all 40 checks across six workflows passed. Rust CI includes strict Clippy, MSRV1.95, Linux x86/ARM, Windows x86/ARM, macOS, protobuf/no-protoc, Arrow ordering and rustdoc. Linux logs report188 early tests,23,261 merge-tree tests and601 exact-head CPU1/Tokio2 tests passed, including the new cache-scope/priority, isolation and cancellation cases. Format and diff checks passed locally; correctness builds/tests run in CI, benchmark builds/measurements on the managed ARM64 VM.

The two36e failures were one test-only field/method typo, fixed in825 without a toolchain increase, warning suppression or weaker assertions. Production metadata code was unchanged by that follow-up.

New/extended CI coverage checks:

• Repeated preparation performs no metadata I/O across supported versions, physical rows and page boundaries.
• Detached locations release the producing I/O scope and use a new request's recorder and priority.
• Cache isolation across files, columns and rows uses different lengths and payloads.
• Truncated metadata fails and then retries without poisoning the cache.
• Repeated committed multi-partition/filter searches preserve row identities and scores without extra warm reads; tied completion order is not assumed.
• Cancellation releases accepted pending I/O with Moka, QuickCache and disabled cache.

Existing coverage includes short reads/cache retry, partial drains, custom/shared/sliced owners, allocation lifetimes, resumable phrase search, late high scores, repeated terms, score floors, wrong-window rejection, full queues, pending sources, worker/merge/write/input errors, cancellation and same-builder retries. Pinned cache cancellation behavior and existing backend regression coverage were inspected; no gates or assertions were weakened.

Historical bfc4bab09 passed all40 checks and all six workflows:184 early regressions,23,210 merge-tree tests and597 exact-head CPU1/Tokio2 cases, including strict Clippy, MSRV1.95, Linux ARM, Windows x86/ARM, macOS, protobuf/no-protoc and Arrow ordering. Those results do not accept the new metadata fix.

The current read-ahead mechanism keeps one request-owned next-window future per distinct term, admitted only when current+next encoded bytes fit the original clause allowance. It polls those futures only while essential work is pending. Matching demand consumes the result/future; skips discard stale predictions. Unused speculative errors are not query errors. Completion clears speculation and releases the finished search state before rescoring. No task/channel/worker is added. Scheduler/coalescing buffers, cache residency, metadata, postings, scratch and other queries are outside the encoded bound. Speculation may overread, so scheduled bytes/ops and phrase tails must be measured; bounded memory alone is insufficient.

Measurement status and preserved negatives

The RSS/per-variant p95 acceptance goal is still unmet. The new uninstrumented screen pairs latest fetched main e0aacd774acf66fba063c01d7a4a4d911e63467e with exact825 reader source. Candidate retains merged cbed ancestry; e0 adds vector-only quantized refinement caching, outside this FTS pool. Both readers use the same frozen676 external dependency pins, harness, release-with-debug profile and allocator. Old beta20 binaries are not relabeled.

Historical825 reader comparison on the same saved writer layouts (main40 / candidate19):

Scenario / metric	Baseline main, initial / return	This PR, selective off / on, saved19	Benefit or remaining cost
Whole-run peak RSS, lower is better	8.750 / 8.595 GiB	8.598 / 5.117 GiB	On: 1.68–1.71x lower RSS
Warm throughput, higher is better	31.750 / 30.704 QPS	41.230 / 28.200 QPS	On: 1.09–1.13x slower
Warm mixed p95, lower is better	2,399 / 2,348 ms	2,463 / 4,560 ms	On: 1.90–1.94x slower
Warm two-term phrase p95, lower is better	1,997 / 1,955 ms	2,915 / 5,712 ms	On: 2.86–2.92x slower
Warm five-term phrase p95, lower is better	3,223 / 3,180 ms	3,342 / 5,871 ms	On: 1.82–1.85x slower

Resource controls: CPU16/C32, hardRAM16.5GiB, indexcache4.125GiB, metadata128MiB, swap0. Each of five cases has288 queries/pass, one first pass and two same-process repeats; warm values are medians of repeats, RSS is the whole invocation including post-run cache snapshots. All4,320 responses passed row/score checks, allowing only documented top100 cutoff ties. No confidence interval, independent rebuild distribution or sustained-service claim follows.

Actual on19 caches include2,147 detached BinaryValueLocation entries (120,232 accounted bytes),89 PositionOffsets entries (~0.34MiB), and2,155 encoded windows. Reuse occurs, but it is insufficient. The on40 fallback guard observed no EncodedPositionWindow entries; its timing differences are not selective-path gains. On19 scheduled operations/bytes remain158.089ops and44.939MiB/query, versus off19 at138.767ops and37.310MiB/query; these are not wire traffic or cache-miss counts.

The saved writers remain 3f03534 / c92eb2e, so this screen qualifies reader behavior only, not current-head writer benefit. A separate unpublished825+bf006eb experiment has now tested partition overlap4/8/16 after metadata caching, keeping2MiB windows,32MiB/partition and preparation concurrency4. Another4,320 responses passed row/score checks. P4/P8/P16 RSS was5.088/5.216/5.599GiB and five-term phrase p95 was5289/5772/4719ms; main initial/return was9.234/9.346GiB and3559/3258ms. All three limits still fail phrase-tail acceptance. This run is sequential, with one reconciled SSH preflight interruption beforeP16: completed cases were retained, only unstarted cases resumed. The ordering and interruption prevent unique attribution of the P4/P16 difference; these larger main RSS anchors do not replace the earlier paired screen. No parameter change is published or adopted. A completed two-case instrumented follow-up checked1,728 responses. In repeats, binary-location preparation summed0.3–0.9ms/query, yet65–72% of offset calls executed a loader. Median per-query preparation share was16.5–32.3% of selective-function wall time; CPU queue elapsed sums were4–10ms. Window waits remain material. These overlapping wait sums are not disjoint fractions or process CPU time, and all instrumented timings/RSS are excluded from performance acceptance. This evidence motivated the current metadata-tier selection fix. It does not increase cache budgets or assume that this alone fixes phrase tails; current-head uninstrumented qualification remains required.

Parent e6bb, against the same frozen d27 main, has now completed a separate five-case uninstrumented screen and4,320 row/score checks. On19 metadata contains2,147 PositionOffsets entries (5,314,700 accounted bytes), with zero such entries in the index payload tier. This confirms the placement fix but not a performance solution. Off/on19 RSS is8.680/5.201GiB, mixed p95 is2795/4384ms, two-term phrase p95 is3403/6435ms, and five-term phrase p95 is3573/6578ms. Main initial/return RSS is8.756/8.726GiB and five-term phrase p95 is3288/3239ms. The on40 fallback guard observed no selective-window entries. Conditions match the bounded288-query screens above; all new-head performance claims remain unverified. The next finite screen compares8188 directly with e6bb on the saved19 layout and retains main initial/return controls. No fresh writer benefit or adoption is claimed.

Historical writer main 3f03534 / candidate c92eb2e demonstrated40→19 committed partitions and22.452→7.328GiB build RSS, with full S3 source-to-S3 committed time439.882→609.455s. Query RSS and phrase tails still regressed. These are frozen historical outputs, not current-head writer benefits or an adopted tradeoff. Main has not been retuned to an equal observed-RSS frontier.

The prior d9cc16a1 Jack2M low-cache regressions remain preserved: baseline/M1 warm C16 was32.87/30.52QPS; baseline/M2 was32.80/20.25QPS. That M2 layout had57 partitions versus main33 and lacked complete final planning. Larger-cache diagnostics do not replace those negatives.

Subsequent saved-layout reader experiments are also rejected:

• Per-block c51 reads reduced RSS but multiplied scheduling and CPU resumptions. All20,736 responses checked rows/scores, yet throughput fell from37.151 to0.761QPS on the same19-partition layout.
• Contiguous2MiB bfc windows passed3,456 response checks but retained adverse phrase tails. Same-binary off/on19 five-term p95 was3282/7453ms.
• A benchmark-only bfc concurrency variant (patch c57e2b4) checked4,320 responses. P4/P8/P16 five-term p95 remained6970/6737/7555ms; main initial/return was3246/3124ms. More partition overlap alone was insufficient.
• A benchmark-only bounded window-size variant (patch f63d4e1) checked another4,320 responses. P16/W2→P16/W8 increased scheduled bytes45.936→84.340MiB/query and five-term p95 from6271→10368ms. Actual8MiB targets differ by clause budget, so cache granularity also changes; this is not a unique-cause attribution.

These screens used CPU16/C32, hardRAM16.5GiB, indexcache4.125GiB, metadata128MiB, swap0, one saved independent index/layout and one first pass plus two same-process repeats. Smaller288-query screens must not be compared numerically with1152-query screens. No prewarm: configured32KiB grouping has effective128-token unplanned fallback; CacheAwareExact can still select an individual token. S3 service cache is uncontrolled. Scheduled bytes/operations are not wire traffic or cache-miss counters. No confidence intervals or sustained-service throughput claim follows.

An instrumented follow-up checked1,728 responses. With2MiB windows, repeat-query preparation took median1.283s for two terms and3.020s for five; the median per-query preparation share was79.5%/68.0% of the selective function's wall time. This excludes outer query setup; overlapping wait sums are not additive wall fractions or CPU/network measurements. Instrumented timings/RSS are excluded from performance acceptance. This evidence motivated the current metadata fix; it does not prove its benefit.

Limits and delivery boundary

Reservations/cache accounting are not hard RSS caps. Oversized documents/runs may be rejected. Cancellation has no synchronous deadline; independently reconstructed overlapping builders need distinct destinations because no cross-process destination lock is provided. Internal FTS partitions are not independently routable Sophon segments.

Low-worker automatic targets, short-text build cost, prewarmed byte directories, released-wheel reads, optimal memory/speed frontiers and Sophon total QN/PE memory/network/skew/scaling remain unqualified. Benchmark output checks support interpretation and do not replace CI correctness acceptance.

The query RSS/per-variant p95 goal remains unmet: current825 evidence still shows unacceptable phrase tails. This phase is draft-only: no default adoption, Ready, external comments/reviews, approval actions, merge, release or deployment.