[MOD-14957] Add tiered SQ8 mean training (3/3)

[dor-forer]

Describe the changes in the pull request

Add mean training to tiered SQ8 while keeping the real HNSW backend present from construction. With a positive QuantNormalizationSetSize, vectors remain searchable in FLAT until the threshold is reached. Tiered then installs the learned mean in the existing SQ8 components and starts migration.

Stack 3/3, based on #1035 (dor-forer-MOD-14957-tiered-sq8-backend): #1034 → #1035 → #1029. The first two PRs contain the FLAT overwrite fix and immediate-backend SQ8 support.

• The factory creates the final SQ8 layout with an initial zero mean when training is requested. The backend pointer, preprocessing layout, and cached distance-context addresses remain stable.
• Tiered owns an optional accumulation state with a double-precision running sum and a threshold. Overwrites and deletions adjust the sum using FLAT's stored vectors, including cosine normalization. The vector count and threshold decision are read under the FLAT lock.
• Finalization computes the FP32 mean and snapshots pending insertion jobs. Under the exclusive main-index lock, HNSW passes the mean to the preprocessor and distance calculator. The calculator computes the squared-mean correction for IP; L2 needs no such correction. Tiered then clears the accumulation state and releases the lock before executing or submitting jobs.
• Named scoped guards handle exclusive locking in finalization, direct writes, and swap-job execution. The guard helper preserves the cumulative lock-acquisition counter used by tests. External shared locks are released in reverse acquisition order.
• Memory estimates include the mean-enabled backend layout and the temporary running sum. Basic metadata continues to come from the backend.

A zero threshold skips training and preserves a supplied quantParams mean, if any. Positive thresholds override a supplied mean, take precedence over flatBufferLimit, and are capped at 102400 vectors. Emptying a trained graph does not restart training. SQ8 getDataByLabel() returns no vectors in every phase, including when a label is buffered in FLAT.

Initialization contract and known behavior

• Mean installation is writer-only: once, before any vector is stored, under the exclusive tiered main lock and before insertion jobs are submitted. Assertions check quantized storage, an empty graph, the mean dimension, and the concrete SQ8 components.
• The setter requires one mean-enabled SQ8 preprocessor in slot 0. Tiered performs cosine normalization in FLAT.
• In write-in-place mode, the threshold-crossing addVector() migrates the accumulated set before returning. This can perform up to 102400 insertions in one call.

Which issues this PR fixes

MOD-14957: tiered SQ8 mean accumulation and initialization.

Validation

Validated fb2c5b0426134c525c4af02c916112197b9c9477 after merging base 4c2f7509, on dorer-intel with GCC 13.3, Debug, and SVS v0.3.2 enabled:

• test_hnsw: 158 focused tiered tests passed, including the SQ8 training cases in test_hnsw_tiered.cpp.

• test_hnsw_sq8: all 133 tests passed.

• test_svs: all 4 selected getDataByLabel tests passed; this binary includes standalone and tiered SVS retrieval tests.

• All three test targets built successfully with no compiler warnings.

• All 463 tracked-file checksums matched the tested source snapshot. Repository-wide make check-format and git diff --check passed.

• The conflict resolution preserves the updated base's SVS retrieval implementation; focused standalone and tiered SVS retrieval tests cover the shared header.

Historical full-suite validation at fa472126: 3,050 Debug CTest cases passed with 8 expected SVS skips; with FP64_TESTS=ON, 3,316 passed with 9 expected skips. Those full suites, RediSearch integration, and ThreadSanitizer were not rerun at this head.

Mark if applicable

• This PR introduces API changes
• This PR introduces serialization changes

---

Note

Medium Risk
Changes tiered HNSW write paths, locking around quantization finalization, and SQ8 distance/preprocess state; mistakes could affect search correctness or concurrency during the training transition.

Overview
Adds deferred SQ8 mean training for tiered HNSW via new TieredHNSWParams.QuantNormalizationSetSize. When set > 0, inserts stay in the FLAT tier (searchable there) while a double-precision running sum tracks component means; HNSW migration jobs are held until the threshold is hit, even if that exceeds flatBufferLimit. At finalization, tiered calls HNSWIndex::setQuantizationMean() under the exclusive main lock to update the existing SQ8 WithNorm preprocessor and distance calculator in place, then submits or synchronously runs the batched insert jobs.

Tiered factory now builds the SQ8 backend immediately with a placeholder zero mean when training is required; 0 skips accumulation and keeps prior immediate-SQ8 behavior (optional quantParams mean). HNSW gains setQuantizationMean, acquireSharedIndexDataGuard, and mutable mean hooks on QuantPreprocessor::setMean / DistanceCalculatorWithNorm::setMeanSumSquares, plus with_mean overloads for memory estimation.

Supporting changes: RAII main/index data locks in tiered paths, getPreprocessorsContainer() exposure, and broad unit coverage for accumulation, finalization, write modes, and concurrent query/migration edge cases.

^{Reviewed by Cursor Bugbot for commit fb2c5b0. Bugbot is set up for automated code reviews on this repo. Configure here.}

[CLAassistant]

All committers have signed the CLA.

[cursor]

Stale Bugbot comment from a previous run.

[cursor]

Stale Bugbot comment from a previous run.

[codecov]

Codecov Report

❌ Patch coverage is 98.86364% with 2 lines in your changes missing coverage. Please review.
✅ Project coverage is 97.45%. Comparing base (4c2f750) to head (fb2c5b0).

Files with missing lines	Patch %	Lines
src/VecSim/algorithms/hnsw/hnsw.h	94.44%	1 Missing ⚠️
src/VecSim/algorithms/hnsw/hnsw_tiered.h	99.13%	1 Missing ⚠️

Additional details and impacted files

@@                            Coverage Diff                             @@
##           dor-forer-MOD-14957-tiered-sq8-backend    #1029      +/-   ##
==========================================================================
+ Coverage                                   97.41%   97.45%   +0.04%     
==========================================================================
  Files                                         141      141              
  Lines                                        8716     8856     +140     
==========================================================================
+ Hits                                         8491     8631     +140     
  Misses                                        225      225              

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.