fix(index): finalize retained symbol HNSW integration

Author: GlitterKill

CHANGELOG.md

@@ -11,7 +11,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ### Changed
 
-- **Incremental symbol embedding refresh**: Only changed `SymbolVectorEmbedding` rows are deleted and reinserted while a healthy HNSW is retained; the index is created only for bootstrap or missing-index recovery. In a direct spike, this was 18.5x faster than the prior drop-and-rebuild path for 50 changed rows out of 26,000, with ANN results durable after close/reopen; this is evidence, not an SLA. Migration 26 copies complete legacy vectors in-database. Legacy `Symbol` embedding columns remain inert for compatibility, and upgraded databases may also retain legacy `Symbol` HNSW indexes until a safe rebuild.
+- **Incremental symbol embedding refresh**: Only changed `SymbolVectorEmbedding` rows are deleted and reinserted while a healthy HNSW is retained; the index is created only for bootstrap or missing-index recovery after at least one complete model row exists. Repository-local cleanup removes only changed-file ownership links, preserves shared Symbol vectors, and deterministically transfers their canonical owner. Symbol ANN queries filter through repository-scoped projected graphs before ranking. In a direct spike, this was 18.5x faster than the prior drop-and-rebuild path for 50 changed rows out of 26,000, with ANN results durable after close/reopen; this is evidence, not an SLA. Migration 26 copies complete legacy vectors in-database. Legacy `Symbol` embedding columns remain inert for compatibility, and upgraded databases may also retain legacy `Symbol` HNSW indexes until a safe rebuild.
 
 ### Fixed
 

docs/architecture.md

@@ -261,15 +261,15 @@ flowchart TD
 
 Read pool enables concurrent multi-session reads (4-6 MCP sessions). Write serialization prevents graph corruption.
 
-### Graph Schema (24 Node Tables, 16 Edge Tables)
+### Graph Schema
 
 **Core nodes:**
 
 | Node Table        | Key Fields                                                                                                                                                        |
 | :---------------- | :---------------------------------------------------------------------------------------------------------------------------------------------------------------- |
 | **Repo**          | repoId, rootPath, configJson, createdAt                                                                                                                           |
 | **File**          | fileId, relPath, byteSize, contentHash, language, lastIndexedAt, directory                                                                                        |
-| **Symbol**        | symbolId, repoId, fileId, kind, name, exported, signatureJson, summary, summaryQuality, summarySource, etag, embeddingJinaCode*, embeddingNomic*                 |
+| **Symbol**        | symbolId, repoId, fileId, kind, name, exported, signatureJson, summary, summaryQuality, summarySource, etag; legacy inline embedding columns remain compatibility-only |
 | **Version**       | versionId, repoId, timestamp, indexedAt                                                                                                                           |
 | **SymbolVersion** | symbolId, versionId, signatureJson, summary                                                                                                                       |
 | **Metrics**       | symbolId, repoId, fanIn, fanOut, churn, testRefs                                                                                                                  |
@@ -301,9 +301,12 @@ Read pool enables concurrent multi-session reads (4-6 MCP sessions). Write seria
 | Node Table          | Key Fields                                            |
 | :------------------ | :---------------------------------------------------- |
 | **SymbolEmbedding** | symbolId, embedding, model (deprecated compatibility table) |
+| **SymbolVectorEmbedding** | embeddingId, repoId, symbolId, model, embeddingVector, cardHash, updatedAt, model-specific numeric vector |
 | **SummaryCache**    | symbolId, summary, provider, model, cardHash, costUsd |
 | **SymbolReference** | referenceId, symbolId, file, line                     |
 
+Production Symbol vectors live in `SymbolVectorEmbedding`, with one complete row per Symbol and model. Model-specific HNSW indexes target this table. Incremental indexing and background semantic repair delete and replace only changed embedding rows while retaining a healthy HNSW; index bootstrap occurs only when the exact table/name/type/property identity is absent and at least one complete row exists. Symbol ANN queries rank candidates inside a repository-scoped projected graph, so matching Symbol-to-File-to-Repo ownership filters the graph before the top-K search.
+
 **Sync, policy, and memory nodes:**
 
 | Node Table         | Key Fields                                                |
@@ -350,7 +353,7 @@ Each module owns a specific domain of queries:
 | `ladybug-clusters.ts`          | Cluster membership, label queries                                                   |
 | `ladybug-processes.ts`         | Process steps, role queries                                                         |
 | `ladybug-embeddings.ts`        | **Deprecated** — legacy SymbolEmbedding node queries                                |
-| `ladybug-symbol-embeddings.ts` | Inline embedding properties on Symbol nodes (replacement for ladybug-embeddings.ts) |
+| `ladybug-symbol-embeddings.ts` | Model-aware rows in `SymbolVectorEmbedding`, including incremental replacement and repository-scoped cleanup |
 | `ladybug-metrics.ts`           | Fan-in/out, churn, test refs                                                        |
 | `ladybug-feedback.ts`          | Agent feedback, audit events, searchText + embeddings for retrieval boosting        |
 | `ladybug-slices.ts`            | Slice handles, lease expiry                                                         |

docs/benchmarks/seed-resolution-evaluation-v2.json

@@ -7,13 +7,13 @@
     "caseCount": 4
   },
   "baseline": {
-    "gitHead": "4d34db7735856179c5f678263561e43e46ca5b0c",
-    "evaluatedSourceDiffSha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
+    "gitHead": "e93ee67311de91bebcb66511063a1d27dcbd3bb0",
+    "evaluatedSourceDiffSha256": "1817d5a641ae2d4d05afcf63f7e7d8777c54b23e706fbd60ba94372aaddb1234",
     "sourceHashes": {
       "src/context/engine.ts": "8f2f70e329bd99f0c00675ef556497e330fde5a6e2fa8fdf0c29041724c94ed3",
       "src/graph/slice/start-node-resolver.ts": "180a7020659ad3c52e4595bee0b1d2257e264b24bf9dbe4abc7bd29b46a67e70",
       "src/retrieval/identifier-extraction.ts": "475acdf8dc8f603e650137ab129a8dc1d15139c6711d87e4529f568b9de5da84",
-      "src/retrieval/orchestrator.ts": "316af9cfedcf2a9cc2c295c152220c8c83d5d393e864b2f4171e1382e435b779",
+      "src/retrieval/orchestrator.ts": "16da041a0f8a241006f3325a72c12c9b942d9c6d917a20d8e3d632945513d667",
       "src/retrieval/seed-resolver.ts": "662604e2076e58cd0f46055298f8296bf63ecb2e4433d714a2a76c714d41efdb"
     },
     "platform": "win32-x64",
@@ -393,8 +393,8 @@
     "sliceStartNodeRecall": 1
   },
   "observedMedianPolicyLatencyMs": {
-    "contextRetrieval": 0.0221,
-    "sliceStartNodes": 0.0082
+    "contextRetrieval": 0.023,
+    "sliceStartNodes": 0.0085
   },
   "reproduction": "npm run benchmark:seed-resolution",
   "check": "npm run build && node --experimental-strip-types scripts/evaluate-seed-resolution.ts --check"

docs/feature-deep-dives/provider-first-indexing.md

@@ -138,16 +138,16 @@ The current full-build path is:
 
 ### Graph Readiness Versus Semantic Readiness
 
-Embeddings, LLM summaries, retrieval-index bootstrap, and semantic enrichment are not part of the first ready gate. They advance a separate semantic readiness state. Provider-first graph finalization skips inline semantic work so the active graph can become graph-ready first.
+Embeddings, LLM summaries, retrieval-index bootstrap, and semantic enrichment are not part of the first ready gate. They advance a separate semantic readiness state. Provider-first graph finalization skips semantic work so the active graph can become graph-ready first.
 
 After graph finalization, SDL-MCP runs the configured semantic readiness refresh against the active DB:
 
 - Summaries run when `semantic.generateSummaries` is enabled.
 - Symbol and FileSummary embeddings run from the configured semantic model plan.
-- Deferred retrieval indexes are rebuilt.
+- Deferred retrieval indexes are ensured or bootstrapped when complete rows exist.
 - Semantic dirty flags are cleared only for completed work; an intentional embedding backlog leaves only the embedding flag dirty.
 
-If semantic refresh fails or an embedding provider degrades, the CLI reports `Semantic readiness: deferred`, records the derived-state error, and aborts readiness finalization. Sub-threshold FileSummary or Symbol vector work is different: SDL-MCP records it as an intentional backlog, continues the remaining semantic lanes and deferred-index work, clears any earlier semantic error plus completed summary work, and leaves `DerivedState.embeddingsDirty` set. The backlog accumulates until it reaches the protected HNSW rebuild floor or an explicit safe rebuild completes it. Mock fallback remains degraded and its rows are not reported as embedded. Repeated provider-first runs that reuse already-current active provider rows run the same post-graph semantic refresh rather than reporting a clean graph prematurely.
+If semantic refresh fails or an embedding provider degrades, the CLI reports `Semantic readiness: deferred`, records the derived-state error, and aborts readiness finalization. Sub-threshold FileSummary vector work is different: SDL-MCP records it as an intentional backlog, continues the remaining semantic lanes and deferred-index work, clears any earlier semantic error plus completed summary work, and leaves `DerivedState.embeddingsDirty` set until the FileSummary HNSW rebuild floor is reached or an explicit safe rebuild completes it. Symbol embeddings do not use that rebuild floor: changed model rows are deleted and reinserted while a healthy `SymbolVectorEmbedding` HNSW remains live. Mock fallback remains degraded and its rows are not reported as embedded. Repeated provider-first runs that reuse already-current active provider rows run the same post-graph semantic refresh rather than reporting a clean graph prematurely.
 
 
 ### Persisted Graph Integrity
@@ -466,7 +466,7 @@ This keeps placeholder metadata, repo links, rare C++ provenance with commas, qu
 
 ## Derived State, Metrics, And Summaries
 
-Semantic refresh is skipped in provider-first post-index finalization and tracked as deferred semantic readiness so index wall time is no longer dominated by the `Summary Embeddings` and `Symbol Embeddings` phases. When semantic refresh is deferred, the deferred index build leaves Symbol FTS, entity FTS, Symbol vectors, and FileSummary vectors for later retrieval readiness/bootstrap work rather than charging those builds to the first provider-first index wall-clock.
+Semantic refresh is skipped in provider-first post-index finalization and tracked as deferred semantic readiness so index wall time is no longer dominated by the `Summary Embeddings` and `Symbol Embeddings` phases. When semantic refresh is deferred, the deferred index build leaves Symbol FTS, entity FTS, initial `SymbolVectorEmbedding` HNSW bootstrap, and FileSummary vectors for later retrieval readiness work rather than charging those builds to the first provider-first index wall-clock.
 
 PageRank and K-core run by default for the readiness-critical centrality signal. Louvain shadow communities remain optional derived enrichment and are policy-skipped above `indexing.algorithmRefresh.louvain.maxCallEdges`, which defaults to `10000` call edges so provider-first full indexes do not spend most of their wall time in LadybugDB community detection.
 

docs/feature-deep-dives/semantic-embeddings-setup.md

@@ -482,7 +482,7 @@ Configure each lane directly; there is no legacy mode switch.
         "conjunctive": false, // true = AND all terms; false = OR
       },
       "vector": {
-        "enabled": true, // Vector search on inline embeddings
+        "enabled": true, // SymbolVectorEmbedding and FileSummary HNSW search
         "topK": 75, // Max candidates per model
         "efs": 200, // Query-time accuracy parameter
       },
@@ -498,8 +498,8 @@ Configure each lane directly; there is no legacy mode switch.
 
 ### How It Works
 
-1. **FTS and vector indexes are created automatically** on DB init when `semantic.enabled: true`. The FTS extension indexes `Symbol.searchText`; vector indexes cover `Symbol.embeddingJinaCode` and `Symbol.embeddingNomic`.
-2. **At query time**, FTS and vector searches run in parallel. Each source produces a ranked candidate list.
+1. **FTS and vector indexes are ensured automatically** on DB init when `semantic.enabled: true`. The FTS extension indexes `Symbol.searchText`; Symbol vector indexes cover model-specific numeric columns on `SymbolVectorEmbedding` after at least one complete row exists.
+2. **At query time**, FTS and vector searches run in parallel. Symbol vector search uses a repository-scoped LadybugDB projected graph, so repository filtering happens before ANN ranking and unrelated repositories cannot consume the top-K window. Each source produces a ranked candidate list.
 3. **RRF fuses** the rank lists: `score(d) = S 1/(k + rank_i(d))` � symbols ranked highly by multiple sources rise to the top.
 4. **If an extension is unavailable** (for example, `fts` or `vector` is not loaded), the system omits that lane, renormalizes the remaining weights, and records the reduced coverage in telemetry.
 
@@ -521,7 +521,7 @@ Retrieval extensions ...................... PASS
 
 ### Migration from SymbolEmbedding
 
-Prior to hybrid retrieval, embeddings were stored in a separate `SymbolEmbedding` node table. Migration m007 automatically copies embeddings to inline Symbol properties (`embeddingJinaCode`, `embeddingNomic`) on DB init. Mock-fallback rows are skipped. The old `SymbolEmbedding` table is deprecated but retained for backward compatibility.
+Prior to hybrid retrieval, embeddings were stored in a separate `SymbolEmbedding` node table. Migration m007 copied those values to inline Symbol properties. Migration m026 copies complete supported inline vectors into model-scoped `SymbolVectorEmbedding` rows. Mock-fallback and incomplete rows are skipped. The old `SymbolEmbedding` table and inline Symbol columns remain compatibility-only; current writes and HNSW indexes use `SymbolVectorEmbedding`.
 
 The current recommended configuration surface is `semantic.retrieval.*`. Retired compatibility knobs are intentionally omitted from this setup guide.
 
@@ -630,14 +630,14 @@ The `intraOpNumThreads` setting is the single most impactful knob after model va
 
 ## Embedding Vector Storage
 
-Embeddings are stored as **inline properties on Symbol nodes** in LadybugDB:
+Symbol embeddings are stored as **model-scoped rows in `SymbolVectorEmbedding`**. FileSummary and AgentFeedback embeddings remain on their own entity nodes.
 
 ```mermaid
 %%{init: {"theme":"base","themeVariables":{"background":"#ffffff","primaryColor":"#E7F8F2","primaryBorderColor":"#0F766E","primaryTextColor":"#102A43","secondaryColor":"#E8F1FF","secondaryBorderColor":"#2563EB","secondaryTextColor":"#102A43","tertiaryColor":"#FFF4D6","tertiaryBorderColor":"#B45309","tertiaryTextColor":"#102A43","lineColor":"#0F766E","textColor":"#102A43","fontFamily":"Inter, ui-sans-serif, system-ui, -apple-system, BlinkMacSystemFont, Segoe UI, sans-serif"},"flowchart":{"curve":"basis","htmlLabels":true}}}%%
 flowchart TD
-    Symbol["Symbol node"] e1@--> Jina["embeddingJinaCode<br/>embeddingJinaCodeCardHash<br/>embeddingJinaCodeUpdatedAt"]
-    Symbol e2@--> Nomic["embeddingNomic<br/>embeddingNomicCardHash<br/>embeddingNomicUpdatedAt"]
-    Symbol e3@--> Jina["embeddingJinaCode<br/>embeddingJinaCodeCardHash<br/>embeddingJinaCodeUpdatedAt"]
+    Symbol["Symbol node"] e1@--> Row["SymbolVectorEmbedding row<br/>repoId + symbolId + model<br/>text + hash + updatedAt"]
+    Row e2@--> Jina["embeddingJinaCodeVec<br/>Jina HNSW"]
+    Row e3@--> Nomic["embeddingNomicVec<br/>Nomic HNSW"]
 
     classDef source fill:#E7F8F2,stroke:#0F766E,stroke-width:2px,color:#102A43;
     classDef process fill:#E8F1FF,stroke:#2563EB,stroke-width:2px,color:#102A43;
@@ -649,6 +649,8 @@ flowchart TD
     class e1,e2,e3 animate;
 ```
 
+After bootstrap, incremental indexing and the background semantic repair worker replace only changed model rows and retain the live Symbol HNSW. Retrieval ranks candidates inside a repository-scoped projected graph. Startup and health checks accept an HNSW only when its table, name, type, and property all match the configured model.
+
 Vectors are compressed using Float16 quantization:
 
 ```text