From 3b3eb1c03acc513991fb89fecc3ad376415be8f2 Mon Sep 17 00:00:00 2001 From: Aaron Queen Date: Fri, 11 Sep 2026 16:10:46 -0600 Subject: [PATCH] docs(design): kernel-only extraction, resolution binding model, greenfield Rust sketch Three linked design docs recording the 2026-09-11 assessment of a Rust rewrite against the fork's PR history, the espn-draft retrieval research and the query audit log: - kernel-only-extraction-plan.md: remove the WASM path; phases for the error-recovery flip, SFC extractors calling the kernel, a parse-tree service for the three read-time consumers, tail-language port/drop decisions, and golden dumps replacing the parity oracle. - resolution-binding-model-plan.md: a per-file bindings table emitted by extraction replaces the five regex families that answer export and binding status (#1721) and ends the #1566/#1790/#1794/#1844 cycle. - greenfield-rust-core-sketch.md: what a from-scratch core must reproduce, what it would buy and cost, and why it is deferred behind the two incremental plans. --- docs/design/greenfield-rust-core-sketch.md | 135 ++++++++++++++++ docs/design/kernel-only-extraction-plan.md | 155 +++++++++++++++++++ docs/design/resolution-binding-model-plan.md | 155 +++++++++++++++++++ 3 files changed, 445 insertions(+) create mode 100644 docs/design/greenfield-rust-core-sketch.md create mode 100644 docs/design/kernel-only-extraction-plan.md create mode 100644 docs/design/resolution-binding-model-plan.md diff --git a/docs/design/greenfield-rust-core-sketch.md b/docs/design/greenfield-rust-core-sketch.md new file mode 100644 index 000000000..ccd5b7e8c --- /dev/null +++ b/docs/design/greenfield-rust-core-sketch.md @@ -0,0 +1,135 @@ +# Greenfield Rust core — design sketch and tradeoff + +**Status:** sketch, not approved. Written 2026-09-11 so the "rewrite from scratch in Rust" option is judged on paper against the two incremental plans it competes with: [kernel-only extraction](kernel-only-extraction-plan.md) and the [resolution binding model](resolution-binding-model-plan.md). + +**Recommendation:** do not start a greenfield core. Do the two incremental plans first. If, after both land, the TypeScript side is a thin driver over a Rust crate, spin that crate into its own repository then. The reasons are in §5. + +## 1. What a replacement must reproduce + +This is the surface a from-scratch core inherits on day one. Every row is a user-visible contract or a persisted artifact; none is optional. + +### 1.1 Persisted graph (`.codegraph/codegraph.db`) + +- Tables `nodes`, `edges`, `files`, `unresolved_refs`, `name_segment_vocab`, `literals`, `project_metadata`, `schema_versions`, plus the FTS5 external-content table `nodes_fts` with its three sync triggers. Schema in `src/db/schema.sql`, ten ordered migrations in `src/db/migrations.ts`. +- Two version axes: schema version (migratable, DDL-only, never backfills content columns) and `EXTRACTION_VERSION` (content shape, drives the re-index advisory in `codegraph status` and `codegraph upgrade`). Currently 28. +- Node identity is a deterministic text id; edge identity is `(source, target, kind, line, col)` with `INSERT OR IGNORE` dedupe. Provenance column separates extracted from `heuristic` (synthesized) edges. Every explore answer and every golden dump depends on these being stable. +- Second database `.codegraph/sessions.db` for transcript search (FTS5 porter, BM25). + +### 1.2 Vocabulary + +`NodeKind` (23 values) and `EdgeKind` (12 values) in `src/types.ts`. Extractors, resolvers, synthesizers, explore ranking, the viewer, and the kernel ABI contract check (`src/extraction/kernel/loader.ts`) all key on the exact strings. + +### 1.3 Extraction + +29 grammars, 20 of them in the Rust kernel already. Nine hand-written slicing extractors for embedded-language files (Vue, Svelte, Astro, Razor, Liquid, Markdown, CFML, MyBatis, DFM). Literal capture, generated-file detection, C/C++ blanking pre-passes, docstring and signature capture, per-file error handling. Inventory and line counts in the kernel-only plan. + +### 1.4 Resolution + +About 30k lines: import resolver, name matcher, alias binding, path aliases, workspace packages, Go modules, 33 framework resolvers under `src/resolution/frameworks/`, and 36 synthesis passes (callback, EventEmitter, React re-render, JSX child, React Native events, router synthesizers for Next, Expo, React Router, TanStack, Vue Router, SvelteKit, C function pointers, GoFrame, Swift/ObjC bridge, tiering). Each was validated against real repos by agent A/B. None is derivable from a spec. + +### 1.5 Graph derivations (`src/graph/`) + +Traversal, symbol lookup (the one "what did the user mean" resolver), named-symbol flow (the one path finder), type hierarchy, dead code, branch guards (query-time AST walk for `WHEN` labels), dynamic boundary report. + +### 1.6 Retrieval and MCP + +- Tools: `codegraph_explore`, `codegraph_sessions` served by default; `search`, `callers`, `callees`, `impact`, `node`, `status`, `files` behind `CODEGRAPH_MCP_TOOLS`. Input schemas in `src/mcp/tools.ts`. +- Explore semantics: budgets tiered by indexed file count (`getExploreBudget`, `getExploreOutputBudget`), allocation across named and discovered files, gap markers, verbatim line-numbered source in Read's shape, flow section, blast radius, staleness and degraded banners, session dedup, did-you-mean for exact names, literal seeding. +- Server instructions in three variants (indexed root, unindexed root, unindexed worktree). The response-shape rule: expected conditions are success-shaped, `isError` only for refusals and malfunctions. +- Process model: detached daemon per project root on a socket or named pipe, stdio proxy with PPID watchdog, query worker pool with per-worker WAL read connections and circuit breaker, single writer lock, liveness and startup watchdogs, refresh launcher, daemon registry under `~/.codegraph/daemons/`. + +### 1.7 Sync + +`fs.watch` based watcher with platform-specific strategies (one recursive watch on macOS and Windows, one inotify watch per directory on Linux with a cap), scope built to match `git ls-files --exclude-standard`, WSL2 `/mnt` force-off, debounced sync, lock-contention retry and degraded state, opt-in git hooks, worktree index-mismatch detection. + +### 1.8 CLI, installer, viewer, sessions, telemetry, upgrade + +- 27 CLI commands including the hidden `prompt-hook` and `serve`. +- Installer for 11 agent hosts with per-host config formats (JSON, JSONC, TOML), marker-fenced instruction blocks, the Cursor `--path` quirk, idempotency and uninstall contracts. +- Viewer: 16 HTTP endpoints plus SSE, own highlighter reading the same grammar as the engine, trail store. +- Sessions: readers for Claude, Codex, Cursor, OpenCode, AGY transcript formats. +- Telemetry, update check, self-upgrade including the Windows locked-binary helper. + +### 1.9 Settings + +About 75 `CODEGRAPH_*` environment variables plus `codegraph.json` (`extensions`, `include`, `includeIgnored`, `exclude`, `deprioritize`, `sessions`). Grouped: + +| Group | Count | Fate under a Rust core | +|---|---|---| +| WASM and V8 workarounds (`KERNEL*`, `WASM_RELAUNCHED`, `ALLOW_UNSAFE_NODE`, `NO_RELAUNCH`) | 7 | Gone with WASM | +| Worker and pool sizing | 8 | Auto-detected from cgroups and load; keep one override | +| Daemon and watchdog timeouts | 12 | Keep as a typed config struct, most never set | +| WAL valve and fast-init | 5 | Keep until the store changes | +| Explore debug and ranking toggles | 7 | Debug flags only, not user settings | +| Paths and host identity | 9 | Keep | +| Install and hook | 8 | Keep | + +The knob count is a symptom of one-off incident fixes, not of the language. A rewrite that keeps the same incident history would grow the same knobs. + +## 2. What the Rust core would look like + +A workspace of crates behind one C ABI and one napi binding: + +``` +codegraph-core/ + crates/ + graph-model NodeKind, EdgeKind, ids, provenance, extraction version + store SQLite (rusqlite, bundled, FTS5), schema, migrations, WAL valve, writer lock + extract tree-sitter walkers for 29 grammars + region slicers for SFC/Markdown/CFML + resolve import resolver, binding model, name matcher, framework resolvers + synth synthesis passes over the stored graph + derive traversal, symbol lookup, flow, hierarchy, dead code, branch guards + retrieve explore ranking, budgets, allocation, formatting + sync notify-based watcher, scope from gitignore, debounce, git hooks + daemon socket server, query pool, watchdogs, registry + mcp JSON-RPC 2.0 over stdio and socket, tool schemas, instructions + cli clap front end + bindings/napi thin surface for the viewer server and installer, which stay in TypeScript +``` + +Design choices that would differ from today: + +- **One extractor per language, no fallback.** Error recovery is whatever the native parser produces. Golden dumps pin it. +- **Bindings as data, not regex.** Extraction emits a per-file binding table (name, kind, target, exported, re-exported-from). Resolution consumes it and never rescans raw source. This is the same model the resolution plan proposes in place. +- **Typed config.** One `Config` struct loaded from `codegraph.json` and a short env allowlist. No ad-hoc `process.env` reads inside modules. +- **Store as the parallelism boundary.** Parse and resolve on rayon; a single writer thread owns the connection; workers send batches over a channel. This is what the current TypeScript pipeline approximates with worker threads and a store worker. +- **Single binary.** CLI, daemon, MCP server, and indexer in one static executable. No Node runtime in the bundle, no `--liftoff-only`, no version gate. + +## 3. What it would buy + +- Fresh-index wall time: the kernel spike measured a 4.4x to 14x parse walk over WASM, and the migration plan's Linux-kernel run puts resolution at 73% of wall. A native resolve with a proper binding table is the remaining large win. Realistic target on the Linux kernel: under 8 minutes on 8 cores against 14.8 today. +- Memory: no per-worker grammar copies, no V8 heap per worker. The Bun probe in the espn-draft research showed how much of today's footprint is runtime, not data. +- Distribution: one binary per platform. The npm thin installer stays as a shim. +- Bugs of the class "handle left open on Windows teardown" and "orphan daemon per test run" become type-system and RAII concerns instead of discipline. + +## 4. What it would cost + +Measured against the repository as of 2026-09-11. + +| Item | Size | Note | +|---|---|---| +| Engine TypeScript to replace | 117k lines | Of which extraction 27k, resolution 30k, MCP 15k | +| Tests encoding fixed incidents | 101k lines | Many anchor to PR numbers; the coverage is the spec | +| Rust already written | 26k lines | Extraction only, reusable as-is | +| Upstream velocity | ~90 first-parent commits/month | 154 of the last 400 are fixes | +| Fork divergence | 166 commits ahead, 13 upstream PRs open | Flow of accuracy fixes in both directions | + +- **The heuristics are the product.** Every framework resolver and synthesizer exists because an agent A/B showed a flow breaking without it. A rewrite re-validates all of them or regresses answers silently, and the repository rule is that a half-bridged flow is worse than none. +- **The upstream relationship ends.** A separate codebase cannot merge upstream fixes or contribute back. Five of the fork's fixes in the last week landed upstream as maintainer re-lands; that channel closes. +- **Time to parity is long.** With the kernel as a head start, extraction is done. Resolution, synthesis, retrieval, daemon, sync, and MCP are each multi-week efforts with a byte-parity bar, done serially because each consumes the previous one's output. +- **Retrieval quality does not move.** The espn-draft audit log shows the organic loss is in ranking and trimming inside explore, at a median 228 ms latency. That code is pure logic and gains nothing from Rust. + +## 5. Decision + +Do not start a greenfield core now. The two incremental plans capture nearly all of the performance and simplification benefit while keeping the tests, the heuristics, and the upstream channel: + +1. [Kernel-only extraction](kernel-only-extraction-plan.md) removes the WASM runtime, the second grammar supply chain, the V8 workarounds, and about 27k lines of TypeScript, and makes the kernel the single producer. +2. [Resolution binding model](resolution-binding-model-plan.md) replaces the three-place export logic with a binding table emitted by extraction, ends the over-matching fix cycle, and is the natural first resolution module to port into the kernel. + +Re-open this sketch when all of the following hold: + +- Both plans have landed and the TypeScript engine is under roughly 40k lines. +- Resolution and synthesis run in the kernel with golden-dump parity. +- A concrete need exists that the Node shell blocks: a single static binary for a host with no Node, or a memory ceiling the daemon cannot meet. + +At that point the crate is already the core, and "greenfield" becomes a repository split, not a rewrite. diff --git a/docs/design/kernel-only-extraction-plan.md b/docs/design/kernel-only-extraction-plan.md new file mode 100644 index 000000000..b6cc768bc --- /dev/null +++ b/docs/design/kernel-only-extraction-plan.md @@ -0,0 +1,155 @@ +# Kernel-only extraction — removing the WASM path + +**Status:** plan, not started. Written 2026-09-11. Companion to [resolution-binding-model-plan.md](resolution-binding-model-plan.md) and [greenfield-rust-core-sketch.md](greenfield-rust-core-sketch.md). Supersedes the "coexistence is permanent" stance in [rust-kernel-migration-plan.md](rust-kernel-migration-plan.md) §4d once approved. + +**Goal:** the Rust kernel (`codegraph-kernel/`) is the only parser and extractor. `web-tree-sitter`, the vendored `.wasm` grammars, the per-language routing table, the error-file deferral, the V8 `--liftoff-only` relaunch, and the Node 25 block are all removed. + +**Why:** the WASM path is no longer a fallback in practice. It is the mandatory path for every Vue, Svelte, Astro and Razor file, for nine tail languages, for three read-time features, and for every file whose parse tree has an error. It costs a second grammar supply chain pinned in lockstep with Cargo.toml, a runtime relaunch that forces a dedicated Node 24 alias on the host, a per-worker grammar heap that made Bun unusable, and about 27k lines of TypeScript. The kernel already produces byte-identical graphs for 20 languages. + +## 1. What the WASM path does today + +Facts as of `f82e4898`. + +| Consumer | Where | Why it needs WASM | +|---|---|---| +| 9 tail languages | `src/extraction/languages/{objc,arkts,pascal,vbnet,cobol,erlang,nix,terraform,solidity}.ts` and the three CFML grammars | No Rust walker | +| SFC extractors | `vue-extractor.ts`, `svelte-extractor.ts`, `astro-extractor.ts`, `razor-extractor.ts` | Slice the file, then construct `TreeSitterExtractor` directly on the block; routed before the kernel branch in `tree-sitter.ts:7321-7376` | +| CFML extractor | `cfml-extractor.ts` | Walks a live CST from `getParser('cfml')` | +| Error-file deferral | `kernel/index.ts:239,320`, `stack.rs` | Kernel throws `defer:`; WASM re-parses. Also the stack-overflow guard | +| Viewer highlighting | `extraction/syntax-tokens.ts` → `ui-server/highlight/index.ts` | Parses a live tree at read time | +| Branch guards (`WHEN` labels) | `graph/branch-guards.ts` | Parses a live tree at query time | +| Explore source ranges | `mcp/explore-source-ranges.ts` | Parses a live tree at query time | +| Parity oracle | 15 `kernel-*-parity` and grammar tests, `scripts/kernel-parity.mjs` | Two-arm compare | +| Loader fail-soft | `kernel/loader.ts` | Any load failure returns null and degrades to WASM | +| Release degradation | `scripts/build-bundle.sh:80-97` | Missing prebuild prints a warning and ships WASM-only | + +The kernel exposes `extract_file`, `contract_info`, `grammar_info`, `cfnptr_scan_files`, `cfnptr_strip_c`. It has no parse-tree API. That is the single largest gap. + +## 2. Design decisions + +### 2.1 Native error recovery becomes canonical + +WASM was declared canonical on files with ERROR nodes only so that parity held by construction (migration plan §4b). The divergence is the UTF-16 versus UTF-8 encoding the parser sees; neither recovery is better. The C/C++ deferral round showed the kernel's own numbers improving as deferral dropped, and `CODEGRAPH_KERNEL_CCPP_ERROR_EXTRACT=1` already extracts from erroring trees natively. Decision: the kernel extracts every file it parses. `EXTRACTION_VERSION` bumps once, and the golden dumps are re-baselined against the kernel. + +The stack-overflow guard in `stack.rs` keeps its defer semantics but the consumer changes: a deferred file is stored with an `errors` entry and zero symbols, the same as a file that fails to parse today. + +### 2.2 The kernel gains a region API, not an SFC parser + +Vue, Svelte, Astro and Razor slicing stays in TypeScript for now. Each already computes `(content, language, lineOffset)` per block and today constructs a `TreeSitterExtractor` on it. They call `extract_file` on the block instead, then rebase positions exactly as they do now. Razor's synthetic `class __RazorCode__ { ... }` wrapper is unchanged. This is a call-site substitution, not a port. + +Moving the slicers themselves into Rust is a later optimization and is not required to delete WASM. + +### 2.3 The kernel gains a parse-tree service for the three read-time consumers + +Add a `parse_tokens(file, content, language)` entry point that returns the flat token stream `syntax-tokens.ts` derives today, and a `walk_guards(file, content, language, ranges)` entry point that returns the guard conditions `branch-guards.ts` computes. `explore-source-ranges.ts` needs only node boundaries and folds into the tokens call. All three return flat buffers through the existing layout mechanism. + +Alternative considered: expose a generic tree handle over napi. Rejected. Per-node boundary crossings are exactly the cost the kernel exists to avoid, and the three consumers need three fixed derivations, not a tree. + +### 2.4 Tail languages: port five, drop four, decide CFML separately + +| Language | Decision | Basis | +|---|---|---| +| Objective-C | Port | 181-line extractor, `tree-sitter-objc` on crates.io, feeds the Swift/ObjC bridge resolver with tests | +| Erlang | Port | 384 lines, tests for arity and behaviour synthesizers, `tree-sitter-erlang` matches the ELP lineage we vendor | +| Nix | Port | 324 lines, option synthesizer test, crate exists | +| Pascal + DFM | Port | 72 lines plus the parser-free DFM extractor, 13 changelog entries show real users | +| Solidity | Port | 282 lines, crate exists, low risk | +| ArkTS | Drop unless a user asks | Crate lineage differs from our harmony-contrib fork; one test file | +| Terraform | Drop unless a user asks | No `tree-sitter-terraform` crate; `tree-sitter-hcl` would need re-validation; no tests | +| VB.NET | Drop | No crate, patched grammar with a C scanner, no tests, no changelog | +| COBOL | Drop | 16 MB vendored fork grammar, no tests; copybook logic would need a port | +| CFML (3 grammars) | Decide with the maintainer | Own grammars for cfquery and cfscript; 508-line extractor walks a live CST; two test files | + +"Drop" means the language is removed from `EXTENSION_MAP` and the README table in the same change, with a changelog entry. It does not mean silently unindexed. + +### 2.5 The loader fails hard + +`kernel/loader.ts` returns null today on missing binary, ABI mismatch or kind-table drift. After this plan a load failure is a fatal startup error with the platform and the expected path in the message. `build-bundle.sh` errors instead of warning when a prebuild is absent. `CODEGRAPH_KERNEL`, `CODEGRAPH_KERNEL_LANGS`, `CODEGRAPH_KERNEL_EXPECT` and `CODEGRAPH_KERNEL_CCPP_ERROR_EXTRACT` are removed. `CODEGRAPH_KERNEL_PATH` stays for source development. + +### 2.6 Platform support is explicit + +Release matrix today: macOS x64 and arm64, Linux glibc x64 and arm64, Windows x64 and arm64. Add `x86_64-unknown-linux-musl` and `aarch64-unknown-linux-musl` so Alpine containers work. Every other platform is unsupported and says so at install time. Source checkouts need a Rust toolchain; `npm test` builds the kernel in `pretest` if the prebuild for the host is missing. + +### 2.7 Golden dumps replace the parity oracle + +`scripts/dump-graph.mjs` already produces a natural-key sorted dump. Check in dumps for a fixed set of fixture repos at pinned commits, one per language family, and add `__tests__/golden-dumps.test.ts` that re-indexes each fixture and compares. This is the regression gate for every extractor change from here on. It replaces the 15 parity test files, which delete. + +## 3. Phases + +Each phase lands on `fork/consolidated` behind the golden-dump gate and leaves the tree shippable. Phases 1 through 3 can proceed in parallel; 4 and 5 depend on all of them. + +### Phase 0: golden-dump gate + +- Pick fixtures: one small repo per language family plus the espn-draft-shaped Vue fixture and a Markdown-heavy fixture. +- Check in dumps generated on the current kernel-routed path. +- Add the test. Wire it into CI on the linux-x64 prebuild job. + +Exit: the gate is green on `fork/consolidated` with no extraction change. + +### Phase 1: error recovery flip + +- Remove the `defer:` throw for parse errors in every walker; keep it for the stack guard. +- Delete `takeDeferredPreParse` and the one-slot memo. +- Bump `EXTRACTION_VERSION`. Re-baseline the golden dumps and record the node and edge deltas per fixture in this document. + +Exit: no file reaches WASM because of an ERROR node. Deferral rate in `scripts/kernel-parity.mjs --max-deferral` reads zero for every routed language. + +### Phase 2: SFC extractors call the kernel + +- Replace `new TreeSitterExtractor(...)` in the four SFC extractors with `extract_file` on the block, then the existing position rebase. +- Add SFC fixtures to the golden set. + +Exit: the espn-draft fixture indexes with zero WASM parses (instrument `getParser` to throw under a test flag). + +### Phase 3: parse-tree service + +- Add `parse_tokens` and `walk_guards` to the kernel with layouts in `layout.ts`. +- Port `syntax-tokens.ts`, `branch-guards.ts` and `explore-source-ranges.ts` to consume them. +- Extend the viewer highlighting parity test (`cg57-highlighting-parity`) to run against the kernel tokens. + +Exit: no `getParser` call outside `src/extraction/`. + +### Phase 4: tail languages + +- Port Objective-C, Erlang, Nix, Pascal, Solidity in that order, each with its own checklist file following the existing `*-kernel-port-checklist.md` pattern and a golden fixture. +- Remove ArkTS, Terraform, VB.NET and COBOL from `EXTENSION_MAP`, the README table, and `grammars.ts`, with a changelog entry under Breaking Changes. +- CFML: separate decision recorded here before Phase 5 starts. + +Exit: every language in `EXTENSION_MAP` has a kernel walker. + +### Phase 5: delete WASM + +- Remove `web-tree-sitter` and `tree-sitter-wasms` from `package.json`, `src/extraction/wasm/`, `copy-assets`, `grammars.ts`, `tree-sitter.ts`'s WASM branch, `TreeSitterExtractor`, `parse-worker.ts`'s Emscripten stderr filter and OOM exit, `resetParser`. +- Remove `wasm-runtime-flags.ts` except `NODE_RUNTIME_FLAGS`, the relaunch in `bin/codegraph.ts`, `command-supervision.ts` if it has no other purpose, the liftoff lines in `npm-shim.js` and `build-bundle.sh`, and the Node 25 block in `node-version-check.ts`. +- Make the loader fatal. Make `build-bundle.sh` fatal on a missing prebuild. Add the musl targets. +- Delete the 15 parity tests, `kernel-parity.mjs`, and the wasm-flag tests. Rewrite `kernel-scaffold.test.ts` for the new semantics. Re-check the four MCP orphan tests that depended on the re-exec process shape. +- Update `server-instructions.ts` only if the language list changes. Update `AGENTS.md` build notes and the `copy-assets` rule. + +Exit: `grep -r web-tree-sitter src __tests__ scripts` is empty. Full suite green on Linux, Windows and macOS. The espn-draft host no longer needs the Node 24 alias. + +## 4. What is removed, by the numbers + +| Item | Lines or count | +|---|---| +| TypeScript language extractors (`languages/*.ts`) | ~25k, minus `c-cpp.ts` (1,793) which stays for preParse blanking | +| `TreeSitterExtractor` and WASM branch in `tree-sitter.ts` | most of 7.4k | +| Vendored `.wasm` grammars | 30 files | +| Parity and WASM tests | 15 files | +| Env vars | 6 | +| Runtime dependencies | 2 of 10 | +| Node runtime constraints | Node 25 block, `--liftoff-only` relaunch, dedicated Node alias on hosts | + +What is added: about 5 Rust walkers for tail languages, two kernel entry points, two musl targets, the golden-dump test, and a `pretest` kernel build for source checkouts. + +## 5. Risks + +- **Error-recovery re-baseline changes graphs.** Expected and accepted. Phase 1 records per-fixture deltas. A regression in a fixture's symbol count is a walker bug to fix, not a reason to keep WASM. +- **Source-checkout DX.** Contributors need cargo. Mitigation: `pretest` builds only when the prebuild is missing, and CI publishes prebuilds on every `fork/consolidated` push so most contributors never compile. +- **Upstream divergence.** Upstream keeps coexistence. This fork's graph output stays byte-identical to upstream's kernel path for the 20 routed languages, so upstream resolution fixes still merge. Extraction changes upstream makes to WASM-only languages will not apply; that is the cost of dropping them. +- **Read-time consumers.** Phase 3 is the least-explored piece. If `walk_guards` turns out to need a general tree, fall back to a narrow node-cursor API scoped to one file, not a global tree handle. +- **Deferred files.** A stack-overflow defer now yields an empty file instead of a WASM parse. Incidence is one known file in clang. + +## 6. Settings after this plan + +Removed: `CODEGRAPH_KERNEL`, `CODEGRAPH_KERNEL_LANGS`, `CODEGRAPH_KERNEL_EXPECT`, `CODEGRAPH_KERNEL_CCPP_ERROR_EXTRACT`, `CODEGRAPH_WASM_RELAUNCHED`, `CODEGRAPH_ALLOW_UNSAFE_NODE`, `CODEGRAPH_NO_RELAUNCH`. +Kept: `CODEGRAPH_KERNEL_PATH` (source dev), `CODEGRAPH_KERNEL_DEBUG`, `CODEGRAPH_KERNEL_CFNPTR`, `CODEGRAPH_PARSE_WORKERS`. diff --git a/docs/design/resolution-binding-model-plan.md b/docs/design/resolution-binding-model-plan.md new file mode 100644 index 000000000..faf1c88b1 --- /dev/null +++ b/docs/design/resolution-binding-model-plan.md @@ -0,0 +1,155 @@ +# Resolution binding model — one source of truth for exports and bindings + +**Status:** plan, not started. Written 2026-09-11. Companion to [kernel-only-extraction-plan.md](kernel-only-extraction-plan.md) (which should land first, so there is one extractor to emit the new facts) and [greenfield-rust-core-sketch.md](greenfield-rust-core-sketch.md). Closes upstream issue #1721 and ends the fix cycle behind #1566, #1790, #1794 and #1844. + +**Goal:** extraction emits a per-file binding table. Resolution consumes it and never rescans raw source to answer "is X exported", "what does N bind to in F", or "is this receiver a known thing". Every resolver predicate that reads source today is replaced by a lookup. + +## 1. The problem, with the evidence + +### 1.1 Five regex families answer one question + +"Is X exported" is decided in five places, none shared: + +| Site | Mechanism | Blind spot | +|---|---|---| +| `languages/typescript.ts:110`, `javascript.ts:68` | AST `export_statement` ancestor walk | `const x; export { x }`, all CommonJS | +| `tree-sitter.ts:2451` mirrored in `codegraph-kernel/src/tsjs/extractors.rs:359` | Regex over file source (`isExportedLater`) | Scoped to Zustand-style stores only; the two hand-mirrored regexes named in #1721 | +| `name-matcher.ts:672` (`isSealedModule`) | Three regexes over comment-stripped source plus a raw-source CommonJS check | Per-file "exports nothing", added by #1720/#1746 | +| `import-resolver.ts:96` (`DEFAULT_EXPORT_BINDING_RE`) | Regex for `export default NAME` | Patches `isExported` inside `FileExportIndex` | +| `alias-binding.ts:105` (`extractLocalExportAliases`) | Regex for `export { X as Y }` | Not shared with any of the above | + +The only persisted fact is `nodes.isExported`. `NodeKind` declares `export` and `EdgeKind` declares `exports`, but no extractor emits either. The export side of the schema exists on paper. + +### 1.2 "What does N bind to in F" is answered by per-reference regexes at resolution time + +`isBoundToBareImport` (`name-matcher.ts:551`), `isBareJsCall` (`:895`) and `isLocallyBoundJsName` (`:932`) each build `RegExp`s over the whole file source per name, memoized per resolution context. `isStaticCFunction` reads C source lines because the extractor records no storage class. The receiver-type inference table (`:1950-2082`) is 40 languages of regex over source. + +### 1.3 The fix cycle + +Each PR in the chain added a new source-reading predicate, narrowed by a corpus measurement, at a different point in the pipeline: + +- #1790 fixed #1566 by dropping every nested identifier-rooted call in both engines. Over-fix. #1794 reported 8 red tests. +- #1844 restores the receiver text and adds `isTsJsNestedCall` as an early-out so nested calls keep only framework evidence. +- #1710 tried the general "drop every unnameable receiver" rule, lost 64 correct edges, and shipped a host-global list instead (#1766). #1790 reversed that judgment; #1844 restores it. +- #1709 filtered the fuzzy candidate set by lexical reachability and manufactured uniqueness (+59 wrong on vite). #1718 moved the check onto the single survivor. The rule "reachability may reject a unique guess, never manufacture one" now lives in a comment at `name-matcher.ts:3220`. +- #1720/#1746 learned that sealed-module status must filter candidates for `imports` ranking but only reject the winner for calls, because removing from a crowd promotes a runner-up. +- #1759 made Zustand store bindings look like local shadows; #1762/#1763 carved them back out. +- #1767 showed that 3,310 edges classified as `exact-match` or `fuzzy` on one repo were really failed import resolution (`.js` specifier to `.ts` source). + +The recurring shape: resolution guesses at binding facts from source text, each guess is a separate predicate, and the predicates interact through candidate-set ordering. Every fix is a new predicate. + +### 1.4 What it costs + +The migration plan's §7a numbers put resolution at 73% of Linux-kernel wall. `settle` (where these predicates run) is 3.6 s at 8 cores after the pool absorbs it, but the predicates also force `readFileCached` on every candidate file and hold source in the worker lane. The larger cost is correctness: the eval runner scores recall only (`__tests__/evaluation/scoring.ts`), so precision regressions are found by ad-hoc corpus runs in PR bodies. + +## 2. Design + +### 2.1 A binding table emitted by extraction + +Extraction emits, per file, a list of bindings. Persisted in a new table: + +``` +bindings( + file_path TEXT NOT NULL, + name TEXT NOT NULL, -- local name as written in this file + kind TEXT NOT NULL, -- 'decl' | 'import' | 'reexport' | 'alias' | 'param' | 'local' + node_id TEXT, -- the declaring node when kind='decl' or 'alias' + target_spec TEXT, -- import/re-export specifier as written + target_name TEXT, -- imported/re-exported name (or '*' / 'default') + exported_as TEXT, -- NULL if not exported; the export name otherwise + export_form TEXT, -- 'esm' | 'esm-later' | 'esm-default' | 'cjs' | 'cjs-object' | 'public' | ... + scope_start INTEGER, scope_end INTEGER, -- line range the binding is visible in + storage TEXT -- language-specific visibility: 'static', 'private', 'pub(crate)', ... +) +``` + +Rules: + +- One row per (file, name, scope). A re-declared name in a nested scope is a separate row with its own scope range. +- `exported_as` is the single answer to "is X exported". The five regex families collapse into the extractor emitting the right `export_form`. +- Per-file "exports nothing" is `NOT EXISTS (SELECT 1 FROM bindings WHERE file_path=? AND exported_as IS NOT NULL)`. `isSealedModule` deletes. +- `kind='import'` with `target_spec` not resolvable to a project file is a bare import. `isBoundToBareImport` becomes a lookup. +- `storage` carries C `static`, Rust visibility, Java/Kotlin/C# access modifiers. `isStaticCFunction` and the `PRIVATE_IS_FILE_LOCAL` table become one predicate over one column. +- `nodes.isExported` stays as a denormalized convenience and is set from the table at store time. `EdgeKind` `exports` starts being emitted, from `bindings` rows with `exported_as`, so `context/index.ts` stops being the sole consumer of a kind nothing produces. + +### 2.2 Resolution reads bindings, not source + +`ResolutionContext` gains `bindingsFor(file)` returning the file's rows, indexed by name and by scope. The strategy ladder in `resolveOneInner` (`index.ts:900`) becomes: + +1. Framework resolvers (unchanged, still first). +2. **Binding lookup**: for reference `N` at line `L` in file `F`, find the innermost `bindings` row for `N` whose scope contains `L`. + - `decl` or `alias` with `node_id`: resolved. Confidence 1.0. + - `import` with a project-resolvable `target_spec`: follow to the target file's `exported_as = target_name` row. Resolved by import. This is `resolveViaImport` with the regex import extractors removed. + - `import` with a bare specifier: unresolved, `status='failed'`, reason `external`. Never falls through to name matching. This is #1713 as a rule instead of a predicate. + - `local` or `param`: unresolved, reason `local`. Never falls through. This subsumes `isLocallyBoundJsName` and the #1762 carve-outs, because a Zustand store accessor is a `decl` row, not a `local` row. +3. **Cross-file name match** only when no binding row exists for `N` in `F`. Candidates are `bindings` rows across the project with `name = N` and `exported_as IS NOT NULL` (or `storage` visible across files), joined to nodes. `isVisibleAcrossFiles`, `isCrossFileReachable`, `isLexicallyReachable` become a single visibility predicate over the row. +4. Fuzzy match keeps the #1718 rule, expressed as: fuzzy may reject the unique candidate on visibility, never filter the set. + +Receiver-typed calls (`a.b()`, `this.x()`) keep `matchMethodCall` and the chain matchers, but receiver identity comes from the binding row for `a` (a `decl` with a `node_id` whose node has a type, or an `import` whose target is a class) before the regex inference table is consulted. The inference table shrinks as languages emit typed bindings. + +### 2.3 Where the receiver-guessing rule lands + +The #1566 / #1790 / #1844 question, "should `values.get()` bind to the sole project `get`", has a fixed answer under this model: only if `values` has a binding row whose target is a project node with a `get` member. An unbound receiver yields unresolved with reason `unknown-receiver`. There is no builtin list and no host-global list; `js-builtins.ts` becomes documentation of why `window` and `document` never have binding rows. + +### 2.4 One extractor + +The binding table is emitted by the kernel walkers only. This is why the kernel-only plan lands first. Emitting it twice, in TypeScript and Rust, would recreate the mirrored-regex trap that #1721 describes. + +## 3. Phases + +### Phase 0: precision in the eval runner + +- Add a precision score to `__tests__/evaluation/scoring.ts`: for a fixed corpus (vite, vitest, svelte, rollup, the ones the PR bodies used), a checked-in list of known-wrong edges that must stay absent and known-right edges that must stay present. This is the gate every later phase runs against. + +Exit: the LOST/GAINED tables from #1713, #1718, #1746 and #1844 are encoded as tests. + +### Phase 1: emit bindings for TS/JS + +- Add the `bindings` table and migration (DDL only, empty until re-index). +- Kernel `tsjs` walker emits rows for declarations, imports, re-exports, aliases, parameters, and block-scoped locals, with `export_form` covering ESM, later-export, default, `module.exports.X`, `module.exports = {}`, and `exports["x"]`. +- Set `nodes.isExported` from the table. Delete `isExportedLater` in both engines. +- Bump `EXTRACTION_VERSION`. + +Exit: `store-exported-later.test.ts` and `commonjs-exports.test.ts` pass against the table, not the regexes. + +### Phase 2: resolution reads bindings for TS/JS + +- Add `bindingsFor` to the context. Replace `isBoundToBareImport`, `isBareJsCall`, `isLocallyBoundJsName`, `isSealedModule`, `DEFAULT_EXPORT_BINDING_RE`, `extractLocalExportAliases`, `extractJSImports` and `extractReExports` with lookups. +- Replace `isTsJsNestedCall` and the host-global list with the receiver rule in §2.3. +- Run the Phase 0 precision corpus and `resolution.test.ts`. + +Exit: no `RegExp` construction over file source remains in `name-matcher.ts` or `import-resolver.ts` for TS/JS. The 8 tests from #1794 and the store tests from #1762 pass with no carve-outs. + +### Phase 3: other languages + +Per language, in order of resolver regex weight: Python, Go, Java/Kotlin, C/C++ (`storage='static'`), Rust (visibility), PHP, Ruby, C#, Swift. Each phase deletes that language's import-extractor regex and its rows in the receiver inference table. + +Exit per language: its `extractXImports` function and inference table entries are deleted. + +### Phase 4: move the binding lookup into the kernel + +With source-reading predicates gone, the resolve step is a join over `bindings`, `nodes` and `unresolved_refs`. Port it into the kernel as a batch entry point that takes a chunk of refs and returns resolved edges, mirroring today's `resolver-worker` chunk contract. The TypeScript `ReferenceResolver` becomes the orchestrator over the kernel and the framework resolvers. This is the P1 item in the migration plan, executed after the model is stable rather than before. + +Exit: `settle` and `read` stages run natively; Linux-kernel resolution under the §7a target. + +## 4. What is removed + +| Item | Location | +|---|---| +| `isExportedLater` (both engines) | `tree-sitter.ts:2451`, `extractors.rs:359` | +| `isSealedModule` and its three regexes | `name-matcher.ts:621-692` | +| `isBoundToBareImport`, `isBareJsCall`, `isLocallyBoundJsName` and their memos | `name-matcher.ts:551-970` | +| `isTsJsNestedCall` early-out and the host-global chain gate | `index.ts:1030`, `name-matcher.ts:3423`, `js-builtins.ts` | +| `DEFAULT_EXPORT_BINDING_RE`, `extractLocalExportAliases` | `import-resolver.ts:96`, `alias-binding.ts:105` | +| Per-language import regex extractors | `import-resolver.ts:898-1174` | +| Receiver inference regex table, as languages migrate | `name-matcher.ts:1950-2082` | +| `strip-comments.ts` once no resolver reads source | 574 lines | + +Estimated: 4k to 6k lines of `src/resolution/` deleted, replaced by one table, one context accessor, and walker emission code in the kernel. + +## 5. Risks + +- **Re-index required.** The table is empty until re-index; readers must union with `nodes.isExported` until then, matching the existing DDL-only migration rule. +- **Emission completeness.** A binding the walker misses becomes an unresolved reference, not a wrong edge. That is the failure direction the maintainer asked for in #1566 and every PR since. Track "unresolved with reason" counts per corpus so misses are visible. +- **Framework resolvers.** They keep their own evidence and run first. Some read source today (`extract` hooks); those are out of scope here but should migrate to bindings when they touch a language that has them. +- **Upstream.** The table and the phase-1 walker change are contributable upstream as the resolution of #1721. Phases 2 onward diverge from upstream's regex predicates and will need care when merging their resolution fixes; most of those fixes become no-ops once the predicate they patch is gone.