test(eval): precision scoring and a pinned-corpus edge gate for resolution - #27
Merged
Merged
Conversation
…ution Phase 0 of docs/design/resolution-binding-model-plan.md. The evaluation runner scored recall only, so a wrong edge — the thing the resolution PR chain (colbymchenry#1713, colbymchenry#1718, colbymchenry#1746, colbymchenry#1844) exists to remove — was invisible to it. scoring.ts gains scoreEdgeCase: an EdgeCase names a kind, a target endpoint (file suffix + symbol) and optionally a source, and expects the edge absent (known-wrong) or present (control). evaluation-edge-scoring.test.ts pins it on a synthetic project with the bare-import and sealed-module shapes and proves it flags a violated case and a missing endpoint. edge-cases.ts encodes the edges the PR bodies named on the commits they measured (full SHAs): vite 8492422b (colbymchenry#1713 self-import + getEnv, colbymchenry#1718 two fuzzy nested calls, colbymchenry#1746 sealed defineConfig -> test-stacktrace.js vite, one relative-import control), vitest 7c818153 (evaluatedModules), svelte 5895c637 (bundle). npm run eval:precision -- <corpus> fetches the pinned commit, indexes, scores, prints the resolved-edge histogram by resolver (the LOST/GAINED methodology) and writes a report; the three baseline reports at codegraph 1498c2f are committed: vite 28,893 edges / fuzzy 16, vitest 75,035 / 28, svelte 70,432 / 159; every absent case held, the control held.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Phase 0 of
docs/design/resolution-binding-model-plan.md: the evaluation runner can now see a wrong edge.scoreEdgeCasein__tests__/evaluation/scoring.ts: an edge case names a kind, a target endpoint and optionally a source, and expects the edge absent (known-wrong) or present (control). A synthetic unit test pins the scorer on the fix(resolution): a name bound to a bare import resolves to no project node colbymchenry/codegraph#1713 bare-import and fix(resolution): a binding in a module that exports nothing is not a cross-file candidate (#1719) colbymchenry/codegraph#1746 sealed-module shapes, including a deliberately violated case and a missing endpoint.__tests__/evaluation/edge-cases.ts: the edges the resolution PR bodies named, on the exact commits they measured (vite, vitest, svelte). Rollup's table listed counts only, so it has no cases yet; fix(extraction): retain nested call evidence without guessed targets colbymchenry/codegraph#1844 is already pinned by an existing test.npm run eval:precision -- <corpus>fetches the pinned commit, indexes it, scores the cases, prints the resolved-edge histogram by resolver, and writes a report. The three baseline reports are committed.Baseline (codegraph
1498c2ff)Verification
evaluation-edge-scoring.test.ts: 5 tests pass.npx tsc --noEmitclean;npm run check:agent-docspasses.