fix(sessions): invalidate exact Cursor parent rewrites

[ScriptedAlchemy]

Summary

• invalidate Cursor parent-dispatch caches on exact-mtime, same-length rewrites
• detect same-path file replacement with stable opened-handle identity on Windows
• preserve append-only delta parsing while replacing the trailing-anchor heuristic with full verified-prefix evidence

Motivation

Fixes #928. Cursor parent-dispatch caches can retain stale models when content is rewritten at the same length and timestamp, or when a file is replaced at the same path.

Changes

• bind identity, metadata, prefix validation, parsing, and revalidation to one opened file handle
• reuse canonical JSONL native identity, change-token, and resume-digest primitives
• content-validate every cached prefix before serving it; metadata timestamps are never the sole rewrite witness
• clear stale model entries before rescanning rewritten content from byte zero
• preserve exact length and mtime in replacement/rewrite fixtures
• cover rewrites whose trailing 4 KiB remains unchanged and unchanged reuse after append

Test plan

• cargo test -p tracedecay-sessions parent_dispatch_index::tests -- --test-threads=1 (17/17; repeated)
• cargo clippy -p tracedecay-sessions --all-targets -- -D warnings
• cargo check -p tracedecay-sessions --target x86_64-pc-windows-gnu
• cargo bench -p tracedecay-sessions --bench cursor_dispatch_model
• cargo test -p tracedecay-sessions (655 passed; five environment-sensitive failures outside the parent-dispatch cache: three existing JSONL same-handle timestamp tests and two Codex frontier tests)
• Windows CI passes without timing sleeps

Checklist

• CHANGELOG.md updated (not required for this scoped regression fix)
• No secrets, credentials, or .env files included
• Breaking changes documented (none)

  

[changeset-bot]

⚠️ No Changeset found

Latest commit: c303b3a

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

[ScriptedAlchemy]

Review against tip f81c9f9. Verdict: needs-fix (correctness by deleting the optimisation the module exists for).

1. hosts/cursor/parent_dispatch_index.rs:213: jsonl_prefix_digest(file, verified_cursor) seeks to 0 and hashes the whole verified prefix on every lookup, and LookupPlan::ServeCached (the zero-I/O path) is gone. The module doc (:3-6) says it exists to stop re-reading parent transcripts from byte zero per subagent batch; this makes every hit O(file).
2. parent_dispatch_index.rs:76-80: ParentFileRevision.change (the JsonlFileChangeToken, which on Unix carries ctime, the field that actually catches a same-length same-mtime rewrite) is stored but never compared in plan_lookup; only identity and len are. The lazy fix: if entry.revision == revision { serve cached } and full-prefix digest only when the token differs (or only on Windows, where the token is last_write_time alone).
3. parent_dispatch_index.rs:263-266: if final_revision != scan.revision { forget; return None } discards a whole scan when the parent is appended to during it, which is the normal case for a live Cursor session, and the next call rescans from zero and can lose the race again. Commit what was verified up to delta.verified_cursor; the next pass picks up the delta.
4. parent_dispatch_index.rs:~580: TEST_ANCHOR_WINDOW_BYTES names a concept the production code just deleted. Delete or rename.
5. Test plan unchecked, including clippy.

Sequencing: after #948 (both in tracedecay-sessions; this rewrites 463 lines that #948 never touches).

[ScriptedAlchemy]

Review findings applied on top of 0f0cd423c, one commit:

• 43d5fc44d fix(sessions): restore the zero-IO Cursor parent cache hit

1 + 2 (digest on every lookup, revision.change never compared). plan_lookup now answers from the stored ParentFileRevision first: when identity, length and the change token are all unchanged and the entry covers the whole file (verified_cursor == revision.len), the cached model is served with no file reads and jsonl_prefix_digest never runs. The digest stays as the fallback for a changed token — the only case it can still tell apart — which is also what covers Windows, where the token is last_write_time alone. The verified_cursor == revision.len half of the guard is load-bearing, not belt-and-braces: it is what stops a scan that was appended to mid-flight (finding 3) from being served as complete.

3 (whole scan discarded on a mid-scan append). Replaced with scanned_prefix_survives(scanned, current): same identity and no truncation. A live parent appended to during the read now commits everything through delta.verified_cursor and the next pass reads only the delta, instead of going back to byte zero and racing again. Only a replaced or truncated file invalidates the scan.

4. TEST_ANCHOR_WINDOW_BYTES -> TEST_UNCHANGED_TAIL_BYTES.

5. Gates run, results below.

One addition beyond the findings, needed to make finding 1 testable: prefix-validation reads were invisible to DispatchScanReceipt, so no test could tell a true cache hit from a full re-hash. They are now charged to the receipt as prefix_digest_bytes (its own field, so bytes_parsed keeps meaning parsed record bytes and append_only_growth_parses_the_delta_and_resolves_late_dispatch is unchanged), with a matching gauge. unchanged_parent_repeated_misses_parse_bytes_once now asserts it is zero on every repeat, so it fails if the fast path stops serving without I/O.

Two new tests, appending_during_a_scan_keeps_the_verified_prefix and replacing_or_truncating_during_a_scan_discards_it, drive scanned_prefix_survives over real files (append, truncate, replace). They exercise the predicate rather than the live race: interleaving a write into the middle of a scan is not reachable deterministically without adding a test hook to production code, and the predicate carries the whole decision.

Gates, all on the pushed tip 43d5fc44d:

• cargo fmt --all --check — clean
• cargo clippy -p tracedecay-sessions --all-targets -- -D warnings — 0 errors, 0 warnings (cc-7616)
• cargo test -p tracedecay-sessions --lib -- parent_dispatch_index — 19 passed, 0 failed (cc-7648)
• cargo test -p tracedecay-sessions --lib -- cursor — 109 passed, 0 failed (cc-7637)

[ScriptedAlchemy]

Architecture review — correctness direction accepted; HOLD for bounded-work and Windows proof

Reviewed 43d5fc44d7d0a9354d7b9546837472889a3f956d; source review only, exact-head CI pending. Binding metadata, parsing and revalidation to one opened handle and invalidating equal-length/equal-mtime rewrites is the right fix. Keep the full-prefix evidence; do not restore the trailing-4KiB heuristic or claim mtime alone proves unchanged content.

P1 performance acceptance: the previous no-I/O cache hit is removed. ParentDispatchIndex::lookup now content-validates the entire cached prefix before every reuse, so repeated lookups over a large parent transcript perform repeated O(prefix) reads even when parsing is delta-only. Attach the benchmark's actual bytes-read, wall time and corpus sizes, including a large parent queried by many subagents, not just a benchmark command. The new prefix_digest_bytes counter must expose that cost. Batch lookups within one verified ingest/read snapshot or use a genuinely authoritative change witness where available; do not cache across unverified rewrites. Keep this work off a latency-sensitive runtime worker and avoid holding an unrelated global registry lock across it.

P1 correctness acceptance: preserve equal length and exact mtime in Windows replacement tests, rewrite only the early prefix while the final 4KiB remains identical, and verify a concurrent mutation during validation/parse refuses or retries without committing mixed models. Append parsing must still consume only new records after verifying the prefix.

The five unrelated failures listed in the body need baseline comparison, not dismissal as 'environment-sensitive'; they must not be introduced by shared native identity helpers. No source-key migration or weakening of #880 admission is required for this cache correction. Target #707 only.

[ScriptedAlchemy]

System-level re-review at 43d5fc44d — Windows exact-rewrite hole in the fast path

I re-read parent_dispatch_index.rs::plan_lookup and the actual JsonlFileChangeToken definitions in runtime/source/jsonl.rs. The source needs a stronger correction than the earlier general performance comment.

P1 — the zero-content-read fast path is unconditional on Windows

plan_lookup returns the cached model when entry.revision == revision && verified_cursor == revision.len, before calling jsonl_prefix_digest. The comment says a Windows rewrite preserving last-write time falls through to the digest, but there is no platform/proof condition implementing that statement. On Windows, JsonlFileChangeToken contains only last_write_time. An in-place complete-file rewrite preserving length and that timestamp therefore keeps native file identity, length and change token identical, and this branch returns the stale model. Opened-handle identity distinguishes replacement files, not writes to the same file.

Permit this shortcut only with an adequate change witness; otherwise perform the exact prefix validation. Do not treat a Unix ctime-containing token and a Windows last-write-only token as the same evidence. Add a Windows regression that primes a newline-terminated file to EOF, changes an early model field to an equal-length value, restores the precise timestamp, closes the writer, and queries again. Assert the returned model is new and the validation receipt accounts for the content reads. Keep the unchanged trailing-4KiB case. No Windows execution is claimed here; this follows directly from the inspected predicate and token fields.

Correction to my earlier wording: the currently inspected source does have a no-content-read cache-hit path, so 'every lookup hashes the full prefix' is not an accurate unconditional description. The problem is whether that shortcut has the proof the platform can actually supply.

Keep the one-open-handle parse/revalidation boundary and the append-only delta path. For performance, batch multiple subagent lookups within one verified observation rather than weakening the rewrite witness. prefix_digest_bytes and parsed bytes must remain distinct: zero parsed records is not proof of zero read work. A parent-dispatch model is derived ingest metadata and must not override the provider/session/stream cursor authority in #948.

Returned CI is pending and profile/coverage queued. This finding takes precedence over treating the PR as correctness-complete pending only benchmarks.

[chatgpt-codex-connector]

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-09-07T05:42:17.102416Z	31c3dde	Draft marked ready

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 31c3dde0e3

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[ScriptedAlchemy]

Accepted-direction HOLD is resolved on pushed head 8b0d18d2b against #707 tip 02de800489.

Correctness and work bounds:

• Exact (identity, length, change token) hits that cover EOF still return with bytes_parsed = 0 and prefix_digest_bytes = 0.
• Windows now adds by-handle FILE_BASIC_INFO::{LastWriteTime, ChangeTime} to the parent revision. The native #[cfg(windows)] test rewrites equal-length bytes, restores the exact modification time, closes/reopens the handle, asserts last-write equality, and strictly asserts ChangeTime changed. No sleep or polling.
• A changed cached revision performs one exact cached-prefix digest before delta parsing. A revision that changes during parsing performs one consumed-prefix digest plus one final revision capture; mutation during that proof is refused rather than polled. Safe append commits the verified prefix and the next lookup parses only the delta; same-length rewrite/truncation refuses the stale model; equal-length/equal-mtime replacement invalidates by native identity.
• The early-prefix rewrite test retains an unchanged trailing payload larger than 4 KiB, so no trailing-window heuristic can pass it.

Measured workload (cargo bench -p tracedecay-sessions --bench cursor_dispatch_model):

• corpus: 5,000 records / 20,970,000-byte parent / 512 lookups (10,736,640,000 bytes for the former full-rescan behavior);
• cold + 511 repeated misses: 20,970,000 bytes parsed once, 0 prefix-digest bytes, 120 ms;
• 512 additional distinct-agent misses on the warm parent: 0 parsed bytes, 0 prefix-digest bytes, 1 ms;
• one append: exactly 20,970,000 prefix-digest bytes, 205 delta bytes parsed, no rescan-from-zero, 17 ms.

Verification:

• cargo test -p tracedecay-sessions --lib -- parent_dispatch_index — 20 passed on final head.
• cargo check -p tracedecay-private-fs -p tracedecay-sessions --tests --target x86_64-pc-windows-gnu — passed on final head, including the native Windows-only test path.
• cargo test -p tracedecay-sessions --locked --lib — 668 passed.
• cargo test -p tracedecay --locked --features test-helpers --test session_suite — 399 passed.
• cargo test -p tracedecay --locked --features test-helpers --test transcript_ingest_suite — 191 passed, 1 ignored. The first attempt correctly failed only because the worktree binary fixture was absent; after cargo build -p tracedecay-cli --bin tracedecay --locked, the unchanged suite passed.
• cargo clippy --workspace --all-targets --locked -- -D warnings — 0 errors, 0 warnings.
• cargo fmt --all --check — clean on final head.

The hosted MSVC Windows build/shards are queued behind the repository-wide CI concurrency holder; this comment does not claim that pending runtime result. PR is ready for review and remains unmerged for the parent.

[ScriptedAlchemy]

Final ancestry update: shared integration fast-forwarded this PR to e149e97ef after merging #707 tip a8c322a02. 8b0d18d2b (the verified fix/evidence head above) is an ancestor; the fast-forward only added the accepted #707 merge and did not change this PR's cache, benchmark, or Windows-token paths.

Rechecked on e149e97ef:

• cargo test -p tracedecay-sessions --lib -- parent_dispatch_index — 20 passed.
• cargo check -p tracedecay-private-fs -p tracedecay-sessions --tests --target x86_64-pc-windows-gnu — passed.
• worktree clean; origin/#707...HEAD = 0 behind / 16 ahead.

The current-head CI run is https://github.com/ScriptedAlchemy/tracedecay/actions/runs/34094861105. It is queued for hosted runners; the superseded prior-head run was cancelled so it cannot hold this one behind stale evidence.

[github-actions]

Performance Comparison codex/tracedecay-total-redesign-plan-reopened → cursor/fix-cursor-parent-cache-f604

Total Elapsed Time: 5.33s → 5.01s (-6.1%)
CPU Baseline: 82.35µs → 86.48µs (+5.0%)
Benchmark ID: index-bench-timing

timing - Execution duration of functions.

+------------------------------------------+----------------------------+----------------------------------+----------------------------------+----------------------------------+------------------------------+
| Function                                 | Calls                      | Avg                              | P95                              | Total                            | % Total                      |
+------------------------------------------+----------------------------+----------------------------------+----------------------------------+----------------------------------+------------------------------+
| tracedecay-index-bench                   | 1 → 1 (+0.0%)              | 5.33s → 5.01s (-6.0%)            | 5.34s → 5.01s (-6.2%)            | 5.33s → 5.01s (-6.0%)            | 100.00% → 100.00% (+0.0%)    |
+------------------------------------------+----------------------------+----------------------------------+----------------------------------+----------------------------------+------------------------------+
| query.artifact.append_pages              | 13 → 13 (+0.0%)            | 188.54ms → 191.98ms (+1.8%)      | 259.65ms → 257.29ms (-0.9%)      | 2.45s → 2.50s (+2.0%)            | 45.95% → 49.81% (+8.4%)      |
+------------------------------------------+----------------------------+----------------------------------+----------------------------------+----------------------------------+------------------------------+
| query.artifact.batch.sqlite              | 13 → 13 (+0.0%)            | 124.10ms → 129.92ms (+4.7%)      | 166.99ms → 177.47ms (+6.3%)      | 1.61s → 1.69s (+5.0%)            | 30.24% → 33.71% (+11.5%)     |
+------------------------------------------+----------------------------+----------------------------------+----------------------------------+----------------------------------+------------------------------+
| query.artifact.batch.postings            | 13 → 13 (+0.0%)            | 110.45ms → 115.84ms (+4.9%)      | 150.08ms → 162.00ms (+7.9%)      | 1.44s → 1.51s (+4.9%)            | 26.92% → 30.06% (+11.7%)     |
+------------------------------------------+----------------------------+----------------------------------+----------------------------------+----------------------------------+------------------------------+
| code_index.workers.install               | 83 → 83 (+0.0%)            | 19.34ms → 15.58ms (-19.4%)       | 64.42ms → 50.27ms (-22.0%) 🚀    | 1.61s → 1.29s (-19.9%)           | 30.10% → 25.81% (-14.3%)     |
+------------------------------------------+----------------------------+----------------------------------+----------------------------------+----------------------------------+------------------------------+
| code_index.build.and_publish             | 2 → 2 (+0.0%)              | 644.92ms → 503.46ms (-21.9%) 🚀  | 685.24ms → 513.28ms (-25.1%) 🚀  | 1.29s → 1.01s (-21.7%) 🚀        | 24.18% → 20.10% (-16.9%)     |
+------------------------------------------+----------------------------+----------------------------------+----------------------------------+----------------------------------+------------------------------+
| domain.canonical.sha256                  | 114526 → 114526 (+0.0%)    | 11.09µs → 8.74µs (-21.2%) 🚀     | 14.68µs → 14.64µs (-0.3%)        | 1.27s → 1.00s (-21.3%) 🚀        | 23.81% → 19.99% (-16.0%)     |
+------------------------------------------+----------------------------+----------------------------------+----------------------------------+----------------------------------+------------------------------+
| query.artifact.prepare_pages             | 13 → 13 (+0.0%)            | 50.63ms → 48.60ms (-4.0%)        | 81.99ms → 65.14ms (-20.6%) 🚀    | 658.22ms → 631.81ms (-4.0%)      | 12.34% → 12.61% (+2.2%)      |
+------------------------------------------+----------------------------+----------------------------------+----------------------------------+----------------------------------+------------------------------+
| query.artifact.finalization.advance_wake | 14 → 14 (+0.0%)            | 39.85ms → 42.37ms (+6.3%)        | 257.56ms → 267.39ms (+3.8%)      | 557.84ms → 593.15ms (+6.3%)      | 10.46% → 11.84% (+13.2%)     |
+------------------------------------------+----------------------------+----------------------------------+----------------------------------+----------------------------------+------------------------------+
| code_index.chunk.index_file              | 276 → 276 (+0.0%)          | 2.81ms → 2.14ms (-23.8%) 🚀      | 5.49ms → 3.14ms (-42.8%) 🚀      | 775.17ms → 589.94ms (-23.9%) 🚀  | 14.53% → 11.77% (-19.0%)     |
+------------------------------------------+----------------------------+----------------------------------+----------------------------------+----------------------------------+------------------------------+
| code_index.extract.parser_artifact       | 276 → 276 (+0.0%)          | 2.57ms → 2.10ms (-18.3%)         | 5.52ms → 3.23ms (-41.5%) 🚀      | 708.07ms → 579.23ms (-18.2%)     | 13.27% → 11.56% (-12.9%)     |
+------------------------------------------+----------------------------+----------------------------------+----------------------------------+----------------------------------+------------------------------+
| query.artifact.batch.parallel_prepare    | 13 → 13 (+0.0%)            | 41.16ms → 39.00ms (-5.2%)        | 68.62ms → 54.56ms (-20.5%) 🚀    | 535.09ms → 506.96ms (-5.3%)      | 10.03% → 10.12% (+0.9%)      |
+------------------------------------------+----------------------------+----------------------------------+----------------------------------+----------------------------------+------------------------------+
| 🆕 code_index.chunk.map_ordered          | 0 → 780 (+100.0%) ⚠️       | 0.00ns → 501.06µs (+100.0%) ⚠️   | 0.00ns → 590.85µs (+100.0%) ⚠️   | 0.00ns → 390.82ms (+100.0%) ⚠️   | 0.00% → 7.80% (+100.0%) ⚠️   |
+------------------------------------------+----------------------------+----------------------------------+----------------------------------+----------------------------------+------------------------------+
| code_index.chunk.build                   | 276 → 276 (+0.0%)          | 1.69ms → 1.33ms (-21.3%) 🚀      | 4.46ms → 1.91ms (-57.2%) 🚀      | 467.43ms → 366.03ms (-21.7%) 🚀  | 8.76% → 7.31% (-16.6%)       |
+------------------------------------------+----------------------------+----------------------------------+----------------------------------+----------------------------------+------------------------------+
| code_index.chunk.identify                | 276 → 276 (+0.0%)          | 1.68ms → 1.31ms (-22.0%) 🚀      | 4.44ms → 1.89ms (-57.4%) 🚀      | 463.09ms → 361.83ms (-21.9%) 🚀  | 8.68% → 7.22% (-16.8%)       |
+------------------------------------------+----------------------------+----------------------------------+----------------------------------+----------------------------------+------------------------------+
| 🗑️ code_index.build.materialize_full     | 1 → 0 (-100.0%) 🚀         | 447.10ms → 0.00ns (-100.0%) 🚀   | 447.22ms → 0.00ns (-100.0%) 🚀   | 447.10ms → 0.00ns (-100.0%) 🚀   | 8.38% → 0.00% (-100.0%) 🚀   |
+------------------------------------------+----------------------------+----------------------------------+----------------------------------+----------------------------------+------------------------------+

---

Generated with hotpath-rs