[ViperJuice]

Summary

Until now, one missing tree-sitter grammar failed the whole repository's staged rebuild, in every language. With this change, staged full and incremental builds leave only the affected language out of the generation, publish everything else, and report the gap per repository. A later sync puts the language back once its grammars load.

The work is split into three commits in the plan's order, so review can follow it:

1. Status fields (feat(readiness)). This commit only adds fields and changes no indexing yet. It adds grammars_for_language(), an availability snapshot, helpers for the grammar_degradation marker, readiness coverage, coverage/coverage_status on health rows, grammars.repositories_degraded, the search_code envelope while degraded, and coverage on symbol_lookup not-found responses. It also fixes two bugs that were already there:
   • A missing tsx grammar no longer lets .tsx files be silently parsed with the TypeScript grammar.
   • A missing javascript grammar now fails .ts files with grammar_unavailable, instead of an unrelated error from the plugin constructor.
2. Opt-in deferral (feat(indexing)):
   • Availability is decided once when the stage starts.
   • Files of an unavailable language are skipped before any plugin runs. A file whose grammar disappears mid-build is deferred instead of failed, also on a host where every grammar loaded when the build started (see the second review follow-up).
   • Every stored row of a degraded language is purged through dispatcher.remove_file, so its FTS rows and vectors go too. The marker is counted from the committed inputs, which keeps the full-walk/incremental parity rule from Incremental repository sync fails when a commit changes files in a disabled language (e.g. YAML) #111.
   • If every admitted file is deferred, the build still refuses to publish, now with code grammar_unavailable.
   • A degraded generation is never archived for upload and never turned into a commit artifact.
3. Repair (feat(indexing)). On sync (before up_to_date), and on a ref-poller tick where the tip has not moved, recoverable_degraded_languages() checks whether any left-out language's grammars now load. If so, it runs an incremental build with an empty change set that indexes only those languages' files. Automatic repair starts only from a clean ready state, so a repair that fails is not retried on every poll.

Owner decisions (as implemented)

1. Readiness: a coverage sub-object is added alongside readiness. The repository readiness state stays ready, and there is no new readiness state.
2. Drop the whole language from the staged generation, with no partial or stale rows. The purge goes through dispatcher.remove_file.
3. Search: search_code switches to the envelope (results, query, coverage, note) only while coverage is degraded, following the indexing-in-progress envelope. symbol_lookup not-found responses carry coverage.
4. Opt-in scope, v1: staged full and incremental builds only. That covers sync, the poller and whole-repository reindex through rebuild_repository_index. These paths stay fail-closed:
   • path-scoped (subdirectory or single-file) reindex through the MCP tool or the gateway, i.e. stage_operation
   • artifact restore
   • the watcher
   • the stdio background index
5. Repair: on sync and on the ref-poller's tip-unchanged check, through an incremental build with an empty change set. It skips _should_full_reindex, so other languages' chunk ids and vectors do not change. chunk_identity_scheme is not touched.

Behaviour on a fully prefetched host

When every grammar loads, behaviour is unchanged:

• The availability snapshot is complete. The walk is still opted in (through a per-build scope, see the second review follow-up), so a grammar lost mid-build is deferred. When nothing changes mid-build, nothing is deferred and index_directory returns the same stats keys as before; test_prefetched_host_unchanged_mid_build_behaves_as_before[full|incremental] covers this.
• No marker is written. The marker step does nothing when there is no marker to clear.
• Readiness reports coverage: {"state": "complete"} and search_code keeps the bare result list. test_prefetched_host_has_no_marker covers this.

A LIBS_DIR_UNSAFE host where every grammar loads also counts as complete coverage.

Status / JSON fields added

• Readiness (RepositoryReadiness.to_dict() and every surface that embeds it): coverage is one of:
  • null when no index could be inspected
  • {"state": "complete"}
  • {"state": "degraded", "unavailable_languages": [...], "missing_grammars": [...], "deferred_files": n, "recoverable": bool, "remediation": "..."}, plus "unreadable": true for a marker this binary cannot parse
• get_status:
  • repositories[].coverage and repositories[].coverage_status
  • grammars.repositories_degraded: [repo_id, ...]
• search_code: while degraded, the envelope {results, query, coverage, note} replaces the bare list. A no-results response gains coverage and note.
• symbol_lookup not-found: coverage and note.
• Staged rebuild result (IndexSyncResult):
  • degraded_languages
  • readiness.coverage on success
  • code: "grammar_unavailable" when every admitted file was deferred
• Index: the index_config key grammar_degradation = {"version": 1, "languages": {lang: {"grammars": [...], "deferred_files": n}}}.

Docs

• docs/SUPPORT_MATRIX.md: the grammar section now describes per-language degradation, coverage, repair, artifacts and scope.
• CHANGELOG.md [Unreleased]: the feature, the two bug fixes, and a rollback note: binaries older than this change ignore the marker and serve a degraded generation as complete until its next rebuild.

Tests

Grammar availability is always faked through grammar_cache.cached_grammars. The tests make no network calls and don't touch the pack.

Commands:

• uv run --extra dev pytest <files> -n 2 --no-cov, run on affected files only.
• git status --untracked-files=all is clean afterwards.

New tests:

• tests/test_grammar_registry.py:
  • grammars_for_language
  • tsx/javascript → typescript, both status and snapshot
  • prefetched host → complete
  • libs_dir_unsafe → complete coverage
  • document-plugin grammars ignored
  • missing javascript → .ts fails with grammar_unavailable
  • opt-in deferral
  • deferral when a grammar is lost mid-build
  • tsx/javascript missing → .ts/.tsx deferred
• tests/test_git_index_manager.py:
  • test_degraded_full_rebuild_publishes_other_languages
  • test_degraded_rebuild_purges_previous_rows_of_degraded_language[full|incremental]: checks files, symbols, chunks and FTS
  • test_incremental_preserves_marker_and_unaffected_chunk_ids
  • test_all_files_deferred_refuses_with_grammar_unavailable
  • test_prefetched_host_has_no_marker
  • test_create_commit_artifact_refuses_degraded_generation
  • test_repair_after_grammar_arrival_indexes_only_deferred_files: the other files' chunk ids are byte-identical before and after, and _should_full_reindex is never consulted
  • test_no_automatic_repair_while_a_failed_sync_is_pending
• tests/test_repository_readiness.py:
  • test_coverage_degraded_keeps_state_ready
  • unreadable marker → degraded
  • no index → coverage: null
• tests/test_tool_handlers_readiness.py:
  • test_search_envelope_carries_coverage_when_degraded[hits|no hits]
  • complete coverage keeps the bare list
  • symbol_lookup not-found carries coverage
  • test_get_status_health_row_coverage
• tests/test_artifact_upload.py: test_upload_refuses_degraded_generation
• tests/test_index_admission_parity.py: test_full_walk_and_incremental_agree_under_grammar_degradation
• tests/test_ref_poller.py: repair on an unchanged tip (recoverable / empty / non-set), and a failed coverage check never syncs
• tests/test_health_surface.py: the expected health-row key set gains coverage and coverage_status

The base is origin/main bd61265, after #139 merged; the rebase was clean.

• Rebased head, 15 affected files with -n 2: 738 passed, 2 failed.
  • One xdist worker crashed during test_index_admission_parity.py::test_failed_rebuilds_keep_only_the_newest_failed_generation.
  • test_v13_data_storage.py::test_internal_cross_repo_coordinator_does_not_confuse_result_shapes hit the 20 s per-file lexical timeout on a two-line hello.py.
  • The host was under contention at the time (load 11–14, swap 7.3 GiB used, memory pressure full avg300≈2). I found no OOM record, so the cause is not verified.
  • Both tests then passed in a serial, isolated retry (-p no:xdist: 2 passed).
• Pre-rebase run of the same files: 739 passed, 1 failed. That failure was another 20 s lexical timeout on a three-line README.md (test_repeated_rebuilds_keep_live_plus_bounded_previous_generations). It passed 3 times in isolation, taking 7 s to 24 s.
• None of these code paths change on a complete-coverage host, but please let CI confirm.
• ruff, black and isort are clean on the changed files. ruff still reports three unused imports in tests/test_health_surface.py; they are already on main and were left alone.

Review follow-up (f64c9d9)

These are fixes for the independent review's findings.

1. Restore can't publish a degraded artifact as complete.
   • An artifact that carries a grammar_degradation marker is refused, before any generation mutation.
   • An artifact missing any committed file this host would store is refused with code artifact_incomplete. The check compares the artifact's own file set against the admitted committed inputs, path sets only.
   • A verified, complete restore clears the marker the stage inherited from a degraded live generation.
2. The all-deferred refusal is decided from the admitted committed inputs. Leftover history and document rows no longer let a generation publish when every source file was deferred.
3. In-process mode with a missing javascript grammar: this did not reproduce. Admission resolves typescript from PluginFactory.get_plugin_availability(), which reports enabled without constructing the plugin, so .ts reaches the deferral check, gets deferred and is recorded. I added the requested sandbox-disabled regression test. It covers both deferral and repair after the grammar arrives, including when a plugin build failed while the grammar was missing.
4. The sweep ignores languages the published marker left out. They are no longer drift, so they never force a full rebuild, and repair stays on the sync/poller path.

Tests:

• 14 affected files with -n 2: 714 passed, 0 failed. Files: test_git_index_manager, test_watcher_sweep, test_index_admission_parity, test_artifact_download, test_sweeper_observability, test_v13_data_reconciliation, test_v13_data_storage, test_v13_prep_panel_repairs, test_watcher_multi_repo, test_grammar_registry, test_repository_readiness, test_tool_handlers_readiness, test_ref_poller, test_artifact_upload.
• The new restore, retained-rows and sweep tests each fail when their fix is reverted.

Review follow-up 2 (b69f219): grammar lost mid-build on a complete host

Earlier commits handled a grammar lost mid-build only when the build's snapshot was already incomplete; on a host that had every grammar at the start, the snapshot was not passed to the walk, so the loss still failed the whole rebuild with grammar_unavailable. That skipped case is now handled:

• The snapshot reaches _full_index and _incremental_index_update through a per-thread scope held around the call (_staged_grammar_scope), also when it is complete. A grammar lost mid-build defers its language exactly as one missing at the start: every row of it is dropped (including files indexed before the loss), the marker records it, the other languages publish, and the existing sync/ref-poller repair re-indexes it once the grammar loads.
• A scope rather than a call keyword: the earlier attempt to always pass the keyword broke 10 existing test_git_index_manager.py tests whose MagicMock(side_effect=...) doubles of _full_index reject unknown keywords. With the scope, no existing test was changed.
• index_directory adds deferred_files/degraded_languages only once a file is deferred when the snapshot is complete, so a prefetched host's walk stats keep their pre-Per-language degradation when a tree-sitter grammar is unavailable #117 shape.
• Path-scoped reindex, artifact restore, the watcher and the stdio background index stay fail-closed, as before.
• docs/SUPPORT_MATRIX.md says a grammar lost partway through a build is handled like one missing at the start.

New tests in tests/test_git_index_manager.py, each failing with the fix removed:

• test_grammar_lost_mid_full_rebuild_defers_language: snapshot complete, rust lost after the first .rs file indexes. Rust is deferred and recorded (deferred_files: 2), no rust file/symbol/chunk/FTS row is published, python and markdown publish, and repair restores rust. Without the fix: failed with grammar_unavailable.
• test_grammar_lost_mid_incremental_build_defers_language: same loss during an incremental build; the python change publishes and unchanged files keep their chunk ids. Without the fix: failed.
• test_prefetched_host_unchanged_mid_build_behaves_as_before[full|incremental]: the walk is opted in with a complete snapshot, yet nothing is deferred, stats carry no deferral keys, no marker is written, coverage is complete and chunk ids are unchanged. Without the fix it fails the opt-in assertion; with only the dispatcher change reverted it fails the stats-shape assertion.

Tests: the same 14 affected files with -n 2: 718 passed, 0 failed (714 before plus the 4 new). ruff, black and isort are clean on the changed files.

Review follow-up 3 (205817a): staged rename when a grammar is lost mid-build

A committed rename (lib.rs -> renamed.rs) in an incremental build still failed the whole rebuild when the grammar disappeared just before the staged move_file. The staged move_file replaced the reindex's grammar_unavailable result with "Renamed committed content was not reindexed", and the rename branch recorded that as a required-mutation failure.

• Preserve the cause. The staged move_file now returns the reindex's grammar_unavailable IndexResult, with its language, instead of the generic error. I chose this over turning renames into remove-plus-add because it is smaller and leaves the rename path's existing semantics unchanged when grammars load.
• Defer it. The incremental rename branch defers that result the same way as an added file, when the build is opted in. After the move the old path has no row. The chunkless row at the new path is dropped with the rest of the language before publish.
• Still fail-closed elsewhere. Without the opt-in, the result still fails the build. Non-staged moves (the watcher, path-scoped reindex) never reindex inside move_file, so they are unchanged.

New tests in tests/test_git_index_manager.py:

• test_grammar_lost_before_staged_rename_defers_language: rust is deferred and recorded (deferred_files: 1), and neither lib.rs nor renamed.rs has a file, symbol, chunk or FTS row. The python change publishes, other chunk ids are unchanged, and repair indexes renamed.rs. Without the fix it fails with the reviewer's exact error ("Failed to move lib.rs -> renamed.rs: Renamed committed content was not reindexed").
• test_staged_rename_with_grammars_available_behaves_as_before: the same rename with every grammar available is still a MOVED staged move. renamed.rs is reindexed with chunks, symbols and FTS rows, there is no marker and coverage is complete. This test passes with or without the fix: it guards the unchanged path.

Tests:

• The same 14 affected files with -n 2: 720 passed, 0 failed.
• The move_file-related files with -n 2: 405 passed, 0 failed. Files: test_v13_data_vectors, test_dispatcher, test_rename_atomicity, test_watcher, test_two_phase_commit, test_dispatcher_advanced, test_reindex_resume, test_incremental_indexer.
• No existing test was changed.

Review follow-up 4 (258f6d3): file_moves record of a deferred staged rename

A deferred staged rename left its file_moves row (lib.rs -> renamed.rs) in the published generation. SQLiteStore.move_file records the move before the reindex runs, and the degradation purge's remove_file does not touch move history.

Fix. The exact move record is now dropped when the rename is deferred:

• SQLiteStore.move_file takes an optional move_record dict, which receives the inserted row id.
• The new delete_file_move(id) deletes exactly that one row.
• The staged move_file passes move_record (staged moves only). When the reindex returns grammar_unavailable, it deletes that record before returning the result.

Unchanged paths:

• Moves with grammars available
• Non-staged moves (watcher, path-scoped)
• remove_file, which still never deletes move history

Why this option, not recording the move only after the reindex succeeds. That alternative would reorder the SQLite primary operation inside two_phase_commit, whose rollback and semantic shadow operation rely on the move and its record being one step. Dropping the one known record is local to the deferred branch, and leaves the successful path unchanged.

Tests:

• test_grammar_lost_before_staged_rename_defers_language now also asserts that the published generation has no lib.rs -> renamed.rs row in file_moves. Without the fix it fails: assert ('lib.rs', 'renamed.rs') not in [('lib.rs', 'renamed.rs')].
• test_staged_rename_with_grammars_available_behaves_as_before asserts the move is still recorded exactly once.
• The same 14 affected files with -n 2: 720 passed, 0 failed.
• The move_file/store-related files with -n 2: 506 passed, 0 failed. Files: the eight from follow-up 3 plus test_sqlite_store, test_collection_provenance and test_chunker_scheme_recovery.

Out of scope here: a Git type change (T status) publishing stale rows is tracked separately.

Issue coverage

Fixes #117. Here is how this PR meets the issue's three acceptance criteria:

• With one grammar missing, the other languages index and publish. The affected language is reported as unavailable in readiness and get_status; it is never silently chunkless.
• A fully prefetched host behaves exactly as today.
• docs/SUPPORT_MATRIX.md describes the degraded state.

The issue's automatic re-index after the grammar becomes available is covered by commit 3.

🤖 Generated with Claude Code

---

^{Open in [code]smith · Need help on this PR? Tag @codesmith-bot with what you need.}

[chatgpt-codex-connector]

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.
To continue using code reviews, add credits to your account and enable them for code reviews in your settings.