Skip to content

Measure actual variable-pair invariant coverage and expose dormant checks - #160

Open
Essoz wants to merge 1 commit into
mainfrom
codex/consistency-check-coverage
Open

Essoz wants to merge 1 commit into
mainfrom
codex/consistency-check-coverage

Conversation

@Essoz

@Essoz Essoz commented Oct 27, 2026

Copy link
Copy Markdown
Collaborator

Fresh-run validation currently cannot distinguish real ConsistencyRelation checks from dormant or skipped variable rules: the coverage observer reports that family as unsupported, while the legacy checker can return success when no comparable variable values were observed.

This extends the validation observer to count the actual checker's collected overlapping variable-value pairs and completed comparisons, without replacing its predicates or learning rules. It records exact variable identities, timestamps/liveness bounds, required types/attributes, precondition results, skipped checks and unvisited pairs. Missing variable types or attributes remain dormant/unvalidated; failed pair collection leaves the opportunity count unknown rather than turning it into zero. API-call counts are explicitly inapplicable to variable-only rules. The observer validates its expected source structure and fails visibly if the checker changes.

Coverage output schema advances from 2 to 3. fully_validated_active means an active checker completed, not that it was violation-free or exhausted all candidates; consumers must also inspect check_passed and unvisited counts. Production inference and checking behavior are unchanged.

Validation:

  • All 12 new regressions fail on main 6dccb47 as unsupported coverage and pass with this change. Cases include real passes and violations, missing required attributes/types, malformed identities, comparison errors, interrupted collection and observer/direct CheckerResult equality.
  • All 38 selected new and existing coverage/contract tests pass on the independent main-based branch.
  • Configured hooks pass. Because the default mypy hook skips tools/tests, the changed tool and new tests were explicitly checked with mypy 1.9, real Torch stubs, --check-untyped-defs --disallow-untyped-defs --disallow-any-generics.
python -m unittest tests.test_consistency_coverage tests.test_fresh_validation_coverage tests.test_fresh_validation_contracts

This is validation tooling, not a performance optimization. It does not establish model-tracer/FSDP compatibility or claim that variable invariants ran in the separate API-only Llama pilot. The signed commit targets main independently; there is no stacked PR base dependency.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant