Skip to content

Preserve evidence required by selective invariant checking - #161

Draft
Essoz wants to merge 4 commits into
mainfrom
codex/selective-policy-evidence
Draft

Essoz wants to merge 4 commits into
mainfrom
codex/selective-policy-evidence

Conversation

@Essoz

@Essoz Essoz commented Oct 27, 2026 •

Copy link
Copy Markdown
Collaborator

Selective instrumentation was omitting fields that its chosen invariants inspect. A fresh native TorchTitan trace produced 61 violated rules even after repairing a separate harness JSON-loading bug: 54 missing-return cases, 1 incomplete distinct-argument case, 5 positional-only/variadic selector cases, and 1 process-local CUDA stream-handle portability failure. Missing context-constructor inputs also made rules dormant.

This compiles the requirements into a union of needed evidence:

  • Returns for ConsistentOutputRelation.
  • Complete arguments for DistinctArgumentRelation and the existing InputOutputParam contract, independent of rule order or narrower overlapping requirements.
  • Context lifecycle events and complete constructor inputs for context preconditions, including inverted conditions.
  • Full argument metadata when the selector cannot represent a required positional-only/variadic argument, unresolved name or absent signature. Supported mappings remain selective; malformed metadata raises explicit errors.

It preserves the invariant set, predicates, literal identity conditions and checker behavior. No rule or field is dropped to force a pass. FSDP/model-tracker compatibility and live-reader issues are outside this PR.

Measured correctness result

Final H100 job 55325270, composed source c2233f1, unchanged 722 rules, 6.2M Llama, one GPU, sequence 256/batch 1:

  • 61 → 1 violated rules; 716 → 719 active rules. Both the 10-step same-seed and 20-step different-seed cohorts have 718 active passes, 1 active violation, 3 dormant rules, 0 errors/unknown outcomes/skips.
  • The remaining rule 614 correctly reports that the learned literal stream handle differs in the new process. Two other rules retain reference-worker-TID dormancy; a third was dormant in the reference.
  • Exact complete observed/direct CheckerResult parity 722/722 per cohort; all report input hashes match audited traces and the frozen rules. All four selective traces pass raw/parsed-row and complete/unique call-pair audits.
  • All three paired training trials and the holdout retain matching state-byte digests, DTensor metadata, losses and RNG evidence.

This is a correctness repair, not a speedup. Correct selective collection takes 22.64x plain steady-step time in this tiny every-step workload. Earlier lower numbers omitted required evidence and are not valid checking results. Full study and evidence provenance.

Engineering validation

Baseline main 6dccb47; this independent main-based PR contains four signed commits. 69 policy/constructor/trace-integrity tests pass on the independent branch. The test-discovery fixture repair also passes both unittest invocation styles locally (28 each); before repair discovery had 3 failures and 3 errors while qualified invocation passed. The composed local pytest selection passes 83 tests. Both invocation styles also pass 28/28 on labmach Python 3.11.4 / Torch 2.16.0.dev20261012+cpu, using the SHA256-verified updated fixture and single-thread settings.

New regressions use real wrappers, JSON policy/trace loading and actual checkers, including negative controls, inversion, literal-ID dormancy, order-independent unions, required-field failures and malformed types. Healthy output/distinct-input/unsupported-argument regressions fail before their respective fixes.

Configured Ruff, mypy, Black and isort hooks pass. New tests/helpers pass explicit stricter mypy. The stronger legacy collect_trace.py check improves 10 baseline findings to 5, with none added; this is not a repository-wide strictness migration.

python -m unittest tests.test_selective_output_policy \
  tests.test_selective_context_requirements tests.test_selective_argument_kinds \
  tests.test_constructor_snapshot tests.test_trace_integrity

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant