[Essoz]

Checking-overhead measurements need a fresh native TorchTitan workload that verifies training behavior and records what instrumentation actually executed. This adds plain, explicitly API-only reference, subclass-reference, and invariant-guided selective modes for the pinned 6,163,712-parameter Llama debug model. It measures synchronized native steps and records parameter/gradient/buffer byte digests, DTensor metadata, losses and RNG evidence. Selective options go through the production JSON-loading boundary; required fields and unsupported cases fail explicitly.

DTensor evidence is limited to one rank. Subclass tracking rejects FSDP-owned DTensors before modifying them. API-only collection reports model tracking disabled. Setup, state verification, queue drain and shutdown stay separate from step timing.

Verified H100 results

Three fresh order-controlled pairs, seed 42, 10 steps, sequence 256, batch 1, sampling_interval=1, warm_up_steps=0; one H100 80GB, Torch 2.16.0.dev20261012+cu130. Corrected selective job 55325270 uses composed source c2233f1 and the unchanged 722 inferred rules.

Steady-step statistic	Plain	Selective API
Median of run medians, steps 3–10	43.861 ms	994.425 ms

Median paired slowdown is 22.64x (+2164% relative to plain), range 22.62–22.67x. The seed 43/20-step holdout is 22.77x. All three pairs and the holdout have matching initial/final tensor-byte digests, DTensor metadata, losses and RNG evidence. This tiny, every-step-instrumented workload does not establish larger-model or interval-10 overhead.

Both validated cohorts have 718 active passes, 1 active violation, 3 dormant rules. Complete observed/direct CheckerResult parity is 722/722, with report inputs matched to audited trace/invariant hashes. The remaining CUDA stream-handle portability violation and literal thread-ID dormancy are retained. All four selective traces have complete, unique call pairs and raw/parsed-row parity.

Selective produces 416,364 records /143,453,745 bytes per 10-step trace, versus 442,918 /255,365,856 for the earlier full-API reference. Record count falls only 6.0%; bytes fall 43.8%. The separate single full-API pair measured 44.570 ms plain vs 5,079.953 ms traced. That cross-job comparison is not a repeated optimization result. Earlier incomplete-policy selective timings are invalid and superseded.

Direct offline checking takes 29.980 s for 10 steps and 54.520 s for 20 steps, separately from training. Concurrent live checking and FSDP model tracking remain unvalidated; no result for either is claimed.

Validation and provenance

• 17 workload tests passed on local Python 3.10/Torch 2.2.2 and the matching lab Torch 2.16 CPU runtime, including DTensor/gradient evidence, identity/storage/RNG preservation, multi-rank rejection, failed steps, tracker-selection contracts and JSON policy loading.
• Configured hooks and explicit strict-scope mypy pass. Tools/tests are checked separately because the default mypy hook skips them. This final results commit is documentation-only; content and diff checks pass, and commits remain signed.
• TorchTitan 2210414902d84d1219fd63ab8efeb24375a2a1a1; baseline main 6dccb47. The measured composed source also includes Fix scalar CUDA hashing and enable it for CUDA tensor dumps #158 (CUDA correctness), Measure actual variable-pair invariant coverage and expose dormant checks #160 (coverage observation), and Preserve evidence required by selective invariant checking #161 (required policy evidence). This PR targets main independently and does not include their production changes.

Full measurements, limitations, hashes and reproduction details. Downloaded source/manifests/traces/results are under outputs/checking-overhead-20261027/ in the experiment owner's checkout; remote copies and exact commands are listed in the report. Large artifacts are excluded from Git; cluster scratch is retention-limited. The corrected component profile is complete; its diagnostic results and limitations follow.

Component profiling completed

H100 job 55325428, same frozen source/policy/control files, steps 3–10, main thread only. A targeted sys.setprofile collector passed thread-isolation tests on allocated Python 3.12.3: 20 background markers executed, 0 profiled. Its complete state/RNG/loss evidence and per-step/API/event counts and bytes match ordinary controls.

Component	Share of separately profiled main-thread window
Metadata conversion	53.15%
Warning management	18.69%
Known bookkeeping helpers	3.66%
Producer buffering	3.08%
Final JSON encoding, including fallback	1.56%
JSON compatibility probes	0.71%
Application/unclassified work and residual	19.14%

These are not ordinary-overhead shares or promised speedups. Profiling expands the selective window 7.961 s→42.005 s (5.28x) and plain 0.354 s→0.712 s (2.01x); callback overhead remains included. It covers 133,440 main-thread records and excludes 105,472 records from the other traced worker, plus writer-thread CPU. All categories reconcile to the full diagnostic window.

Actionable work: 1,243,312 attribute reads and warning contexts over 8 steps (155,414 per step); integer positional keys predictably reject the first orjson attempt; and 34,104 separate JSON compatibility probes. Prefer reducing repeated metadata/warning work, preserving its behavior, then avoiding predictable fallback exceptions and eligible primitive probes. Changing encoders must preserve NaN/Infinity, large-integer and error behavior. No production optimization or speedup is claimed here.

The prior cProfile job 55325378 is retained as rejected timing attribution because CPython 3.12 captured background threads and mixed caller accounting. Its percentages are not used. The corrected profile, raw traces, runtime self-test, typed analysis tools and reproducible commands are saved under outputs/checking-overhead-20261027/job-55325428 and the full report. Analysis-helper strict typing/lint/format and eight raw/owner audit fixtures pass. This PR update is a signed documentation-only checkpoint.