Skip to content

Qualify two-GPU training and document TrainCheck coverage gaps - #166

Open
Essoz wants to merge 3 commits into
codex/rsi-pilot-20261030from
codex/ring18-distributed-probe
Open

Essoz wants to merge 3 commits into
codex/rsi-pilot-20261030from
codex/ring18-distributed-probe

Conversation

@Essoz

@Essoz Essoz commented Oct 31, 2026

Copy link
Copy Markdown
Collaborator

The earlier RSI pilot kept the trainer and distributed integration outside the agent's editable surface. This adds a reproducible two-GPU qualification workload for the implementation-contract evaluation in #165, and records what the current instrumenter/checker actually supports.

The probe uses a 5,504-parameter Transformer with DDP, gradient accumulation, unequal valid-token counts per rank, clipping, AdamW, a scheduler, and FP32/BF16 modes. An independent combined-batch gradient calculation distinguishes correct global normalization from a local-mean fault. Fresh processes save/resume complete per-rank checkpoints; a typed comparator checks all stored tensor bits, scalar/container types and fields without normalization. API and active subclass tracing have explicit update scopes and suppress bookkeeping.

Measured on Ring18's two A16 GPUs, Python 3.10.14 / Torch 2.2.2+cu121:

  • 13 runs completed; all 9 complete checkpoint comparisons pass, covering two ranks and 274 tensor pairs per comparison. Plain/API/subclass states, plain↔subclass checkpoint interchange, and plain/subclass BF16 agree exactly, including complete checkpoint-file hashes.
  • Healthy FP32 gradient max error 4.47035e-8; injected local-mean fault 0.0695832. Faulty ranks still agree with each other, so rank equality alone misses this error.
  • Rank-local API inference yields 592/593 active rules. Fresh healthy runs have 2/2 alarms; fault runs 1/2, with no fault-only violated rule. These diagnostics did not distinguish this known fault.
  • Subclass samples expose stale timestamps: every recorded post-update sample in full FP32/BF16 runs predates that update's AdamW.step. First-update before/after times match despite changed hashes for all 27 parameters per rank. No core timestamp fix or rule filtering is included.
  • Total supervised workload wall time 211.29 s. This is qualification, not an optimization or representative overhead claim; short-loop timing and trace-size observations are documented with their limitations.

Validation: 28 focused tests pass locally and 28 on Ring18 with CUDA visible, configured hooks pass, explicit strict mypy passes over seven modules, and 188 exported artifact hashes verify. The CUDA-hidden configuration separately preserves 26 passes / 2 tracer-startup failures caused by metadata inspection initializing CUDA; the passing configuration does not fix that existing compatibility gap. All study processes have exited and GPUs are idle.

Scope: no FSDP, stochastic dropout, DataLoader cursor recovery, partial accumulation windows, changed world size, FP16 scaler/overflow or paired agent study. Optimizer state is verified independently but not registered with the model tracker; no cross-rank invariant checker is supplied. The reused supervisor's process-group RSS excludes separate torchrun sessions, and its size limit covers API traces only; these limitations are recorded rather than presented as full enforcement guarantees. Large traces/checkpoints remain outside Git.

Stack dependency: based on open #164 (codex/rsi-pilot-20261030) to use its integrated core and supervision changes. This PR contains only the distributed probe, regression tests, comparator and measurement report; merge/rebase onto main after #164 lands and recheck the composed result. It does not close #165 or establish agent benefit.

Full reproduction, source revisions, scope and artifact locations: measurement report.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant