Repository navigation
S25-J: round-3 integration — sharded requant, images, verl cuts, device families, stage-1 launcher - #278
Draft
michaellchung wants to merge 40 commits into
Draft
S25-J: round-3 integration — sharded requant, images, verl cuts, device families, stage-1 launcher#278michaellchung wants to merge 40 commits into
michaellchung wants to merge 40 commits into
Conversation
The port (yeto/rl/engine/sampler_engine.py) has start/load/served_version, generate with per-token log-probs, teacher-forced score, and the expert-count and weight-memory introspection the post-load checks need. Provenance and the run-wide SamplingParams (repetition penalties refused unless allowed) live next to it. Adapters register in yeto/rl/adapters/sampler_engines. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…S25 N7.6) A-TRN report #8: restoring optimizer state on PPU can silently reset the Adam step. A cut now records a backend-neutral resume manifest (Adam step per group, LR scheduler progress and applied LRs, exp_avg/exp_avg_sq norm and sha256 of fixed sampled parameters, global step, data watermark). After loading, every rank must match it; after the first step every counter must advance by exactly the steps taken. Any mismatch raises ResumeAssertionError (a CutError); a cut without the manifest is refused. - yeto/rl/engine/resume_manifest.py: neutral types, checks, file form, torch.optim reader (FSDP2 DTensor local shards). - Miles: cut shards report the rank state; save_cut embeds the manifest, restore_cut checks it before the digest comparison, the first train_step after a restore checks the advance. - verl/FSDP2: worker binding (actor_module_fsdp/actor_optimizer/ actor_lr_scheduler). - CPU tests: train 3 + save + resume 2 equals train 5 (torch and FSDP2 over gloo, 2 ranks); reset step, partial reset, fresh moments, reset scheduler and a non-advancing first step are all refused. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
New yeto/rl/quant/mxfp4.py implements the compressed-tensors mxfp4-pack-quantized rules: E8M0 shared scale per 32 inputs (BF16 amax rounded to a power of two at mantissa 1.75, minus 2), FP4 E2M1 ties to even with saturation at 6, sign bit for w < 0, even column in the low nibble. requant.py now selects a QuantScheme from the template's quantization_config.format: w4a8-int4-channel (int-quantized, unchanged behaviour) or mxfp4-g32-rtn (weight_packed + weight_scale, refuses AWQ smoothing). compare_checkpoints reports FP4 and E8M0 code match shares. The release CLI takes its scheme tag from the template. Checked on CPU against yetalabs/apex-flash-1-mxfp4-experts@70e86a0: layer-10 routed experts (296 modules) equal in every FP4 and E8M0 code. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
compat_group(card, hbm_gib) gives e.g. ascend-910b4-64g, so 32 GiB and 64 GiB 910B4 islands never merge; without HBM the group is unchanged. Inventory nodes derive and record their group and refuse an unknown HBM variant at parse time. M890 is 144 GiB as read on the PPU machine. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
ResourceSampler reads a CardCollector of the island's device family (YETO_RL_DEVICE_FAMILY, NVML when unset). npu-smi samples carry utilization, HBM used, power, temperature and health per chip; an unusable tool writes one available:false record with the reason. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
SampleIndexEntry gets an optional provenance record that index lines carry only when set; old lines read as unknown. The payload format and GroupWriter move to the neutral yeto.rl.engine.buffer_payload (re-exported by the Miles adapter) so the engine sampler island writes the same payloads. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Groups the backend's recorded train and rollout log-probs by provenance key and reports ratio quantiles, TIS clip share, |dlogp| mean and p99, k3, finish reasons and truncated share, also per length bucket. yeto still computes no importance weight. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ers (S25 N2.3) PortReleaseLoader loads releases through the SamplerEngine port and serves them only after the expert-count, fixed-prompt and log-prob probe checks and a served-version read-back pass; failures write sampler_load_check_failed, stop generation and restart on retry. The island refuses repetition penalties and a max_tokens the engine cannot serve, and can cut looping completions as loop_abort. The SGLang (PPU) and vLLM (910B) adapters carry the A-INF vendor guards: weight_packed and dense-format refusal, MX_PATCH for MXFP4, dp8+NEXTN refusal, DeepGEMM warm-up, exit 134 after stop, start timeout, host-memory floor, eager refusal, restart-only reload on vllm-ascend. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A SamplerEngineError raised by a check (fixed prompts, log-prob probe) escaped the loader, so it was not counted and the loader kept ready=True from the previous load while the engine held unchecked weights. The loader now marks the engine not ready before each load and counts such an error as a failed check. The probe base is updated only after every check of the load passed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…5/N10.6) - SamplerEngine port: optional TokenSamplerEngine.generate_tokens (token-id prompts, stop token ids); SGLang (input_ids), vLLM (id-list prompts) and the fake engine implement it. - RecordingBackend: a gateway SessionBackend that samples the engine directly and keeps one append-only token stream per session, with loss mask, log-probs, policy version and provenance per turn; SGLang-style meta_info in the reply; per-turn render drift check; optional CompactionRL context compaction reusing yeto.rl.compaction. - RecordsProxy: the existing Gateway (Responses/Chat translation, chains, signed sampling) over the records backend, with an aiohttp shell (Responses SSE reuses the Codex bridge encoder). - chat_templates: template registry (glm5, qwen3), output parsers and the token alignment check against apply_chat_template. GLM5.3-Flash tokenizer is not available; glm5 is checked with the GLM-4.7-Flash tokenizer. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The turn's policy version was read after generate_tokens returned, so a reload during the call recorded the new version for tokens the old one may have sampled. The backend now reads the version before and after the call and refuses the turn when it changed. Signed repetition_penalty is passed to SamplingParams so check() refuses it instead of dropping it silently, and prompt_tokens is computed under the session lock. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…5 N7.7) The Megatron learner, the Miles value island and the FSDP2 plan no longer hard-code torch.cuda and NCCL. yeto.accel maps the inventory device family (cuda, ppu, npu, cpu) to a torch device type and a collective backend; PPU runs as cuda + NCCL over ACCL-P and NPU as torch_npu + HCCL (neither verified on its machine). The SSH island launcher exports YETO_DEVICE_FAMILY from the node inventory so a PPU rank does not rely on auto-detection. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
verl had no cut path, so its resume assertions were never applied. The verl adapter now writes a neutral CutManifest (same progress/data/ledger/ outer and resume sections as Miles) over one yeto-owned file per FSDP rank (local DTensor shards, optimizer state, LR scheduler, RNG). Restore needs a fresh trainer at the same world size, checks each file checksum and the loaded-state digest, then the resume assertions; the first round must advance every Adam step by the optimizer steps per round and the scheduler by one. verl itself is unchanged. Review fixes on the same assertions: - Miles compared the data watermark with a value read from the same cut (T-REV2). The watermark is now compared with the data cursor that actually landed: the driver after its seek, the trainer rebuild after its live re-read (confirm_resumed_data_cursor). The first step after a restore refuses an unconfirmed cursor. - A DP-change restore (restore_cut_resharded) now checks the run-wide counters: global step, every rank's Adam step, scheduler progress and LRs against the cut, plus the first-step and data-cursor checks. Neutral: restore_problems(check_data_watermark=), data_watermark_problems, run_wide_reference, first_step_problems(optimizer_steps=); a cut may name fewer data cursor fields (cursor_fields), sample_offset stays required. CPU tests: FSDP2 over gloo with 2 ranks, 3 rounds + cut + restore + 2 rounds equals 5 rounds bit for bit; a reset Adam step on one rank is refused on every rank; watermark, first-round and refusal cases. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… (S25 N8.2, N8.3, N8.5b) - quant/stacked.py: trainer-native stacked experts (gate_up_proj [E,2I,H], down_proj [E,H,I]) map to the per-expert source names; the converter reads a stacked source directory directly, slicing each expert on read. - quant/shards.py: each rank quantizes the rows it holds into a part file; ranks merge disjoint output shards; rank 0 writes index and metadata. The result is byte-equal to the single-process converter (W4A8 and MXFP4). - adapters/miles/quant_shards.py (Megatron grouped and sequential expert names, EP offset, ETP=1) and adapters/verl/quant_shards.py (FSDP2 Shard(0) DTensors) produce the neutral source slices. - quant/schemes.py: one place for scheme names and release_dir_for(); MXFP4 releases go to policy-releases-mxfp4-g32-rtn, W4A8 keeps policy-releases-w4a8. Records add awq_source and expert_matrices; the releaser refuses a target directory of another scheme; PolicyFollower (scheme=...) refuses a release of another scheme before reading it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…count (S25 N2) The release expert count is read from the release index before the engine loads anything, so a mismatch leaves the engine on its checked version. The loader used to mark the engine not ready anyway, which stopped generation and forced a restart on the next load. Now the release is refused: the engine stays ready, the failure event carries engine_touched=false and the kept version, and the next good release loads in place. Refusals still count toward max_consecutive_failures, so a run that only publishes bad releases stops. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…EV3) download_bytes and download_dir removed the VM staging files in a finally block, so a failing cleanup replaced the transfer's own error. The cleanup now runs after the transfer; when the transfer raised, a cleanup failure is logged and the transfer's error propagates. Also drop a duplicate import and give os.environ.get a string default in the Miles agent function (PLW1508). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…emory estimate S25 N0.2, N1.6, N7.1b. Nothing is built: both images are written only. N0.2: docker/ppu-train (x86_64, PPU trainer + SGLang sampler, on the Alibaba inference-xpu-pytorch-opt v0.5.17 base, plus Megatron-LM, mbridge, Miles glm5_next and verl) and docker/npu-sampler (aarch64, vllm-ascend 0.30.0rc1 on CANN 9.x). Each has a versions.lock.json that carries the versions and the decisions behind them, and docker/image-assert/assert_image_versions.py checks the image against its lock at every container start (and at build time for ppu-train), exiting 1 before the engine starts. The npu1 CANN gap (host 8.0.1 vs 9.x) is handled by the container's own CANN user-space, with a host driver upgrade kept as the fallback; the driver minimum is recorded, not enforced, until the first NPU probe measures it. N1.6: the node inventory requires cpu_arch, accepts a top-level images map (cpu_arch -> image), and refuses a node whose image is a yeto catalog image of another architecture or device family. The SSH trainer island compares uname -m with cpu_arch before any start. N7.1b: the memory estimator gains framework="megatron" (TP/PP/EP/CP, distributed optimizer by group, PP in-flight micro batches, mHC residual streams, the DSA dense fallback) and labels every output as derived, not measured. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The previous commit overwrote yeto/launch/images.py (learner image selection and the Miles source setup), which broke every import of yeto.launcher. That file is restored unchanged and the new catalog lives in its own module. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…h run yeto-stage1 plan expands examples/stage1/glm53-flash-ppu.yaml and its node inventory into the Miles/Megatron layout and arguments, the verl FSDP2 overrides, the SGLang TP8 server command (built by the sglang-server adapter), the MXFP4 re-quantizer argv, the OSS paths and the ordered per-node steps, and checks that card counts, scheme and release-dir names, revisions, max_tokens and penalties agree. It opens no SSH connection. yeto-stage1 run refuses while a check error or an open gap (trainer image, sampler entry, glm5_next recipe, SSH launch path, head access to OSS) remains, and needs --confirm-machines-free. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
verl cannot drop one sample, so HarborAgentLoop returns an ABORTED trajectory as a one-token placeholder with mask 0 and reward 0, and verl's GRPO counted it in its prompt group's mean and std. The new yeto_grpo estimator, registered through verl's estimator registry, moves every row without a loss token into a group of its own and then calls verl's GRPO function unchanged: the placeholder's advantage is 0 and its siblings' baseline is theirs alone. verl is not modified. Miles keeps its check_no_aborted filter (drops the group, samples a new one); both backends now hold the same rule that an ABORTED sample never enters a group mean. The test compiles verl's own GRPO function from the pinned fork source. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… (S25 N10.5) verl builds a new agent loop for every sample, and HarborAgentLoop called the YETO_AGENTIC_RECORDER factory in each constructor. The loop now keeps the recorder per event loop (LoopLocal) and calls the factory once per loop. ProxyTokenRecorder is the TokenRecorder over the N10.5 records proxy: it serves the proxy over HTTP in the worker, refuses trainer sampling that differs from the proxy's signed sampling, turns the recorded segments into TokenTurns with policy version and source, and forgets the trajectory after close. recorder_from_env builds it from YETO_RECORDS_PROXY. The loop also opens one recorder session per sample (verl repeats a row n times with the same trajectory id), refuses an estimator other than yeto_grpo, puts the sample's sampler source into extra_fields, turns an invalidated recording into ABORTED, and no longer lets a failing recorder close hide the trial's own error. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ghts-only The E2 harness replays the frozen batch drawn at the cut after arm A moved the live cursor, so rebuild_same_shape takes the resumed cursor from its caller when given (default: the live cursor). verl rank files hold only tensors and plain containers, so torch.load uses weights_only=True. The driver reads its trainer defensively for test doubles. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
yeto/launch must not import the adapters (import boundaries); the planner builds the SGLang argv with the sglang-server adapter and parses the island argv with the Miles island parser, so it lives in yeto/rl. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The SSH trainer island now exports YETO_DEVICE_FAMILY from its inventory nodes, so the resource sampler picks the card collector of the node's family (npu-smi, ppu-smi, NVML) instead of defaulting to NVML. The variable is the one T-N77 introduced for yeto.accel (defined in gpu_spec); the telemetry reads it under that name and the old YETO_RL_DEVICE_FAMILY name is gone. A run env that sets another family than the inventory is refused. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… (S25 N2.3, N2.5) Readers: an engine that does not return prompt ids (SGLang text input) writes response-only tokens with prompt_tokens_known=False. The Miles buffer reader now rebuilds the prompt ids with the trainer's tokenizer (complete_prompt_tokens, loaded on first need) and records that the trainer tokenized them; without a tokenizer such a sample is refused instead of training the response as the prompt. The source record stays in the sample metadata. offpolicy_source: verl emits the event per step from the trainer batch (old and rollout log-probs, response mask, per-row provenance from the HarborAgentLoop extra field; rows without it group under unknown), in both the sync and the fully async trainer. Miles' training process does not hand its log-probs back, so the Miles reader reports the per-source sample, truncation and loop-abort shares as rollout metrics, without ratio fields. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The prompt-id rebuild loaded the tokenizer with its own trust_remote_code=True call, which the production-tree check refuses. It now uses rollout_meta_hook's Miles tokenizer load, the single allowed one. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…islands, OSS relay - engine_sampler gains main(): builds the engine sampler island from a JSON spec (engine factory, prompts JSONL, reward, store, post-load checks). Adapters are built by yeto.rl.adapters.sampler_engines.from_spec, so the neutral core names no adapter. - Miles adapter: --rl-model-recipe glm5_next with --rl-transformer-impl (transformer_engine default, local = N7.5 TE-free fallback); the recipe's model arguments replace the Bridge geometry. - SshTrainerSpec: run_script mode with the sky task variables, runtime docker (image, mounts, device args; refused without device args), rendezvous from the inventory address instead of the SSH alias. The launcher builds the held-back ssh: trainer island from --rl-node-inventory. - Head OSS access through a relay node (ossutil over SSH), switched by YETO_RL_STORE_OSS_RELAY; unset keeps the direct path. - yeto-stage1: sampler spec step, trainer steps from the same SshTrainerSpec, store_access relay check, service steps, and a new gap for the Miles full-parameter one-node DP=1 rule (shared run_config function). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> # Conflicts: # yeto/launch/ssh_trainer_island.py
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> # Conflicts: # yeto/launch/node_inventory.py
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> # Conflicts: # yeto/launch/ssh_trainer_island.py
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> # Conflicts: # yeto/launch/ssh_trainer_island.py
…ry from the scheme gpu_spec.DEVICE_FAMILY_ENV is the only name of YETO_DEVICE_FAMILY: drop the accel alias and import it directly in stage1 and the tests. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…s from the store A full-parameter island with --rl-sample-role trainer is trainer-only: a fixed partition with no rollout GPU, which Miles runs as --debug-train-only (no engine, no weight sync to engines). Any node count and DP are accepted. Islands with engines keep the Milestone-1 rule (one node, DP=1, dedicated engines); a full-parameter trainer never shares cards with engines. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…gures with it (N7.1c) The Megatron estimate takes optimizer_offload_fraction (Adam state, and the FP32 master unless offload_master is off) and reports the offloaded bytes as host memory per node. yeto-stage1 plan now uses the estimator instead of its own static formula, checks the plan's 78.7 GiB/card and 1196 GiB/node figures (both derived) against it, and reports a gap: the Miles island runs full-parameter mode without the distributed optimizer and without optimizer offload, which the estimator puts at 243 GiB/card. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ffload Close stage-1 gap N7.1d. The full-parameter Miles island turned the distributed optimizer off and had no optimizer-offload flag, so the plan's TP4 x PP3 x EP8 x DP2 layout ran at about 243 GiB/card (estimate) against 132.5 GiB usable. - New island/launcher flag --rl-optimizer-offload-fraction (TrainMemoryConfig.optimizer_offload_fraction). It translates to --use-distributed-optimizer --use-precision-aware-optimizer --optimizer-cpu-offload --overlap-cpu-optimizer-d2h-h2d --optimizer-offload-fraction F; Megatron's validate_args requires the precision-aware optimizer with CPU offload. - Whole FP32 masters are forced only for the full-parameter sync, which reads them; a no-sync trainer island keeps Miles' distributed optimizer. Offload together with the full-parameter sync is refused. - Cut plugin: accept the precision-aware optimizer when it is the FP32 CPU-offload path (needs Miles s25-distopt for the HybridDeviceOptimizer state); the GPU path stays refused. - yeto-stage1: the island argv carries the plan's offload fraction; the island memory estimate is derived from the island argv; plan errors when the island optimizer differs from megatron_args. plan now reports only N0.2 and N0.2b. - Tests: translation, flag forwarding, stage-1 gaps, and the DistOpt PP2/EP2 cut tests run against both the image pin and s25-distopt, plus per-rank resume-manifest sampling under the sharded optimizer. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Integration branch for S25 round 3 (merges PR-B #270, PR-C #271, PR-E #273 on top of #277, so their commits also appear here).
Testing: remote full suite 6493 passed / 0 failed. Nothing validated on PPU/NPU hardware. Companion Miles fork branches s25-no-te / s25-distopt are local only.
Draft; do not merge before #262–#277.
🤖 Generated with Claude Code