Repository navigation
S25-A: SSH node pool, OSS store, W4A8 requant, full-param trainer island - #269
Draft
michaellchung wants to merge 13 commits into
Draft
michaellchung wants to merge 13 commits into
michaellchung wants to merge 13 commits into
Conversation
rl-ssh-node-pool D2/D3. The card catalog gains 910B2/910B3/910C and the T-Head PPU cards 810E/M890, plus the launcher device family (cuda, npu, ppu), which is not the torch device type: a PPU reports device type cuda and family ppu. An unknown card name is now refused (canonical_card) instead of falling back silently to cuda, so an unlisted Ascend or PPU card can no longer run down the NVIDIA path. yeto/hw/card_tools.py adapts nvidia-smi, npu-smi and ppu-smi to one record (index, card, HBM GiB, health) and refuses a card, count or HBM difference before any accelerator process starts. The ppu-smi format is not verified against a real machine; both the NVML CSV and a key-value form are accepted and the docstring records that state. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
rl-ssh-node-pool D1: one file describes every node yeto reaches by SSH, on-premises or rented. Each node states its SSH target, device family, card, card count, HBM GiB and host memory GiB; a jump host, fabric, interface, mounts, image, runtime and declared price are optional. Validation refuses the launch before any SSH: a missing device family or host memory, an unknown card, a card of another family, a bad fabric or runtime, a docker node without an image, and a repeated node name or SSH target. Node.ssh_options() carries ProxyJump on the command line and never writes to the user's SSH configuration. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
rl-ssh-node-pool D5: island_comm_env maps (device family, fabric, interface) to the environment of an SSH-pool island. An npu island gets HCCL_SOCKET_IFNAME, GLOO_SOCKET_IFNAME and HCCL_CONNECT_TIMEOUT and no NCCL variables; cuda and ppu islands keep the NCCL names (ACCL-P uses them); a fabric the inventory declares (roce, ib) is never disabled. Task N1.4 checked the variable names first: HCCL_SOCKET_IFNAME, GLOO_SOCKET_IFNAME and HCCL_CONNECT_TIMEOUT appear in the Ascend deployment guides and the CANN variable reference. HCCL_IF_IP, named in design D5, is not confirmed, so this module does not write it. The cloud path (multinode_env_prelude, network tier) is unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…mily The harness accepted H200 islands only. An island now records any card the yeto catalog knows (prepare --accelerator, default H200, so existing plans and their digests are unchanged), and every island of a plan must record the same card. Before the host setup, start() runs the device-family preflight (_check_island_cards): it reads the card tool over SSH (nvidia-smi, npu-smi info, ppu-smi) and refuses the launch when the card name, the card count or the declared HBM (prepare --hbm-gib) differs. The remote host setup runs the card tool of the plan's family instead of nvidia-smi; the cuda lines are unchanged. --jit-cache-root stays cuda only, because the cache identity comes from the nvidia-smi runtime identity. Tests use recorded and stub card tool output and a fake ssh; no test reaches a real machine. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
rl-ssh-node-pool: the launcher reads its SSH nodes from an inventory file. The flag joins the RL flag schema (CLI layer) and _prepare_rl_args loads and validates the file before any cloud or SSH work, so an unknown card or a missing device family refuses the launch at submit time. The parsed nodes stay on args.rl_node_inventory_nodes for the SSH harness. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The proposal, design, tasks and spec delta of the S25 SSH node pool, as drafted in infra-drafts/SESSION25. The CPU parts of N1.1, N1.2, N1.4 and the N1.3 half that needs no machine are implemented in the commits above; the real-node smoke (N1.3) and the head reachability preflight (N1.5) are not. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…checks rl-cross-site-sample-store N3.1-N3.4 (CPU part): - yeto/cloud/oss_store.py: OssStore for oss://BUCKET[/PREFIX] through the S3-compatible API, selected by YETO_RL_STORE_IMPL. Always staged: the store root is a local directory mirrored to the bucket. Credentials come from environment variables or an owner-only JSON file. Creates send x-oss-forbid-overwrite. Not tested against OSS. - Large files move in parts: multipart upload read from the file with pread, aborted on failure; parallel ranged downloads pinned to the ETag. Part size and concurrency come from YETO_RL_STORE_PART_MB and YETO_RL_STORE_PART_CONCURRENCY. TreeMirror and release downloads stream files instead of holding them in memory. - Sample buffer: on a store without reliable cross-host creates, each sampler island writes its own shard (shards/<name>/<pos>.json). The trainer, the one consumer, merges the shards into the index and keeps shard and shard_pos per record; shard_positions() gives the per-shard form of a watermark. Releases carry the trainer's per-shard read positions, and a shard sampler caps its backlog on its own shard. - Policy releases record every weight file with its size. A reader loads a release only when all files exist with those sizes; the follower otherwise keeps the served release. The mirror uploads data files, then RELEASE.json, then LATEST.json. - The trainer scheduler no longer pins the store scheme to the trainer's cloud: it accepts modal-volume://, s3:// or oss:// when the trainer and every sampler island reach the store (static check by store kind). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Proposal, design, tasks and spec delta for the OSS cross-site store. N3.1-N3.4 are marked done; the design notes where the implementation differs from the draft (trainer-side shard merge, scalar watermark with a derived per-shard vector). N3.5 (OSS transfer measurement) and N3.6 (head backlog view of a sharded buffer) remain open. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Full-parameter training needs every trained BF16 policy turned back into the W4A8 checkpoint the vllm-ascend sampler loads. The original artefact (apex-awq-w4a8-20261012-v10/checkpoint) is compressed-tensors int-quantized, not msModelSlim: routed expert linears hold int4 values in I8 with a BF16 per-channel scale [out, 1]; all other tensors stay high precision, with routers stored F32 and AWQ smoothing folded in. yeto/rl/quant: - int4: symmetric per-channel quantization, scale = amax / 7.5. - awq: fixed AWQ factors from recipe.yaml + awq-scales/ (option R2); AwqSmoothing.none() is the no-AWQ fallback (R1). - requant: layout template taken from the original checkpoint, per-tensor transform (row slices give the same rows), shard-by-shard converter with an output-shard split plus finalize, tensor-by-tensor comparison, CLI. The rules reproduce layer 30 of the original bit for bit (router, shared experts, norm, one routed expert) from the BF16 source; the full-model comparison is not run yet. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…7.1, N7.2) yeto/rl/fullparam/memory.py derives the per-card device and per-node host memory of one FSDP2 step (sharded state, gathered layers, activations, the HF dense DSA terms) for offload none/optimizer/full and refuses plans that exceed the inventory. Model shapes are counted on the meta device. yeto/rl/fullparam/fsdp2.py shards an HF causal LM per decoder layer with mixed precision and optional CPU offload, runs one training step, and writes the same plan as verl FSDP2 overrides (lora_rank=0). Host-step optimizer offload and Ulysses without a flash-attention path are refused. Tests: one step on two gloo ranks on CPU (tiny llama; tiny glm5_next when transformers has it: the DSA indexer receives no gradient). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ed-shard backlog (S25 N7.3, N3.6) yeto/launch/ssh_trainer_island.py implements the TrainerIslands port over SSH node-inventory nodes: card preflight per device family, one torchrun per node in its own session with the incarnation, collective and run environment, node 0's log as the event stream, release and orphan release by process group. The spec is checked against the full-parameter memory estimate before any SSH. trainer_loop dispatches cloud "ssh" to it. yeto/rl/quant/release.py turns each BF16 policy release into a W4A8 release in policy-releases-w4a8/ (same commit order, derived_from record, idempotent on restart); the island runs it as a side process on node 0 plus one pass after the trainer ends. PolicyReleaseStore takes a release_dir; PolicyRelease gains derived_from. N3.6: the head's CommittedBacklog also counts sampler shard records the trainer has not merged yet (UnmergedShards, read-only), so a stopped trainer can be started again on an OSS store. Adds the rl-npu-ppu-trainer change (PPU trains, NPU samples). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…L literals Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…5 review) - PolicyReleaseStore no longer drops the cache of another release directory (the W4A8 cache under the default one) when it prunes. - The trainer's shard merge keeps each shard in its own order, so a restarted merger never skips or repeats a record after a clock step. - S3ObjectStore.read_file reads a small object with one unpinned GET and reports an object pruned after the HEAD as missing. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Testing: Full remote test suite on the stack tip: 0 failures. Not validated on PPU/NPU hardware (machines busy).
Part of the S25 stack (NPU+PPU heterogeneous GLM5.3-Flash full-param RL). Draft; do not merge before the S24 stack (#262–#268).
🤖 Generated with Claude Code