Repository navigation
rl-trainer-start-stop: sample buffer, trainer lifecycle and scheduler, sampler/trainer islands, convertible nodes - #265
Draft
michaellchung wants to merge 46 commits into
Draft
michaellchung wants to merge 46 commits into
michaellchung wants to merge 46 commits into
Conversation
RoundCutCheckpoint.save takes a final-cut trigger and LocalOnlySync passes its final_trigger through; resume treats SCHEDULED like MANUAL (rl-trainer-start-stop L3.2, design D3). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit 198bf71)
TrainerLifecycle, DurableSampleBuffer and TrainerScheduler are neutral ports (rl-trainer-start-stop D1). DriverLifecycle builds a fresh driver per incarnation, JOINs before restoring the last cut, holds the driver at round boundaries, drains with the cut trigger of the stop reason (STOPPED -> SCHEDULED, PREEMPTED -> PREEMPT) and LEAVEs with the P5 code (L3.1, L3.3, L3.4). CPU fakes record JOIN/LEAVE and cut the fake trainer through save_cut/restore_cut; tests run three start/stop cycles. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit eeaa187)
FileSampleBuffer keeps an append-only index next to the sample payloads in the cut storage; reopen after process exit, torn-line recovery, select, commit, backlog statistics and polled arrival callbacks (rl-trainer-start- stop L4.1, L4.3). The consumed watermark goes into the cut record and a restart resets the buffer to it (L4.2). CPU fakes: a buffer-fed rollout pool and a sampler that loads each published version (L4.4). Tests cover the crash-after-consuming scenario and samplers following the trainer. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit 81b49f8)
ConsistentCut carries an optional buffer_watermark (rl-trainer-start-stop D4); RoundCutCheckpoint takes it from a buffer_watermark callable. The lifecycle now holds the driver after each publication, so samplers see the newest version while the trainer waits, and commits a round's samples through the rollout pool's optional commit_round verb. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit a51fb3a)
- elastic syncer: outer_step lists the deltas rejected since the previous step; every tape line gets time_unix (transfer lines too); accepted and carried deltas get arrival_offset_s; outer_step gets step_seconds and last_arrival_island. The coordinator stays clock-free: TapeClock stamps lines when the server flushes them, so ledger golden cases are unchanged. - yeto/rl/engine/merge_metrics.py folds the elastic tape into per-step weight shares, carried share/discount and rejected share by reason; wandb_tape logs these once per outer step instead of per raw record. - island side: rl_sync_timing (submit_ts, global_ready_ts, recv_seconds, sync_wait_seconds) for strict and elastic; full-parameter round events carry the same fields; decoupled remote_quorum_wait_seconds is now the measured blocked time instead of 0.0. - decoupling golden: strict tape gains one rl_sync_timing event. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit 766c927)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit 619016a)
L5.1-L5.3 (rl-trainer-start-stop): Q_on = A_hard + 1 by default, or min(Q_s1, A_hard + 1) with the optional alpha; stop when the backlog is empty and idle for T_s; T_s from the last trainer_start_summary with a configured fallback; the idle start time and the decision inputs go to the controller journal so a restarted head continues the idle time. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit 2352deb)
TrainerLoop ticks S1Scheduler on the buffer backlog and the lifecycle state, starts a new incarnation from the newest cut, trains at round boundaries, and stops with a SCHEDULED cut and the P5 LEAVE code. Starts and stops are journaled next to the scheduler state; after a head restart the loop continues, numbers the next incarnation after the last one and releases an orphaned trainer. The launcher runs it beside the fleet controller only with --rl-trainer-scheduler s1 (default off) and refuses the switch until a cloud trainer binding exists. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit 6b444d9)
Sampler islands write each finished group (whole samples with per-token behavior log-probs and weight versions) to the sample buffer through a rollout-function plugin; trainer islands select their rounds from the buffer with the max_policy_age gate instead of generating. Round cuts commit and record the consumed watermark of the rounds below the cut, and a restore continues at it. No Miles fork change. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit c34b0da)
…uler rl-trainer-start-stop L7.1-L7.3. Neutral core in yeto/rl/engine/convertible.py: node classes with the class rule (no spot or low-precision convertible node), a conversion port with a fixed call order in both directions, the P0-S cut rule for in-flight trajectories, pool rollout capacity with the last-rollout-node guard, and a scheduler that turns S1 start/stop into conversions. Miles/SGLang port uses the existing colocate offload/onload calls. Launcher flag --rl-node-class refuses bad classes before launch. CPU tests with fakes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit c8a437e)
RemoteTrainerLifecycle runs each trainer incarnation on its own island: the cloud layer launches it with the incarnation and the cut to restore in its environment, commands (train N rounds, drain with the stop reason's cut trigger and LEAVE code) go through a per-incarnation file in the run's checkpoint store, and readiness, trained rounds, the consumed watermark and the final cut come back as rl_trainer_lifecycle events on the island's event stream. The island side (IslandTrainerAgent) wraps the built driver in DriverLifecycle; the Miles entry runs it when YETO_TRAINER_CONTROL=1. The Modal binding is the default binding_factory: the launcher holds back --rl-trainer-island, spawns one call per incarnation, cancels it at release and checks the call ended; call ids are kept in the run dir so a restarted head releases an orphan. The head reads the modal-volume store through the SDK and counts the backlog from the consumed watermark the trainer committed (running incarnation's report, else the newest cut). Sky trainer islands are refused for now. Scheduler S1 now stops with FINISHED when the data ended and less than one batch is left, in any startable or training state, and treats a partial batch as idle; the loop ends and journals it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit 5ff3b06)
…ght requests, admission MilesPublisher.load_version loads a published version into the engines and returns the digest of their common check_weights(checksum) body; published_checksum gives the digest of the last full publication. MilesRolloutPool.set_admission closes/opens harness admission. engine_requests_inflight counts running and queued engine requests. resume_rollout onloads the weights tag directly: onload_weights is a no-op without 'weight' in --offload-rollout-level while the release frees weights. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit d944da3)
…rsion load_version now stamps the version under the update lock (shared helper with _publish) and, in LoRA mode, loads the adapter into each engine's LoRA pool before the read-back, so the checksum covers the adapter. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit 2a21396)
- --rl-sample-role both|sampler|trainer (default both, unchanged): neutral SampleBufferConfig on RLRunConfig; launcher checks and forwards the flags (single-island no-sync launches sharing one --rl-checkpoint-store). - Sampler island: Miles --debug-rollout-only (no trainer model, no trainer GPU), BufferWriterRolloutFn writes groups; SamplerLoop loads each policy release and waits at a backlog cap, keeping the last version while the trainer is stopped; marks the data end after the last round. - Policy releases through the store: the trainer's publisher also exports an HF checkpoint (actor.export_hf) per published version with its buffer read position; LATEST is the commit point; republish is a no-op, a rewound lineage is a new release. The sampler loads it under the controller update lock (update_weights_from_disk, stamp + read back the version). - Groups read from the buffer are renumbered by buffer position, so group and sample indexes stay unique across sampler rounds and restarts. - connect_island_ray forwards the buffer environment to Ray workers. - Full-parameter mode only; LoRA buffer islands are refused. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit 5fd3f3b)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit adc2417)
G-L7.4 first GPU run: the LoRA warm-up generate inside load_version ran with the KV cache still released and crashed the SGLang scheduler. The port now loads (load_version), onloads the KV cache, then reads the checksum (MilesPublisher.served_checksum, which warms the LoRA adapter first). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit faef230)
- --rl-island-sample-role ISLAND:ROLE (repeatable): one launch of a sampler island and a trainer island over one --rl-checkpoint-store. Needs --rl-single-island-no-sync (no syncer); each island runs as a one-learner no-sync island (learner 0 of 1) with its own --rl-sample-role and only its own role option, while the cluster / Modal function name, tape subdir and volumes keep the island index. The held-back trainer island of --rl-trainer-scheduler is therefore never counted as a syncer learner. - --tuning full is accepted for sampler/trainer islands only; the island command carries --parameter-mode full without the full-parameter sync, and the Miles island entry accepts full mode on a no-sync buffer island. - run_with_trainer_loop: when the fleet (the samplers) ends, the loop sees data_ended and keeps ticking up to --rl-trainer-drain-seconds (default 3600) so the trainer consumes the rest of the buffer and finishes. - dry-run plan: per-island sample_role, held_by_trainer_scheduler, and the trainer_scheduler record; it runs the scheduler launch checks. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit e12c6f7)
--rl-trainer-start-seconds-fixed keeps the configured T_s after a start was measured (the measured value is still journaled), and --rl-sampler-min-round-seconds throttles the sampler island so the trainer becomes idle and scheduler S1 can stop it more than once in a short run. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit 4649cc4)
The island parser accepted --parameter-mode full on a sampler/trainer island, but run_miles refused it (expert-full tuning needs the sync) and bound the full-parameter identity only inside the policy-sync block. Skip the refusal for buffer islands and bind the identity for them as well. Found by the L5.4 GPU run s24-l54-pair-20261014c. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit bd81194)
A sampler island asks Miles for --debug-rollout-only, so Miles places the rollout engines only; the placement check refused that as a rewritten fixed-partition placement. Skip the check when the launch asked for it. Found by the L5.4 GPU run s24-l54-pair-20261014e. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit 7174b9f)
The rollout metadata hook pushes group records to the named Ray actor the trainer driver creates; a sampler island has no driver, so its first round failed to look up yeto_rollout_meta. Create the sink on the sampler island and drop the records after each round. Found by the L5.4 GPU run s24-l54-pair-20261014f. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit 3e6eca4)
Miles post-processes every round for the trainer's data-parallel layout, which a trainer actor normally sets on the rollout executor. A sampler island has no trainer actor, so RolloutExecutor.get raised KeyError(None). Set DP=1 (the buffer trainer's full-parameter layout) at sampler start. Found by the L5.4 GPU run s24-l54-pair-20261014g. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit 183b724)
created_at is int(time.time()); a run crossing a second boundary made the stock and CompactionRL raw outputs differ (flaky in a full run). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit dce70b3)
A sampler/trainer (no outer sync) island in --parameter-mode full got the LoRA policy state, so its first publish exported LoRA tensors and failed with 'Megatron produced an empty or duplicate LoRA mapping' (L5.4 GPU run s24-l54-pair-20261014i, all four incarnations). Such an island now gets MilesFullParameterLocalPolicyState: each trainer rank hashes its model shard (export_full_digest plugin), the head combines the shard hashes into a trainer-resident state; the weights reach the engines through Miles update_weights as before. No LoRA export, no apply. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit db1315b)
The loop started a new trainer incarnation about 15 s after each failure, without a limit (L5.4 run i). After --rl-trainer-max-failures (default 3) failed incarnations in a row with no trained round between them, the loop releases the trainer, journals gave_up and raises TrainerLoopGaveUp; the fleet controller is aborted and the launcher exits 8. A trained round resets the count. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit 387d05b)
A trainer island that reads its rounds from the sample buffer never called --rollout-sample-filter-path / --rollout-all-samples-process-path, so the yeto rollout metadata never reached the driver and every incarnation failed with 'no rollout metadata for rollout 0'. Found by the L5.4 GPU run s24-l54-pair-20261014j. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit 212a9be)
The held-back Modal function keeps the environment built at fleet launch, so every incarnation reported phase 'submitted' at the run's launch time. Each spawn now sets YETO_START_SUBMITTED_UNIX itself, so the island's trainer_start_summary.total_s (the scheduler's T_s) measures this start. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit 6160480)
A round cut refused every full-parameter trainer ('no LoRA adapter
parameters', L5.4 GPU run k), so a trainer the scheduler stopped could not
continue. A trainer without LoRA adapters now cuts every trainable
parameter: the model copies, the named optimizer state (FP32 main and
moments, same fork-M5 helpers), the LR scheduler, RNG and counters, as for
LoRA. A mix of LoRA adapters and other trainable parameters is still
refused. The shard summary records the tuning ('lora' or 'full').
CPU test: a full-parameter trainer stops at a SCHEDULED cut, a fresh
incarnation restores it, continues the policy version and the buffer
watermark, trains no group twice and ends bitwise equal to a run that
never stopped.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
(cherry picked from commit 2748b9b)
The scheduler could only start and stop a Modal trainer island; check() refused a sky one. A sky trainer island is now one cluster per incarnation (<island>-i<n>), launched from the held-back task with the incarnation environment and torn down (verified) at release. - SkyTrainerIslands: launch submits sky.launch and provisions on a daemon thread (alive while provisioning), tails the job log for the island events; release cancels a launch still provisioning; a restarted head releases an orphan by its recorded cluster. - S3StoreFiles: the head's command channel and cut/buffer reads over an s3:// checkpoint store (the island mounts the bucket). - Each incarnation's events go to <events dir>/<island>-i<n>.jsonl (Modal too), where the launcher reads trainer_start for the manifest; a record replayed by a reconnected log tail is skipped. - check(): a sky trainer island needs an s3:// store; Verda refused. - --rl-trainer-ready-seconds (default 1800): a sky start includes provisioning. - The sky island submit (model-store config) moves to sky_ops.submit_island_launch, shared by the fleet and the binding. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit f05e0053961e4176ce911e52fd9833a66f104a37)
…store A LoRA trainer island now releases only its adapter (run_plugin write_lora_adapter over the snapshot publisher) with a file sha256 and the trainer engines' lora:* checksum digest. The sampler island hot-swaps the adapter (unload + load_lora_adapter under the update lock), waits for the first release before generating, checks the file digest before loading and reads the engines' adapter checksum back against the trainer's. The launch refusal of LoRA buffer islands is removed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit 658b7cc3d20b07509575a3395d88898ff68d3135)
After a LoRA load the sampler sends one fixed greedy request under the adapter and records the answer digest and its token log-probability sum in rl_sampler_policy_loaded, so a run shows the served policy change with the version. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit 4b1be2aaab47a6c1e22af4be09724c8f321bfad1)
The LoRA release export gathers over the trainer's NCCL groups, which a colocated trainer has released when it publishes (CUDA invalid argument in all_gather_object on the first GPU run). Require a partitioned placement. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit 9a16d57a232d642559192d7d6525eca95e1167f2)
SGLang keeps the adapter files open, and open files on a Modal Volume make every later volume reload fail, so the sampler never saw a new release. The engines now load a local copy; the previous copy is removed after its adapter is unloaded. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit be5ef64f6bf5a0203ac848c65804b7164b21a951)
A colocated trainer publishes asleep (process groups destroyed, memory paused). The LoRA release export now reloads the groups and disables the memory saver around the adapter write, as the fork's update_weights does for an asleep actor. Colocated LoRA trainer islands with more than one GPU stay refused. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit 71df2b3efb135ac9537e4cebe1714b0ff4d81403)
A LoRA sampler serves no adapter before the first policy release, so the sample buffer stays empty and S1 never starts the trainer: the pair waits forever (seen on Nebius, s24-tsky-l54-20261015a). The loop now starts a trainer while the store has no release (journal record first_policy) and keeps that incarnation until it trained a round, so it is not idle-stopped right after its warm-up publication. The cloud binding probes policy-releases/LATEST.json for LoRA runs only; a full-parameter sampler serves the base model. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit b26e7e2fdb84e6d25dcc30aa0b06e6da59d22474)
A bucket mount (goofys) cannot append to or rewrite an object, so the sample buffer's index.jsonl stalled on an s3:// store (L5.4 b on Nebius). Shared state that one host rewrites while another reads it now goes through an object API: - yeto/rl/engine/object_store.py: neutral ObjectStore port (read, write, create-if-absent, exists, list by prefix after a key, delete) and a directory backend. - yeto/cloud/s3_store.py: S3 backend (boto3, configurable endpoint for MinIO, conditional put for create) and S3Store, the bucket mount plus `objects`. Islands use only explicit store credentials (YETO_RL_STORE_S3_*), never the default AWS chain; the launcher refuses a sample-role s3 launch without them. - Sample buffer: one object per index record (sample-buffer/index/<pos>.json), positions taken with create-if-absent; the jsonl index stays for local directories and Modal Volumes. Watermark and dedup semantics are unchanged. - Trainer commands and the cut pointer are read through the object API on the island; the policy release pointer is read by key and release weights are downloaded to a local cache on the sampler. - The head reads the buffer by object key on sky instead of mirroring the index. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit bd598c9e3df5c2930e4ca960dd6edccf829e6d3e)
A bucket mount (goofys) writes an empty "<dir>/" object for each directory it creates; the sampler's release download read it as a key and failed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit 193fef17749fc1847d11bb1fcebc34c54b870bce)
The cut shards went through the goofys mount at 112-133 s per cut. A one-node island with an s3:// store now keeps the store root on local disk and mirrors it by object (TreeMirror): commit uploads new and changed files with the pointer files (LATEST, LATEST.json) last and deletes pruned ones; reload downloads the newest committed cut. Snapshot, cut and release code is unchanged. Multi-node islands keep the mount (ranks on other nodes write to their own disks). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit 65d93d53ebd7752e679c7e342bfae85e8ca29dbf)
… the store Neutral core for the convertible pool (rl-trainer-start-stop L7.5): - rank_nodes/select_group: hard conditions, network domain, layout fit and GPU type, in-flight trajectories, rollout capacity (GOAL 7.1 item 14); ConversionScheduler uses it and journals the ranking. - Sample buffer shards: one index per appender host, merged by the head. - convertible_remote: node agent (rollout rounds, conversion, one trainer incarnation per conversion), head TrainerIslands port and S1 gate. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit 75e1edfa3dace2a5b14f621a7db008b06283a944)
- Islands declared --rl-node-class islandM=convertible run as one pool: no held trainer island, S1 starts at the conversion time (60 s until one is measured), ConvertibleGate holds a start no node may take. - Head binding (yeto/launch/convertible_pool.py): node profiles from --gpu, memory estimate as the level-1 limit, shard merge before each backlog, only samples within A_hard count. - Miles node: ConvertibleRolloutFn switches between writing its shard and reading the buffer; run_convertible_node boots the engines, generates, and runs one fresh driver per conversion (DriverHandoffPort). - A node reader never creates the shared index (store commit race). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit f79848abc9bf7e39677e6ccf20a194e578a59e83)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit 717d43f26b75ed66130078310eb25d1ca5c38d7d)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit 9851510f6a55d91570b67e4b315f1703581565e2)
GPU memory per node from memory_estimate.gpu_capacity_gib (GiB, as the estimate); trainer_loop passes the node classes and the trainer precision, so convertible_pool imports neither the launcher nor trainer_loop. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit a8808b5a441061821354400f57ebd533951acd2e)
…writes GPU run s24-conv-20261015b: the driver's first publication ran onload_kv on engines whose KV cache was still resident (SGLang KeyError 'kv_cache'). The handoff port now releases the KV cache and CUDA graphs (weights stay) and wakes the trainer, the state of a fresh boot. Node store writes are retried on OS errors (a Modal mount refused one rename); status writes are best effort. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit 389ea2c6f863e31704a483e2b52b4c8178467c67)
Spec for openspec change rl-trainer-start-stop, taken from s24-sampler-spec 7bd416e2 (PR #258). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This was referenced Oct 15, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on PR-4. Implements openspec change
rl-trainer-start-stop(spec as the last commit).SCHEDULED, elastic JOIN/LEAVE, durable sample buffer with arrival polling, real Miles path wiring through--rollout-function-path.yeto/rl/engine/sample_buffer.pykeeps the object/JSONL index layouts and threads the convertible shard index throughJsonlIndex(as in integration merge 5d75197b).GPU evidence: L5.4 (Modal and sky), L7.4, LoRA sampler runs.
🤖 Generated with Claude Code