Skip to content

S24: sampler island, trainer start/stop, decoupled PPO, identity tiering - #261

Draft
michaellchung wants to merge 64 commits into
mainfrom
s24-impl
Draft

S24: sampler island, trainer start/stop, decoupled PPO, identity tiering#261
michaellchung wants to merge 64 commits into
mainfrom
s24-impl

Conversation

@michaellchung

@michaellchung michaellchung commented Oct 15, 2026

Copy link
Copy Markdown
Contributor

Summary

S24 implementation of the four openspec changes in #258:

  • rl-trainer-cold-start-cost: start-phase events and T_s; setup reads the image manifest; working model prefetch; lighter Miles import chain (no megatron_utils in light actors, lazy modelopt); port race fix (defect 9); opt-in sampler SGLang snapshot mode. New ports image 44d41ef-2fa8801.
  • rl-offpolicy-correction-passthrough: neutral CorrectionSpec; decoupled PPO mapped to Miles (--use-tis) and verl (bypass_mode=False, rollout_is=token); refusal of settings that break it; start-version gating; offpolicy_gate events for both backends; max_policy_age cap 4.
  • rl-trainer-start-stop: trainer lifecycle / durable sample buffer / scheduler ports; S1 scheduler in the head loop (--rl-trainer-scheduler s1); remote Modal trainer islands with a store command channel; sampler-only islands and weight release through the store; full-parameter round cut; merge metrics and sync timing; convertible nodes.
  • rl-identity-tiering: semantic_sha256, tolerant plugin hashes, run-level code pack (--rl-code-pack git-patch).

Test plan

  • Local full suite (flock): 5937 passed, 0 failed (empty baseline at c4ae38a).
  • GPU (Modal H100): L8.1, L8.2, L6.11, L6.10, L7.4, L2.6, L2.2, L1.1 step 1, D4 timing, L5.4 (2 SCHEDULED stops, version continuity, no retraining). Evidence: SESSION24 gpu reviews.
  • Not verified: Miles drop branch on GPU; Modal real snapshot restore; TP>1 snapshot; data-end stop path on GPU; sky trainer islands.

🤖 Generated with Claude Code

chowhsu and others added 30 commits October 14, 2026 13:38
@chowhsu @claude
Add the SCHEDULED consistent-cut trigger for scheduler-ordered stops
198bf71 
RoundCutCheckpoint.save takes a final-cut trigger and LocalOnlySync passes
its final_trigger through; resume treats SCHEDULED like MANUAL
(rl-trainer-start-stop L3.2, design D3).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@chowhsu @claude
rl-identity-tiering L9.1/L9.2: semantic hash dual write, tolerant Plu…
17b98c3 
…ginRef

- AlgorithmSpec.semantic_sha256(): v2 spec with plugin refs reduced to path.
- Run manifest and rl_driver_start dual-write semantic_sha256 and
  code_pack_digest (transition: attested source tree hash, mode source-tree).
- PluginRef.sha256 is optional; a stale hash is a warning event
  (rl_plugin_source_mismatch), no longer a reject (also grpo_knobs).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@chowhsu @claude
Add trainer lifecycle, buffer and scheduler ports over IslandDriver
eeaa187 
TrainerLifecycle, DurableSampleBuffer and TrainerScheduler are neutral
ports (rl-trainer-start-stop D1). DriverLifecycle builds a fresh driver per
incarnation, JOINs before restoring the last cut, holds the driver at round
boundaries, drains with the cut trigger of the stop reason (STOPPED ->
SCHEDULED, PREEMPTED -> PREEMPT) and LEAVEs with the P5 code (L3.1, L3.3,
L3.4). CPU fakes record JOIN/LEAVE and cut the fake trainer through
save_cut/restore_cut; tests run three start/stop cycles.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@chowhsu @claude
Record trainer start phases; setup reads the image manifest; fix mode…
4a39de3 
…l prefetch

- L0.1/L0.2 (rl-trainer-cold-start-cost): seven trainer_start_phase events
  (submitted, container_ready, setup_done, ray_ready, actor_import_done,
  actor_init_done, first_step_done) and a trainer_start_summary with
  total_s; the launcher copies the summaries into run_manifest.json
  (trainer_start) so the scheduler can read T_s of the last run.
- L2.3: island setup checks peft and sglang against
  /opt/yeto/image-manifest.json without importing them; a missing entry
  falls back to the import check and records image_manifest_fallback; a
  mismatch fails setup before Ray starts.
- L2.4 (code only, no image built): the ports image build puts the pinned
  peft wheel into site-packages and records it in the image manifest.
- L2.5: the Hub prefetch uses snapshot_download (huggingface-cli exits 1 in
  the ports image) and records model_prefetch_failed with the exit code.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@chowhsu @claude
Record the sample-buffer watermark in the cut; wait after publications
a51fb3a 
ConsistentCut carries an optional buffer_watermark (rl-trainer-start-stop
D4); RoundCutCheckpoint takes it from a buffer_watermark callable. The
lifecycle now holds the driver after each publication, so samplers see the
newest version while the trainer waits, and commits a round's samples
through the rollout pool's optional commit_round verb.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@chowhsu @claude
Add the durable file sample buffer with arrival polling
81b49f8 
FileSampleBuffer keeps an append-only index next to the sample payloads in
the cut storage; reopen after process exit, torn-line recovery, select,
commit, backlog statistics and polled arrival callbacks (rl-trainer-start-
stop L4.1, L4.3). The consumed watermark goes into the cut record and a
restart resets the buffer to it (L4.2). CPU fakes: a buffer-fed rollout pool
and a sampler that loads each published version (L4.4). Tests cover the
crash-after-consuming scenario and samplers following the trainer.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@chowhsu @claude
Add merge metrics, island sync timing and syncer tape clock (L5.5-L5.7)
766c927 
- elastic syncer: outer_step lists the deltas rejected since the previous
  step; every tape line gets time_unix (transfer lines too); accepted and
  carried deltas get arrival_offset_s; outer_step gets step_seconds and
  last_arrival_island. The coordinator stays clock-free: TapeClock stamps
  lines when the server flushes them, so ledger golden cases are unchanged.
- yeto/rl/engine/merge_metrics.py folds the elastic tape into per-step
  weight shares, carried share/discount and rejected share by reason;
  wandb_tape logs these once per outer step instead of per raw record.
- island side: rl_sync_timing (submit_ts, global_ready_ts, recv_seconds,
  sync_wait_seconds) for strict and elastic; full-parameter round events
  carry the same fields; decoupled remote_quorum_wait_seconds is now the
  measured blocked time instead of 0.0.
- decoupling golden: strict tape gains one rl_sync_timing event.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@chowhsu @claude
Drop Megatron-Bridge's eager modelopt import in the ports image layer
2838755 
The ports image overlay layer now writes megatron/bridge/__init__.py from
docker/miles-ports/megatron_bridge_init.py: the Megatron-Bridge 8cd3466 file
without its top-level `import modelopt.torch`, which cost 21-30 s per
trainer process. Bridge modules that use modelopt import it themselves.
The image manifest records the patch. Takes effect on the next image build
(rl-trainer-cold-start-cost L2.2).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@chowhsu @claude
Add neutral CorrectionSpec fields and keep the correction group out o…
b8eb808 
…f the phase-1 hash

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@chowhsu @claude
Map correction settings to Miles, refuse decoupled-PPO breakers, reco…
25d07c1 
…rd and compare per island

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@chowhsu @claude
Raise policy-age cap to 4, emit offpolicy_gate, add off-policy correc…
5976f1b 
…tion tests

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@chowhsu @claude
Make the submitted-phase clock a seam; update tests for the new prefe…
dc96be6 
…tch and envs

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@chowhsu @claude
Load fake lifecycle cuts with weights_only=True
619016a 
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@chowhsu @claude
Add scheduler S1: backlog start, ski-rental stop, journaled idle time
2352deb 
L5.1-L5.3 (rl-trainer-start-stop): Q_on = A_hard + 1 by default, or
min(Q_s1, A_hard + 1) with the optional alpha; stop when the backlog is
empty and idle for T_s; T_s from the last trainer_start_summary with a
configured fallback; the idle start time and the decision inputs go to the
controller journal so a restarted head continues the idle time.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@chowhsu @claude
Keep plugin sources and default tapes unchanged: build offpolicy_gate…
3ffea04 
… in the driver

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@chowhsu @claude
Add gated opt-in SGLang snapshot mode for sampler engines (L8.2)
b5e5328 
--rl-sampler-snapshot (off by default) starts the fixed-partition rollout
engines with enable_memory_saver + enable_weights_cpu_backup via an inline
--sglang-config override. Launch is refused unless the shipped
sampler_snapshot_compat.json records a pass for the pinned SGLang commit on
the NCCL (distributed) weight-update path. The L8.1 tensor-path pass is
recorded; the distributed-path record waits for the L8.2 GPU check.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@chowhsu @claude
rl-identity-tiering L9.4: yeto/codepack.py run code pack (git-patch m…
98d88d1 
…ode)

Deterministic tar of commit + dirty tree (untracked included), sha256
digest, pluggable pack stores, checked unpack, run-level resolve that
reuses the manifest pack, and the explicit upgrade rule (L9.6).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@chowhsu @claude
Merge branch 's24-miles-import' into s24-image
0ffd52a 
Resolve make_layer.py: keep both the --wheel unpacking (peft) and the
megatron_bridge __init__.py override.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@chowhsu @claude
Record trainer_start T_s for non-echo sky islands
8c9ff33 
Read the island tapes that the pre-teardown diagnostics fetched into
~/yeto-diag/<cluster>/ and merge their trainer_start_summary into the run
manifest, so every backend gives the scheduler T_s.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@chowhsu @claude
rl-identity-tiering L9.4-L9.6: launcher uses the run code pack
50cb213 
--rl-code-pack git-patch makes the pack at the first start, stores it in
the checkpoint store and records it in the run manifest (code_pack_mode
git-patch). Later starts of the run take the manifest pack; its unpacked
tree is the sky workdir and the Modal add_local_dir. Islands get
--code-pack-digest and refuse to start when the tree differs.
--rl-code-pack-upgrade REASON switches packs only with an unchanged
semantic hash and records old -> new in the manifest.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@chowhsu @claude
Run the trainer scheduler in the head loop behind --rl-trainer-scheduler
6b444d9 
TrainerLoop ticks S1Scheduler on the buffer backlog and the lifecycle state,
starts a new incarnation from the newest cut, trains at round boundaries,
and stops with a SCHEDULED cut and the P5 LEAVE code. Starts and stops are
journaled next to the scheduler state; after a head restart the loop
continues, numbers the next incarnation after the last one and releases an
orphaned trainer. The launcher runs it beside the fleet controller only
with --rl-trainer-scheduler s1 (default off) and refuses the switch until a
cloud trainer binding exists.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@chowhsu @claude
Regenerate decoupling golden samples for the phase-1 correction hash …
69a2722 
…(D8)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@chowhsu @claude
Wire the Miles rollout through the durable sample buffer (L4.5)
c34b0da 
Sampler islands write each finished group (whole samples with per-token
behavior log-probs and weight versions) to the sample buffer through a
rollout-function plugin; trainer islands select their rounds from the buffer
with the max_policy_age gate instead of generating. Round cuts commit and
record the consumed watermark of the rounds below the cut, and a restore
continues at it. No Miles fork change.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@chowhsu @claude
Map off-policy correction to verl fully_async and add CPU parity checks
da796bb 
- Add yeto/rl/adapters/verl/offpolicy_correction.py (role offpolicy_correction):
  design D3 rows as algorithm.rollout_correction overrides, prox_anchor
  first_local_step, bypass_mode=True refused, warnings for unmappable fields
  and for the MIS-mask denominator.
- fully_async_plan replaces the rollout_correction overrides with the
  mapping (decoupled PPO: bypass_mode=False, rollout_is=token, threshold=C).
- RoundCollector fills the offpolicy_gate carry-over keys (trajectory
  granularity, whole-group drop over A_hard); the driver builds the event.
- CPU parity tests load the verl fork and Miles correction functions from
  source: TIS (lower 0) and IcePop weights match; verl rejection sampling
  shrinks the loss mask; actor.use_rollout_log_probs has no reader.
- Example specs decoupled-ppo-c2 (A_hard 2) and -agentic (A_hard 4).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@chowhsu @claude
Keep the sampler snapshot gate on the island side only
05c359c 
The launcher layer may not import adapters and reads placement only from
island args; the island translation refuses before any Ray/GPU work.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@chowhsu @claude
Pin Miles 17aeb5cb5 and ports image 17aeb5c-2fa8801
47e3052 
The image layer adds peft 0.20.0 and the lazy-modelopt Megatron-Bridge
__init__.py; Miles s24-miles-import keeps light actors off megatron_utils
and holds the master port until use. Regenerate the decoupling golden
samples (backend identity and runtime fingerprint only) and record the
build.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@chowhsu @claude
Record the L8.2 NCCL-path snapshot compatibility pass
0a5241f 
G-L8.2 (2x H100, SGLang 2fa880182): flags on vs off agree in text and
full-model checksum across two update_weights_from_distributed updates,
including a release/resume and a rebuilt NCCL group before the second one.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@chowhsu @claude
Add convertible nodes: node classes, conversion port, cut rule, sched…
c8a437e 
…uler

rl-trainer-start-stop L7.1-L7.3. Neutral core in yeto/rl/engine/convertible.py:
node classes with the class rule (no spot or low-precision convertible node),
a conversion port with a fixed call order in both directions, the P0-S cut rule
for in-flight trajectories, pool rollout capacity with the last-rollout-node
guard, and a scheduler that turns S1 start/stop into conversions. Miles/SGLang
port uses the existing colocate offload/onload calls. Launcher flag
--rl-node-class refuses bad classes before launch. CPU tests with fakes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@chowhsu
Merge branch 's24-l4-wiring' into s24-l5-remote
95ba2e8
@chowhsu @claude
Label offpolicy_gate events with the batch's engine backend
8e5884a 
RolloutBatchHandle gains a backend field; the Miles and verl adapters fill
it, and the driver labels offpolicy_gate with it ('unknown' when unset)
instead of assuming 'miles'.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
chowhsu and others added 29 commits October 14, 2026 17:44
@chowhsu @claude
Keep the trainer island binding free of an import cycle
adc2417 
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@chowhsu @claude
Read the convertible node's weight checksum after the KV onload
faef230 
G-L7.4 first GPU run: the LoRA warm-up generate inside load_version ran with
the KV cache still released and crashed the SGLang scheduler. The port now
loads (load_version), onloads the KV cache, then reads the checksum
(MilesPublisher.served_checksum, which warms the LoRA adapter first).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@chowhsu
Merge branch 's24-l5-measure' into s24-integ
34102b2 
# Conflicts:
#	openspec/changes/yeto-framework-decoupling/hash-migration.md
@chowhsu
Merge branch 's24-l9-identity' into s24-integ
108e66c
@chowhsu
Merge branch 's24-l8-snapshot' into s24-integ
83bff47
@chowhsu
Merge branch 's24-l5-remote' into s24-integ
3643236
@chowhsu
Merge branch 's24-sampler-mode' into s24-integ
9bdcef9
@chowhsu
Merge branch 's24-l7-convert' into s24-integ
e89bcf7
@chowhsu @claude
Compose the L5.4 sampler/trainer pair in one launch
e12c6f7 
- --rl-island-sample-role ISLAND:ROLE (repeatable): one launch of a sampler
  island and a trainer island over one --rl-checkpoint-store. Needs
  --rl-single-island-no-sync (no syncer); each island runs as a one-learner
  no-sync island (learner 0 of 1) with its own --rl-sample-role and only its
  own role option, while the cluster / Modal function name, tape subdir and
  volumes keep the island index. The held-back trainer island of
  --rl-trainer-scheduler is therefore never counted as a syncer learner.
- --tuning full is accepted for sampler/trainer islands only; the island
  command carries --parameter-mode full without the full-parameter sync, and
  the Miles island entry accepts full mode on a no-sync buffer island.
- run_with_trainer_loop: when the fleet (the samplers) ends, the loop sees
  data_ended and keeps ticking up to --rl-trainer-drain-seconds (default
  3600) so the trainer consumes the rest of the buffer and finishes.
- dry-run plan: per-island sample_role, held_by_trainer_scheduler, and the
  trainer_scheduler record; it runs the scheduler launch checks.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@chowhsu @claude
Create the Modal volume when the code pack is first stored
06d617b 
The launcher stores the run code pack before the island creates the
checkpoint volume, so a first launch with --rl-code-pack git-patch on a
new modal-volume:// store failed with NotFoundError (found on GPU, L6.10).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@chowhsu @claude
Pin Miles 44d41efd0 and ports image 44d41ef-2fa8801 (lazy modelopt)
f013058 
rl-trainer-cold-start-cost L2.2 did not pass on 17aeb5c-2fa8801: an
import trace in the image showed that the trainer gets modelopt from
Megatron-LM a84b105 (megatron/training/checkpointing.py and training.py
import it at module top), not from Megatron-Bridge.

- The image layer now applies docker/miles-ports/
  megatron_lm_lazy_modelopt.patch to the base's /root/Megatron-LM files
  (make_layer.py patched_files, --megatron-lm-src). The modelopt helpers
  are imported where they are used. The checkpoint save/load helpers are
  skipped while modelopt is not imported: they are no-ops for a model
  that ModelOpt has not converted, and only ModelOpt converts a model.
  post_training/checkpointing.py imports modelopt per function, so
  get_model's has_modelopt_state file check no longer needs it.
- Miles s24-miles-import2 (44d41efd0): the megatron_utils package
  __init__ stays light, so RayWorkerManager no longer loads megatron.bridge
  when it unpickles placement-group and worker-spec objects.
- Regenerate the decoupling golden samples (backend identity and runtime
  fingerprint only), record the build, update hash-migration.md.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@chowhsu @claude
Add check-run levers for the L5.4 start/stop GPU check
4649cc4 
--rl-trainer-start-seconds-fixed keeps the configured T_s after a start was
measured (the measured value is still journaled), and
--rl-sampler-min-round-seconds throttles the sampler island so the trainer
becomes idle and scheduler S1 can stop it more than once in a short run.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@chowhsu @claude
Create the Modal Volume when the code pack is stored
0d473e7 
The code pack is uploaded before the first island starts, so the run's
checkpoint Volume may not exist yet; the launch failed with NotFoundError.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@chowhsu @claude
Let a no-sync buffer island train every parameter in run_miles
bd81194 
The island parser accepted --parameter-mode full on a sampler/trainer
island, but run_miles refused it (expert-full tuning needs the sync) and
bound the full-parameter identity only inside the policy-sync block. Skip
the refusal for buffer islands and bind the identity for them as well.
Found by the L5.4 GPU run s24-l54-pair-20261014c.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@chowhsu @claude
Keep the off-policy gate inputs on the Miles rollout handle
8dec16e 
_carry_over_fields whitelisted the carry-over keys but not the gate
inputs (dropped lengths, granularity, truncation by turn), so a real
Miles island never taped offpolicy_gate (found on GPU, L6.10).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@chowhsu @claude
Accept --debug-rollout-only on a fixed-partition sampler island
7174b9f 
A sampler island asks Miles for --debug-rollout-only, so Miles places the
rollout engines only; the placement check refused that as a rewritten
fixed-partition placement. Skip the check when the launch asked for it.
Found by the L5.4 GPU run s24-l54-pair-20261014e.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@chowhsu @claude
Give the sampler island its rollout metadata sink
3e6eca4 
The rollout metadata hook pushes group records to the named Ray actor the
trainer driver creates; a sampler island has no driver, so its first round
failed to look up yeto_rollout_meta. Create the sink on the sampler island
and drop the records after each round. Found by the L5.4 GPU run
s24-l54-pair-20261014f.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@chowhsu @claude
Set the sampler island's train parallel config for Miles post-processing
183b724 
Miles post-processes every round for the trainer's data-parallel layout,
which a trainer actor normally sets on the rollout executor. A sampler
island has no trainer actor, so RolloutExecutor.get raised KeyError(None).
Set DP=1 (the buffer trainer's full-parameter layout) at sampler start.
Found by the L5.4 GPU run s24-l54-pair-20261014g.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@chowhsu @claude
Merge s24-image into s24-integ
555286f 
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@chowhsu
Merge branch 's24-gpu-l610' into s24-integ
e50885d
@chowhsu @claude
Put RayMetadataSink.discard on the Ray sink, not the dir source
79280be 
3e6eca4 added the method to DirMetadataSource by mistake; the sampler
island failed with AttributeError in run s24-l54-pair-20261014h.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@chowhsu @claude
Count Miles' own stale buffer drops in offpolicy_gate
439705e 
Under --fully-async Miles drops stale groups inside its own buffer; yeto
saw them only as stale_dropped_groups in the carry-over. Report them in
offpolicy_gate under a separate reason, backend_stale_dropped, so they
stay apart from the yeto gate drops (absent when Miles gives no count).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@chowhsu @claude
Pin the clock in the compaction byte-identity test
dce70b3 
created_at is int(time.time()); a run crossing a second boundary made
the stock and CompactionRL raw outputs differ (flaky in a full run).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@chowhsu @claude
Publish a no-sync full-parameter island through a full-parameter digest
db1315b 
A sampler/trainer (no outer sync) island in --parameter-mode full got the
LoRA policy state, so its first publish exported LoRA tensors and failed
with 'Megatron produced an empty or duplicate LoRA mapping' (L5.4 GPU run
s24-l54-pair-20261014i, all four incarnations). Such an island now gets
MilesFullParameterLocalPolicyState: each trainer rank hashes its model
shard (export_full_digest plugin), the head combines the shard hashes into
a trainer-resident state; the weights reach the engines through Miles
update_weights as before. No LoRA export, no apply.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@chowhsu @claude
Stop the trainer scheduler after repeated failed incarnations
387d05b 
The loop started a new trainer incarnation about 15 s after each failure,
without a limit (L5.4 run i). After --rl-trainer-max-failures (default 3)
failed incarnations in a row with no trained round between them, the loop
releases the trainer, journals gave_up and raises TrainerLoopGaveUp; the
fleet controller is aborted and the launcher exits 8. A trained round
resets the count.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@chowhsu @claude
Run Miles' end-of-round hooks in the sample buffer reader
212a9be 
A trainer island that reads its rounds from the sample buffer never called
--rollout-sample-filter-path / --rollout-all-samples-process-path, so the yeto
rollout metadata never reached the driver and every incarnation failed with
'no rollout metadata for rollout 0'. Found by the L5.4 GPU run
s24-l54-pair-20261014j.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@chowhsu @claude
Merge s24-integ (dce70b3) into s24-gpu-l54
4345e7b 
Brings in the start-phase events (L0.1-L0.2), the new image pins and the
L6.10 fixes. Both sides fixed the code-pack volume creation; the s24-integ
version of yeto/codepack.py and its test is kept.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@chowhsu @claude
Time a held trainer island's start from its spawn
6160480 
The held-back Modal function keeps the environment built at fleet launch,
so every incarnation reported phase 'submitted' at the run's launch time.
Each spawn now sets YETO_START_SUBMITTED_UNIX itself, so the island's
trainer_start_summary.total_s (the scheduler's T_s) measures this start.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@chowhsu @claude
Cut a full-parameter trainer: carry every trainable parameter
2748b9b 
A round cut refused every full-parameter trainer ('no LoRA adapter
parameters', L5.4 GPU run k), so a trainer the scheduler stopped could not
continue. A trainer without LoRA adapters now cuts every trainable
parameter: the model copies, the named optimizer state (FP32 main and
moments, same fork-M5 helpers), the LR scheduler, RNG and counters, as for
LoRA. A mix of LoRA adapters and other trainable parameters is still
refused. The shard summary records the tuning ('lora' or 'full').

CPU test: a full-parameter trainer stops at a SCHEDULED cut, a fresh
incarnation restores it, continues the policy version and the buffer
watermark, trains no group twice and ends bitwise equal to a run that
never stopped.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@michaellchung

michaellchung commented Oct 15, 2026

Copy link
Copy Markdown
Contributor Author

Superseded by the stacked draft PRs #262 (cold start) → #263 (off-policy correction) → #264 (identity tiering) → #265 (start/stop) → #266 (TB2/27B run support). Each implementation PR carries its own openspec change. Keeping this one open for reference until the stack is reviewed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@michaellchung @chowhsu