Repository navigation
rl-offpolicy-correction-passthrough: neutral correction group, Miles/verl mapping, hashed (L6b) - #263
Draft
michaellchung wants to merge 13 commits into
Draft
michaellchung wants to merge 13 commits into
michaellchung wants to merge 13 commits into
Conversation
…f the phase-1 hash Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit b8eb808)
…rd and compare per island Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit 25d07c1)
…tion tests Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit 5976f1b)
… in the driver Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit 3ffea04)
…(D8) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit 69a2722)
- Add yeto/rl/adapters/verl/offpolicy_correction.py (role offpolicy_correction): design D3 rows as algorithm.rollout_correction overrides, prox_anchor first_local_step, bypass_mode=True refused, warnings for unmappable fields and for the MIS-mask denominator. - fully_async_plan replaces the rollout_correction overrides with the mapping (decoupled PPO: bypass_mode=False, rollout_is=token, threshold=C). - RoundCollector fills the offpolicy_gate carry-over keys (trajectory granularity, whole-group drop over A_hard); the driver builds the event. - CPU parity tests load the verl fork and Miles correction functions from source: TIS (lower 0) and IcePop weights match; verl rejection sampling shrinks the loss mask; actor.use_rollout_log_probs has no reader. - Example specs decoupled-ppo-c2 (A_hard 2) and -agentic (A_hard 4). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit da796bb)
RolloutBatchHandle gains a backend field; the Miles and verl adapters fill
it, and the driver labels offpolicy_gate with it ('unknown' when unset)
instead of assuming 'miles'.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
(cherry picked from commit 8e5884a)
The fully_async verl island now appends correction_record (group, verl values, prox_anchor) and its correction_config_warning events to the island tape, as the Miles island entry does (D8). Default specs leave the tape unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit d3a423a)
_carry_over_fields whitelisted the carry-over keys but not the gate inputs (dropped lengths, granularity, truncation by turn), so a real Miles island never taped offpolicy_gate (found on GPU, L6.10). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit 8dec16e)
Under --fully-async Miles drops stale groups inside its own buffer; yeto saw them only as stale_dropped_groups in the carry-over. Report them in offpolicy_gate under a separate reason, backend_stale_dropped, so they stay apart from the yeto gate drops (absent when Miles gives no count). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit 439705e)
rl-offpolicy-correction-passthrough phase 2. AlgorithmSpec.sha256() now hashes the correction group as written; a default group keeps the pre-D1 form, so default specs keep their hash. A correction_config_warning (unmappable field, backend without a mapping, island difference) now refuses the launch, and each island refuses the fields its backend cannot map after the record is on the tape. Golden files are regenerated with the version-boundary commit. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit 6d3e72f9bb014c22b0cf39d7362db4a3d2a16eba)
6d3e72f9 puts the off-policy correction group back into AlgorithmSpec.sha256(). On the integration branch the golden samples were regenerated later, together with island contract v3 (rl-identity-tiering, 4702dfbe), and hash-migration.md records both under the phase 2 version boundary entry. This stacked PR lands L6b before that, so regenerate grpo_tis.json here (algorithm hash and execution-profile contract hash only) with python tests/decoupling_golden.py --write. The other 7 samples are unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Spec for openspec change rl-offpolicy-correction-passthrough, taken from s24-sampler-spec 7bd416e2 (PR #258). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This was referenced Oct 15, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on PR-1. Implements openspec change
rl-offpolicy-correction-passthrough(spec as the last commit).level,lower,upper,veto,batch_normalize,ratio_mask) inAlgorithmSpec; Miles and verl adapters map them, refuse decoupled-PPO breakers, and record/compare them per island.offpolicy_gatebuilt in the driver;max_policy_age(cap 4); monitoring.grpo_tis.jsonfor the L6b hash. On the integration branch this regeneration happened later, together with island contract v3;hash-migration.mdrecords it under the phase 2 boundary entry that PR-4 adds.GPU evidence: L6.11 (verl single island) and L6.10.
🤖 Generated with Claude Code