Repository navigation
Bind algorithm and data correctly for full-parameter islands; SecRLEnv on ports - #253
Merged
Merged
Conversation
This was referenced Oct 14, 2026
michaellchung
force-pushed
the
s23-pr-11-remove-legacy-engine
branch
from
October 14, 2026 11:31
0cb2801 to
5637d41
Compare
michaellchung
changed the base branch from
s23-pr-11-remove-legacy-engine
to
main
October 14, 2026 11:32
Islands may train on their own task lists (P3 per-island tasks), so the contract the full-parameter islands compare no longer carries --data, its revision or the prompt file hash, and --prompt-data joins the island-local Miles flags. Schema v4 -> v5; old test runs do not resume (P0-D5). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The island contract already carries the AlgorithmSpec hash (P0-4.8), but the full-parameter role streams bound only the backend identity, so two islands with different algorithms could share a dense or SAO session. RoleStreamConfig now requires algorithm_spec_sha256 and the session contract hashes it (schema and domain v2 -> v3). The dense sweep and both SAO roles pass the island's spec hash; the SAO role pairing is checked at bind time, once that hash is known (still before Ray starts). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
StrictRlBridge.run/_run_round were the legacy engine's island loop; the ports IslandDriver drives the bridge through engine.bridges.StrictAvgSync at each boundary, and the legacy decoupled loop went with the legacy engine. The bridge's IslandRuntime now lists only what the bridge calls (initialize, apply_global_policy, record_local_round). The protocol tests that need a self-driven fake island use tests/strict_bridge_loop.py. rl/bridge.py stays the strict-avg protocol client (PULL/PUSH/BCAST, round audit); engine/bridges.py holds the sync sessions that use it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
P0-1.7b.4-Q1: with legacy removed, the SecRLEnv group filter (yeto.rl.harness.codex.reward.check_group) was refused by the ports AlgorithmSpec, so SecRLEnv agents could not train. - algorithm: neutral filter name secrlenv_group_check; the dotted path normalizes to it; it is a supported and expressible filter. - binding: map the name to the dotted path; the fixed-input call check is skipped for it (it needs signed episode evidence), argv and identity checks still run. - Miles capabilities declare it (not yet run on GPU on ports). - launcher: compute the expected spec hash after the agent contract binds its filter and over-sampling count, so it matches the island's. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
P0-1.7b.4-Q1: move the same-task infrastructure retry of the removed legacy rollout to the ports engine, with the same contract (exact task batch, one reserved refill slot, one retry per round, authenticated 503-retry ledger, sanitized terminal errors, drain before raising). - secrlenv_retry: Miles-independent data-source mixin, argv binding and terminal cleanup. - secrlenv_miles: the Miles data source and rollout function classes (--data-source-path, --rollout-function-path); the round's retry counters become rl/secrlenv/* rollout metrics. - config: SecRLEnv agents and the group check must go together, with over-sampling = batch + 1 and no policy-age limit; the SecRLEnv generate wrapper reads the agent flags like the upstream one. - reward / codex_generate: mark a restored or infrastructure sample FAILED instead of ABORTED. Pinned Miles drops a group with an aborted sample before the dynamic filter runs, so the group check would never schedule the retry; FAILED still never trains. Generate pin updated. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Stop the round's group tasks on a terminal error: after Miles' abort, cancel and await the generate_and_rm_group tasks the exited rollout loop left on the event loop, then check for episode residue (one timeout bounds the whole cleanup). - Raise the round's first terminal error: Miles swallows an exception inside a generate task, so the round later failed on the refill instead. The reward function and the generate wrapper record the first UntrustedOutcome on args; the rollout function raises it (sanitized). The record is cleared at each round start. Generate pin updated. - Drop the aborted-group counter and the "secrlenv_aborted" reason: SecRLEnv samples are marked FAILED, and Miles drops aborted groups before the group check. - One status-flattening helper (reward.sample_statuses) for the group check and the retry data source. - Tests: a driver shaped like generate_rollout_async (swallowed task error, refill without retry, stuck group tasks); regression for the spec hash computed before the contract bound its filter. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
michaellchung
marked this pull request as ready for review
October 14, 2026 11:32
michaellchung
force-pushed
the
s23-pr-12-secrlenv-ports
branch
from
October 14, 2026 11:32
a5c72fb to
defb6cc
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Restores SecRLEnv training on ports (group check + same-task retry), binds the AlgorithmSpec hash into full-parameter role-stream sessions, drops training data from the full-parameter contract, and leaves one island loop.
Motivation
P0-1.7b.4-Q1 / P0-1.7b.4b; P3 needs per-island data; rl-code-quality-cleanup 2.3.
What changed
--data, its revision or prompt hash (schema v4 -> v5).RoleStreamConfig.algorithm_spec_sha256required; session contract schema/domain v2 -> v3.StrictRlBridgeself-driven loop removed; portsIslandDriveris the only island loop.secrlenv_group_checkinengine/algorithm.py; Miles binding/capabilities accept it.secrlenv_retry.py(neutral mixin) +secrlenv_miles.py(Miles data source and rollout function): exact task batch, one refill slot, one retry per round, authenticated 503 ledger, drain before raising.UntrustedOutcome, drop the aborted-group counter.Behavior changes / compatibility
aborted_groupsis gone; such groups count asfailed_groups; reasonsecrlenv_abortedremoved.secrlenv_group_checkis declared in Miles capabilities but not yet run on GPU on ports (scheduled in P0-1.11 GPU run).Tests
Local CPU tests: every test file this PR adds or changes: 302 passed, 1 skipped in 59.37s. Full suite with cargo (Rust syncer tests): 11 failed, the same 11 as on s23-final.
Known expected failures until the ports image is rebuilt for the new Miles pin (P0-1.11 GPU run):
test_rl_ports_image(2),test_rl_backend_identity::test_golden_samples_keep_both_old_hashes_and_carry_the_identity, and 8test_decoupling_goldensamples (golden is regenerated after the rebuild).Depends on
#252.
Not verified on GPU
None of this PR has run on GPU yet. The regression GPU run for the S23 stack is P0-1.11 GPU run (in progress at the time of writing); results will be linked here before merge.
🤖 Generated with Claude Code