Repository navigation
Stop requesting chunks for envs that finished their last episode - #1673
bbutin-nvidia wants to merge 3 commits into
Conversation
|
Holding finished envs instead of querying the server makes sense for the staggered case. One thing the description does not say: while an env is held, do its termination and success terms still run, and can a held env be counted again after its episode has ended? Since the banana runs use 100 episodes, printing the success count per run with an interval, before and after, would let a reader check the "agree within run-to-run noise" line directly. Also, #1664 makes policies hold the last action for inactive envs too, so it may be worth checking that the two hold paths do not overlap. |
[Medium impact] The changes appear safe to merge; no actionable issues were found. SummaryThe OpenPI and Cosmos clients now skip action requests for environments that have finished their last assigned episode and hold their robots in place instead.
No actionable issues found. Reviews (1) · Last reviewed commit: "Stop requesting chunks for envs that fin..." · Reviewed by Greptile |
| import torch | ||
|
|
||
|
|
||
| def idle_env_ids(env: gym.Env) -> list[int]: |
There was a problem hiding this comment.
🟡 Second way to hold inactive envs
VLMPolicy.get_action (isaaclab_arena_vlm_agent/vlm_agent_policy.py:263-288, from #1648) already finds inactive envs from active_episode_mask, but it holds them with their last action rather than with hold_action_from_action_manager. After this PR, the VLM and chunk-replay policies detect and hold idle envs in different ways. Your helper also handles relative arm terms and envs that never got an episode (VLM still sends one request for those). Could VLMPolicy switch to idle_env_ids and hold_action_from_action_manager, here or in a follow-up? Then there would be one path to reconcile with #1664.
| Args: | ||
| env: Environment, optionally gym-wrapped. | ||
| """ | ||
| active_episode_mask = getattr(env.unwrapped, "active_episode_mask", None) |
There was a problem hiding this comment.
🔵 getattr fallback for active_episode_mask
Every env that rollout_policy steps is an IsaacLabArenaManagerBasedRLEnv, which always defines active_episode_mask. rollout_policy already reads reset_env_ids directly, and VLMPolicy reads the mask directly. The fallback seems to exist only for the _fake_env helpers in the pi0 and Cosmos tests. In production it would quietly turn off the skip if the property were ever renamed. Could we read the property directly, add active_episode_mask to those two fakes, and drop test_idle_env_ids_is_empty_for_an_env_without_episode_tracking and the last sentence of the docstring?
def idle_env_ids(env: gym.Env) -> list[int]:
...
return (~env.unwrapped.active_episode_mask).nonzero(as_tuple=True)[0].tolist()| elif isinstance(term, DifferentialInverseKinematicsAction) and term.cfg.controller.use_relative_mode: | ||
| term_hold_actions.append(torch.zeros_like(term.raw_actions)) | ||
| else: | ||
| term_hold_actions.append(term.raw_actions.clone()) |
There was a problem hiding this comment.
🔵 Catch-all branch doesn't hold every term type
Repeating the last raw action only holds terms that take absolute targets. For a JointVelocityAction, JointEffortAction, relative OperationalSpaceControllerAction or NonHolonomicAction term, the term keeps applying its last command and the robot keeps moving. That contradicts "keeps every env's robot where it is". Only DROID reaches this code today, so nothing breaks now. Could the else branch assert that the term is a type known to hold with its last command (for example BinaryJointAction or an absolute DifferentialInverseKinematicsAction)? A new embodiment would then fail loudly. Alternatively, the docstring could be narrowed to the term types this function supports.
| env.reset() | ||
| hold_action = hold_action_from_action_manager(env) | ||
| assert hold_action.shape == (NUM_ENVS, env.unwrapped.action_manager.total_action_dim) | ||
| assert torch.all(hold_action == 0.0) |
There was a problem hiding this comment.
🟡 Test passes without the relative branch
Right after env.reset(), every term's raw_actions is zero, so the catch-all branch (repeat the last raw action) also returns zeros. This test would still pass if the RelativeJointPositionAction branch were deleted, and it pays for a second maple-table env build. If you step once with a non-zero action first, it checks the real failure, where repeating the last delta keeps the arm moving:
env.reset()
action_dim = env.unwrapped.action_manager.total_action_dim
env.step(torch.full((NUM_ENVS, action_dim), 0.1, device=env.unwrapped.device))
hold_action = hold_action_from_action_manager(env)
assert hold_action.shape == (NUM_ENVS, action_dim)
assert torch.all(hold_action[:, :7] == 0.0), "relative arm terms hold with a zero delta, not the last delta"
🤖 Isaac Lab-Arena Review BotSummaryThis PR stops Design, Boundaries & Scope
Findings🟡 Warning: Test Coverage and CI Cost
VerdictMinor fixes needed |
With an episode limit, the Arena env stops replacing finished episodes once the limit is reached but keeps stepping those envs until the remaining episodes end. RemoteChunkReplayPolicy kept requesting chunks for them; those actions were discarded, so the inference was wasted, a large share of a Run with many envs and few episodes per env. The client now skips envs outside the env's active_episode_mask: it drops their cached chunk, requests nothing and applies the action that holds the robot in place until the Run ends. A reset restarts them. The policy prints the number of held env-steps when it closes. Signed-off-by: Benjamin Butin <bbutin@nvidia.com>
…ask directly Read active_episode_mask without a fallback, since every Arena env defines it, and give the pi0 and Cosmos test fakes the mask. Hold joint velocity and non-holonomic terms at zero, repeat the last command only for binary joint and absolute IK terms, and reject any other term type instead of repeating a command that would keep the robot moving. Step the relative-term test with a non-zero action first so it fails if the relative branch is removed. Signed-off-by: Benjamin Butin <bbutin@nvidia.com>
Use idle_env_ids and hold_action_from_action_manager instead of a private last-action cache, so the VLM and chunk-replay policies share one hold path. An environment that never got an episode now holds from its first step without sending a request. Signed-off-by: Benjamin Butin <bbutin@nvidia.com>
ba47ac5 to
6f1cdd7
Compare
Stop the Cosmos and OpenPI clients from requesting action chunks for environments that finished their last episode and are only waiting for the other environments to finish. Those environments hold their robot in place instead.
Why this change?
A Run with
rollout_limit.num_episodesstops assigning new episodes once the limit is reached, but the simulation keeps stepping every environment until the last assigned episode ends.RemoteChunkReplayPolicy, shared byPi0RemotePolicyandCosmosRemotePolicy, sends one request per environment whenever its chunk runs out, so it kept asking the server for actions that the env then discarded.The waste grows with the number of environments and shrinks with the number of episodes per environment. With 100 envs and 100 episodes every env runs exactly one episode, so an env that succeeds early sits idle for the rest of the slowest episode while still requesting chunks. On the RoboLab banana task (85 % success) that was 53 % of all env-steps, and on OSMO the same run spent 6.0 min in inference, 2.4 min of it with this change.
What changes?
isaaclab_arena/policy/idle_envs.pyaddsidle_env_ids(env), the environments outside the Arena env'sactive_episode_mask, andhold_action_from_action_manager(env), the action that keeps every robot where it is for one step: current joint positions for absolute joint-position terms, zero for relative joint-position and relative IK terms, and the last raw action for anything else such as a binary gripper.RemoteChunkReplayPolicy.get_actionskips the idle environments. It drops their cached chunk, sends no request, and returns the hold action for them. The runner'sreseton those envs clears the state as before, so an env that is assigned a new episode fetches a fresh chunk. Without an episode limit the mask is always full and nothing changes.idle_stepsand prints the total when it closes, so a Run's log shows how much inference was skipped.VLMPolicyswitches from its own last-action cache (Skip VLM requests for environments without an active episode #1648) to the same two helpers, so the VLM and chunk-replay policies share one hold path. One behaviour change: withnum_episodesbelownum_envs, an environment that never gets an episode now holds from its first step and sends no request, where it used to request once.ActionSchedulerrather than to this client.Validation
isaaclab_arena_openpi/tests/test_pi0_remote_policy.py: a new test marks one of two envs idle, checks that only the active env refetches when its chunk runs out, that the idle env gets the hold action and counts 15 held steps, and that a reset restarts its requests. 10 tests pass.isaaclab_arena_vlm_agent/tests/test_vlm_agent_policy.py: the three inactive-env tests patch the hold action and check that idle envs send no request from the first step. 113 VLM tests pass.isaaclab_arena/tests/test_idle_envs.py: two unit tests foridle_env_ids, and two Isaac Sim tests on the maple-table DROID environments checking that the hold action equals the current arm joint positions and keeps the arm within 0.02 rad over five steps fordroid_abs_joint_pos, and is zero fordroid_rel_joint_pos. 4 tests pass natively.pre-commitclean.