Skip to content

Stop requesting chunks for envs that finished their last episode - #1673

Open
bbutin-nvidia wants to merge 3 commits into
mainfrom
bbutin/disable_inference_when_done
Open

bbutin-nvidia wants to merge 3 commits into
mainfrom
bbutin/disable_inference_when_done

Conversation

@bbutin-nvidia

@bbutin-nvidia bbutin-nvidia commented Nov 12, 2026 •

Copy link
Copy Markdown
Collaborator

Stop the Cosmos and OpenPI clients from requesting action chunks for environments that finished their last episode and are only waiting for the other environments to finish. Those environments hold their robot in place instead.

Why this change?

A Run with rollout_limit.num_episodes stops assigning new episodes once the limit is reached, but the simulation keeps stepping every environment until the last assigned episode ends. RemoteChunkReplayPolicy, shared by Pi0RemotePolicy and CosmosRemotePolicy, sends one request per environment whenever its chunk runs out, so it kept asking the server for actions that the env then discarded.

The waste grows with the number of environments and shrinks with the number of episodes per environment. With 100 envs and 100 episodes every env runs exactly one episode, so an env that succeeds early sits idle for the rest of the slowest episode while still requesting chunks. On the RoboLab banana task (85 % success) that was 53 % of all env-steps, and on OSMO the same run spent 6.0 min in inference, 2.4 min of it with this change.

What changes?

  • isaaclab_arena/policy/idle_envs.py adds idle_env_ids(env), the environments outside the Arena env's active_episode_mask, and hold_action_from_action_manager(env), the action that keeps every robot where it is for one step: current joint positions for absolute joint-position terms, zero for relative joint-position and relative IK terms, and the last raw action for anything else such as a binary gripper.
  • RemoteChunkReplayPolicy.get_action skips the idle environments. It drops their cached chunk, sends no request, and returns the hold action for them. The runner's reset on those envs clears the state as before, so an env that is assigned a new episode fetches a fresh chunk. Without an episode limit the mask is always full and nothing changes.
  • The policy counts held env-steps per environment in idle_steps and prints the total when it closes, so a Run's log shows how much inference was skipped.
  • Held environments are stepped as before, so their termination and success terms still run. The env's reset path only records, counts and restarts environments with an active episode, so a held env that terminates again is neither recorded nor counted, and it is never assigned a new episode. This PR does not change that.
  • VLMPolicy switches from its own last-action cache (Skip VLM requests for environments without an active episode #1648) to the same two helpers, so the VLM and chunk-replay policies share one hold path. One behaviour change: with num_episodes below num_envs, an environment that never gets an episode now holds from its first step and sends no request, where it used to request once.
  • GR00T is not covered. Its client sends one batched request for all environments, so skipping envs there means subsetting the batch, which is a change to ActionScheduler rather than to this client.

Validation

  • isaaclab_arena_openpi/tests/test_pi0_remote_policy.py: a new test marks one of two envs idle, checks that only the active env refetches when its chunk runs out, that the idle env gets the hold action and counts 15 held steps, and that a reset restarts its requests. 10 tests pass.
  • isaaclab_arena_vlm_agent/tests/test_vlm_agent_policy.py: the three inactive-env tests patch the hold action and check that idle envs send no request from the first step. 113 VLM tests pass.
  • isaaclab_arena/tests/test_idle_envs.py: two unit tests for idle_env_ids, and two Isaac Sim tests on the maple-table DROID environments checking that the hold action equals the current arm joint positions and keeps the arm within 0.02 rad over five steps for droid_abs_joint_pos, and is zero for droid_rel_joint_pos. 4 tests pass natively.
  • Measured on the OSMO L40S pool with pi0.5 on two RoboLab tasks, 20 and 100 envs, 100 and 300 episodes: inference time drops 7 to 59 % per run where envs finish staggered (banana), and is unchanged where nearly every env runs to the step limit (sauce at 20 envs). Success counts with 95 % intervals, same tasks and seeds, before and after (the policy samples its own noise, so two unbatched runs differ by as much as these do):
run main this change
banana, 20 envs, 100 ep 85/100 (85 ± 7 %) 84/100 (84 ± 7 %)
banana, 20 envs, 300 ep 251/300 (84 ± 4 %) 256/300 (85 ± 4 %)
banana, 100 envs, 100 ep 83/100 (83 ± 7 %) 86/100 (86 ± 7 %)
banana, 100 envs, 300 ep 251/300 (84 ± 4 %) 252/300 (84 ± 4 %)
sauce, 20 envs, 100 ep 52/100 (52 ± 10 %) 40/100 (40 ± 10 %)
sauce, 20 envs, 300 ep 142/300 (47 ± 6 %) 124/300 (41 ± 6 %)
sauce, 100 envs, 100 ep 41/100 (41 ± 10 %) 38/100 (38 ± 10 %)
sauce, 100 envs, 300 ep 98/300 (33 ± 5 %) 108/300 (36 ± 5 %)
  • pre-commit clean.

@sattyamjjain

Copy link
Copy Markdown

Holding finished envs instead of querying the server makes sense for the staggered case. One thing the description does not say: while an env is held, do its termination and success terms still run, and can a held env be counted again after its episode has ended? Since the banana runs use 100 episodes, printing the success count per run with an interval, before and after, would let a reader check the "agree within run-to-run noise" line directly. Also, #1664 makes policies hold the last action for inactive envs too, so it may be worth checking that the two hold paths do not overlap.

@greptile-apps

greptile-apps Bot commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

RetriggerView in GreptileConfidence Score: 5/5

[Medium impact] The changes appear safe to merge; no actionable issues were found.

Summary

The OpenPI and Cosmos clients now skip action requests for environments that have finished their last assigned episode and hold their robots in place instead.

  • Cosmos and OpenPI clients hold finished environments without requesting chunks.

No actionable issues found.

Reviews (1) · Last reviewed commit: "Stop requesting chunks for envs that fin..." · Reviewed by Greptile

import torch


def idle_env_ids(env: gym.Env) -> list[int]:

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Second way to hold inactive envs

VLMPolicy.get_action (isaaclab_arena_vlm_agent/vlm_agent_policy.py:263-288, from #1648) already finds inactive envs from active_episode_mask, but it holds them with their last action rather than with hold_action_from_action_manager. After this PR, the VLM and chunk-replay policies detect and hold idle envs in different ways. Your helper also handles relative arm terms and envs that never got an episode (VLM still sends one request for those). Could VLMPolicy switch to idle_env_ids and hold_action_from_action_manager, here or in a follow-up? Then there would be one path to reconcile with #1664.

Comment thread isaaclab_arena/policy/idle_envs.py Outdated
Args:
env: Environment, optionally gym-wrapped.
"""
active_episode_mask = getattr(env.unwrapped, "active_episode_mask", None)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 getattr fallback for active_episode_mask

Every env that rollout_policy steps is an IsaacLabArenaManagerBasedRLEnv, which always defines active_episode_mask. rollout_policy already reads reset_env_ids directly, and VLMPolicy reads the mask directly. The fallback seems to exist only for the _fake_env helpers in the pi0 and Cosmos tests. In production it would quietly turn off the skip if the property were ever renamed. Could we read the property directly, add active_episode_mask to those two fakes, and drop test_idle_env_ids_is_empty_for_an_env_without_episode_tracking and the last sentence of the docstring?

def idle_env_ids(env: gym.Env) -> list[int]:
    ...
    return (~env.unwrapped.active_episode_mask).nonzero(as_tuple=True)[0].tolist()

elif isinstance(term, DifferentialInverseKinematicsAction) and term.cfg.controller.use_relative_mode:
term_hold_actions.append(torch.zeros_like(term.raw_actions))
else:
term_hold_actions.append(term.raw_actions.clone())

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Catch-all branch doesn't hold every term type

Repeating the last raw action only holds terms that take absolute targets. For a JointVelocityAction, JointEffortAction, relative OperationalSpaceControllerAction or NonHolonomicAction term, the term keeps applying its last command and the robot keeps moving. That contradicts "keeps every env's robot where it is". Only DROID reaches this code today, so nothing breaks now. Could the else branch assert that the term is a type known to hold with its last command (for example BinaryJointAction or an absolute DifferentialInverseKinematicsAction)? A new embodiment would then fail loudly. Alternatively, the docstring could be narrowed to the term types this function supports.

Comment thread isaaclab_arena/tests/test_idle_envs.py Outdated
env.reset()
hold_action = hold_action_from_action_manager(env)
assert hold_action.shape == (NUM_ENVS, env.unwrapped.action_manager.total_action_dim)
assert torch.all(hold_action == 0.0)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Test passes without the relative branch

Right after env.reset(), every term's raw_actions is zero, so the catch-all branch (repeat the last raw action) also returns zeros. This test would still pass if the RelativeJointPositionAction branch were deleted, and it pays for a second maple-table env build. If you step once with a non-zero action first, it checks the real failure, where repeating the last delta keeps the arm moving:

env.reset()
action_dim = env.unwrapped.action_manager.total_action_dim
env.step(torch.full((NUM_ENVS, action_dim), 0.1, device=env.unwrapped.device))
hold_action = hold_action_from_action_manager(env)
assert hold_action.shape == (NUM_ENVS, action_dim)
assert torch.all(hold_action[:, :7] == 0.0), "relative arm terms hold with a zero delta, not the last delta"

@arena-review-bot

Copy link
Copy Markdown
Contributor

🤖 Isaac Lab-Arena Review Bot

Summary

This PR stops RemoteChunkReplayPolicy (pi0 and Cosmos) from requesting chunks for envs outside active_episode_mask. Those envs get a hold action from the new hold_action_from_action_manager instead. The change to get_action is small and correct, and the new pi0 unit test covers the skip, refetch and reset logic cheaply. My main concern is that this adds a second, different way of holding inactive envs alongside the one VLMPolicy already has. Also, one of the two new sim tests passes even without the code path it is meant to check.

Design, Boundaries & Scope

  • Two ways to hold inactive envs. VLMPolicy.get_action (isaaclab_arena_vlm_agent/vlm_agent_policy.py:263-288, merged in Skip VLM requests for environments without an active episode #1648) already reads active_episode_mask and holds inactive envs with their last action. This PR adds idle_env_ids and an action-manager-based hold, used only by RemoteChunkReplayPolicy. The PR's hold covers more cases: relative arm terms, and envs that never got an episode when num_episodes < num_envs (VLM still sends one request for those). That makes it the better candidate for a single shared path. Could VLMPolicy use it too? The overlap with Hold the last action of inactive environments in every policy #1664 was already raised in the earlier comment on this PR, so I won't repeat it here.
  • Who should own the hold action? hold_action_from_action_manager lives in isaaclab_arena/policy/, but it encodes knowledge of each action-term type and reads private Isaac Lab members (_asset, _joint_ids, _offset, _scale). IsaacLabArenaManagerBasedRLEnv owns both active_episode_mask and the action manager. Would it be simpler for the env's step() to replace inactive envs' actions with the hold action? Then every policy, including GR00T and DreamZero, would hold the same way, and policies would only need to skip inference. This is an open question, not a blocker.

Findings

🟡 Warning: isaaclab_arena/policy/idle_envs.py:19: This is a second inactive-env detector and hold strategy next to VLMPolicy's; see the design note above.
🟡 Warning: isaaclab_arena/tests/test_idle_envs.py:78: The relative-joint-position test passes even if the RelativeJointPositionAction branch is deleted, because every raw_actions is zero right after reset.
🔵 Improvement: isaaclab_arena/policy/idle_envs.py:28: The getattr(..., "active_episode_mask", None) fallback only seems to serve the pi0 and Cosmos test fakes. Production envs always define the property.
🔵 Improvement: isaaclab_arena/policy/idle_envs.py:64: For velocity, effort or relative OSC terms, the catch-all branch keeps applying the last command, which contradicts the docstring.

Test Coverage and CI Cost

  • _test_hold_action_is_zero_for_relative_joint_position_terms builds a second maple-table env but doesn't protect the behavior it names (details inline). Stepping once with a non-zero action before taking the hold action fixes that without adding cost.
  • CI limitation: when I reviewed c2a32db, Pre-commit had passed, but the image build was still running and the test jobs had not run. I couldn't confirm that the new sim tests pass in CI.

Verdict

Minor fixes needed

With an episode limit, the Arena env stops replacing finished episodes once
the limit is reached but keeps stepping those envs until the remaining
episodes end. RemoteChunkReplayPolicy kept requesting chunks for them; those
actions were discarded, so the inference was wasted, a large share of a Run
with many envs and few episodes per env.

The client now skips envs outside the env's active_episode_mask: it drops
their cached chunk, requests nothing and applies the action that holds the
robot in place until the Run ends. A reset restarts them. The policy prints
the number of held env-steps when it closes.

Signed-off-by: Benjamin Butin <bbutin@nvidia.com>
…ask directly

Read active_episode_mask without a fallback, since every Arena env defines
it, and give the pi0 and Cosmos test fakes the mask. Hold joint velocity and
non-holonomic terms at zero, repeat the last command only for binary joint
and absolute IK terms, and reject any other term type instead of repeating a
command that would keep the robot moving. Step the relative-term test with a
non-zero action first so it fails if the relative branch is removed.

Signed-off-by: Benjamin Butin <bbutin@nvidia.com>
Use idle_env_ids and hold_action_from_action_manager instead of a private
last-action cache, so the VLM and chunk-replay policies share one hold path.
An environment that never got an episode now holds from its first step
without sending a request.

Signed-off-by: Benjamin Butin <bbutin@nvidia.com>
@bbutin-nvidia
bbutin-nvidia force-pushed the bbutin/disable_inference_when_done branch from ba47ac5 to 6f1cdd7 Compare November 18, 2026 20:11

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants