Skip to content

Batch the chunk requests of environments that request on the same step - #1674

Draft
bbutin-nvidia wants to merge 1 commit into
mainfrom
bbutin/inference_batching
Draft

bbutin-nvidia wants to merge 1 commit into
mainfrom
bbutin/inference_batching

Conversation

@bbutin-nvidia

Copy link
Copy Markdown
Collaborator

Send the chunk requests of all environments that need a chunk on the same step as one batched request, and add the openpi server that answers them. Add batch_chunk_fetches, max_batch_size and batch_sizes to the client configs, and the -b, -s and -u flags to run_openpi_server.sh.

Why this change?

The openpi wire protocol carries one observation per request. RemoteChunkReplayPolicy, shared by Pi0RemotePolicy and CosmosRemotePolicy, therefore sent one websocket request per environment on every step on which chunks ran out, and the server ran the model once per request. Each request pays the websocket round trip, msgpack encoding, the input transforms and a model call for a batch of one. With 100 environments and a 15-step chunk about seven environments request on a typical step, and all of them on the first step and after a synchronized reset.

Measured on OSMO L40S nodes with pi0.5 on two RoboLab tasks at 100 envs (100 and 300 episodes), with finished environments parked (#1673): batching to a fixed size of 8 cut inference time 22 to 74 % against one request per environment and made 7 to 14 times fewer server calls.

What changes?

  • isaaclab_arena_openpi/server/serve_batched_policy.py is the batching server. It runs inside the existing openpi server image next to upstream's scripts/serve_policy.py, accepts the same arguments plus --max-batch-size and --batch-sizes, and wraps the openpi Policy: a request of the form {"batch": [obs, ...]} applies the input transforms per observation, calls sample_actions once on the stacked batch and applies the output transforms per observation. Single-observation requests pass through unchanged. The server advertises batch_inference, max_batch_size and batch_sizes in its connect metadata.
  • JAX compiles the model once per batch shape (a few minutes for pi0.5), so a batch is padded by repeating its last observation: to the next power of two up to --max-batch-size by default, or to the smallest of the fixed shapes given with --batch-sizes 8 16.
  • RemoteChunkReplayPolicy.get_action collects the environments whose chunk ran out and, when the server advertises batch_inference, sends them as batched requests of at most the server's or the config's limit. Against any other server, such as upstream openpi's or the Cosmos server, each environment keeps its own request. The policy counts chunk requests and server calls and prints both when it closes.
  • Pi0RemotePolicyCfg and CosmosRemotePolicyCfg gain batch_chunk_fetches (default true) and max_batch_size (default None, the server's limit). Pi0RemotePolicyCfg also gains batch_sizes, which the OSMO server task passes to the server.
  • run_openpi_server.sh mounts the server script and serves it by default. -b <size> sets the batch limit (32), -s 8,16 sets fixed shapes, -u serves upstream's script.
  • Pi0ServerTask on OSMO embeds the script into the server container through the task files and runs it with the client's max_batch_size (32 when unset) and batch_sizes, so submitted Experiments get batched inference without a new server image. max_batch_size: null on the server task serves upstream's script.

Fixed shapes are only worth it when the number of environments requesting per step is close to the batch size. In the same OSMO benchmark, 20-env runs with a fixed batch of 8 or 16 were 67 to 300 % slower on inference, because one or two real requests per step were padded to a full batch. The docs say so; the default power-of-two padding does not have this problem.

Usage example

Before:

./isaaclab_arena_openpi/docker/run_openpi_server.sh          # one observation per request

After:

./isaaclab_arena_openpi/docker/run_openpi_server.sh          # batched server, limit 32, power-of-two padding
./isaaclab_arena_openpi/docker/run_openpi_server.sh -s 8,16  # pad every batch to 8 or 16
./isaaclab_arena_openpi/docker/run_openpi_server.sh -u       # upstream serve_policy.py
policy:
  type: isaaclab_arena_openpi.policy.pi0_remote_policy.Pi0RemotePolicy
  max_batch_size: 16          # cap the environments per request
  batch_sizes: [8, 16]        # OSMO: start the server with fixed shapes
  batch_chunk_fetches: false  # one request per environment, as before

Validation

  • isaaclab_arena_openpi/tests/test_serve_batched_policy.py (10 tests, fake JAX policy): padding to powers of two and to fixed sizes, stack/unstack round trip with padding, pass-through of single requests, one model call per batched request, splitting above the limit, metadata.
  • isaaclab_arena_openpi/tests/test_pi0_remote_policy.py (6 new): one request for 16 envs, splitting at the server limit (8, 8, 4) and at the config limit (4, 4, 2), only the reset envs request after a partial reset, batching disabled by config, and one request per env against a server without the metadata.
  • isaaclab_arena/tests/test_osmo_experiment_workflow.py: the server task embeds the script and runs it with the client's limit and shapes; max_batch_size: null serves upstream's script. 69 tests pass. Cosmos tests pass. pre-commit clean.
  • End to end: the local RTX 6000 Ada and OSMO L40S benchmark runs behind the numbers above used this server and client, with finished environments parked. Success rates matched the unbatched runs within the run-to-run noise.
  • This change conflicts trivially with Add a flag for the GPU memory the pi0 server reserves #1672 (run_openpi_server.sh) and Stop requesting chunks for envs that finished their last episode #1673 (RemoteChunkReplayPolicy.get_action); whichever lands second needs a small rebase.

The openpi wire protocol carries one observation per request, so
RemoteChunkReplayPolicy sent one websocket request per environment on every
step on which chunks ran out and the server ran the model once per request.
The policy now packs the observations of every environment that needs a
chunk on a step into one {"batch": [obs, ...]} request whenever the connected
server advertises batch_inference in its connect metadata, and falls back to
one request per environment otherwise, so upstream openpi's serve_policy.py
and the Cosmos server keep working unchanged.

isaaclab_arena_openpi/server/serve_batched_policy.py is the batching server.
It runs inside the openpi server image next to upstream's serve_policy.py,
accepts the same arguments plus --max-batch-size and --batch-sizes, applies
the input transforms per observation, samples actions once for the stacked
batch and applies the output transforms per observation. Batches are padded
to the next power of two by default, or to the smallest listed fixed shape,
since JAX compiles once per batch shape. run_openpi_server.sh serves it by
default (-b batch limit, -s fixed shapes, -u upstream script). The OSMO pi0
server task embeds the script into the server container and starts it with
the client's max_batch_size and batch_sizes.

Pi0RemotePolicyCfg and CosmosRemotePolicyCfg gain batch_chunk_fetches
(default true) and max_batch_size; Pi0RemotePolicyCfg also gains batch_sizes
for the OSMO server. The policy prints, when it closes, how many chunk
requests it sent in how many server calls.

Signed-off-by: Benjamin Butin <bbutin@nvidia.com>
@sattyamjjain

Copy link
Copy Markdown

The 100-env numbers look worth it, and flagging the 20-env slowdown is useful. One question on outputs: does the batching OpenPI server return the same chunk for an observation whether it is batched with 7 others or sent alone? In bf16, batch size can shift the numerics slightly, so a quick check of success counts on the same seeds with fixed batch 8 against batch 1 would show whether rates from the two paths can be compared.

@bbutin-nvidia

bbutin-nvidia commented Nov 17, 2026 •

Copy link
Copy Markdown
Collaborator Author

The 100-env numbers look worth it, and flagging the 20-env slowdown is useful. One question on outputs: does the batching OpenPI server return the same chunk for an observation whether it is batched with 7 others or sent alone? In bf16, batch size can shift the numerics slightly, so a quick check of success counts on the same seeds with fixed batch 8 against batch 1 would show whether rates from the two paths can be compared.

From my understanding, the pi0.5 introduces its own random noise, so even right now if you send twice the same observation you can get a different result. So the exact success rate is probably not a good metric to look at.

However according to claude, you are right that the batching can introduce an additional change:


Where batch size can change numerics. Batch elements are independent in the network, so there is no cross-sample mixing, and padding repeats a real observation rather than zeros, so no garbage rows. The only effect is XLA picking different matmul kernels and reduction orders per batch shape. The premise is right that this is in bf16, since openpi runs the model in bfloat16 by default, so the drift is larger than it would be in fp32. It is the same class of difference as serving the unbatched model on a different GPU or JAX version, and it is far smaller than the sampling noise the policy injects on purpose.


The experiments I ran did not show a meaningful difference in `successrate:

  ┌────────────────┬────────────────────┬─────────┬──────────┬────────────┐
  │ task, episodes │ unbatched (parked) │ batch 8 │ batch 16 │ batch 8/16 │
  ├────────────────┼────────────────────┼─────────┼──────────┼────────────┤
  │ banana, 100 ep │ 0.86               │ 0.88    │ 0.82     │ 0.92       │
  ├────────────────┼────────────────────┼─────────┼──────────┼────────────┤
  │ banana, 300 ep │ 0.84               │ 0.87    │ 0.89     │ 0.80       │
  ├────────────────┼────────────────────┼─────────┼──────────┼────────────┤
  │ sauce, 100 ep  │ 0.38               │ 0.44    │ 0.38     │ 0.35       │
  ├────────────────┼────────────────────┼─────────┼──────────┼────────────┤
  │ sauce, 300 ep  │ 0.36               │ 0.40    │ 0.38     │ 0.35       │
  └────────────────┴────────────────────┴─────────┴──────────┴────────────┘

num_envs = env.unwrapped.num_envs
self._maybe_init_per_env_state(num_envs)

requesting_env_ids = [

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Non-blocking. I see generalization capability in fetching the chunks on demand. Depend on you to do followup MRs for my suggestions.

Fetching chunks only for the envs whose chunk ran out is the same job ActionChunkScheduler (isaaclab_arena/policy/action_scheduling/) does for GR00T. Right now the two paths are implemented separately. Can we refactor them to unify?

GR00T has the opposite problem from the one this PR fixes. Its scheduler runs inference for all num_envs whenever any env runs out, then throws away the results for envs that didn't need one. When envs are out of sync, most of each GR00T call is wasted.

In a follow-up, could this move into the shared scheduler? The scheduler would pass the "needs a new chunk" mask to the fetch callback, and callers would fetch only those envs. Then:

  • We have one implementation for the same core logic
  • GR00T would send only the envs that need a chunk. Its server can accept any batch size, so it needs no padding.
  • RemoteChunkReplayPolicy could use ActionChunkScheduler instead of its own _cached_action_chunks / _next_chunk_steps lists. The new chunk_requests / server_calls counters could come from the scheduler's existing fetch statistics.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants