Repository navigation
Batch the chunk requests of environments that request on the same step - #1674
bbutin-nvidia wants to merge 1 commit into
Conversation
The openpi wire protocol carries one observation per request, so
RemoteChunkReplayPolicy sent one websocket request per environment on every
step on which chunks ran out and the server ran the model once per request.
The policy now packs the observations of every environment that needs a
chunk on a step into one {"batch": [obs, ...]} request whenever the connected
server advertises batch_inference in its connect metadata, and falls back to
one request per environment otherwise, so upstream openpi's serve_policy.py
and the Cosmos server keep working unchanged.
isaaclab_arena_openpi/server/serve_batched_policy.py is the batching server.
It runs inside the openpi server image next to upstream's serve_policy.py,
accepts the same arguments plus --max-batch-size and --batch-sizes, applies
the input transforms per observation, samples actions once for the stacked
batch and applies the output transforms per observation. Batches are padded
to the next power of two by default, or to the smallest listed fixed shape,
since JAX compiles once per batch shape. run_openpi_server.sh serves it by
default (-b batch limit, -s fixed shapes, -u upstream script). The OSMO pi0
server task embeds the script into the server container and starts it with
the client's max_batch_size and batch_sizes.
Pi0RemotePolicyCfg and CosmosRemotePolicyCfg gain batch_chunk_fetches
(default true) and max_batch_size; Pi0RemotePolicyCfg also gains batch_sizes
for the OSMO server. The policy prints, when it closes, how many chunk
requests it sent in how many server calls.
Signed-off-by: Benjamin Butin <bbutin@nvidia.com>
|
The 100-env numbers look worth it, and flagging the 20-env slowdown is useful. One question on outputs: does the batching OpenPI server return the same chunk for an observation whether it is batched with 7 others or sent alone? In bf16, batch size can shift the numerics slightly, so a quick check of success counts on the same seeds with fixed batch 8 against batch 1 would show whether rates from the two paths can be compared. |
From my understanding, the pi0.5 introduces its own random noise, so even right now if you send twice the same observation you can get a different result. So the exact success rate is probably not a good metric to look at. However according to claude, you are right that the batching can introduce an additional change: Where batch size can change numerics. Batch elements are independent in the network, so there is no cross-sample mixing, and padding repeats a real observation rather than zeros, so no garbage rows. The only effect is XLA picking different matmul kernels and reduction orders per batch shape. The premise is right that this is in bf16, since openpi runs the model in bfloat16 by default, so the drift is larger than it would be in fp32. It is the same class of difference as serving the unbatched model on a different GPU or JAX version, and it is far smaller than the sampling noise the policy injects on purpose. The experiments I ran did not show a meaningful difference in `successrate: |
| num_envs = env.unwrapped.num_envs | ||
| self._maybe_init_per_env_state(num_envs) | ||
|
|
||
| requesting_env_ids = [ |
There was a problem hiding this comment.
Non-blocking. I see generalization capability in fetching the chunks on demand. Depend on you to do followup MRs for my suggestions.
Fetching chunks only for the envs whose chunk ran out is the same job ActionChunkScheduler (isaaclab_arena/policy/action_scheduling/) does for GR00T. Right now the two paths are implemented separately. Can we refactor them to unify?
GR00T has the opposite problem from the one this PR fixes. Its scheduler runs inference for all num_envs whenever any env runs out, then throws away the results for envs that didn't need one. When envs are out of sync, most of each GR00T call is wasted.
In a follow-up, could this move into the shared scheduler? The scheduler would pass the "needs a new chunk" mask to the fetch callback, and callers would fetch only those envs. Then:
- We have one implementation for the same core logic
- GR00T would send only the envs that need a chunk. Its server can accept any batch size, so it needs no padding.
- RemoteChunkReplayPolicy could use ActionChunkScheduler instead of its own
_cached_action_chunks / _next_chunk_stepslists. The new chunk_requests / server_calls counters could come from the scheduler's existing fetch statistics.
Send the chunk requests of all environments that need a chunk on the same step as one batched request, and add the openpi server that answers them. Add
batch_chunk_fetches,max_batch_sizeandbatch_sizesto the client configs, and the-b,-sand-uflags torun_openpi_server.sh.Why this change?
The openpi wire protocol carries one observation per request.
RemoteChunkReplayPolicy, shared byPi0RemotePolicyandCosmosRemotePolicy, therefore sent one websocket request per environment on every step on which chunks ran out, and the server ran the model once per request. Each request pays the websocket round trip, msgpack encoding, the input transforms and a model call for a batch of one. With 100 environments and a 15-step chunk about seven environments request on a typical step, and all of them on the first step and after a synchronized reset.Measured on OSMO L40S nodes with pi0.5 on two RoboLab tasks at 100 envs (100 and 300 episodes), with finished environments parked (#1673): batching to a fixed size of 8 cut inference time 22 to 74 % against one request per environment and made 7 to 14 times fewer server calls.
What changes?
isaaclab_arena_openpi/server/serve_batched_policy.pyis the batching server. It runs inside the existing openpi server image next to upstream'sscripts/serve_policy.py, accepts the same arguments plus--max-batch-sizeand--batch-sizes, and wraps the openpiPolicy: a request of the form{"batch": [obs, ...]}applies the input transforms per observation, callssample_actionsonce on the stacked batch and applies the output transforms per observation. Single-observation requests pass through unchanged. The server advertisesbatch_inference,max_batch_sizeandbatch_sizesin its connect metadata.--max-batch-sizeby default, or to the smallest of the fixed shapes given with--batch-sizes 8 16.RemoteChunkReplayPolicy.get_actioncollects the environments whose chunk ran out and, when the server advertisesbatch_inference, sends them as batched requests of at most the server's or the config's limit. Against any other server, such as upstream openpi's or the Cosmos server, each environment keeps its own request. The policy counts chunk requests and server calls and prints both when it closes.Pi0RemotePolicyCfgandCosmosRemotePolicyCfggainbatch_chunk_fetches(default true) andmax_batch_size(default None, the server's limit).Pi0RemotePolicyCfgalso gainsbatch_sizes, which the OSMO server task passes to the server.run_openpi_server.shmounts the server script and serves it by default.-b <size>sets the batch limit (32),-s 8,16sets fixed shapes,-userves upstream's script.Pi0ServerTaskon OSMO embeds the script into the server container through the task files and runs it with the client'smax_batch_size(32 when unset) andbatch_sizes, so submitted Experiments get batched inference without a new server image.max_batch_size: nullon the server task serves upstream's script.Fixed shapes are only worth it when the number of environments requesting per step is close to the batch size. In the same OSMO benchmark, 20-env runs with a fixed batch of 8 or 16 were 67 to 300 % slower on inference, because one or two real requests per step were padded to a full batch. The docs say so; the default power-of-two padding does not have this problem.
Usage example
Before:
./isaaclab_arena_openpi/docker/run_openpi_server.sh # one observation per requestAfter:
Validation
isaaclab_arena_openpi/tests/test_serve_batched_policy.py(10 tests, fake JAX policy): padding to powers of two and to fixed sizes, stack/unstack round trip with padding, pass-through of single requests, one model call per batched request, splitting above the limit, metadata.isaaclab_arena_openpi/tests/test_pi0_remote_policy.py(6 new): one request for 16 envs, splitting at the server limit (8, 8, 4) and at the config limit (4, 4, 2), only the reset envs request after a partial reset, batching disabled by config, and one request per env against a server without the metadata.isaaclab_arena/tests/test_osmo_experiment_workflow.py: the server task embeds the script and runs it with the client's limit and shapes;max_batch_size: nullserves upstream's script. 69 tests pass. Cosmos tests pass.pre-commitclean.run_openpi_server.sh) and Stop requesting chunks for envs that finished their last episode #1673 (RemoteChunkReplayPolicy.get_action); whichever lands second needs a small rebase.