Skip to content

Add a flag for the GPU memory the pi0 server reserves - #1672

Open
bbutin-nvidia wants to merge 3 commits into
mainfrom
bbutin/expose_gpu_usage
Open

bbutin-nvidia wants to merge 3 commits into
mainfrom
bbutin/expose_gpu_usage

Conversation

@bbutin-nvidia

Copy link
Copy Markdown
Collaborator

Add a -m flag to run_openpi_server.sh that sets how much GPU memory the pi0 server reserves, as a fraction (-m 0.75) or an absolute size (-m 12GB). The default stays half the GPU, and the OSMO pi0 server task takes the same setting from the client policy.

Why this change?

The openpi server starts JAX with XLA_PYTHON_CLIENT_MEM_FRACTION=0.5, hardcoded in both run_openpi_server.sh and Pi0ServerTask. Half the GPU is a reasonable split when a simulation shares the card, but it is the wrong amount in other setups: a server alone on a 24 GB card wants more headroom for larger batches, a server next to a 100-env camera rollout on a 48 GB card wants less. Changing it today means editing the script.

What changes?

  • isaaclab_arena_openpi/server/gpu_memory_fraction.sh is a small shell helper that converts a reservation into the fraction JAX reads. A fraction in (0, 1] passes through. A size such as 12GB, 12GiB, 12G, 6000MB, 6000MiB or 6000M (GB and MB are read as GiB and MiB, the units nvidia-smi reports) is divided by the total memory of the first visible GPU, so 12GB reserves 12 GiB on any card. Values that are zero, above 1, above the GPU's memory, or unitless are rejected.
  • run_openpi_server.sh -m <memory> sources that helper and passes the resulting fraction into the container. Without -m the behaviour is unchanged (0.5).
  • Pi0ServerTaskCfg.gpu_memory (default "0.5") carries the same value on OSMO. The task validates the format at submit time, embeds the helper into the server container next to the entry script, and the entry script runs it on the node, so an absolute size is converted against the GPU the server actually lands on.
  • Pi0RemotePolicyCfg.server_gpu_memory (default None) lets an Experiment Definition set the server's reservation, the same way policy_variant already selects the served checkpoint. Pi0ServerTask.task_cfg_for_policy forwards it.

Usage example

Before:

# edit XLA_PYTHON_CLIENT_MEM_FRACTION=0.5 in run_openpi_server.sh, then
./isaaclab_arena_openpi/docker/run_openpi_server.sh

After:

./isaaclab_arena_openpi/docker/run_openpi_server.sh -m 0.75   # 75 % of the GPU
./isaaclab_arena_openpi/docker/run_openpi_server.sh -m 12GB   # 12 GiB
# Experiment Definition submitted to OSMO
policy:
  type: isaaclab_arena_openpi.policy.pi0_remote_policy.Pi0RemotePolicy
  server_gpu_memory: 12GB

Validation

  • isaaclab_arena_openpi/tests/test_gpu_memory_fraction.py runs the helper with a fake nvidia-smi reporting a 48 GB GPU: fractions pass through, sizes divide by the total, and out-of-range or malformed values exit non-zero. isaaclab_arena/tests/test_osmo_experiment_workflow.py checks that the client's server_gpu_memory reaches the OSMO entry script and that the helper is embedded, and that Pi0ServerTaskCfg rejects invalid values. 96 tests in the OSMO module and 23 in the helper module pass on the host.
  • Ran the helper on this machine (RTX 6000 Ada, 49140 MiB): 0.75 -> 0.75, 12GB -> 0.2501, 6000MB -> 0.1221; 100GB, 1.5 and abc are rejected.
  • pre-commit clean.
  • Not verified: an actual server start with -m through Docker, and an OSMO submission with server_gpu_memory; both paths only change the value of the environment variable the server already reads.

run_openpi_server.sh -m sets the memory JAX reserves on the server, either as a
fraction of the GPU (-m 0.75) or as an absolute size (-m 12GB, read as GiB);
the default stays half the GPU. A shared shell helper,
isaaclab_arena_openpi/server/gpu_memory_fraction.sh, converts the value into
XLA_PYTHON_CLIENT_MEM_FRACTION against the GPU it runs on. The OSMO pi0 server
task embeds that helper and applies the same setting from the client policy's
new server_gpu_memory field.

Signed-off-by: Benjamin Butin <bbutin@nvidia.com>

@alexmillane alexmillane left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks mainly good. However we add a setting the pi0 client config which can generate confusing behaviour when running locally. I think we need to do something a bit different there.

Comment thread osmo/tasks/pi0_server_task.py Resolved
Comment thread osmo/tasks/pi0_server_task.py Resolved
Drop the OSMO pi0 server task setting and the client-side server_gpu_memory
field: the client never read it, so it was silently ignored on local runs.
Drop the docs section as well; the launcher's help text documents the flag.

Signed-off-by: Benjamin Butin <bbutin@nvidia.com>
@bbutin-nvidia
bbutin-nvidia marked this pull request as ready for review November 18, 2026 17:28
@greptile-apps

greptile-apps Bot commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

RetriggerView in GreptileConfidence Score: 4/5

[Medium impact] The change reaches users starting openpi through the local Docker launcher.

The PR appears safe to merge, with a non-blocking concern for machines with different-sized GPUs.

Findings

  1. P2 GPU sizes can be mismatched ▶

Summary

Adds a configurable GPU-memory reservation to the local OpenPI launcher while keeping the default at half the GPU.

  • The local server launcher lets operators choose its GPU memory reservation.

bbutin-nvidia intentionally removed the OSMO settings in the final commit because the client did not read them. The remaining non-blocking concern is matching the queried GPU to the GPU used by the server.

Reviews (1) · Last reviewed commit: "Limit the GPU memory flag to the local o..." · Reviewed by Greptile

G|GB|GiB) mebibytes=$(awk -v a="$amount" 'BEGIN { printf "%.3f", a * 1024 }') ;;
*) mebibytes="$amount" ;;
esac
total_mebibytes="$(nvidia-smi --query-gpu=memory.total --format=csv,noheader,nounits | head -n 1 | tr -d '[:space:]')"

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 GPU sizes can be mismatched. This query uses the host's first GPU, but the launcher exposes every GPU without selecting the same device for JAX. On a machine with different-sized GPUs, CUDA's device order need not match nvidia-smi's order. A 12GB request can therefore reserve a different amount on the card running the model.

Tie the query and the server to the same GPU, or reject absolute sizes when the visible GPUs have different memory totals.

echo "GPU memory $memory must be above 0 and at most the GPU's $total_mebibytes MiB" >&2
return 1
fi
awk -v m="$mebibytes" -v t="$total_mebibytes" 'BEGIN { printf "%.4f\n", m / t }'

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Comma decimal in some locales

awk's printf uses the LC_NUMERIC locale, and Ubuntu's default mawk does too. With a comma-decimal locale such as LC_NUMERIC=en_DK.utf8, -m 12GB passes XLA_PYTHON_CLIENT_MEM_FRACTION=0,2501 to the container. I tested JAX 0.5.3 on a GPU machine: it can't parse that value, logs a CUDA plugin error and falls back to CPU. The server starts, but it runs pi0 on CPU. Could we set LC_ALL=C on this call and on the awk call on line 28?

Suggested change
awk -v m="$mebibytes" -v t="$total_mebibytes" 'BEGIN { printf "%.4f\n", m / t }'
LC_ALL=C awk -v m="$mebibytes" -v t="$total_mebibytes" 'BEGIN { printf "%.4f\n", m / t }'

# Usage:
# bash gpu_memory_fraction.sh 12GB # prints 0.2501 on a 48 GB GPU
# source gpu_memory_fraction.sh && gpu_memory_fraction 0.75
set -euo pipefail

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Sourcing changes the caller's shell options

The header suggests source gpu_memory_fraction.sh && gpu_memory_fraction 0.75. Sourcing runs this line, which turns on errexit and nounset in the caller's shell. In an interactive terminal, the next failing command (for example gpu_memory_fraction 2) then closes it. run_openpi_server.sh already sets these options itself. Could this line move inside the if [[ "${BASH_SOURCE[0]}" == "$0" ]] block at the bottom?

G|GB|GiB) mebibytes=$(awk -v a="$amount" 'BEGIN { printf "%.3f", a * 1024 }') ;;
*) mebibytes="$amount" ;;
esac
total_mebibytes="$(nvidia-smi --query-gpu=memory.total --format=csv,noheader,nounits | head -n 1 | tr -d '[:space:]')"

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 head -n 1 can end the standalone run with no message

With pipefail and errexit on, head can exit before nvidia-smi writes the lines for the other GPUs. nvidia-smi then gets SIGPIPE, and bash gpu_memory_fraction.sh 12GB exits 141 with no output. I reproduced this with a fake 8-GPU nvidia-smi that writes one line at a time. The launcher's $(...) call isn't affected. Asking nvidia-smi for GPU 0 directly removes the head call:

Suggested change
total_mebibytes="$(nvidia-smi --query-gpu=memory.total --format=csv,noheader,nounits | head -n 1 | tr -d '[:space:]')"
total_mebibytes="$(nvidia-smi --id=0 --query-gpu=memory.total --format=csv,noheader,nounits | tr -d '[:space:]')"

PORT="8000"
FORCE_REBUILD=false
GPU_MEMORY="0.5"
SERVER_DIR="$(cd "${SCRIPT_DIR}/../server" && pwd)"

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Does the helper still need a server/ directory?

The new isaaclab_arena_openpi/server/ directory made sense when the OSMO task copied this helper into the server container. Since bcfd8bd, run_openpi_server.sh is its only caller. Could it move next to the launcher as docker/gpu_memory_fraction.sh, be sourced from ${SCRIPT_DIR}, and let this SERVER_DIR line go?

@arena-review-bot

Copy link
Copy Markdown
Contributor

🤖 Isaac Lab-Arena Review Bot

Summary

This PR adds -m to run_openpi_server.sh, so the local openpi server's JAX memory reservation can be a fraction (0.75) or an absolute size (12GB). A new gpu_memory_fraction.sh helper does the conversion, and a pytest file covers it. The default (0.5) is unchanged and never calls nvidia-smi. The OSMO and client-config parts from the earlier review were removed in bcfd8bd, so those threads no longer apply. The main problem is that a size can come out with a comma decimal under some locales, and JAX then quietly falls back to CPU.

The PR description still describes the removed parts (Pi0ServerTaskCfg.gpu_memory, Pi0RemotePolicyCfg.server_gpu_memory, the OSMO test, the server_gpu_memory YAML example). Please update it to match the current three-file change.

Findings

🟡 Warning: isaaclab_arena_openpi/server/gpu_memory_fraction.sh:40: awk's printf uses the LC_NUMERIC locale. With a comma-decimal locale, -m 12GB produces 0,2501. JAX 0.5.3 can't parse that, logs a CUDA plugin error and serves on CPU. Setting LC_ALL=C here and on line 28 fixes it.

🔵 Improvement: isaaclab_arena_openpi/server/gpu_memory_fraction.sh:12: the header documents sourcing the helper, but sourcing it turns on set -euo pipefail in the caller's shell. This line could move into the run-as-a-script block.

🔵 Improvement: isaaclab_arena_openpi/server/gpu_memory_fraction.sh:31: with pipefail on, | head -n 1 can make a standalone run exit 141 with no message on multi-GPU hosts. Querying nvidia-smi --id=0 avoids the pipe.

🔵 Improvement: isaaclab_arena_openpi/docker/run_openpi_server.sh:27: now that the OSMO task no longer copies the helper into the server container, it could live in docker/ next to its only caller.

Test Coverage and CI Cost

The new cases are cheap: one bash subprocess each, with no simulator. However, no pytest step in .github/workflows/ci.yml or uv-ci.yml includes isaaclab_arena_openpi/tests/, so these 23 cases only run when someone runs them by hand. The existing openpi tests are in the same situation. The new cases need only bash and awk, so they would fit the Phase 1 step. That is a workflow change, so it's the maintainers' decision.

I ran the helper locally with a fake nvidia-smi, and checked JAX's handling of 0,2501 with JAX 0.5.3 on a GPU machine. I did not start the Docker server with -m, and I did not check this PR's CI results.

Verdict

Minor fixes needed

The launcher exposes every GPU to the server, so a size such as 12GB only
converts to one fraction when all visible GPUs report the same total. With
mixed GPUs the helper now asks for a fraction instead.

Signed-off-by: Benjamin Butin <bbutin@nvidia.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants