Repository navigation
Add a flag for the GPU memory the pi0 server reserves - #1672
bbutin-nvidia wants to merge 3 commits into
Conversation
run_openpi_server.sh -m sets the memory JAX reserves on the server, either as a fraction of the GPU (-m 0.75) or as an absolute size (-m 12GB, read as GiB); the default stays half the GPU. A shared shell helper, isaaclab_arena_openpi/server/gpu_memory_fraction.sh, converts the value into XLA_PYTHON_CLIENT_MEM_FRACTION against the GPU it runs on. The OSMO pi0 server task embeds that helper and applies the same setting from the client policy's new server_gpu_memory field. Signed-off-by: Benjamin Butin <bbutin@nvidia.com>
alexmillane
left a comment
There was a problem hiding this comment.
Looks mainly good. However we add a setting the pi0 client config which can generate confusing behaviour when running locally. I think we need to do something a bit different there.
Drop the OSMO pi0 server task setting and the client-side server_gpu_memory field: the client never read it, so it was silently ignored on local runs. Drop the docs section as well; the launcher's help text documents the flag. Signed-off-by: Benjamin Butin <bbutin@nvidia.com>
[Medium impact] The change reaches users starting openpi through the local Docker launcher. The PR appears safe to merge, with a non-blocking concern for machines with different-sized GPUs. Findings
SummaryAdds a configurable GPU-memory reservation to the local OpenPI launcher while keeping the default at half the GPU.
bbutin-nvidia intentionally removed the OSMO settings in the final commit because the client did not read them. The remaining non-blocking concern is matching the queried GPU to the GPU used by the server. Reviews (1) · Last reviewed commit: "Limit the GPU memory flag to the local o..." · Reviewed by Greptile |
| G|GB|GiB) mebibytes=$(awk -v a="$amount" 'BEGIN { printf "%.3f", a * 1024 }') ;; | ||
| *) mebibytes="$amount" ;; | ||
| esac | ||
| total_mebibytes="$(nvidia-smi --query-gpu=memory.total --format=csv,noheader,nounits | head -n 1 | tr -d '[:space:]')" |
There was a problem hiding this comment.
GPU sizes can be mismatched. This query uses the host's first GPU, but the launcher exposes every GPU without selecting the same device for JAX. On a machine with different-sized GPUs, CUDA's device order need not match
nvidia-smi's order. A 12GB request can therefore reserve a different amount on the card running the model.
Tie the query and the server to the same GPU, or reject absolute sizes when the visible GPUs have different memory totals.
| echo "GPU memory $memory must be above 0 and at most the GPU's $total_mebibytes MiB" >&2 | ||
| return 1 | ||
| fi | ||
| awk -v m="$mebibytes" -v t="$total_mebibytes" 'BEGIN { printf "%.4f\n", m / t }' |
There was a problem hiding this comment.
🟡 Comma decimal in some locales
awk's printf uses the LC_NUMERIC locale, and Ubuntu's default mawk does too. With a comma-decimal locale such as LC_NUMERIC=en_DK.utf8, -m 12GB passes XLA_PYTHON_CLIENT_MEM_FRACTION=0,2501 to the container. I tested JAX 0.5.3 on a GPU machine: it can't parse that value, logs a CUDA plugin error and falls back to CPU. The server starts, but it runs pi0 on CPU. Could we set LC_ALL=C on this call and on the awk call on line 28?
| awk -v m="$mebibytes" -v t="$total_mebibytes" 'BEGIN { printf "%.4f\n", m / t }' | |
| LC_ALL=C awk -v m="$mebibytes" -v t="$total_mebibytes" 'BEGIN { printf "%.4f\n", m / t }' |
| # Usage: | ||
| # bash gpu_memory_fraction.sh 12GB # prints 0.2501 on a 48 GB GPU | ||
| # source gpu_memory_fraction.sh && gpu_memory_fraction 0.75 | ||
| set -euo pipefail |
There was a problem hiding this comment.
🔵 Sourcing changes the caller's shell options
The header suggests source gpu_memory_fraction.sh && gpu_memory_fraction 0.75. Sourcing runs this line, which turns on errexit and nounset in the caller's shell. In an interactive terminal, the next failing command (for example gpu_memory_fraction 2) then closes it. run_openpi_server.sh already sets these options itself. Could this line move inside the if [[ "${BASH_SOURCE[0]}" == "$0" ]] block at the bottom?
| G|GB|GiB) mebibytes=$(awk -v a="$amount" 'BEGIN { printf "%.3f", a * 1024 }') ;; | ||
| *) mebibytes="$amount" ;; | ||
| esac | ||
| total_mebibytes="$(nvidia-smi --query-gpu=memory.total --format=csv,noheader,nounits | head -n 1 | tr -d '[:space:]')" |
There was a problem hiding this comment.
🔵 head -n 1 can end the standalone run with no message
With pipefail and errexit on, head can exit before nvidia-smi writes the lines for the other GPUs. nvidia-smi then gets SIGPIPE, and bash gpu_memory_fraction.sh 12GB exits 141 with no output. I reproduced this with a fake 8-GPU nvidia-smi that writes one line at a time. The launcher's $(...) call isn't affected. Asking nvidia-smi for GPU 0 directly removes the head call:
| total_mebibytes="$(nvidia-smi --query-gpu=memory.total --format=csv,noheader,nounits | head -n 1 | tr -d '[:space:]')" | |
| total_mebibytes="$(nvidia-smi --id=0 --query-gpu=memory.total --format=csv,noheader,nounits | tr -d '[:space:]')" |
| PORT="8000" | ||
| FORCE_REBUILD=false | ||
| GPU_MEMORY="0.5" | ||
| SERVER_DIR="$(cd "${SCRIPT_DIR}/../server" && pwd)" |
There was a problem hiding this comment.
🔵 Does the helper still need a server/ directory?
The new isaaclab_arena_openpi/server/ directory made sense when the OSMO task copied this helper into the server container. Since bcfd8bd, run_openpi_server.sh is its only caller. Could it move next to the launcher as docker/gpu_memory_fraction.sh, be sourced from ${SCRIPT_DIR}, and let this SERVER_DIR line go?
🤖 Isaac Lab-Arena Review BotSummaryThis PR adds The PR description still describes the removed parts ( Findings🟡 Warning: isaaclab_arena_openpi/server/gpu_memory_fraction.sh:40: 🔵 Improvement: isaaclab_arena_openpi/server/gpu_memory_fraction.sh:12: the header documents sourcing the helper, but sourcing it turns on 🔵 Improvement: isaaclab_arena_openpi/server/gpu_memory_fraction.sh:31: with 🔵 Improvement: isaaclab_arena_openpi/docker/run_openpi_server.sh:27: now that the OSMO task no longer copies the helper into the server container, it could live in Test Coverage and CI CostThe new cases are cheap: one I ran the helper locally with a fake VerdictMinor fixes needed |
The launcher exposes every GPU to the server, so a size such as 12GB only converts to one fraction when all visible GPUs report the same total. With mixed GPUs the helper now asks for a fraction instead. Signed-off-by: Benjamin Butin <bbutin@nvidia.com>
Add a
-mflag torun_openpi_server.shthat sets how much GPU memory the pi0 server reserves, as a fraction (-m 0.75) or an absolute size (-m 12GB). The default stays half the GPU, and the OSMO pi0 server task takes the same setting from the client policy.Why this change?
The openpi server starts JAX with
XLA_PYTHON_CLIENT_MEM_FRACTION=0.5, hardcoded in bothrun_openpi_server.shandPi0ServerTask. Half the GPU is a reasonable split when a simulation shares the card, but it is the wrong amount in other setups: a server alone on a 24 GB card wants more headroom for larger batches, a server next to a 100-env camera rollout on a 48 GB card wants less. Changing it today means editing the script.What changes?
isaaclab_arena_openpi/server/gpu_memory_fraction.shis a small shell helper that converts a reservation into the fraction JAX reads. A fraction in(0, 1]passes through. A size such as12GB,12GiB,12G,6000MB,6000MiBor6000M(GB and MB are read as GiB and MiB, the unitsnvidia-smireports) is divided by the total memory of the first visible GPU, so12GBreserves 12 GiB on any card. Values that are zero, above 1, above the GPU's memory, or unitless are rejected.run_openpi_server.sh -m <memory>sources that helper and passes the resulting fraction into the container. Without-mthe behaviour is unchanged (0.5).Pi0ServerTaskCfg.gpu_memory(default"0.5") carries the same value on OSMO. The task validates the format at submit time, embeds the helper into the server container next to the entry script, and the entry script runs it on the node, so an absolute size is converted against the GPU the server actually lands on.Pi0RemotePolicyCfg.server_gpu_memory(defaultNone) lets an Experiment Definition set the server's reservation, the same waypolicy_variantalready selects the served checkpoint.Pi0ServerTask.task_cfg_for_policyforwards it.Usage example
Before:
# edit XLA_PYTHON_CLIENT_MEM_FRACTION=0.5 in run_openpi_server.sh, then ./isaaclab_arena_openpi/docker/run_openpi_server.shAfter:
Validation
isaaclab_arena_openpi/tests/test_gpu_memory_fraction.pyruns the helper with a fakenvidia-smireporting a 48 GB GPU: fractions pass through, sizes divide by the total, and out-of-range or malformed values exit non-zero.isaaclab_arena/tests/test_osmo_experiment_workflow.pychecks that the client'sserver_gpu_memoryreaches the OSMO entry script and that the helper is embedded, and thatPi0ServerTaskCfgrejects invalid values. 96 tests in the OSMO module and 23 in the helper module pass on the host.0.75 -> 0.75,12GB -> 0.2501,6000MB -> 0.1221;100GB,1.5andabcare rejected.pre-commitclean.-mthrough Docker, and an OSMO submission withserver_gpu_memory; both paths only change the value of the environment variable the server already reads.