Skip to content

Fix hang in L-island kitchen scenes - #1697

Merged
zhx06 merged 3 commits into
mainfrom
zxiao/fix/deadlock-background-physics-scene
Nov 18, 2026
Merged

zhx06 merged 3 commits into
mainfrom
zxiao/fix/deadlock-background-physics-scene

Conversation

@zhx06

@zhx06 zhx06 commented Nov 18, 2026 •

Copy link
Copy Markdown
Collaborator

Fix a hang when a background USD contains its own PhysicsScene: deactivate those scenes when the background spawns.

Why this change?

kitchen_l_island.usda, used by replicator_kitchen_l_island, has a PhysicsScene under its default prim. Background is spawned into every environment, so 30 environments add 30 PhysX scenes next to the simulation's /physicsScene. The first physics step in PhysxManager._warmup_and_create_views then never returns. Isaac Lab removes existing scenes only when SimulationContext is created, which is before assets are spawned.

This is the hang that made the nightly placement-subset run time out at kitchen_bench_replicator_l_island_bowl_sink (#1696).

Fixes #[issue number]

What changes?

_spawn_from_usd_with_resettable_nested_physics, the spawner Background uses by default, now deactivates the physics scenes under the spawned background before it is cloned. The simulation's /physicsScene is then the only active scene.

Backgrounds created with reset_nested_physics=False keep Isaac Lab's spawner and are not covered. None of them has its own scene today. The asset should still drop its scene.

Validation

  • kitchen_bench_replicator_l_island_bowl_sink and kitchen_bench_replicator_l_island_mustard_bowl with 30 environments hang on main and complete with this change.
  • placement_subset_experiment.yaml completes all 10 Runs and writes the placement KPIs.
  • With one environment, the trajectories are bit-identical to main.
  • Not run on OSMO.

@zhx06 zhx06 changed the title deactivate physics scenes authored in background USDs Deactivate Physics Scenes Authored in Background USDs Nov 18, 2026
@zhx06
zhx06 marked this pull request as ready for review November 18, 2026 00:25
@greptile-apps

greptile-apps Bot commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

RetriggerView in GreptileConfidence Score: 5/5

[Medium impact] The PR appears safe to merge; no actionable issues were found.

Summary

The PR turns off physics scenes embedded in spawned backgrounds to prevent multi-environment simulations from hanging.

  • Background spawns leave their own physics scenes inactive.

zhx06 explicitly leaves backgrounds with reset_nested_physics=False unchanged, noting that none currently contains its own scene. No actionable issues were found.

Reviews (1) · Last reviewed commit: "narrow the background physics-scene fix" · Reviewed by Greptile

Comment thread isaaclab_arena/assets/background.py Resolved
@arena-review-bot

Copy link
Copy Markdown
Contributor

🤖 Isaac Lab-Arena Review Bot

Summary

The spawner Background uses by default, _spawn_from_usd_with_resettable_nested_physics, now deactivates any PhysicsScene prims authored in the background USD before the background is cloned into each environment. That leaves /physicsScene as the only active scene, which fixes the 30-environment hang with replicator_kitchen_l_island. The fix is small and well targeted. My comments are minor.

Design, Boundaries & Scope

  • The fix covers backgrounds only. Isaac Lab deletes physics scenes only in SimulationContext._init_usd_physics_scene, which runs before any assets are spawned. Any spawned USD with a scene under its default prim (an object or a robot, not just a background) can still add extra PhysX scenes.
    • Is it worth opening an Isaac Lab issue for scenes that spawned USDs bring in?
    • Separately, it's worth asking the asset owners to remove the scene from kitchen_l_island.usda; the PR description already says the asset should drop it.
  • The fix is tied to reset_nested_physics. See the inline comment on background.py:152.

Findings

🔵 Improvement: isaaclab_arena/assets/background.py:152 — Scenes are only deactivated when reset_nested_physics=True, and the code doesn't say so. Suggest noting this on the flag and in the _get_spawn_cfg docstring.
🔵 Improvement: isaaclab_arena/tests/test_background_physics_reset.py:470 — The test doesn't cover the ordering that the comment at background.py:151 relies on: a scene inside a subtree that gets de-instanced. Covering it takes a few more lines and adds no CI cost.

Other notes:

  • Overlap with Remove faulty scene the tracked placement experiments #1696. That open PR removes kitchen_bench_replicator_l_island_bowl_sink from placement_subset_experiment.yaml and blames the stall on PhysX SDF cooking. If this PR fixes the hang, that removal may not be needed. Worth confirming on Remove faulty scene the tracked placement experiments #1696 that it's the same hang.
  • PR description. It still contains the template placeholder Fixes #[issue number].
  • CI. When I reviewed, the test jobs for this head had not run yet because the image build was still in progress. Pre-commit passed. I couldn't confirm that the new test passes in CI.

Verdict

Ship it

@zhx06 zhx06 changed the title Deactivate Physics Scenes Authored in Background USDs Fix hang in L-island kitchen scenes Nov 18, 2026

@david-tingdahl-nvidia david-tingdahl-nvidia left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for fixing this

Comment thread isaaclab_arena/assets/background.py Resolved
Signed-off-by: zhx06 <zihaox@nvidia.com>
Signed-off-by: zhx06 <zihaox@nvidia.com>
Signed-off-by: zhx06 <zihaox@nvidia.com>
@zhx06
zhx06 force-pushed the zxiao/fix/deadlock-background-physics-scene branch from 547f081 to 73c79bd Compare November 18, 2026 16:53
@zhx06
zhx06 merged commit ccc08da into main Nov 18, 2026
10 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants