Skip to content

NV: lock GA10x's PCIe link at Gen1 (RTX A6000 behind Thunderbolt 5) - #246

Merged
xji3 merged 2 commits into
usb2from
a6000-fix
Oct 21, 2026
Merged

xji3 merged 2 commits into
usb2from
a6000-fix

Conversation

@xji3

@xji3 xji3 commented Oct 21, 2026 •

Copy link
Copy Markdown
Member

What

On Ampere (GA10x, architecture 0x17) the NV boot now adds RMPcieLinkSpeed=0x800002AA to the registry table it hands GSP-RM, which locks the PCIe link at Gen1 at load. BEAGLE_NV_GA10X_LINK_LOCK=0 sends tinygrad's table instead. Ada and Blackwell are unchanged.

Why

On an RTX A6000 (GA102, 10de:2230) in an Aoostar AG03 (Intel JHL9480, Thunderbolt 5), GSP-RM 570.144 leaves the link at Gen1 x4 by GSP_INIT_DONE (target still Gen4; the card trains at Gen4 x4 cold). The link then dies under BAR1 traffic within seconds: the Mac's root port logs AER and "Link timeout ... Disabling the port", every read returns all ones, and the crash guard holds. The value is the one tinygrad PR #18215 found for Ada and Blackwell behind TB5 bridges. A Gen4 lock (0x80000295) broke the boot before INIT_DONE.

Changes

  • TinyGPUNVBoot.h: nv_ga10x_link_lock(); the registry entry count is no longer hard-coded at 2.
  • oracle/nv_init_helper.py: the same table on the goldens' tinygrad side.
  • golden_boot.py: GA102 cases built from the A6000's own replies (early boot, NV_FLCN.init_sw on its VBIOS, the whole init_sw with and without the lock), compared with the card's L0 recording when the machine has it; a check that SET_REGISTRY carries the lock exactly on GA10x; three perturbations (the value, the gate, the opt-out). Cases whose captured VBIOS is not on the machine now print SKIPPED instead of crashing the test.
  • TinyGPU.md: the Ampere row, the lock (it lasts until the enclosure is power-cycled), the opt-out, and that TinyGPU.app c0d024f9 serves only display-class (0x0300) cards.

Testing

On the RTX A6000 in the AG03:

  • Without the lock: the link was lost on every cold boot tried (2 of 2).
  • With it: the testing plan's NVIDIA matrix passed, about 80 boots in all with no failure and no hold:
    • cold and warm boots; SP and DP acceptance (200 repeats exact); SP and DP sweeps to 256 states;
    • instances, threads and cycles; both out-of-memory paths;
    • three soaks (64,000 / 20,000 / 5,000 reps) and 20 warm boots;
    • D1 SP 13/13 and DP 12/12;
    • kill-at-idle (guard verdict clean) and the P4 warm-GPU recovery.
  • This PR's build, no env, after a power cycle: NV1 cold PASS on both usb2 and matrix_vector (logL -1498.89812 = CPU, clean teardown).
  • Offline: golden_boot.py all identical on both branches (GA102 cases equal to the card's recording). run_goldens.sh fails only the goldens that need Docker or the author's private VBIOS and tables, the same set as before.

Also: Linux CI (second commit, usb2 only)

usb2's Linux CI has failed on every push since the TinyGPU backend landed (101ad60 included): BUILD_TINYGPU defaulted ON everywhere, and libhmsbeagle/GPU/CMakeLists.txt then requires nvcc, so a plain cmake .. on Ubuntu stopped at configure. The second commit ports matrix_vector's default (from its merge of usb2, a56795b): TinyGPU is ON by default only on macOS where nvcc and ptxas are found. -DBUILD_TINYGPU=ON still requires them. Checked locally: the default is unchanged on a Mac with the tools, and the build completes with TinyGPU off. matrix_vector (#247) already has this and passes CI.

Notes

  • Only one card and one enclosure have run this. Other GA10x boards and enclosures boot the same way but are untested. The cost of running at Gen1 is not measured: no unlocked boot of this card survives long enough to compare.
  • The GA102 golden cases need the A6000's VBIOS and recording in ~/.beagle/tinygpu. Those are NVIDIA firmware and stay out of git, so on other machines these cases print SKIPPED.
  • The same fix is open against both branches: a6000-fix → usb2 and mv-a6000-fix → matrix_vector. The matrix_vector cherry-pick resolved a conflict in golden_boot.py's main() in that branch's style.

🤖 Generated with Claude Code

xji3 and others added 2 commits October 21, 2026 10:33
On an RTX A6000 (GA102, 10de:2230) in an Aoostar AG03 (Intel JHL9480, Thunderbolt 5),
GSP-RM 570.144 leaves the link at Gen1 x4 by GSP_INIT_DONE (target still Gen4; the card
trains at Gen4 x4 cold), and the link then dies under BAR1 traffic within seconds: the
Mac's root port logs AER and a link timeout and disables the port, every read returns
all ones, and the crash guard holds. Locking the link at load with
RMPcieLinkSpeed=0x800002AA (gens 2-6 disabled; tinygrad PR #18215's value for Ada and
Blackwell behind TB5 bridges) fixes it: the whole hardware matrix (16.4-16.11), the
soaks, 20 warm boots, D1 SP and DP, kill-at-idle and the P4 recovery all passed on the
A6000 with it. A Gen4 lock (0x80000295) broke the boot before INIT_DONE.

- TinyGPUNVBoot.h: nv_ga10x_link_lock(); on architecture 0x17 the registry table gets
  the entry after tinygrad's two (the count no longer hard-coded at 2).
  BEAGLE_NV_GA10X_LINK_LOCK=0 sends tinygrad's table.
- oracle/nv_init_helper.py: the same table for the goldens' tinygrad side.
- golden_boot.py: GA102 cases from the A6000's own replies (early, NV_FLCN.init_sw on
  its VBIOS, the whole init_sw with and without the lock), each compared with the
  card's L0 recording when this Mac has it (as a prefix when the recording has no
  init_sw marker); a check that SET_REGISTRY carries the lock exactly on GA10x;
  perturbations of the value, the gate and the opt-out. Cases whose captured VBIOS is
  not on this Mac (the AD107 dumps) are reported SKIPPED instead of crashing the test.
- TinyGPU.md: the Ampere row, the lock and its stickiness, the opt-out, and that
  TinyGPU.app c0d024f9 serves only display-class (0x0300) cards.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
BUILD_TINYGPU defaulted ON everywhere, and libhmsbeagle/GPU/CMakeLists.txt then
requires nvcc, so a plain `cmake ..` on Linux stopped at configure ("nvcc not
found"): usb2's Linux CI has failed on every push since the TinyGPU backend
landed. matrix_vector already defaults it ON only on macOS where its kernel
tools (nvcc, ptxas) are found (its merge of usb2, a56795b); this is that block.
-DBUILD_TINYGPU=ON still requires them; on a Mac with the tools the default is
unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@xji3
xji3 merged commit 859c663 into usb2 Oct 21, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant