Skip to content

NV: lock GA10x's PCIe link at Gen1 (RTX A6000 behind Thunderbolt 5) [matrix_vector] - #247

Merged
xji3 merged 1 commit into
matrix_vectorfrom
mv-a6000-fix
Oct 21, 2026
Merged

xji3 merged 1 commit into
matrix_vectorfrom
mv-a6000-fix

Conversation

@xji3

@xji3 xji3 commented Oct 21, 2026

Copy link
Copy Markdown
Member

What

On Ampere (GA10x, architecture 0x17) the NV boot now adds RMPcieLinkSpeed=0x800002AA to the registry table it hands GSP-RM, which locks the PCIe link at Gen1 at load. BEAGLE_NV_GA10X_LINK_LOCK=0 sends tinygrad's table instead. Ada and Blackwell are unchanged.

Why

On an RTX A6000 (GA102, 10de:2230) in an Aoostar AG03 (Intel JHL9480, Thunderbolt 5), GSP-RM 570.144 leaves the link at Gen1 x4 by GSP_INIT_DONE (target still Gen4; the card trains at Gen4 x4 cold). The link then dies under BAR1 traffic within seconds: the Mac's root port logs AER and "Link timeout ... Disabling the port", every read returns all ones, and the crash guard holds. The value is the one tinygrad PR #18215 found for Ada and Blackwell behind TB5 bridges. A Gen4 lock (0x80000295) broke the boot before INIT_DONE.

Changes

  • TinyGPUNVBoot.h: nv_ga10x_link_lock(); the registry entry count is no longer hard-coded at 2.
  • oracle/nv_init_helper.py: the same table on the goldens' tinygrad side.
  • golden_boot.py: GA102 cases built from the A6000's own replies (early boot, NV_FLCN.init_sw on its VBIOS, the whole init_sw with and without the lock), compared with the card's L0 recording when the machine has it; a check that SET_REGISTRY carries the lock exactly on GA10x; three perturbations (the value, the gate, the opt-out). Cases whose captured VBIOS is not on the machine now print SKIPPED instead of crashing the test.
  • TinyGPU.md: the Ampere row, the lock (it lasts until the enclosure is power-cycled), the opt-out, and that TinyGPU.app c0d024f9 serves only display-class (0x0300) cards.

Testing

On the RTX A6000 in the AG03:

  • Without the lock: the link was lost on every cold boot tried (2 of 2).
  • With it: the testing plan's NVIDIA matrix passed, about 80 boots in all with no failure and no hold:
    • cold and warm boots; SP and DP acceptance (200 repeats exact); SP and DP sweeps to 256 states;
    • instances, threads and cycles; both out-of-memory paths;
    • three soaks (64,000 / 20,000 / 5,000 reps) and 20 warm boots;
    • D1 SP 13/13 and DP 12/12;
    • kill-at-idle (guard verdict clean) and the P4 warm-GPU recovery.
  • This PR's build, no env, after a power cycle: NV1 cold PASS on both usb2 and matrix_vector (logL -1498.89812 = CPU, clean teardown).
  • Offline: golden_boot.py all identical on both branches (GA102 cases equal to the card's recording). run_goldens.sh fails only the goldens that need Docker or the author's private VBIOS and tables, the same set as before.

Notes

  • Only one card and one enclosure have run this. Other GA10x boards and enclosures boot the same way but are untested. The cost of running at Gen1 is not measured: no unlocked boot of this card survives long enough to compare.
  • The GA102 golden cases need the A6000's VBIOS and recording in ~/.beagle/tinygpu. Those are NVIDIA firmware and stay out of git, so on other machines these cases print SKIPPED.
  • The same fix is open against both branches: a6000-fix → usb2 and mv-a6000-fix → matrix_vector. The matrix_vector cherry-pick resolved a conflict in golden_boot.py's main() in that branch's style.

🤖 Generated with Claude Code

On an RTX A6000 (GA102, 10de:2230) in an Aoostar AG03 (Intel JHL9480, Thunderbolt 5),
GSP-RM 570.144 leaves the link at Gen1 x4 by GSP_INIT_DONE (target still Gen4; the card
trains at Gen4 x4 cold), and the link then dies under BAR1 traffic within seconds: the
Mac's root port logs AER and a link timeout and disables the port, every read returns
all ones, and the crash guard holds. Locking the link at load with
RMPcieLinkSpeed=0x800002AA (gens 2-6 disabled; tinygrad PR #18215's value for Ada and
Blackwell behind TB5 bridges) fixes it: the whole hardware matrix (16.4-16.11), the
soaks, 20 warm boots, D1 SP and DP, kill-at-idle and the P4 recovery all passed on the
A6000 with it. A Gen4 lock (0x80000295) broke the boot before INIT_DONE.

- TinyGPUNVBoot.h: nv_ga10x_link_lock(); on architecture 0x17 the registry table gets
  the entry after tinygrad's two (the count no longer hard-coded at 2).
  BEAGLE_NV_GA10X_LINK_LOCK=0 sends tinygrad's table.
- oracle/nv_init_helper.py: the same table for the goldens' tinygrad side.
- golden_boot.py: GA102 cases from the A6000's own replies (early, NV_FLCN.init_sw on
  its VBIOS, the whole init_sw with and without the lock), each compared with the
  card's L0 recording when this Mac has it (as a prefix when the recording has no
  init_sw marker); a check that SET_REGISTRY carries the lock exactly on GA10x;
  perturbations of the value, the gate and the opt-out. Cases whose captured VBIOS is
  not on this Mac (the AD107 dumps) are reported SKIPPED instead of crashing the test.
- TinyGPU.md: the Ampere row, the lock and its stickiness, the opt-out, and that
  TinyGPU.app c0d024f9 serves only display-class (0x0300) cards.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@xji3
xji3 merged commit f65a92a into matrix_vector Oct 21, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant