Skip to content

NV: lock GA10x's PCIe link at Gen1 (RTX A6000 behind Thunderbolt 5) [matrix_vector] #247

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
xji3 merged 1 commit into from
Oct 21, 2026
Merged

NV: lock GA10x's PCIe link at Gen1 (RTX A6000 behind Thunderbolt 5) [matrix_vector] #247

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
26 changes: 23 additions & 3 deletions libhmsbeagle/GPU/TinyGPU.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,11 +24,13 @@ oracle.
|---|---|---|
| Ada (AD10x) | `0x26xx`-`0x28xx` | tested on an RTX 4060 (AD107); the other AD10x boot the same way, untested |
| Blackwell (GB20x) | `0x2bxx`-`0x2dxx`, `0x2fxx` | tested on an RTX 5070 (GB205); the other GB20x boot the same way, untested |
| Ampere (GA10x) | `0x22xx`, `0x24xx`, `0x25xx` | boots as Ada does (TODO.md plan step G1), untested: no Ampere card has run it |
| Ampere (GA10x) | `0x22xx`, `0x24xx`, `0x25xx` | tested on an RTX A6000 (GA102, `10de:2230`, 48 GiB) in an Aoostar AG03 (Intel JHL9480, Thunderbolt 5; the link trains at Gen4 x4), with the Gen1 link lock below; the other GA10x boot the same way, untested |
| AMD Navi 31 (gfx1100) | `1002:744c` | tested on an RX 7900 XT |
| AMD Navi 48 (gfx1201) | `1002:7550` | tested on an RX 9070 XT (XFX); any other AMD device ID is refused before anything is sent to the card, and IP versions other than these two cards' at its boot |

These are tinygrad's NVIDIA families. Any other NVIDIA GPU (Turing, GA100, Hopper, GB100) is refused at
These are tinygrad's NVIDIA families. TinyGPU.app `c0d024f9` serves only display-class cards (its driver matches PCI
class `0x0300`), so an NVIDIA card that enumerates as a 3D controller (`0x0302`: the A2, A16, A30, A40 and A100, say) is
not served at all. Any other NVIDIA GPU (Turing, GA100, Hopper, GB100) is refused at
`beagleCreateInstance`, with nothing written to it. The NVIDIA boot also needs a 256 MiB BAR1, as GeForce cards come: a
card with another size (data-center and some professional cards have a larger one) is refused at the start of its boot
(`BarLayoutError`), before any firmware is set up.
Expand All @@ -38,6 +40,23 @@ architecture (sm_86, sm_89, sm_120), on AMD 18 HSACOs per architecture (gfx1100,
well as single (TODO.md plan steps C16, A7); consumer GPUs run double precision natively, at a fraction of the single rate. A
GPU whose architecture has no cubin is refused at `beagleCreateInstance`.

### Ampere: the Gen1 link lock

On GA10x the plugin adds `RMPcieLinkSpeed=0x800002AA` to the registry table it hands GSP-RM (gens 2 to 6 disabled, locked
at load: the value tinygrad PR #18215 found for Ada and Blackwell behind Thunderbolt 5 bridges). Without it, the RTX A6000
behind the AG03's JHL9480 came out of GSP-RM's boot at Gen1 x4 with its target still Gen4, and the link then died under the
first seconds of BAR1 traffic: the Mac's root port logged AER and a link timeout and disabled the port, every read returned
all ones, and the crash guard held. With the lock every boot passed (October 2026): the hardware matrix, the soaks and 20
warm boots, D1 in both precisions, the crash guard at idle and the P4 recovery. The link then runs at Gen1 (2.5 GT/s x4).
What that costs is not measured, since no unlocked boot of this card survives long enough to compare; the A6000 ran 0.165 ms
per evaluation at 4 states, single precision. A Gen4 lock (`0x80000295`) broke the boot before GSP-RM was up.

- The lock lasts until the enclosure is power-cycled, so a boot of another configuration after it is not a clean
comparison until then.
- `BEAGLE_NV_GA10X_LINK_LOCK=0` sends tinygrad's table instead (for diagnosis).
- The enclosure and bridge matter: an RTX 3080 in a Thunderbolt 3 enclosure (Titan Ridge) was reported to boot stock
tinygrad (tinygrad PR #16262), so the lock may be unnecessary there. It is on for every GA10x all the same.

## Requirements

- macOS on Apple silicon, with the eGPU attached.
Expand Down Expand Up @@ -120,7 +139,7 @@ BEAGLE itself, as at exit, so it needs no power cycle.
Power-cycle the eGPU (unplug it, then plug it in again) only then, and always before killing a holding guard. A GPU left
warm by an earlier process that was not torn down (WPR2 still up) is refused with nothing written:
`WARM GPU: WPR2 is up ... Power-cycle the eGPU`. The exception (plan step P4, on Ada and Ampere GPUs; the default since two
clean recoveries on the RTX 4060, untested on Ampere): a warm GPU whose GSP-RM was unloaded, so that only the teardown is
clean recoveries on the RTX 4060, and one on the RTX A6000): a warm GPU whose GSP-RM was unloaded, so that only the teardown is
missing (after `BEAGLE_NV_TEARDOWN=0`, or a teardown that failed after the unload), is torn down at boot and then boots.
Only reads come before that decision. NVIDIA's driver refuses such a GPU instead; `BEAGLE_NV_RECOVER=0` does too.

Expand All @@ -140,6 +159,7 @@ For users:
| `BEAGLE_NV_TEARDOWN=0` | the GSP unload only, no teardown: the next boot then needs a power cycle (for diagnosis) |
| `BEAGLE_NV_UNLOAD_LEVEL=0` | the LEVEL_0 unload instead of FAST_UNLOAD (a fallback) |
| `BEAGLE_NV_RECOVER=0` | a warm Ada or Ampere GPU whose GSP is suspended or halted is refused, as NVIDIA's driver does, instead of torn down at boot |
| `BEAGLE_NV_GA10X_LINK_LOCK=0` | Ampere: no Gen1 link lock in the registry table (tinygrad's table; see Supported GPUs) |
| `BEAGLE_NV_GUARD` | the crash guard's path (default: next to the plugin); `BEAGLE_AMD_GUARD` for the AMD C++ boot |
| `APL_REMOTE_SOCK` | TinyGPU.app's socket (default `$TMPDIR/tinygpu.sock`, as tinygrad) |
| `BEAGLE_AMD_DATA_MB` | AMD: the VRAM pool, in MiB (default: half the VRAM) |
Expand Down
26 changes: 20 additions & 6 deletions libhmsbeagle/GPU/TinyGPUNVBoot.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,7 @@
#include <cerrno>
#include <cstdint>
#include <cstdio>
#include <cstdlib>
#include <cstring>
#include <memory>
#include <string>
Expand Down Expand Up @@ -655,10 +656,23 @@ inline void nv_boot_rpc_set_gsp_system_info(NVBootDev& d, NVGspBoot& g) {
g.cmd_q->send_rpc(nv::NV_VGPU_MSG_FUNCTION_GSP_SET_SYSTEM_INFO, nv_bytes(data));
}

// rpc_set_registry_table (ip.py:615-627): RMForcePcieConfigSave and RMSecBusResetEnable, 1 each
inline void nv_boot_rpc_set_registry_table(NVGspBoot& g) {
const std::pair<const char*, uint32_t> table[] = {{"RMForcePcieConfigSave", 0x1}, {"RMSecBusResetEnable", 0x1}};
const size_t hdr_size = sizeof(nv::struct_PACKED_REGISTRY_TABLE), entries_size = sizeof(nv::struct_PACKED_REGISTRY_ENTRY) * 2;
// BEAGLE's Ampere fix (reports/A6000_fix_plan_2026-10-19.md, Phase C): on GA10x (architecture 0x17) GSP-RM 570.144 leaves the
// link at Gen1 x4 by GSP_INIT_DONE through a TB5 bridge (the RTX A6000 in an AG03, JHL9480, which trains at Gen4 x4 cold), and
// the link then dies under BAR1 traffic: the root port logs AER and a link timeout and disables the port. RMPcieLinkSpeed
// 0x800002AA (gens 2-6 disabled, locked at load; tinygrad PR #18215's value for Ada and Blackwell behind TB5) holds it at Gen1,
// and every boot then passed (Phase B3). The lock lasts until the enclosure is power-cycled. BEAGLE_NV_GA10X_LINK_LOCK=0 sends
// tinygrad's table instead.
inline bool nv_ga10x_link_lock(uint32_t architecture) {
const char* e = getenv("BEAGLE_NV_GA10X_LINK_LOCK");
return architecture == 0x17 && !(e && strcmp(e, "0") == 0);
}

// rpc_set_registry_table (ip.py:615-627): RMForcePcieConfigSave and RMSecBusResetEnable, 1 each, then on GA10x the link lock
// (nv_init_helper's _rpc_set_registry_table_ga10x)
inline void nv_boot_rpc_set_registry_table(const NVBootDev& d, NVGspBoot& g) {
std::vector<std::pair<const char*, uint32_t>> table = {{"RMForcePcieConfigSave", 0x1}, {"RMSecBusResetEnable", 0x1}};
if (nv_ga10x_link_lock(d.architecture)) table.push_back({"RMPcieLinkSpeed", 0x800002AA});
const size_t hdr_size = sizeof(nv::struct_PACKED_REGISTRY_TABLE), entries_size = sizeof(nv::struct_PACKED_REGISTRY_ENTRY) * table.size();
std::vector<uint8_t> entries, data;
for (const auto& kv : table) {
nv::struct_PACKED_REGISTRY_ENTRY e{};
Expand All @@ -670,7 +684,7 @@ inline void nv_boot_rpc_set_registry_table(NVGspBoot& g) {
}
nv::struct_PACKED_REGISTRY_TABLE header{};
header.size = (uint32_t)(hdr_size + entries.size() + data.size());
header.numEntries = 2;
header.numEntries = (uint32_t)table.size();
std::vector<uint8_t> msg = nv_bytes(header);
msg.insert(msg.end(), entries.begin(), entries.end());
msg.insert(msg.end(), data.begin(), data.end());
Expand All @@ -683,7 +697,7 @@ inline void nv_boot_gsp_init_sw(NVBootDev& d, NVGspBoot& g, const NVFlcnImages*
nv_boot_init_libos_args(d, g);
nv_boot_init_wpr_meta(d, g, flcn);
nv_boot_rpc_set_gsp_system_info(d, g);
nv_boot_rpc_set_registry_table(g);
nv_boot_rpc_set_registry_table(d, g);
g.gpfifo_class = nv_gpu::AMPERE_CHANNEL_GPFIFO_A; g.compute_class = nv_gpu::AMPERE_COMPUTE_B; g.dma_class = nv_gpu::AMPERE_DMA_COPY_B;
const std::string fam = d.chip_name.substr(0, 2);
g.viddec_class = fam == "AD" ? nv_gpu::NVC9B0_VIDEO_DECODER : fam == "GB" ? nv_gpu::NVCFB0_VIDEO_DECODER : 0; // .get(): None
Expand Down
Loading
Loading