Copy-engine support

[dsidler]

No description provided.

[dsidler]

@mawad-amd not sure how useful the changes and copy egine path for examples 07 and 10 are. I might revert those and only keep the changes for message-passing example

[mawad-amd]

Thanks, David! I think we will want to keep the benchmark you used for your results. The message passing one would be a great introductory example.

Will review this PR soon.

[mawad-amd]

Some minor comments. Thanks a lot for this David!

Please fix/resolve conflict, and just make sure CI passes and merge.

[mawad-amd]

This will likely break the simulation backend but let's merge it and see.

[mawad-amd]

Also let's pin this to a known good sha

[mawad-amd]

CI failure triage

Two distinct classes here — 1 real, 10 infra. TL;DR: re-run the 10, the Gluon one needs a fix.

The real failure — External Gluon Validation Test

Job log:

00:03:36 INFO  [Iris] [0/2] [torch_allocator] TorchAllocator: init heap_size=1.0GB device=0
ERROR code: 1 Failed (File: .../rocm-xio/src/endpoints/sdma-ep/anvil.hip, Line: 239)

Both ranks die at the same instant right after TorchAllocator init — this is during iris init, not inside the test. It's branch-specific: External Gluon Validation is green on #562, #553 and #544, and only fails here.

anvil.hip:239 (rocm-xio @ cbe97e6) is the status check on:

HSAKMT_STATUS createStatus = hsaKmtCreateQueueExt(
  localNodeId, HSA_QUEUE_SDMA_BY_ENG_ID, DEFAULT_QUEUE_PERCENTAGE,
  DEFAULT_PRIORITY, engineId, queueBuffer_, SDMA_QUEUE_SIZE, nullptr, &queue_);
CHECK_HSAKMT_SUCCESS(createStatus, "Failed");   // <-- 239

ERROR code: 1 is HSAKMT_STATUS_ERROR, i.e. KFD rejected engineId.

Suspected cause — an unguarded -1 -> uint32_t conversion:

anvil.hpp:79     int getSdmaEngineId(...)                       // returns int, can be -1
anvil.hip:659      fallbackEngineId = (src == dst) ? 0 : -1;
anvil.hip:661      engineId = getRecommendedSdmaEngineId(src, dst, fallback)
                   // returns fallback (-1) at 582/598/611/630/634/648:
                   // node-props fail, NumIOLinks == 0, iolink query fail,
                   // RecSdmaEngIdMask == 0, no link matching dstNodeId
anvil.hip:491    const uint32_t engineId = getSdmaEngineId(...)  // -1 -> 0xFFFFFFFF
sdma-ep.hip:49   uint32_t engineId = ...getSdmaEngineId(...)     // same
sdma-ep.hip:80   uint32_t engineId = ...getSdmaEngineId(...)     // same

There is no guard on a negative engineId anywhere on that path.

Why it would trigger on this runner: getMappedSdmaEngineId is MI300X-specific — it reads sysfs xgmi_physical_id and indexes a hardcoded mi300xOamMap, and throws when that node is absent, so MI350X lands on the -1 fallback. getRecommendedSdmaEngineId then needs a KFD io-link from src to dst node; with only a subset of GPUs visible in the container (this job ran 2 ranks), "no link matching dstNodeId" returns -1 -> 0xFFFFFFFF -> KFD rejects.

This is inferred from reading the source — I have not reproduced it on hardware. Cheap confirmation: re-run with ROCM_XIO_SDMA_DEBUG=1 and read the SDMA_DEBUG map ... engineId= line that anvil.hip prints just before the create. engineId=-1 / 4294967295 confirms it.

Possible fix: validate before the cast — if getSdmaEngineId() returns < 0, either fail with a real message or fall back to engine 0 / plain HSA_QUEUE_SDMA rather than BY_ENG_ID.

The other 10 look like infra flakes

All the Test <suite> (N ranks, git install) jobs plus New examples (4 ranks, editable):

1. Same inputs, different outcome. Same iris SHA (288aba9) and identical resolved deps (tritonBLAS fa3f508, rocm-xio cbe97e6). Test ccl passed at 8 ranks and failed at 1 rank. Same split for unittests (8r/4r pass, 1r/2r fail) and context (8r/2r pass, 1r/4r fail).
2. Time-clustered. Every failure started between 00:02:24 and 00:10:59. Everything outside that window is green, including Test ops (4 ranks) which started 00:12:22 and passed.
3. No test output. The process dies ~0.5s after collected 30 items, before any test runs. One is SIGABRT (-6), the rest exit 1.
4. Unrelated suites fail together — e.g. test_allocate_symmetric_returns_peer_bases is basic heap allocation, unrelated to sDMA.
5. Six jobs overlapped on a single runner inside that window, all drawing from the same 8-GPU bitmap allocator.

The exact mechanism is not recoverable from the logs since the child's stdout is lost when it dies — but (1) is fairly strong evidence it is not the diff.

Unrelated to this PR, but worth noting: rocm-xio is installed from its default branch and tritonBLAS from a moving branch ref, so CI is not reproducible across time. Did not cause this (the commits matched across pass and fail), but it is a latent flake source.

---

This comment was AI-generated. The source trace is mechanical and checkable; the root-cause attribution is a hypothesis, not a reproduced result — please verify before acting on it.

[dsidler]

b603a74 is a workaround. i'll plan test a fix on the xio side such that HIP_VISIBLE_DEVICES works.

[mawad-amd]

Thanks!