feat(network): v2 weights bridge, relay hardening and deterministic golden guard

.github/workflows/ci.yml

@@ -66,6 +66,18 @@ jobs:
           enable-cache: true
           cache-dependency-glob: "uv.lock"
       - run: uv sync --locked --all-extras --group dev
+      # Golden leaf roots are per CPU class; record which CPU ran this shard.
+      - name: Runner CPU
+        shell: bash
+        run: |
+          set -euo pipefail
+          MODEL=$(lscpu | sed -n 's/^Model name: *//p' | head -n1)
+          VENDOR=$(lscpu | sed -n 's/^Vendor ID: *//p' | head -n1)
+          if lscpu | grep -E '^Flags:' | grep -qw avx512f; then AVX512F=yes; else AVX512F=no; fi
+          TORCH=$(uv run --frozen python -c "import torch;print(torch.__version__, torch.get_num_threads())")
+          LINE="cpu: ${MODEL} | vendor: ${VENDOR} | avx512f: ${AVX512F} | torch, threads: ${TORCH}"
+          echo "$LINE"
+          echo "shard ${{ matrix.shard }} $LINE" >> "$GITHUB_STEP_SUMMARY"
       - name: Sharded suite
         env:
           SHARD: ${{ matrix.shard }}

.gitignore

@@ -5,6 +5,7 @@ __pycache__/
 .ruff_cache/
 .pytest_cache/
 .hypothesis/
+.evidence/
 
 # audit/report/plan artifacts (kept out of the product repo)
 experiments/results/

.publishignore

@@ -14,6 +14,6 @@ id_(ed25519|rsa)
 \.(pem|key)$
 # Local authority, custody and recovery material; public nested data stays publishable.
 ^\.(omo|herdr-web-ui)/
-(^|/)(private|secrets|roles|private-roles|role-keys|private-keys|authority|authority-state|portable-authority|portable-bundle|portable-bundles|snapshots|preserved-snapshots|evidence|service-export|integration-root|rescue)/
+(^|/)(private|secrets|roles|private-roles|role-keys|private-keys|authority|authority-state|portable-authority|portable-bundle|portable-bundles|snapshots|preserved-snapshots|evidence|\.evidence|service-export|integration-root|rescue)/
 \.(seed|priv|p12|pfx|pkcs8)$
 (^|/)(private[-_]key|admin\.token|drain\.token|challenge\.db(-wal|-shm)?|bundle\.json)$

README.md

@@ -184,7 +184,7 @@ CI publishes and attests both images to GHCR after each green build on `main`. T
 uv run --frozen python scripts/vast_e2e.py --key-file ~/.vast-key --max-usd 6 --max-minutes 90
 ```
 
-The script rents a single 2x RTX 5090 host with the pinned GPU image and syncs the current source. It runs the GPU self-check, the CUDA test selectors and a full Network v2 round trip (service, multi-GPU miner, auditor replay). It then rescues the results and always deletes the instance. Add `--dry-run` to see the selected offer and its worst-case cost.
+The script rents a single 2x RTX 5090 host with the pinned GPU image and syncs the current source. It runs the GPU self-check, the CUDA test selectors and a full Network v2 round trip (service, multi-GPU miner, auditor replay). It then rescues the results and always deletes the instance. Add `--dry-run` to see the selected offer and its worst-case cost; it makes read-only provider GET calls (credit and offer search) and creates nothing.
 
 ## Components
 
@@ -228,6 +228,7 @@ tests/        one directory per package
 ## Contributing
 
 - `uv sync --frozen --all-extras`, then `bash scripts/check.sh` before opening a pull request.
+- `bash scripts/mine-smoke-v2.sh` checks the network-v2 miner CLI chain end to end on CPU, offline (~2 min; see [docs/network-v2.md](docs/network-v2.md#cpu-smoke)).
 - Verification stays bitwise. Tolerance checks, obfuscation or trusted-hardware assumptions are out of scope for the security model.
 - When a route or message changes, update [docs/challenge-routes.md](docs/challenge-routes.md) and regenerate schemas with `uv run --frozen python -m hypertrain.protocol.schema docs/schemas`.
 - Publishing goes through `scripts/publish.sh`, which runs a strict secret scan and requires an out-of-band approval token.

deploy/challenge-profile-row.example.toml

@@ -0,0 +1,24 @@
+# Example challenge profile with hypertrain active (config/challenges.toml in CortexLM/cortex,
+# served by the gateway at GET /v1/trust). The live profile is version 1 with bounty = 2000 and
+# proof = 8000 bps; here proof drops to 7000 to give hypertrain 1000. The shares must sum to
+# exactly 10000. `version` is 3 because a third challenge needs version >= 3; bump it above the
+# live value. It is a whole file: replace the live profile with it, never append it (duplicate
+# keys). Restart the master between two epochs.
+# Leave `introduced_epoch` at or below the current epoch. No key: Cortex signs nothing here.
+version = 3
+introduced_epoch = 0
+
+[[challenges]]
+id = "bounty"
+emission_share_bps = 2000
+policy = "all_metagraph_hotkeys"
+
+[[challenges]]
+id = "proof"
+emission_share_bps = 7000
+policy = "all_metagraph_hotkeys"
+
+[[challenges]]
+id = "hypertrain"
+emission_share_bps = 1000
+policy = "all_metagraph_hotkeys"

deploy/registry-row.toml

@@ -1,9 +1,10 @@
 # Row to append to deploy/challenges/registry.toml in CortexLM/cortex (unsigned operator
-# registry, cortex@d738424 src/cortex/challenges/registry.py). Registered but absent from the
-# signed trust root = burn-in without emission. Secrets in
+# registry, src/cortex/challenges/registry.py). Registered but absent from the challenge profile
+# (config/challenges.toml) = burn-in without emission. Secrets in
 # $BASE_CHALLENGE_SECRETS_HOST_DIR/hypertrain/ (0700, owner 65532): internal.token, admin.token,
-# worker.token, coord.key (K_coord seed, never the Cortex leaf seed).
-# HYPERTRAIN_OWNER_HOTKEY is the ss58 of K_owner; replace before enabling.
+# worker.token, coord.key (K_coord seed).
+# HYPERTRAIN_OWNER_HOTKEY is the ss58 of hypertrain's K_owner (RunManifest signer); replace
+# before enabling.
 version = 1
 
 [[challenge]]

docs/challenge-routes.md

@@ -33,7 +33,7 @@ Errors have the shape `{"detail": str}`.
 | `CHALLENGE_ADMIN_TOKEN_FILE` | Bearer for admin, beacon push and aggregator routes |
 | `CHALLENGE_WORKER_TOKEN_FILE` | Bearer for auditors and referees |
 | `CHALLENGE_MASTER_URL` | Used for `GET /v1/metagraph/latest`, which returns `{"hotkeys": {ss58: uid}}` and is cached 30 s |
-| `HYPERTRAIN_COORD_KEY_FILE` | K_coord sr25519 seed: 32 raw bytes or 64 hex. It is never the Cortex leaf seed |
+| `HYPERTRAIN_COORD_KEY_FILE` | K_coord sr25519 seed: 32 raw bytes or 64 hex |
 | `HYPERTRAIN_OWNER_HOTKEY` | ss58 of K_owner, which signs RunManifest |
 | `HYPERTRAIN_GENESIS_UNIX` | Ledger `Params`. Default: quicknet genesis |
 | `HYPERTRAIN_EPOCH_SECONDS` | Ledger `Params`. Default 4320 |

docs/findings/v2-weights-bridge.md

@@ -0,0 +1,84 @@
+# Finding: network-v2 rewards never reach `get_weights`
+
+Status: **bridged** (fix option 2 below). Before the fix (base `1818f9e`) a matured
+contract-2 reward was never paid as weight. End-to-end proof:
+`tests/challenge/test_v2_weights_bridge.py::test_v2_matured_reward_reaches_get_weights`
+(slow, no longer xfail). Unit proofs: `tests/challenge/test_v2_weights_pay.py`.
+
+## Bridge
+`ChallengeStore.weights` computes the v1 answer, then `_pay_v2` fills the mass it left as
+`units_burned_this_epoch` from `EscrowV2.payable(round)`. `round` is the drand round of
+`epoch_at` under the run's beacon (`(epoch_at - genesis_time) // period + 1`), so the answer
+never reads the wall clock. Each row is moved `available -> paid` with `EscrowV2.pay` inside
+the transaction that writes `weights_runs`; the weight goes to the origin's hotkey.
+- Paid: free (`available`) units of live reward origins (`escrow_finalized`) whose
+  `mature_at <= round`, oldest maturity first, still held by the owner the origin was issued to.
+- Never paid: `reward_pending`, `admission_locked`, `dispute_locked`, `burned`; shadow
+  (bootstrap) origins and test genesis, which stay collateral (founder decision, 2026-10-19);
+  units transferred to another owner or credited to a referee.
+- Mass: `full_share_mass` stays 10^6 per store; weights + burned == 10^6. Units over the
+  budget stay `available` and pay in later epochs. v1 entitlements are paid first.
+- Idempotent: a stored epoch answer is returned before any v2 work. A failed persist rolls
+  the pay back; the retry recomputes and pays once. The `weights_runs_from` meta key tells
+  such a retry (the v1 journal already holds the epoch) from a genuine pre-multi-run answer.
+- Deregistered hotkey: `registered_weights` (`app.py`) still drops it after the store
+  answered. The escrow units are already `paid`, so they burn to UID0 at the master and are
+  not owed again (the existing `ponytail`). No freeze was added.
+- Child runs (`runs/<id>/`) pay and commit in their own database before the root commits.
+  If the root then fails, its retry is not legacy (`weights_runs_from`), so it reads the
+  child's stored answer and merges it again: the child's units are emitted once.
+
+## Code path
+- `GET /internal/v1/get_weights` (`src/hypertrain/challenge/app.py:864`) calls `store.weights`
+  (`app.py:874`), then the metagraph filter `registered_weights` (`app.py:884`).
+- `ChallengeStore.weights` (`src/hypertrain/challenge/store.py:1992`) returns a stored
+  `weights_runs` answer first. Otherwise it computes the v1 answer with
+  `self.ledger.get_weights` (`store.py:2029`, `src/hypertrain/ledger/ledger.py:490`) and,
+  when a live v2 reward was finalized, fills it with `_pay_v2` (call `store.py:2033`,
+  definition `store.py:2043`). Child runs merge through `_merge_answers` (`store.py:322`).
+- `_pay_v2` reads `EscrowV2.payable` (`src/hypertrain/ledger/escrow_v2.py:973`) and moves each
+  row with `EscrowV2.pay` (`escrow_v2.py:964`, called at `store.py:2077`), skipping a new
+  hotkey once the answer holds `MAX_ENTRIES` (`store.py:2064`).
+- `finalize_v2` (`store.py:8132`) issues live origins with `escrow.reward_finalize`
+  (`store.py:8267`); `push_beacon` (`store.py:663`) matures them to `available`.
+- `weights_runs_from` is set on the first v2-era call to one past the newest epoch the v1
+  journal answered, so no epoch the journal served before multi-run hosting is re-paid.
+
+Line numbers match the tree of this note's commit; re-grep `def weights`, `_pay_v2`, `def payable` after edits.
+
+## Not addressed (relay review, one line each)
+- B4: grants are idempotent per live assignment; nothing revokes one when the original arrives.
+- B5: `relay-disabled` is keyed by registry epoch, so a rotation re-enables a disabled key.
+- B6: the relay accepts at `now == d_upload` (`now > deadline`) while the master refuses
+  `>= d_upload`; no honest relay is disabled by it.
+
+## Test
+Reuses `_graduate_four` + `_live_checkpoint` (`tests/e2e/test_service_network_v2_e2e.py`).
+They finalize live round 0 via `finalize_v2` (status `SETTLED`, four origins of 250000).
+The test pushes the beacon past `mature_at`, reads every `available` round-0 unit, then calls
+the route and asserts the exact weight vector (four hotkeys at 250000), `units_paid` and the
+burned remainder. The fixture has no internal token, so the test points the app `Config` at a
+tmp token file. Run time ~150 s (measured 149 s on the host), so it is marked `slow`.
+
+Before the bridge (base `1818f9e`, earlier `> 0` assertion, no xfail marker):
+```
+>       assert response.json()["weights"].get(hotkey, 0) > 0, response.text
+E       AssertionError: {"challenge_slug":"hypertrain",...,"epoch":0,"full_share_mass":1000000,
+"metadata":{...,"ledger_minted":0,"ledger_paid":0,"ledger_pending":0,"reason":"ok",
+"round_at":0,"units_burned_this_epoch":1000000,"units_paid":0},"weights":{}}
+1 failed, 3 warnings in 139.48s
+```
+With `xfail(strict=True)`: `1 xfailed, 3 warnings in 142.30s`.
+
+## Fix options (option 2 implemented)
+1. Mirror v2 into the v1 ledger. `finalize_v2` adds one ledger entitlement per origin
+   (new `Ledger` event kind). Vesting stays in the ledger. Touches `challenge/store.py`
+   (`finalize_v2`), `ledger/ledger.py`. Risk: the same vesting is kept in two places, and an
+   EscrowV2 dispute or burn must also claw back the ledger.
+2. Answer weights from EscrowV2. `ChallengeStore.weights` adds, for v2 runs, the units that
+   matured up to `epoch_at` per origin hotkey. It moves them `available` -> `paid` with
+   `EscrowV2.pay` in the same transaction, then merges via `_merge_answers`. Touches
+   `challenge/store.py` (`weights`, `_merge_answers`), `ledger/escrow_v2.py` (a query by
+   maturity round). Escrow stays the only source of truth.
+3. Product decision: v2 units are not emissions. Document that `get_weights` pays only v1,
+   and stop promising v2 rewards. Touches `docs/` only.

docs/mfu-multigpu.md

@@ -33,7 +33,7 @@ peak, tokens/s, per-step times, `rank_peak_mem_gib` (whole round), `rank_step_pe
 |------|--------|--------|
 | `--reduction zero2o` | Gradients reduced to their ZeRO-2 owner in buckets issued from post-accumulate grad hooks, so communication overlaps the rest of backward. | Bitwise equal to `zero2` (same adds, same order). |
 | `--act-ckpt` (`HT_ACT_CKPT=1`) | Per-layer activation checkpointing. | Same kernels on recompute: bitwise equal. |
-| sdpa-flash | Selected on (sm_120, torch 2.14.0+cu130) from the qualification record in `tests/trainer/data/attn_qualify_sm120.json`. | Qualified by cross-process equality. |
+| sdpa-flash | Selected on (sm_120, torch 2.14.0+cu130) from the qualification record in `tests/trainer/data/attn_qualify_sm120.json`. When the flash kernel rejects `enable_gqa`, the first accepting path of `attention.PATHS` is probed once per (device, dtype, group, head dim) and logged as `attention backend: <path>`; a dtype or head dim no path accepts (e.g. fp32) stops the run (set `HT_ATTN_BACKEND=math` for such models); inputs whose layout the chosen kernel still rejects stop the run with the kernel error (no math fallback). | Qualified by cross-process equality (flash path only). A fallback path is a function of hardware, torch and geometry, so ranks agree, but it is not qualified. |
 | `--compile` (`HT_COMPILE=1`) | One Inductor graph per layer. | Leaves differ from eager (fusion changes rounding), so a compiled run replays only under the same flag. Both modes replay bitwise. |
 | `--micro-batch N` | Overrides the profile's per-rank micro_batch. | Part of the run; recorded as `HT_MFU_MICRO_BATCH`. |
 | `--nccl-pcie` | setdefaults `NCCL_P2P_DISABLE=1`, Ring, Simple, 4-8 channels, 8 MiB buffers. | Collectives are copies: no numeric effect. |
@@ -65,7 +65,10 @@ host with `pcie_bw` >= 50.
 - **Host PCIe.** The same dp2 config measured 11.2-11.5 % on a host where NCCL took ~3.8 s of GPU
   time per step whether P2P was on or off, against 37.2 % on a `pcie_bw` 53.8 (PCIe Gen5) host.
   `--profile mfu` therefore filters offers on `pcie_bw` and requires `cuda_max_good >= 13.2`
-  (the image is CUDA 13.2; a 13.0 host refused create with HTTP 400).
+  (the image is CUDA 13.2; a 13.0 host refused create with HTTP 400). Every profile also keeps
+  only the drivers in `scripts/genuine_cuda_ci.py` `DRIVERS`; MFU does not compare replays, so
+  add `--any-driver` to widen the market. Evidence goes to `.evidence/vast/` unless
+  `--evidence-dir` is given.
 - **Round-end memory, not step memory.** `max_memory_allocated` over a round (~29.9 GiB at
   micro_batch 4) is dominated by the round end: full weights and optimizer state, delta, error
   feedback and the compressed payload. Inside the steps the peak is ~20 GiB, which is why
@@ -75,3 +78,27 @@ host with `pcie_bw` >= 50.
   reduce-scatter-shaped owner reduction and a larger micro_batch at dp4 are the next steps.
 - **Elementwise work.** In the compiled trace, CUDA time outside GEMM/attention/NCCL (fp32 adds,
   fills, casts, checkpoint recompute) is still the largest bucket.
+
+## All GPU proofs on one host (`--profile proofs`)
+
+One 2x RTX 5090 instance, one upload, one DELETE and one parsed absence proof at the end. The
+image is the digest-pinned `IMAGE` of `scripts/vast_e2e.py`; Vast pulls it before ssh is ready,
+so the per-step seconds in `out/steps.tsv` (name, rc, seconds) exclude the pull.
+
+```bash
+uv run --frozen python scripts/vast_e2e.py --profile proofs --key-file /var/lib/openbot/.vast-key \
+  --max-usd 8 --max-minutes 45 --dry-run
+```
+
+`--dry-run` prints the chosen offer, then the step list (`PROOF_PLAN`), the pytest node ids
+(`PROOF_TESTS`), the caps and the worst-case cost, without creating anything (it does make read-only GET calls: credit and up to 8 offer pages). `remote_env` is printed with values masked (`KEY=***`). A real run exits 4 when it passed on a driver outside `QUALIFIED_DRIVERS`: the result is valid but backs no bitwise-replay claim. Steps: the
+gpu-e2e flow (network round, auditor replay, repeat; `HT_ATTN_BACKEND=math` because the fixture
+model is fp32 with head_dim 4), then with `HT_ATTN_BACKEND` unset: `select_backend` must pick
+sdpa-flash, the GPU-only tests (a skip fails), and a 1B bf16 dp2 round plus replay through
+`run_mfu_dist.py`, checked by `check_gpu_evidence.py --kind mfu` and `attention_backend ==
+sdpa-flash`. `--boot-retries` defaults to 0 here.
+
+Drivers: the default is `--any-driver`, and the offer quote and `harness-result.json` carry
+`driver_qualified` (false when the driver is not in `scripts/genuine_cuda_ci.py` `DRIVERS`, so
+no bitwise claim is made). `--eight-gpu-qualified` rents 8x RTX 5090 with a qualified driver
+only; nothing else selects 8 GPUs for this profile.