[MaxGhenis]

PE_UK_PRIVATE_HF_READ_TOKEN no longer reaches any CI job that runs on a pull request. In the push-only job that holds it, the fetch now runs before any of the variant's code.

Problem

ci.yml's replay-engine job installs each certified variant's pinned third-party packages and runs them. It also held the dataset token in a push-only "fetch" step that ran after that code, so a compromised package could redirect the fetch through $GITHUB_ENV, $GITHUB_PATH or a planted file. An independent stack review flagged this, and #210 confirmed it.

Review of the first version of this PR found more.

Pull requests received the token too. A debug re-run of this PR's own pull_request CI (job 119587048984) evaluated secrets.PE_UK_PRIVATE_HF_READ_TOKEN to a masked, non-empty value. The step that named it was then skipped, but the runner already had the token. The same leg then installed and ran the variant's packages.

That applies to every same-repository PR, including the bot's results PRs, which are exactly where a new release's pins first run. GitHub's docs say secrets a workflow doesn't reference are scrubbed, but referenced ones can be harvested.

Ordering alone can't protect the token where variant code runs beside it. The runner user has passwordless sudo, and the runner holds the job's secrets in memory until the job ends. The tj-actions compromise read secrets from exactly there. This repo's own uk_replay_sandbox.sh header already says so.

What changes

• New .github/workflows/uk-replay-determinism.yml, on pushes to main only. It holds the real measure-year replay and the token. It lists the certified variants the same way ci.yml does, from the index alone. In its real-replay job:
  • The fetch is the first step after the checkout, uv's setup and the read of the index. It runs before anything of the variant's is installed.
  • It calls uv by setup-uv's absolute uv-path under /usr/bin/env -i, keeping only PATH=/usr/bin:/bin, HOME, VARIANT and HF_HUB_CACHE.
  • uv runs with --no-project --isolated --no-config --no-cache --python /usr/bin/python3 --exclude-newer 2026-10-15T00:00:00Z, and Python runs with -I. Every package that loads next to the token is frozen at that date; uv picks the newest version before it.
  • The token goes to Python on stdin through the shell's own printf, never as a process argument.
  • HF_HUB_CACHE follows the same rule uk_replay_variants.find_dataset_bytes uses to find the dataset.
  • Install and replay run only when the fetch succeeded.
• ci.yml names no dataset token. replay-engine keeps the install, the installed-environment check, the guards, the premises and the diagnostics on PRs and pushes. A PR run posts a notice that no real replay ran. The step that only reads the index is renamed to say so, and the install step gets the name "Install the variant's exact environment".
• docs/uk_replay/RELEASE_REPLAY.md says where the real replay runs, what the step order secures and what it leaves open. It drops the false claim that pull requests never receive the token.

What is still open

In real-replay, the variant's engine still runs as the runner user while the token is in the runner's memory, so a compromised certified freeze could read it with sudo. The fix is to run the install and the replay as uk_replay_sandbox.sh's unprivileged account, as uk-replay.yml does for candidates. That is a redesign of that script and job, so it gets its own PR. Both are said in the job's comment and in the docs.

Separately, ci.yml's push-only publish-db still names the Supabase secrets. I did not test whether jobs that don't name a secret receive it, so I'm raising this to check, not claiming it.

Invariants (tested)

1. No workflow with a pull_request or pull_request_target trigger contains the token's name. Exactly three steps across all workflows hold it, each under a fixed condition.

2. In real-replay, the fetch comes before every step that touches .venv-variant. Only the checkout, setup-uv and the index read come before it, and none of them runs uv, uvx, pip, venv, pipeline or $REQUIREMENTS.

3. The script runs under bash -e, the shell GitHub's log shows for it, with uv replaced by a stub that records what it receives. The job environment would redirect an unshielded uv or Python: impostor uv/env/python3 first on PATH, HF_ENDPOINT, a proxy, UV_INDEX_URL, UV_PYTHON, LD_LIBRARY_PATH and PYTHONPATH=.. Even so, uv receives exactly:

   • {PATH, HOME, VARIANT, HF_HUB_CACHE}, apart from what Python itself adds at startup (LC_CTYPE; __CF_USER_TEXT_ENCODING on macOS)
   • the exact argv shown above
   • the token on stdin

   The only lines that name the token are the emptiness check and the printf. BASH_ENV, LD_PRELOAD or a bash earlier on PATH would act before the script's first line, so only the step order can keep variant code from setting them.

4. The download asks for exactly the variant's indexed dataset with the token. fetched=true is written only after it succeeds. A failed download fails the step and reports nothing. With no token or no sealed namespace, the step posts a notice and never runs uv.

5. Differential test: for all 8 combinations of HF_HUB_CACHE/HF_HOME/XDG_CACHE_HOME unset or set to a plain path, the cache the fetch names equals the one find_dataset_bytes searches. An empty HF_HOME, or ~ inside a value, would make the two disagree. The hosted image sets none of these variables.

6. Every job in the new workflow has read-only contents permission and nothing more, and both workflows list the matrix from the index.

Each of these mutations fails at least one test:

• the token back in ci.yml
• a pull_request trigger on the new workflow
• an install before the fetch
• dropping env -i, -I or --exclude-newer
• bare uv
• /usr/bin/printf instead of the shell's own
• the token in argv
• the XDG cache rule
• a job-level token
• contents: write

Checked on a real runner

The real fetch only runs on pushes, so this PR's CI cannot exercise it. A throwaway branch ran the step's exact script on ubuntu-latest with uv 0.13.4 (run 39697771059; the branch is deleted). An earlier step had planted these:

• impostor uv/python3/env on $GITHUB_PATH
• HF_ENDPOINT, UV_INDEX_URL, UV_DEFAULT_INDEX, UV_PYTHON, proxies and XDG_CACHE_HOME in $GITHUB_ENV
• a hostile uv.toml in the checkout and in ~/.config/uv
• a fake huggingface_hub package in the checkout, with PYTHONPATH=.

What happened:

• No impostor ran, and PyPI served the packages.
• The private repo with a dummy token got a 401 / GatedRepoError from huggingface.co itself.
• A public file landed in /home/runner/.cache/huggingface/hub, where find_dataset_bytes looks.

The run predates --exclude-newer; I checked that resolution locally (all 15 packages). One reviewer separately confirmed with real uv that Python receives the token on stdin. The real token's fetch and replay first run on the push that merges this, and I'll watch that run.

Reviews

Two independent Opus 5.5 reviews of the first version (Codex lanes were closed) both asked to change the header, which overclaimed. This version addresses that and their test findings.

• Not changed: the cache rules for an empty HF_HOME and for ~ inside a value. The test's docstring now names that domain.
• Not changed: the actions are still on version tags rather than commit SHAs, and setup-uv's version isn't pinned, both repo-wide practice. Left for a follow-up.

Locally: the queue, CI-gate, engine-matrix, determinism and results-PR test files give 166 passed. ruff format --check is clean, and actionlint reports only the existing SC2129 style hint on the copied index-read step.

axiom: n/a: CI only

🤖 Generated with Claude Code

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated
policyengine-scorecard	Ready	Preview	Oct 22, 2026 3:04am UTC