Skip to content

feat(standard): read a data provider's token from the environment in a headless run - #153

Merged
aitianshi merged 1 commit into
mainfrom
fix/151-hf-token-env
Nov 16, 2026
Merged

aitianshi merged 1 commit into
mainfrom
fix/151-hf-token-env

Conversation

@aitianshi

Copy link
Copy Markdown
Collaborator

Closes #151

What

A headless nullius standard run or prove (and --shard, the same path) on this machine takes a data-provider token the benchmark declares from the process environment, under the provider's variable (HF_TOKEN for Hugging Face), before the provider's store. A stored token still works when the variable is unset or empty.

  • run.Licences takes an optional environ; at construction it keeps only the values of providers the benchmark declares, and get returns one before reading the store.
  • cli.licences passes the process environment only when the run is not --from-run and no host line (NULLIUS_LICENCES) was handed. App runs and host runs behave as before.
  • Scope: data-provider tokens only. Other benchmark licences still come from the benchmark's store or the host line, never the environment.

The token takes the same path as a stored one: driver_secrets to the declaring driver alone (a provider token is validated as to: driver), and into the run's redaction list. Every child process still gets only run.KEEP from the environment, so the agent, the relay and setup or conformance processes cannot inherit it.

Touches

  • app/nullius/standard/run.py (Licences), app/nullius/standard/cli.py (licences, module docstring)
  • app/tests/test_secrets.py, app/tests/test_standard_environments.py, the envbench probe agent (reports whether it sees HF_TOKEN)
  • docs/cli/run.md, docs/settings/data-providers.md

Verified

  • New tests: environment token with an empty store; environment wins over a stored token; stored token used when the variable is unset, empty or blank; --from-run reads nothing; a host line wins; a benchmark declaring no provider (or the same name without a provider) gets nothing. End to end on the envbench fixture, local and container backends: the declaring driver gets the token, the agent sees neither its name nor HF_TOKEN (set in the parent process too), and no job file contains it.
  • python3 app/tests/parallel.py: 1996 tests in 116 modules, OK. DockerRun in test_standard_environments with NULLIUS_TEST_DOCKER=1: 2 tests, OK. No app/web change, so the route suite was not run.
  • Real run, temporary NULLIUS_HOME with an empty store, HF_ENDPOINT=https://huggingface.co: nullius standard run gaia oracle --dataset gaia-validation-level1 --limit 1 with HF_TOKEN in the environment passed (1/1, reward 1.0); the token is in no job or profile file. The same command without HF_TOKEN ended in EnvironmentSecretMissingError.

Risk

A shell that exports a stale HF_TOKEN now overrides the token saved in Settings for headless standard runs (not for app runs). That is the documented precedence, as for model keys and HF_ENDPOINT; unset the variable to use the stored one. Reverting the commit restores store-only reads.

…a headless run

A headless standard run or prove takes a data-provider token the benchmark
declares (HF_TOKEN for Hugging Face) from the process environment before
the provider's store. --from-run and host runs are unchanged.
@github-actions

Copy link
Copy Markdown

🟡 PR judge · 7 of 7 yes · #153 ↔ #151

Merge risk 🔴 MAJOR the judge's reading of the diff; app/nullius/standard/cli.py is in the path of every run
Priority 🔴 EXTREME the deepest path is app/nullius/standard/cli.py
Question Verdict
Changes do what the issue asks · 7 of 7 criteria ✅ yes
Issue and pull request describe the same change ✅ yes
Description matches the diff, no more, no less ✅ yes
Diff stays in scope ✅ yes
Tests cover the change, or Verified says how ✅ yes
Docs follow the change ✅ yes
Risk named in the description ✅ yes
Before and after shown –
Acceptance criteria, 7 of 7 met
# Criterion Verdict
1 A standard run on this machine uses HF_TOKEN from the environment for a benchmark whose environment declares the Hugg… ✅ met
2 The environment value wins over a stored token; a stored token still works when the variable is unset or empty ✅ met
3 The token reaches only the declaring benchmark's driver, on both local and --backend container; the agent, the rela… ✅ met
4 A benchmark that declares no data provider gets no HF_TOKEN from the environment ✅ met
5 A run started by the app (--from-run) and a run on a host behave as today ✅ met
6 docs/settings/data-providers.md and docs/cli/run.md say that a headless standard run reads the provider's variable … ✅ met
7 Tests cover the environment token, its precedence over the store, that it reaches only the driver, and the undeclared c… ✅ met
Checks the script runs itself, 4 of 4 pass
No machine attribution trailer in the commits or the description ✓
New code under app/, docs/, .github/ or .claude/ ✓
No Flask import under the backend package ✓
No key-shaped string in the diff ✓

@aitianshi
aitianshi merged commit c02a5e4 into main Nov 16, 2026
2 checks passed
@aitianshi
aitianshi deleted the fix/151-hf-token-env branch November 16, 2026 13:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Standard: a headless run ignores HF_TOKEN in the environment, so a CI cannot give GAIA its token

1 participant