Repository navigation
feat(standard): read a data provider's token from the environment in a headless run - #153
Merged
Merged
Conversation
…a headless run A headless standard run or prove takes a data-provider token the benchmark declares (HF_TOKEN for Hugging Face) from the process environment before the provider's store. --from-run and host runs are unchanged.
🟡 PR judge · 7 of 7 yes · #153 ↔ #151
Acceptance criteria, 7 of 7 met
Checks the script runs itself, 4 of 4 pass
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #151
What
A headless
nullius standard runorprove(and--shard, the same path) on this machine takes a data-provider token the benchmark declares from the process environment, under the provider's variable (HF_TOKENfor Hugging Face), before the provider's store. A stored token still works when the variable is unset or empty.run.Licencestakes an optionalenviron; at construction it keeps only the values of providers the benchmark declares, andgetreturns one before reading the store.cli.licencespasses the process environment only when the run is not--from-runand no host line (NULLIUS_LICENCES) was handed. App runs and host runs behave as before.The token takes the same path as a stored one:
driver_secretsto the declaring driver alone (a provider token is validated asto: driver), and into the run's redaction list. Every child process still gets onlyrun.KEEPfrom the environment, so the agent, the relay and setup or conformance processes cannot inherit it.Touches
app/nullius/standard/run.py(Licences),app/nullius/standard/cli.py(licences, module docstring)app/tests/test_secrets.py,app/tests/test_standard_environments.py, the envbench probe agent (reports whether it seesHF_TOKEN)docs/cli/run.md,docs/settings/data-providers.mdVerified
--from-runreads nothing; a host line wins; a benchmark declaring no provider (or the same name without a provider) gets nothing. End to end on the envbench fixture, local and container backends: the declaring driver gets the token, the agent sees neither its name norHF_TOKEN(set in the parent process too), and no job file contains it.python3 app/tests/parallel.py: 1996 tests in 116 modules, OK.DockerRunintest_standard_environmentswithNULLIUS_TEST_DOCKER=1: 2 tests, OK. Noapp/webchange, so the route suite was not run.NULLIUS_HOMEwith an empty store,HF_ENDPOINT=https://huggingface.co:nullius standard run gaia oracle --dataset gaia-validation-level1 --limit 1withHF_TOKENin the environment passed (1/1, reward 1.0); the token is in no job or profile file. The same command withoutHF_TOKENended inEnvironmentSecretMissingError.Risk
A shell that exports a stale
HF_TOKENnow overrides the token saved in Settings for headless standard runs (not for app runs). That is the documented precedence, as for model keys andHF_ENDPOINT; unset the variable to use the stored one. Reverting the commit restores store-only reads.