Provide an API to resolve, verify and load the datasets .py certifies #667

Problem

.py's bundle manifest (policyengine/data/bundle/manifest.json) records, for each country, the certified data release (repository, revision, files and their sha256) and the exact country-package versions each extra pins. Repos that run models on that data keep re-deriving this information in their own code instead of asking .py for it.

The clearest example is calibration-diagnostics, which moved to policyengine[us]==6.4.1 in PolicyEngine/calibration-diagnostics#222. frontend/scripts/hosted_release.py implements:

• certifying_package: reads the .py version and extra that a checked-in hosted_release.json names as the certifier. That file repeats the release ID, Hugging Face commit, file name, sha256 and package versions.
• py_pinned_packages: parses the installed .py distribution's Requires-Dist entries and evaluates the extra markers to get the exact policyengine-us, policyengine-core and spm-calculator versions.
• py_bundle_manifest: opens policyengine/data/bundle/manifest.json by path through importlib.metadata.
• validate_py_certification: matches the release ID, data build ID, dataset sha256 and URI, repository, release and build manifest paths, revision, model version and every file digest against data_releases["us"].
• download_certification and verify_h5: download the release manifest, build manifest and H5 with huggingface_hub, then hash them.

verify_hosted_runtime.py and generate_variable_catalog.py use these functions. The same repo's evaluation harness, in evaluation_harness/adapters/microcosm.py, hard-codes the same release ID and sha256 a second time and has its own download and sha256 check.

Other repos do similar work:

• policybench, policyengine_runtime.py: matches Requires-Dist with a regex to find the model pin, and reads the manifest by path, with a fallback for the per-country manifest files that 4.x shipped.
• obbba-household-by-household, export_microcosm_explorer.py: reads data_releases["us"] by path.
• policyengine-sim-api, release_bundle.py: walks the raw dict from get_current_bundle() for package versions, dataset URIs and digests.
• uk-triple-lock, datasets.py: hard-codes release manifest and dataset digests, with its own download, hashing and DatasetMismatch error.
• uk-autumn-budget-dashboard-2026, bundle_adoption.py: reads data_releases["uk"]["certified_data_artifact"]["sha256"].

All of these depend on the manifest's internal layout, which has already changed once (from per-country manifests in 4.x to data_releases in 6.x). Each also needs a manual edit whenever .py certifies a new release.

What .py already provides

Most of the building blocks exist, and all of the following were already in 6.4.1:

• policyengine.bundle.get_current_bundle() and load_bundle_manifest() return the manifest as a dict.
• policyengine.provenance.get_release_manifest(country) returns a typed CountryReleaseManifest.
• policyengine.bundle.bundle_install_requirements(countries=[...]) returns exact requirement strings.
• policyengine.bundle.inspect_bundle_status() and policyengine bundle status|verify compare installed versions, and the dataset files listed in a bundle receipt, against the bundle. They return status dicts.
• policyengine.provenance.materialize_dataset(country, name, data_dir=...) downloads a certified file, verifies its sha256 and reuses a matching local copy.
• policyengine.tax_benefit_models.{us,uk}.managed_microsimulation(dataset=...) builds a simulation from a certified dataset. The US version also applies .py's stored-input renames (would_claim_wic to takes_up_wic_if_eligible) and legacy SPM declarations, which a consumer that constructs policyengine_us.Microsimulation directly does not get.

None of the consumers above uses these for the whole flow. These gaps send them back to their own code:

• Verifying a local file. There is no public way to check a file the caller already has (from the Hugging Face cache, or a --dataset-file argument) against the certified sha256 without downloading it again. _reuse_or_download_bundle_files is private, and it replaces a mismatching file rather than refusing it. A local path passed to materialize_dataset or managed_microsimulation needs allow_unmanaged=True, and then loses its certified identity (bundle_dataset is None).
• Failing on a version mismatch. No function raises when the installed environment differs from the pins. inspect_bundle_status returns dicts, and the country model version only emits a UserWarning when the installed country package differs from the pin.
• Checking a release. No call answers "does this .py certify this repository, revision and sha256?". Consumers compare the fields themselves.
• Pins as a mapping. Pins are available as requirement strings or as the raw packages and extras blocks, but not as a {package: version} mapping for one extra.
• Manifest identity. The bundle records no digest for the US release_manifest.json, and the US revisions are tag names (populace-us-2024-spm-20260915), not commit SHAs. calibration-diagnostics therefore pins the commit (8ab57ffc…, which the tag currently resolves to) and its own manifest digests.
• Cache location. materialize_dataset defaults to ./data, relative to the working directory.
• Error types. Errors are BundleError, DatasetMaterializationError, DataReleaseManifestUnavailableError or plain ValueError, all ValueError subclasses. A version mismatch, a digest mismatch and an unknown dataset can be told apart only by their messages.

Proposed functionality

This is a list of capabilities, not a fixed design. One documented module that builds on the code above would cover it:

• List the datasets certified for a country.
• Get a certified release's identity: repository, repository type, revision (tag and commit), file path, sha256, and the release and build manifests with their digests.
• Get the exact country-package versions the installed .py pins for an extra, as a mapping.
• Verify the installed environment against those pins, and raise on mismatch.
• Check whether a given repository, revision and sha256 is a release this .py certifies.
• Download a certified dataset or release file, with digest verification and caching in a user cache directory or the Hugging Face cache.
• Verify a local file against a certified record without downloading it, and keep its certified identity.
• Load a verified file into a simulation through the same path as managed_microsimulation, so the stored-input renames and SPM handling apply.
• Raise typed errors on mismatch, such as a package version mismatch, a dataset digest mismatch or an uncertified dataset.
• Optionally, a CLI equivalent (for example policyengine bundle verify --dataset-file …) for image builds.

Why it helps

The certified release would have one source of truth. Consumers would stop copying release IDs and digests into their own JSON and constants, and would stop depending on the manifest's layout. An upgrade would then mean bumping .py in each repo. calibration-diagnostics alone could drop most of hosted_release.py, hosted_release.json and the release constants in its harness.

Priority

Low; this is a nice-to-have. Nothing is blocked, and the consumers above work today with their own code.

Open questions

• Private datasets and authentication: the UK default lives in policyengine/policyengine-uk-data-private. Should the API accept a token argument as well as HUGGING_FACE_TOKEN, and report which credential it needs before it starts a download?
• Offline use: should verifying and loading work without network access when the files are already local? This includes the release manifest that get_data_release_manifest currently fetches.
• Revisions and digests: should the bundle record a commit SHA for each artifact, as it already does for the UK release manifest revision, plus a digest of each release manifest?
• JSON digests: should the JSON manifests be identified by byte sha256 (what .py records) or by a canonical-JSON digest (what calibration-diagnostics records)?
• Historical bundles: should the API also read older bundles through load_bundle_manifest(version), so a consumer can check a release that an earlier .py certified?
• First adopters: which consumers would adopt this first? calibration-diagnostics (hosted runtime and harness) and policybench look like the smallest changes.

Related: #270 (the manifest layer), #502 (unified dataset materialization), #415 (local files in managed_microsimulation) and #591 (year-file identity).

🤖 Generated with Claude Code