Support bundle-pinned local data release manifests

changelog.d/703.added.md

@@ -0,0 +1 @@
+Support opt-in local data release manifests pinned by the installed bundle TRO, with fail-closed verification and a distinct runtime certification basis for offline Budget-day runners. Reject conflicting TRO hash assertions and require local and fetched manifest artifacts to agree with the installed bundle.

docs/bundles.md

@@ -116,6 +116,68 @@ policyengine bundle manifest 4.19.1
 `install` targeted. Use `--venv` or `--python` only to inspect a different
 target explicitly.
 
+## Local release manifests for Budget-day runners
+
+Set `POLICYENGINE_UK_DATA_MANIFEST_FILE` before starting a generation process to
+read the UK release manifest from that file. The same setting is available as
+`POLICYENGINE_<COUNTRY>_DATA_MANIFEST_FILE` for other bundled countries. When set,
+it always selects the local file, avoiding a Hugging Face request and its timeout.
+An unset setting preserves the existing fetch and certification behavior.
+
+The trust anchor is the `trov:sha256` on
+`composition/1/artifact/data_release_manifest` in the installed country's
+`data/bundle/{country}.trace.tro.jsonld`. The file's **raw bytes** must match that
+pin. A missing pin, unreadable file, altered bytes, or certification mismatch
+stops the process. Every definition of the artifact ID in the TRO must agree on
+one digest. After revision normalization, the manifest's certified/default
+artifact and all artifacts shared with the bundle must agree with the bundle's
+locations and pinned hashes. This artifact check also applies to fetched
+manifests. The model, core, build ID and fingerprint checks still apply;
+selecting a local file cannot extend the bundle's certification.
+
+At G0 each morning, while Hugging Face is available, use the exact installed
+release that the runner will use and download and verify the manifest before
+enabling the setting. Keep the Hugging Face token in `HUGGING_FACE_TOKEN`,
+`HF_TOKEN` or `HUGGINGFACE_HUB_TOKEN`:
+
+```bash
+env -u POLICYENGINE_UK_DATA_MANIFEST_FILE POLICYENGINE_SKIP_COUNTRY_IMPORTS=1 python - <<'PY' &&
+import hashlib
+from pathlib import Path
+
+import requests
+from policyengine.provenance.manifest import (
+    bundled_data_release_manifest_sha256, get_release_manifest,
+    hugging_face_auth_headers, https_release_manifest_uri,
+)
+
+expected = bundled_data_release_manifest_sha256("uk")
+response = requests.get(
+    https_release_manifest_uri(get_release_manifest("uk").data_package),
+    headers=hugging_face_auth_headers(), timeout=30,
+)
+response.raise_for_status()
+actual = hashlib.sha256(response.content).hexdigest()
+if actual != expected:
+    raise RuntimeError(f"Manifest SHA-256: expected {expected}; actual {actual}")
+target = Path("data/uk-release-manifest.json")
+target.parent.mkdir(parents=True, exist_ok=True)
+staged = target.with_suffix(".json.tmp")
+staged.write_bytes(response.content)
+staged.replace(target)
+print(f"Verified {target}: {actual}")
+PY
+export POLICYENGINE_UK_DATA_MANIFEST_FILE="$PWD/data/uk-release-manifest.json"
+```
+
+Start a fresh runner process with that setting and record its receipt at G0.
+Runtime certification and TRACE output use a distinct basis ending in
+`_local_manifest`, such as `exact_build_model_version_local_manifest`; packaged
+execution receipts record it as `runtime.data_certification_basis`. The
+receipt's certificate keeps the bundle's published basis so certificate
+verification remains reproducible. See [execution receipts](receipts.md).
+Refresh and verify the file whenever the installed certified release changes.
+
 ## Bundle-only PRs
 
 Run:

docs/receipts.md

@@ -47,6 +47,25 @@ receipt_json = receipt.to_canonical_json()
 
 For a country `Microsimulation` built with `managed_microsimulation()`, call `build_execution_receipt_for_microsimulation(sim, years=..., reform=..., baseline=...)`. It reads the dataset that was loaded from `sim.policyengine_bundle`. The file must still hash to the SHA-256 verified when it was loaded, and the receipt must name the dataset, URI, revision and package recorded then; otherwise it is refused, because it would describe bytes that did not run. A simulation loaded from an unmanaged file or URI recorded no load-time hash, so it cannot be receipted. The microsimulation does not record its reform, so pass the reform you built it with.
 
+With `POLICYENGINE_UK_DATA_MANIFEST_FILE` enabled, a packaged-bundle receipt
+rechecks runtime certification using the local file and records its distinct
+`*_local_manifest` basis in `runtime.data_certification_basis`. Local-file
+verification errors stop receipt creation. The certificate's
+`compatibility_basis` stays the published bundle's assertion, which
+`verify_certificate` can reproduce. Like the other runtime context, this source
+annotation is excluded from execution and receipt digests. Caller-supplied
+bundles do not claim this packaged-bundle runtime annotation.
+
+The execution-receipt schema accepts the annotation as an optional string;
+compatibility bases in the certificate and TRACE TRO are also strings. The UK
+Budget dashboard's receipt adapter uses this package's receipt model and
+`verify_certificate`, so it needs no basis allowlist change. When adopting this
+release, refresh the dashboard's vendored
+`tests/fixtures/w4/execution_receipt.v1.schema.json`: its
+`tests/test_released_receipt.py` checks exact equality against the installed
+schema. Older consumers that forbid extra runtime fields need the updated
+schema before reading these receipts.
+
 There are two ways to bind a policyengine-uk `Scenario`:
 - A scenario built from `parameter_changes` is bound by content with `scenario_reform_identity(scenario)`.
 - `Scenario.from_reform(d)` wraps `d` in a closure that cannot be bound by its code, so bind `d` itself with `reform_identity(d, "policyengine-uk.reform_dict/v1")`.

src/policyengine/data/schemas/execution_receipt.v1.schema.json

@@ -473,6 +473,19 @@
           "default": null,
           "title": "Ci Run Url"
         },
+        "data_certification_basis": {
+          "anyOf": [
+            {
+              "type": "string"
+            },
+            {
+              "type": "null"
+            }
+          ],
+          "default": null,
+          "description": "Runtime certification basis when an opt-in local data manifest was verified against the packaged bundle TRACE TRO. The certificate's compatibility_basis remains the published bundle's assertion.",
+          "title": "Data Certification Basis"
+        },
         "platform": {
           "title": "Platform",
           "type": "string"

src/policyengine/provenance/certification.py

@@ -62,6 +62,12 @@
     fetch_pypi_wheel_metadata,
     release_build_id,
 )
+from policyengine.provenance.manifest import (
+    RELEASE_SCOPED_ARTIFACT_KINDS as RELEASE_SCOPED_ARTIFACT_KINDS,
+)
+from policyengine.provenance.manifest import (
+    artifact_path_for_country_manifest as artifact_path_for_country_manifest,
+)
 
 MANIFEST_URI_PATTERN = re.compile(
     r"^hf://(?P<repo_type>dataset|model)/(?P<repo_id>[^@]+)@(?P<revision>[^/]+)"
@@ -113,9 +119,6 @@
     CONSTITUENCY_ASSET_SPEC.weight_matrix_filename,
     LOCAL_AUTHORITY_ASSET_SPEC.weight_matrix_filename,
 )
-# Artifact kinds published beside the release manifest rather than at the repo
-# root, so their manifest paths are relative to the release directory.
-RELEASE_SCOPED_ARTIFACT_KINDS = frozenset({"diagnostics", "evidence"})
 US_STATE_CODES = (
     "AL",
     "AK",
@@ -526,19 +529,6 @@ def head_release_file(
     return response.status_code == 200
 
 
-def artifact_path_for_country_manifest(artifact, uri_parts: dict) -> str:
-    path = artifact.path
-    release_dir = release_manifest_dir(uri_parts)
-    if (
-        artifact.kind in RELEASE_SCOPED_ARTIFACT_KINDS
-        and artifact.repo_id == uri_parts["repo_id"]
-        and artifact.revision == uri_parts["revision"]
-        and not path.startswith(f"{release_dir}/")
-    ):
-        return f"{release_dir}/{path}"
-    return path
-
-
 def artifact_reference_url(
     reference: dict, uri_parts: dict, prefix: str = "datasets/"
 ) -> str: