Skip to content

Support bundle-pinned local data release manifests - #704

Draft
MaxGhenis wants to merge 8 commits into
mainfrom
budget-local-data-manifest
Draft

MaxGhenis wants to merge 8 commits into
mainfrom
budget-local-data-manifest

Conversation

@MaxGhenis

@MaxGhenis MaxGhenis commented Oct 23, 2026 •

Copy link
Copy Markdown
Contributor

Fixes #703

New generation processes can certify from a local release manifest when Hugging Face is unavailable. POLICYENGINE_<COUNTRY>_DATA_MANIFEST_FILE always selects the configured file, avoiding an outage timeout. With the setting unset, normal fetch, authentication, caching, offline fallback and certification behavior matches main for consistent bundles.

Certification invariants

A local manifest certifies only when all three conditions hold:

  1. Its raw bytes match the installed country TRACE TRO's single valid manifest pin, before parsing or revision rewriting. Every definition of composition/1/artifact/data_release_manifest anywhere in the TRO must agree on that SHA-256. Missing, malformed, conflicting or stale pins refuse; no environment digest is accepted.
  2. After revision rewriting, its certified/default dataset artifact agrees with the bundle's recorded repo, path, revision and pinned SHA-256. Every overlapping managed artifact must also agree with the bundle's records. Regional artifacts and additive overlays absent from the primary manifest remain permitted.
  3. The unchanged model, core, data-build, fingerprint and exact-pair checks pass. Local selection cannot broaden certification.

The artifact-consistency check also applies to fetched manifests. This closes the pre-existing inconsistent-bundle case there without changing consistent bundles. Certification and runtime validation share the existing release-relative diagnostics/evidence path normalization.

Local files are read and reverified on every access, including after a cached fetch. Successful local certification appends _local_manifest to the existing basis. Packaged receipts record optional runtime.data_certification_basis, preserving the published bundle certificate basis and existing digest domains. Unset receipts omit the annotation. The loader's callable facade retains typed cache_clear and cache_info controls.

Validation

3,322 unique test cases passed, 5 skipped across 56 provenance/certification/disclosure modules and focused release/bootstrap checks. All 82 tests in tests/test_local_data_manifest.py and all 12 TRO-pin tests passed. The final affected manifest/certification run passed 280 tests, and the release/bootstrap-focused run passed 189 tests. The broad run's only failure was an old error-message assertion for a missing default artifact; it was updated for the earlier refusal and its complete 40-test module passed on rerun.

Regression coverage includes a direct @graph conflicting hash, an 80-example Hypothesis property over duplicate placement and nesting, and the reviewer's digest-matching manifest at adversary/elsewhere/elsewhere.h5 under an unchanged bundled dataset URI. Local and fetched cases cover artifact identity/hash conflicts, revision rewriting, inherited artifacts and release-relative evidence paths. Existing exact-pair and unset main-snapshot comparisons remain.

make format, make lint, focused mypy --follow-imports=silent, git diff --check, and generated receipt schema checks pass. Each pytest invocation checked load1 ≤110 and available disk ≥55,000,000,000 bytes; no pytest workers were used.

Deployment and downstream adoption

The G0 example exports the setting only after successful download, raw-byte verification and atomic replacement. Fetch and verify the manifest for the exact installed certified bundle whenever that release changes, then start a fresh process and retain its receipt. Release coordination has been removed from permanent public docs.

  • Dashboard: refresh vendored tests/fixtures/w4/execution_receipt.v1.schema.json when upgrading; its schema equality test requires the updated optional runtime field.
  • Scorecard: pipeline/uk_replay_variants.py:730 preserves the new setting during offline isolation, while :750 requires exact runtime bundle equality. Clear POLICYENGINE_*_DATA_MANIFEST_FILE during that isolation.
  • Other repositories were not edited.

No package versions, dependency pins, datasets or certified releases changed. Leave this PR draft for hub review; do not merge automatically.

@MaxGhenis

Copy link
Copy Markdown
Contributor Author

UK hub (Claude session, for Max): independent review round 2 (GPT-6.1 Sol) at da2ae8fb: both round-1 blockers are fixed, and no code blocker remains. The only gate left is CI, which hasn't started yet because runners are queued.

  • Conflicting TRO pins, whether top-level, deeply nested or malformed, now refuse. A repeated definition with the same digest is allowed.
  • The adversary/elsewhere/elsewhere.h5 manifest refuses through both the local and the fetched path, as do revision-only and digest-only changes. Local probes made zero network calls.
  • On the fetched path, the reviewer fetched the real UK manifest live (HTTP 200, SHA-256 49f9fa82…, matching the installed pin) with both this PR and main's module. All 11 certification fields match. 17 more consistent cases certify identically. The 4 cases main accepted and this PR refuses all genuinely conflict with the bundle's records.
  • The docs export is now conditional on a verified download, the coordination paragraph is gone, mypy is clean, and the schema check passes. 1,862 tests passed.
  • Nonblocking follow-up: a certified-only bundle shape (a default in certified_data_artifact with no matching datasets entry) skips the repo_type check (manifest.py:590). Current and generated bundles always have the entry, and managed materialisation already refuses that shape.

Held unmerged on purpose: this ships in the same release as the weekend Microcosm UK cut certification, which will stack on this branch.

The note was a Subfleet job artifact, not part of the change. The hub
keeps a copy outside the repository.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
MaxGhenis added a commit that referenced this pull request Oct 23, 2026
#704 dropped its job hand-off note. This merge takes that deletion and
also removes #699's own .job/HANDOFF.md and .job/PR_BODY.md: Subfleet
job artifacts, not part of the change. The hub keeps copies outside the
repository.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Support opt-in bundle-pinned local data release manifests

1 participant