Publish v2 catalog regions from checked-in reference files and the certified data release

.github/workflows/seed-v2-database.yml

@@ -66,3 +66,6 @@ jobs:
         run: uv run python scripts/publish_v2_metadata_catalog.py
         env:
           V2_DATA_WRITE_DATABASE_URL: ${{ secrets.V2_DATA_WRITE_DATABASE_URL }}
+          # Reads the certified UK data release, whose repository is private,
+          # to check each new catalog version's region filters.
+          HUGGING_FACE_TOKEN: ${{ secrets.PE_UK_PRIVATE_HF_READ_TOKEN }}

.gitignore

@@ -23,3 +23,8 @@ gha-creds-*.json
 ## vscode settings
 /.vscode
 *.db.init.lock
+
+# coverage.py data and reports written by `make test`
+.coverage
+.coverage.*
+coverage.xml

changelog.d/v2plan-1f-123-region-reference-files.changed.md

@@ -0,0 +1 @@
+Catalog publication reads regions from the Public API's checked-in reference files, one per country, instead of PolicyEngine.py's region registry. Each region takes its dataset from the certified data release and a row filter from a scoping rule: state_fips or congressional_district_geoid for US states and districts, the UK region column for UK nations, and constituency_code_oa or la_code_oa on the local-area dataset the release routes UK constituencies and local authorities to. The UK now publishes 1,016 regions, with 650 constituencies and 361 local authorities (including Ards and North Down) by ONS code and Northern Ireland as country/northern_ireland; the national regions are "United States" and "United Kingdom". US places are listed in the reference file but left out of the catalog until they can run, and the publication evidence counts them. Before it publishes a catalog version that is not stored yet, the publication command reads the HDF5 metadata of the certified datasets the filters use, from Hugging Face at their pinned revisions, and stops if a filter's column is missing; the seed workflow's publication step receives the UK data read token as HUGGING_FACE_TOKEN for this. A catalog version that is already stored keeps the regions it was published with, so region file changes take effect in the next catalog version; republishing it logs the difference in region rows instead of failing, its evidence counts the stored regions and lists the kept version under kept_regions, and every other table must still match.

docs/engineering/uk-private-hugging-face-credential.md

@@ -51,6 +51,14 @@ Cloud Run candidate resolver verifies that the exact deployed revision binds
 `HUGGING_FACE_TOKEN` to `pe-uk-private-hf-read-token:latest`, without printing
 the credential.
 
+The v2 catalog publication step of `.github/workflows/seed-v2-database.yml`
+also receives the organization secret as `HUGGING_FACE_TOKEN`, step-scoped.
+Before it publishes a new catalog version, the step reads, through one
+authenticated Hugging Face file system, the HDF5 metadata of the certified
+datasets that version's region filters use: the UK national and local-area
+datasets in the private UK data repository, and the public US dataset. It
+downloads only the 1 MB blocks that hold that metadata, not whole files.
+
 The simulation-entry Cloud Run service does not receive this credential. It
 submits requests to Modal and does not download model data.
 

docs/migration/stage-9-v2-metadata.md

@@ -64,8 +64,14 @@ must execute in this order:
    Alembic chain, confirms all heads are current, and detects ungenerated model
    changes.
 3. Remove the migration URL from the command environment.
-4. Supply only `V2_DATA_WRITE_DATABASE_URL` and the same target identity, then
-   run `uv run python scripts/publish_v2_metadata_catalog.py`.
+4. Supply only `V2_DATA_WRITE_DATABASE_URL`, the same target identity and
+   `HUGGING_FACE_TOKEN` (from the organization secret
+   `PE_UK_PRIVATE_HF_READ_TOKEN`), then run
+   `uv run python scripts/publish_v2_metadata_catalog.py`. Before publishing a
+   catalog version that is not stored yet, the command reads the certified data
+   artifacts its regions' filters use and stops if one cannot select a region.
+   It stops before reading anything if a UK version needs checking and
+   `HUGGING_FACE_TOKEN` is unset, because the UK data repository is private.
 5. Require the command to finish successfully and retain its non-secret JSON
    evidence before creating the candidate revision.
 6. Deploy the Cloud Run candidate with the target identity and
@@ -101,20 +107,50 @@ does not reorder, deduplicate, or repair package output. A value superseded on
 the immediately following day therefore has equal start and end dates and is a
 valid one-day interval.
 
-The US national region uses `populace_us_2024`. A US state,
-congressional-district, or place region uses its PolicyEngine.py regional
-alternative when one is available. Until a later reviewed change removes the
-fallback, a subnational region without such an alternative uses
-`populace_us_2024`. The command emits one summary warning containing only the
-affected region types and counts. The UK national and country regions use
-`microcosm_uk_2024_25`, the default certified by PolicyEngine.py 6.10.0; a
-country filters that national file. UK constituency and local-authority
-regions use the dataset PolicyEngine.py routes them to,
-`microcosm_uk_2024_25_dense`, because the national file has no area codes.
-
-Review the fallback summary on every new PolicyEngine.py release. A changed
-count may identify an upstream catalog change that needs a reviewed dataset
-selection even when publication otherwise succeeds.
+Regions come from the Public API's checked-in reference files, one per country
+(`policyengine_api/data/v2/catalog/region_reference_files/`), not from
+PolicyEngine.py's region registry. Each row lists a region's code, label, type,
+parent, US state, the value that selects its households, and the v1 identifier
+it replaces. A scoping rule per country and region type gives each region its
+dataset from the certified data release and its filter:
+
+| Regions | Dataset | Filter |
+| --- | --- | --- |
+| US national | the release's national dataset (`populace_us_2024`) | none |
+| US states and congressional districts | the release's state and district dataset | `row_filter` on `state_fips` or `congressional_district_geoid` |
+| UK national | the release's national dataset (`microcosm_uk_2024_25`) | none |
+| UK nations | the release's national dataset | `row_filter` on `region` |
+| UK constituencies and local authorities | the release's local-area dataset (`microcosm_uk_2024_25_dense`), which carries their codes | `row_filter` on `constituency_code_oa` or `la_code_oa`, value the ONS code |
+
+If the release names no dataset for a US state or district, those regions use
+the default dataset and the command emits one summary warning with the affected
+region types and counts. A UK constituency or local authority cannot fall back,
+since the national file has no area codes: a release that routes no dataset to
+their type stops publication. Region types without a rule, such as US places
+until the data can select them, are left out and counted in the evidence's
+`omitted_region_summaries`.
+
+Before a new catalog version is published, the command reads the HDF5 metadata
+of each dataset the row filters use, with ranged requests at the pinned
+revision, and stops if a filter's column is missing from its dataset's
+household columns. It does not check that a column holds rows for every
+region's value.
+
+Review the fallback and omitted-region summaries on every new PolicyEngine.py
+release. A changed count may identify an upstream catalog change that needs a
+reviewed dataset selection even when publication otherwise succeeds.
+
+A catalog version is identified by its country and PolicyEngine.py version and
+never changes once published. When the command republishes a stored version
+whose staged regions differ from the stored ones, it keeps the stored regions
+unchanged, prints a warning with the country, the version and the counts of
+staged region rows the stored version lacks and of stored rows not staged, and
+continues. Its evidence then describes what is stored: the region count is the
+stored one, `kept_regions` lists the country, version, stored and staged region
+counts, and the fallback and omitted-region summaries leave that country out,
+since they describe staged regions that were not published. Every other table
+must still match exactly. Region reference changes therefore take effect in the
+next catalog version that is published.
 
 ## Validation evidence and retry
 
@@ -123,7 +159,9 @@ Successful JSON evidence contains:
 - the canonical PolicyEngine.py version;
 - the dependency versions derived from its packaged manifest;
 - catalog entity counts;
-- US fallback counts;
+- fallback counts and omitted-region counts by country and region type, and
+  the stored and staged region counts of any stored version whose regions were
+  kept;
 - elapsed publication time; and
 - a success outcome.
 
@@ -142,8 +180,11 @@ confirms existing simulation and report record counts are unchanged.
 A failed attempt rolls back all catalog changes. Retry the same artifact only
 after correcting the external failure. A matching retry preserves identifiers,
 content, and row counts. Different normalized content under the same
-PolicyEngine.py version fails for operator review; it is never silently
-replaced. A later PolicyEngine.py version adds a catalog version without
+PolicyEngine.py version fails for operator review, with an error naming the
+first table that differs; it is never silently replaced. Regions are the
+exception: they come from the Public API's reference files, so a stored version
+keeps the regions it was published with (see "Source and regional dataset
+behavior"). A later PolicyEngine.py version adds a catalog version without
 deleting or modifying earlier versioned rows, including their logical datasets,
 regions, model descriptions, current-law IDs, and time-period options.
 

policyengine_api/data/v2/catalog/certified_data_inspection.py

@@ -0,0 +1,140 @@
+"""Read household column names from certified HDF5 dataset files.
+
+The artifacts are read from Hugging Face at their pinned revisions through
+ranged requests. Only the 1 MB blocks holding the HDF5 metadata and the
+column-name arrays are downloaded, a small part of each file; the household
+arrays themselves are never read.
+"""
+
+from __future__ import annotations
+
+from typing import TYPE_CHECKING, BinaryIO, Protocol, TypeVar
+from urllib.parse import quote
+
+import h5py
+from huggingface_hub import HfFileSystem
+
+from policyengine_api.data.v2.catalog.errors import CertifiedDataReadError
+
+if TYPE_CHECKING:
+    from collections.abc import Callable
+    from contextlib import AbstractContextManager
+
+    from policyengine_api.data.v2.catalog.records import CertifiedArtifact
+
+HOUSEHOLD_TABLE = "household"
+# PyTables names the columns a table stores together "values_block_<n>"; their
+# real names live only in pickled attributes.
+VALUE_BLOCK_PREFIX = "values_block_"
+TABLE_INDEX_COLUMN = "index"
+
+_Result = TypeVar("_Result")
+
+
+# HDF5 metadata is read in many small, scattered pieces; caching whole blocks of
+# this size keeps the number of ranged requests low.
+READ_BLOCK_BYTES = 1 << 20
+
+
+class ArtifactFileSystem(Protocol):
+    """Opens a file in a Hugging Face repository at a revision for reading."""
+
+    def open(
+        self, path: str, mode: str, *, block_size: int, cache_type: str
+    ) -> AbstractContextManager[BinaryIO]:
+        """Return a readable, seekable binary file that caches blocks it reads."""
+        ...
+
+
+def huggingface_path(artifact: CertifiedArtifact) -> str:
+    """Return the Hugging Face file system path of an artifact at its revision."""
+    prefix = "datasets/" if artifact.repo_type == "dataset" else ""
+    revision = quote(artifact.revision, safe="")
+    return f"{prefix}{artifact.repo_id}@{revision}/{artifact.path}"
+
+
+def _text(value: object) -> str:
+    return value.decode("utf-8") if isinstance(value, bytes) else str(value)
+
+
+def _is_frame(group: h5py.Group) -> bool:
+    return "axis0" in group or "table" in group
+
+
+def _frame_columns(group: h5py.Group) -> frozenset[str]:
+    """Return the column names of one pandas data frame stored in HDF5."""
+    if "axis0" in group:
+        return frozenset(_text(value) for value in group["axis0"][()])
+    names = group["table"].dtype.names or ()
+    if any(name.startswith(VALUE_BLOCK_PREFIX) for name in names):
+        msg = (
+            f"household table {group.name.lstrip('/')} stores columns in value "
+            "blocks, whose names cannot be read without unpickling"
+        )
+        raise CertifiedDataReadError(msg)
+    return frozenset(name for name in names if name != TABLE_INDEX_COLUMN)
+
+
+def household_columns_in_hdf5(h5: h5py.File) -> frozenset[str]:
+    """Return the household column names of a dataset file.
+
+    A pandas file stores the household frame directly, or one frame per year
+    (then only columns every year has count). A policyengine-core file stores
+    one top-level entry per variable.
+    """
+    if HOUSEHOLD_TABLE not in h5:
+        return frozenset(h5.keys())
+    household = h5[HOUSEHOLD_TABLE]
+    if not isinstance(household, h5py.Group):
+        msg = "household entry is not an HDF5 group"
+        raise CertifiedDataReadError(msg)
+    if _is_frame(household):
+        return _frame_columns(household)
+    frames = [
+        entry
+        for entry in household.values()
+        if isinstance(entry, h5py.Group) and _is_frame(entry)
+    ]
+    if not frames:
+        msg = "household group holds no data frame"
+        raise CertifiedDataReadError(msg)
+    return frozenset.intersection(*(_frame_columns(frame) for frame in frames))
+
+
+def _hugging_face_file_system(token: str | None) -> ArtifactFileSystem:
+    return HfFileSystem(token=token)
+
+
+class HuggingFaceDataInspector:
+    """Reads certified artifacts from Hugging Face with an optional read token."""
+
+    def __init__(
+        self,
+        *,
+        token: str | None,
+        file_system_factory: Callable[[str | None], ArtifactFileSystem] = (
+            _hugging_face_file_system
+        ),
+    ) -> None:
+        """Build the file system once; a token of None reads public data only."""
+        self._file_system = file_system_factory(token)
+
+    def _read(
+        self,
+        artifact: CertifiedArtifact,
+        read: Callable[[h5py.File], _Result],
+    ) -> _Result:
+        with (
+            self._file_system.open(
+                huggingface_path(artifact),
+                "rb",
+                block_size=READ_BLOCK_BYTES,
+                cache_type="blockcache",
+            ) as handle,
+            h5py.File(handle, "r") as h5,
+        ):
+            return read(h5)
+
+    def household_columns(self, artifact: CertifiedArtifact) -> frozenset[str]:
+        """Return the household column names of a dataset artifact."""
+        return self._read(artifact, household_columns_in_hdf5)

policyengine_api/data/v2/catalog/errors.py

@@ -0,0 +1,14 @@
+"""Errors raised while catalog sources are read and normalized."""
+
+from __future__ import annotations
+
+
+class CatalogExtractionError(RuntimeError):
+    """Raised before database access when source metadata cannot be normalized."""
+
+
+class CertifiedDataReadError(RuntimeError):
+    """Raised when a certified artifact does not have the structure expected.
+
+    Its message names only HDF5 entries and counts, so it is safe to print.
+    """