Certify the Microcosm UK national and local-area pair with policyengine-uk 2.132.6

.github/workflows/certify-uk-release.yaml

@@ -7,7 +7,8 @@
 # mandatory gates on that exact commit, replayed by policyengine-scorecard,
 # and recorded in the uk-qualification-ledger branch. Promotion is
 # serialized: it readies the newest promotable candidate PR that advances
-# main, closes superseded ones, and (cos d1344) fast-forwards main to its
+# main, closes superseded ones and those of pairs from another data line
+# (which are never promoted), and (cos d1344) fast-forwards main to its
 # qualified commit once every check on it has passed, which succeeds only
 # while main is still the commit it was built on; otherwise the catch-up
 # requalifies it on current main.

.github/workflows/push.yaml

@@ -316,6 +316,11 @@ jobs:
           VERSION: ${{ needs.Publish.outputs.version }}
           HUGGING_FACE_TOKEN: ${{ secrets.HUGGING_FACE_TOKEN }}
         run: python scripts/release_consumers.py scorecard-request --version "$VERSION" > "$RUNNER_TEMP/scorecard-request.json"
+      - name: Warn that no released-bundle replay was requested
+        if: steps.released.outcome == 'failure' || steps.scorecard-request.outcome == 'failure'
+        env:
+          VERSION: ${{ needs.Publish.outputs.version }}
+        run: echo "::warning title=No released-bundle replay::The scorecard's replay request for policyengine $VERSION could not be built; see the steps above."
       - name: Generate GitHub App token for the scorecard
         if: steps.scorecard-request.outcome == 'success'
         id: scorecard-token

README.md

@@ -70,9 +70,10 @@ print(f"Total UC spending: £{agg.result / 1e9:.1f}bn")
 For baseline-vs-reform comparisons, see `pe.uk.economic_impact_analysis`
 and its US counterpart.
 
-UK population data is stored in a private Hugging Face model repository. Set
-`HUGGING_FACE_TOKEN` to a token from an account with access before running UK
-population examples. To download the raw `.h5` file directly, see
+UK population data is stored in private Hugging Face repositories: the
+Microcosm default in a dataset repository, and the enhanced FRS in a model
+repository. Set `HUGGING_FACE_TOKEN` to a token from an account with access
+before running UK population examples. To download the raw `.h5` file directly, see
 [Microsimulation](docs/microsim.md#uk-private-data-and-raw-h5-access).
 
 ## Documentation

changelog.d/certify-uk-microcosm-uk-2024-25-national-20261019T083323Z-6cba5d71.changed.md

@@ -0,0 +1 @@
+Switch the UK default dataset from the Enhanced FRS 2024-25 (policyengine-uk-data 1.58.1) to Microcosm UK 2024-25 (`microcosm_uk_2024_25`, cut `microcosm-uk-2024-25-national-20261019T083323Z-6cba5d71`), certified with policyengine-uk 2.132.6 and policyengine-core 3.33.3, the versions it was built with. Constituency and local-authority regions now load the Microcosm local-area release (`microcosm_uk_2024_25_dense`, cut `microcosm-uk-2024-25-dense-20261019T000432Z-ee5c2dbe`) and are in the default UK region registry. UK results change with the data. The national file has no constituency or local-authority codes, so constituency and local-authority outputs (`compute_uk_geography_disclosure` at those levels, `compute_uk_constituency_impacts`, `compute_uk_local_authority_impacts`) need simulations on `microcosm_uk_2024_25_dense`, and receipts for those runs are uncertified: the certificate covers the national dataset only. The Enhanced FRS 2024-25 datasets stay loadable by name. Data certificates now say how they cover policyengine-core: by default one follows the bundle's core pin (`core_binding: bundle_pin`, as the UK certificate does; the Microcosm releases claim core 3.33.3), and certification and `bundle.py check --include-tros --strict-tros` warn when the pin is outside a release's core claims, as it is for the US release today; `--core-version` binds the core instead (`core_binding: certified`). With a Hugging Face token set, importing policyengine with any policyengine-uk other than 2.132.6 raises, as it did before for versions the Enhanced FRS release did not claim. The certified default is below the Universal Credit caseload floor (0.897 of the floor's 6.7m benchmark, policyengine-uk-data's derivation; Microcosm calibrates to DWP Stat-Xplore's 6.2m), an exception accepted and recorded in the data certification skill. UK candidate promotion now closes the pull requests of pairs from another data line than main's certificate, at the commit their receipt qualified, and recovery no longer re-dispatches attempts for such pairs.

changelog.d/certify-uk-microcosm-uk-2024-25-national-20261019T083323Z-6cba5d71.removed.md

@@ -0,0 +1 @@
+The Enhanced FRS weight matrices (`parliamentary_constituency_weights`, `local_authority_weights`, `local_geography_weights`) no longer resolve by name, now that the UK default is Microcosm UK and its area regions use the Microcosm local-area release.

docs/bundles.md

@@ -127,6 +127,14 @@ python scripts/bundle.py update-packages \
   --uk 2.91.0
 ```
 
+By default a data certificate follows the core pin (`core_binding:
+bundle_pin`), so `--core` can move it; `bundle.py check --include-tros
+--strict-tros` then warns when a data release does not claim the new core. A
+certificate that binds its core (`certify-data --core-version`,
+`core_binding: certified`) instead fails that check until the country is
+re-certified for the new core. See "The core" in the data certification
+skill.
+
 To certify a new data release from a data-producer manifest, run:
 
 ```bash

docs/countries.md

@@ -33,7 +33,7 @@ Override in any output with `income_variable=`.
 | | Dataset |
 |---|---|
 | US | Populace US 2024 (`populace_us_2024.h5`) |
-| UK | Enhanced FRS 2024-25 (`enhanced_frs_2024_25.h5`, policyengine-uk-data) |
+| UK | Microcosm UK 2024-25 (`microcosm_uk_2024_25.h5`); constituency and local-authority regions use its local-area release (`microcosm_uk_2024_25_dense.h5`) |
 
 ## State / regional breakdown
 

docs/disclosure.md

@@ -22,6 +22,10 @@ impact.findings  # what blocks publication, if anything
 impact.public  # the rows that may be published, or None while blocked
 ```
 
+For constituency and local-authority levels, `baseline` and `reform` must run
+on a dataset whose households carry those codes: the local-area dataset
+`microcosm_uk_2024_25_dense`, not the Microcosm national default.
+
 The UK datasets derive from surveys under the UK Data Service End User
 Licence. The defaults below follow its disclosure rules. You can raise them
 for a study with stricter conditions; you cannot lower them.

docs/engineering/runbooks/uk-qualification.md

@@ -118,3 +118,23 @@ write here, such as renaming another branch into main to make a run against
 another base look like the candidate's: main is not protected, so they
 could push it directly. Treat any such change as an incident, not as
 something the gate will catch.
+
+## A pair from another data line
+
+Promotion compares only pairs on main's data line (its UK certificate's data
+package and repository). While Microcosm UK is the default, the
+policyengine-uk-data candidates the workflow resolves are `other_line`. An
+unforced run of one stops there and writes nothing to the ledger. Recovery and
+check refreshes skip such pairs, and promotion closes their open pull requests;
+the ledger keeps their evidence.
+
+`force` still qualifies such a pair, for evidence only. Its candidate commit
+drops main's area routing (`--drop-regional-release`), because the routing
+belongs to the line the pair would replace, and leaves the test constants that
+pin main's certificate as they are, so the candidate's own tests fail. Its pull
+request says it is never promoted, and promotion closes it once its receipt is
+published, at the commit that receipt qualified. The catch-up does not recover
+such a run: re-run it by hand with `force`. A run whose attempt dies before it
+publishes a receipt leaves its pull request open; close that one by hand. Changing the data line
+is a reviewed certification pull request (see
+`docs/engineering/skills/data-certification.md`), not a promotion.

docs/engineering/skills/data-certification.md

@@ -80,9 +80,30 @@ Microcosm national release with one of:
   `dataset_overlays.uk` entry.
 
 Either way, certification checks that the regional release serves the pinned
-model, and the pinned core when `--core-version` is given, by its own build or
-compatibility claims (a `--qualified-pair-id` names the primary release only),
-and HEAD-checks every artifact it carries.
+model by its own build or compatibility claims (a `--qualified-pair-id` names
+the primary release only), checks its core claims as for the primary release
+(below), and HEAD-checks every artifact it carries.
+
+### The core
+
+The bundle has one core pin, `packages.policyengine-core`, shared by every
+country. A certificate says how it covers it:
+
+- **`core_binding: bundle_pin` (the default, no `--core-version`).** The
+  certificate names no core and certifies whatever core the bundle pins, so
+  the pin can move with another country without re-certifying this one.
+  Certification checks the pin against each release's build-time core and
+  core claims, and warns, without refusing, when the pin is outside them.
+  `bundle.py check --include-tros --strict-tros` warns the same way when the
+  pin later moves outside the claims. Receipts then certify the pin without
+  the release's claim; the warning, the certificate and its TRO say so.
+- **`core_binding: certified` (`--core-version`).** The certificate names the
+  core (`certified_for_core_version`). Certification refuses a core that a
+  release neither was built with nor claims, and the strict check refuses a
+  pin that differs from the named core, so moving the core needs a
+  re-certification first. A qualified exact pair always binds its core.
+
+Use the default unless a release must not run on any other core.
 
 ### Re-certifying a routed bundle
 
@@ -92,6 +113,19 @@ a run without `--regional-manifest-uri` is refused; pass the current regional
 release again (or its replacement), or `--drop-regional-release` to remove the
 routing on purpose. The two flags cannot be combined.
 
+An overlay may never shadow a certified dataset name, so certification removes
+any `dataset_overlays.<country>` entry whose name the new release certifies. A
+rollback to policyengine-uk-data with `--drop-regional-release`, for example,
+removes the enhanced FRS overlays. `bundle.py check` refuses a bundle with an
+overlay that shadows a certified name.
+
+Removed overlays do not come back by themselves. Certification warns about
+every dataset name that stops resolving. After a certification that stops
+certifying a name that must stay loadable (rolling forward after a rollback,
+or a ship-together run after a national-first one), re-add its overlay from
+the previous bundle, as #685 did for `enhanced_frs_2024_25`, `frs_2024_25` and
+their `_tiny` files.
+
 ### US
 
 For US Populace certification, certify the Populace release manifest directly:
@@ -182,9 +216,34 @@ python scripts/check_uk_uc_floor.py --dataset <candidate .h5 or bundle dataset n
 
 The benchmarks and their sources are in
 `src/policyengine/data/qualification/uk_uc_floor.json`. The receipt holds
-weighted national totals only. Before quoting any other figure from UK data
-in a PR, apply the UKDS output rules: no unit-record values, no cells under
-10 unweighted cases, no differencing.
+weighted national totals only.
+
+Before quoting any other figure from UK data in a PR, apply the UKDS output
+rules: no unit-record values, no cells under 10 unweighted cases, no
+differencing.
+
+**Recorded exceptions (policyengine.py#685, October 2026).**
+
+- **UC floor.** The Microcosm UK national cut
+  `microcosm-uk-2024-25-national-20261019T083323Z-6cba5d71` became the default
+  with benefit units at 0.897 of the floor's 6.7m benchmark; spend passed at
+  0.959. The floor's benchmarks are not the figures Microcosm calibrates to
+  (DWP Stat-Xplore's 6.2m households on Universal Credit in payment, and its own
+  OBR spend line). Their definitions are to be settled when the UK data checks
+  move to the data pipelines. Until then, a later Microcosm cut with the same
+  gap needs the same explicit acceptance in its pull request, recorded here.
+- **Comparison gates.** The metrics review and the scorecard replay measure a
+  candidate against the previous certified pair, and a change of data line
+  moves the totals by design, so they were not run for that switch; its pull
+  request compares the two lines' weighted totals instead. The other gates,
+  reform responsiveness included, were run on the national cut.
+- **Budget-critical inputs: needs review.** `incapacity_benefit_reported` is
+  missing from the Microcosm file but was a column of the enhanced FRS. The
+  enhanced FRS column carries no values (Incapacity Benefit is closed), so
+  both datasets give zero. Accepted by María Juaristi in #685.
+- **Reform responsiveness: needs review.** The metrics tool's small-cell rule
+  withholds a standard reform's figures, so the gate cannot recompute that
+  reform's verdict. Accepted by María Juaristi in #685.
 
 ## Release runbooks
 

docs/households.md

@@ -106,7 +106,7 @@ separate benefit inputs; they do not override SPM measurement composition.
 
 These changes require the coordinated canonical country/calculator bundle, which
 the packaged production manifest now pins and certifies: `policyengine-us`
-2.2.1, `policyengine-core` 3.33.0, `spm-calculator` 1.0.0, and the US data
+2.2.1, `policyengine-core` 3.33.3, `spm-calculator` 1.0.0, and the US data
 release `populace-us-2024-spm-20260915`. Local-wheel development manifest
 fixtures remain explicitly uncertified. The measurement receipt identifies a
 calculation; it does not certify a population dataset or establish publication

docs/microsim.md

@@ -136,9 +136,11 @@ dataset = datasets["populace_us_2024_2026"]
 
 The default US dataset is **Populace US 2024** — a Populace-built dataset
 calibrated to IRS, CMS, SNAP, Census, and other administrative totals. The
-current UK certified default is **Enhanced FRS 2024–25**, supplied by
-`policyengine-uk-data`. **Populace UK 2023** remains available as a named,
-non-default bundle dataset.
+current UK certified default is **Microcosm UK 2024–25**, a Microcosm build
+calibrated to HMRC, DWP, OBR and other administrative totals; constituency and
+local-authority regions use its local-area release. **Enhanced FRS 2024–25**
+(`enhanced_frs_2024_25`, policyengine-uk-data 1.58.1) and **Populace UK 2023**
+remain available as named, non-default bundle datasets.
 
 PolicyEngine.py obtains the repository type, immutable revision, and SHA-256
 from the installed release bundle. An existing file in the configured data
@@ -199,9 +201,10 @@ ca = Simulation(
 ### UK private data and raw h5 access
 
 UK population data uses licensed Family Resources Survey inputs. The default
-UK release bundle points to the private
-`policyengine/policyengine-uk-data-private` Hugging Face repository. Set
-`HUGGING_FACE_TOKEN` to a token from a Hugging Face account with access:
+UK release bundle points to the private `policyengine/populace-uk-private`
+Hugging Face repository, and the Enhanced FRS datasets to
+`policyengine/policyengine-uk-data-private`. Set `HUGGING_FACE_TOKEN` to a
+token from a Hugging Face account with access:
 
 ```bash
 export HUGGING_FACE_TOKEN=hf_...
@@ -216,11 +219,11 @@ import policyengine as pe
 from policyengine.core import Simulation
 
 datasets = pe.uk.ensure_datasets(
-    datasets=["enhanced_frs_2024_25"],
+    datasets=["microcosm_uk_2024_25"],
     years=[2026],
     data_folder="./data",
 )
-dataset = datasets["enhanced_frs_2024_25_2026"]
+dataset = datasets["microcosm_uk_2024_25_2026"]
 
 simulation = Simulation(
     dataset=dataset,
@@ -237,7 +240,7 @@ from policyengine.provenance import materialize_dataset
 
 result = materialize_dataset(
     "uk",
-    "enhanced_frs_2024_25",
+    "microcosm_uk_2024_25",
 )
 
 print(result.path)
@@ -261,7 +264,7 @@ projected year's tables to policyengine-uk as observed data would make the
 State Pension follow CPI instead.
 
 Each projected year file therefore keeps its data year, `dataset.data_year`
-(2024 for Enhanced FRS 2024–25), and that year's tables,
+(2024 for Microcosm UK 2024–25), and that year's tables,
 `dataset.data_year_data`. `Simulation.run()` projects the data year's tables
 forward as policyengine-uk does, and uses the file's own tables for the
 simulated year. A run of a year file then gives the same result, record by

docs/outputs.md

@@ -273,6 +273,11 @@ control: each number passes a floor on the survey households behind it and
 carries a 90% interval. Use it for anything you publish. See
 [Disclosure control](disclosure.md).
 
+Constituency and local-authority levels need simulations on a dataset whose
+households carry those codes. The Microcosm national default does not: run
+them on the local-area dataset the bundle routes those regions to,
+`microcosm_uk_2024_25_dense`. Regions can come from either.
+
 ```python
 from policyengine.outputs import compute_uk_geography_disclosure