[juaristi22]

Fixes #880

Part of #728. This makes the executor ready for policyengine.py 6.10.0 (PolicyEngine/policyengine.py#685), which makes the Microcosm national dataset the UK default and routes constituency and local-authority regions to the Microcosm local-area dataset. Every change below applies only when uk_area_regions_routed() is true. It is false on the pinned 6.7.5, so this can merge before the bump and changes nothing until then.

Same-Repository Draft PR Check

• This PR is a draft.
• This PR is from a branch in PolicyEngine/policyengine-sim-api, not a personal fork.

Summary

On 6.10.0 the national file has no area codes. Without this PR, the bump fails every UK national and nation-level comparison, the image smoke and the deploy sentinel.

1. National and nation-level breakdowns (simulation_runtime.py, simulation_output_builder.py).
   • When: the UK dataset a run loads has no area codes. Detection returns None only then, and only when routed.
   • What: the run builds a second baseline and reform on the routed dataset, with the same policies and the same nation filter. constituency_impact and local_authority_impact come from that pair; every other figure still comes from the national pair.
   • Checks: the routed dataset must carry constituency_code_oa and la_code_oa, and its authority codes must match one packaged boundary version (detect_uk_area_dataset_metadata). The builder computes the second pair the first time a breakdown needs it.
   • Which dataset: routed_uk_area_dataset_name() finds it the way policyengine.py's region_dataset_name does, and both area types must route to the same one.
2. Image smokes (smoke_app.py, v2_static_runtime_files_smoke.py).
   • When: the installed default has no la_code_oa and the bundle routes area regions.
   • What: the smokes download the routed dataset next to the default and read its codes instead. materialize_dataset checks the download against the bundle's SHA-256.
   • Secrets and memory: both smoke functions now mount hf_secret, the secret the image build already uses for the private UK data. The Stage 12 smoke asks for 2 GiB instead of 512 MiB, because reading the routed file's codes peaked at about 0.5 GB locally.
3. Stage 12 (output_planning.py, aggregation.py). A routed national or nation-level report plans no area columns, and its aggregate leaves the two breakdowns out (omit_uk_area_breakdowns). Production is unaffected.
4. Names and codes (_registry_uk_area_region_code).
   • Matching: once routed, constituency/… and local_authority/… match the region registry by GSS code in any case, or by name ignoring case.
   • Source: policyengine.py builds the registry from its packaged constituencies_2024.csv and local_authorities_2023.csv, and each region's label is the name in that list.
   • Rejections: no weight-matrix region is built once routed. An area outside the registry, such as one of the 17 LAD22-only districts, gets Unsupported UK region.
   • policyengine-api names: all 650 constituency names and all 360 local-authority names in its CSVs match policyengine.py's lists. Those lists also have N09000011.

How it combines with the 6.10.0 bump

• Order: merge this first. The bump PR (auto/update-policyengine-6.10.0, pyproject.toml and uv.lock only) then needs no edits. If it opens before this merges, rerun its CI afterwards.
• First use in a container: on the bump, a UK national or nation-level request downloads the local-area file (about 720 MB) from the same private repository as the default, and builds a 2026 year file from it. Constituency and local-authority runs already do this on 6.10.0.
• Smoke and sentinel: the image smoke downloads the same file inside the smoke function. The deploy's sentinel_expected job runs the sentinel's UK national request in-process, so it computes the second pair too (see Cost).
• policyengine-api: its follow-up (catalog default, area regions on the local-area file, N09000011, Microcosm label) still waits until sim-api serves 6.10.0.
• Numbers change: constituency and local-authority results move from enhanced FRS weight matrices to row filters on the Microcosm local-area file.

Testing

• Lint: ruff format --check projects/*/src libs/*/src and ruff check . (0.15.22) pass.
• Pinned 6.7.5: 612 passed. The files run were test_uk_local_authority_metadata, test_simulation_output_builder, test_stage12_output_planning, test_stage12_runtime, test_smoke_app, test_stage12_modal_app, test_region_group_resolution, test_uk_weight_matrix_guard, test_simulation_output_geographic, test_stage12_qualification, test_stage12_worker_validation, test_microdata_weight_boundaries, test_runtime_evaluated_annotations, test_job_progress, test_precompute, test_stage12_cache_precompute, test_stage12_coordination, test_release_sentinel, test_segmented_national and test_segmented_national_reduce.
• 6.10.0: the same files ran in a separate environment with policyengine[models]==6.10.0 (policyengine-uk 2.132.6, policyengine-core 3.33.3). 611 passed and 1 failed: test_v2_image_retains_required_runtime_dependencies compares the pyproject pin with the installed bundle, so it passes once the bump moves the pin. Three existing planning tests describe unrouted reports and now pin routing off, so they pass on either release.
• End to end on 6.10.0 with the private data, through run_simulation_impl with the sentinel's uk_basic_rate_21p_2026 request:
  • uk: 650 constituency and 361 local-authority records.
  • country/wales: 32 and 22.
  • constituency/Aldershot and local_authority/maidstone, addressed by name: resolve and run.
  • local_authority/E07000026: Unsupported UK region.
  • The smoke logic on the real files: it picks the routed file and reads LAD23.
• Not run locally: the full suite, the Docker builds, the integration tests, the Modal image smoke and the deploy sentinel. CI and the deploy run them.

Cost

These are for the sentinel's UK national request on a 10-core Apple laptop with 32 GB. Nothing was measured on Modal.

• Year files already built: 103 s in total. The national analysis took 20 s and loading the local-area file 5 s. The output_uk_constituency span, which now includes computing the second pair, took 72 s.
• Year files rebuilt from the downloaded files: 191 s. Loading took 18 s for the national file and 46 s for the local-area file, and the constituency span took 93 s.
• Download: on a home connection, downloading the local-area file and building its year file took about 130 s, 33 s of it CPU.
• Memory: peak resident memory was about 12 GiB in the runs with year files built (the cold run read lower on macOS). That fits run_simulation's 32 GiB. It also fits the 16 GB ubuntu-latest runner of sentinel_expected, though without much room.

Risks and limits

• Cost per request. The local-area file is about 2.2 times the national file's size. In the warm run, the second pair took about 70 s against 20 s for the national analysis.
• Budget windows. UK budget-window children are ordinary run_simulation calls, so each year also runs the second pair, although the window reads only the budget.
• Warm containers. policyengine.py's in-process simulation cache (LRU, 100 entries) now keeps four simulations per UK national request instead of two.
• Stage 12. Routed national reports leave the two breakdowns out, so the shadow comparison and the manual parity qualification will report them as missing. Stage 12 area reports load the local-area file through the same region resolution.
• Observability. The second pair adds dataset_resolution and dataset_load spans with dataset_role=uk_area, and simulation_build spans with simulation_kind set to uk_area_baseline or uk_area_reform. Its calculation counts towards output_uk_constituency.
• Use disclosure-controlled UK geography outputs for constituency and local authority impacts #860. It also edits simulation_output_builder.py, UK_GEOGRAPHIC_DATASET_VARIABLES and changelog_entry.yaml, so whichever PR merges second needs a small rebase:
  • Use disclosure-controlled UK geography outputs for constituency and local authority impacts #860's _uk_geography_disclosure_output() should take its baseline and reform from _uk_area_breakdown_inputs().
  • Use disclosure-controlled UK geography outputs for constituency and local authority impacts #860 reads region_code_oa and source_household_id. The local-area file carries both; the national file carries only source_household_id.
• UK economy runs fail for every constituency, local authority and Northern Ireland #858. On 6.10.0 this makes its constituency/{name} and local_authority/{name} requests resolve. country/ni, policyengine-api's spelling, stays unsupported, because policyengine.py names that region country/northern_ireland.

Follow-ups (under #728)

• After the bump: delete _build_uk_weight_replacement_region, _require_certified_uk_weight_matrix and tests/test_uk_weight_matrix_guard.py. 6.7.5 needs them until then.
• Worker start-up: check the routed dataset when the worker starts, by installing or pre-downloading it. Today a run checks it when it loads, and the image smoke checks it before merge.
• Provenance: record the local-area dataset and its revision in outputs. This is an OpenAPI change.
• Budget windows: let budget-window children skip the area breakdowns.
• Stage 12: add a second run for routed national reports.
• Northern Ireland: map country/ni to country/northern_ireland.

🤖 Generated with Claude Code