Serve UK constituency and local-authority results from the routed local-area dataset

changelog_entry.yaml

@@ -1,15 +1,22 @@
 - bump: patch
   changes:
     added:
+      - Added breakdown "local_areas" to the simulation request for UK requests whose region is absent, uk or one of country/england, country/scotland, country/wales and country/northern_ireland, refused with 422 for other regions (including country/ni), other countries, include_cliffs=true, segmented=true or a region_group (include_cliffs=false stays accepted); the job runs one baseline and reform on the local-area dataset alone, with the region's filter, and returns a LocalAreaBreakdownOutput (kind local_areas, model_version, policyengine_version, dataset, data_version, and constituency_impact and local_authority_impact as lists of their own record types), which the job status result now accepts alongside SingleYearMacroOutput
+      - Declared local_area_breakdown version 1 in the routing bundle manifest of apps whose policyengine.py bundle routes both UK area region types to one local-area release dataset (regional_release_kind local_area; an area_coded_national bundle, or one routing the two types to different datasets, does not count as routed), and made the gateway refuse breakdown with 400, before spawning, for any route whose manifest does not declare it, including legacy routes and routes pinned to an older version, since older apps ignore the field and would return a national result
+      - Made a routed local-area dataset fail as it loads, before any simulation, when it lacks any of the four household columns the area outputs read (constituency_code_oa, la_code_oa, region_code_oa and source_household_id), for breakdown, constituency and local-authority runs alike, in the runtime and in the Stage 12 worker; Stage 12 copies the same columns
+      - Kept breakdown requests out of the Stage 12 shadow comparison, which skips them as unsupported_options, and added beta integration checks of a UK local-area breakdown and of a UK national result without area data
       - Added four registered dependency rules, checked by the locked-installs quality guard with proof fixtures; every project commits a uv.lock that PR CI checks with uv lock --check, every install in workflows, scripts, Makefiles, Dockerfiles, Compose files and Modal images installs the locked versions, every test command runs through uv run --frozen, and the daily update-dependencies workflow relocks every project; every uv sync and uv run now passes --frozen, and the five sites that cannot install from a lock are registered exceptions
+      - Added optional disclosure fields to UK constituency and local-authority records (disclosure_tier, reported or approximate, and the 90% interval bounds of the average and relative income change); records stored before this change still validate
       - Added the rule sim.boundaries.sqlalchemy-and-alembic-only, with proof from the psycopg adapter of 08ebe1ca; code connects to databases only through SQLAlchemy, never by importing a database driver, and runtime code never changes the schema, which changes only through Alembic migrations; a guard parses every file under projects/*/src, libs/*/src and .github/scripts and rejects driver imports, alembic imports outside a migrations directory, create_all and drop_all, create and drop on tables, SQLAlchemy DDL statements and DDL text in string literals, which extends the Stage 12 DDL check's pattern to the gateway and every library
       - Made the budget's reform-minus-baseline totals log a warning when the baseline and reform household frames carry different weights, since the change is then weighted by the reform's weights alone
-      - Added weighted-microdata rules to the registry, with proof from the #767 household count; the executor's code never aggregates a weight column as a value, reads raw values from weighted frames only in simulation_output_common.py (the UK local-authority breakdown's plain household frame is its one recorded exception there, in a list that may only shrink), and its output-builder tests build entity frames through one shared factory with non-uniform weights, while a doc-only rule records that weights, values, labels and counts each allow different operations
+      - Added weighted-microdata rules to the registry, with proof from the #767 household count; the executor's code never aggregates a weight column as a value, reads raw values from weighted frames only in simulation_output_common.py, where any other read needs a recorded exception in a list that may only shrink and is now empty, and its output-builder tests build entity frames through one shared factory with non-uniform weights, while a doc-only rule records that weights, values, labels and counts each allow different operations
       - Made CLAUDE.md and the Copilot instructions thin pointers to AGENTS.md, required workflow logic longer than one line to live in tested and shellchecked scripts, and registered the tracking-issue rule
       - Added a rules registry (rules/registry.toml) with a standard-library tool that validates it, checks every rule is enforced where it claims, and renders docs/generated/rules.md and the Rules block in AGENTS.md
       - Ran Stage 12 US national simulations as 20 region-group calls each, and built an independent, immutable Stage 12 cache of three annual US datasets and 60 current-law baselines before each deployment, publishing it only after a recorded determinism check and recording its release in the v2 version manifest under a new key, while the existing key keeps a copy that earlier readers accept
       - Made the budget refuse a household count outside its country's plausible range (above 200 million in the US or 60 million in the UK, negative or not finite), raising ImplausibleEntityCountError instead of publishing a count like #767's
     changed:
+      - Removed the UK constituency and local-authority weight-matrix region fallback (_build_uk_weight_replacement_region, _require_certified_uk_weight_matrix and its Stage 12 call); each deployment rebuilds only the app for the current policyengine.py version, now 6.10.0, which routes those regions to the local-area dataset, and apps deployed for older versions are not rebuilt and keep their own code; on a bundle that does not route them, an area region is now an unsupported region
+      - Made UK-wide and nation-level runs on a policyengine.py bundle that routes UK area regions to a local-area release dataset (6.10.0) load only the national dataset and return null for constituency_impact and local_authority_impact, whose results come from a breakdown=local_areas run instead; on a bundle that does not route them, those runs still compute both from the national dataset, and Stage 12 leaves the same breakdowns out of the same reports
       - Enabled every ruff rule (select ALL), ignoring only COM812 and ISC001, which conflict with ruff format, and the D203 and D213 halves of the mutually exclusive docstring pairs; complex functions, long parameter lists and magic numbers were split into smaller helpers and grouped parameters, each behind new characterization tests, across the contract and persistence validation, entry settings, app factory and Stage 12 adapter, gateway route resolution, Stage 12 coordination, workers, parity and qualification, the precompute and national reduce, the routing publisher and its command line, the release sentinel, the package update helper and the shared rules tool; Modal function, FastAPI route, Pydantic validator and command-line signatures and every OpenAPI document are unchanged; the contract library's batch-state builder and SPM selection resolver take their route values as one grouped argument, the simulation observability initializer drops a service_version parameter no caller passed, and no PLR0913 suppression is needed
       - Enabled ruff's argument, idiom and pytest-style rules (ARG, ASYNC, B, C4, FBT, PD, PERF, PGH, PLW, PT, PTH, RUF, SIM); unused arguments are removed, underscored or explained, FastAPI auth dependencies use Annotated, re-raised errors keep their cause, and boolean options of in-repo helpers are keyword-only; OpenAPI documents are unchanged, and the shared rules tool's unittest tests are exempt from pytest-style rules; tests take the unused-argument rules (ARG001, ARG002, ARG005) as file-level exceptions, since their fakes and stubs mirror real signatures; best-effort telemetry and cleanup calls that swallow any error use contextlib.suppress, with the reason as a comment
       - Enabled ruff's structure and import rules (INP, PLC, SLF, N, A, FAST, E); long lines are wrapped, deferred imports that Modal images, test seams, import cycles or heavy dependencies need carry a reason, script directories are declared to ruff as namespace packages, four internal exceptions gained an Error suffix, and published field names, OpenAPI documents and logged error types are unchanged; tests are also exempt from SLF001, INP001 and PLC0415; heavy or optional packages that load only inside functions are listed in ruff's banned-module-level-imports, which keeps them out of module-level imports (TID253) and lets their deferred imports go without a suppression, and the Modal utility programs take INP001 file-wide
@@ -27,6 +34,7 @@
       - Ran the full PR workflow and the Modal image smoke only on the lowest unmerged pull request in a stack, behind a new CI complete gate job that fails on the others
     fixed:
       - Stopped trusting the compromised eMudhra (emSign) intermediate CA certificates; every lockfile now pins certifi 2026.10.19, which drops emSign ECC Root CA - G3, every Modal image upgrades Debian's ca-certificates package right after its base layer, and the Modal images and the Cloud Run entrypoint image set GRPC_DEFAULT_SSL_ROOTS_FILE_PATH so grpcio verifies servers against the operating-system CA bundle instead of the CA list compiled into it
+      - Built UK constituency and local-authority results with policyengine's disclosure-controlled compute_uk_geography_disclosure instead of the deprecated, unprotected functions, in one release nested in regions and the nation; an area that rests on fewer than 10 survey households, is dominated by one, would be recovered by subtraction, or has a non-positive baseline income is left out of the list, so small areas no longer show exploding relative changes such as +1372% for a £192 loss
       - Preserved bounded reform stop dates in UK and US simulations while retaining current-law baseline artifact reuse
       - Resolved PolicyEngine-managed regional datasets using bundled data versions instead of release-manifest commit hashes when loading from GCS
       - Added a Modal app-release bundle backfill for missing data artifact revisions
@@ -35,7 +43,8 @@
       - Added a dedicated read-only Hugging Face credential for private UK runtime datasets, validated it before synchronization, and tested its worker configuration
       - Accepted a UK dataset without area codes only when the policyengine.py bundle routes constituency and local-authority regions to another dataset, and kept requiring area codes on every constituency and local-authority run
       - Raised a clear error, never an omission, when a UK constituency or local-authority breakdown is built over a dataset without area codes
-      - Bound UK constituency and local-authority weight matrices to the certified data release of the selected dataset instead of the unversioned bucket copy, and stopped runs before they start when the matrix lacks the run year or household dimension
+      - Resolved UK constituency and local-authority regions by name or by GSS code in any case once policyengine.py routes them, as row filters from its region registry, so policyengine-api's constituency/{name} and local_authority/{name} requests keep working without weight matrices, and the 17 districts replaced in April 2023 are unsupported regions
+      - Made the executor and Stage 12 image smokes check the routed dataset's area codes, and that its household table has all four columns the area outputs read (reading only the column names), downloading it with the Hugging Face credential, when the installed UK default carries none, and stopped Stage 12 copying area codes for a routed UK national or nation-level report, whose aggregate leaves the two area breakdowns out
       - Released each Stage 12 database session after its operation instead of holding pooled sessions through calculations, retried opening each Stage 12 coordinator and worker database connection that the pooler refused without ever retrying a statement or commit, and reported Stage 12 failures to telemetry before recording them in the database so a failed write cannot hide the original error
       - Accepted a null SPM provenance in Stage 12 aggregate artifacts, so UK and other reports without SPM calculations are saved, while every supplied provenance record is still validated
       - Recorded a redacted, bounded Stage 12 failure cause naming the exception, failing stage, simulation role and message instead of only the exception class, and kept a failed worker's recorded cause on its simulation and report

docs/engineering/skills/microdata-weights.md

@@ -82,7 +82,15 @@ Each finding needs a decision, recorded in the PR:
 Recorded exceptions are listed in
 `rules/baselines/sim-microdata-weight-stripping.txt`. That list may only
 shrink, so adding one is a deliberate change to the rule's baseline. Today it
-holds one: `_unweighted_household_frame`, the plain household frame the UK
-local-authority breakdown hands policyengine's longwise geography impact. It
-goes once that breakdown hands policyengine the simulations, as the
-constituency and US congressional district breakdowns already do.
+is empty: the last one, `_unweighted_household_frame`, went when the UK
+constituency and local-authority breakdowns moved to policyengine's
+disclosure-controlled geography output, which takes the simulations.
+
+The UK constituency and local-authority breakdowns are built only from the
+pair they are given: a constituency or local-authority run's own pair, or,
+for a UK-wide or nation-level region, the pair of a separate local-area
+breakdown run (`LocalAreaBreakdownBuilder`) on the local-area dataset.
+That pair's frames carry the local-area dataset's own weights, so the
+breakdowns aggregate them like any other run's. A UK-wide or nation-level run
+on a bundle that routes area regions builds no breakdown and never mixes the
+national file's weights with the local-area file's.

docs/generated/rules.md

@@ -66,9 +66,9 @@ Every rule that governs `policyengine-sim-api` is listed here with how it is enf
 ### Ratcheted (existing violations still tolerated)
 
 - `sim.ci.multiline-run-is-script`: 13 remaining in `rules/baselines/ci-multiline-run.txt`
-- `sim.microdata.weight-stripping-confined`: 1 remaining in `rules/baselines/sim-microdata-weight-stripping.txt`
+- `sim.microdata.weight-stripping-confined`: 0 remaining in `rules/baselines/sim-microdata-weight-stripping.txt`
 - `sim.quality.pyright-apis-integ`: 4 remaining in `rules/baselines/pyright-apis-integ.txt`
-- `sim.quality.pyright-executor`: 77 remaining in `rules/baselines/pyright-executor.txt`
+- `sim.quality.pyright-executor`: 68 remaining in `rules/baselines/pyright-executor.txt`
 - `sim.quality.pyright-gateway`: 0 remaining in `rules/baselines/pyright-gateway.txt`
 
 ### Exceptions and review dates

docs/migration/stage-12-modal-worker-foundation.md

@@ -212,10 +212,23 @@ aggregate calculations. Extra columns are permitted; missing planned columns
 fail the temporary Stage 12 report without affecting the production result.
 
 The plan distinguishes calculated variables from columns copied directly from
-the source dataset. UK geographic reports require the household columns
-`constituency_code_oa` and `la_code_oa`. Workers do not request those columns as
-calculated PolicyEngine variables, but they must be present in each materialized
-artifact; otherwise validation fails before geographic aggregation begins.
+the source dataset. UK geographic reports require the household columns that
+the constituency and local-authority outputs read (`UK_AREA_DATASET_COLUMNS`):
+`constituency_code_oa`, `la_code_oa`, `region_code_oa` and
+`source_household_id`. Workers do not request those columns as calculated
+PolicyEngine variables, but they must be present in each materialized artifact;
+otherwise validation fails before geographic aggregation begins.
+
+The exception is a UK national or nation-level report whose bundle routes
+constituency and local-authority regions to another dataset. Its national
+dataset may carry no area codes, so the plan copies none, and its aggregate
+leaves the constituency and local-authority breakdowns out. The production
+national run leaves them out too: both decide with
+`uk_area_breakdowns_omitted`, so their results still compare. Production
+computes those breakdowns in a separate `breakdown: "local_areas"` request on
+the routed dataset. The entry adapts no such request into a Stage 12 report
+(skip reason `unsupported_options`), and Stage 12 has no local-area report
+yet, so the shadow comparison does not cover the breakdown numbers.
 
 Aggregation uses the retained output tables without rerunning either model.
 The precomputed baseline and reform simulations retain their original policy

libs/policyengine-simulation-contract/src/policyengine_simulation_contract/gateway_models.py

@@ -19,7 +19,13 @@
     JobStage,
 )
 from policyengine_simulation_contract.json_types import JsonObject
-from policyengine_simulation_contract.macro_output import SingleYearMacroOutput
+from policyengine_simulation_contract.local_area_breakdown import (
+    local_area_breakdown_error,
+)
+from policyengine_simulation_contract.macro_output import (
+    LocalAreaBreakdownOutput,
+    SingleYearMacroOutput,
+)
 from policyengine_simulation_contract.spm import (
     SPMCapability,
     SPMComparisonProvenance,
@@ -153,9 +159,17 @@ def enforce_max_payload_size(cls, value: object) -> object:
 
 
 class SimulationRequest(GatewayRequestBase):
-    """Request model for simulation submission."""
+    """Request model for simulation submission.
+
+    ``breakdown: "local_areas"`` asks a UK-wide or nation run for its
+    constituency and local-authority impacts alone, computed on the
+    local-area dataset; the job's result is then a
+    ``LocalAreaBreakdownOutput``. Without it, the job runs the usual
+    comparison.
+    """
 
     region: str | None = None
+    breakdown: Literal["local_areas"] | None = None
 
     @model_validator(mode="after")
     def region_xor_region_group(self) -> "SimulationRequest":
@@ -165,6 +179,22 @@ def region_xor_region_group(self) -> "SimulationRequest":
             raise ValueError(msg)
         return self
 
+    @model_validator(mode="after")
+    def local_area_breakdown_scope(self) -> "SimulationRequest":
+        """Reject a breakdown outside a UK-wide or nation run of its own."""
+        if self.breakdown is None:
+            return self
+        error = local_area_breakdown_error(
+            country=self.country,
+            region=self.region,
+            region_group=self.region_group,
+            include_cliffs=self.include_cliffs,
+            segmented=self.segmented,
+        )
+        if error is not None:
+            raise ValueError(error)
+        return self
+
 
 class PolicyEngineBundle(BaseModel):
     """Resolved runtime provenance returned by the gateway."""
@@ -212,7 +242,10 @@ class JobStatusResponse(BaseModel):
     """Response model for job status polling."""
 
     status: Literal["running", "complete", "failed"]
-    result: SingleYearMacroOutput | None = None
+    # A breakdown request's result is a LocalAreaBreakdownOutput; every other
+    # request's is a SingleYearMacroOutput. Both forbid extra fields, so a
+    # payload validates as one of them only.
+    result: SingleYearMacroOutput | LocalAreaBreakdownOutput | None = None
     errors: list[SPMErrorDetail] | None = None
     error: str | None = None
     resolved_app_name: str | None = None