UK #1334: draw UC take-up by award band, and measure the HMRC income rows over each person's year

changelog.d/1334-uc-take-up-by-award.added.md

@@ -0,0 +1 @@
+The UK spine now draws Universal Credit take-up by award band and family type (microcosm#1334). `frs_take_up` and `spi_benefit_coherence` drew `would_claim_uc` at one contract rate spread uniformly over the age population whatever the award, which at design weights put the caseload above DWP's for single people without children, couples and the smallest awards and below it for lone parents, so the national solve met DWP's caseloads by down-weighting low-income families. A new stage, `uc_take_up`, runs after `uc_capital_coherence` and before `uc_legacy_closure_take_up`. One engine materialization at the calibration year gives each benefit unit the award it would be paid if it claimed (after the benefit cap and the deduction attributes `uc_deduction_attributes` assigns), whether the engine moves it off its legacy benefits, and DWP's family type. Within cells of family type by DWP's £100 monthly award band, pooled until the drawn units rest on 30 distinct source households, a reporter claims and the other entitled units of the age population claim at r = (T − F)/D, where T is DWP's calendar-2025 count (the Stat-Xplore facts the registry's payment-band rows average, vendored as `dwp_uc_award_band_facts.json`), F the other entitled units' expected claims (units moved off legacy benefits at their Move to UC claim probability) and D the drawn weight, all over Great Britain; units claim in the order of their identity uniform, so each pool realizes T to within one unit's weight, and Northern Ireland uses the same threshold. Units with no award keep the single-rate draw. A stage gate, `uk_stage_uc_take_up`, replaces `would_claim_uc` in the terminal take-up signal gate, and the roster moves from 47 to 48 stages. María's ruling of 2026-10-21 lifts the U8 freeze for entitled units; the take-up contract records it and keeps 0.55 for units with no award. A unit reporting a legacy benefit that the engine does not move at the legacy closure is not drawn onto UC, since policyengine-uk ends the kept benefit once the unit claims UC; the receipt reports, at the stage's weights over Great Britain, how many working-age units reporting Housing Benefit keep it before and after the redraw, beside DWP's 2025 caseload.

changelog.d/1344-uk-withhold-thin-family-rests.fixed.md

@@ -0,0 +1 @@
+A published UK total less some of its published parts can no longer give back a cell that fewer than ten source households carry (Vahid's second review of microcosm#1344, item 17, and four adversarial reviews of the fix). `uk_withheld_row_sets` only asked whether published rows give back a combination of withheld rows, so the Universal Credit family total less its bound payment bands gave back the unbound band above £2,500 a month, which no row carries. A residual check now runs before the span check and on each of its passes. A total's parts are the published rows inside it, and households whose parts hold the same shares form an atom, which every combination of the total and its parts treats alike. A test comes first: when every atom's share column lies within a tenth of the rounding share of the span of the atoms that rest on ten or more source households (as a unit column, with each row scaled, and summed over its stored entries), no combination of the total and its parts with weights between minus one and one can be thin, and the total is clear (a family total over bands whose every cell rests on ten households or more). Otherwise the check takes the total less any set of members whose shares stay within one (an exact branch-and-bound that tries every set, with a part of constant share also counted at share one; past a work cap, or for a total too large to search, greedy groups instead, and a warning names it), less any one part, and less any two parts, which gives back two overlapping parts' overlap. While one of these rests on one to nine source households, the part with the fewest carriers among those it subtracts is withheld. The note on a row withheld this way says it would give back "a row or a cell". Not checked: a total that is no row (a sum of published regions), a part that reaches outside its total, and, on a total the first test does not clear, combinations at other weights than these.

changelog.d/uk-annual-income-measures.added.md

@@ -0,0 +1,5 @@
+The UK measure resolver now computes each HMRC income measure over a year of income as well as at the current rate: the 12 months before each FRS interview, applied to the engine's amounts at the calibration year, as a proxy for HMRC's tax year. Seven annual measures sit beside their current ones: `employment_income_annual`, `self_employment_income_after_loss_relief_annual`, `private_pension_income_annual`, `state_pension_annual`, `total_income_after_loss_relief_annual`, `income_tax_annual` and `tax_band_annual`. Each is its current measure computed on one more policyengine-uk simulation per resolver, whose inputs are each person's year from the `frs_annual_income_spells` descriptors, so the engine's own rules give the tax on that year's income.
+
+policyengine-uk projects a dataset's inputs to its later years as it loads it, so the year's inputs are set at the measurement year: pay, profit, private pensions and contributory JSA, ESA and Incapacity Benefit at the engine's current amount times the annual share, plus the other spells carried from the dataset's year by the engine's own factor for that input, which must be one factor for every row. The State Pension takes the share of the fiscal year paid to someone who attains State Pension age within it (`months_since_state_pension_age`, measured to 6 October), and the earnings-tested carer benefits keep their baseline values. Pension contributions and salary sacrifice also keep their current values: for a part-year entrant they can be large against the year's pay, which lowers `income_tax_annual` a little. The simulation refuses if it does not read back an input it was given.
+
+A frame without the descriptors refuses every annual measure, with one exception for the rule-1 scorer: the incumbent it scores against, built before the descriptors existed, reads its current values as its annual ones (a share of 1 and no other spells), and the score receipt names that route.

changelog.d/uk-annual-income-national-rows.changed.md

@@ -0,0 +1 @@
+The national HMRC income rows now measure each person's income over a year, as HMRC's statistics count income over the tax year. The year is a proxy for the tax year: the 12 months before each FRS interview (April 2024 to March 2025), applied to the engine's 2025 amounts and fitted to HMRC targets for calendar 2025 (the SPI 2023-24 rows uprated to it, and the Income Tax liabilities rows' calendar-2025 window). The 49 declarations of `hmrc_spi`, `hmrc_spi_region`, `hmrc_itl` and `hmrc_itl_marginal_rate` read the annual measures for their values, their total-income bands and filters, their taxpayer test and their rate band (`employment_income_annual`, `self_employment_income_after_loss_relief_annual`, `private_pension_income_annual`, `state_pension_annual`, `total_income_after_loss_relief_annual`, `income_tax_annual`, `tax_band_annual`), and `obr.income_tax` reads `income_tax_annual`. Dividends, property and savings income have no spells and stay as they are. Target ids, facts, values and metric names do not change: only the person-side measure each row adds up moves, so a recent leaver whose year of income makes them liable to income tax counts as a taxpayer, and a part-year earner sits in the band of what they received. Poverty, benefits and the engine's published tax keep reading current income. Reform costings on files built from this code are lower: in 2025 the engine's income tax on current income falls 1.0% (£324.7bn to £321.5bn against the build before this change, on policyengine-uk 2.147.2; now below OBR's receipts by construction, since the calibration fits OBR's total on the year's income), and static costings of threshold changes fall 1.7% to 1.9% (a £1,000 rise in the personal allowance from £8.98bn to £8.81bn, and 1p on the basic rate from £6.62bn to £6.51bn), closer to HMRC's 2025-26 ready reckoner (January 2025, which includes behaviour: £6.55bn for 1p, and £0.78bn for a £100 rise and £9.65bn for a 10% rise, which put £1,000 near £7.7bn to £7.8bn). The five local `hmrc` rows, the CGT taxable-income bands and the salary-sacrifice reliefs stay on current income for now; the local rows follow once the local line computes its metrics from their declarations.

changelog.d/uk-annual-income-spells.added.md

@@ -0,0 +1,3 @@
+The UK spine now records each adult's year of income beside their current income. HMRC's income statistics count the income each taxpayer received over a tax year (they cover individuals liable to UK income tax), while the spine's income columns are current, annualised incomes (usual weekly pay times 365.25 / 7), so a person who started, left or changed a job during the year sits in another HMRC band than their current pay gives, or in none. The year the descriptors describe is a proxy for the tax year: the 12 months before each interview (April 2024 to March 2025). A new stage, `frs_annual_income_spells`, runs after `frs_hmrc_spine_leaves` and before the SPI support channel. It reads the FRS 2024-25 adult, job and household tabs (pinned by size and sha256, read selectively) and writes six person descriptors: the share of the year paid at today's employee pay and at today's self-employment profit; annual pay from other spells, chiefly a recent leaver's last job at its usual gross pay, scaled to the DV pay concept by the stage's measured ratio and uprated by average earnings, else a donor's rate; ceased self-employment; and the shares of the year a private pension (after retiring in it) and contributory JSA, ESA and Incapacity Benefit (after leaving work) were in payment. No current column changes, so poverty, benefits and the engine's tax are unchanged; the descriptors are inputs for the calibration's annual HMRC measures, which follow separately.
+
+On the SPI channel, `hmrc_spi_income_spine` holds a drawn row's descriptors to its drawn incomes. The tape's income is the tax year's, so each share is 1, and the twin's other-spell amounts stay only where the tape drew none of that income; an undrawn copy keeps its twin's descriptors. A new release-blocking stage gate, `uk_stage_frs_annual_income_spells`, checks the descriptors' invariants, the measured usual-to-DV pay ratio (0.95 to 1.10), annual over current employee earners (1.0 to 1.15) and, where ten or more households carry it, that at least 0.7 of the pay outside the current job is observed rather than drawn from a donor. The receipt follows UKDS disclosure control: a support band per spell source, and weighted figures only over ten or more households. The descriptors join the export surface's reviewed extra columns, the SPI income family requires the stage before it, and the roster moves from 48 to 49 stages.

docs/uk-full-build-graph.md

@@ -66,11 +66,11 @@ For smaller outputs, add `--dataset-households 150000` to use the common informe
 
 ## Graph owners and shared contracts
 
-The spine graph is the 47-stage source roster declared by the UK spec, built by `uk_runtime.spine_build` (the spine tool moved into the package; `tools/build_uk_frs_spine.py` is a six-line shim over it). The superseded `frs_hmrc_retained_leaves` and `hmrc_spi_income` stages and the `UK_SPINE_EXCLUSIONS` list that hid them are removed; the active HMRC path is `frs_hmrc_spine_leaves`, `spi_support_channel`, `spi_income_band_donors` and `hmrc_spi_income_spine`, and the FRS HMRC leaf columns come from `uk_runtime.frs_hmrc_source`. The spine's assembled and transferred gate batteries are graph nodes (`uk_runtime.graph_evidence`); stage evidence and fit-weight records are read back from the content store through the shared `stage_evidence` artifact rather than from in-memory collectors; the HMRC replay sidecar is rebuilt from the SPI stage's checkpoint metadata; the spine sidecar records the operation inventory and the graph manifest, and the graph manifest is saved as `spine.graph.json` under the checkpoint root.
+The spine graph is the 48-stage source roster declared by the UK spec, built by `uk_runtime.spine_build` (the spine tool moved into the package; `tools/build_uk_frs_spine.py` is a six-line shim over it). The superseded `frs_hmrc_retained_leaves` and `hmrc_spi_income` stages and the `UK_SPINE_EXCLUSIONS` list that hid them are removed; the active HMRC path is `frs_hmrc_spine_leaves`, `spi_support_channel`, `spi_income_band_donors` and `hmrc_spi_income_spine`, and the FRS HMRC leaf columns come from `uk_runtime.frs_hmrc_source`. The spine's assembled and transferred gate batteries are graph nodes (`uk_runtime.graph_evidence`); stage evidence and fit-weight records are read back from the content store through the shared `stage_evidence` artifact rather than from in-memory collectors; the HMRC replay sidecar is rebuilt from the SPI stage's checkpoint metadata; the spine sidecar records the operation inventory and the graph manifest, and the graph manifest is saved as `spine.graph.json` under the checkpoint root.
 
 | Operation | Graph owner |
 | --- | --- |
-| Raw FRS and donor preparation, enrichment, support channels | The 47 UK spine stage nodes and their declared composite operations |
+| Raw FRS and donor preparation, enrichment, support channels | The 48 UK spine stage nodes and their declared composite operations |
 | Assembled and transferred spine gates | `spine.gates.assembled`, `spine.gates.transferred` |
 | Bound checkpoint admission | `uk.full.spine_checkpoint`, when resuming a saved spine |
 | Pool sample and mass normalization | `uk.full.sample`, `uk.full.normalize` |

packages/microcosm-build/src/microcosm/build/source_manifest.py

@@ -51,6 +51,7 @@
         "assign_bus_pass_eligibility",
         "assign_clipped_normal",
         "assign_component_take_up_residual",
+        "assign_award_band_take_up_residual",
         "assign_family_claims_by_child_age",
         "assign_opt_out_by_charge_income",
         "assign_period_constant",
@@ -81,6 +82,7 @@
         "declare_income_reference_offset",
         "derive",
         "derive_adult_care_inputs",
+        "derive_annual_income_spells",
         "derive_childcare_inputs",
         "derive_child_support_inputs",
         "derive_disability_benefits",

packages/microcosm-build/src/microcosm/build/spec_engine/schema/sources.schema.json

@@ -254,6 +254,84 @@
   },
   "source_parameter_record": {
    "oneOf": [
+    {
+     "type": "object",
+     "additionalProperties": false,
+     "required": [
+      "kind",
+      "outputs",
+      "window_months",
+      "leaving_date_share",
+      "unknown_leaving_months",
+      "donor_recent_start_months",
+      "minimum_donors",
+      "donor_age_edges",
+      "seed",
+      "salt",
+      "earnings_index",
+      "earnings_index_from",
+      "earnings_index_to",
+      "maximum_uprating_months"
+     ],
+     "properties": {
+      "kind": {
+       "const": "derive_annual_income_spells"
+      },
+      "outputs": {
+       "type": "array",
+       "items": {
+        "type": "string"
+       },
+       "minItems": 1
+      },
+      "window_months": {
+       "type": "integer",
+       "minimum": 1
+      },
+      "leaving_date_share": {
+       "type": "number",
+       "minimum": 0,
+       "maximum": 1
+      },
+      "unknown_leaving_months": {
+       "type": "number",
+       "minimum": 0
+      },
+      "donor_recent_start_months": {
+       "type": "integer",
+       "minimum": 1
+      },
+      "minimum_donors": {
+       "type": "integer",
+       "minimum": 1
+      },
+      "donor_age_edges": {
+       "type": "array",
+       "items": {
+        "type": "integer"
+       }
+      },
+      "seed": {
+       "type": "integer"
+      },
+      "salt": {
+       "type": "string"
+      },
+      "earnings_index": {
+       "type": "string"
+      },
+      "earnings_index_from": {
+       "type": "string"
+      },
+      "earnings_index_to": {
+       "type": "string"
+      },
+      "maximum_uprating_months": {
+       "type": "integer",
+       "minimum": 0
+      }
+     }
+    },
     {
      "type": "object",
      "additionalProperties": false,
@@ -6496,6 +6574,90 @@
       }
      }
     },
+    {
+     "type": "object",
+     "additionalProperties": false,
+     "required": [
+      "kind",
+      "output",
+      "draw",
+      "seed",
+      "anchor",
+      "population",
+      "award",
+      "family_type",
+      "award_bands",
+      "targets",
+      "pooling",
+      "minimum_pool_sources",
+      "drawn_units",
+      "undrawn_claims",
+      "weight_mapping",
+      "residual_rate",
+      "solve_scope",
+      "not_drawn"
+     ],
+     "properties": {
+      "kind": {
+       "const": "assign_award_band_take_up_residual"
+      },
+      "output": {
+       "type": "string"
+      },
+      "draw": {
+       "type": "string"
+      },
+      "seed": {
+       "type": "integer"
+      },
+      "anchor": {
+       "type": "string"
+      },
+      "population": {
+       "type": "string"
+      },
+      "award": {
+       "type": "string"
+      },
+      "family_type": {
+       "type": "string"
+      },
+      "award_bands": {
+       "type": "string"
+      },
+      "targets": {
+       "type": "string"
+      },
+      "pooling": {
+       "type": "string"
+      },
+      "minimum_pool_sources": {
+       "type": "integer",
+       "minimum": 1
+      },
+      "drawn_units": {
+       "type": "string"
+      },
+      "undrawn_claims": {
+       "type": "string"
+      },
+      "weight_mapping": {
+       "type": "string"
+      },
+      "residual_rate": {
+       "type": "string"
+      },
+      "solve_scope": {
+       "type": "string",
+       "enum": [
+        "great_britain"
+       ]
+      },
+      "not_drawn": {
+       "type": "string"
+      }
+     }
+    },
     {
      "type": "object",
      "additionalProperties": false,

packages/microcosm-build/src/microcosm/build/target_materialization.py

@@ -982,7 +982,9 @@ def _refuse_ignored_measurement_period(
     read at one of those two and is refused. So is the frame's period on a
     value the frame does not hold at that period: the engine computes it at
     the calibration period, so the row would be measured there under the
-    declared period.
+    declared period. Only the value terms are checked: a filter, condition or
+    band on an engine-computed variable is still read at the calibration
+    period under a declared frame period.
     """
 
     declared = binding.get("measurement_period")

packages/microcosm-build/src/microcosm/build/uk/country_package.json

@@ -107,6 +107,11 @@
       "kind": "legacy_json",
       "schema_id": "legacy_json"
     },
+    {
+      "path": "dwp_uc_award_band_facts.json",
+      "kind": "legacy_json",
+      "schema_id": "legacy_json"
+    },
     {
       "path": "obr_hvcts_costing_facts.json",
       "kind": "legacy_json",