From 04b4561e0ac216fc2717765faee71debb40ab962 Mon Sep 17 00:00:00 2001 From: Max Ghenis Date: Fri, 23 Oct 2026 13:19:19 -0400 Subject: [PATCH] Correct 19 October planning and validation docs pages Consolidate the 19 corrected source PR net diffs against main, preserving 249 corrected claims and excluding #1724. Replace their separate release fragments with one changed fragment listing the pages. Record the pending #2573 input implementations on the child-maintenance and bereavement pages without treating unmerged engine code as available. Local checks skipped: uk-hub is held back by the route-a-release-rerun-d1811 host reservation. All 19 source PR required checks passed before batching. Co-authored-by: Vahid Ahmadi <57112303+vahid-ahmadi@users.noreply.github.com> --- ...planning-validation-corrections.changed.md | 10 ++ .../capital-gains-distribution-plan.md | 119 ++++++++----- docs/book/assumptions/mp30k-uk-switch-plan.md | 103 ++++++++--- .../non-uk-benefit-receipt-plan.md | 109 +++++++----- docs/book/assumptions/takeup-analysis-plan.md | 122 ++++++++----- .../assumptions/uk-pipeline-alignment-plan.md | 145 +++++++++++----- docs/book/policy/future-date-announcements.md | 67 +++++--- docs/book/policy/indirect-tax-effects-plan.md | 75 +++++--- .../programs/gov/cms/child-maintenance.md | 54 ++++-- .../gov/dfe/regional-childcare-plan.md | 157 ++++++++++------- .../gov/dwp/bereavement-support-plan.md | 129 +++++++++----- .../gov/dwp/cold-weather-payment-plan.md | 83 +++++---- docs/book/programs/gov/dwp/pip-points-plan.md | 82 +++++---- docs/book/programs/gov/hmrc/firm-vat.md | 43 +++-- .../programs/gov/hmrc/pension-tax-relief.md | 161 +++++++++++++----- .../gov/hmrc/pensions-comparison-stbm.md | 47 +++-- .../gov/ofgem/warm-home-discount-plan.md | 88 ++++++---- .../policyengine-py-household-impact.md | 132 +++++++------- .../reform-impacts-dashboard-plan.md | 138 ++++++++++----- docs/book/validation/ubuntu-vs-macos.md | 104 +++++++---- 20 files changed, 1276 insertions(+), 692 deletions(-) create mode 100644 changelog.d/october-planning-validation-corrections.changed.md diff --git a/changelog.d/october-planning-validation-corrections.changed.md b/changelog.d/october-planning-validation-corrections.changed.md new file mode 100644 index 0000000000..a3bbb046db --- /dev/null +++ b/changelog.d/october-planning-validation-corrections.changed.md @@ -0,0 +1,10 @@ +- Corrected 19 October planning and validation documentation pages: firm-level + VAT, child maintenance, the default dataset switch, indirect tax effects, + non-UK benefit receipt, pension tax relief, the STBM pension comparison, + UK pipeline alignment, PIP points, regional childcare programmes, + future-dated policy announcements, effective take-up analysis, + policyengine.py household-impact tests, Ubuntu versus macOS reproducibility, + capital gains distribution, the reform impacts dashboard, Warm Home Discount, + Cold Weather Payment and bereavement support. Updated legal rules, model and + data-pipeline descriptions, the Microcosm UK default, and the causes of the + household-impact and cross-platform failures. diff --git a/docs/book/assumptions/capital-gains-distribution-plan.md b/docs/book/assumptions/capital-gains-distribution-plan.md index 5e47427100..fd9d8cc558 100644 --- a/docs/book/assumptions/capital-gains-distribution-plan.md +++ b/docs/book/assumptions/capital-gains-distribution-plan.md @@ -6,20 +6,29 @@ (model gains jointly across income groups) and [#817](https://github.com/PolicyEngine/policyengine-uk/issues/817) (don't impute gains to households with no wealth). The imputation lives -in the data build, not in this repo. Most of what this page proposed -in May 2026 has since been built in -[Microcosm UK](https://github.com/PolicyEngine/microcosm), which builds -the replacement for the Enhanced FRS. The certified UK default dataset is -still the Enhanced FRS from +in the data build, not in this repo. Most of what this page set out to +fix in May 2026 has since been addressed in +[Microcosm UK](https://github.com/PolicyEngine/microcosm), by a different +method from the one it proposed. Microcosm UK 2024-25 replaced the +Enhanced FRS from [policyengine-uk-data](https://github.com/PolicyEngine/policyengine-uk-data) -(release 1.58.0). What remains is listed at the end of this page. +as the certified UK default dataset on 21 October 2026 +([policyengine.py#685](https://github.com/PolicyEngine/policyengine.py/pull/685), +released in policyengine.py 6.10.0); the Enhanced FRS can still be loaded +by name. What remains is listed at the end of this page. ``` ## How the model uses capital gains -`capital_gains_before_response` is a person-level input, uprated by OBR -per-capita GDP growth. `capital_gains_tax` charges it at the main CGT -rates, with three components charged at their own rates (all added by +`capital_gains_before_response` is a person-level input, uprated through +`gov.economic_assumptions.indices.capital_gains`, which follows OBR +per-capita GDP growth unless a simulation supplies a +`CapitalGainsProjection` +([#2274](https://github.com/PolicyEngine/policyengine-uk/pull/2274); see +the [capital gains projection guide](https://github.com/PolicyEngine/policyengine-uk/blob/main/docs/capital-gains-projection.md)). +`capital_gains_tax` charges `capital_gains` (these gains plus any +behavioural response) at the main CGT rates, with three components +charged at their own rates (all added by [#1861](https://github.com/PolicyEngine/policyengine-uk/pull/1861)): - `capital_gains_badr`: gains qualifying for Business Asset Disposal @@ -36,32 +45,39 @@ Behavioural responses: `capital_gains_behavioural_response` changes realisations using an elasticity with respect to the retention rate (`relative_capital_gains_retention_rate_change`) or the marginal tax rate. Gains use `gov.simulation.capital_gains_responses.elasticity`, -which is 0 by default, so there is no response unless a user sets it. +which is 1.0 by default (since +[#2419](https://github.com/PolicyEngine/policyengine-uk/pull/2419), +October 2026), so a reform that changes a person's marginal rate on gains +changes their realisations unless a user sets it to 0; the +[capital gains realisation response](capital-gains-responses.md) page +explains it. When `gov.simulation.capital_gains_responses.separate_badr_elasticity` is true (false by default), gains qualifying for BADR respond at `badr_elasticity` instead (1.4, the value the OBR used for these gains -in its January 2025 costing), through `capital_gains_badr_elasticity`. +in its costing of the October 2024 rate rises, published in January +2025), through `capital_gains_badr_elasticity`. ## How the gains are imputed ### Enhanced FRS: one distribution per income band -The Enhanced FRS (policyengine-uk-data, the certified default) fits a -spline to the Advani and Summers gains percentiles within each income -band and draws each household's gains from its band. The spline alone -cannot produce gains above about £2m, so policyengine-uk-data main (not -yet in a release) also stacks 30 donor households per HMRC -size-of-gain band and calibrates to HMRC counts and amounts of gains by -band. #818 pointed out the problems with the -band-by-band fit: - -- each band's upper tail was fitted on few observations, so the tails - were noisy and the joint distribution of income and gains had breaks - at band boundaries; -- two people in the same band drew from the same distribution however +The Enhanced FRS (policyengine-uk-data, the certified default until +21 October 2026) fits a spline to the Advani and Summers gains +percentiles within each income band and draws each household's gains +from its band. The spline alone cannot produce gains above about £2m, so +policyengine-uk-data also stacks 30 donor households for each HMRC +size-of-gain band from £500,000 up (from £12,300 up before release +1.59.0) and calibrates to HMRC counts and amounts of gains by band. #818 +asks for gains to be modelled across all income groups together rather +than each group independently, to avoid overfitting. Fitting band by band +means that: + +- each band is fitted on its own, so the joint distribution of income + and gains has breaks at band boundaries; +- two people in the same band draw from the same distribution however different their incomes; -- gains depended on income only, not on wealth, so they could be - imputed to households with no assets (#817). +- gains depend on income only, not on wealth, so they can be imputed to + households with no assets (#817). ### Microcosm UK @@ -70,35 +86,47 @@ Microcosm builds gains in six stages (see the UK 1. **Support split** (`cgt_support_split`). Within each income band of HMRC Capital Gains Tax statistics Table 3, the households with the - most investable wealth (financial wealth, business wealth and - property other than the main home) are split into lighter copies, so + most investable wealth (gross financial wealth, corporate wealth such + as shares and investment funds, and property other than the main + home) are split into lighter copies, so that there are enough rows to hold the largest gains. -2. **Who has gains** (`cgt_incidence_clone`). The set and order of - gainers come from the incidence distribution in Advani and Summers - (2020), *Capital Gains and UK Inequality* (CAGE Working Paper 465). +2. **Who might have gains** (`cgt_incidence_clone`). Every household is + cloned at half its weight, and the clone's oldest adult draws a prior + gain from the Advani and Summers (2020) gains percentiles for their + income band (*Capital Gains and UK Inequality*, CAGE Working Paper + 465). 3. **How much** (`hmrc_cgt_gains_spine`). Amounts are redrawn so that gains match HMRC Table 3 for 2024-25, which counts taxpayers and gains **jointly by size of gain and taxable income**, conditioned on - Tables 1, 2.1a, 5 and 6. -4. **Weight anchor** (`cgt_incidence_anchor`). Clones below the annual - exempt amount carry more weighted people than HMRC publishes, so the - excess weight is moved back to each clone's paired original. + Tables 1, 2.1a, 5 and 6. Within each income band, age group and + region group, the largest gains go to the gainers ranked highest on + household investable wealth (weighted three quarters) and the prior + gain (one quarter). +4. **Weight anchor** (`cgt_incidence_anchor`). After the redraw, clones + with gains at or below the annual exempt amount, or with losses, + carry far more weight than the Advani and Summers reporter + composition implies for the number of liable gainers (HMRC publishes + no count of them), so the excess weight is moved back to each clone's + paired original. 5. **Residential property** (`cgt_residential_split`). Gains on UK residential property are fitted to Table 8 (2024-25 residential property disposals) by splitting households into weighted copies, which fills `capital_gains_residential_property`. -6. **What kind** (`hmrc_cgt_asset_type_spine`). Of the remaining gains, - those qualifying for BADR or Investors' Relief are fitted to Table - 4.1, which fills `capital_gains_badr`, and the rest are given asset - types from Table 7 (gains by asset type, 2023-24). +6. **What kind** (`hmrc_cgt_asset_type_spine`). Of the other liable + gainers, those claiming BADR or Investors' Relief are fitted band by + band to Table 4.1, which fills `capital_gains_badr`. Every one of them + is also given a main asset type from Table 7 (gains by asset type, + 2023-24), for diagnostics only. Carried interest (HMRC Table 9) is not + imputed. -All the HMRC tables come from the Capital Gains Tax statistics July -2026 release, through Microcosm's pinned Chronicle feed. +All the HMRC tables come from the 2026 release of the Capital Gains Tax +statistics (published 27 August 2026), through Microcosm's pinned +Chronicle feed. Against #818: gains are now fitted to the published joint distribution of gain size and taxable income, not band by band. Against #817: -the largest gains are placed on the wealthiest households in each -income band. Microcosm's release checks also include a projection check +within each income band, the largest gains go mainly to the wealthiest +households, through the wealth-weighted ranking in stage 3. Microcosm's release checks also include a projection check on people just below the annual exempt amount (`uk_cgt_projection_entrants` in `gates.json`), so that uprating does not push implausible numbers of them into paying CGT in @@ -110,8 +138,9 @@ later years. positive gains sit on households with zero investable wealth. Neither repo asserts this directly yet, so #817 should stay open until it does. -- **Close #818** once a Microcosm UK dataset is the certified default - and its diagnostics confirm the Table 3 fit; link them from the issue. +- **Close #818** once the certified Microcosm UK dataset's diagnostics + confirm the Table 3 fit (Microcosm UK has been the certified default + since 21 October 2026); link them from the issue. - **Elasticities by group.** Responses use one main elasticity and one BADR elasticity. With the joint distribution in place, an elasticity that varies by income or size of gain becomes possible, at the cost diff --git a/docs/book/assumptions/mp30k-uk-switch-plan.md b/docs/book/assumptions/mp30k-uk-switch-plan.md index 39f9d64841..374e9209f3 100644 --- a/docs/book/assumptions/mp30k-uk-switch-plan.md +++ b/docs/book/assumptions/mp30k-uk-switch-plan.md @@ -11,25 +11,34 @@ same switch and rollback path to Microcosm UK. ## Current state -PolicyEngine UK has no built-in default dataset. `Simulation()` uses, in order: +PolicyEngine UK has no built-in default dataset. An inline `situation`, if +passed, is used on its own. Otherwise `Simulation()` uses, in order: 1. an explicit `dataset=` argument; 2. the `POLICYENGINE_UK_DEFAULT_DATASET` environment variable (`DEFAULT_DATASET_ENV_VAR` in `tax_benefit_system.py`); -3. otherwise, unless an inline `situation` is passed, it raises a `ValueError` - (`get_default_dataset_url` in `simulation.py`). +3. otherwise it raises a `ValueError` (`get_default_dataset_url` in + `simulation.py`). -The test suite sets the variable to the private -`hf://policyengine/policyengine-uk-data-private/enhanced_frs_2023_24.h5@1.40.3` -when a Hugging Face token is available (`policyengine_uk/tests/conftest.py`). -So the eFRS is still the dataset the model is tested against, and the -default in downstream tools such as policyengine.py. +When a Hugging Face token is available and the variable is not already set, +the test suite sets it to the certified dataset pinned by commit and SHA-256 in +`policyengine_uk/tests/certified_dataset.yaml` (read by +`policyengine_uk/tests/conftest.py`): `enhanced_frs_2024_25.h5` from +policyengine-uk-data 1.58.1, in the private +`policyengine/policyengine-uk-data-private` repository. So the eFRS is still +the dataset this repository's tests run on. Direct microsimulation through `Microsimulation()` in this package is deprecated in favour of [policyengine.py](https://github.com/PolicyEngine/policyengine.py), whose managed bundle pins certified datasets (see the repository README). The default most users meet is therefore the one that bundle pins, so a switch has -to be made there as well as in this repository's tests. +to be made there as well as in this repository's tests. policyengine.py made +that switch in release 6.10.0 (21 October 2026): its UK default is now +Microcosm UK 2024-25 (`microcosm_uk_2024_25`), and the Enhanced FRS 2024-25 +stays loadable by name. It did so under its own bundle certification (its +6.10.0 changelog records an accepted exception to its Universal Credit +caseload floor); this page does not record whether the gates below were run. +This repository's tests remain on the eFRS. ## What the switch needs @@ -41,14 +50,31 @@ to be made there as well as in this repository's tests. because `_pre_encode_enum_columns` (in `simulation.py`) walks each year's tables. - The dataset's `years` drive uprating. Microcosm's base year is 2024-25, - one year later than the eFRS file used in tests, so uprating paths from the - earlier base year need checking. -- Enum columns must use the labels in this repo's variable definitions. - Unknown labels do not encode correctly. + the same as the pinned eFRS file the tests use (`enhanced_frs_2024_25.h5`); + `test_latest_data_smoke.py` still loads the unpinned + `enhanced_frs_2023_24.h5`. +- String enum columns must use the member names in this repo's variable + definitions (for example `OWNED_OUTRIGHT`, not "Owned outright"). An + unknown name raises a `ValueError` when the dataset is encoded + (policyengine-core's `Enum.encode`); integer-coded columns are not + checked. +- This package's `hf://` loader (policyengine-core's + `download_huggingface_dataset`) and the test pin's downloader (`materialise` + in `policyengine_uk/utils/aggregate_impact/dataset.py`) both request Hugging + Face model repositories (`repo_type="model"`), and the pin file has no + repository-type field. Microcosm UK files sit in a dataset repository + (`policyengine/populace-uk-private`), so pinning one for the tests + (criterion 4) needs both to accept dataset repositories first, and loading + one by `hf://` needs the loader to. ### Variables -- `policyengine_uk/variables/input/` is the data contract. A switch should +- `policyengine_uk/variables/input/` holds many of the inputs, but the + loader (`build_from_multi_year_dataset` in `simulation.py`) sets every + dataset column named after any model variable as an input and skips the + rest, and many dataset inputs are defined elsewhere (for example + `tenure_type`, `household_weight`, `would_claim_uc` and the `*_reported` + benefit amounts). A switch should report the input columns present in one dataset but not the other, and decide for each whether a missing column is acceptable. - Take-up and other reported-versus-simulated inputs have to come from the @@ -73,28 +99,53 @@ A documented gate must pass before any environment's default changes: 3. **Reforms:** at least three standard reforms (for example +1p on the basic rate, +£5 a week on the UC standard allowance, abolishing the two-child limit) give the same sign and are within 5% of each other. -4. **Tests:** the suite passes with the new default, including - `test_latest_data_smoke.py`, `test_deterministic_variables.py` and - `test_no_economic_assumptions.py`. +4. **Tests:** the suite passes with the new dataset pinned in + `policyengine_uk/tests/certified_dataset.yaml`, with the reform + regression fixtures in `policyengine_uk/tests/microsimulation/reforms_config.yaml` + moved to the same dataset. `test_latest_data_smoke.py` loads a + hard-coded, unpinned `enhanced_frs_2023_24.h5`; + `test_deterministic_variables.py` uses inline situations and + `test_no_economic_assumptions.py` an inline situation and a hand-built + one-person dataset, so none of those three tests the new default as + written. 5. **Documentation:** the `assumptions/` section records the switch and any remaining aggregate gaps. ## Rollback path -1. Dataset choice stays a runtime setting: no code in this repo depends on - which dataset is loaded. +1. Dataset choice stays a runtime setting: no code in this repo branches on + which dataset is loaded. One parameter is tied to the dataset, though: + `gov.simulation.vat.survey_consumption_coverage` is calibrated to one + Microcosm UK 2024-25 national cut only + (`microcosm-uk-2024-25-national-20261002T230158Z-5c6b3f68`; policyengine.py + 6.10.0 pins the later cut + `microcosm-uk-2024-25-national-20261019T083323Z-6cba5d71`), puts 2024-25 + VAT 38% (policyengine-uk-data 1.56.16) or 44% (1.58.0) above receipts on + the Enhanced FRS 2024-25, and has to be re-derived when the dataset it + should match changes (`docs/engineering/vat-coverage.md`). 2. While both are in use, CI runs the dataset-dependent tests on both. -3. The default is set per environment through - `POLICYENGINE_UK_DEFAULT_DATASET`, with the eFRS file kept available. -4. Rolling back is changing that variable and redeploying, with no code - change here. This assumes the eFRS file stays published while the switch - beds in. +3. The default is set per environment: for direct use of this package + through `POLICYENGINE_UK_DEFAULT_DATASET`; for this repository's tests + through the pin in `policyengine_uk/tests/certified_dataset.yaml` (with + the reform regression fixtures recording the same dataset); and in + policyengine.py through its bundle manifest's `default_dataset`, which + changes with a policyengine.py release. The eFRS file is kept available. +4. Rolling back is changing that setting and redeploying (in policyengine.py, + a new bundle release). In this repository it also means re-pinning the + test dataset and regression fixtures if the tests have switched, and + re-deriving dataset-calibrated parameters such as the VAT coverage above. + This assumes the eFRS file stays published while the switch beds in. ## Acceptance PolicyEngine UK switches its default only after the gates above pass, with a -rollback that needs no code change in this repo. Until then either dataset can -be used by passing `dataset=` or setting the environment variable. +rollback that needs no change in this repo beyond re-pinning the test +dataset and re-deriving dataset-calibrated parameters. Until then either +dataset can be used by passing `dataset=` or setting the environment +variable. Microcosm UK files sit in a Hugging Face dataset repository, and +this package's `hf://` loader requests model repositories (policyengine-core's +`download_huggingface_dataset` passes `repo_type="model"`), so download a +Microcosm file and pass its local path. ## References diff --git a/docs/book/assumptions/non-uk-benefit-receipt-plan.md b/docs/book/assumptions/non-uk-benefit-receipt-plan.md index 928177417f..148ff5a56a 100644 --- a/docs/book/assumptions/non-uk-benefit-receipt-plan.md +++ b/docs/book/assumptions/non-uk-benefit-receipt-plan.md @@ -4,14 +4,19 @@ **Planning page.** The data behind PolicyEngine UK is calibrated so that household benefit totals match published **DWP/HMRC/OBR totals**. Some of that outturn doesn't go to **UK private households**. It -goes to people living abroad (exported pensions), people in +goes to people living abroad (exported pensions) and to people in institutional accommodation outside the FRS sampling frame (care homes, -hostels), and a small administrative-leakage residual. This page +hostels). Fraud and error overpayments are a separate issue: claimants +receive them but aren't entitled to them. This page sets out the proposed treatment, tracked in [#842](https://github.com/PolicyEngine/policyengine-uk/issues/842) (still -open, October 2026). Calibration targets are now set in -[Microcosm UK](https://github.com/PolicyEngine/microcosm), so most of the -work below belongs there. +open, October 2026). Microcosm UK, the dataset in policyengine.py's +certified UK bundle, sets its calibration targets in +[Microcosm](https://github.com/PolicyEngine/microcosm), so most of the work +below belongs there. policyengine-uk-data's Enhanced FRS, which this +repository's microsimulation test jobs and aggregate-impact report still run +on, keeps its own target set, which still includes the OBR State Pension +line. ``` ## Why this matters @@ -24,28 +29,38 @@ on a household the FRS samples. It doesn't: State Pension they built up. DWP's benefit expenditure tables report the amount paid to recipients abroad, and Stat-Xplore gives the caseload by country. None of these people are in the FRS. -- **Disability Living Allowance and PIP paid abroad**: limited cases - under reciprocal agreements (EU and EFTA countries and a handful of - others). -- **Institutional residents**: care-home residents receive most - benefits but the FRS doesn't sample care homes. Their entitlements +- **Disability and carer benefits paid abroad**: limited cases. The DLA care + component, the PIP daily living part, Attendance Allowance and Carer's + Allowance can be paid to people living in an EEA country or Switzerland + who are covered by the Withdrawal Agreement; the mobility parts can't be + paid abroad. DWP's tables report each benefit's spending outside the UK. +- **Institutional residents**: care-home residents keep their State + Pension and can get some other benefits (Pension Credit, or Attendance + Allowance if they pay for their own care), but the FRS doesn't sample + care homes. Their entitlements feed into the DWP outturn but not into PolicyEngine's `state_pension` totals. -- **Administrative leakage**: fraud, error and advance payments that - appear in DWP cashflow but never reach an entitled person. Where a target includes these groups, calibration pushes their spending onto FRS households and **overstates** what UK private households receive. -**Already done for State Pension.** Microcosm no longer binds the OBR State +**Fraud and error** is a different case. DWP counts overpayments as benefit +spending although they go beyond what claimants are entitled to: 3.2% of +benefit spending in its FYE 2026 estimates, against 0.4% underpaid through +official error. That +money does reach households, so it doesn't overstate what they receive, but +it matters where the model simulates entitlement rather than receipt. + +**Already done for State Pension paid abroad.** Microcosm no longer binds the OBR State Pension line. State Pension is now bound to resident figures from DWP Stat-Xplore (Great Britain) and the Northern Ireland Department for Communities, and the OBR line and DWP's "paid outside UK" line are kept as -diagnostics. The plan is -[microcosm#1069](https://github.com/PolicyEngine/microcosm/issues/1069) and -the change landed in +diagnostics. The plan began as +[microcosm#1069](https://github.com/PolicyEngine/microcosm/issues/1069), which +now tracks the open follow-ups, and the change landed in [microcosm#1084](https://github.com/PolicyEngine/microcosm/pull/1084) -(merged 2 October 2026). The same question is open for other programmes. +(merged 2 October 2026). The resident series still includes care-home +residents. The same question is open for other programmes. ## Scope @@ -53,29 +68,42 @@ the change landed in For each benefit programme the calibration targets, identify the share of the published total that goes outside UK private households. DWP and HMRC -publish enough data to assemble a first-pass table: - -| Programme | UK private households | Overseas | Institutional | Admin / error | -|-----------|----------------------|----------|---------------|---------------| -| State Pension | published | DWP overseas tables | -- | -- | -| Pension Credit | published | small | minimal | minor | +publish enough data to assemble a first-pass table. The fraud and error +column is recorded separately: that money reaches households, so it isn't +part of the gap. + +| Programme | Published total | Overseas | Institutional | Fraud and error | +|-----------|-----------------|----------|---------------|-----------------| +| State Pension | published | DWP "paid outside UK" line | care-home residents | F&E published | +| Pension Credit | published | small (Great Britain residence required) | care-home residents (unquantified) | F&E published | | Universal Credit | published | minimal | small | F&E published | -| Housing Benefit | published | -- | care-home component | F&E published | -| Child Benefit | published | EU-treaty residue | -- | -- | -| Disability Living Allowance / PIP | published | reciprocal residue | -- | -- | -| Attendance Allowance | published | -- | care-home residue | -- | -| Winter Fuel Payment | published | overseas eligible cohort | -- | -- | +| Housing Benefit | published | -- | hostels (Housing Benefit doesn't pay for a care-home place) | F&E published | +| Child Benefit | published | EEA, Switzerland and agreement-country residue | -- | F&E published (HMRC annual report and accounts) | +| Disability Living Allowance / PIP | published | DWP "outside UK" lines | care-home residue (self-funders; both mobility parts) | F&E published | +| Attendance Allowance | published | DWP "outside UK" line | care-home residue (self-funders) | F&E published | +| Winter Fuel Payment | published | none since winter 2025-26 (recipients must be ordinarily resident in England, Wales or Northern Ireland) | care-home residents | -- | Sources: DWP *Benefit expenditure and caseload tables* and Stat-Xplore (payments abroad), the DWP *Fraud and Error in the Benefit System* annual -release, and programme-specific breakdowns. +release, HMRC's annual report and accounts (Child Benefit fraud and error), +programme-specific breakdowns, and the rules on care homes +([Housing Benefit Regulations 2006 reg. 9(1)(k) and (4)](https://www.legislation.gov.uk/uksi/2006/213/regulation/9) +and the [pension-age equivalent](https://www.legislation.gov.uk/uksi/2006/214/regulation/9); +[PIP Regulations 2013 reg. 28](https://www.legislation.gov.uk/uksi/2013/377/regulation/28); +[DLA Regulations 1991 reg. 9](https://www.legislation.gov.uk/uksi/1991/2890/regulation/9)) +and on Winter Fuel Payment residence +([SI 2025/969 reg. 2](https://www.legislation.gov.uk/uksi/2025/969/regulation/2) +for England and Wales; +[SR 2025/142 reg. 2](https://www.legislation.gov.uk/nisr/2025/142/regulation/2) +for Northern Ireland). ### Phase 2: subtract from calibration targets Once the gap is quantified per programme, each **calibration target** should be the **UK-private-household part** of the total, not the full -total. Microcosm's State Pension change does this by binding a resident -series instead. Where no resident series exists, the target can carry an +total. Microcosm's State Pension change does this for pensions paid abroad +by binding a resident series instead; that series still includes care-home +residents. Where no resident series exists, the target can carry an explicit private-household share (0–1, default 1) with a cited source for each override. @@ -110,12 +138,12 @@ changes in this repo are small: ## Open questions -- Should the `private_household_share` be year-varying (e.g. overseas - State Pension share rose post-EU exit) or held flat? -- For Universal Credit, where the FRS *does* sample some - institutionalised people (in supported accommodation), what's the - correct private-household-share value: strictly < 1, or 1 minus a - smaller residual? +- Should the `private_household_share` be year-varying (e.g. the overseas + share of State Pension spending was 4.3% in 2019-20 and 3.9% in 2024-25 in + DWP's outturn tables) or held flat? +- For Universal Credit, if the FRS samples some people in supported + accommodation, what's the correct private-household-share value: + strictly < 1, or 1 minus a smaller residual? - For the household-calculator (single-household) interface, the private-household-share correction shouldn't apply, because the user represents their own household. Confirm the correction is only on the @@ -124,7 +152,8 @@ changes in this repo are small: ## References - DWP, [Benefit expenditure and caseload tables](https://www.gov.uk/government/collections/benefit-expenditure-and-caseload-tables): spending by benefit, including payments abroad. -- DWP, [Fraud and Error in the Benefit System](https://www.gov.uk/government/collections/fraud-and-error-in-the-benefit-system): administrative leakage estimates. +- DWP, [Fraud and Error in the Benefit System](https://www.gov.uk/government/collections/fraud-and-error-in-the-benefit-system): estimates of benefit overpaid through fraud and error and underpaid through official error. - DWP, [Stat-Xplore](https://stat-xplore.dwp.gov.uk/): caseloads by benefit, including State Pension recipients by country of residence. -- HMRC, [Tax credits and Child Benefit statistics](https://www.gov.uk/government/collections/personal-tax-credits-statistics): EU-treaty Child Benefit residue. -- Issue: [#842](https://github.com/PolicyEngine/policyengine-uk/issues/842). Related: [#1621](https://github.com/PolicyEngine/policyengine-uk/issues/1621) (UK pipeline alignment) and [microcosm#1069](https://github.com/PolicyEngine/microcosm/issues/1069) (State Pension on resident figures). +- HMRC, [Annual reports and accounts](https://www.gov.uk/government/collections/hmrcs-annual-report-and-accounts): Child Benefit error and fraud estimates. +- GOV.UK, [Child Benefit if you leave the UK](https://www.gov.uk/child-benefit-abroad): when Child Benefit is paid for families abroad (EEA, Switzerland and social security agreement countries). +- Issue: [#842](https://github.com/PolicyEngine/policyengine-uk/issues/842). Related: [#1621](https://github.com/PolicyEngine/policyengine-uk/issues/1621) (UK pipeline alignment) and [microcosm#1069](https://github.com/PolicyEngine/microcosm/issues/1069) (UK pensions follow-ups after #1084). diff --git a/docs/book/assumptions/takeup-analysis-plan.md b/docs/book/assumptions/takeup-analysis-plan.md index 64cd8e5679..519da82353 100644 --- a/docs/book/assumptions/takeup-analysis-plan.md +++ b/docs/book/assumptions/takeup-analysis-plan.md @@ -10,41 +10,62 @@ and per-programme target list. ``` ```{admonition} Status, October 2026 -The analysis below has not been run yet. Take-up flags are drawn in the -data build, not in this repository. The certified UK default dataset is -still the Enhanced FRS built by +The analysis below has not been run yet. Stochastic take-up flags are +drawn in the data build, not in this repository; the Housing Benefit, +Income Support, tax credit and Council Tax Reduction flags are formulas +here (see below). Since policyengine.py 6.10.0 (21 October 2026) the +certified UK default dataset is Microcosm UK (`microcosm_uk_2024_25`), +built by [Microcosm](https://github.com/PolicyEngine/microcosm). The +Enhanced FRS built by [`policyengine-uk-data`](https://github.com/PolicyEngine/policyengine-uk-data) -(release 1.58.0). [Microcosm UK](https://github.com/PolicyEngine/microcosm) -builds the replacement national dataset with the same take-up design. -Both draw Universal Credit, Pension Credit and Child Benefit take-up -anchored on reported receipt, so the analysis should run on whichever -dataset is the certified default at the time. +stays loadable by name at release 1.58.1, and it is the dataset this +repository's credentialed tests pin +(`policyengine_uk/tests/certified_dataset.yaml`). Both draw Universal +Credit, Pension Credit and Child Benefit take-up anchored on reported +receipt, and Microcosm adds some draws the Enhanced FRS lacks (see the +table), so the analysis should run on whichever dataset is the certified +default at the time. ``` ## Why this matters PolicyEngine UK does not compute benefit take-up from scratch. The data -build **seeds** stochastic take-up flags (`would_claim_*`) per programme -from published target rates, then runs them through: +build **seeds** stochastic take-up flags (`would_claim_*`) for most +programmes from target rates (some published take-up estimates, some +fitted to caseload or spending, and Universal Credit's 0.55, which has no +cited source in policyengine-uk-data and is held in Microcosm for parity +with the earlier pipeline), then runs them through: 1. **Reported anchoring.** Universal Credit, Pension Credit and Child Benefit flags are inputs, drawn so that units reporting receipt claim. Housing Benefit, Income Support, the tax credits and Council - Tax Reduction instead derive their flags at runtime from reported - receipt (or `claims_all_entitled_benefits`); see + Tax Reduction instead derive their flags at runtime: on data-built + simulations they equal reported receipt, and hand-entered households + also claim when `claims_all_entitled_benefits` is true; see [#1621](https://github.com/PolicyEngine/policyengine-uk/issues/1621). -2. **Reweighting** to match aggregate caseload and expenditure - targets. -3. **SPI integration** for income-tax-relevant variables (which can +2. **SPI integration** for income-tax-relevant variables (which can shift the income distribution and therefore who falls into each benefit's eligible population). +3. **Reweighting** to match aggregate caseload and expenditure + targets. + +Both builds integrate the SPI before reweighting. Pension Credit take-up +(both builds), and Child Benefit and Carer's Allowance take-up +(Microcosm), are redrawn after the SPI step. By the end of the pipeline the *effective* take-up rate, the share of the eligible population that ends up with `would_claim_X == True`, can -diverge from the seed rate by a few percentage points. The seeded rates -themselves were calibrated against pre-reweighted aggregates, so the -real interest is how the post-pipeline shares stack up against -published outturn share figures. +diverge a long way from the seed rate: the initial analysis linked below +found 74.9% of the benefit units entitled to Universal Credit in 2026 +receiving it, against a 55% seed (Enhanced FRS 2023-24, policyengine-uk +2.53.1). Part of the gap is by construction, because the data builds +apply most seed rates to every benefit unit, not only to eligible ones. +Some seed rates are published take-up estimates (Pension Credit, Child +Benefit, Scottish Child Payment, Marriage Allowance); others were fitted +to caseload or spending targets, some at calibrated weights (Tax-Free +Childcare in policyengine-uk-data, the Universal Credit childcare element +in Microcosm). The real interest is how the post-pipeline shares stack up +against published outturn share figures. ## Where the seed rates live @@ -52,10 +73,13 @@ The take-up rate parameters in this repository (for example `gov.dwp.universal_credit.takeup_rate` and `gov.dwp.housing_benefit.takeup`) are reference values: no formula reads them. The rates the data build draws from are in `policyengine_uk_data/parameters/take_up/` in -policyengine-uk-data and in `build/uk/take_up_contract.json` in -Microcosm. The analysis should read the seed rate from the data build +policyengine-uk-data and in +`packages/microcosm-build/src/microcosm/build/uk/take_up_contract.json` +in Microcosm. The analysis should read the seed rate from the data build that produced the dataset, and check the matching parameter here is in -sync. +sync. Two already differ: Pension Credit is 0.7 here against 0.62 in +policyengine-uk-data from April 2023, and Scottish Child Payment for +children aged 6 and over is 0.86 here against 0.85 in both builds. ## Programmes to cover @@ -64,26 +88,28 @@ Per #1354, with current PolicyEngine UK take-up handling: | Programme | Variable | Take-up flag | Reference parameter here | Handling | |-----------|----------|--------------|--------------------------|----------| | Universal Credit | `universal_credit` | `would_claim_uc` (input) | `gov.dwp.universal_credit.takeup_rate` | Drawn in the data build, anchored on reported receipt | -| UC childcare element | `uc_childcare_element_claimed` | `would_claim_uc_childcare` (input) | – | Drawn in Microcosm at single and couple rates; the Enhanced FRS does not set it, so it defaults to true | +| UC on legacy closure | `claims_uc_at_legacy_closure` | `would_claim_uc_at_legacy_closure` (input) | – | Drawn in Microcosm at DWP's Move to Universal Credit claim rates by legacy benefit combination; the Enhanced FRS does not set it, so it defaults to true | +| UC childcare element | `uc_childcare_element` (claimed via `uc_childcare_element_claimed`) | `would_claim_uc_childcare` (input) | – | Drawn in Microcosm at single and couple rates; the Enhanced FRS does not set it, so it defaults to true | | Pension Credit | `pension_credit` | `would_claim_pc` (input) | `gov.dwp.pension_credit.takeup` | Drawn in the data build, anchored on reported receipt | | Child Benefit | `child_benefit` | `would_claim_child_benefit` (input) | `gov.hmrc.child_benefit.takeup.overall` and `.by_age` | Drawn in the data build, anchored on reported receipt | | Housing Benefit | `housing_benefit` | `would_claim_housing_benefit` (formula) | `gov.dwp.housing_benefit.takeup` | Follows reported receipt (#1621) | -| Income Support | `income_support` | `would_claim_IS` (formula) | `gov.dwp.income_support.takeup` | Follows reported receipt (#1621) | -| Child Tax Credit | `child_tax_credit` | `would_claim_CTC` (formula) | `gov.dwp.tax_credits.child_tax_credit.takeup` | Follows reported receipt; scheme ended 2025-04-06 | -| Working Tax Credit | `working_tax_credit` | `would_claim_WTC` (formula) | `gov.dwp.tax_credits.working_tax_credit.takeup` | Follows reported receipt; scheme ended 2025-04-06 | -| Council Tax Reduction | `council_tax_reduction` | `would_claim_council_tax_reduction` (formula) | – | Local schemes simulated; take-up follows reported receipt (#1669 open) | +| Income Support | `income_support` | `would_claim_IS` (formula) | `gov.dwp.income_support.takeup` | Follows reported receipt (#1621); scheme closed 2026-04-01 | +| Income-based JSA / income-related ESA | `jsa_income`, `esa_income` | none | `gov.dwp.JSA.income.takeup`, `gov.dwp.ESA.income.takeup_estimate_fye_2019` | Paid on reported awards that pass capital and work tests; income-based JSA closed 2026-04-01 | +| Child Tax Credit | `child_tax_credit` | `would_claim_CTC` (formula) | `gov.dwp.tax_credits.child_tax_credit.takeup` | Follows reported receipt; scheme ended on 5 April 2025 | +| Working Tax Credit | `working_tax_credit` | `would_claim_WTC` (formula) | `gov.dwp.tax_credits.working_tax_credit.takeup` | Follows reported receipt; scheme ended on 5 April 2025 | +| Council Tax Reduction | `council_tax_reduction` | `would_claim_council_tax_reduction` (formula) | – | Simulated for England's pension-age scheme, the Scottish and Welsh schemes and five English councils' working-age schemes, reported reductions elsewhere; on data-built simulations take-up follows reported receipt | | Tax-Free Childcare | `tax_free_childcare` | `would_claim_tfc` (input) | – | Drawn in the data build at a flat rate | | Universal childcare entitlement | `universal_childcare_entitlement` | `would_claim_universal_childcare` (input) | – | Drawn in the data build at a flat rate | | Extended childcare entitlement | `extended_childcare_entitlement` | `would_claim_extended_childcare` (input) | – | Drawn in the data build at a flat rate | | Targeted childcare entitlement | `targeted_childcare_entitlement` | `would_claim_targeted_childcare` (input) | – | Drawn in the data build at a flat rate | | Marriage Allowance | `marriage_allowance` | `would_claim_marriage_allowance` (input) | `gov.hmrc.income_tax.allowances.marriage_allowance.takeup_rate` (0.5, the post-2019 steady state) | Drawn in the data build at a flat rate | | Scottish Child Payment | `scottish_child_payment` | `would_claim_scp` (input) | `gov.social_security_scotland.scottish_child_payment.takeup_rate` (under 6, 6 and over) | Drawn in the data build by age | -| Carer's Allowance | `carers_allowance` | `would_claim_carers_allowance` (input) | – | Set from reported receipt in Microcosm; the Enhanced FRS does not set it, so it defaults to true | +| Carer's Allowance | `carers_allowance` | `would_claim_carers_allowance` (input) | – | Set from reported receipt in Microcosm, whose main branch since 20 October 2026 also draws a claim at 0.6 for carers outside Scotland whom only the earnings limit keeps from the allowance; the Enhanced FRS does not set it, so it defaults to true | | 16 to 19 Bursary Fund | `bursary_fund_16_to_19` | `would_claim_bursary_fund_16_to_19` (input) | – | Defaults to true | | Student support grants | `adult_dependants_grant`, `childcare_grant`, `parents_learning_allowance`, `travel_grant`, `disabled_students_allowance` | `would_claim_adult_dependants_grant`, `would_claim_childcare_grant`, `would_claim_parents_learning_allowance`, `would_claim_travel_grant`, `would_claim_disabled_students_allowance` (inputs) | – | Default to true | | Care to Learn | `care_to_learn` | `would_claim_care_to_learn` (input) | – | Defaults to true | -| State Pension | `state_pension` | n/a (effectively universal) | – | See [state-pension.md](../programs/gov/dwp/state-pension.md) | -| PIP / DLA / AA / SDA | various | none | – | Category inputs set from reported receipt; see [disability-legacy-benefits.md](../programs/gov/dwp/disability-legacy-benefits.md) | +| State Pension | `state_pension` | none | – | Paid from reported State Pension (`state_pension_reported`); see [state-pension.md](../programs/gov/dwp/state-pension.md) | +| PIP / DLA / AA / SDA | various | none | – | PIP, DLA and AA category inputs set from reported receipt (PIP categories can also come from points); SDA paid at the maximum rate where `sda_reported` is positive; see [disability-legacy-benefits.md](../programs/gov/dwp/disability-legacy-benefits.md) | ## Methodology @@ -107,21 +133,26 @@ For each programme above, the analysis should produce: 4. **Pipeline drift:** `effective − seed` per programme, with a sign convention so positive means more claimants than seeded. -The published reference rates change roughly annually. The analysis -should be repeatable, ideally as a notebook under +Most published reference rates change roughly annually (HMRC's tax +credit take-up rates stop at 2017 to 2018). The analysis should be +repeatable, ideally as a notebook under [`docs/book/validation/`](../validation/) so future builds can re-run -it and update the gaps table. +it and update the gaps table. A notebook needs its own `file:` entry +under Validation in `myst.yml` to appear in the docs, since only +Markdown pages are picked up by folder. ## Output A new validation notebook `docs/book/validation/take-up-rates.ipynb` should: -- pull each programme's seed take-up parameter, +- pull each programme's seed take-up rate from the data build that + produced the dataset, - compute the effective rate against the built dataset, -- compare to the most recent published share (DWP take-up tables for - income-related benefits, HMRC Tax-Free Childcare and tax credits - statistics for the rest), +- compare to the most recent published share (DWP's income-related + benefits take-up tables; HMRC's Child Benefit, Tax-Free Childcare and + tax relief statistics; Scottish Government take-up rates; DfE + childcare statistics), - flag drifts of more than ±2 percentage points for follow-up. The first run is the deliverable for #1354. After that the notebook @@ -132,20 +163,21 @@ data refresh. - **Reported-anchored take-up** is in both data builds for UC, Pension Credit and Child Benefit. The programmes whose flags still follow - reported receipt at runtime are the most likely source of drift - between seed and effective rates. + reported receipt at runtime have no seed draw: their effective rate is + the share of eligible units that report receipt, which the analysis + should compare with the published rate directly. - The **`would_claim_*` input-only conversion** (also in #1621) would give the analysis a cleaner ladder: with all of `would_claim_*` as - inputs, the effective rate is exactly the share assigned by the - data build's take-up stage, and pipeline drift becomes a pure - reweighting question. + inputs, every flag comes from the data build's take-up stage, and + pipeline drift comes only from the steps after it (SPI income + imputation and reweighting). ## References - Issue: [#1354](https://github.com/PolicyEngine/policyengine-uk/issues/1354). -- Initial analysis (UC and CTC): -- DWP, [Income-related benefits: estimates of take-up](https://www.gov.uk/government/collections/income-related-benefits-estimates-of-take-up): the main reference rates for Housing Benefit, Income Support, Pension Credit and Universal Credit. +- Initial analysis (UC and CTC): (policyengine-uk 2.53.1; it measured take-up by setting `gov.dwp.universal_credit.takeup_rate` to 1, which no longer changes take-up on main). +- DWP, [Income-related benefits: estimates of take-up](https://www.gov.uk/government/collections/income-related-benefits-estimates-of-take-up--2): reference rates for Pension Credit and pensioners' Housing Benefit, and (from the financial year ending 2025 edition) a composite Universal Credit (or equivalent benefits) rate. - HMRC, [Personal tax credits statistics](https://www.gov.uk/government/collections/personal-tax-credits-statistics). -- HMRC, [Child Benefit statistics](https://www.gov.uk/government/collections/child-benefit-statistics) and Marriage Allowance statistics. +- HMRC, [Child Benefit: annual release](https://www.gov.uk/government/collections/child-benefit-geographical-statistics), and Marriage Allowance claimants and cost in HMRC's [tax relief statistics](https://www.gov.uk/government/statistics/tax-reliefs/tax-relief-statistics-january-2026). - Pipeline alignment: [#1621](https://github.com/PolicyEngine/policyengine-uk/issues/1621). - Cross-links: [`state-pension.md`](../programs/gov/dwp/state-pension.md), [`disability-legacy-benefits.md`](../programs/gov/dwp/disability-legacy-benefits.md), [`tax-credits.md`](../programs/gov/dwp/tax-credits.md). diff --git a/docs/book/assumptions/uk-pipeline-alignment-plan.md b/docs/book/assumptions/uk-pipeline-alignment-plan.md index 34def30814..50d2f87982 100644 --- a/docs/book/assumptions/uk-pipeline-alignment-plan.md +++ b/docs/book/assumptions/uk-pipeline-alignment-plan.md @@ -15,10 +15,11 @@ The target data pipeline is now [Microcosm UK](https://github.com/PolicyEngine/microcosm), which replaced the Enhanced FRS build in `policyengine-uk-data`; its UK national line is built on the Family Resources Survey 2024-25 (`microcosm_uk_2024_25`). -Gaps 2 and 3 below, and PR 4 of the sequence, are implemented there; -gap 1 is still open on the model side. The aggregate residuals are the -May 2026 Enhanced FRS measurements and have not been re-measured on the -Microcosm release. +Gaps 2 and 3 below, and PR 4 of the sequence, are implemented there (and +were in policyengine-uk-data in April 2026, gap 3 in narrower form); +gap 1 is still open on the model side. The aggregate residuals other than +State Pension are the figures #1621 reported for 2025 on Enhanced FRS 1.50.3 +in April 2026, and none has been re-measured on the Microcosm release. ``` ## Architectural gaps vs US @@ -39,19 +40,57 @@ UK's `frs_take_up` stage). But several still derive from `_reported` at runtime: - `would_claim_housing_benefit` +- `would_claim_council_tax_reduction` - `would_claim_IS` - `would_claim_WTC` - `would_claim_CTC` -The runtime pattern is `claims_all_entitled_benefits | (foo_reported > 0)`, -which means a reform expanding eligibility only reaches existing -`_reported > 0` claimants. Non-claimants who newly qualify are -invisible unless the user manually sets -`claims_all_entitled_benefits = True` everywhere. +In hand-entered situations each is `claims_all_entitled_benefits | (foo_reported > 0)` +(for Income Support only the claimant's or partner's report counts), and +`claims_all_entitled_benefits` is true for a benefit unit that reports under +£1 across seven means-tested benefits. In data-built simulations +`claims_all_entitled_benefits` is false for every benefit unit, and Housing +Benefit and Council Tax Reduction ignore it, so a reform expanding +eligibility only reaches existing `_reported > 0` claimants. Setting +`claims_all_entitled_benefits` to true on a data-built simulation does not +change this: Housing Benefit and Council Tax Reduction ignore it there, and +the eligibility tests for Child Tax Credit, Working Tax Credit, Income +Support and working-age Housing Benefit (`is_CTC_eligible`, +`is_WTC_eligible`, `income_support_eligible`, `housing_benefit_eligible`) +also require a reported award. In years when tax credits are active it does +turn on `would_claim_CTC` for families that report no Child Tax Credit. +Unlike the family and disabled child elements, `CTC_child_element` does not +check `is_CTC_eligible`, so a family with children that claims Child Tax +Credit without being eligible has `tax_credits` (the child element, less the +income taper) but no `child_tax_credit`. That amount enters its Housing +Benefit and Council Tax Reduction means tests, and the tax credit award that +the 2024 Winter Fuel Payment and Pension Age Winter Heating Payment tests +read. This already happens on `main` in tax-credit years with nothing +overridden. In hand-entered situations the flag is true for every family +that reports none of the seven benefits, so each such family with children +claims Child Tax Credit. In both kinds of simulation, a family that reports +Child Tax Credit but claims Universal Credit still claims it although +ineligible. So does a family whose legacy benefits have closed, but on +`main` that happens only once tax credits are no longer active, so its +`tax_credits` is zero unless a reform leaves tax credits active after a +legacy benefit has closed. Otherwise, the same eligibility +tests mean that a hand-entered family reporting none of the seven benefits +can receive, of these five benefits, only Council Tax Reduction and +pension-age Housing Benefit. **Fix**: convert each to input-only `default_value = True` (matching `would_claim_uc`), and add each to Microcosm UK's reported-anchored -take-up stages. Tax credits stopped paying ongoing awards on 5 April +take-up stages. Because of those eligibility tests, the only new awards +this gives families that report none of these benefits are Council Tax +Reduction and pension-age Housing Benefit (the model's pension-age route, +which includes mixed-age couples that can still claim on it). For years when +tax credits are active, `CTC_child_element` needs an `is_CTC_eligible` gate. +The fix would spread the problem but does not create it: without the gate, +a family with children drawn as a Child Tax Credit claimant without a report +gets the child element, less the income taper, as `tax_credits` but no Child +Tax Credit, and that amount counts as income in its Council Tax Reduction +and any Housing Benefit means test. +Tax credits stopped paying ongoing awards on 5 April 2025, and `gov.dwp.tax_credits.active` is false from 2025-04-06 (see [tax-credits.md](../programs/gov/dwp/tax-credits.md)), so the WTC and CTC fix matters only for simulations of earlier years. @@ -62,9 +101,13 @@ fix matters only for simulations of earlier years. and `child_benefit_take_up` stages keep every benefit unit that reports receipt as a claimant and draw the rest to the target rate ([`source_stages.json` at `75167a6`](https://github.com/PolicyEngine/microcosm/blob/75167a68/packages/microcosm-build/src/microcosm/build/uk/source_stages.json)). -The description below is the Enhanced FRS behaviour this replaced. +The description below is the Enhanced FRS behaviour before +[policyengine-uk-data#359](https://github.com/PolicyEngine/policyengine-uk-data/pull/359) +(April 2026), which anchored `would_claim_uc`, `would_claim_pc` and +`would_claim_child_benefit` on reported receipt, the same three flags +Microcosm UK anchors. -The Enhanced FRS take-up assignment was a pure random draw: +Until then the Enhanced FRS take-up assignment was a pure random draw: ```python pe_benunit["would_claim_uc"] = generator.random(len(pe_benunit)) < universal_credit_rate @@ -74,27 +117,36 @@ This discards information: respondents who reported receiving UC clearly took it up, and should have `would_claim_uc = True` with certainty. -**Fix**: port `policyengine_us_data/utils/takeup.py::assign_takeup_with_reported_anchors` +**Fix** (done in policyengine-uk-data#359): port +`policyengine_us_data/utils/takeup.py::assign_takeup_with_reported_anchors` into `policyengine-uk-data` and apply it to each take-up variable where -a `_reported` column exists. This tightens per-variable calibration +a `_reported` column exists (`would_claim_uc`, `would_claim_pc`, +`would_claim_child_benefit`). This tightens per-variable calibration without changing the target rates. ### 3. Second-stage imputation gap on SPI donors -**Done in Microcosm UK.** `hmrc_spi_income_spine` trains an FRS-only -second stage on the new SPI incomes (its outputs include `gift_aid` and -`charitable_investment_gifts`), `spi_housing_shell` redraws tenure, +**Done in Microcosm UK.** `hmrc_spi_income_spine` draws `gift_aid` and +`charitable_investment_gifts` from the SPI jointly with the new incomes, and +an FRS-only second stage, trained on the FRS rows, predicts pension +contributions, tax-free savings income and the `*_reported` benefits for the +SPI rows from those incomes. `spi_housing_shell` redraws tenure, dwelling and council tax band for the SPI households, and the wealth, spending and VAT stages now run after the SPI block, so they are drawn for the new incomes rather than inherited from the FRS parent. The -description below is the Enhanced FRS behaviour this replaced. +description below is the Enhanced FRS behaviour before April 2026: +policyengine-uk-data#347 then added `gift_aid` and +`charitable_investment_gifts` to the SPI imputation, and #362 added a +second-stage QRF (`datasets/imputations/frs_only.py`) for pension +contributions, tax-free savings income and the `*_reported` benefits; rent +and wealth on SPI rows still come from the FRS parent there. `policyengine_uk_data/datasets/imputations/income.py::impute_income` -trains a QRF that replaces only six income variables on SPI donor +trained a QRF that replaced only six income variables on SPI donor rows; everything else (rent, wealth, pension contributions, gift aid, -benefits-reported flags) stays as whatever middle-income FRS donor +benefits-reported flags) stayed as whatever middle-income FRS donor happened to be sampled. So a synthetic SPI donor ending up with £2M -imputed self-employment income still carries an unchanged middle-FRS- +imputed self-employment income still carried an unchanged middle-FRS- donor's savings balance and zero gift aid. The US equivalent ([`policyengine-us-data#589`](https://github.com/PolicyEngine/policyengine-us-data/pull/589)) @@ -112,7 +164,14 @@ predictors = demographics + 6 income vars, outputs = a wide set of `charitable_investment_gifts`, `other_deductions`, `employee_pension_contributions*`, `employer_pension_contributions`, total wealth components, `capital_allowances`, `deficiency_relief`, -and the full list of `*_reported` benefits. +and the full list of `*_reported` benefits. policyengine-uk-data#362 (April +2026) added this stage with a narrower output list: pension contributions, +tax-free savings income and the `*_reported` benefits. Microcosm UK's second +stage had the same outputs at `75167a6`, less the incapacity benefit and +maternity allowance reports, which its base data lacks; it has since added +trading losses, six owner-manager company variables and the War Widow's +Pension report +([`source_stages.json` at `54b514f`](https://github.com/PolicyEngine/microcosm/blob/54b514f9/packages/microcosm-build/src/microcosm/build/uk/source_stages.json)). ## Specific aggregate residuals @@ -121,7 +180,9 @@ in [PR #1618](https://github.com/PolicyEngine/policyengine-uk/pull/1618), new State Pension pro-rating + Protected Payment in [PR #1634](https://github.com/PolicyEngine/policyengine-uk/pull/1634)), the residual benefit-aggregate gaps against OBR on the Enhanced FRS -(measured May 2026, except the State Pension row, which is taken from +(for the rows other than State Pension, the figures #1621 reported for 2025 +on Enhanced FRS 1.50.3 in April 2026, which predates policyengine-uk-data#347, +#359 and #362; the State Pension row is taken from the State Pension page; none re-measured on Microcosm UK) were: | Variable | Model | Target | Gap | Likely cause | @@ -130,18 +191,20 @@ the State Pension page; none re-measured on Microcosm UK) were: | `esa_contrib` | ~£5.56 bn | ~£2.5 bn | **+£3.1 bn** | Large. Likely reported-based take-up plus no phase-out for migrated cases. | | `attendance_allowance` | ~£8.76 bn | ~£7.0 bn | +£1.8 bn | Cap-rate or take-up issue. | | `tax_free_childcare` | ~£0.41 bn | ~£0.9 bn | **-£0.5 bn** | Understated by about half: under-imputed eligible families. | -| `state_pension` | £119.0 bn | £131.3 bn | -£12.2 bn | 2024-25, against DWP's Spring Forecast 2026 less payments abroad, on policyengine-uk-data-private 1.57.4, as documented in [state-pension.md](../programs/gov/dwp/state-pension.md). The shortfall arises in building the Enhanced FRS ([policyengine-uk-data#493](https://github.com/PolicyEngine/policyengine-uk-data/issues/493)). | +| `state_pension` | £119.0 bn | £131.3 bn | -£12.2 bn | 2024-25, against DWP's Spring Forecast 2026 less payments abroad, on policyengine-uk-data-private 1.57.4, as documented in [state-pension.md](../programs/gov/dwp/state-pension.md). Most of the shortfall arises in building the Enhanced FRS ([policyengine-uk-data#493](https://github.com/PolicyEngine/policyengine-uk-data/issues/493)). | Each warrants a scoped PR pair (model + data) once the architectural fixes above land. -## Items already fixed since #1621 was filed +## Small items from #1621 -The issue body flagged three small bugs; all are now resolved: +The issue body flagged three small items; two are resolved: -- `tax_credits/min_benefit.yaml` unit was `currency-USD`; it is now `currency-GBP`. -- `benunit_weekly_hours` label said "Average" but the formula sums; label now reads "Total weekly hours worked by adults in the benefit unit". -- `state_pension_type` BASIC/NEW classification (the `is_SP_age` heuristic in the issue) was rewritten in PR #1618. +- `tax_credits/min_benefit.yaml` unit was `currency-USD`; it is now `currency-GBP` (#1627). +- `benunit_weekly_hours` label said "Average" but the formula sums; label now reads "Total weekly hours worked by adults in the benefit unit" (#1627). +- Some `~` operations on Python bools raised `DeprecationWarning: Bitwise inversion '~' on bool`, which becomes an error in Python 3.16; no fix is tracked. + +Separately, the `state_pension_type` BASIC/NEW classification was rewritten in PR #1618. This page reflects the *current* gap rather than the issue body, so the next reviewer working through the queue doesn't waste time on @@ -151,19 +214,23 @@ already-fixed items. 1. **Convert remaining `would_claim_*` formulas to input-only** (model side) + reported-anchored assignment in Microcosm UK's `frs_take_up` - stage. Still open: the four formulas above read `_reported` at + stage. Still open: the five formulas above read `_reported` at runtime on `main`. -2. ~~Port `assign_takeup_with_reported_anchors`~~: done in Microcosm UK - (gap 2). +2. ~~Port `assign_takeup_with_reported_anchors`~~: done in + policyengine-uk-data#359 and in Microcosm UK (gap 2). 3. ~~Add second-stage QRF for FRS-only variables on SPI donors~~: done - in Microcosm UK (gap 3). -4. ~~Add `gift_aid` to `IMPUTATIONS`~~: done in Microcosm UK, where - `gift_aid` is a stage-2 output. + in policyengine-uk-data#362 and in Microcosm UK (gap 3), both with a + narrower output list than proposed above; Microcosm UK also redraws + housing and runs the wealth, spending and VAT stages after the SPI + block. +4. ~~Add `gift_aid` to `IMPUTATIONS`~~: done in policyengine-uk-data#347 + and in Microcosm UK, where `gift_aid` is a stage-1 SPI output. 5. **Residual benefit-aggregate follow-ups**: re-measure on the - Microcosm release first, then separate small PRs for - IS phase-out (analogous to the WTC/CTC reactive scheme close-out), - ESA contrib investigation, AA calibration, TFC under-imputation, - the State Pension data-side fix (#1632, policyengine-uk-data#493). + Microcosm release first, then separate small PRs for ESA contrib + investigation, AA calibration, TFC under-imputation, the State Pension + data-side fix (#1632, policyengine-uk-data#493). The IS phase-out + landed in #1631 (`gov.dwp.income_support.active` is false from + 2026-04-01). ## References diff --git a/docs/book/policy/future-date-announcements.md b/docs/book/policy/future-date-announcements.md index 8e91ec8ce7..e12b48e4d0 100644 --- a/docs/book/policy/future-date-announcements.md +++ b/docs/book/policy/future-date-announcements.md @@ -8,17 +8,18 @@ values keyed by effective date. This was originally requested in [#636](https://github.com/PolicyEngine/policyengine-uk/issues/636) in the context of the Truss "growth plan" announcements in autumn 2022. -The feature has been live and well-exercised since the model moved to -date-keyed parameter values; this page documents the current usage -pattern so that future analyses don't need to reinvent it. +Nothing new was needed: parameter values have been keyed by effective +date since the model began in 2020, and the parameter files already carry +many future-dated rows. This page documents the current usage pattern so +that future analyses don't need to reinvent it. ## How it works Every parameter is a `values:` map from effective date to value. When the tax-benefit system is built, `convert_to_fiscal_year_parameters` -(`policyengine_uk/utils/parameters.py`) turns each parameter into one -value per model year, where model year 2026 is the fiscal year 6 April -2026 to 5 April 2027: +(`policyengine_uk/utils/parameters.py`) turns each parameter under `gov` +into one value per model year from 2015 to 2040, where model year 2026 is +the fiscal year 6 April 2026 to 5 April 2027: - by default, the value in force on 30 April of that year; - for parameters with `fiscal_year_blend: true` in their metadata (fuel @@ -26,7 +27,9 @@ value per model year, where model year 2026 is the fiscal year 6 April change part-way through the year counts for the share of the year it applies; - parameters with `preserve_calendar_dates: true` keep their statutory - dates, and their formulas annualise explicitly. + dates, and the code that reads them handles the dates itself (the + alcohol and tobacco duty formulas, for example, split each fiscal year + into dated segments with `policyengine_uk/utils/excise.py`). Example from [`gov/hmrc/income_tax/allowances/personal_allowance/amount.yaml`](https://github.com/PolicyEngine/policyengine-uk/blob/main/policyengine_uk/parameters/gov/hmrc/income_tax/allowances/personal_allowance/amount.yaml): @@ -53,15 +56,16 @@ The 2030-04-06 row records a future-dated decision: the freeze to 2026 s. 10. The file has no uprating. From the tax year after its last row, `policyengine_uk/parameters/gov/hmrc/create_threshold_indexation.py` applies the Income Tax Act 2007 s. 57 CPI indexation year by year. So -model years up to 2030 return £12,570, and indexation starts in 2031. +model years 2021 to 2030 return £12,570, and indexation starts in 2031. ## Patterns for common cases ### Announced but reversed before taking effect Some announcements are walked back before the effective date. The -Truss "growth plan" basic-rate cut to 19% (announced September 2022, -reversed October 2022) is the canonical example. To represent this +basic-rate cut to 19%, which the Truss "growth plan" of 23 September 2022 +brought forward to April 2023 and which was reversed on 17 October 2022, +is the canonical example. To represent this honestly: 1. When the announcement is made, add the announced value at the @@ -78,23 +82,32 @@ honestly: The baseline then follows what was finally enacted. PolicyEngine UK has no "policy as announced on date X" switch, so to score an announcement as -it stood before a reversal, apply it as a reform (see -the [excise duties page](../../engineering/excise-duties.md) for how -dated reforms are annualised). +it stood before a reversal, apply it as a reform (see "Dated changes" in +the [scenarios guide](../usage/scenarios.md) for how dated reforms are +annualised, and the [excise duties page](../../engineering/excise-duties.md) +for duties that keep their calendar dates). ### Phased-in or staggered changes -Phased rate changes are the most common future-dating pattern. Each -phase is a row at its effective date with a separate `reference`. The -`national_insurance/class_1/rates/employee/main.yaml` trajectory is a -good worked example: 13.25% from April 2022 under the Health and Social -Care Levy, back to 12% from November 2022 when the levy was repealed, -then the cuts to 10% in January 2024 and 8% in April 2024. +Phased rate changes are a common future-dating pattern. Each phase is a +row at its effective date with a separate `reference`. The Class 1 +employee main rate (`gov.hmrc.national_insurance.class_1.employee_scale[1].rate`, +in `national_insurance/class_1/employee_scale.yaml`) is a good worked +example: 13.25% from April 2022 under the Health and Social Care Levy Act +2021 (a temporary 1.25 percentage point rise for 2022-23, ahead of a +separate levy due from April 2023), back to 12% for earnings paid from +6 November 2022 under the Health and Social Care Levy (Repeal) Act 2022, +then the cuts to 10% in January 2024 and 8% in April 2024. The rate has +no `fiscal_year_blend`, so each model year takes the 30 April rate: +13.25% for 2022, 12% for 2023 and 8% from 2024. The November 2022 and +January 2024 changes do not show in annual results +([#1807](https://github.com/PolicyEngine/policyengine-uk/issues/1807)). ### Announced abolitions For an abolition announced ahead of its effective date (for example the -2025 Budget removal of the two-child limit, in +removal of the two-child limit, announced at the 2025 Budget and since +enacted by the Universal Credit (Removal of Two Child Limit) Act 2026, in [`gov/dwp/universal_credit/elements/child/limit/child_count.yaml`](https://github.com/PolicyEngine/policyengine-uk/blob/main/policyengine_uk/parameters/gov/dwp/universal_credit/elements/child/limit/child_count.yaml)): - The relevant parameter (`gov.dwp.universal_credit.elements.child.limit.child_count`) @@ -115,17 +128,19 @@ is £650 from 2022, £900 from 2023 and goes back to zero from 1. **Cite the announcement in `metadata.reference`** at every future-dated row. Where possible, cite both the announcement and the legislation that enacts it once it is published. The 2022-04-06 - row of `national_insurance/class_1/rates/employee/main.yaml`, for - example, cites both the Health and Social Care Levy Act 2021 and the - Budget document. + row of the Class 1 employee main rate + (`national_insurance/class_1/employee_scale.yaml`), for example, cites + both the Health and Social Care Levy Act 2021 and the Budget document. 2. **Don't extrapolate**. Each future-dated row should reflect a specific announcement. If a parameter would otherwise move with uprating, let the uprating handle it. Adding a "frozen forever" row is a policy assertion that needs its own reference. 3. **Check how the parameter is annualised.** A change dated part-way through a fiscal year only counts for part of the year if the - parameter has `fiscal_year_blend: true`; otherwise the model year takes - the value in force on 30 April. + parameter has `fiscal_year_blend: true`, or has + `preserve_calendar_dates: true` and a formula that splits the year (as + the alcohol and tobacco duties do); otherwise the model year takes the + value in force on 30 April. ## What this isn't @@ -142,6 +157,6 @@ baseline tree with an effective-date row. ## References - Issue [#636](https://github.com/PolicyEngine/policyengine-uk/issues/636): the original request, from the Truss growth plan period. -- PolicyEngine research: [Tax cuts in Prime Minister Truss's growth plan](https://www.policyengine.org/uk/research/tax-cuts-in-prime-minister-trusss-growth-plan), a worked example of modelling an announcement before its reversal. +- PolicyEngine research: [Tax cuts in Prime Minister Truss's Growth Plan 2022](https://www.policyengine.org/uk/research/tax-cuts-in-prime-minister-trusss-growth-plan), a worked example of modelling an announcement before its reversal. - [PolicyEngine Core](https://github.com/PolicyEngine/policyengine-core): the date-keyed parameter mechanism underneath. - `convert_to_fiscal_year_parameters` in `policyengine_uk/utils/parameters.py`: how dated values become model-year values. diff --git a/docs/book/policy/indirect-tax-effects-plan.md b/docs/book/policy/indirect-tax-effects-plan.md index 7c1fb0b296..a6a9692780 100644 --- a/docs/book/policy/indirect-tax-effects-plan.md +++ b/docs/book/policy/indirect-tax-effects-plan.md @@ -21,27 +21,38 @@ Right now PolicyEngine UK computes: 1. **Direct effect**: the full structural change to income tax, NI, UC and so on under the reform. -2. **Behavioural response on labour supply**: modelled with the +2. **Behavioural response on labour supply**: available through two + separate routes. `employment_income_behavioral_response` applies the substitution and income elasticities under - `gov.simulation.labour_supply_responses` (`policyengine_uk/dynamics/labour_supply.py`, - `employment_income_behavioral_response`). + `gov.simulation.labour_supply_responses` (both 0 by default). + Separately, calling `Simulation.apply_dynamics` on a simulation with a + baseline runs `apply_labour_supply_responses` + (`policyengine_uk/dynamics/labour_supply.py`), which applies the OBR's + elasticities by demographic group when + `gov.dynamic.obr_labour_supply_assumptions` is true, as it is by + default. 3. **Indirect-tax response**: *not modelled*. Households' consumption bundles are read from the dataset and don't shift in response to income changes. -#1114 reports that UKMOD handles this in its indirect tax (TCO) module with -an aggregate **consumption elasticity of 0.8**: a 1% rise in net income gives a 0.8% rise in -consumption, which then flows through VAT and excise revenue. That figure -has not yet been checked against UKMOD's own documentation, so it needs a -primary source before it is used as a default. +#1114 suggests that UKMOD handles this in its indirect taxation (TCO) +extension with a consumption elasticity of 0.8, citing only a ChatGPT +conversation. UKMOD's own +[TCO technical note](https://www.microsimulation.ac.uk/wp-content/uploads/2026/01/TCO-extension-v3.pdf) +(revised October 2025) does not give 0.8. It describes three alternative +assumptions: constant income shares (an income elasticity of +1 for every +commodity), constant expenditure shares (nominal savings held fixed, so the +whole change in disposable income goes to spending) and constant quantities +(an income elasticity of 0). Any default therefore needs another primary +source. ## Scope ### Phase 1: household-level consumption response - New parameter `gov/simulation/indirect_tax_response/consumption_elasticity.yaml`, - defaulting to 0 (no response) until a sourced value such as UKMOD's is - confirmed and cited in its metadata. + defaulting to 0 (no response) until a sourced value is confirmed and + cited in its metadata. - New variable `consumption_response_factor` (Household, YEAR) = `1 + elasticity * (net_income_change / baseline_net_income)`. - Apply the factor to the twelve COICOP categories that `consumption` @@ -51,7 +62,8 @@ primary source before it is used as a default. `reduced_rate_vat_consumption`), so it picks up the change. - The duties do not read those categories. Fuel duty reads `petrol_litres` and `diesel_litres`, which divide the `petrol_spending` - and `diesel_spending` inputs by pump prices. Alcohol duty reads the + and `diesel_spending` inputs by pump prices, and the `lpg_kg` and + `natural_gas_kg` inputs. Alcohol duty reads the `*_litres` inputs under `variables/input/consumption/alcohol/`, and tobacco duty reads `cigarettes_per_week` and the `*_grams_per_week` inputs under `variables/input/consumption/tobacco/`. The same factor @@ -60,10 +72,11 @@ primary source before it is used as a default. ### Phase 2: heterogeneity by category - A single economy-wide elasticity hides that some categories (food - staples, domestic energy) are nearly income-inelastic while others - (recreation, restaurants) are not. Replace the single + staples, domestic energy) are less income-elastic than others + (recreation, restaurants). Replace the single `consumption_elasticity.yaml` with a per-category set, calibrated to - ONS *Living Costs and Food Survey* income-elasticity estimates. + income elasticities estimated from the ONS *Living Costs and Food + Survey*. ### Phase 3: incidence link @@ -93,10 +106,11 @@ primary source before it is used as a default. Each `*_consumption` input variable currently reads straight from the dataset. Two options for the hook: -1. **Decorate at the input layer**: add `defined_for` / - `default_formula` that multiplies by `consumption_response_factor`. - This is clean but ties the consumption inputs to indirect-tax - modelling. +1. **Decorate at the input layer**: give each input a formula that + multiplies by `consumption_response_factor`. This ties the consumption + inputs to indirect-tax modelling, and policyengine-core does not run a + formula for a period the dataset already supplies, so it would not + change dataset values. 2. **Insert a derived layer**: keep `*_consumption_baseline` (reads from data) and introduce `*_consumption` formulas that apply the response factor. This is the safer route: users who do not need the indirect @@ -109,28 +123,33 @@ Recommendation: route (2). The dataset-side variable becomes ### Tests -- A reform that drops income tax for low-income households should - produce a *positive* `consumer_incident_tax_revenue_change` aggregate - (more take-home pay -> more consumption -> more VAT). +- Once Phase 3 derives `consumer_incident_tax_revenue_change` from the + response, a reform that drops income tax for low-income households should + produce a *positive* aggregate (more take-home pay -> more consumption -> + more VAT). Today the variable spreads the amount set by the + `consumer_incident_tax_change` parameter (in £bn) over households in + proportion to their consumption, so in a `policyengine_uk` simulation + with positive weighted consumption its weighted aggregate equals that + amount whatever the reform does to income. - A reform that raises NI should produce a *negative* aggregate. - The aggregate should track elasticity × (sum of net income change × VAT-equivalent rate) to within a small tolerance. ## Open questions -- Should the elasticity vary by household income decile? Empirical - evidence suggests low-income households have higher marginal - consumption propensity. +- Should the elasticity vary by household income decile, for example if + lower-income households spend a larger share of extra income? - Should saving be modelled explicitly (the residual after consumption)? - Phase 1 implicitly assumes any income change above the elasticity - applies to saving. + Phase 1 implicitly assumes that whatever part of an income change is not + consumed (a share of 1 − elasticity × baseline consumption / baseline + net income) is saved. - How does this interact with the existing labour-supply response so we don't double-count? ## References -- UKMOD documentation, to confirm the 0.8 consumption elasticity reported in #1114 (not yet checked). -- ONS [Living Costs and Food Survey](https://www.ons.gov.uk/peoplepopulationandcommunity/personalandhouseholdfinances/expenditure/bulletins/familyspendingintheuk/latest): income-elasticity calibration source for Phase 2. +- UKMOD, [Indirect Taxation (TCO) Extension technical note](https://www.microsimulation.ac.uk/wp-content/uploads/2026/01/TCO-extension-v3.pdf) (revised October 2025): its three expenditure assumptions; it does not give the 0.8 reported in #1114. +- ONS, [Family spending in the UK](https://www.ons.gov.uk/peoplepopulationandcommunity/personalandhouseholdfinances/expenditure/bulletins/familyspendingintheuk/latest): the Living Costs and Food Survey's headline publication; Phase 2 income elasticities would be estimated from LCF microdata. - HMRC, [VAT annual statistics](https://www.gov.uk/government/statistics/value-added-tax-vat-annual-statistics): calibration target for the aggregate. - Existing infrastructure: [`consumer_incident_tax_revenue_change`](https://github.com/PolicyEngine/policyengine-uk/blob/main/policyengine_uk/variables/contrib/policyengine/consumer_incident_tax_revenue_change.py) and the `gov/simulation/labour_supply_responses/` parameter tree for the parallel labour-supply pattern. - Related issue: [#1114](https://github.com/PolicyEngine/policyengine-uk/issues/1114). diff --git a/docs/book/programs/gov/cms/child-maintenance.md b/docs/book/programs/gov/cms/child-maintenance.md index 7dc437cd20..9dfc844c37 100644 --- a/docs/book/programs/gov/cms/child-maintenance.md +++ b/docs/book/programs/gov/cms/child-maintenance.md @@ -3,7 +3,7 @@ ```{warning} **Not yet modelled in detail.** PolicyEngine UK has two generic person-level inputs, `maintenance_income` and `maintenance_expenses`. They do not tell -child maintenance apart from maintenance between former partners, and the +child maintenance apart from spousal maintenance, and the model does not calculate a Child Maintenance Service (CMS) award. This page sets out the scope for doing that, tracked in [#669](https://github.com/PolicyEngine/policyengine-uk/issues/669) (still @@ -33,24 +33,38 @@ For **child** maintenance that is the right answer for UC: - **Paying parent:** maintenance paid is not deducted, so the paying parent is means-tested on their full earnings. -This asymmetry is the work-incentive problem described by the Centre for Social -Justice in [*The Hidden Parent Poverty Trap: Child Maintenance and Universal -Credit*][csj]. The model already reproduces it, because neither input reaches -UC. - -The gap is **maintenance between former spouses or civil partners**. -Reg. 66(1)(d) counts it as unearned income in UC, but the model cannot see it, -because it shares `maintenance_income` with child maintenance. Those -recipients' UC is overstated. +The paying-parent side of this asymmetry is the work-incentive problem +described by the Centre for Social Justice in [*The Hidden Parent Poverty Trap: +Child Maintenance and Universal Credit*][csj]. The model reproduces UC's side +of it, because neither input reaches UC. It does not reproduce the combined +marginal rates, because `maintenance_expenses` is a fixed input that does not +rise with earnings. + +The gap is **maintenance from a spouse, civil partner, former spouse or former +civil partner** under a court order or a maintenance agreement. +Reg. 66(1)(d) counts it as unearned income in UC, but the model cannot see it: +no maintenance input reaches UC, and `maintenance_income` does not tell it +apart from child maintenance +([policyengine-uk-data#627](https://github.com/PolicyEngine/policyengine-uk-data/issues/627) +discusses where FRS-based datasets record it). Those recipients' UC is +overstated. ## Scope +[PR #2573](https://github.com/PolicyEngine/policyengine-uk/pull/2573), +implementing [#2572](https://github.com/PolicyEngine/policyengine-uk/issues/2572), +proposes keeping `maintenance_income` for child maintenance received and +adding `spousal_maintenance_income` for spousal maintenance. That +implementation is pending; it would replace the input split proposed in +Phases 1 and 2 below while leaving the CMS calculation in Phase 3 unmodelled. + ### Phase 1: separate the inputs - New input `child_maintenance_received` (Person, GBP, annual). - New input `child_maintenance_paid` (Person, GBP, annual). -- Keep `maintenance_income` and `maintenance_expenses` for maintenance - between former partners, and say so in their documentation. +- Keep `maintenance_income` and `maintenance_expenses` for spousal + maintenance (received from, or paid to, a spouse, civil partner, former + spouse or former civil partner), and say so in their documentation. - In the data pipeline ([Microcosm UK](https://github.com/PolicyEngine/microcosm)), route the FRS child-maintenance items to the new inputs and the spousal items to the @@ -79,13 +93,21 @@ parent's gross weekly income | £100.01 to £199.99 | Reduced | Formula | | £200 to £3,000 | Basic | 12% / 16% / 19% of income up to £800 for 1 / 2 / 3+ children, then 9% / 12% / 15% of income from £800 to £3,000 | | Above £3,000 | Basic, capped | Income above £3,000 is ignored; the receiving parent can apply to the courts | -| Not enough information | Default | £38 / £51 / £64 | +| Not enough information | Default | £39 / £51 / £64 ([SI 2012/2677 reg. 49](https://www.legislation.gov.uk/uksi/2012/2677/regulation/49); GOV.UK's guide shows £38 for one child) | Then: -- **Shared care:** the amount falls by 1/7 for 52 to 103 nights a year, 2/7 - for 104 to 155, 3/7 for 156 to 174, and by half plus £7 a week for 175 or - more. +- **Shared care:** for the basic and reduced rates, the amount payable to + each carer falls by 1/7 where a child spends 52 to 103 nights a year with + the paying parent, 2/7 for 104 to 155, 3/7 for 156 to 174 and 1/2 for 175 + or more; where one carer looks after more than one qualifying child, the + fractions for those children are added and divided by the number of them + (para. 7(5)). It then falls + by a further £7 a week for each child with 175 or more nights, but not + below £7 a week in total (Sch. 1 para. 7). Where the flat rate applies + because the paying parent or their partner gets a prescribed benefit + (para. 4(1)(b) or (c) or 4(2)), 52 or more nights with a child makes the + amount payable to that child's carer nil (para. 8). - **Other children:** an adjustment applies for other children living with the paying parent. diff --git a/docs/book/programs/gov/dfe/regional-childcare-plan.md b/docs/book/programs/gov/dfe/regional-childcare-plan.md index 225a5291e6..4a1efeb7f5 100644 --- a/docs/book/programs/gov/dfe/regional-childcare-plan.md +++ b/docs/book/programs/gov/dfe/regional-childcare-plan.md @@ -14,25 +14,30 @@ and the broader [#1644 audit][issue-1644]. ## What is modelled today -The current childcare coverage is captured in -[`policyengine_uk/programs.yaml`](https://github.com/PolicyEngine/policyengine-uk/blob/main/policyengine_uk/programs.yaml) (see the -[#1644 split into per-programme rows][issue-1644]) and works as follows: +The current childcare coverage is recorded in +[`policyengine_uk/programs.yaml`](https://github.com/PolicyEngine/policyengine-uk/blob/main/policyengine_uk/programs.yaml), +where [#1703](https://github.com/PolicyEngine/policyengine-uk/pull/1703) +split the old umbrella entry into per-programme rows after the +[#1644 audit][issue-1644]. Care to Learn and the Scottish funded ELC have +no row yet. Coverage works as follows: - **UK-wide**: Tax-Free Childcare (HMRC); Universal Credit childcare element (DWP); Working Tax Credit childcare element (legacy, HMRC - administered, modelled under `gov/dwp/`; see + administered, ended 5 April 2025, modelled under `gov/dwp/`; see [tax-credits.md](./../dwp/tax-credits.md)). - **England-only** (gate on `country == ENGLAND`): Universal Childcare Entitlement (15 hours for 3- and 4-year-olds), Extended Childcare Entitlement (working parents, now reaching down to children aged nine months), Targeted Childcare Entitlement (2-year-olds - in families on qualifying benefits or low incomes), Care to Learn, - Student Finance England Childcare Grant. + in families on qualifying benefits, or on Universal Credit or tax + credits with income below a limit), Care to Learn, Student Finance + England Childcare Grant. - **Scotland**: eligibility for the universal funded ELC for 3- and - 4-year-olds (`scottish_funded_early_learning_childcare_eligible`, - including the statutory start-date rule for 3-year-olds) and its 1,140 - hours a year, added in [#1643][pr-1643]. No funding value is attached - yet, so it does not reach household income. + 4-year-olds (`scottish_funded_early_learning_childcare_eligible`, with + the statutory start date for 3-year-olds taken as an input, + `scottish_funded_early_learning_childcare_start_date_reached`) and its + 1,140 hours a year, added in [#1643][pr-1643]. No funding value is + attached yet, so it does not reach household income. - **Wales, Northern Ireland**: no programmes modelled. ## Proposed scope by jurisdiction @@ -41,16 +46,25 @@ The current childcare coverage is captured in The [#1644 audit][issue-1644] flagged three Scottish gaps: -1. **Eligible-two-year-old strand** of the funded ELC programme - (vulnerable or low-income 2-year-olds under the - [Provision of Early Learning and Childcare (Specified Children) - (Scotland) Order 2014][sssi-196-2014]). Modelled the same way as the - 3- and 4-year-old strand but gated on UC, tax credits or other - qualifying benefits. -2. **Student Awards Agency Scotland (SAAS) Lone Parents' Grant, - Dependants' Grant, and Childcare Fund** for full-time HE students: - the Scottish equivalent of Student Finance England's Childcare - Grant. Income-tested. +1. **Eligible-two-year-old strand** of the funded ELC programme: + 2-year-olds who are looked after, under a kinship care order or with + an appointed guardian ([Children and Young People (Scotland) Act 2014 + s. 47(3)][cyp-47]), or whose parent receives a qualifying benefit + under the [Provision of Early Learning and Childcare (Specified + Children) (Scotland) Order 2014][ssi-196-2014]. It would be modelled + the same way as the 3- and 4-year-old strand but gated on those + conditions. The qualifying benefits include Universal Credit with + earned income of up to £850 an assessment period; Child Tax Credit + was removed from the list from 6 April 2025 ([SSI 2025/82][ssi-82-2025]). +2. **Student Awards Agency Scotland (SAAS) support for student + parents**: the Childcare Funds that universities and colleges pay out + (a Lone Parents' Childcare Grant of up to £1,215 a year, not + income-assessed, plus discretionary help assessed on need), the + closest Scottish equivalent of Student Finance England's Childcare + Grant ([SAAS childcare fund guidance, 2026 to 2027][saas-childcare]); + and SAAS's income-assessed Lone Parents' Grant (up to £1,305). The + audit also listed SAAS's Dependants' Grant, but that supports adult + dependants rather than children. 3. **Support for young parents in further education**, the role Care to Learn plays in England. The [Education Maintenance Allowance][scot-ema] still runs in Scotland for 16- to 19-year-olds, applied for through @@ -61,14 +75,21 @@ The [#1644 audit][issue-1644] flagged three Scottish gaps: The audit identified four Welsh programmes: -1. **Childcare Offer for Wales**: 30 hours a week for working parents - of 3- and 4-year-olds (term time plus some holiday weeks). +1. **Childcare Offer for Wales**: up to 30 hours a week of combined + nursery education (at least 10 hours) and childcare, for up to 48 + weeks a year, for 3- and 4-year-olds of working parents or parents in + further or higher education + ([gov.wales](https://www.gov.wales/get-30-hours-childcare-3-and-4-year-olds)). Modelled the same as the English Extended Childcare Entitlement with Welsh-specific income tests and hour caps. Statutory basis: [Childcare Funding (Wales) Act 2019][wales-act]. -2. **Flying Start**: a universal offer in disadvantaged areas for - 2- to 3-year-olds (12.5 hours a week). Eligibility is geographic, - based on lower layer super output areas. +2. **Flying Start**: funded childcare for 2- to 3-year-olds in some of + the most disadvantaged areas of Wales (12.5 hours a week, 39 weeks a + year), expanded in phases since 2022 with the aim of reaching all + 2-year-olds + ([Welsh Government guidance](https://www.gov.wales/flying-start-childcare-guidance-html)). + Eligibility is geographic, based on lower layer super output areas + chosen by each local authority. 3. **Student Finance Wales Childcare Grant**, parallel to the English Childcare Grant. 4. **Welsh EMA**, parallel to the Scottish EMA. @@ -77,8 +98,12 @@ The audit identified four Welsh programmes: The audit identified four NI programmes: -1. **Pre-school Education Programme (PSEP)**: 12.5 hours a week in the - year before school. Universal. +1. **Pre-school Education Programme (PSEP)**: a funded place in the + immediate pre-school year for every child whose family wants one. + In December 2024, 60% of children had 12.5 hours a week and 40% had + 22.5 hours; the Department of Education is moving places to 22.5 + hours, starting with 77 settings from September 2025 + ([Department of Education](https://www.education-ni.gov.uk/news/education-minister-announces-first-step-major-expansion-pre-school-education-provision)). 2. **Sure Start**: younger children in disadvantaged areas, including a programme for 2-year-olds. 3. **Student Finance NI Childcare Grant**. @@ -88,10 +113,11 @@ The audit identified four NI programmes: For each new programme: -- New parameter tree under `parameters/gov///` - capturing the income tests, hours offered, and weekly funding rates. -- New variable(s) under `variables/gov///` gated on - the appropriate `country == ...` check. +- New parameter tree under the administering body (as #1643 did with + `parameters/gov/scottish_government//`) capturing the + income tests, hours offered, and weekly funding rates. +- New variable(s) under the matching `variables/gov///` + folder, gated on the appropriate `country == ...` check. - A `_eligible` companion variable that the new programme variable composes with the rate parameter, the same pattern as the English entitlement trio. @@ -100,8 +126,10 @@ For each new programme: ### Structural items deferred to this work -The [#1644 audit][issue-1644] also flagged two structural items that -become more pressing once the regional programmes land: +The [#1644 audit][issue-1644] also listed four structural findings. +#1703 has since split the `programs.yaml` umbrella row, and one (a +shared `in_england` helper) is minor; the other two become more pressing +once the regional programmes land: - `weeks_per_year` and `childcare_funding_rate` currently sit at `gov.dfe.*` root but encode English term-week and funding-rate @@ -118,19 +146,25 @@ per-programme work. ## Data needs -Most regional programmes are point-in-time per-hour funding rather -than per-household payments, so the data side is comparatively easy: - -- **FRS** provides `region` and `country` at the household level, - enough to gate eligibility. -- **Welsh Government Statistical Bulletins** publish Flying Start - and Childcare Offer caseloads by local authority. -- **Scottish Government Childcare and Early Years statistics** provide - Scottish funded ELC caseloads. +Several regional programmes (the Scottish two-year-old strand, the +Childcare Offer, Flying Start and PSEP) are funded hours rather than +cash payments, so their data side is comparatively easy: + +- **FRS** provides the household's `region`, from which the model + derives `country`; that is enough to gate eligibility. +- **Welsh Government** publishes Flying Start official statistics with + local authority breakdowns, and an annual Childcare Offer for Wales + report that gives the number of children using the Offer by local + authority. +- **Scottish Government Funded Early Learning and Childcare Statistics** + (the annual ELC census) provide funded ELC registrations, including + eligible two-year-olds. - **Department of Education NI** publishes PSEP caseloads. -- The SAAS, Student Finance Wales and NI childcare grants need the same - treatment as the English `childcare_grant`: calibration against - published award totals. +- The SAAS, Student Finance Wales and Student Finance NI childcare + grants need published award totals to check the modelled amounts + against. The English `childcare_grant` has no calibration target yet: + the Student Loans Company targets in policyengine-uk-data cover + Parents' Learning Allowance and Adult Dependants' Grant only. The aggregating challenge is choosing whether to expose **subsidised hours** or **subsidised cost** as the primary household-level @@ -144,27 +178,36 @@ entitlements already enter `household_benefits` and `gov_spending`. splits Universal / Extended / Targeted), or as one variable with the age-and-eligibility logic inside? Recommendation: separate, mirroring the English pattern for queryability. -- Welsh Flying Start is geographic. Microcosm UK, the dataset being - built to replace the Enhanced FRS, assigns each household in England - and Wales to a 2021 Output Area, which nests in LSOAs, so eligibility - by area becomes possible on that dataset once the Flying Start area - list is sourced. +- Welsh Flying Start is geographic. Microcosm UK, the default UK dataset + in the policyengine.py bundle since release 6.10.0 (21 October 2026), + places each household in its local-area build in one atomic area + (a 2021 Output Area in England and Wales, a 2022 Output Area in + Scotland, a 2021 Data Zone in Northern Ireland) and derives larger + geographies such as LSOAs from it. The national file carries no small + areas. Eligibility by area becomes possible on the local-area build + once the Flying Start LSOA list is sourced. - EMA ended in England but still runs in Scotland, Wales and Northern - Ireland. Today `child_ema` and `adult_ema` are reported amounts with - no formula. Should the devolved schemes be modelled under the existing - `gov.dfe.care_to_learn` umbrella or as separate regional schemes? + Ireland. Today `child_ema` and `adult_ema` (under + `variables/gov/dwp/`) are reported FRS amounts with no formula. Should + the devolved schemes sit alongside the England-only + `gov.dfe.care_to_learn` tree, or be separate schemes under each + administration? ## References - Audit: [#1644](https://github.com/PolicyEngine/policyengine-uk/issues/1644), the full childcare coverage audit. - Tracking: [#1008](https://github.com/PolicyEngine/policyengine-uk/issues/1008), the original Scotland, Wales and NI gap report. -- Recent landed work: [#1643][pr-1643], Scottish universal funded ELC. -- Existing England docs (for the pattern to mirror): the entitlement variables under [`policyengine_uk/variables/gov/dfe/`](https://github.com/PolicyEngine/policyengine-uk/blob/main/policyengine_uk/variables/gov/dfe). +- Recent landed work: [#1643][pr-1643], Scottish universal funded ELC; [#1703](https://github.com/PolicyEngine/policyengine-uk/pull/1703), per-programme `programs.yaml` rows. +- Existing England implementation (the pattern to mirror): the entitlement variables under [`policyengine_uk/variables/gov/dfe/`](https://github.com/PolicyEngine/policyengine-uk/blob/main/policyengine_uk/variables/gov/dfe). - Welsh childcare statute: [Childcare Funding (Wales) Act 2019][wales-act]. -- Scottish eligible-2-year-old statute: [SSSI 2014/196][sssi-196-2014]. +- Scottish eligible-2-year-old legislation: [Children and Young People (Scotland) Act 2014 s. 47(3)][cyp-47] and [SSI 2014/196][ssi-196-2014], as amended by [SSI 2025/82][ssi-82-2025]. +- SAAS, [University Childcare Fund guidance, 2026 to 2027][saas-childcare]. [issue-1644]: https://github.com/PolicyEngine/policyengine-uk/issues/1644 [pr-1643]: https://github.com/PolicyEngine/policyengine-uk/pull/1643 [wales-act]: https://www.legislation.gov.uk/anaw/2019/1/contents -[sssi-196-2014]: https://www.legislation.gov.uk/ssi/2014/196/contents/made +[ssi-196-2014]: https://www.legislation.gov.uk/ssi/2014/196/contents +[ssi-82-2025]: https://www.legislation.gov.uk/ssi/2025/82/made +[cyp-47]: https://www.legislation.gov.uk/asp/2014/8/section/47 +[saas-childcare]: https://www.saas.gov.uk/files/311/saas-childcare-fund-guidance.pdf [scot-ema]: https://www.mygov.scot/ema diff --git a/docs/book/programs/gov/dwp/bereavement-support-plan.md b/docs/book/programs/gov/dwp/bereavement-support-plan.md index 9499139048..6477232479 100644 --- a/docs/book/programs/gov/dwp/bereavement-support-plan.md +++ b/docs/book/programs/gov/dwp/bereavement-support-plan.md @@ -12,8 +12,9 @@ the related historical-WPA ask in [#465](https://github.com/PolicyEngine/policyengine-uk/issues/465). Updated October 2026: still reported-only; the facts below were checked against GOV.UK and legislation.gov.uk. The data notes cover both the -certified Enhanced FRS (policyengine-uk-data) and Microcosm UK, the -dataset being built to replace it. +Enhanced FRS (policyengine-uk-data, which policyengine-uk's own tests +still pin at 1.58.1) and Microcosm UK, which replaced it as the certified +default UK dataset in policyengine.py 6.10.0 (21 October 2026). ``` ## What's currently modelled @@ -36,25 +37,42 @@ That means: - Reform analyses changing BSP rates have no household-level surface. - Anyone bereaved after the FRS reference week never appears as a BSP recipient even if eligible. -- The legacy WPA has no variable of its own. Both the Enhanced FRS +- The legacy WPA has no variable of its own. The Enhanced FRS ([`frs.py`](https://github.com/PolicyEngine/policyengine-uk-data/blob/main/policyengine_uk_data/datasets/frs.py)) - and Microcosm UK fill `bsp_reported` from FRS benefit codes 6 and 9 - ([`frs_spine.py`](https://github.com/PolicyEngine/microcosm/blob/75167a68/packages/microcosm-build/src/microcosm/build/uk_runtime/frs_spine.py#L825)), - and Microcosm's own + fills `bsp_reported` from FRS benefit codes 6 and 9, and so did + Microcosm UK up to its certified 19 October 2026 build + ([`frs_spine.py`](https://github.com/PolicyEngine/microcosm/blob/75167a68/packages/microcosm-build/src/microcosm/build/uk_runtime/frs_spine.py#L825)); + Microcosm's own [source notes](https://github.com/PolicyEngine/microcosm/blob/d1fda720/packages/microcosm-build/src/microcosm/build/uk/spec/sources.yaml#L1870) - say code 6 mixes benefits and code 9 is tax-free War Widow's Pension. So the reported "BSP" also carries legacy - widow's benefits and War Widow's Pension. + say code 6 mixes benefits and code 9 is tax-free War Widow's Pension. + The FRS labels code 6 "Bereavement Support Payment / Widowed Parent's + Allowance" and code 9 "War Widow's/Widower's Pension", so wherever both + codes are summed the reported "BSP" also carries Widowed Parent's + Allowance and War Widow's Pension. Microcosm's main branch now puts + code 9 in `war_widows_pension_reported` and keeps only code 6 in + `bsp_reported` + ([microcosm#1333](https://github.com/PolicyEngine/microcosm/pull/1333)). - Since [#2382](https://github.com/PolicyEngine/policyengine-uk/pull/2382) (2.135.7), War Widow's Pension has its own variables, `war_widows_pension_reported` and `war_widows_pension`. `bsp_reported` keeps CPI uprating only until the datasets move FRS code 9 out of it - ([policyengine-uk-data#578](https://github.com/PolicyEngine/policyengine-uk-data/issues/578), - [microcosm#1222](https://github.com/PolicyEngine/microcosm/issues/1222)); + (done on Microcosm's main branch in microcosm#1333; still open for the + Enhanced FRS in + [policyengine-uk-data#578](https://github.com/PolicyEngine/policyengine-uk-data/issues/578), + and [microcosm#1222](https://github.com/PolicyEngine/microcosm/issues/1222) + is still open); [#2381](https://github.com/PolicyEngine/policyengine-uk/issues/2381) tracks freezing it after that. ## Proposed scope +[PR #2573](https://github.com/PolicyEngine/policyengine-uk/pull/2573), +implementing [#2572](https://github.com/PolicyEngine/policyengine-uk/issues/2572), +proposes `widowed_parents_allowance_reported` and +`widowed_parents_allowance`, with the appropriate tax and benefit-income +treatment. This reported-input implementation is pending; the eligibility +and rule-based amount described in Phase 2 would still be needed for reforms. + ### Phase 1: rule-based BSP formula The BSP under the [Pensions Act 2014, Part 5][pa-2014-5] and the @@ -62,18 +80,27 @@ The BSP under the [Pensions Act 2014, Part 5][pa-2014-5] and the pays: - A **lump sum** of either £2,500 (standard rate) or £3,500 (higher - rate, with a dependent child). -- **18 monthly payments** of £100 (standard) or £350 (higher), for a - death on or after 6 April 2017 - ([GOV.UK](https://www.gov.uk/bereavement-support-payment/what-youll-get)). - -The higher rate applies if, when the partner died, the claimant was -getting (or entitled to) Child Benefit for a child living with them, or -was pregnant. The + rate), paid only if the claim is made within 12 months of the death + (SI 2017/410 reg 3(2) and (5)). +- Up to **18 monthly payments** of £100 (standard) or £350 (higher), all + 18 only if the claim is made within 3 months of the death + ([GOV.UK](https://www.gov.uk/bereavement-support-payment/what-youll-get)), + for a death on or after 6 April 2017, when Pensions Act 2014 s. 30 came + fully into force. Someone who reaches State Pension age within 18 + months of the death may get fewer. + +The higher rate applies if the claimant was pregnant or entitled to +Child Benefit when the partner died, or later became entitled to Child +Benefit for a child who lived with them or the partner immediately +before the death (SI 2017/410 reg 4). The [Bereavement Benefits (Remedial) Order 2023](https://www.legislation.gov.uk/uksi/2023/134/contents) (SI 2023/134) extended BSP and WPA to cohabiting partners with children, with effect back to 30 August 2018, after the Supreme Court's -2018 McLaughlin judgment. It didn't change the number of payments. +2018 McLaughlin judgment (on WPA) and the High Court's 2020 Jackson +judgment (on BSP). It kept the 18-month maximum, but a cohabiting +partner whose partner died between 6 April 2017 and 29 August 2018 gets +18 monthly payments less one for each month that had passed by +29 August 2018 (SI 2017/410 reg 2(7) to (11)). To model this we need: @@ -82,8 +109,10 @@ To model this we need: the [single initial payment](https://www.gov.uk/bereavement-support-payment/what-youll-get) is paid once. A flag for "bereaved in the last 18 months" would stay true across two consecutive annual periods and pay the lump sum twice. -- `bsp_higher_rate_eligible` (Person, YEAR): whether they have a - dependent child. +- `bsp_higher_rate_eligible` (Person, YEAR): whether they were pregnant + or entitled to Child Benefit when the partner died, or became entitled + afterwards for a child who lived with them or the partner + (SI 2017/410 reg 4). - `bsp_months_remaining` (Person, YEAR): how many of the 18 monthly instalments fall within the simulation year (0 to 12). This is where the 18-month window belongs. @@ -113,24 +142,37 @@ been £350, £3,500, £100 and £2,500 since April 2017, and the ### Phase 2: historical WPA under the same formula Widowed Parent's Allowance (under [Social Security Contributions and -Benefits Act 1992 s. 39A][sscba-39a]) was paid for deaths from April -2001 up to 5 April 2017. The benefit: - -- Continued for as long as the surviving spouse had a dependent child. -- Was uprated annually. +Benefits Act 1992 s. 39A][sscba-39a]) is paid for deaths before 6 April +2017 (s. 39A(1)(a)), except where a woman's husband died before 9 April +2001 (s. 39A(1A)). The benefit: + +- Continues while the survivor is entitled to Child Benefit for a + qualifying child (or is pregnant) and is under State Pension age, and + ends on remarriage or a new civil or cohabiting partnership + (s. 39A(4) and (5)). +- Is uprated each April (£156.65 a week from April 2026, SI 2026/148 + art 18). - Was replaced by BSP for deaths from 6 April 2017. For historical analyses the model should: -- Add `wpa_eligible` (Person, YEAR): partner died between April 2001 - and 5 April 2017, the survivor is under State Pension age and is - entitled to Child Benefit. +- Add `wpa_eligible` (Person, YEAR): the partner died before 6 April + 2017 (but not a woman's husband before 9 April 2001), the partner met + the contribution conditions, and the survivor is under State Pension + age, has not remarried or formed a new partnership, and is entitled to + Child Benefit for a qualifying child or is pregnant (SSCBA 1992 + s. 39A). - Add `widowed_parents_allowance` (Person, YEAR) as a derived formula reading the WPA rate parameter (which would need backfilling under `gov/dwp/wpa/rate.yaml`). -WPA has no new awards, but existing awards continue while the -surviving parent has a dependent child, so some recipients remain. In +WPA is closed to deaths from 6 April 2017, but claims for earlier +deaths are still possible: the 2023 Remedial Order brought about 3,400 +successful WPA claims from cohabiting partners between February 2023 and +November 2024 +([DWP, July 2025](https://www.gov.uk/government/publications/bereavement-benefit-remedial-order-uptake-february-2023-to-november-2024)). +Existing awards continue while the surviving parent is entitled to Child +Benefit and under State Pension age, so some recipients remain. In the current data they sit inside `bsp_reported` (see above), so they are already counted in income; the gap is that they can't be told apart from BSP or reformed separately. @@ -138,14 +180,16 @@ apart from BSP or reformed separately. ### Phase 3: data-side bereavement inputs Neither `bsp_initial_payment_due` nor `bsp_months_remaining` exists in -the FRS. The FRS is a cross-section, not a panel, and doesn't record -when a partner died, so these inputs would need imputing in the data (Microcosm UK, as the -replacement dataset): +the FRS. The FRS is a cross-section, not a panel, and its benefits +documentation shows no record of when a partner died, so these inputs +would need imputing in the data (Microcosm UK, now the certified default +UK dataset): - start from FRS respondents who report BSP (once it is separated from the other codes, as above) and from widowed respondents; -- calibrate to DWP's BSP and WPA caseloads ([Stat-Xplore](https://stat-xplore.dwp.gov.uk/)) by age, sex - and region. +- calibrate to DWP's BSP caseloads ([Stat-Xplore](https://stat-xplore.dwp.gov.uk/), + by age, gender and geography, from April 2017) and to WPA caseloads + where DWP still publishes them. This is the heaviest of the three phases; without it the new variables work for the household calculator (which can input them @@ -156,10 +200,12 @@ directly) but not for microsim aggregates. - WPA recipients who would have been on BSP under the new rules: do we want a reform-side parameter "treat all WPA receipts as if they were BSP" for counterfactual analysis? -- The 2023 Remedial Order extended BSP to cohabiting partners with - children, with effect from 30 August 2018. Eligibility should not - require marriage or civil partnership for deaths from that date where - there are children. +- The 2023 Remedial Order extended BSP to cohabiting partners, with + effect from 30 August 2018. Eligibility should not require marriage or + civil partnership where the survivor was pregnant or entitled to Child + Benefit (Pensions Act 2014 s. 30(1)(aa)), for deaths on or after + 6 April 2017; for deaths before 30 August 2018 the monthly payments + are cut as in SI 2017/410 reg 2(7) to (11). ## References @@ -169,7 +215,8 @@ directly) but not for microsim aggregates. - [Bereavement Benefits (Remedial) Order 2023 (SI 2023/134)](https://www.legislation.gov.uk/uksi/2023/134/contents): cohabiting-parent extension, effective from 30 August 2018. - [SSCBA 1992 s. 39A][sscba-39a]: legacy WPA basis. - gov.uk, [Bereavement Support Payment](https://www.gov.uk/bereavement-support-payment) and [Widowed Parent's Allowance](https://www.gov.uk/widowed-parents-allowance). -- DWP, [Stat-Xplore](https://stat-xplore.dwp.gov.uk/): Bereavement Support Payment and Widowed Parent's Allowance caseloads. +- DWP, [Stat-Xplore](https://stat-xplore.dwp.gov.uk/): Bereavement Support Payment claims and recipients from April 2017. +- DWP, [Uptake of backdated BSP and WPA under the Bereavement Benefits (Remedial) Order 2023](https://www.gov.uk/government/publications/bereavement-benefit-remedial-order-uptake-february-2023-to-november-2024) (July 2025). - Issues: [#466](https://github.com/PolicyEngine/policyengine-uk/issues/466) (BSP formula), [#465](https://github.com/PolicyEngine/policyengine-uk/issues/465) (historical WPA). [pa-2014-5]: https://www.legislation.gov.uk/ukpga/2014/19/part/5 diff --git a/docs/book/programs/gov/dwp/cold-weather-payment-plan.md b/docs/book/programs/gov/dwp/cold-weather-payment-plan.md index b24fd6315b..591ff5c6c8 100644 --- a/docs/book/programs/gov/dwp/cold-weather-payment-plan.md +++ b/docs/book/programs/gov/dwp/cold-weather-payment-plan.md @@ -14,7 +14,8 @@ In Scotland the equivalent is the **Winter Heating Payment**, a flat annual payment (£62.00 from April 2026, regulation 9 of [SSI 2023/16](https://www.legislation.gov.uk/ssi/2023/16/contents) as amended), which -replaced Cold Weather Payment there in 2023. It isn't modelled either. +replaced Cold Weather Payment there from winter 2022-23 (first paid in +February 2023). It isn't modelled either. The separate Pension Age Winter Heating Payment, which replaced Winter Fuel Payment in Scotland, is modelled under `gov.social_security_scotland.pawhp`. Updated October 2026: amounts, @@ -24,7 +25,7 @@ legislation and the Scotland and Northern Ireland position checked. ## What CWP is Defined in [The Social Fund Cold Weather Payments (General) -Regulations 1988][cwp-1988] (SI 1988/1724). A household qualifies if +Regulations 1988][cwp-1988] (SI 1988/1724). A claimant qualifies if **all** of the following hold: - The address is in a postcode linked to a weather station where the @@ -33,15 +34,21 @@ Regulations 1988][cwp-1988] (SI 1988/1724). A household qualifies if - A member of the benefit unit receives a **qualifying benefit**: Pension Credit; income-related ESA in the support or work-related activity group, or with a severe or enhanced disability premium, a - pensioner premium, a disabled child or a child under 5 (the - Regulations extend the same premium test to Income Support and - income-based JSA); Universal Credit when neither the claimant nor a - partner is employed or gainfully self-employed and someone has - limited capability for work or there is a child under 5, or, whether - working or not, when the award includes a disabled child amount; or - Support for Mortgage Interest with one of those premiums or children. - The [GOV.UK eligibility page](https://www.gov.uk/cold-weather-payment/eligibility) - has the exact conditions. + pensioner premium, a disabled child or a child under 5 (Income Support + and income-based JSA qualify on a similar test, under which the + disability, higher pensioner and disabled child premiums also count); + Universal Credit when neither the claimant nor a partner is employed + or gainfully self-employed and someone has limited capability for + work or there is a child under 5, or, whether working or not, when the + award includes a disabled child amount; or Support for Mortgage + Interest, either while treated as entitled to Pension Credit (no + further condition) or with one of those premiums or children. The + [GOV.UK eligibility page](https://www.gov.uk/cold-weather-payment/eligibility) + summarises them; regulation 1A has the full conditions. +- The home is in England or Wales, and the claimant doesn't live in a + care home, an independent hospital or certain other accommodation, + unless the family includes a child under 5 or child tax credit + includes a disability element (regulation 1A(4) and (8)). The payment is £25 per 7-day cold period, and each further cold period triggers another payment. Spending varies a lot with the winter: in @@ -54,22 +61,25 @@ weekly triggers, from which payments can be estimated. ## Why this is harder than other DWP benefits -The eligibility test combines an administrative benefit check (clean -for PolicyEngine) with a **geographic weather event** (no FRS -analogue). Modelling the weather side has three plausible approaches: +The eligibility test combines an administrative benefit check (mostly +covered by existing PolicyEngine variables, though Support for Mortgage +Interest and the legacy pensioner premium aren't modelled) with a +**geographic weather event** (no FRS analogue). Modelling the weather side has three plausible approaches: 1. **Expected-value parameterisation**: a single national - `cold_weather_payments_expected_periods` parameter capturing the - long-run-average number of 7-day cold periods triggered per winter. - Multiply by £25 and by the qualifying household flag to produce the expected - annual payment. Easy to implement, accurate in aggregate, wrong in + `gov.dwp.cold_weather_payment.expected_periods_per_year` parameter + capturing the long-run-average number of 7-day cold periods triggered + per winter. Multiply by £25 and by the qualifying benefit-unit flag to + produce the expected annual payment. Easy to implement, accurate in aggregate, wrong in any specific winter. 2. **Regional weather mixing**: a per-region weather-event - distribution (for example, the North East has more cold periods than London on - average), parameterised from Met Office historic data. Captures the + distribution (for example, the North East has more cold periods than + London on average), parameterised from DWP's trigger counts by + weather station. Captures the distributional skew without forecasting any particular winter. 3. **Year-specific lookups**: tabulate the DWP-published *Cold Weather - Payment statistics* by region by year and use those as historical + Payment estimates* by weather station by year (mapped to regions + through DWP's postcode-district lists) and use those as historical inputs. Most accurate for back-cast simulations; needs ongoing updating. @@ -99,8 +109,8 @@ benefits) at the cost of smoothing year-to-year weather variance. ### Phase 2: regional weather mixing Add a `gov.dwp.cold_weather_payment.expected_periods_by_region` table -parameterised from Met Office observation data, and a household-side -lookup by `region`. +parameterised from DWP's trigger counts by weather station, and a +household-side lookup by `region`. ### Phase 3: Scotland's Winter Heating Payment @@ -113,9 +123,12 @@ child-under-5 conditions), assessed in a qualifying week (2 to 8 November in 2026), but the payment is a flat annual amount with no weather trigger. So: -- `winter_heating_payment_eligible`: the Phase 1 eligibility rule plus - residence in Scotland. Reuse the Phase 1 variable rather than a second - copy of the benefit conditions. +- `winter_heating_payment_eligible`: the Phase 1 benefit conditions, + without Cold Weather Payment's care-home and England-and-Wales + conditions (and without the child tax credit disability element, + which Winter Heating Payment dropped in 2025), plus residence in + Scotland in the qualifying week. Reuse the Phase 1 benefit-condition + variable rather than a second copy of the benefit conditions. - `winter_heating_payment` = `amount` × eligible, with `amount` uprated each year (£62.00 from April 2026). @@ -123,15 +136,23 @@ weather trigger. So: - DWP, [Cold Weather Payment statistics](https://www.gov.uk/government/collections/social-fund-cold-weather-payments): estimated eligible numbers at the start of each season and weekly triggers by weather station. The main calibration source. - Social Security Scotland's Winter Heating Payment statistics: Phase 3 calibration. -- Met Office, [UK climate historic stations](https://www.metoffice.gov.uk/research/climate/maps-and-data/historic-station-data): Phase 2 regional expected-period parameterisation. - -The FRS gives `region` cleanly, so the geographic gate is no extra -data work for Phases 1 and 2. +- DWP's trigger counts by weather station (Table 3 of each season's + *Cold Weather Payment estimates*), mapped to regions through DWP's + postcode-district lists: Phase 2 regional expected-period + parameterisation. The Met Office's + [historic station data](https://www.metoffice.gov.uk/research/climate/maps-and-data/historic-station-data) + gives only monthly series for 37 stations, so it can serve as a + cross-check but can't count 7-day cold periods. + +The FRS records `region`, from which the model derives `country`, so the +England-and-Wales and Scotland gates need no extra data. Region is a +coarse stand-in for the weather-station areas that trigger payments, so +Phase 2 models a regional average, not individual stations. ## Open questions - The qualifying-benefit list is narrower than for the Warm Home - Discount in England and Wales, which now covers everyone on + Discount in England and Wales, which now covers bill-payers on Universal Credit, Housing Benefit, income-related ESA or Pension Credit. Model the eligibility as one composite variable, shared with the Scottish Winter Heating Payment, or as separate flags by route diff --git a/docs/book/programs/gov/dwp/pip-points-plan.md b/docs/book/programs/gov/dwp/pip-points-plan.md index 67a20e8091..79400b2b94 100644 --- a/docs/book/programs/gov/dwp/pip-points-plan.md +++ b/docs/book/programs/gov/dwp/pip-points-plan.md @@ -1,34 +1,43 @@ -# PIP point system (planned) +# PIP point system (partly modelled) ```{warning} -**Not yet modelled.** PolicyEngine UK currently takes the PIP daily-living -and mobility **band** as a categorical input (`pip_dl_category` and -`pip_m_category`, each `STANDARD | ENHANCED | NONE`) and multiplies by -the published weekly rate. The **points-system** logic that determines -which band a claimant falls into (ten daily-living activities and two -mobility activities, with descriptors worth 0 to 12 points under the -legislation) is **not** modelled. +**Partly modelled.** PolicyEngine UK sets the PIP daily-living and +mobility **band** in `pip_dl_category` and `pip_m_category` (each +`STANDARD | ENHANCED | NONE`) and multiplies by the published weekly rate. +A band can be given as an input or derived from total points +(`pip_dl_points`, `pip_m_points`) against the 8- and 12-point thresholds. +The activity-level logic (ten daily-living activities and two mobility +activities, with descriptors worth 0 to 12 points under the legislation) +is **not** modelled. This page captures the proposed scope, tracked under [#1069](https://github.com/PolicyEngine/policyengine-uk/issues/1069). ``` ```{admonition} Status, October 2026 -A first step is open as +A first step merged on 21 October 2026 as [#1661](https://github.com/PolicyEngine/policyengine-uk/pull/1661): total points inputs (`pip_dl_points`, `pip_m_points`) and the 8- and 12-point -thresholds under `gov.dwp.pip.{daily_living,mobility}.points_thresholds`, -with the category derived from points only when it is not given as an -input. That is a simpler form of Phase 1 below: no per-activity scores -and no `use_point_system` switch, since a category input already takes -precedence over the formula. - -PR #1661 is waiting on a data-side path. Its companion, +thresholds under `gov.dwp.pip.{daily_living,mobility}.points_thresholds`. +The category is derived from points only when no category is given as an +input for the year. A category given for a year is stored for everyone in +the simulation, with NONE for anyone not given one. Otherwise the person's +latest assessed points (not -1) for that year or an earlier one set the +category, except that +a category given for an earlier year carries forward unless those points are +for a later year than it; with neither, the category is NONE +(`policyengine_uk/utils/pip.py`). That is a simpler form of Phase 1 below: +no per-activity scores and no `use_point_system` switch, since a category +input already takes precedence over the formula. + +No data path supplies points yet. The datasets store categories, not +points, so points play no part in microsimulation. The companion PR, [policyengine-uk-data #408](https://github.com/PolicyEngine/policyengine-uk-data/pull/408) (points imputed from the observed categories), was closed in June 2026 in the uk-data archive clean-up, with PIP points derivation to move into Microcosm UK's source stages ([microcosm #145](https://github.com/PolicyEngine/microcosm/issues/145)) -once the input variables exist here. +once the input variables existed here. Microcosm #145 closed on 16 October +2026 with that item dropped, because the variables did not exist yet. ``` ## What PIP scoring actually is @@ -40,7 +49,7 @@ Security (Personal Independence Payment) Regulations 2013): - **Part 2: Daily living activities**: ten activities (preparing food, taking nutrition, managing therapy, washing and bathing, managing toilet needs, dressing and undressing, communicating verbally, - reading and understanding signs and symbols, engaging with other + reading and understanding signs, symbols and words, engaging with other people face-to-face, making budgeting decisions). Each activity has a set of descriptors with point values from 0 up to 12; the top value varies by activity. @@ -66,10 +75,12 @@ For population-aggregate simulation, the categorical bands are sufficient because: - the FRS doesn't capture activity-level point scores at all, -- DWP's published caseload reports band counts (standard / enhanced) - rather than activity scores, -- the data (both the certified Enhanced FRS from policyengine-uk-data - and Microcosm UK, which is being built to replace it) sets +- DWP's regular PIP statistics report the caseload by award level + (standard / enhanced) rather than activity scores; per-activity figures + appear only in one-off publications such as the evidence-pack tables + cited under Phase 3, +- the data (both Microcosm UK, the certified default since October 2026, + and the Enhanced FRS from policyengine-uk-data, which it replaced) sets `pip_dl_category` and `pip_m_category` by comparing the weekly PIP daily-living and mobility amounts respondents report in the FRS with the standard and enhanced rates. @@ -90,7 +101,9 @@ Three reform scenarios that the categorical model can't answer: this; the categorical band wouldn't shift unless an analyst manually re-bands a synthetic distribution. 2. **Threshold reform**: changing the band thresholds (for example raising - the standard-rate floor from 8 to 10 points). Same limitation. + the standard-rate floor from 8 to 10 points). The thresholds are now + parameters, so this works for households entered with points, but not in + microsimulation, where the data supply bands rather than points. 3. **Targeted analysis by descriptor**: distributional questions like "how does the enhanced cohort split between mobility activities" need the underlying activity scores. @@ -99,11 +112,9 @@ Three reform scenarios that the categorical model can't answer: ### Phase 1: points-to-category mapping -- Add the threshold parameters under `gov/dwp/pip/`: - - `daily_living/standard_threshold.yaml` (8) - - `daily_living/enhanced_threshold.yaml` (12) - - `mobility/standard_threshold.yaml` (8) - - `mobility/enhanced_threshold.yaml` (12) +- Threshold parameters: done in #1661 as + `gov.dwp.pip.{daily_living,mobility}.points_thresholds.{standard,enhanced}` + (8 and 12). - Add per-activity score inputs as `pip_dl_activity_1_score` ... `pip_dl_activity_10_score`, `pip_m_activity_1_score`, `pip_m_activity_2_score`. Type: integer (Person, YEAR), default 0. @@ -137,9 +148,13 @@ machinery (for reform analysis) get the activity-level surface. - The FRS provides PIP payment amounts but not descriptor selections. The data (Microcosm UK, as the replacement dataset) would need to impute descriptor selections per claimant that are consistent with the observed band. The plausible - approach is to draw uniformly across descriptor combinations that - sum to the observed band, weighted by DWP-published "activities most - commonly scoring above zero" tables. + approach is to draw descriptor combinations whose total falls in the + observed band, weighted by the point-score distributions for + each daily-living activity that DWP published in the + [Pathways to Work evidence pack](https://www.gov.uk/government/consultations/pathways-to-work-reforming-benefits-and-support-to-get-britain-working-green-paper/pathways-to-work-evidence-pack-chapter-2-reforming-the-structure) + (chapter 2, Tables 2.27 and 2.28: working-age, normal-rules claimants + awarded the daily living component in England, Wales and abroad, January + 2025 caseload; the pack has no per-activity mobility table). - This phase is the heaviest and is only required to make Phase 2 reforms work on the calibrated microsim. For the household calculator, users can input activity scores directly. @@ -148,7 +163,8 @@ machinery (for reform analysis) get the activity-level surface. - Should the score inputs default to `None` (sentinel) or 0? Sentinel is cleaner when the points machinery is opt-in via the - `use_point_system` flag. + `use_point_system` flag. (#1661's total points inputs use -1 for "not + assessed".) - DWP's PIP review process distinguishes "ongoing" vs "fixed-term" awards. Should the model expose the award duration, or treat all current awards as ongoing? (Current model assumes ongoing.) @@ -164,7 +180,7 @@ machinery (for reform analysis) get the activity-level surface. - DWP, [Modernising support for independent living: the health and disability green paper (April 2024)][green-paper]: proposals on the descriptors. - DWP, [Pathways to Work: Reforming Benefits and Support to Get Britain Working (March 2025)](https://www.gov.uk/government/consultations/pathways-to-work-reforming-benefits-and-support-to-get-britain-working-green-paper): the four-point proposal. - Issue: [#1069](https://github.com/PolicyEngine/policyengine-uk/issues/1069). -- Existing model: `pip_dl_category`, `pip_m_category`, `pip_dl`, `pip_m`. +- Existing model: `pip_dl_category`, `pip_m_category`, `pip_dl_points`, `pip_m_points`, `pip_dl`, `pip_m`, and the thresholds under `gov.dwp.pip.{daily_living,mobility}.points_thresholds`. [si-377]: https://www.legislation.gov.uk/uksi/2013/377 [assess-guide]: https://www.gov.uk/government/publications/personal-independence-payment-assessment-guide-for-assessment-providers diff --git a/docs/book/programs/gov/hmrc/firm-vat.md b/docs/book/programs/gov/hmrc/firm-vat.md index 9ae9e79977..b06be28580 100644 --- a/docs/book/programs/gov/hmrc/firm-vat.md +++ b/docs/book/programs/gov/hmrc/firm-vat.md @@ -7,9 +7,10 @@ consumption*: the `vat` variable applies `gov.hmrc.vat.standard_rate` and spending (`full_rate_vat_consumption`, `reduced_rate_vat_consumption`), then grosses the result up to VAT receipts. There is no business side. This page sets out the scope for adding firm-level VAT and a `Firm` entity, tracked in -[#1320](https://github.com/PolicyEngine/policyengine-uk/issues/1320) and -[#1333](https://github.com/PolicyEngine/policyengine-uk/issues/1333). Both -issues are still open (October 2026). +[#1320](https://github.com/PolicyEngine/policyengine-uk/issues/1320), which is +still open (October 2026). The companion analysis issue, +[#1333](https://github.com/PolicyEngine/policyengine-uk/issues/1333), was +closed on 20 October 2026 as a duplicate of #1320. ``` ## Why firm-level VAT? @@ -84,17 +85,27 @@ Firm = build_entity( "key": "owner", "plural": "owners", "label": "Owner", - "doc": "Person or household that owns the firm.", + "doc": "Person who owns the firm.", } ], - containing_entities=["state"], + containing_entities=[], ) ``` The `containing_entities` set deliberately does *not* include `household` because incorporated firms have no household; the link for sole traders is via an explicit `firm_owning_household_id` input -variable rather than entity nesting. +variable rather than entity nesting. (The existing `Household` and +`BenUnit` entities list a `"state"` containing entity, but PolicyEngine UK +has no `state` entity, so the firm should not copy it.) In policyengine-core +only people can be members of a group entity, so the owner role holds people. +Core also places each person in exactly one instance of every group entity, +so a person cannot own two firms and everyone belongs to some firm: a +situation that lists firms gives anyone it leaves out a one-person firm, one +that lists households but no firms puts all its people in a single firm, and +a dataset build must assign every person a firm. The design therefore needs +a way to represent people who own no firm and incorporated firms with no +owner in the data. ### Variables @@ -144,27 +155,25 @@ component sits against our current data: | Component | Public source | FRS coverage | Datalab source | |-----------|---------------|--------------|----------------| -| Firm turnover distribution | ONS *Business Demography* | sole-trader / partnership turnover via FRS self-employment income | HMRC Datalab: anonymised VAT returns | -| Industry sector | SIC codes from ONS *Inter-Departmental Business Register (IDBR)* | not in FRS | HMRC Datalab | +| Firm turnover distribution | ONS *UK business: activity, size and location* | sole-trader / partnership profit (not turnover) via FRS self-employment income | HMRC Datalab: anonymised VAT returns | +| Industry sector | SIC codes from ONS *Inter-Departmental Business Register (IDBR)* | main-job SIC 2007 division only (`sic_industry_division`); no firm-level sector | HMRC Datalab | | VAT registration share | HMRC *VAT statistics* (annual) | not in FRS | HMRC Datalab | | Input VAT shares | ONS *Supply and Use tables* | not applicable | HMRC Datalab | ### Proposed imputation approach -1. Build a **synthetic firm panel** weighted to ONS Business Demography - counts by SIC sector and turnover band. +1. Build a **synthetic firm panel** weighted to ONS *UK business: + activity, size and location* counts by SIC sector and turnover band. 2. Attach to each firm a registration status, supply split, and intermediate-consumption ratio drawn from HMRC VAT statistics and ONS Supply and Use tables. 3. Calibrate the panel so that summed `firm_net_vat_liability` matches - the HMRC VAT receipts outturn for the year. This mirrors the - second-stage imputation pattern discussed in - [#1621](https://github.com/PolicyEngine/policyengine-uk/issues/1621). + the HMRC VAT receipts outturn for the year. 4. For sole traders and partnerships, link the synthetic firm row back to the FRS household that owns it via self-employment income. -The companion repo [`policyengine/uk-vatlab`][vatlab] hosts the analysis -files (#1333) used to size each step before they are productionised. +The companion repo [`policyengine/uk-vatlab`][vatlab] hosts the firm-level +VAT analysis files linked from #1333. ## Effort estimate @@ -193,10 +202,10 @@ files (#1333) used to size each step before they are productionised. ## References -- HMRC, [VAT rates](https://www.gov.uk/vat-rates) and [VAT registration thresholds](https://www.gov.uk/vat-registration-thresholds). +- HMRC, [VAT rates](https://www.gov.uk/vat-rates) and [VAT thresholds](https://www.gov.uk/how-vat-works/vat-thresholds). - [Value Added Tax Act 1994](https://www.legislation.gov.uk/ukpga/1994/23/contents): primary statute. - HMRC, [VAT statistics](https://www.gov.uk/government/statistics/value-added-tax-vat-annual-statistics): annual outturn used for calibration. -- ONS, [UK Business: Activity, Size and Location](https://www.ons.gov.uk/businessindustryandtrade/business/activitysizeandlocation) (the IDBR-based business demography release). +- ONS, [UK business: activity, size and location](https://www.ons.gov.uk/businessindustryandtrade/business/activitysizeandlocation/datasets/ukbusinessactivitysizeandlocation): counts of VAT- and/or PAYE-based enterprises by industry (SIC) and turnover band from the Inter-Departmental Business Register (IDBR). - ONS, [Supply and Use tables](https://www.ons.gov.uk/economy/nationalaccounts/supplyandusetables): intermediate consumption shares. - Companion analysis repo: [`policyengine/uk-vatlab`][vatlab] (#1333). diff --git a/docs/book/programs/gov/hmrc/pension-tax-relief.md b/docs/book/programs/gov/hmrc/pension-tax-relief.md index 1762ca37f6..b2636f48a7 100644 --- a/docs/book/programs/gov/hmrc/pension-tax-relief.md +++ b/docs/book/programs/gov/hmrc/pension-tax-relief.md @@ -5,7 +5,7 @@ pension contributions and addresses the confusion reported in [#704](https://github.com/PolicyEngine/policyengine-uk/issues/704) about whether higher-rate taxpayers receive the right amount of relief. #704 is still open (October 2026); the figures below were re-run -on policyengine-uk 2.140.0. +on policyengine-uk 2.148.2. ## What the model does @@ -17,13 +17,14 @@ by their pension contributions, up to the annual limit for relief in - the person's relevant UK earnings ([s. 189(2)](https://www.legislation.gov.uk/ukpga/2004/12/section/189)), - which the model takes as employment income, plus employer-paid Statutory - Sick Pay and Statutory Maternity Pay (`employment_benefits`, listed as - relevant UK earnings in HMRC's + which the model takes as taxable employment income: employment income + plus employer-paid Statutory Sick Pay and Statutory Maternity Pay + (`employment_benefits`, listed as relevant UK earnings in HMRC's [PTM044100](https://www.gov.uk/hmrc-internal-manuals/pensions-tax-manual/ptm044100)), - plus self-employment profit (a trading loss counts as zero, and does not - reduce employment earnings). Statutory Paternity Pay is not yet counted; - open pull request + less allowable employment expenses; plus trading profit after capital + allowances and the trading allowance but before losses (a trading loss + counts as zero, and does not reduce employment earnings). Statutory + Paternity Pay is not yet counted; open pull request [#2368](https://github.com/PolicyEngine/policyengine-uk/pull/2368) would add it to taxable employment pay; - raised towards `gov.hmrc.income_tax.reliefs.pension_contribution.basic_amount` @@ -46,10 +47,10 @@ PolicyEngine does not represent the £720 claim as a payment, and with no taxable income the deduction saves no income tax. Statutory Sick Pay and Statutory Maternity Pay were added to relevant earnings by [#2464](https://github.com/PolicyEngine/policyengine-uk/pull/2464), merged on -19 October 2026. One open pull request would change this section once merged: -[#2488](https://github.com/PolicyEngine/policyengine-uk/pull/2488) would apply -the same relief-at-source-only rule to the pension uplift in adjusted total -income. +19 October 2026. +[#2488](https://github.com/PolicyEngine/policyengine-uk/pull/2488), merged on +20 October 2026, applied the same relief-at-source-only rule to the pension +uplift in adjusted total income. The annual allowance does not limit the relief. `pension_annual_allowance` is the standard annual allowance (£60,000 from 6 April 2023). From 2016-17 it @@ -58,17 +59,26 @@ is tapered only when both the statutory adjusted income threshold income (`pension_annual_allowance_threshold_income`, above the taper level less the standard allowance, £200,000 from 2023-24) are over their limits, by £1 for every £2 of adjusted income above the taper level, -rounded down to whole pounds, to a minimum of £10,000 +rounded down to whole pounds, to a minimum of £10,000 (£4,000 from 2020-21 +to 2022-23) ([Finance Act 2004 s. 228ZA](https://www.legislation.gov.uk/ukpga/2004/12/section/228ZA); parameters under `gov.hmrc.income_tax.allowances.annual_allowance`). See `docs/engineering/pension-annual-allowance.md` for the income conventions. Unused allowance carried forward from the previous three tax years ([s. 228A](https://www.legislation.gov.uk/ukpga/2004/12/section/228A)) is -the input `unused_pension_annual_allowance`, which is added to the allowance -when the charge is worked out. It defaults to nil, because survey data does -not record past pension input amounts, and the model does not derive it from -earlier years. The money purchase annual allowance, the £10,000 limit on -money purchase savings after pensions are first flexibly accessed +`unused_pension_annual_allowance`, which is added to the allowance when the +charge is worked out. It can be supplied for a year. Otherwise, because +survey data does not record past pension input amounts, from 2020-21 the +model estimates it: it assumes that in each of the five previous years the +pension input and the incomes tested for the taper were this year's (times +`gov.simulation.pensions.prior_year_input_ratio`, 1, for the input), deflated +by average earnings growth, and carries forward what the last three years' +allowances leave unused, after each year's excess has used the unused +allowance of the three years before it, earliest first (s. 228A(7)). The +estimate is nil for someone with no pension input this year, and nil before +2020-21. The money purchase +annual allowance, the £10,000 limit on money purchase savings after pensions +are first flexibly accessed ([s. 227ZA](https://www.legislation.gov.uk/ukpga/2004/12/section/227ZA)), is not modelled. @@ -78,16 +88,48 @@ separately attract the annual allowance charge, the excess at the basic, higher and additional rates on the slices that fall in each band when it is added on top of the person's income after allowances ([Finance Act 2004 s. 227](https://www.legislation.gov.uk/ukpga/2004/12/section/227)). +Relief-at-source contributions are not deducted from that income. Under +s. 192(4) they raise the basic and higher rate limits instead, but only for +someone with income that would otherwise be taxed at the higher or additional +rate. Savings from 2017-18, dividends and, from 2027-28, property income have +their own rates and do not count. For a Scottish taxpayer, s. 192(4A) raises +the Scottish basic rate limit and the limits above it where non-savings income +would otherwise be taxed at a Scottish rate above the Scottish basic rate. +Section 227(4C) then applies the raised limits to the charge, and the excess +itself does not count towards that test (s. 227(5)). So an employee on +£45,000 who pays £20,000 gross into a personal pension, with £50,000 of +employer contributions, pays a £2,946 charge on the +£10,000 excess in 2025-26: the excess sits on top of £32,430 of income after +the personal allowance, which is below the £37,700 basic rate limit, so the +limits are not raised and £4,730 of the excess falls in the higher rate +band. Paid through a net pay arrangement, the same contributions would be +deducted first and the charge would be £2,000. HMRC's +[PTM056120](https://www.gov.uk/hmrc-internal-manuals/pensions-tax-manual/ptm056120) +describes the limits as raised by any relief-at-source contribution, without +the s. 192(4) condition, and says the charge is the same whichever way the +contribution is paid; on that reading this charge would also be £2,000. The +model follows the statutory condition. The amount tested against the allowance is `pension_contributions_for_annual_allowance`, which counts member contributions made before age 75, employer contributions and salary sacrifice ([Finance Act 2004 s. 233](https://www.legislation.gov.uk/ukpga/2004/12/section/233)). For example, an employee on £100,000 who pays £70,000 into a personal pension in 2025-26 gets £70,000 of relief and a £2,000 charge on the £10,000 excess, -which falls in the basic rate band. With £10,000 of unused allowance +which falls in the basic rate band. The model's default estimate of unused +allowance for this person is nil, since the estimated input in each earlier +year also exceeds that year's allowance; with £10,000 of unused allowance brought forward, there is no charge. By default the pension input amount is -cash contributions: defined-benefit accrual counts only if supplied through -`pension_contributions_for_annual_allowance`. +cash contributions: `defined_benefit_pension_share` defaults to nil, which +treats every contribution as money purchase. To count defined-benefit +accrual, supply `defined_benefit_pension_input_amount` (for example from a +pension savings statement) together with `defined_benefit_pension_share`, the +share of workplace contributions it replaces; without the share it is added +on top of contributions that still count as money purchase. A share supplied +alone leaves the total unchanged before age 75, since the input amount is +then estimated as those contributions times +`gov.simulation.pensions.defined_benefit_input_per_contribution` (1). +Supplying `pension_contributions_for_annual_allowance` overrides the whole +total. Relieved contributions also reduce `adjusted_net_income`, so they restore personal allowance lost above £100,000, as in law @@ -96,18 +138,19 @@ An employee on £110,000 has a £7,570 personal allowance in 2025-26, and the full £12,570 if they pay £10,000 gross into a personal pension. So the headline behaviour is: **up to the relief limit, pension -contributions reduce taxable income £1-for-£1**. The income-tax saving -that the model reports is the contribution × the person's marginal -rate, plus any personal allowance restored. +contributions reduce taxable income £1-for-£1**. Where the deduction comes +off one kind of income taxed at one rate, as in the worked examples below, +the income-tax saving that the model reports is the contribution × the +person's marginal rate, plus any personal allowance restored. ## Worked examples -The four scenarios reported in #704 (all 2025-26, all self-employed, -single person aged 40, no other income, contributions entered as -`personal_pension_contributions`). The "model" column was re-run in -October 2026 on policyengine-uk 2.140.0: +The four scenarios reported in #704 (all self-employed, no other income), +re-run for 2025-26 for a single person aged 40 with contributions entered as +`personal_pension_contributions`; #704 gives no year or age. The "model" +column was re-run in October 2026 on policyengine-uk 2.148.2: -| ID | Self-employed income | Pension contribution | Expected income-tax change | What the model returns | Why | +| ID | Self-employed income | Pension contribution | Total relief as an income-tax change | What the model returns | Why | |-------|---------------------|----------------------|-----------------------------|------------------------|-----| | 11800 | £30,000 | £0 | baseline | baseline | – | | 11799 | £30,000 | £3,000 | -£600 | -£600 | £3k contribution × 20% basic rate | @@ -116,45 +159,71 @@ October 2026 on policyengine-uk 2.140.0: This is exactly the behaviour reported. The user's expectation that higher-rate cases should only see -£1,200 (£6k × 20%) corresponds to a -misunderstanding of how UK pension tax relief works in aggregate: the -total tax relief is full marginal rate, even though it is paid in two +misunderstanding of how UK pension tax relief works in aggregate: for +someone like this, whose taxable income is at least the contribution and is +all taxed at the basic, higher or additional rate, the total tax relief is +the full marginal rate, even though it is paid in two parts in the real world (relief at source plus higher-rate top-up via Self Assessment). ## Relief at source vs marginal-rate relief The user-facing UK pension framework has two parallel mechanisms that -add up to the same total relief: +together make up the total relief: 1. **Relief at source (RAS)**: the pension provider claims the basic-rate portion (20%) directly from HMRC. The contributor pays from net income and the pension grows by the gross amount. For a basic-rate taxpayer, this is the *entire* relief. -2. **Higher-rate top-up**: for taxpayers at higher or additional - rates, the extra band difference (20% for higher rate, 25% for - additional) is reclaimed via Self Assessment. The mechanism is - technically a **basic-rate-band extension**: HMRC widens the basic - rate band by the gross contribution, pushing income that would - otherwise be taxed at the higher rate down into the basic band. - -PolicyEngine computes the **total** relief (both mechanisms together) -because it operates on annual aggregates rather than walking the -cash-flow steps. That matches HMRC's published statistics on the -*fiscal cost of pension tax relief*, which also report the total -across both mechanisms. +2. **Higher-rate top-up**: for taxpayers with income taxed at the higher + or additional rate itself (not at the savings, dividend or property + rates; see above), the extra relief (in England, Wales and Northern + Ireland, 20% for higher rate and 25% for additional rate) is claimed + through Self Assessment or directly from HMRC. The mechanism is technically a + **rate-band extension**: the basic rate limit and the higher rate + limit are both raised by the gross contribution + ([Finance Act 2004 s. 192(4)](https://www.legislation.gov.uk/ukpga/2004/12/section/192)), + pushing income that would otherwise be taxed at the higher or + additional rate down a band. + +PolicyEngine computes the **total** relief (both mechanisms together) as a +reduction in income tax, because it operates on annual aggregates rather +than walking the cash-flow steps. It gives less than the law where +basic-rate relief at source is worth more than the tax the deduction saves: +on contributions above the person's taxable income, such as the £720 on a +non-taxpayer's £3,600 contribution, and on contributions that come off +income taxed below 20%, such as dividends at the dividend ordinary rate or a +Scottish taxpayer's starter rate band. The provider still claims 20% in those +cases (s. 192(1)-(3)), but the model has less or no tax to reduce. Where +the only income that would be taxed above the basic rate limit is savings +(from 2017-18), dividends or (from 2027-28) property income, s. 192(4) raises +no limits, so the relief in law is the 20% the provider claims, while the +model's deduction saves tax at the rates of the income it removes. The model +can then give more or less than the law. An employee on £15,000 with £60,000 +of dividends who pays £10,000 gross into a personal pension saves £3,648 of +income tax in the model in 2025-26, against £2,000 of relief at source in law +(raising the limits regardless of the condition would give £4,500: the +£2,000 plus £2,500 less tax on dividends moved from the 33.75% upper rate to +the 8.75% ordinary rate). With £12,000 of earnings and £40,000 of dividends, +the same contribution saves £1,308 in the model, below the £2,000 in law. +Otherwise this matches HMRC's published +statistics on the *fiscal cost of pension tax relief*, which also report the +total across both mechanisms. If you want to see the user-visible tax bill reduction separately from the pension-provider's RAS claim, you'd need to split the existing relief into two output variables (a basic-rate-portion and a -higher-rate-top-up portion). This is a presentation question rather -than a calculation question: the totals are unchanged. +higher-rate-top-up portion). Outside the cases above where the model's +total differs from the law, this is a presentation question rather than a +calculation question: the totals are unchanged. ## Marginal tax rate display #704 also reported a 0% marginal tax rate where 20% or 40% was expected. The model's `marginal_tax_rate` variable ([`variables/household/marginal_tax_rate.py`](https://github.com/PolicyEngine/policyengine-uk/blob/main/policyengine_uk/variables/household/marginal_tax_rate.py)) -raises each adult's `employment_income` by +raises the `employment_income` of each of the household's two eldest adults +in turn (`gov.simulation.marginal_tax_rate_adults`) by `gov.simulation.marginal_tax_rate_delta` (£1,000, or 0.1% of employment income where that is larger) and measures the share of the rise that does not reach `household_net_income`. Re-run in October 2026, it returns diff --git a/docs/book/programs/gov/hmrc/pensions-comparison-stbm.md b/docs/book/programs/gov/hmrc/pensions-comparison-stbm.md index 67ce942385..3eddd435fa 100644 --- a/docs/book/programs/gov/hmrc/pensions-comparison-stbm.md +++ b/docs/book/programs/gov/hmrc/pensions-comparison-stbm.md @@ -20,23 +20,36 @@ more detail on the PolicyEngine side. | | PolicyEngine UK | STBM | |---|---|---| -| How relief is given | Reduces taxable income by the contribution (`pension_contributions_relief`) | Treats contributions as paid net of basic rate: grosses them up by 1/(1 − basic rate) and widens every rate band above the starting band by the gross amount, as under relief at source | +| How relief is given | Reduces taxable income by the contribution (`pension_contributions_relief`) | Treats contributions as paid net of basic rate: grosses them up by 1/(1 − basic rate) and raises the basic-rate limit and every rate limit above it by the gross amount, as under relief at source. For savings and dividend income it raises every limit, including the starting-rate limit for savings and the top of the 0% dividend band (the dividend allowance), neither of which relief at source raises | | Contributions that get relief | The person's own contributions | Employee contributions, additional voluntary contributions **and employer contributions** | | Basic-rate relief added to the pot | Not shown separately | Recorded separately (`pension_relief_at_source`) | -| Earnings limit | Relief capped at relevant UK earnings: employment income, employer-paid Statutory Sick Pay and Statutory Maternity Pay, and self-employment profit | No earnings cap; £3,600 limit when total income is below it | +| Earnings limit | Relief capped at relevant UK earnings (employment income, employer-paid Statutory Sick Pay and Statutory Maternity Pay, and self-employment profit), or £3,600 for relief-at-source contributions if earnings are lower | No earnings cap; when total income is below a basic-amount parameter the limit is that amount (£10,000 in its 2023-24 to 2026-27 systems, £3,600 before) | | Annual allowance | £60,000 (`pension_annual_allowance`), tapered only when adjusted income is above £260,000 and threshold income is above £200,000, as in FA 2004 s. 228ZA | Tapered on total income above a threshold-income parameter | | Contributions above the allowance | Annual allowance charge on the excess, taxed as the top slice of income (`personal_pension_contributions_tax`) | Relief stops at the allowance; no separate charge | -| Employer contributions in the data | Read from the data (`employer_pension_contributions`) | Imputed when missing from the FRS, using the ONS ASHE employer-contribution bands (`src/Pensions.jl`) | -| Carry-forward of unused allowance | Input `unused_pension_annual_allowance` (FA 2004 s. 228A) adds to the allowance; defaults to nil, as the survey data has no past pension input amounts | Not modelled | +| Employer contributions in the data | An input (`employer_pension_contributions`). Microcosm UK draws it from the ONS ASHE employer-contribution bands (Table P10); the Enhanced FRS sets it to three times employee contributions for survey respondents and imputes it with a quantile regression forest on the high-income records whose incomes it imputes from the SPI | Imputed when missing from the FRS, using the ONS ASHE employer-contribution bands (`src/Pensions.jl`) | +| Carry-forward of unused allowance | `unused_pension_annual_allowance` (FA 2004 s. 228A) adds to the allowance. As the survey data has no past pension input amounts, from 2020-21 the default estimates it from an assumed five-year history (this year's input and taper incomes deflated by average earnings growth); before 2020-21 it is nil | Not modelled | + +The STBM column describes its Scottish systems, its default. In its +rest-of-UK systems from 2023-24, `non_savings_basic_rate` points at the 40% +rate (`params/sys_2023_24_ruk.jl` onwards), so there the gross-up uses 40% +and, for non-savings income, only the higher rate limit (£125,140, where the +additional rate starts) rises, not the basic-rate limit. In every system all +savings and dividend limits rise. ## Where the two models differ ### 1. Reducing taxable income versus widening the bands -For an income-tax payer whose contribution stays inside one band, the two -methods give the same total relief. They can differ when the person pays -little or no income tax: STBM still records the basic-rate addition to the -pot, which PolicyEngine UK does not show. +In STBM's Scottish systems, for an income-tax payer with only non-savings +income whose contribution stays inside one band above the Scottish starter +rate, the two methods give the same total relief. They can differ when the +person pays little or no income tax: STBM still records the basic-rate +addition to the pot, which PolicyEngine UK does not show. They also differ +when dividends exceed the dividend allowance, because STBM widens the 0% +dividend band by the gross contribution and PolicyEngine UK does not. +Savings income in the starting-rate band does not make them differ: that +band shrinks as taxable non-savings income rises, so PolicyEngine UK's +deduction frees as much of it as STBM's higher starting-rate limit. **The personal allowance taper.** In law, gross pension contributions reduce adjusted net income @@ -61,7 +74,7 @@ employee's bands further than the law does. The legal taper ([FA 2004 s. 228ZA](https://www.legislation.gov.uk/ukpga/2004/12/section/228ZA)) uses "adjusted income", which adds employer contributions, and applies only -when "threshold income" is also above £200,000 (from 2023-24). +when "threshold income" is also above £200,000 (from 2020-21). - PolicyEngine UK applies both tests (`pension_annual_allowance_adjusted_income` and @@ -70,8 +83,9 @@ when "threshold income" is also above £200,000 (from 2023-24). income of £300,000 but threshold income of £200,000, so keeps the full £60,000 allowance. Its limits: by default the pension input amount is cash contributions, so defined-benefit accrual is counted only if supplied - as an input, and all salary sacrifice is treated as made on or after - 9 July 2015. + as an input, and salary sacrifice is treated as made on or after + 9 July 2015 unless `pension_contributions_via_salary_sacrifice_pre_2015` is + supplied. - STBM tapers on total income above a threshold parameter. Both affect only very high earners. @@ -80,7 +94,11 @@ PolicyEngine UK does not model the money purchase annual allowance, the £10,000 limit on money purchase savings after pensions are first flexibly accessed ([FA 2004 s. 227ZA](https://www.legislation.gov.uk/ukpga/2004/12/section/227ZA)). -It always uses the default chargeable amount. +Its annual allowance charge is always on the default chargeable amount, +the pension input amount above the annual allowance (s. 227ZA(3)), never the +alternative chargeable amount (s. 227B), which replaces it when the money +purchase annual allowance is exceeded and the alternative amount is the +larger (s. 227ZA(1)). ### 4. Lifetime and lump sum allowances @@ -92,8 +110,9 @@ are drawn, not to contributions. ## What neither model does - Working out carry-forward of unused annual allowance from the previous - three years. PolicyEngine UK accepts the result as an input but does not - derive it from past years. + three years' actual pension input. PolicyEngine UK accepts it as an input + and otherwise estimates it from an assumed history, not from recorded past + years. - Behavioural responses: contributions are held fixed when tax rates change. - Decumulation: crystallisation, the tax-free lump sum, and annuity versus drawdown choices. diff --git a/docs/book/programs/gov/ofgem/warm-home-discount-plan.md b/docs/book/programs/gov/ofgem/warm-home-discount-plan.md index 151871ce26..7f179f7833 100644 --- a/docs/book/programs/gov/ofgem/warm-home-discount-plan.md +++ b/docs/book/programs/gov/ofgem/warm-home-discount-plan.md @@ -3,7 +3,9 @@ ```{warning} **Not yet modelled.** The Warm Home Discount (WHD) is a £150 rebate off a household's electricity bill each winter, paid by energy suppliers -under schemes that Ofgem administers. PolicyEngine UK doesn't model it. +under schemes that Ofgem administers. PolicyEngine UK doesn't model it; +draft [#2504](https://github.com/PolicyEngine/policyengine-uk/pull/2504) +implements Phase 1 below. This page sets out the proposed scope, tracked in [#502](https://github.com/PolicyEngine/policyengine-uk/issues/502). Updated October 2026: in England and Wales eligibility is now based on @@ -21,13 +23,19 @@ They replace the [2022 Regulations](https://www.legislation.gov.uk/uksi/2022/772/contents) (SI 2022/772), which applied to both England and Wales. -Under the 2022 Regulations, the means-tested group had to pass a -"high cost to heat" test based on property data. For winter 2025-26 -that test was removed, so every household on a qualifying means-tested +Under the eligibility statements made under the 2022 Regulations, the +means-tested group (Core Group 2) had to pass a "high cost to heat" test +based on property data. For winter 2025-26 the eligibility statement +removed that test, so every household on a qualifying means-tested benefit became eligible. The government estimated this would bring -2.7 million more households into the scheme, taking it to more than -6 million households in Great Britain -([government response, June 2025](https://assets.publishing.service.gov.uk/media/6852e6e9679778c74ec15e82/expanding-the-warm-home-discount-scheme-2025-to-2026-government-response.pdf)). +2.7 million more households into the scheme +([government response, June 2025](https://assets.publishing.service.gov.uk/media/6852e6e9679778c74ec15e82/expanding-the-warm-home-discount-scheme-2025-to-2026-government-response.pdf)), +taking it to around 6.1 million households in Great Britain +([impact assessment](https://assets.publishing.service.gov.uk/media/68529908ff16d05c5e6aa678/expanding-the-warm-home-discount-scheme-2025-to-2026-impact-assessment.pdf)). +For winter 2025-26 (qualifying date 24 August 2025) the qualifying +benefits also included income-based Jobseeker's Allowance and Income +Support, with Pension Credit Guarantee Credit in a separate core group +([eligibility statement, 2025 to 2026](https://www.gov.uk/government/publications/warm-home-discount-eligibility-statement-england-and-wales-2025-to-2026-scheme-year/warm-home-discount-eligibility-statement-england-and-wales-2025-to-2026)). For winter 2026-27, GOV.UK ([eligibility in England and Wales](https://www.gov.uk/the-warm-home-discount-scheme/if-you-live-in-england-and-wales)) @@ -41,7 +49,9 @@ their partner gets one of: - income-related Employment and Support Allowance; - Pension Credit. -The rebate is applied automatically. +Most eligible households get the rebate automatically through data +matching; the rest are sent a letter and must call the helpline by +26 February 2027 to confirm their details. ### Scotland @@ -53,12 +63,18 @@ The scheme runs under separate Scottish regulations. (SI 2022/1073). Both 2026 Regulations set the rebate at £150. For winter 2026-27 ([eligibility in Scotland](https://www.gov.uk/the-warm-home-discount-scheme/if-you-live-in-scotland)): -- **Core group, automatic:** Pension Credit; Universal Credit with an - extra amount for a disability or health condition; income-related ESA - with a disability or pensioner premium; or a Support for Mortgage - Interest loan with one of those premiums. Also people responsible for - a child under 5 who are unemployed and on Universal Credit or - income-related ESA, or have a Support for Mortgage Interest loan. +- **Core group, automatic.** Under Schedule 1 to the Regulations: + anyone on Pension Credit; Universal Credit claimants with the disabled + child addition, or who are not in work and have a child under 5 or + limited capability for work (with or without work-related activity); + income-related ESA claimants with a child under 5, a pensioner, severe + disability or enhanced disability premium, or limited capability for + work or work-related activity; Income Support or income-based JSA + claimants with a child under 5 or a pensioner, higher pensioner, + disability, severe disability, enhanced disability or disabled child + premium; and people receiving loans for mortgage interest through + Pension Credit, or through one of those legacy benefits with a child + under 5 or a qualifying premium. GOV.UK's summary is narrower. - **Broader group, by application** to the energy supplier, which can set its own extra criteria. @@ -76,22 +92,29 @@ extends to Great Britain only. Now a rule on benefit receipt: - Parameter `gov.ofgem.warm_home_discount.amount`: £150, the - "prescribed rebate" in both sets of 2026 Regulations (£140 before - winter 2022-23). -- Parameter listing the qualifying benefits, so a reform can widen or - narrow the list. + "prescribed rebate" in both sets of 2026 Regulations (£150 since + winter 2022-23; under the Warm Home Discount Regulations 2011 it was + £140 from 2014-15 to 2021-22, after £120, £130 and £135 in the first + three scheme years). +- Parameter listing the qualifying benefits by scheme year, since the + 2025-26 and 2026-27 lists differ, so a reform can widen or narrow the + list. - Variable `warm_home_discount` (Household, YEAR): the amount if the household is in England or Wales and any benefit unit receives a qualifying benefit. Read receipt from the modelled benefits, for example `universal_credit`, `housing_benefit`, `esa_income` and - `pension_credit`, so that take-up carries through. -- No separate take-up flag, since the rebate is paid without a claim. + `pension_credit`, so that take-up carries through (`esa_income` is + reported receipt, screened by capital and work tests). +- No separate take-up flag, since most rebates are paid through data + matching without a claim (a minority must phone the helpline). ### Phase 2: Scotland core group -The core group can be built from the same benefit variables plus the -Universal Credit disability elements, the ESA premiums and the age of -the youngest child. +The core group can be built from the same benefit variables plus +Universal Credit's disabled child and limited capability for work +assessments, whether the claimant is in work, whether there is a child +under 5, and legacy-benefit premiums. ESA's pensioner premium and loans +for mortgage interest have no variables yet. ### Phase 3: Scotland broader group @@ -100,8 +123,8 @@ published recipient numbers for Scotland, on top of the eligible group. ## Data and calibration -- Ofgem's [Warm Home Discount pages](https://www.ofgem.gov.uk/environmental-and-social-schemes/warm-home-discount-whd): - recipients and spend by group and country. +- Ofgem's [Warm Home Discount reports and data](https://www.ofgem.gov.uk/environmental-and-social-schemes/warm-home-discount-whd/warm-home-discount-whd-reports-and-data) + and annual reports: recipients and spend by group and country. - DWP benefit caseloads, already used to calibrate the qualifying benefits. - The 2025-26 impact assessment and government response give the @@ -110,10 +133,11 @@ published recipient numbers for Scotland, on top of the eligible group. ## Open questions -- **Timing.** The scheme year runs from October to March, with the - qualifying date in August. A model year (April to March) contains one - winter's rebate, so pay the rebate in the year that contains the - winter. +- **Timing.** The scheme year runs from April to March, but rebates are + paid in the winter: letters go out from late October and suppliers + apply the discount by 31 March, with the qualifying date in August. A + model year (6 April to 5 April) contains one scheme year's rebate, so + pay it in the model year that contains the winter. - **Who counts as the household.** The rebate attaches to an electricity account. In multi-benefit-unit households, decide whether one qualifying benefit unit is enough (simplest, and likely right for @@ -130,7 +154,9 @@ published recipient numbers for Scotland, on top of the eligible group. - [The Warm Home Discount (England and Wales) Regulations 2022 (SI 2022/772)](https://www.legislation.gov.uk/uksi/2022/772/contents). - [The Warm Home Discount (Scotland) Regulations 2026 (SI 2026/488)](https://www.legislation.gov.uk/uksi/2026/488/contents). - [The Warm Home Discount (Scotland) Regulations 2022 (SI 2022/1073)](https://www.legislation.gov.uk/uksi/2022/1073/contents). -- DESNZ, [Expanding the Warm Home Discount scheme, 2025 to 2026: government response](https://assets.publishing.service.gov.uk/media/6852e6e9679778c74ec15e82/expanding-the-warm-home-discount-scheme-2025-to-2026-government-response.pdf). +- DESNZ, [Expanding the Warm Home Discount scheme, 2025 to 2026: government response](https://assets.publishing.service.gov.uk/media/6852e6e9679778c74ec15e82/expanding-the-warm-home-discount-scheme-2025-to-2026-government-response.pdf) and [impact assessment](https://assets.publishing.service.gov.uk/media/68529908ff16d05c5e6aa678/expanding-the-warm-home-discount-scheme-2025-to-2026-impact-assessment.pdf). +- DESNZ, [Warm Home Discount eligibility statement, England and Wales, 2025 to 2026](https://www.gov.uk/government/publications/warm-home-discount-eligibility-statement-england-and-wales-2025-to-2026-scheme-year/warm-home-discount-eligibility-statement-england-and-wales-2025-to-2026). +- [The Warm Home Discount Regulations 2011 (SI 2011/1033)](https://www.legislation.gov.uk/uksi/2011/1033/regulation/2): earlier rebate amounts. - GOV.UK, [Warm Home Discount Scheme](https://www.gov.uk/the-warm-home-discount-scheme). -- Ofgem, [Warm Home Discount scheme pages and reports](https://www.ofgem.gov.uk/environmental-and-social-schemes/warm-home-discount-whd). +- Ofgem, [Warm Home Discount scheme pages](https://www.ofgem.gov.uk/environmental-and-social-schemes/warm-home-discount-whd) and [reports and data](https://www.ofgem.gov.uk/environmental-and-social-schemes/warm-home-discount-whd/warm-home-discount-whd-reports-and-data). - Issue: [#502](https://github.com/PolicyEngine/policyengine-uk/issues/502). diff --git a/docs/book/validation/policyengine-py-household-impact.md b/docs/book/validation/policyengine-py-household-impact.md index 1926c110cc..3cede5ab7e 100644 --- a/docs/book/validation/policyengine-py-household-impact.md +++ b/docs/book/validation/policyengine-py-household-impact.md @@ -2,26 +2,57 @@ ```{note} **Resolved.** [#1628](https://github.com/PolicyEngine/policyengine-uk/issues/1628) -reported that two `policyengine.py` household-impact integration tests -started asserting `0.0` for expected-positive tax and child-benefit -values after bumping the UK extra to `policyengine-uk==2.88.0`. This -page documents the diagnosis from the PolicyEngine UK side and the -guidance for the upstream `policyengine.py` repo. It is not a model -bug: the UK model produces the expected non-zero values when called -correctly. - -Re-checked in October 2026 on policyengine-uk 2.123.7: both scenarios -below still return £7,486 and £1,354.60. The upstream reproducer -(policyengine.py#280) was closed without merging. policyengine.py has -since replaced the `TestUKHouseholdImpact` tests with -`TestUKCalculateHousehold` in `tests/test_household_impact.py`, which -passes an explicit `year` to every case and pins policyengine-uk -2.125.1. #1628 can be closed once those tests are confirmed green. +reported that two `policyengine.py` household-impact integration tests, +which assert that income tax and child benefit are positive, got `0.0` +after +[policyengine.py#280](https://github.com/PolicyEngine/policyengine.py/pull/280) +moved the UK extra from `policyengine-uk==2.74.0` to `==2.88.0` and +raised the `policyengine_core` floor from 3.23.6 to 3.24.1. + +It was not a PolicyEngine UK bug. policyengine-core 3.24.0 to 3.24.3 +discarded inputs that a country package set before applying a reform +during simulation construction, the pattern `policyengine_uk.Simulation` +uses. +[policyengine-core#475](https://github.com/PolicyEngine/policyengine-core/pull/475) +fixed this in core 3.24.4, and +[policyengine.py#283](https://github.com/PolicyEngine/policyengine.py/pull/283) +raised policyengine.py's core floor to 3.25.0, both on 18 April 2026. + +On 22 October 2026 all eight UK cases in policyengine.py's +`TestUKCalculateHousehold` (`tests/test_household_impact.py`, which +replaced `TestUKHouseholdImpact` in the v4 launch) passed on its main CI +with policyengine-uk 2.132.6, so #1628 can be closed. ``` +## What went wrong + +policyengine-core's changelog for 3.24.4 describes the bug: its cache +invalidation after `apply_reform` "wiped every variable's +`_memory_storage._arrays`, which also wiped user-provided dataset inputs +loaded via `set_input`. Country-package subclasses calling `set_input` +during construction and then applying a structural reform (the +`policyengine-uk` pattern) silently lost their datasets." In the failing +tests, inputs such as age, `employment_income` and `would_claim_*` were +gone before any calculation ran, so tax and benefits came out as zero. + +Two other explanations were considered and do not fit the harness code +at policyengine.py#280's head (`6772ecc`): + +- **The dataset-required change** (policyengine-uk 2.76.0, where + `Simulation()` with no situation, no dataset and no + `POLICYENGINE_UK_DEFAULT_DATASET` raises a `ValueError`). The harness + passed an explicit dataset (`Microsimulation(dataset=...)` in + `src/policyengine/tax_benefit_models/uk/model.py`), so it never reached + that path. +- **The default period** (`default_input_period` and + `default_calculation_period`, 2025 in both 2.74.0 and 2.88.0). The + harness passed `period=` to every calculation, and the failing tests + built 2026 households. + ## Reproduction from the UK side -Running the two scenarios directly against `policyengine-uk` main: +Two comparable scenarios, run directly against `policyengine-uk` main for +2025 (the failing tests used 2026 and a one-adult, two-child family): ```python from policyengine_uk import Simulation @@ -53,67 +84,20 @@ print("child_benefit:", sim.calculate("child_benefit", YEAR).sum()) # -> 1354.60 ``` -Both return the expected non-zero values across the range 2022-2025 -(the income-tax figure is invariant because the personal allowance is -£12,570 throughout this window). So **the underlying model is healthy**. - -## What changed between 2.74 and 2.88 - -The change in that window that most plausibly explains the upstream -failure is the **dataset-required** invariant, added in 2.76.0 -("Stop defaulting to private dataset"): `Simulation()` with no -`situation=`, no explicit `dataset=` and no -`POLICYENGINE_UK_DEFAULT_DATASET` environment variable raises -`ValueError: Simulation() requires an explicit dataset when no -situation is provided`. If the `policyengine.py` harness was catching -that error and falling through to a degenerate code path (a -zero-filled `Microsimulation`), the tests would see zeros. - -The **default-period** contract is a second thing to rule out, although -it did not change in that window (both 2.74.0 and 2.88.0 default to -2025): - -```python -# policyengine_uk/simulation.py -class Simulation(CoreSimulation): - default_input_period: int = 2025 - default_calculation_period: int = 2025 -``` - -The failing tests built 2024 households. Any calculation that relied on -the default period would compute 2025, and any tax or benefit input -populated only for 2024 would be missing for that year. - -## Guidance for the upstream fix - -The `policyengine.py` household-impact harness should: - -1. Pass an **explicit `period`** to every `calculate(...)` call - matching the year encoded in the situation YAML, rather than relying - on the model default. -2. Build single-household cases with **`Simulation(situation=...)`** - rather than a dataset-backed simulation. They don't need a dataset - and shouldn't bring in the default-dataset env-var dependency. -3. **Assert against the year encoded in the test fixture**, not the - model's `default_input_period`. The fixture-encoded year is what the - test author intended. - -The two failing tests should produce ~£7,486 income tax for a -£50k-earning single adult and ~£1,354.60 of child benefit for a -couple-with-one-child benefit unit at 2025 rates. - -## Why this is documented in `policyengine-uk` rather than fixed there - -The actual fix lives in `policyengine.py`'s test harness and is -explicitly out of scope for this repo (sister-repo edits aren't -made from `policyengine-uk`). This page exists so the next reviewer -who sees `#1628` and is unfamiliar with the policyengine.py setup has -a one-stop diagnosis instead of having to re-derive it. +Re-checked on 23 October 2026 on policyengine-uk 2.148.2 (policyengine-core +3.33.2): both scenarios return £7,486 and £1,354.60. Both also return +non-zero values for every year from 2022 to 2025. The income-tax figure is +the same each year because the personal allowance (£12,570), the basic +rate limit (£37,700) and the 20% basic rate are unchanged from 2021-22; +child benefit follows the eldest-child rate (£1,133.60 in 2022, £1,248.00 +in 2023, £1,331.20 in 2024). ## References - Issue: [#1628](https://github.com/PolicyEngine/policyengine-uk/issues/1628). -- Reproducer: [`PolicyEngine/policyengine.py#280`](https://github.com/PolicyEngine/policyengine.py/pull/280). +- Where the failure showed: [`PolicyEngine/policyengine.py#280`](https://github.com/PolicyEngine/policyengine.py/pull/280) (closed without merging, superseded by #282). +- The fix: [`PolicyEngine/policyengine-core#475`](https://github.com/PolicyEngine/policyengine-core/pull/475) (core 3.24.4) and [`PolicyEngine/policyengine.py#283`](https://github.com/PolicyEngine/policyengine.py/pull/283) (core floor 3.25.0). - Default-period contract: [`policyengine_uk/simulation.py`](https://github.com/PolicyEngine/policyengine-uk/blob/main/policyengine_uk/simulation.py) (`default_input_period`, `default_calculation_period`). - Default-dataset env var contract: `POLICYENGINE_UK_DEFAULT_DATASET`, - defined in [`policyengine_uk/tax_benefit_system.py`](https://github.com/PolicyEngine/policyengine-uk/blob/main/policyengine_uk/tax_benefit_system.py). + defined in [`policyengine_uk/tax_benefit_system.py`](https://github.com/PolicyEngine/policyengine-uk/blob/main/policyengine_uk/tax_benefit_system.py) + and read in `policyengine_uk/simulation.py`. diff --git a/docs/book/validation/reform-impacts-dashboard-plan.md b/docs/book/validation/reform-impacts-dashboard-plan.md index 316ace197c..529b198808 100644 --- a/docs/book/validation/reform-impacts-dashboard-plan.md +++ b/docs/book/validation/reform-impacts-dashboard-plan.md @@ -4,53 +4,86 @@ **Planning page.** Tracks [#806](https://github.com/PolicyEngine/policyengine-uk/issues/806), which asks for a dashboard tracking the budgetary impact of -high-importance reforms over time and across model / data versions. -The repo already has the **fixtures**; what's missing is a published -view onto them. Updated October 2026: still not built, and the planned -dataset move to Microcosm UK (below) makes it more useful. +high-importance reforms over time and across model versions. +The repo already has the **fixtures**, and each one now records its +earlier values with the model and data that produced them; what's +missing is a published view onto them. Updated October 2026: still not +built, and the planned dataset move to Microcosm UK (below) makes it more +useful. ``` ## What exists today -[`policyengine_uk/tests/microsimulation/reforms_config.yaml`](../../../policyengine_uk/tests/microsimulation/reforms_config.yaml) -is a list of canonical reform-impact regression fixtures with the -shape: +[`policyengine_uk/tests/microsimulation/reforms_config.yaml`](https://github.com/PolicyEngine/policyengine-uk/blob/main/policyengine_uk/tests/microsimulation/reforms_config.yaml) +names the dataset the fixtures were measured on and lists the canonical +reform-impact regression fixtures. Each fixture keeps its earlier values +under `receipts`, with the model and data that produced each. Abridged, +at policyengine-uk 2.148.2: ```yaml +dataset: + data_version: 1.58.1 + path: enhanced_frs_2024_25.h5 + revision: 2526edbbb687a3b54dbb11272e5389e4b62d2ac2 + sha256: 056e80fd8e9151d78f432a25cde863340baee81304c6216a37fb575ce8563784 +reforms: - name: Raise basic rate by 1pp - expected_impact: 7.8 # £bn fiscal impact at the reference year + expected_impact: 7.1 parameters: gov.hmrc.income_tax.rates.uk[0].rate: 0.21 + receipts: + - impact: 7.8 + data: enhanced_frs_2022_23, unpinned (latest at the time) + model: 2.34.5 (a1832dc77, #1205, 2025-07-11) + # ... four more receipts ... + - impact: 7.115 + data: 1.58.1 enhanced_frs_2024_25 + model: 2.135.7 (merged 7717438bf, main 7ad002f54, 2026-10-18) +# ... - name: Reduce Universal Credit taper rate to 20% - expected_impact: -18.1 + expected_impact: -22.0 tolerance: 3.0 parameters: gov.dwp.universal_credit.means_test.reduction_rate: 0.2 + receipts: + # ... ``` Each fixture is run by -[`tests/microsimulation/test_reform_impacts.py`](../../../policyengine_uk/tests/microsimulation/test_reform_impacts.py), +[`policyengine_uk/tests/microsimulation/test_reform_impacts.py`](https://github.com/PolicyEngine/policyengine-uk/blob/main/policyengine_uk/tests/microsimulation/test_reform_impacts.py), which compares the change in `gov_balance` in 2029 with `expected_impact` (£bn), within `tolerance` (default £1bn). The tests are marked `microsimulation`, so they run in CI's test suite only when a dataset is available: `policyengine_uk/tests/conftest.py` points -`POLICYENGINE_UK_DEFAULT_DATASET` at the Enhanced FRS 2023-24 file -(`enhanced_frs_2023_24.h5@1.40.3`) when a Hugging Face token is set. -The suite requires that data in the merge queue and on main, and skips -these tests on pull requests without credentials. So CI already -catches *unintended* impact changes before they reach main. But the -signal is only -pass or fail, and when a change is intended, someone reruns -[`update_reform_impacts.py`](../../../policyengine_uk/tests/microsimulation/update_reform_impacts.py) -and the old value is overwritten. The fixtures are regression values -taken from the model's own output, not external costings (the file says -so at the top). - -What's missing is a **time series**: the same eight reforms run on -every commit / version, plotted against the published OBR/RF -benchmark for the same reform, with annotations for the relevant -infrastructure changes (model versions, dataset refreshes, EFO -updates). +`POLICYENGINE_UK_DEFAULT_DATASET` at the certified dataset pinned in +`policyengine_uk/tests/certified_dataset.yaml` when a Hugging Face token +is set. That dataset is the Enhanced FRS 2024-25 +(`enhanced_frs_2024_25.h5` from policyengine-uk-data 1.58.1), pinned by +Hugging Face commit and checked against its SHA-256. The suite requires +that data in the merge queue and on main, and skips these tests on pull +requests without credentials. So CI already catches *unintended* impact +changes before they reach main. But the test itself is only pass or fail. +The `Aggregate impact` workflow +(`.github/workflows/aggregate-impact.yaml`) also costs these reforms on +each pull request's base and head for 2025-26 to 2031-32 and posts the +comparison as a pull request comment, but nothing keeps those results +once the pull request is done. When a change is intended, someone reruns +[`update_reform_impacts.py`](https://github.com/PolicyEngine/policyengine-uk/blob/main/policyengine_uk/tests/microsimulation/update_reform_impacts.py), +which overwrites `expected_impact`. The earlier values survive only +because each fixture also keeps a `receipts` list: the script does not +add to it, but a test requires the latest receipt to be on the pinned +dataset and to match `expected_impact`. The fixtures are regression +values taken from the model's own output, not external costings (the +file says so at the top). + +What's missing is a **time series** that keeps itself: the receipts hold +only the values someone chose to record, for 2029, and the +aggregate-impact comments aren't stored. The plan is the same eight +reforms run on every commit or version, plotted against a published +external costing where one exists (for the tax-rate changes, HMRC's +[ready reckoner](https://www.gov.uk/government/statistics/direct-effects-of-illustrative-tax-changes)), +with annotations for the relevant infrastructure changes (model +versions, dataset refreshes, EFO updates). ## Proposed shape @@ -72,17 +105,21 @@ A new entry point under `tests/microsimulation/` (or, better, a GitHub Actions workflow that doesn't run inside the test suite) walks `reforms_config.yaml`, runs each reform, and writes a row per fixture. Schedule: weekly + on every release tag. +(`python -m policyengine_uk.utils.aggregate_impact measure` already does +the costing, for the eight fixtures and two more reforms; the new part is +running it on a schedule and keeping the results.) ### Publication Three publication paths, in order of effort: 1. **Static page in the docs site**: the simplest. Convert the CSV - to a Plotly chart (matching the existing `validation/*.ipynb` - pattern) and render it under `docs/book/validation/`. Updates - when the docs site rebuilds. + to a Plotly chart (as `validation/validation.ipynb` does) and render + it under `docs/book/validation/`. A notebook must be added to + `myst.yml` by name, since only Markdown pages are picked up by + folder. Updates when the docs site rebuilds. 2. **Living JSON endpoint**: push the latest aggregate JSON to a - stable URL (HuggingFace dataset or a `gh-pages` artefact) so the + stable URL (a HuggingFace dataset or a GitHub Pages artefact) so the policyengine.org app can render it without needing the docs build. 3. **policyengine.org dashboard tile**: a tile on the marketing site that shows the latest impact alongside the OBR/RF benchmark. Most @@ -93,9 +130,13 @@ Three publication paths, in order of effort: For each reform: - **Latest observed impact** (£bn, current model). -- **Published external benchmark** (OBR EFO costing where one - exists; RF *No half measures*-style costing as a comparator). Annotate - the publication date. +- **Published external benchmark** where one exists, with its + publication date: HMRC's ready reckoner (*Direct effects of + illustrative tax changes*) for the income tax, National Insurance and + VAT changes, and OBR or Resolution Foundation costings (such as RF's + *No half measures* on the two-child limit) for the policies they + cover. Any published figure must follow the aggregate-impact privacy + rules (`policyengine_uk/utils/aggregate_impact/privacy.py`). - **Time series** of observed impact across the last N model releases. - **Annotations** for each known regime shift: EFO release, dataset refresh, methodology PR. The reforms_config history is the natural @@ -107,7 +148,8 @@ Use the existing `reforms_config.yaml` entries as the v1 dashboard content: 1. Raise basic rate by 1pp. -2. Raise higher rate by 1pp. +2. Raise the higher rate to 42% (the fixture's name says "by 1pp", but + the rate is 40%, so the rise is 2pp). 3. Raise the personal allowance to £13,000 (the fixture's name says "by ~800GBP/year", but the allowance is frozen at £12,570 to 2030-31, so in 2029 the rise is £430). @@ -117,23 +159,26 @@ content: 7. Raise VAT standard rate by 2pp. 8. Raise additional rate by 3pp. -These cover three of the four big revenue heads (income tax, NI, VAT) -plus a UC parameter that matters disproportionately for distributional -analysis. +These cover income tax (four fixtures), National Insurance and VAT, +plus two benefit parameters: the Child Benefit rate for additional +children and the Universal Credit taper. Add to v2 over time as published OBR / RF reforms become available benchmarks: -- Remove the two-child limit (an external comparison is drafted in - [#1724](https://github.com/PolicyEngine/policyengine-uk/pull/1724)). -- Freeze the Personal Allowance for an additional year. +- Reinstate the two-child limit. The baseline removes it from 6 April + 2026, and [child-poverty-tcl.md](child-poverty-tcl.md) compares its + removal with the DWP, Budget 2025 and Resolution Foundation estimates. +- Freeze the Personal Allowance for an additional year (to 2031-32, so + it needs a later reference year than 2029). - 1pp employer NI change. ## Why now The UK data pipeline is moving from the Enhanced FRS to [Microcosm UK](https://github.com/PolicyEngine/microcosm), while the -fixtures still run on the Enhanced FRS 2023-24. Moving the +fixtures still run on the Enhanced FRS (2024-25, from +policyengine-uk-data 1.58.1). Moving the fixtures' dataset will shift every expected impact at once, and model changes to VAT, fuel duty and other bases are landing at the same time. A history that records the dataset alongside each value is the easiest @@ -152,7 +197,7 @@ relevant pages are: - [nowcasting-comparison.md](../assumptions/nowcasting-comparison.md) (PolicyEngine vs RF). - [rf-nowcasting-methodology.md](../assumptions/rf-nowcasting-methodology.md) (RF methodology detail). -- [#1724](https://github.com/PolicyEngine/policyengine-uk/pull/1724) (child poverty against OBR and Resolution Foundation benchmarks, in review). +- [child-poverty-tcl.md](child-poverty-tcl.md) (the child-poverty impact and cost of removing the two-child limit, against DWP, Budget 2025 and Resolution Foundation estimates). ## Open questions @@ -169,6 +214,7 @@ relevant pages are: ## References - Issue: [#806](https://github.com/PolicyEngine/policyengine-uk/issues/806). -- Existing fixture list: [`reforms_config.yaml`](../../../policyengine_uk/tests/microsimulation/reforms_config.yaml). -- Existing regression runner: [`test_reform_impacts.py`](../../../policyengine_uk/tests/microsimulation/test_reform_impacts.py). -- Script that rewrites the expected values: [`update_reform_impacts.py`](../../../policyengine_uk/tests/microsimulation/update_reform_impacts.py). +- Existing fixture list: [`reforms_config.yaml`](https://github.com/PolicyEngine/policyengine-uk/blob/main/policyengine_uk/tests/microsimulation/reforms_config.yaml). +- Existing regression runner: [`test_reform_impacts.py`](https://github.com/PolicyEngine/policyengine-uk/blob/main/policyengine_uk/tests/microsimulation/test_reform_impacts.py). +- Script that rewrites the expected values: [`update_reform_impacts.py`](https://github.com/PolicyEngine/policyengine-uk/blob/main/policyengine_uk/tests/microsimulation/update_reform_impacts.py). +- Per-pull-request costings: [`.github/workflows/aggregate-impact.yaml`](https://github.com/PolicyEngine/policyengine-uk/blob/main/.github/workflows/aggregate-impact.yaml) and [`policyengine_uk/utils/aggregate_impact/`](https://github.com/PolicyEngine/policyengine-uk/tree/main/policyengine_uk/utils/aggregate_impact). diff --git a/docs/book/validation/ubuntu-vs-macos.md b/docs/book/validation/ubuntu-vs-macos.md index 8f2d867ad2..d3a8b4e2cd 100644 --- a/docs/book/validation/ubuntu-vs-macos.md +++ b/docs/book/validation/ubuntu-vs-macos.md @@ -2,65 +2,93 @@ ```{note} Issue [#1209](https://github.com/PolicyEngine/policyengine-uk/issues/1209) -reported that PolicyEngine UK produced different aggregates on -`ubuntu-latest` and `macos-latest` for the same dataset and the same -`policyengine-uk` version. The cause was in parameter uprating. As of -October 2026 the parameter tree is byte-identical across macOS and -Linux, and CI has moved back to Linux. This page records the diagnosis, -the fix and how to check for a regression. +reported that PolicyEngine UK gave different results on `ubuntu-latest` +and `macos-latest` for the same dataset and the same `policyengine-uk` +version. The cause was in parameter uprating. As of October 2026 the +parameter tree is byte-identical across macOS and Linux, and CI has moved +back to Linux. This page records the diagnosis, the fix and how to check +for a regression. ``` ## Symptom -The divergence was first spotted in +This divergence was spotted in [#1205](https://github.com/PolicyEngine/policyengine-uk/pull/1205), -whose results differed between the two runners. A diagnostic -repository that ran the same code on both runners and stored the full -entity dataset as artifacts showed that: +whose results differed between the two runners. It was the second such +case: [#1091](https://github.com/PolicyEngine/policyengine-uk/issues/1091) +(April 2025) had found benefit parameters that uprated on macOS but not +on Ubuntu, and [#1092](https://github.com/PolicyEngine/policyengine-uk/pull/1092) +fixed that one. A diagnostic repository that ran the same code on both +runners and stored the full entity dataset as artifacts showed that: - the input data were identical on both operating systems; - a limited subset of parameters was uprated on macOS but not on - Ubuntu (the parameter-tree dumps are attached to - [#1209](https://github.com/PolicyEngine/policyengine-uk/issues/1209#issuecomment-3062418736)). + Ubuntu. Parameter-tree dumps attached to a + [later comment on #1209](https://github.com/PolicyEngine/policyengine-uk/issues/1209#issuecomment-3062418736) + show 82 Department for Work and Pensions parameters, all uprated + through `gov.benefit_uprating_cpi`, that were uprated on macOS but + kept their set values on Ubuntu. -The effect on weighted aggregates was small, but it broke +No record gives the size of the effect on aggregates, but it broke reproducibility. ## Cause +The dumps show where the trees differed: the 82 parameters differ only +in the main tree, while their `parameters.baseline` copies match. At the +time `gov.benefit_uprating_cpi` held only a 2005 value and was itself +uprated from `gov.obr.consumer_price_index`, so these were chained +upratings (a parameter uprated from an index that is itself uprated), the +case #1092 had found in April 2025. policyengine-core then loaded each +parameter directory in unsorted `os.listdir` order, which can differ +between file systems, and uprating depended on the order in which it +reached the parameters, so a parameter reached before its uprating index +had been extended got no new values. +[policyengine-core#490](https://github.com/PolicyEngine/policyengine-core/issues/490) +later described the same failure in policyengine-us. The order the 2025 +runners actually listed the files in was never recorded, so this is +inferred from the code and the dumps rather than traced. + [#1223](https://github.com/PolicyEngine/policyengine-uk/pull/1223) -(closed without merging) traced it to an **import-time ordering -problem**: the uprating growth-factor table -(`BASELINE_GROWFACTORS = create_policyengine_uprating_factors_table()`) -was computed when its module was imported, before the parameter tree -had been fully processed. Whether the tree was ready by then depended -on the order in which modules and parameter files were loaded, and that -order differed between the two operating systems. +(closed without merging) proposed a different cause: that the uprating +growth-factor table (`BASELINE_GROWFACTORS`) was computed at import time, +before the system was initialised. That table only read parameter values +and, as called (`print_diff=False`), returned the stored +`uprating_growth_factors.csv`, so it could not change the parameter tree. ## How it was resolved - [#1092](https://github.com/PolicyEngine/policyengine-uk/pull/1092) - and [#1207](https://github.com/PolicyEngine/policyengine-uk/pull/1207) - ran the Test job on `macos-latest` in 2025, so that CI matched the - platform that gave the expected results. + (April 2025) added `macos-latest` alongside `ubuntu-latest` to the + pull-request Test job and fixed the first case, #1091. When #1209 + appeared, [#1205](https://github.com/PolicyEngine/policyengine-uk/pull/1205) + dropped `ubuntu-latest` from that job to unblock itself, and + [#1207](https://github.com/PolicyEngine/policyengine-uk/pull/1207) + moved the main-branch Test job to `macos-latest` (both July 2025). - [#1254](https://github.com/PolicyEngine/policyengine-uk/pull/1254) (multi-year datasets, with uprating as a separate step) removed the - import-time `BASELINE_GROWFACTORS` table. Uprating is now applied by - `apply_uprating` in + import-time `BASELINE_GROWFACTORS` table that #1223 had suspected. + Dataset uprating is now applied by `apply_uprating` in [`policyengine_uk/data/economic_assumptions.py`](https://github.com/PolicyEngine/policyengine-uk/blob/main/policyengine_uk/data/economic_assumptions.py). -- Later fixes to the same path: + Parameter uprating is policyengine-core's `uprate_parameters`, called + from `policyengine_uk/tax_benefit_system.py`. +- Later fixes to the dataset uprating path: [#1523](https://github.com/PolicyEngine/policyengine-uk/pull/1523) (no redundant dataset copy in `apply_uprating`), [#1543](https://github.com/PolicyEngine/policyengine-uk/pull/1543) and [#1544](https://github.com/PolicyEngine/policyengine-uk/pull/1544) (`uprate_rent` lookups). - policyengine-core now sorts each directory listing when it loads the - parameter tree, so the load order no longer depends on the file - system. + parameter tree, and uprates each parameter after the index it is + uprated from + ([policyengine-core#491](https://github.com/PolicyEngine/policyengine-core/pull/491), + May 2026), so neither load order nor the file system can change + uprated values. On 3 October 2026 the Test job moved from macOS to `ubuntu-24.04-arm` (commit -[`67c63861`](https://github.com/PolicyEngine/policyengine-uk/commit/67c638613)). +[`67c63861`](https://github.com/PolicyEngine/policyengine-uk/commit/67c638613), +in [#2115](https://github.com/PolicyEngine/policyengine-uk/pull/2115)). That change verified, on main at `3c48247e`, that: - the full parameter tree (2,706 parameter objects and 43,666 dated @@ -75,16 +103,17 @@ That change verified, on main at `3c48247e`, that: ## Checking for a regression CI now runs the test suite (`.github/workflows/test-suite.yaml`: the -YAML tests and the Python tests, split over four shards) on Linux arm64 -(`ubuntu-24.04-arm`), and the install and smoke-import checks on x86 -`ubuntu-latest`. There is no macOS job. If an OS difference is +YAML tests in one job and the Python tests split over five shards) on +Linux arm64 (`ubuntu-24.04-arm`), and the install and smoke-import checks +on x86 `ubuntu-latest`. There is no macOS job. If an OS difference is suspected again: 1. Dump the processed parameter tree on each platform (every parameter and every dated value, including `parameters.baseline`) and compare checksums. This is the check the October 2026 move relied on. -2. If the trees differ, look first at anything that runs at import time - or depends on file or dictionary order. +2. If the trees differ, look first at chained uprating (a parameter + uprated from an index that is itself uprated) and at anything else + that depends on file or dictionary order. 3. If the trees match but results differ, compare the uprated dataset variables next. #1209 showed the input data themselves were the same on both systems. @@ -92,7 +121,8 @@ suspected again: ## References - Tracking issue: [#1209](https://github.com/PolicyEngine/policyengine-uk/issues/1209). -- Where the difference was first seen: [#1205](https://github.com/PolicyEngine/policyengine-uk/pull/1205). -- Diagnosis: [#1223](https://github.com/PolicyEngine/policyengine-uk/pull/1223) (closed). -- CI moved to macOS: #1092, #1207; moved back to Linux: [`67c63861`](https://github.com/PolicyEngine/policyengine-uk/commit/67c638613). +- Where this difference was seen: [#1205](https://github.com/PolicyEngine/policyengine-uk/pull/1205); the earlier case: [#1091](https://github.com/PolicyEngine/policyengine-uk/issues/1091), fixed by #1092. +- A proposed cause that did not hold: [#1223](https://github.com/PolicyEngine/policyengine-uk/pull/1223) (closed without merging). +- Order-independent uprating: [policyengine-core#490](https://github.com/PolicyEngine/policyengine-core/issues/490), [#491](https://github.com/PolicyEngine/policyengine-core/pull/491). +- CI moved to macOS: #1092 (macOS alongside Ubuntu), #1205 and #1207 (macOS only); moved back to Linux: [`67c63861`](https://github.com/PolicyEngine/policyengine-uk/commit/67c638613) (#2115). - Uprating rebuild and follow-ups: #1254, #1523, #1543, #1544.