Correct 19 planning and validation docs pages from the October batch

changelog.d/october-planning-validation-corrections.changed.md

@@ -0,0 +1,10 @@
+- Corrected 19 October planning and validation documentation pages: firm-level
+  VAT, child maintenance, the default dataset switch, indirect tax effects,
+  non-UK benefit receipt, pension tax relief, the STBM pension comparison,
+  UK pipeline alignment, PIP points, regional childcare programmes,
+  future-dated policy announcements, effective take-up analysis,
+  policyengine.py household-impact tests, Ubuntu versus macOS reproducibility,
+  capital gains distribution, the reform impacts dashboard, Warm Home Discount,
+  Cold Weather Payment and bereavement support. Updated legal rules, model and
+  data-pipeline descriptions, the Microcosm UK default, and the causes of the
+  household-impact and cross-platform failures.

docs/book/assumptions/capital-gains-distribution-plan.md

@@ -6,20 +6,29 @@
 (model gains jointly across income groups) and
 [#817](https://github.com/PolicyEngine/policyengine-uk/issues/817)
 (don't impute gains to households with no wealth). The imputation lives
-in the data build, not in this repo. Most of what this page proposed
-in May 2026 has since been built in
-[Microcosm UK](https://github.com/PolicyEngine/microcosm), which builds
-the replacement for the Enhanced FRS. The certified UK default dataset is
-still the Enhanced FRS from
+in the data build, not in this repo. Most of what this page set out to
+fix in May 2026 has since been addressed in
+[Microcosm UK](https://github.com/PolicyEngine/microcosm), by a different
+method from the one it proposed. Microcosm UK 2024-25 replaced the
+Enhanced FRS from
 [policyengine-uk-data](https://github.com/PolicyEngine/policyengine-uk-data)
-(release 1.58.0). What remains is listed at the end of this page.
+as the certified UK default dataset on 21 October 2026
+([policyengine.py#685](https://github.com/PolicyEngine/policyengine.py/pull/685),
+released in policyengine.py 6.10.0); the Enhanced FRS can still be loaded
+by name. What remains is listed at the end of this page.
 ```
 
 ## How the model uses capital gains
 
-`capital_gains_before_response` is a person-level input, uprated by OBR
-per-capita GDP growth. `capital_gains_tax` charges it at the main CGT
-rates, with three components charged at their own rates (all added by
+`capital_gains_before_response` is a person-level input, uprated through
+`gov.economic_assumptions.indices.capital_gains`, which follows OBR
+per-capita GDP growth unless a simulation supplies a
+`CapitalGainsProjection`
+([#2274](https://github.com/PolicyEngine/policyengine-uk/pull/2274); see
+the [capital gains projection guide](https://github.com/PolicyEngine/policyengine-uk/blob/main/docs/capital-gains-projection.md)).
+`capital_gains_tax` charges `capital_gains` (these gains plus any
+behavioural response) at the main CGT rates, with three components
+charged at their own rates (all added by
 [#1861](https://github.com/PolicyEngine/policyengine-uk/pull/1861)):
 
 - `capital_gains_badr`: gains qualifying for Business Asset Disposal
@@ -36,32 +45,39 @@ Behavioural responses: `capital_gains_behavioural_response` changes
 realisations using an elasticity with respect to the retention rate
 (`relative_capital_gains_retention_rate_change`) or the marginal tax
 rate. Gains use `gov.simulation.capital_gains_responses.elasticity`,
-which is 0 by default, so there is no response unless a user sets it.
+which is 1.0 by default (since
+[#2419](https://github.com/PolicyEngine/policyengine-uk/pull/2419),
+October 2026), so a reform that changes a person's marginal rate on gains
+changes their realisations unless a user sets it to 0; the
+[capital gains realisation response](capital-gains-responses.md) page
+explains it.
 When `gov.simulation.capital_gains_responses.separate_badr_elasticity`
 is true (false by default), gains qualifying for BADR respond at
 `badr_elasticity` instead (1.4, the value the OBR used for these gains
-in its January 2025 costing), through `capital_gains_badr_elasticity`.
+in its costing of the October 2024 rate rises, published in January
+2025), through `capital_gains_badr_elasticity`.
 
 ## How the gains are imputed
 
 ### Enhanced FRS: one distribution per income band
 
-The Enhanced FRS (policyengine-uk-data, the certified default) fits a
-spline to the Advani and Summers gains percentiles within each income
-band and draws each household's gains from its band. The spline alone
-cannot produce gains above about £2m, so policyengine-uk-data main (not
-yet in a release) also stacks 30 donor households per HMRC
-size-of-gain band and calibrates to HMRC counts and amounts of gains by
-band. #818 pointed out the problems with the
-band-by-band fit:
-
-- each band's upper tail was fitted on few observations, so the tails
-  were noisy and the joint distribution of income and gains had breaks
-  at band boundaries;
-- two people in the same band drew from the same distribution however
+The Enhanced FRS (policyengine-uk-data, the certified default until
+21 October 2026) fits a spline to the Advani and Summers gains
+percentiles within each income band and draws each household's gains
+from its band. The spline alone cannot produce gains above about £2m, so
+policyengine-uk-data also stacks 30 donor households for each HMRC
+size-of-gain band from £500,000 up (from £12,300 up before release
+1.59.0) and calibrates to HMRC counts and amounts of gains by band. #818
+asks for gains to be modelled across all income groups together rather
+than each group independently, to avoid overfitting. Fitting band by band
+means that:
+
+- each band is fitted on its own, so the joint distribution of income
+  and gains has breaks at band boundaries;
+- two people in the same band draw from the same distribution however
   different their incomes;
-- gains depended on income only, not on wealth, so they could be
-  imputed to households with no assets (#817).
+- gains depend on income only, not on wealth, so they can be imputed to
+  households with no assets (#817).
 
 ### Microcosm UK
 
@@ -70,35 +86,47 @@ Microcosm builds gains in six stages (see the UK
 
 1. **Support split** (`cgt_support_split`). Within each income band of
    HMRC Capital Gains Tax statistics Table 3, the households with the
-   most investable wealth (financial wealth, business wealth and
-   property other than the main home) are split into lighter copies, so
+   most investable wealth (gross financial wealth, corporate wealth such
+   as shares and investment funds, and property other than the main
+   home) are split into lighter copies, so
    that there are enough rows to hold the largest gains.
-2. **Who has gains** (`cgt_incidence_clone`). The set and order of
-   gainers come from the incidence distribution in Advani and Summers
-   (2020), *Capital Gains and UK Inequality* (CAGE Working Paper 465).
+2. **Who might have gains** (`cgt_incidence_clone`). Every household is
+   cloned at half its weight, and the clone's oldest adult draws a prior
+   gain from the Advani and Summers (2020) gains percentiles for their
+   income band (*Capital Gains and UK Inequality*, CAGE Working Paper
+   465).
 3. **How much** (`hmrc_cgt_gains_spine`). Amounts are redrawn so that
    gains match HMRC Table 3 for 2024-25, which counts taxpayers and
    gains **jointly by size of gain and taxable income**, conditioned on
-   Tables 1, 2.1a, 5 and 6.
-4. **Weight anchor** (`cgt_incidence_anchor`). Clones below the annual
-   exempt amount carry more weighted people than HMRC publishes, so the
-   excess weight is moved back to each clone's paired original.
+   Tables 1, 2.1a, 5 and 6. Within each income band, age group and
+   region group, the largest gains go to the gainers ranked highest on
+   household investable wealth (weighted three quarters) and the prior
+   gain (one quarter).
+4. **Weight anchor** (`cgt_incidence_anchor`). After the redraw, clones
+   with gains at or below the annual exempt amount, or with losses,
+   carry far more weight than the Advani and Summers reporter
+   composition implies for the number of liable gainers (HMRC publishes
+   no count of them), so the excess weight is moved back to each clone's
+   paired original.
 5. **Residential property** (`cgt_residential_split`). Gains on UK
    residential property are fitted to Table 8 (2024-25 residential
    property disposals) by splitting households into weighted copies,
    which fills `capital_gains_residential_property`.
-6. **What kind** (`hmrc_cgt_asset_type_spine`). Of the remaining gains,
-   those qualifying for BADR or Investors' Relief are fitted to Table
-   4.1, which fills `capital_gains_badr`, and the rest are given asset
-   types from Table 7 (gains by asset type, 2023-24).
+6. **What kind** (`hmrc_cgt_asset_type_spine`). Of the other liable
+   gainers, those claiming BADR or Investors' Relief are fitted band by
+   band to Table 4.1, which fills `capital_gains_badr`. Every one of them
+   is also given a main asset type from Table 7 (gains by asset type,
+   2023-24), for diagnostics only. Carried interest (HMRC Table 9) is not
+   imputed.
 
-All the HMRC tables come from the Capital Gains Tax statistics July
-2026 release, through Microcosm's pinned Chronicle feed.
+All the HMRC tables come from the 2026 release of the Capital Gains Tax
+statistics (published 27 August 2026), through Microcosm's pinned
+Chronicle feed.
 
 Against #818: gains are now fitted to the published joint distribution
 of gain size and taxable income, not band by band. Against #817:
-the largest gains are placed on the wealthiest households in each
-income band. Microcosm's release checks also include a projection check
+within each income band, the largest gains go mainly to the wealthiest
+households, through the wealth-weighted ranking in stage 3. Microcosm's release checks also include a projection check
 on people just below the annual exempt amount
 (`uk_cgt_projection_entrants` in `gates.json`), so that
 uprating does not push implausible numbers of them into paying CGT in
@@ -110,8 +138,9 @@ later years.
   positive gains sit on households with zero investable wealth. Neither
   repo asserts this directly yet, so #817 should stay open until it
   does.
-- **Close #818** once a Microcosm UK dataset is the certified default
-  and its diagnostics confirm the Table 3 fit; link them from the issue.
+- **Close #818** once the certified Microcosm UK dataset's diagnostics
+  confirm the Table 3 fit (Microcosm UK has been the certified default
+  since 21 October 2026); link them from the issue.
 - **Elasticities by group.** Responses use one main elasticity and one
   BADR elasticity. With the joint distribution in place, an elasticity
   that varies by income or size of gain becomes possible, at the cost

docs/book/assumptions/mp30k-uk-switch-plan.md

@@ -11,25 +11,34 @@ same switch and rollback path to Microcosm UK.
 
 ## Current state
 
-PolicyEngine UK has no built-in default dataset. `Simulation()` uses, in order:
+PolicyEngine UK has no built-in default dataset. An inline `situation`, if
+passed, is used on its own. Otherwise `Simulation()` uses, in order:
 
 1. an explicit `dataset=` argument;
 2. the `POLICYENGINE_UK_DEFAULT_DATASET` environment variable
    (`DEFAULT_DATASET_ENV_VAR` in `tax_benefit_system.py`);
-3. otherwise, unless an inline `situation` is passed, it raises a `ValueError`
-   (`get_default_dataset_url` in `simulation.py`).
+3. otherwise it raises a `ValueError` (`get_default_dataset_url` in
+   `simulation.py`).
 
-The test suite sets the variable to the private
-`hf://policyengine/policyengine-uk-data-private/enhanced_frs_2023_24.h5@1.40.3`
-when a Hugging Face token is available (`policyengine_uk/tests/conftest.py`).
-So the eFRS is still the dataset the model is tested against, and the
-default in downstream tools such as policyengine.py.
+When a Hugging Face token is available and the variable is not already set,
+the test suite sets it to the certified dataset pinned by commit and SHA-256 in
+`policyengine_uk/tests/certified_dataset.yaml` (read by
+`policyengine_uk/tests/conftest.py`): `enhanced_frs_2024_25.h5` from
+policyengine-uk-data 1.58.1, in the private
+`policyengine/policyengine-uk-data-private` repository. So the eFRS is still
+the dataset this repository's tests run on.
 
 Direct microsimulation through `Microsimulation()` in this package is
 deprecated in favour of [policyengine.py](https://github.com/PolicyEngine/policyengine.py),
 whose managed bundle pins certified datasets (see the repository README). The
 default most users meet is therefore the one that bundle pins, so a switch has
-to be made there as well as in this repository's tests.
+to be made there as well as in this repository's tests. policyengine.py made
+that switch in release 6.10.0 (21 October 2026): its UK default is now
+Microcosm UK 2024-25 (`microcosm_uk_2024_25`), and the Enhanced FRS 2024-25
+stays loadable by name. It did so under its own bundle certification (its
+6.10.0 changelog records an accepted exception to its Universal Credit
+caseload floor); this page does not record whether the gates below were run.
+This repository's tests remain on the eFRS.
 
 ## What the switch needs
 
@@ -41,14 +50,31 @@ to be made there as well as in this repository's tests.
   because `_pre_encode_enum_columns` (in `simulation.py`) walks each year's
   tables.
 - The dataset's `years` drive uprating. Microcosm's base year is 2024-25,
-  one year later than the eFRS file used in tests, so uprating paths from the
-  earlier base year need checking.
-- Enum columns must use the labels in this repo's variable definitions.
-  Unknown labels do not encode correctly.
+  the same as the pinned eFRS file the tests use (`enhanced_frs_2024_25.h5`);
+  `test_latest_data_smoke.py` still loads the unpinned
+  `enhanced_frs_2023_24.h5`.
+- String enum columns must use the member names in this repo's variable
+  definitions (for example `OWNED_OUTRIGHT`, not "Owned outright"). An
+  unknown name raises a `ValueError` when the dataset is encoded
+  (policyengine-core's `Enum.encode`); integer-coded columns are not
+  checked.
+- This package's `hf://` loader (policyengine-core's
+  `download_huggingface_dataset`) and the test pin's downloader (`materialise`
+  in `policyengine_uk/utils/aggregate_impact/dataset.py`) both request Hugging
+  Face model repositories (`repo_type="model"`), and the pin file has no
+  repository-type field. Microcosm UK files sit in a dataset repository
+  (`policyengine/populace-uk-private`), so pinning one for the tests
+  (criterion 4) needs both to accept dataset repositories first, and loading
+  one by `hf://` needs the loader to.
 
 ### Variables
 
-- `policyengine_uk/variables/input/` is the data contract. A switch should
+- `policyengine_uk/variables/input/` holds many of the inputs, but the
+  loader (`build_from_multi_year_dataset` in `simulation.py`) sets every
+  dataset column named after any model variable as an input and skips the
+  rest, and many dataset inputs are defined elsewhere (for example
+  `tenure_type`, `household_weight`, `would_claim_uc` and the `*_reported`
+  benefit amounts). A switch should
   report the input columns present in one dataset but not the other, and
   decide for each whether a missing column is acceptable.
 - Take-up and other reported-versus-simulated inputs have to come from the
@@ -73,28 +99,53 @@ A documented gate must pass before any environment's default changes:
 3. **Reforms:** at least three standard reforms (for example +1p on the basic
    rate, +£5 a week on the UC standard allowance, abolishing the two-child
    limit) give the same sign and are within 5% of each other.
-4. **Tests:** the suite passes with the new default, including
-   `test_latest_data_smoke.py`, `test_deterministic_variables.py` and
-   `test_no_economic_assumptions.py`.
+4. **Tests:** the suite passes with the new dataset pinned in
+   `policyengine_uk/tests/certified_dataset.yaml`, with the reform
+   regression fixtures in `policyengine_uk/tests/microsimulation/reforms_config.yaml`
+   moved to the same dataset. `test_latest_data_smoke.py` loads a
+   hard-coded, unpinned `enhanced_frs_2023_24.h5`;
+   `test_deterministic_variables.py` uses inline situations and
+   `test_no_economic_assumptions.py` an inline situation and a hand-built
+   one-person dataset, so none of those three tests the new default as
+   written.
 5. **Documentation:** the `assumptions/` section records the switch and any
    remaining aggregate gaps.
 
 ## Rollback path
 
-1. Dataset choice stays a runtime setting: no code in this repo depends on
-   which dataset is loaded.
+1. Dataset choice stays a runtime setting: no code in this repo branches on
+   which dataset is loaded. One parameter is tied to the dataset, though:
+   `gov.simulation.vat.survey_consumption_coverage` is calibrated to one
+   Microcosm UK 2024-25 national cut only
+   (`microcosm-uk-2024-25-national-20261002T230158Z-5c6b3f68`; policyengine.py
+   6.10.0 pins the later cut
+   `microcosm-uk-2024-25-national-20261019T083323Z-6cba5d71`), puts 2024-25
+   VAT 38% (policyengine-uk-data 1.56.16) or 44% (1.58.0) above receipts on
+   the Enhanced FRS 2024-25, and has to be re-derived when the dataset it
+   should match changes (`docs/engineering/vat-coverage.md`).
 2. While both are in use, CI runs the dataset-dependent tests on both.
-3. The default is set per environment through
-   `POLICYENGINE_UK_DEFAULT_DATASET`, with the eFRS file kept available.
-4. Rolling back is changing that variable and redeploying, with no code
-   change here. This assumes the eFRS file stays published while the switch
-   beds in.
+3. The default is set per environment: for direct use of this package
+   through `POLICYENGINE_UK_DEFAULT_DATASET`; for this repository's tests
+   through the pin in `policyengine_uk/tests/certified_dataset.yaml` (with
+   the reform regression fixtures recording the same dataset); and in
+   policyengine.py through its bundle manifest's `default_dataset`, which
+   changes with a policyengine.py release. The eFRS file is kept available.
+4. Rolling back is changing that setting and redeploying (in policyengine.py,
+   a new bundle release). In this repository it also means re-pinning the
+   test dataset and regression fixtures if the tests have switched, and
+   re-deriving dataset-calibrated parameters such as the VAT coverage above.
+   This assumes the eFRS file stays published while the switch beds in.
 
 ## Acceptance
 
 PolicyEngine UK switches its default only after the gates above pass, with a
-rollback that needs no code change in this repo. Until then either dataset can
-be used by passing `dataset=` or setting the environment variable.
+rollback that needs no change in this repo beyond re-pinning the test
+dataset and re-deriving dataset-calibrated parameters. Until then either
+dataset can be used by passing `dataset=` or setting the environment
+variable. Microcosm UK files sit in a Hugging Face dataset repository, and
+this package's `hf://` loader requests model repositories (policyengine-core's
+`download_huggingface_dataset` passes `repo_type="model"`), so download a
+Microcosm file and pass its local path.
 
 ## References