Skip to content

Report each pull request's effect on national aggregates on the certified dataset - #2335

Merged
MaxGhenis merged 30 commits into
mainfrom
aggregate-impact-report
Oct 18, 2026
Merged

MaxGhenis merged 30 commits into
mainfrom
aggregate-impact-report

Conversation

@MaxGhenis

@MaxGhenis MaxGhenis commented Oct 11, 2026 •

Copy link
Copy Markdown
Collaborator

Fixes #2334

This is workstream W3 of the Autumn Budget 2026 CI plan. Max ruled cos d1601 A on 2026-10-17: publish exact base-to-head changes, including changes between model versions, as policyengine-api does; level protection stays. The PR remains a draft pending the UK hub’s independent review and CI. GPT-6.1 Sol approved the pre-adoption head, 1191ac5a4 (delta review 11, no findings), after re-reviews 7–10.

What it adds

  1. An aggregate-impact report. The module is policyengine_uk.utils.aggregate_impact, and the workflow is .github/workflows/aggregate-impact.yaml. On every PR it measures the base and the head on the same certified dataset, on the same runner image. Each side syncs its own uv.lock and runs the PR head's copy of the tool. The report covers every year from 2025-26 to 2031-32:

    • receipts by tax head;
    • spending and caseloads by benefit;
    • poverty: absolute and relative, BHC and AHC, for everyone, children, working-age adults and pensioners;
    • the Gini coefficient;
    • mean income in each side's own deciles, and household-matched decile changes within the base deciles;
    • ten standard reform costings: the eight fixtures, plus a 10% rise in the Universal Credit standard allowance and in the State Pension.

    Base and head are measured in one job. The dataset is licensed survey microdata, so only aggregates leave that job; a test enforces it (see "Data protection" below).

    It posts one comment and updates it on later runs, including after a failed run. It fails on:

    • a crash;
    • a different dataset on the two sides;
    • non-finite figures;
    • any check that the head breaks where the base passes it, judged per year and per problem.

    Moves above the flag thresholds are listed, not failed.

  2. A fiscal partition, policyengine_uk/data/fiscal_partition.yaml.

  3. Immutable test data and a mandatory-data mode, in tests/conftest.py.

    • tests/certified_dataset.yaml pins uk-data 1.58.1's enhanced_frs_2024_25.h5 by Hugging Face commit 2526edbb (sha256 056e80fd…). policyengine.py bundle uk-6.4.3 certified that artifact for model 2.130.1.
    • Data tests verify those bytes once per process before they run.
    • With POLICYENGINE_UK_REQUIRE_TEST_DATA=1, every data test fails in setup when the dataset is unavailable, and runs only on the pinned file. The one named exception is test_latest_data_smoke.py, which exists to read the latest data; it is reported in the pytest header.
    • Without the flag, tests skip as before. Credential-free unit tests are unaffected.
    • W2 wires the flag into the existing workflows; this PR does not touch them.
  4. Reform fixtures rebased onto 1.58.1. reforms_config.yaml keeps every earlier value as a receipt, with the model and data that produced it: the unpinned values, 1.40.3 and 1.58.0 at 2.126.8, and 1.58.1 at 2.132.6. The 1.40.3 receipts show how much of each move is the data.

Invariants the report enforces

  • Closure. For each of gov_tax, household_tax, pre_budget_change_household_tax, gov_spending and household_benefits, the model's own component list is exactly the partition's heads plus its declared double counts, each listed once. Each head's declared entity is its variable's entity.

  • Identities, household by household.

    • Heads plus double counts equal each model total, respecting dropped_when (council tax abolition).
    • gov_balance equals gov_tax − gov_spending.
    • Household net income equals market income plus benefits, less taxes and pension contributions.
    • Geographic exclusivity holds once declared double counts are netted out of their containers.
  • Counted once, household by household. Each component is raised by £1 per unit, and every head of its totals is read in every household. A two-point slope test separates a passport (slope 0) from containment (slope 1) for heads allowed to respond. The rules:

    • its own head rises by exactly £1;
    • no other tax head moves;
    • no other benefit rises (means tests may claw the rise back);
    • the model's own total moves by what its heads moved.

    A head that breaks a rule, or that the slope test screens, is measured again twice:

    • with every other head held at baseline;
    • with every other head held at its value after the first point, while the component moves to the second.

    It fails where either measurement still shows the response, so a containment gated by another head (a passport) is caught too. A response that both measurements clear is reported as mediated. The other heads that moved in those households are listed as candidates, not as a proven route. For example, Incapacity Benefit lowers Carer's Allowance through the overlapping-benefit rules, and Universal Credit counts Carer's Allowance as income but, wrongly (Count Incapacity Benefit and SDA as Universal Credit unearned income (UC transitional regulations reg. 25) #2424), not Incapacity Benefit.

    Further rules:

    • A declared double count must still be contained in its head, or the check fails as stale.
    • The check reuses one simulation, restores every input it overrode, and requires the recomputed baseline to match the first run exactly.

    Documented limits:

    • a head that adds a component back while tapering it pound for pound;
    • a mediated fall that offsets or outweighs a containment in every perturbed household (tested as a limit).
  • Bounds and finiteness. Weights are finite and non-negative. Poverty rates lie in [0, 1]. Every figure is finite, reform costings included. Heads declared sign: nonnegative are never negative.

  • Determinism. A rebuilt unreformed simulation reproduces the baseline gov_balance exactly, in every year.

  • Reform responsiveness. Each standard reform names the variable it must move, the direction and a minimum (standard_reforms.yaml). A reform that moves its target by less, by nothing or the wrong way fails responsive:<id> for that year, so a reform the model silently ignores cannot pass as a small number.

    • Relative reforms are built from each side's own baseline parameters: that model year's value times the factor.
  • Comparison.

    • Comparing a side with itself changes nothing and flags nothing.
    • Changes are antisymmetric.
    • A flag fires exactly when |head − base| > max(absolute, relative × |base|).
    • Required checks must run, and a base is required.
    • The comment always carries its marker and fits GitHub's limit.

Thresholds adopted under cos d1344

d1344 delegates these calls to the workstream, with an independent review agreeing. The full basis is in the files.

  • Responsiveness minimums (standard_reforms.yaml). Each is a fifth of the reform's smallest yearly effect, 2025-26 to 2031-32, on the certified pair, rounded down to two significant figures. parse_reforms refuses a minimum that doesn't follow from the recorded smallest_effect, and python -m policyengine_uk.utils.aggregate_impact calibrate recomputes both.
    • Measured on uk-data 1.58.1 with policyengine-uk 2.130.1 on core 3.33.0 (the certified model) and with 2.132.6 on core 3.33.2. All 70 reform-year target changes were identical between the two.
    • What the check must catch: the backtest the plan cites found the UC standard allowance rise returning exactly £0 (2.45.1–2.75.3) and the State Pension rise barely moving (2.88.10–2.102.6).
    • What it must not catch: the fixture receipts show 2029-30 costings moving by up to 18% between data and model versions, and 38% for VAT +2pp (data). A fifth fails only after an 80% loss.
    • Changes: the personal allowance minimum falls from £1bn to £0.35bn, because its smallest year, 2031-32, is worth £1.78bn. VAT +2pp rises from £4bn to £6.1bn. The other eight move by less.
    • Alternatives recorded: a tenth, a half, per-year minimums, an external anchor, and a sign check alone.

Initial calibration before adoption (1191ac5a4; models 2.130.1–2.132.6 on uk-data 1.58.1). This table records the original minimums; the merged-model changes appear in the adoption update below.

Reform Smallest yearly effect Year Minimum
Basic rate +1pp £6.37bn 2025-26 £1.2bn
Higher rate +1pp £3.55bn 2025-26 £0.71bn
Personal allowance 13,000 −£1.78bn 2031-32 £0.35bn
Child benefit additional +£25/wk £1.59bn 2031-32 £0.31bn
UC taper 20% £20.5bn 2025-26 £4.1bn
Employee NICs main 10% £11.3bn 2025-26 £2.2bn
VAT +2pp £30.9bn 2025-26 £6.1bn
Additional rate +3pp £2.80bn 2025-26 £0.55bn
UC standard allowance +10% £3.04bn 2025-26 £0.60bn
State Pension +10% £12.3bn 2025-26 £2.4bn
  • Flag thresholds (report_spec.yaml), a judgement recorded with alternatives:

    • £ totals: max(£100m, 0.1%);
    • reform costings and their targets: max(£100m, 1% of the base costing);
    • caseloads: max(5,000, 0.5%);
    • household £ figures: max(£10, 0.1%);
    • rates: 0.1pp; Gini: 0.001.

    Flags only decide which moves lead the comment. Every changed figure is listed whether or not it is flagged. When the comment would exceed GitHub's limit, sections are left out, and those figures are in the report artifact.

    • Repeated runs of the same pair are identical, so no positive threshold flags noise.
    • To revisit with the flag counts from the first two weeks of comments.

Data protection

An earlier version of this workflow (commits 073e808 to 8d3e482) uploaded each side's measurement as an Actions artifact, including household-level arrays, and cached the dataset in the Actions cache. Those artifacts and the cache were deleted on 11 October, and fd94415 removes both:

  • both sides are measured in one job, and only the report directory is uploaded (two result.json files of aggregates, the comment and the summary);

  • the dataset is never cached;

  • household detail is written under private/, which is deleted after the comparison;

  • small cells (privacy.py, which also states the threat model and its deliberate exceptions). A head total, caseload or declared double count built from 1 to 9 records is published as suppressed, and its years are listed. Its raw value never enters the record. A suppressed head's caseload is suppressed with it, since a caseload of zero would show that none of its records is positive.

    A total holding a suppressed figure is rebuilt that year from its other components, household by household. It is never the model's total less the suppressed figure, and the components it leaves out are recorded with their signs under excludes. Earlier versions subtracted the figure and gated that on the identities holding. Sol's re-review 7 showed that a £1 identity tolerance, together with a benefit-uprating setting, still left a residual from which the figure could be recovered. A rebuilt total is the published parts' sum exactly, so no formula or tolerance matters. What each total is rebuilt from:

    • A model total: its other heads and declared double counts.
    • A balance aggregate: its other heads.
    • gov_balance: the rebuilt gov_tax less the rebuilt gov_spending.
    • Household net income: market income plus the rebuilt household_benefits, less the rebuilt household_tax and pension contributions (its own adds and subtracts). It is suppressed if those can't be read.
    • The HBAI total: rebuilt from its own component lists without the suppressed ones, then reset at zero. Before the reset, the lists must reproduce the model's income to within half a penny for every household that doesn't hold the figure. Otherwise the total is suppressed (council tax when it is abolished, for example).
    • Decile incomes: the rebuilt household net income.

    Checks whose verdict could depend on a suppressed figure leave its holders out, or aren't run.

    • The identities (verdict, count and gap), the HBAI reproduction check and the overlap guards are judged only on households that hold no suppressed figure that year. With a formula that scales its components, whether a holder's household differs would otherwise depend on the figure. The cost: a bug confined to those 1 to 9 households goes unseen that year.
    • A suppressed head's sign check isn't run that year, because it would say whether any of its few records is negative.
    • A count of fewer than 10 non-finite values is given as "fewer than 10".
    • The published guard drops a few-unit double count's coefficient.
    • The counted-once verdict for a few-unit component is kept, as the only containment test for it. Its counts, responses and coefficients are not published.

    Poverty, inequality and deciles are withheld in a year when HBAI income holds a suppressed figure. Households are ranked into deciles on HBAI income, so which decile a holder falls into would depend on the figure. The comparison says so for that year.

    A standard reform's costing and target change are withheld for a year (published as suppressed, and the responsiveness detail gives no amount) when the reform:

    • changes any record of a component suppressed in the baseline, by any amount, even if the changes cancel nationally. An unchanged record compares exactly equal, because the rebuilt baseline reproduces the first run exactly;
    • leaves any other component with 1 to 9 records;
    • changes gov_balance in 1 to 9 households, or its target variable in 1 to 9 records, by more than half a penny.

    Re-review 7's cases fail without these rules and pass with them: two suppressed pensions whose changes cancel, a £4 pension raised 10%, and an allowance rise that leaves one taxpayer.

    Comparisons. Where the two sides leave out different components, additive totals are lined up by taking the union out of both, from published values only. The change is then the published parts' change, and the total is marked †. A change that can't be shown is rendered "withheld", never "no change": a mean or the clipped HBAI total with different exclusions, or a figure suppressed on either side. Decile rows are withheld when the two incomes leave out different components. When every year is withheld, the decile section says why.

    Under d1601 A (Max, 2026-10-17), exact national changes between base and head are published as policyengine-api does, including changes between model versions or datasets. This includes a change from 1 to 9 changed records of a total with 10 or more. Level protection stays; decile rows and reform figures retain their existing small-change withholding. This ruling required text updates only: the report already behaves this way.

    Guard details give a small count as "fewer than 10", and the published result drops small unit counts, household counts and shares. compare treats a suppressed cell as neither non-finite nor flagged. The W4 long format carries suppressed and, for each total, the excludes list with each entry's sign in exclude_signs.

    On certified 1.58.1 at 1191ac5a4, Disabled Students' Allowance's total and caseload are suppressed in all seven years (14 of 1,484 rows in the W4 format). Sixteen figures are rebuilt without it in every year (112 cells):

    • gov_spending, household_benefits and gov_balance;
    • the spending and balance aggregates;
    • household net income;
    • the ten decile mean incomes.

    Against f8dfecd4f's subtraction, these moved only by the identity tolerance that subtraction carried: decile means by under £0.002 per household, and the national totals by £600 to £6,700. Every value is identical between 092444913 and 1191ac5a4. DSA isn't part of HBAI income, so the HBAI total, market income, poverty and Gini are published as measured, and the decile ranking is unaffected. No standard reform touches it or leaves a small cell, so every costing is published. The only failing check is sign:vat, which fails on main too. DSA's own sign check is skipped. The measurement took 391s. The W4 fixture is budget-metrics-uk-data-1.58.1-pe-uk-1191ac5a4.json.

An earlier comment on this PR and six earlier report artifacts published Disabled Students' Allowance figures built from fewer than 10 records, before suppression existed. The comment's current text suppresses them. Under Max's standing rule (d932), old revisions and artifacts holding survey small cells need no action.

test_aggregate_impact_workflow.py fails if the workflow uploads anything else, uses the cache, or splits the measurement across jobs. It also fails if the comment step stops running after a failure: the step writes a FAIL comment when the comparison wrote none, so an earlier passing comment can't stand.

Tests

  • Property-based (Hypothesis):
    • the data plan over random environments: mandatory mode never skips and runs only on the pin;
    • pin parsing and byte verification against random tampering;
    • partition mutations and head-order invariance;
    • counted-once on a fake simulation with known containment:
      • the coefficient is 2 exactly when a head contains another;
      • in a model where Incapacity Benefit lowers Carer's Allowance and UC reads both, the guard fails exactly when UC responds to Incapacity Benefit directly, whatever share of Carer's Allowance it reads;
      • examples: a containment gated by a passport on another head fails. A mediated fall that cancels or outweighs a containment is a tested limit;
    • small cells:
      • a cell from 1 to 9 records never reveals its amount;
      • two datasets that differ only in a suppressed head publish exactly the same figures, for any value, sign and formula scaling. This is tested on the rebuild, and through measure_year on a stub simulation, including every check's status and detail and the decile incomes. Adding negative awards to this test found the caseload leak above;
      • repeated reports with changing published heads say nothing about it;
      • no published change depends on it, whichever side suppresses it;
      • any change to any suppressed pension record, whether or not it is cancelled nationally, publishes no reform figure;
      • the rebuilt HBAI income never depends on the suppressed component;
    • responsiveness symmetry;
    • calibrated minimums: within 10% below the fraction, two significant figures, monotone;
    • comparison identity, antisymmetry and thresholds; comment size.
  • Examples for small cells:
    • suppression end to end through compare, the comment and the W4 format;
    • published guard output keeps no small counts or shares;
    • totals are rebuilt without a suppressed head with the right sign, and decile incomes are rebuilt household by household;
    • re-review 7's cases, each with Sol's own numbers: a 0.01% benefit uplift, a one-record corporate SDLT double count (gov_tax less receipts gives nothing), cancelling pension changes, a £4 pension, the allowance rise that leaves one taxpayer, reform costings from 3, 9 and 10 changed households, abolished council tax of £1,200, £0.50 and £0.01, and an identity whose only failure is a holder;
    • a suppressed HBAI component withholds deciles, poverty and the Gini;
    • withheld comparisons render as "withheld" and count as changed;
    • a suppression transition (10 records on the base, 9 on the head) aligns both sides from published values;
    • no head of the shipped partition is part of household market income.
  • Differential:
    • the partition against the model's own lists;
    • decile impacts against a pandas groupby;
    • Gini against microdf;
    • head totals against Microsimulation.calculate(...).sum().
  • On certified data (microsimulation marker): test_aggregate_impact_certified.py runs 2026-27 with the counted-once check on every tax component and five benefits, and the two relative reforms.

Pre-adoption validation at 1191ac5a4:

  • pytest policyengine_uk/tests/aggregate_impact policyengine_uk/tests/code_health/test_aggregate_impact_workflow.py -m "not microsimulation" -n 4: 192 passed without credentials, at 1191ac5a4.
  • The full certified measurement on the 1.58.1 pin at 1191ac5a4 (above).
  • test_aggregate_impact_certified.py in mandatory mode on the 1.58.1 pin: 9 passed at 1191ac5a4.
  • test_aggregate_impact_certified.py with POLICYENGINE_UK_REQUIRE_TEST_DATA=1 on the 1.58.1 pin: 9 passed.
  • Before the latest commits, the 39 microsimulation tests passed on the 1.58.1 pin in mandatory mode.
  • A full guard run on 2026-27: 314 checks pass, 15 warn (inert heads) and 1 fails (sign:vat, pre-existing).

The full Python suite was not run locally; CI runs it.

Findings recorded before adoption at 1191ac5a4

Not in this PR

  • W2 sets POLICYENGINE_UK_REQUIRE_TEST_DATA in the existing workflows.
  • A data-comparison mode for uk-data PRs (different dataset bytes, same model) is Compare two datasets on one model in the aggregate-impact tool #2425, stacked on this one.
  • The uk-data gates come in separate uk-data PRs for the d833 batch: the model-fixed comparison, final-fit and target-coverage gates, single-age and HBAI checks, and OBR ingestion.

axiom: n/a: CI, test-data and reporting infrastructure; no policy change

🤖 Generated with Claude Code

Round 2 adoption update — newer main merged and recalibrated

Current head: a6f736468a752ddf8a63f3735f261b24e603b041. Merged main 7ad002f549680a9ca024c896532697d532a3a92a (policyengine-uk 2.135.7) in 7717438bffbd5aa45fff464466b5d043e473052b, on top of round-one head 7ac83a4cdf4ee72812bdc7157ae4935bd4599578, preserving the author and original history. The reforms_config.yaml conflict is resolved. The PR remains a draft pending the UK hub's independent review and hosted CI; no merge or readiness action is requested. #2425 remains unchanged.

d1601 A remains as ruled by Max on 2026-10-17: publish exact base-to-head changes, including changes between model versions or datasets, as policyengine-api does; level protection stays. No national-change protection was added.

Historical round-one evidence only: at 7ac83a4cdf4ee72812bdc7157ae4935bd4599578 on model 2.135.6, the focused/workflow suite passed 192 tests and mandatory pinned-data suite passed 33 tests. Its full seven-year invariant harness completed, with the State Pension / Universal Credit counted-once finding in 2029-30 and the existing negative-VAT finding retained. Round one's minimum changes were higher rate £710m → £700m and additional rate £550m → £560m. Those tests and findings are historical evidence; the fresh results below describe the newer merged tree.

Closure and partition. Runtime registry closure and structure checks pass for all five totals: gov_tax 26, household_tax 27, pre_budget_change_household_tax 17, gov_spending 55, household_benefits 54. Main f1a01e4d54f13bdd1847bde52cffca7422873c3f (#2382, merged at 50063e1da71495ebaf7aa767f049000ead3cc1b3) adds war_widows_pension to both spending lists. The partition adds that one person-level spending head, giving 55 spending heads. VAT #2187 changes grossing formulas but no component list. Tax heads and the existing corporate_sdlt declared double count stay under #2235/#2159.

Reform receipts and centres. Verified the certified uk-data 1.58.1 bytes first: SHA256 056e80fd8e9151d78f432a25cde863340baee81304c6216a37fb575ce8563784. Recalibrated with model 2.135.7, Core 3.33.2, on the same pin. Preserved all 41 historical receipt mappings, including all round-one receipts and main's complete incoming VAT mapping in its original schema (plus its source commit), then appended eight new 2029 gov_balance receipts in the round-one impact / data / model format. The main VAT mapping is historical evidence whose dataset/model are not inferred. Refreshed 1 expected-impact centre(s); parameters and tolerances stay.

Reform 2.135.6 receipt £bn New receipt £bn Expected old → new £bn
Raise basic rate by 1pp 7.115 7.115 7.1 → 7.1
Raise higher rate by 1pp 4.490 4.490 4.5 → 4.5
Raise personal allowance by ~800GBP/year -4.195 -4.195 -4.2 → -4.2
Raise child benefit by 25GBP/week per additional child -1.442 -1.442 -1.4 → -1.4
Reduce Universal Credit taper rate to 20% -21.956 -21.956 -22.0 → -22.0
Raise Class 1 main employee NICs rate to 10% 12.463 12.463 12.5 → 12.5
Raise VAT standard rate by 2pp 34.041 27.581 34.0 → 27.6
Raise additional rate by 3pp 3.656 3.656 3.7 → 3.7

The official python -m policyengine_uk.utils.aggregate_impact calibrate .job/calibration-measurement/result.json command passed and all ten outputs match the applied calibration. Only 1 responsiveness minimum(s) change from round one under the existing rule: one fifth of the smallest directed yearly target effect over 2025-26 to 2031-32, rounded down to two significant figures. Each changed minimum and its stored smallest_effect were updated together; all other historical calibration fields stay.

Reform Fresh smallest yearly effect £ Year 2.135.6 minimum £ New minimum £
vat_standard_plus_2pp 25,006,282,297 2025-26 6,100,000,000 5,000,000,000

Smallest yearly effects moving by more than 10% since 2.135.6:

  • vat_standard_plus_2pp: smallest yearly effect £30,863,018,576 → £25,006,282,297 (-18.98%). VAT grossing divisor changed from 0.38 to household share 0.7 times survey coverage 0.67 (=0.469). Holding the consumption base fixed, VAT reform effects scale by 0.38/0.469. Cause: main 98ab77093b0f48e4cb50677c2c9403a5533631f9 (Split VAT grossing into household share and survey coverage #2187). This attribution is an inference from the formula and measured ratio.

Fresh local validation on the merged model:

  • Focused PR/workflow suite, pytest policyengine_uk/tests/aggregate_impact policyengine_uk/tests/code_health/test_aggregate_impact_workflow.py -m 'not microsimulation' -n 4 --dist worksteal -o xfail_strict=true: 192 passed, 0 failed, 0 errors, 0 skipped (192 distinct nodes).
  • Mandatory pinned-data suite, POLICYENGINE_UK_REQUIRE_TEST_DATA=1 pytest policyengine_uk/tests/aggregate_impact/test_aggregate_impact_certified.py policyengine_uk/tests/microsimulation policyengine_uk/tests/test_capital_gains_tax_components.py -m microsimulation -n 0 -o xfail_strict=true: 33 passed, 0 failed, 0 errors, 0 skipped (33 distinct nodes).
  • Seven-year calibration measurement: all ten reforms have complete target coverage; closure, identities, determinism, responsiveness and reform execution pass. Counted-once was disabled for this calibration measurement. Only aggregate artifacts were written.
  • The full seven-year counted-once harness was explicitly skipped for round two as requested; its prior 2.135.6 run remains historical evidence. Only 2029-30 was rerun: 1 fail, 83 pass; failing component(s): state_pension. The 2029 counted-once command exits 1 for its recorded failure; no guard pass is claimed.
    The round-one State Pension / Universal Credit finding is confirmed unchanged. The public failure signature and common-component statuses match round one; the affected small-cell count remains protected. Attribution to main issue #1947 remains provisional. Added components: war_widows_pension pass. Declared double counts pass: True; reset reproduces the baseline: True.
  • make format and lint passed before the calibration commit; uvx ruff format --check . passed before pushing.

The calibration measurement separately records 1 failing ancillary check(s) and 17 warning(s). Failing IDs: sign:vat. Negative VAT remains the existing negative-consumption/data finding uk-data #565.

The adoption adds receipt/calibration and partition metadata changes without editing model policy formulas or parameters beyond the merged main. All suites and microsimulations ran in the foreground, with no more than four pytest workers and one microsimulation at a time. The complete repository Python suite was not run locally. Hosted CI supplies the per-PR base/head report; independent review belongs to the hub. No all-seven-year counted-once pass is claimed.

MaxGhenis and others added 4 commits October 11, 2026 13:43
The report measures one model checkout on the pinned certified dataset over
2025-26 to 2031-32: receipts by tax head, spending and caseloads by benefit,
poverty by measure and group, inequality, own-decile incomes and the
standard reform costings. It checks them against a fiscal partition of the
model's totals (policyengine_uk/data/fiscal_partition.yaml):
- closure: each total's components are exactly the partition's heads and
  declared double counts;
- accounting identities, household by household;
- a counted-once check, which perturbs each component by £1 and requires
  each tax total to move by exactly £1 per unit, and each spending total by
  at most £1.

compare then sets a base measurement against a head one. It fails on a
crash, a different dataset, non-finite figures or a check the base passes
and the head breaks. It flags moves above the proposed thresholds, and
renders a pull request comment.

The partition's tax heads follow the #2159 work's specification, encoded
for main before that fix: corporate_sdlt is a declared double count inside
expected_sdlt. Spending heads cover gov_spending and household_benefits.

Test data: conftest pins enhanced_frs_2024_25 from uk-data 1.58.0 by
Hugging Face commit and SHA-256 (tests/certified_dataset.yaml), verifies the
bytes before data tests run, and adds a mandatory-data mode
(POLICYENGINE_UK_REQUIRE_TEST_DATA). In that mode, data tests fail rather
than skip without data, and run only on the pinned dataset. The reform-cost
fixtures are rebased onto 1.58.0, keeping each earlier value as a receipt
with the model and data that produced it.

Refs #2334.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
.github/workflows/aggregate-impact.yaml measures the pull request's base
and head on the same runner image. Each side syncs its own uv.lock and runs
the head's copy of the tool against its own model, on the pinned dataset
whose bytes the tool verifies. The report job then compares the two, writes
the job summary and one updated comment, and fails on structural failures.
Fork and Dependabot runs, which cannot read the private dataset, record a
skip instead.

Refs #2334.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sections with no change collapse into one line, an unchanged decile table
into one sentence, and the counted-once summary lists only real violations
(a tax total moving by other than £1 per unit, or a spending total by more
than £1), not the means-test offsets every benefit shows.

Refs #2334.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Refs #2334.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
MaxGhenis added a commit that referenced this pull request Oct 11, 2026
The aggregate impact report (#2335) adds gov_spending, household_benefits
and spending_heads to fiscal_partition.yaml. The tax tests now check the
three tax totals only, require that spending heads borrow no tax head id or
variable and that no spending head names a tax total, and apply dropped_when
to the tax totals alone. Checked against #2335's file with these tax
sections merged in: this file's tests and #2335's partition and guard tests
pass.

The vat head records that 76 households have negative VAT on
policyengine-uk-data 1.58.0 (consumption or the full-rate expenditure share
is negative in the data; policyengine-uk-data issue 565).

Refs #2159

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
MaxGhenis added a commit that referenced this pull request Oct 11, 2026
…lookup

- Set POLICYENGINE_UK_REQUIRE_TEST_DATA=1 (W3's conftest switch, #2335)
  wherever require-data is on: main pushes, merge queue groups and
  releases. A test that needs the dataset then fails with its own message
  instead of skipping. Unused until #2335 lands.
- A manual dispatch of the main workflow always tests; the merge queue
  lookup applies only to pushes, and a failed lookup parses to zero.
- Document the re-run path's residual and where docs-only pull requests
  build the documentation; the workflow-shape test covers the release
  workflow's require-data and the lookup.

From the GPT-6.1 Sol delta review of 96e9cec (APPROVE, non-blocking).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
MaxGhenis and others added 2 commits October 11, 2026 19:24
The standard reforms move into the tool (standard_reforms.yaml), so a
vendored copy carries them: the eight reform-cost fixtures, plus a 10% rise
in the Universal Credit standard allowance and in the State Pension. Each
names the variable it must move, the direction and a minimum. A reform that
moves its target by less, by nothing or the wrong way in any year fails
responsive:<id>, so a reform the model silently ignores cannot pass as a
small number.

A relative reform is built from each side's own baseline parameters: that
model year's value times the factor, for every year of the horizon.

On uk-data 1.58.0 all ten respond in every year, 2025-26 to 2031-32. The
UC standard allowance rise adds £3.0bn to £3.4bn of Universal Credit, and
the State Pension rise £12.4bn to £14.4bn.

Refs #2334.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The certified dataset is licensed survey microdata, and only aggregates may
be published. The workflow uploaded each side's measurement as an artifact,
including the household-level arrays the decile comparison uses, and cached
the dataset in the Actions cache. Both are removed:

- base and head are now measured in one job, so the household detail never
  travels between jobs; only the report directory (the two result.json
  files, the comment and the summary) is uploaded;
- the dataset is downloaded on each run and never cached;
- measure writes household detail under private/, apart from result.json;
- check details give counts and weighted totals, not one household's value.

test_aggregate_impact_workflow.py fails if the workflow uploads anything
but the report, uses the Actions cache, copies anything private into the
report, or splits the measurement across jobs.

Refs #2334.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@PolicyEngine PolicyEngine deleted a comment from github-actions Bot Oct 11, 2026
MaxGhenis added a commit that referenced this pull request Oct 11, 2026
The aggregate impact report (#2335) adds gov_spending, household_benefits
and spending_heads to fiscal_partition.yaml. The tax tests now check the
three tax totals only, require that spending heads borrow no tax head id or
variable and that no spending head names a tax total, and apply dropped_when
to the tax totals alone. Checked against #2335's file with these tax
sections merged in: this file's tests and #2335's partition and guard tests
pass.

The vat head records that 76 households have negative VAT on
policyengine-uk-data 1.58.0 (consumption or the full-rate expenditure share
is negative in the data; policyengine-uk-data issue 565).

Refs #2159

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@github-actions

github-actions Bot commented Oct 11, 2026 •

Copy link
Copy Markdown
Contributor

Aggregate impact on certified data: pass

Base 7ad002f549 against head 5fdfbc1956, both on enhanced_frs_2024_25.h5 (policyengine-uk-data 1.58.1, sha256 056e80fd8e91), fiscal years 2025-26 to 2031-32. Each year is a full microsimulation run on each side.

Checks: 0 broken by this change, 1 failing on both sides, 0 fixed. Moves flagged for review: 0 (thresholds in report_spec.yaml; a flag does not fail the job).

Workflow run and full report

Notes

  • In the years a figure from fewer than 10 records is suppressed (spending.disabled_students_allowance), every total containing it, and the decile incomes, are rebuilt from their other components, so no published total includes it. Those totals are marked †. A change is withheld where the two sides leave out different figures and can't be lined up.

Moves flagged for review

None.

Spending by benefit and service

Measure 2025-26 2026-27 2027-28 2028-29 2029-30 2030-31 2031-32
Disabled Students' Allowance suppressed suppressed suppressed suppressed suppressed suppressed suppressed

Caseloads

Measure 2025-26 2026-27 2027-28 2028-29 2029-30 2030-31 2031-32
Disabled Students' Allowance suppressed suppressed suppressed suppressed suppressed suppressed suppressed

Household net income by decile

No household's net income changes in any year.

No change in any year: receipts by tax head; government balance, each component counted once; model totals; components the model counts twice; household income totals; population; poverty rates; people in poverty; inequality; mean household net income in each side's own deciles; standard reform costings (change in gov_balance); standard reforms: change in each reform's target.

Checks failing on both base and head (1)
  • VAT is never negative (sign:vat): 2025-26: 76 units below zero, a weighted £-54,797,972 in total; 2026-27: 76 units below zero, a weighted £-56,271,349 in total; 2027-28: 76 units below zero, a weighted £-57,609,146 in total
Counted-once check

Each of 84 components was raised by £1 per unit in 2026-27, and every head of its totals was read household by household: its own head must rise by exactly £1, no other tax head may move, no other benefit may rise, and the model's own total must move by what its heads moved. A head that breaks a rule is measured again twice: with every other head held at its baseline value, and with every other head held at its value after the first £1 while the component rises again. It fails where either still shows the response. 0 failed; 0 were skipped.

  • incapacity_benefit: pension_credit responds only through other heads in fewer than 10 households (other heads that moved there: carers_allowance; gov_spending; not a failure)
  • incapacity_benefit: pension_credit responds only through other heads in fewer than 10 households (other heads that moved there: carers_allowance; household_benefits; not a failure)
  • incapacity_benefit: universal_credit responds only through other heads in 72 households (other heads that moved there: carers_allowance, carer_support_payment; gov_spending; not a failure)
  • incapacity_benefit: universal_credit responds only through other heads in 72 households (other heads that moved there: carers_allowance, carer_support_payment; household_benefits; not a failure)
  • maternity_allowance: pension_credit responds only through other heads in fewer than 10 households (other heads that moved there: carers_allowance; gov_spending; not a failure)
  • maternity_allowance: pension_credit responds only through other heads in fewer than 10 households (other heads that moved there: carers_allowance; household_benefits; not a failure)
  • declared double count corporate_sdlt is still contained in expected_sdlt (coefficient 1.000; Corporate SDLT is counted twice in household and government tax totals #2159)
Provenance
Side Commit Runtime Environment hash Partition hash Time
base main 7ad002f549 uk 2.135.7, core 3.33.2, py 3.13.16 aarch64 4465a4b5e606 0d0426ee9e38 894.63s
head #2335 merged into main 5fdfbc1956 uk 2.135.7, core 3.33.2, py 3.13.16 aarch64 4465a4b5e606 0d0426ee9e38 887.15s

Dataset: hf://policyengine/policyengine-uk-data-private/enhanced_frs_2024_25.h5@2526edbbb687a3b54dbb11272e5389e4b62d2ac2, sha256 056e80fd8e9151d78f432a25cde863340baee81304c6216a37fb575ce8563784 (verified on both sides), policyengine-uk-data 1.58.1.

Receipts by tax head: head levels
Measure (head) 2025-26 2026-27 2027-28 2028-29 2029-30 2030-31 2031-32
Income tax £298.30bn £318.11bn £335.36bn £350.32bn £365.75bn £383.77bn £402.05bn
National Insurance (employee and self-employed) £52.58bn £54.94bn £56.75bn £58.48bn £60.51bn £62.45bn £64.90bn
National Insurance (employer) £152.64bn £159.18bn £164.14bn £168.73bn £175.57bn £181.08bn £188.22bn
Employer NI change borne by capital £0.00bn £0.00bn £0.00bn £0.00bn £0.00bn £0.00bn £0.00bn
Employer NI change borne by consumers £0.00bn £0.00bn £0.00bn £0.00bn £0.00bn £0.00bn £0.00bn
Capital gains tax £12.62bn £13.06bn £13.55bn £14.04bn £14.54bn £15.09bn £15.65bn
Student loan repayments £7.83bn £8.12bn £8.35bn £8.56bn £8.79bn £9.04bn £9.38bn
VAT £253.71bn £260.53bn £266.72bn £273.14bn £279.83bn £286.71bn £293.71bn
Change in VAT from its baseline £0.00bn £0.00bn £0.00bn £0.00bn £0.00bn £0.00bn £0.00bn
VAT on private school fees £0.00bn £0.00bn £0.00bn £0.00bn £0.00bn £0.00bn £0.00bn
Fuel duties £24.53bn £24.64bn £26.64bn £26.69bn £26.42bn £25.76bn £26.35bn
Alcohol duties £0.00bn £0.00bn £0.00bn £0.00bn £0.00bn £0.00bn £0.00bn
Tobacco duties £0.00bn £0.00bn £0.00bn £0.00bn £0.00bn £0.00bn £0.00bn
Vehicle excise duties £0.00bn £0.00bn £0.00bn £0.00bn £0.00bn £0.00bn £0.00bn
TV licence fee £4.28bn £4.43bn £4.44bn £4.46bn £4.49bn £4.51bn £4.53bn
Council tax £49.56bn £52.45bn £55.78bn £59.39bn £62.98bn £66.74bn £70.70bn
High value council tax surcharge £0.00bn £0.00bn £0.00bn £0.48bn £0.50bn £0.51bn £0.52bn
Northern Ireland domestic rates £0.55bn £0.58bn £0.61bn £0.63bn £0.67bn £0.70bn £0.73bn
Business rates £31.73bn £31.73bn £31.73bn £31.73bn £31.73bn £31.73bn £31.73bn
Stamp duty land tax (own purchases, and corporate until the £10.26bn £10.84bn £11.32bn £11.80bn £12.29bn £12.83bn £13.42bn
Land transaction tax (Wales) £0.00bn £0.00bn £0.00bn £0.00bn £0.00bn £0.01bn £0.01bn
Land and buildings transaction tax (Scotland) £0.01bn £0.02bn £0.02bn £0.02bn £0.02bn £0.02bn £0.02bn
Unspecified tax change borne by capital £0.00bn £0.00bn £0.00bn £0.00bn £0.00bn £0.00bn £0.00bn
Unspecified tax change borne by consumers £0.00bn £0.00bn £0.00bn £0.00bn £0.00bn £0.00bn £0.00bn
Wealth tax £0.00bn £0.00bn £0.00bn £0.00bn £0.00bn £0.00bn £0.00bn
Wealth tax excluding the main residence £0.00bn £0.00bn £0.00bn £0.00bn £0.00bn £0.00bn £0.00bn
Land value tax £0.00bn £0.00bn £0.00bn £0.00bn £0.00bn £0.00bn £0.00bn
Carbon tax £0.00bn £0.00bn £0.00bn £0.00bn £0.00bn £0.00bn £0.00bn
Spending by benefit and service: head levels
Measure (head) 2025-26 2026-27 2027-28 2028-29 2029-30 2030-31 2031-32
State Pension £122.50bn £127.12bn £128.28bn £130.21bn £134.05bn £138.02bn £142.08bn
Pension Credit £6.68bn £7.06bn £7.04bn £7.11bn £7.23bn £7.37bn £7.46bn
Winter Fuel Payment £1.70bn £1.68bn £1.59bn £1.57bn £1.57bn £1.55bn £1.52bn
Pension Age Winter Heating Payment (Scotland) £0.19bn £0.20bn £0.20bn £0.21bn £0.21bn £0.22bn £0.22bn
Universal Credit £79.17bn £83.50bn £85.32bn £87.08bn £88.02bn £89.45bn £90.80bn
Housing Benefit £8.60bn £8.93bn £9.20bn £9.34bn £9.62bn £9.88bn £10.17bn
Council Tax Reduction £1.99bn £2.03bn £2.08bn £2.13bn £2.18bn £2.24bn £2.29bn
Employment and Support Allowance (income-related) £2.12bn £2.18bn £2.23bn £2.28bn £2.33bn £2.39bn £2.45bn
Employment and Support Allowance (contribution-based) £5.22bn £5.36bn £5.49bn £5.62bn £5.76bn £5.90bn £6.04bn
Jobseeker's Allowance (income-based) £0.00bn £0.00bn £0.00bn £0.00bn £0.00bn £0.00bn £0.00bn
Jobseeker's Allowance (contribution-based) £0.29bn £0.30bn £0.30bn £0.31bn £0.32bn £0.33bn £0.33bn
Income Support £0.12bn £0.00bn £0.00bn £0.00bn £0.00bn £0.00bn £0.00bn
Working Tax Credit £0.00bn £0.00bn £0.00bn £0.00bn £0.00bn £0.00bn £0.00bn
Child Tax Credit £0.00bn £0.00bn £0.00bn £0.00bn £0.00bn £0.00bn £0.00bn
Incapacity Benefit £0.00bn £0.00bn £0.00bn £0.00bn £0.00bn £0.00bn £0.00bn
Personal Independence Payment £27.05bn £28.19bn £28.94bn £29.64bn £30.37bn £31.11bn £31.87bn
Disability Living Allowance £7.52bn £7.83bn £8.04bn £8.24bn £8.44bn £8.65bn £8.86bn
Attendance Allowance £8.71bn £9.08bn £9.32bn £9.55bn £9.78bn £10.02bn £10.26bn
Carer's Allowance £4.50bn £4.69bn £4.81bn £4.93bn £5.05bn £5.17bn £5.29bn
Carer Support Payment (Scotland) £0.35bn £0.37bn £0.38bn £0.39bn £0.40bn £0.41bn £0.41bn
Carer's Allowance Supplement (Scotland) £0.00bn £0.05bn £0.05bn £0.05bn £0.05bn £0.05bn £0.05bn
Severe Disablement Allowance £0.03bn £0.03bn £0.03bn £0.03bn £0.03bn £0.03bn £0.03bn
Industrial Injuries Disablement Benefit £0.51bn £0.52bn £0.53bn £0.54bn £0.56bn £0.57bn £0.59bn
Armed Forces Compensation Scheme £0.61bn £0.62bn £0.64bn £0.65bn £0.67bn £0.69bn £0.70bn
War Widow's or Widower's Pension £0.00bn £0.00bn £0.00bn £0.00bn £0.00bn £0.00bn £0.00bn
Child Benefit £16.23bn £16.91bn £17.32bn £17.73bn £18.16bn £18.60bn £19.04bn
Scottish Child Payment £0.50bn £0.52bn £0.53bn £0.55bn £0.56bn £0.57bn £0.58bn
Two Child Limit Payment (Scotland, counterfactual) £0.00bn £0.00bn £0.00bn £0.00bn £0.00bn £0.00bn £0.00bn
Healthy Start vouchers £0.05bn £0.05bn £0.05bn £0.05bn £0.05bn £0.05bn £0.05bn
Maternity Allowance £0.00bn £0.00bn £0.00bn £0.00bn £0.00bn £0.00bn £0.00bn
Statutory Maternity Pay £3.32bn £3.41bn £3.49bn £3.57bn £3.66bn £3.75bn £3.84bn
Statutory Paternity Pay £0.00bn £0.00bn £0.00bn £0.00bn £0.00bn £0.00bn £0.00bn
Statutory Sick Pay £1.13bn £1.16bn £1.19bn £1.22bn £1.25bn £1.28bn £1.31bn
Sure Start Maternity Grant £0.01bn £0.01bn £0.01bn £0.01bn £0.01bn £0.01bn £0.01bn
Bereavement Support Payment £0.67bn £0.69bn £0.70bn £0.72bn £0.74bn £0.76bn £0.78bn
Tax-Free Childcare £0.55bn £0.56bn £0.57bn £0.58bn £0.60bn £0.61bn £0.59bn
Extended childcare entitlement £2.65bn £3.09bn £3.17bn £3.24bn £3.33bn £3.41bn £3.49bn
Universal childcare entitlement £3.27bn £3.39bn £3.48bn £3.57bn £3.66bn £3.77bn £3.85bn
Targeted childcare entitlement £0.57bn £0.59bn £0.61bn £0.62bn £0.64bn £0.66bn £0.66bn
Care to Learn £0.00bn £0.00bn £0.00bn £0.00bn £0.00bn £0.00bn £0.00bn
Childcare Grant £0.00bn £0.00bn £0.00bn £0.00bn £0.00bn £0.00bn £0.00bn
Parents' Learning Allowance £0.20bn £0.20bn £0.20bn £0.20bn £0.20bn £0.20bn £0.20bn
Adult Dependants' Grant £0.06bn £0.06bn £0.04bn £0.04bn £0.04bn £0.04bn £0.04bn
Travel Grant £0.00bn £0.00bn £0.00bn £0.00bn £0.00bn £0.00bn £0.00bn
Disabled Students' Allowance suppressed suppressed suppressed suppressed suppressed suppressed suppressed
16 to 19 Bursary Fund £0.00bn £0.00bn £0.00bn £0.00bn £0.00bn £0.00bn £0.00bn
NHS spending £204.68bn £205.46bn £206.22bn £207.04bn £207.96bn £208.89bn £209.79bn
State education spending £0.02bn £0.02bn £0.02bn £0.02bn £0.02bn £0.02bn £0.02bn
Transport subsidy spending £25.15bn £25.25bn £25.34bn £25.44bn £25.55bn £25.67bn £25.78bn
Cost-of-living support payment £0.00bn £0.00bn £0.00bn £0.00bn £0.00bn £0.00bn £0.00bn
Energy Bills Rebate £0.00bn £0.00bn £0.00bn £0.00bn £0.00bn £0.00bn £0.00bn
Energy Price Guarantee subsidy £0.00bn £0.00bn £0.00bn £0.00bn £0.00bn £0.00bn £0.00bn
Basic income £0.00bn £0.00bn £0.00bn £0.00bn £0.00bn £0.00bn £0.00bn
Disability basic income £0.00bn £0.00bn £0.00bn £0.00bn £0.00bn £0.00bn £0.00bn
Other public spending budget change £0.00bn £0.00bn £0.00bn £0.00bn £0.00bn £0.00bn £0.00bn
Caseloads: head levels
Measure (head) 2025-26 2026-27 2027-28 2028-29 2029-30 2030-31 2031-32
State Pension 11,308k 11,208k 10,906k 10,812k 10,860k 10,908k 10,955k
Pension Credit 1,491k 1,489k 1,477k 1,455k 1,457k 1,454k 1,451k
Winter Fuel Payment 7,091k 7,008k 6,593k 6,509k 6,491k 6,377k 6,242k
Pension Age Winter Heating Payment (Scotland) 806k 807k 807k 805k 809k 812k 816k
Universal Credit 6,372k 6,379k 6,409k 6,420k 6,395k 6,387k 6,371k
Housing Benefit 1,254k 1,253k 1,254k 1,240k 1,245k 1,247k 1,249k
Council Tax Reduction 3,187k 3,212k 3,225k 3,247k 3,263k 3,282k 3,296k
Employment and Support Allowance (income-related) 187k 188k 188k 189k 190k 190k 191k
Employment and Support Allowance (contribution-based) 722k 724k 727k 730k 733k 737k 740k
Jobseeker's Allowance (income-based) 0k 0k 0k 0k 0k 0k 0k
Jobseeker's Allowance (contribution-based) 67k 67k 68k 68k 68k 69k 69k
Income Support 25k 0k 0k 0k 0k 0k 0k
Working Tax Credit 0k 0k 0k 0k 0k 0k 0k
Child Tax Credit 0k 0k 0k 0k 0k 0k 0k
Incapacity Benefit 0k 0k 0k 0k 0k 0k 0k
Personal Independence Payment 3,581k 3,595k 3,608k 3,623k 3,639k 3,655k 3,671k
Disability Living Allowance 1,296k 1,301k 1,306k 1,311k 1,317k 1,323k 1,329k
Attendance Allowance 1,701k 1,708k 1,714k 1,721k 1,729k 1,736k 1,744k
Carer's Allowance 1,038k 1,042k 1,046k 1,050k 1,055k 1,059k 1,064k
Carer Support Payment (Scotland) 81k 82k 82k 82k 83k 83k 83k
Carer's Allowance Supplement (Scotland) 0k 82k 82k 82k 83k 83k 83k
Severe Disablement Allowance 4k 4k 4k 4k 4k 4k 4k
Industrial Injuries Disablement Benefit 137k 138k 139k 139k 140k 140k 141k
Armed Forces Compensation Scheme 94k 94k 94k 95k 95k 95k 96k
War Widow's or Widower's Pension 0k 0k 0k 0k 0k 0k 0k
Child Benefit 8,136k 8,165k 8,185k 8,215k 8,249k 8,282k 8,314k
Scottish Child Payment 352k 354k 355k 356k 357k 358k 357k
Two Child Limit Payment (Scotland, counterfactual) 0k 0k 0k 0k 0k 0k 0k
Healthy Start vouchers 200k 201k 202k 203k 204k 204k 205k
Maternity Allowance 0k 0k 0k 0k 0k 0k 0k
Statutory Maternity Pay 334k 335k 336k 338k 339k 341k 342k
Statutory Paternity Pay 0k 0k 0k 0k 0k 0k 0k
Statutory Sick Pay 182k 183k 183k 184k 185k 186k 186k
Sure Start Maternity Grant 26k 26k 26k 27k 27k 27k 27k
Bereavement Support Payment 98k 98k 99k 99k 100k 100k 101k
Tax-Free Childcare 1,017k 1,028k 1,028k 1,033k 1,034k 1,038k 1,023k
Extended childcare entitlement 665k 671k 675k 677k 682k 686k 687k
Universal childcare entitlement 986k 989k 992k 997k 1,002k 1,013k 1,015k
Targeted childcare entitlement 119k 120k 120k 121k 121k 122k 121k
Care to Learn 0k 0k 0k 0k 0k 0k 0k
Childcare Grant 0k 0k 0k 0k 0k 0k 0k
Parents' Learning Allowance 98k 99k 99k 100k 100k 100k 101k
Adult Dependants' Grant 18k 18k 12k 13k 13k 13k 13k
Travel Grant 0k 0k 0k 0k 0k 0k 0k
Disabled Students' Allowance suppressed suppressed suppressed suppressed suppressed suppressed suppressed
16 to 19 Bursary Fund 0k 0k 0k 0k 0k 0k 0k
Cost-of-living support payment 0k 0k 0k 0k 0k 0k 0k
Energy Bills Rebate 0k 0k 0k 0k 0k 0k 0k
Energy Price Guarantee subsidy 0k 0k 0k 0k 0k 0k 0k
Basic income 0k 0k 0k 0k 0k 0k 0k
Disability basic income 0k 0k 0k 0k 0k 0k 0k
Government balance, each component counted once: head levels
Measure (head) 2025-26 2026-27 2027-28 2028-29 2029-30 2030-31 2031-32
Government receipts, each component once £898.60bn £938.62bn £975.41bn £1,008.49bn £1,044.08bn £1,080.93bn £1,121.92bn
Government spending £536.94bn † £551.10bn † £557.43bn † £564.51bn † £573.07bn † £582.33bn † £591.49bn †
Government balance, each receipt once £361.67bn † £387.52bn † £417.98bn † £443.98bn † £471.01bn † £498.60bn † £530.43bn †
Benefits in household_benefits but not in gov_spending £0.00bn £0.00bn £0.00bn £0.00bn £0.00bn £0.00bn £0.00bn
Model totals: head levels
Measure (head) 2025-26 2026-27 2027-28 2028-29 2029-30 2030-31 2031-32
gov_tax £901.73bn £941.75bn £978.54bn £1,011.62bn £1,047.21bn £1,084.06bn £1,125.05bn
household_tax £495.38bn £522.04bn £547.68bn £569.74bn £591.81bn £616.26bn £643.12bn
pre_budget_change_household_tax £487.56bn £513.92bn £539.33bn £560.70bn £582.53bn £606.72bn £633.22bn
gov_spending £536.94bn † £551.10bn † £557.43bn † £564.51bn † £573.07bn † £582.33bn † £591.49bn †
household_benefits £536.94bn † £551.10bn † £557.43bn † £564.51bn † £573.07bn † £582.33bn † £591.49bn †
gov_balance £364.80bn † £390.65bn † £421.11bn † £447.11bn † £474.14bn † £501.73bn † £533.56bn †
Components the model counts twice: head levels
Measure (head) 2025-26 2026-27 2027-28 2028-29 2029-30 2030-31 2031-32
corporate_sdlt, counted again inside expected_sdlt £3.13bn £3.13bn £3.13bn £3.13bn £3.13bn £3.13bn £3.13bn
Household income totals: head levels
Measure (head) 2025-26 2026-27 2027-28 2028-29 2029-30 2030-31 2031-32
Household market income £1,657.06bn £1,719.25bn £1,771.68bn £1,821.99bn £1,887.41bn £1,945.59bn £2,017.37bn
Household net income £1,659.34bn † £1,707.55bn † £1,739.55bn † £1,773.81bn † £1,810.06bn † £1,851.71bn † £1,903.31bn †
Household net income (HBAI definition) £1,468.17bn £1,514.88bn £1,547.36bn £1,580.55bn £1,614.82bn £1,654.01bn £1,704.49bn
Population: head levels
Measure (head) 2025-26 2026-27 2027-28 2028-29 2029-30 2030-31 2031-32
People 69,514k 69,778k 70,036k 70,316k 70,626k 70,944k 71,249k
Benefit units 35,247k 35,381k 35,512k 35,654k 35,811k 35,972k 36,127k
Households 31,167k 31,285k 31,401k 31,526k 31,665k 31,808k 31,944k
Poverty rates: head levels
Measure (head) 2025-26 2026-27 2027-28 2028-29 2029-30 2030-31 2031-32
Absolute poverty, before housing costs: everyone 15.9% 14.9% 15.5% 15.6% 15.7% 15.8% 15.7%
Absolute poverty, before housing costs: children 23.7% 20.9% 21.6% 21.4% 21.6% 21.8% 21.7%
Absolute poverty, before housing costs: working-age adults 13.8% 13.4% 14.0% 14.1% 14.2% 14.3% 14.3%
Absolute poverty, before housing costs: pensioners 13.6% 13.1% 13.2% 13.4% 13.5% 13.6% 13.5%
Absolute poverty, after housing costs: everyone 20.9% 20.4% 21.0% 21.2% 21.4% 21.7% 21.8%
Absolute poverty, after housing costs: children 31.8% 30.0% 30.7% 31.0% 31.2% 31.6% 31.8%
Absolute poverty, after housing costs: working-age adults 19.4% 19.2% 20.0% 20.2% 20.5% 20.7% 20.8%
Absolute poverty, after housing costs: pensioners 12.5% 12.2% 12.0% 12.2% 12.2% 12.3% 12.3%
Relative poverty, before housing costs: everyone 17.0% 16.4% 16.5% 16.6% 16.8% 16.8% 16.9%
Relative poverty, before housing costs: children 25.7% 23.5% 23.3% 23.2% 23.4% 23.5% 23.7%
Relative poverty, before housing costs: working-age adults 14.5% 14.4% 14.7% 14.8% 15.1% 15.1% 15.2%
Relative poverty, before housing costs: pensioners 14.8% 14.9% 14.4% 14.4% 14.3% 14.4% 14.6%
Relative poverty, after housing costs: everyone 21.5% 21.1% 21.3% 21.4% 21.5% 21.6% 21.8%
Relative poverty, after housing costs: children 32.6% 31.3% 31.1% 31.1% 31.4% 31.4% 31.8%
Relative poverty, after housing costs: working-age adults 19.8% 19.8% 20.2% 20.3% 20.5% 20.7% 20.8%
Relative poverty, after housing costs: pensioners 13.3% 12.8% 12.6% 12.7% 12.3% 12.2% 12.3%
People in poverty: head levels
Measure (head) 2025-26 2026-27 2027-28 2028-29 2029-30 2030-31 2031-32
Absolute poverty, before housing costs: everyone 11,050k 10,426k 10,847k 10,945k 11,075k 11,205k 11,215k
Absolute poverty, before housing costs: children 3,559k 3,140k 3,260k 3,252k 3,297k 3,333k 3,333k
Absolute poverty, before housing costs: working-age adults 5,864k 5,737k 6,070k 6,164k 6,236k 6,314k 6,319k
Absolute poverty, before housing costs: pensioners 1,627k 1,549k 1,517k 1,528k 1,542k 1,558k 1,563k
Absolute poverty, after housing costs: everyone 14,536k 14,210k 14,688k 14,922k 15,145k 15,380k 15,517k
Absolute poverty, after housing costs: children 4,765k 4,513k 4,641k 4,697k 4,753k 4,831k 4,894k
Absolute poverty, after housing costs: working-age adults 8,275k 8,251k 8,665k 8,836k 8,991k 9,138k 9,200k
Absolute poverty, after housing costs: pensioners 1,496k 1,445k 1,382k 1,389k 1,401k 1,411k 1,423k
Relative poverty, before housing costs: everyone 11,807k 11,456k 11,570k 11,652k 11,832k 11,919k 12,065k
Relative poverty, before housing costs: children 3,863k 3,538k 3,520k 3,528k 3,567k 3,592k 3,642k
Relative poverty, before housing costs: working-age adults 6,178k 6,160k 6,389k 6,483k 6,623k 6,674k 6,741k
Relative poverty, before housing costs: pensioners 1,767k 1,758k 1,661k 1,642k 1,642k 1,653k 1,682k
Relative poverty, after housing costs: everyone 14,918k 14,717k 14,897k 15,056k 15,215k 15,345k 15,518k
Relative poverty, after housing costs: children 4,886k 4,714k 4,695k 4,721k 4,789k 4,814k 4,894k
Relative poverty, after housing costs: working-age adults 8,442k 8,497k 8,756k 8,884k 9,018k 9,123k 9,201k
Relative poverty, after housing costs: pensioners 1,590k 1,507k 1,446k 1,451k 1,408k 1,408k 1,423k
Inequality: head levels
Measure (head) 2025-26 2026-27 2027-28 2028-29 2029-30 2030-31 2031-32
Gini coefficient, equivalised HBAI income before housing costs 0.3076 0.3053 0.3067 0.3074 0.3079 0.3086 0.3096
Mean household net income in each side's own deciles: head levels
Measure (head) 2025-26 2026-27 2027-28 2028-29 2029-30 2030-31 2031-32
Decile 1: mean household_net_income per household £17,701 † £17,986 † £17,930 † £18,005 † £18,312 † £18,507 † £18,668 †
Decile 2: mean household_net_income per household £28,345 † £29,006 † £28,643 † £28,959 † £29,297 † £29,667 † £30,264 †
Decile 3: mean household_net_income per household £34,596 † £35,840 † £36,513 † £36,996 † £37,351 † £38,068 † £38,716 †
Decile 4: mean household_net_income per household £39,340 † £40,981 † £41,875 † £42,608 † £43,161 † £43,755 † £44,885 †
Decile 5: mean household_net_income per household £44,541 † £46,019 † £46,949 † £47,578 † £48,334 † £49,282 † £49,907 †
Decile 6: mean household_net_income per household £48,356 † £49,451 † £50,382 † £51,209 † £52,330 † £53,133 † £54,591 †
Decile 7: mean household_net_income per household £56,413 † £57,572 † £58,474 † £59,440 † £60,063 † £61,464 † £63,069 †
Decile 8: mean household_net_income per household £69,292 † £70,485 † £71,702 † £73,027 † £74,348 † £75,820 † £77,601 †
Decile 9: mean household_net_income per household £79,707 † £82,059 † £83,307 † £84,456 † £86,255 † £87,808 † £90,034 †
Decile 10: mean household_net_income per household £117,835 † £120,650 † £122,572 † £124,724 † £127,210 † £129,846 † £132,991 †
Standard reform costings (change in gov_balance): head levels
Measure (head) 2025-26 2026-27 2027-28 2028-29 2029-30 2030-31 2031-32
Raise basic rate by 1pp £6.22bn £6.50bn £6.71bn £6.90bn £7.11bn £7.35bn £7.62bn
Raise higher rate by 1pp £3.54bn £3.82bn £4.05bn £4.27bn £4.49bn £4.76bn £4.96bn
Raise personal allowance by ~800GBP/year £-3.84bn £-3.98bn £-4.06bn £-4.11bn £-4.20bn £-4.32bn £-2.04bn
Raise child benefit by 25GBP/week per additional child £-2.00bn £-1.76bn £-1.64bn £-1.54bn £-1.44bn £-1.34bn £-1.24bn
Reduce Universal Credit taper rate to 20% £-20.54bn £-21.10bn £-21.45bn £-21.88bn £-21.96bn £-22.33bn £-22.81bn
Raise Class 1 main employee NICs rate to 10% £11.06bn £11.51bn £11.83bn £12.14bn £12.46bn £12.80bn £13.27bn
Raise VAT standard rate by 2pp £25.01bn £25.68bn £26.29bn £26.92bn £27.58bn £28.26bn £28.95bn
Raise additional rate by 3pp £2.83bn £3.07bn £3.26bn £3.46bn £3.66bn £3.89bn £4.21bn
Raise the Universal Credit standard allowance by 10% £-3.04bn £-3.17bn £-3.25bn £-3.32bn £-3.37bn £-3.43bn £-3.47bn
Raise the State Pension by 10% £-8.93bn £-9.14bn £-9.06bn £-9.12bn £-9.31bn £-9.55bn £-9.82bn
Standard reforms: change in each reform's target: head levels
Measure (head) 2025-26 2026-27 2027-28 2028-29 2029-30 2030-31 2031-32
Raise basic rate by 1pp: change in income_tax £6.36bn £6.65bn £6.87bn £7.07bn £7.29bn £7.53bn £7.81bn
Raise higher rate by 1pp: change in income_tax £3.54bn £3.83bn £4.06bn £4.28bn £4.50bn £4.77bn £4.97bn
Raise personal allowance by ~800GBP/year: change in income_tax £-3.97bn £-4.12bn £-4.22bn £-4.27bn £-4.36bn £-4.48bn £-1.79bn
Raise child benefit by 25GBP/week per additional child: change in child_benefit £2.34bn £2.15bn £2.03bn £1.92bn £1.82bn £1.71bn £1.59bn
Reduce Universal Credit taper rate to 20%: change in universal_credit £20.53bn £21.09bn £21.44bn £21.87bn £21.95bn £22.32bn £22.80bn
Raise Class 1 main employee NICs rate to 10%: change in national_insurance £11.33bn £11.79bn £12.13bn £12.44bn £12.77bn £13.12bn £13.61bn
Raise VAT standard rate by 2pp: change in vat £25.01bn £25.68bn £26.29bn £26.92bn £27.58bn £28.26bn £28.95bn
Raise additional rate by 3pp: change in income_tax £2.83bn £3.07bn £3.26bn £3.46bn £3.66bn £3.89bn £4.21bn
Raise the Universal Credit standard allowance by 10%: change in universal_credit £3.04bn £3.17bn £3.25bn £3.32bn £3.38bn £3.43bn £3.48bn
Raise the State Pension by 10%: change in state_pension £12.25bn £12.71bn £12.83bn £13.02bn £13.41bn £13.80bn £14.21bn

MaxGhenis and others added 3 commits October 11, 2026 19:44
Counted-once check. It judged each total's net movement, which a means-test
fall, or the eligibility-jump exclusion, could use to hide a double count.
It now reads each head's response household by household:
- a component's own head must rise by exactly £1;
- no other tax head may move;
- no other benefit may rise in any household (rises and falls are not
  netted);
- the model's own total must move by what its heads moved, plus one for a
  declared double count.
That last rule catches a total whose formula adds a zero component twice.
A head that declares a read may respond, but not one for one. The snapshot
of supplied inputs is taken after formula results are discarded, so an
earlier result is never frozen as an input.

Comparison.
- A missing or crashed base now fails.
- A check failing in a year where the base passes is a regression, even if
  it already failed in another year.
- A check the base ran that is missing from the head fails (or is noted,
  when the partition changed), and the head must have run the core checks.
- A figure on one side only is shown as added or removed, never as no
  change.
- A non-finite head figure fails directly.
- Households must match one to one.

Measurement.
- Finiteness is checked after the reform costings.
- A rebuilt baseline must match in every year.
- A skipped check is recorded as skipped.
- metrics_version covers the effective spec, horizon and reforms.
- Geographic exclusivity guards are checked on the data.

Test data.
- Data tests are identified by their marker, so the data-free tests under
  microsimulation/ run without credentials.
- The setup hook runs before pytest's skip handling, so a skip marker
  cannot turn missing data into a skip.
- The latest-data smoke test is a named exception to the pin.

The partition's notes now describe main before the #2159 fix correctly.

Refs #2334.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Qualification compares a candidate's report with the previous pair's. With
only a status per check, a check that fails in more years than before looks
unchanged; failing_scopes lets it be told apart.

Refs #2334.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The tool digest was read from disk when a measurement finished, so editing
the tool during a run stamped the result with code that had not produced
it. It is now taken when the module loads.

Refs #2334.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@MaxGhenis

Copy link
Copy Markdown
Collaborator Author

Two fixes from this PR's W3 report change the lists fiscal_partition.yaml closes against. Here is what the partition needs once each lands (checked against this PR's head 9b33fe1):

  1. Count Scottish Child Payment, Healthy Start and the Two Child Limit Payment in government spending #2366 (fixes Scottish Child Payment and Healthy Start are household benefits but missing from government spending #2338) adds scottish_child_payment, healthy_start_vouchers and two_child_limit_payment to GOV_SPENDING_VARIABLES.
  2. Count statutory sick, maternity and paternity pay as Universal Credit earned income #2365 (fixes Universal Credit rises with SSP and SMP: tax on them is deducted but they aren't counted as earnings #2337) adds statutory_paternity_pay next to statutory_maternity_pay in four lists: HOUSEHOLD_BENEFIT_VARIABLES, GOV_SPENDING_VARIABLES, HBAI_HOUSEHOLD_NET_INCOME_ADDS and hbai_benefits.adds.
    • The partition needs a statutory_paternity_pay spending head with totals: [gov_spending, household_benefits], like the SSP and SMP heads.
    • uk-data never fills SPP, so no survey aggregate moves.

Neither PR touches tests/conftest.py, aggregate-impact.yaml or reforms_config.yaml.

…hanges

Merges origin/main (1a31ef1).

- conftest: main's --shard selection and shard header now sit in the same
  hooks as the data plan, instead of two definitions of each hook. The pin
  is found through the installed package, so CI's tests that copy conftest
  into a scratch directory still load it.
- Pin: the certified pair is now policyengine-uk 2.130.1 with uk-data
  1.58.1 (policyengine.py bundle uk-6.4.3): enhanced_frs_2024_25.h5 at
  Hugging Face commit 2526edbb, sha256 056e80fd. The reform-cost fixtures
  are recomputed on it with this model (2.132.6), each new value added as a
  receipt.
- Partition: Scottish Child Payment, Healthy Start and the Two Child Limit
  Payment are now in gov_spending (#2366), Statutory Paternity Pay is a new
  spending head (#2365), and the Pension Age Winter Heating Payment is in
  household_benefits (#2047). Closure caught each one.
- The SSP/SMP Universal Credit interaction is fixed on main (#2365), so the
  certified test no longer expects it to fail.
- The workflow test moves to code_health/ and skips in an installed wheel,
  which has no .github, as the other CI-script tests do.
- The dataset is projected to 2032 (#2376), so 2031-32 is no longer
  extrapolated differently; the README says so.

Refs #2334.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
MaxGhenis and others added 2 commits October 15, 2026 06:37
Re-review fixes:
- The counted-once guard judges each household: its own head rises by
  exactly what was added, no other tax head moves, no other benefit rises,
  and the model total moves by its heads. Readers and zero-baseline
  benefits use a two-point slope test, so a passport (slope 0) is told apart
  from containment (slope 1).
- A head that breaks a rule, or a reader with a linear response that is not
  one for one, is measured again with every other head held at baseline.
  Responses carried only by another head are reported as mediated, not
  failed. Incapacity Benefit raises Universal Credit only through the
  Carer's Allowance overlap, which reflects a missing UC income source
  (#2424), not a double count.
- Components that are non-zero only in zero-weight households are skipped.
- Check details suppress cells below 10 records; the geography guard nets
  declared double counts out of their containers.
- Closure, counted-once and responsiveness checks are scoped per problem
  and per year, so a comparison sees new failures, not just "still fails".
- The tool digest is taken at load; the definitions digest covers parsed
  spec fields; household detail is written under private/.
- The workflow posts a fallback comment when the comparison crashes and
  always replaces an earlier comment.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Each standard reform's minimum is now a fifth of its smallest yearly
effect over 2025-26 to 2031-32 on uk-data 1.58.1, measured with
policyengine-uk 2.130.1 on core 3.33.0 (the certified model) and with
2.132.6 on core 3.33.2; every target change was identical between the
two. Minimums are rounded down to two significant figures, and
parse_reforms refuses a minimum that does not follow from its recorded
smallest_effect. A new `calibrate` command recomputes both from fresh
measurements.

The personal allowance minimum falls from £1bn to £0.35bn: its smallest
year, 2031-32, is worth £1.78bn, so the old minimum was 56% of the
effect. VAT's rises from £4bn to £6.1bn.

The flag thresholds are kept and their basis recorded: flags only order
the comment, every change is listed, and runs are deterministic.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
MaxGhenis and others added 8 commits October 15, 2026 07:51
- A head total or caseload built from fewer than 10 records is published
  as suppressed (privacy.py): one recipient's total and caseload give the
  award and the weight. On certified 1.58.1 this hides Disabled Students'
  Allowance. Compare, render and the W4 long format carry the flag; a
  suppressed figure is neither non-finite nor flagged.
- Counted-once messages, problem counts and shares, mediated counts and
  perturbed-unit counts keep no figure under 10 in the published result.
- The direct-effect step also holds the other heads at their values after
  the first point while the component moves to the second, so a head
  gated by a passport on another head (h = c * (b > 0)) keeps its slope
  and fails. Mediated records list the other heads that moved, not a
  proven route; the documented limit covers falls that outweigh a rise.
- The PR comment is replaced on any failed run, even before the
  comparison.
- Wording: the half alternative for reform minimums, the single-minimum
  weakness, the VAT receipts ratio by data version, and the 1% reform
  flag threshold.

On certified 1.58.1 (2026-27) the guard gives the same verdicts: 314
pass, 15 warn, 1 fail (sign:vat, pre-existing).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sol review of #2425: a head from fewer than 10 records was suppressed,
but a total containing it, less the published heads, gave it back (one
rare £123.45 benefit with weight 2 recovered as £246.90). Each published
total that contains a suppressed head in a year (model totals, balance
aggregates, gov_balance and the household income totals) is now rounded
to the nearest £1bn, or suppressed if the head is worth £0.5bn or more.
Totals without one stay exact; the W4 long format flags rounded rows.

On certified 1.58.1 Disabled Students' Allowance is suppressed in every
year, so spending, benefit and balance totals are published to the
nearest £1bn.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Decile rows publish no household weight or base income, round changes
  to what the comment shows, and publish no change where fewer than 10
  weighted households change. In a year with a suppressed head, decile
  mean incomes are rounded to £1,000.
- A rounded total's exact change is read from each side's private detail
  and published only when no suppressed head inside it changed; otherwise
  it is shown as withheld and not flagged. A suppressed head never shows
  a change.
- Records, perturbed units and problem households count only records
  with weight.
- A failing determinism check gives no amounts.
- A component perturbed in fewer than 10 units publishes no responses,
  coefficients or shares.
- The comment describes the second hold, and a head-only suppressed figure
  shows as suppressed.
- The VAT receipts comment says only the 62% is a same-model ratio.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ostics (Sol re-review 4)

- A protected total's exact change is withheld when any suppressed head
  inside it, or the total itself, is suppressed on one side only.
- Identity and overlap diagnostics count only households with weight.
- Decile relative changes are published to 0.1%, so the all row can't pin
  the base income total to within £0.5bn.
- Decile changes that round to £0 are described as such, not as no change.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… re-review 5)

Rounding a total that contains a suppressed head let repeated reports
narrow the head down: each rounded level less the exact published heads
bounds it. Every total containing a suppressed head is now published
without it that year, with the heads it leaves out and their signs under
excludes; no published figure depends on the head at all.

- Signs follow each total: model and balance totals add their heads,
  gov_balance subtracts spending heads, household net income adds benefits
  and subtracts taxes. Market income holds no head. The HBAI total resets
  negative incomes, so it is recomputed household by household before the
  reset, or suppressed if the model can't support that.
- Decile incomes leave the suppressed heads out per household, so decile
  levels and changes carry none of them; the decile rounding is gone.
- The comparison aligns sides that leave out different heads using
  published values only, so the private hidden.json and private_delta are
  gone. The comment marks such totals with a dagger, and the W4 format
  lists the excluded heads.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…view 6)

- A total is published without a suppressed head only when every identity
  behind it holds that year; otherwise it is suppressed, since a formula
  that scales its components would let total less head reveal the head.
  Decile incomes follow the same rule.
- The HBAI total is rebuilt only where its component lists reproduce the
  model's income household by household (council tax abolished, say).
- Poverty and inequality are measured on HBAI income: in a year when a
  suppressed head is part of it, they are suppressed too.
- A standard reform that moves a suppressed head in a year publishes
  neither its costing nor its target change for that year, and its
  responsiveness detail gives no amount.
- The comparison lines up different exclusions only for additive totals,
  and withholds decile rows when the two sides' incomes leave out
  different heads.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Totals holding a suppressed head or double count are rebuilt household by
  household from their other components, never as the model's total less the
  suppressed one, so no tolerance or scaling formula leaves a residual.
- Declared double counts get the same small-cell suppression as heads.
- Identity checks and the HBAI reproduction check leave out households that
  hold a suppressed figure, so their verdicts and details can't depend on it.
- Deciles, poverty and the Gini are withheld while HBAI income holds one.
- Reform figures for a year are withheld when the reform changes any record
  of a suppressed component, leaves a new small cell, or changes gov_balance
  or its target in 1 to 9 households.
- Withheld comparisons render as "withheld", not "no change"; decile notes
  show when every year is withheld.
- privacy.py states the threat model and the d1601 residual.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
MaxGhenis added a commit that referenced this pull request Oct 16, 2026
… after the model loaded

Merges aggregate-impact-report at 0924449 (components rebuilt from their
published parts, double counts protected, reform figures withheld from small
cells, withheld comparisons shown as such). Conflicts were the decile branch
in compare(), the empty-deciles text in render, and the appended tests; both
sides are kept.

Sol re-review 7 of #2425, P3 6: `python -m policyengine_uk.utils.aggregate_impact`
imports policyengine_uk, which builds the model and caches its parameters
before the command can fingerprint its files. `measure` now records
`model_loaded_before_fingerprint`; the data axis refuses a side where it is
true or missing, and the model axis warns. The README's data-axis commands
use a copy of the tool, as CI does.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- The sign check of a head from fewer than 10 records isn't run that year,
  and its caseload is suppressed with it: either would say whether its few
  records are negative.
- A count of fewer than 10 non-finite values is given as "fewer than 10".
- Overlap guards leave out households holding a suppressed figure, as the
  identities do.
- The published guard drops a few-unit double count's coefficient, and the
  double_count_present detail comes from it.
- A comparison withheld in every year says why instead of "household detail
  missing".
- privacy.py says "no published total includes it" and names the deliberate
  exceptions (formula responses, the counted-once verdict, no dominance rule).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
MaxGhenis added a commit that referenced this pull request Oct 16, 2026
Conflict in compare(): the withheld years are now worked out before the
axis branch, and the data axis keeps its own note.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
MaxGhenis and others added 2 commits October 16, 2026 00:52
…view 9)

The published guard drops a few-unit double count's coefficient, and the
comment formatted it as a number. It now says it isn't shown, and the test
renders the stripped guard. privacy.py names the double-count-present
verdict alongside the counted-once one.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- The certified household-detail test treats `excludes` as metadata, not a
  per-household column.
- `excludes` is set through a helper that rebinds the record's map. Main's
  cached-array scan read `x.setdefault("excludes", {})[year] = ...` as a write
  into a simulation array: a call whose first argument is a string looks
  like `entity("variable", period)`.
- The W4 long format gains `exclude_signs` ({metric id: +1 or -1}) beside
  `excludes`, so additive totals with different exclusions can be lined up
  from published figures.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@MaxGhenis

Copy link
Copy Markdown
Collaborator Author

Max ruled d1601: A (2026-10-17 13:23Z, recorded in cos; relayed by the UK hub).

publish exact base-to-head changes as policyengine-api already does; level protection stays; covers changes between model versions and between datasets (#2425)

This unblocks this PR for its 10/19 live date, and #2425 after it. When it's ready, hand it to the UK hub with a handoffs.jsonl line. uk-data's #602 pins #2425's head 6406b2ce, and the batch-3 release (d1621, go given 10/17) is being assembled with that pin. If #2425's head moves before the release build, the hub needs to know.

@MaxGhenis

Copy link
Copy Markdown
Collaborator Author

The UK hub is adopting this PR (d1280 fallback: the W3 session that owned it is no longer live). Fleet ops asked that it land before 10/19.

  • d1601 is ruled A (Max, 2026-10-17): exact base-to-head changes are published, as in policyengine-api, and level protection stays. This head already behaves that way, so no behaviour changes. Only the "still open" wording in privacy.py and in this body will be updated.

  • Main is about 234 commits ahead of this branch's base. The adoption build merges main, then re-runs on the merged tree:

    • the fiscal-partition closure checks;
    • the reform calibration;
    • the data tests, against the pinned certified 1.58.1 file.

    It will report any partition or receipt change, with the commit that caused it.

  • Then an independent review at the new head, then a head-pinned merge-queue enqueue. Compare two datasets on one model in the aggregate-impact tool #2425 follows after this lands.

@MaxGhenis

MaxGhenis commented Oct 18, 2026 •

Copy link
Copy Markdown
Collaborator Author

Adoption handoff at 7ac83a4cdf4ee72812bdc7157ae4935bd4599578, pushed once successfully. Still Draft for the UK hub's independent review and CI. #2425 is untouched.

Merged main 62a0117f9 (2.135.6), recorded d1601 A as text only, and recalibrated on the verified certified uk-data 1.58.1 bytes. All five component lists are unchanged, so no partition edit was needed. Appended eight receipts and refreshed three expected centers without changing tolerances. Only the rule-derived higher-rate minimum (£710m→£700m) and additional-rate minimum (£550m→£560m) changed; the other eight are unchanged. The body contains the receipts, causes and evidence.

Validation:

  • 192 focused + 33 mandatory pinned-data tests passed (225 distinct nodes), including the workflow/privacy and aggregates-only tests. Post-calibration reform-unit coverage is 19 passes plus the exact failed node's passing seeded retry; the original Hypothesis input-generation health-check failure remains recorded.
  • All seven years 2025-26 through 2031-32 executed. Closure, identities, determinism and all ten reform responses pass every year. Six years pass all 83 counted-once components; 2029 has 82 passes and one retained State Pension→UC rise finding. Every declared double-count check and exact baseline restoration passes. No all-seven-year pass is claimed.
  • make format/lint passed before commits; final uvx ruff format --check . passed (1779 files).

Open items for the hub:

  • Concurrent main advance blocks CI/merge. After validation against the requested 62a0117f9 (2.135.6), main advanced to 7ad002f54 (2.135.7; Split VAT grossing into household share and survey coverage #2187 VAT coverage / Give War Widow's Pension its own CPI-uprated variable ahead of freezing BSP #2382 BSP / War Widow uprating). A read-only merge check confirms a conflict in reforms_config.yaml. The hub must reconcile the receipts and recalibrate/test that newer model; this evidence covers 2.135.6. .job/MAIN-ADVANCED.md and the exact fixture diff are retained. The one allowed push is already complete.
  • The protected 2029 finding is consistent with existing main #1947. An executed fabricated real-formula MOCK reproduces pension +£1→UC +£0.10 through Marriage Allowance rounding. The affected certified records' exact route was not traced, so attribution remains provisional. .job/ISSUE-state-pension-uc.md records evidence and limits; no guard relaxation or main-model fix was made.
  • The existing negative-VAT/data uk-data #565 finding remains documented with an executed MOCK. Do not file duplicate issues without reconciling these existing reports.
  • Fix evidence passed; no Code/Docs/Aggregate impact run exists at this head because of the merge conflict. Independent hub review remains pending. The complete repository suite and optional local main/head comparison were not run; the hosted aggregate comparison is blocked until the new-main conflict is resolved. National neutrality was not measured locally.

.job/HANDOFF.md, the one-line handoff record and all aggregate evidence are in the assigned workspace. .job/adopt-2335.bundle is verified recovery for this exact head; source .job/env.sh for its workspace-local Git metadata. All managed test/simulation processes have ended and .hypothesis/ was removed. No merge or readiness action was taken.

@MaxGhenis

Copy link
Copy Markdown
Collaborator Author

Round 2 handoff at a6f736468a752ddf8a63f3735f261b24e603b041, pushed once successfully on attempt 1. Still Draft for the UK hub's independent review and CI. #2425 is untouched; no merge or readiness action was taken.

Merged main 7ad002f549680a9ca024c896532697d532a3a92a (2.135.7) in 7717438bf; the final pre-push fetch confirmed that SHA was unchanged. Resolved reforms_config.yaml while preserving all 40 PR receipts and main's complete new VAT fixture in its original format. Appended eight fresh 2029 receipts for 2.135.7 on verified certified uk-data 1.58.1 (49 historical/current receipt mappings total).

  • Partition: added only war_widows_pension to both spending totals, following f1a01e4d5 (Give War Widow's Pension its own CPI-uprated variable ahead of freezing BSP #2382). All five static/runtime closure checks pass; component counts 26/27/17/55/54. VAT Split VAT grossing into household share and survey coverage #2187 changes formulas but no component list. Corporate SDLT's declaration stays.
  • Calibration: official CLI passed and all ten outputs match. Only VAT changes: expected centre £34.0bn → £27.6bn, minimum £6.1bn → £5.0bn, smallest yearly target effect £30.863bn → £25.006bn (2025–26). Its 18.98% fall is the only >10% shift since 2.135.6 and matches Split VAT grossing into household share and survey coverage #2187 (98ab77093), which changes the VAT divisor 0.38 → 0.7 × 0.67 = 0.469, within float32 precision. Other nine minimums, all parameters and all tolerances stay.
  • Fresh tests: 192 focused/workflow + 33 mandatory pinned-data tests passed, 225 distinct nodes, no failures/errors/skips; data run serial with POLICYENGINE_UK_REQUIRE_TEST_DATA=1. Certified SHA256 verified before runs. make format/lint and uvx ruff format --check . passed (1785 files).
  • 2029–30 counted-once: 83 pass / 1 retained failure. The protected State Pension→UC failure signature and every common-component status match round one; the added War Widow component passes. Declared double counts and exact baseline restoration pass. The guard command exits 1 for that finding; no guard pass is claimed. Main #1947 attribution remains provisional because affected certified-record routes are untraced.
  • The seven-year calibration measurement has complete reform-target coverage and passing closure/identity/determinism/responsiveness checks. The full seven-year counted-once rerun was skipped as requested; only 2029 was rerun. Existing negative VAT uk-data #565 persists. War Widow's Pension is inert on this pin.

The PR body now records current-main receipts, minimums, causes and tests. Independent hub review and hosted CI remain pending; the full repository suite and optional local main/head comparison were not run. d1601 A remains unchanged: exact changes are published and level protection stays.

.job/HANDOFF.md, .job/handoffs.jsonl, aggregate evidence and verified .job/adopt-2335-r2.bundle are in the assigned workspace. Source .job/env.sh for the workspace-local Git metadata. All managed test/simulation jobs have ended; temporary UV/Hypothesis caches are being removed before the job ends. No dataset build or survey-record artifact was produced.

@MaxGhenis
MaxGhenis marked this pull request as ready for review October 18, 2026 22:15
@MaxGhenis
MaxGhenis added this pull request to the merge queue Oct 18, 2026
Merged via the queue into main with commit 772198d Oct 18, 2026
17 of 18 checks passed
@MaxGhenis
MaxGhenis deleted the aggregate-impact-report branch October 18, 2026 23:02
@MaxGhenis

Copy link
Copy Markdown
Collaborator Author

Landed by the UK hub through the merge queue, pinned to head a6f736468a752ddf8a63f3735f261b24e603b041. The hub adopted this PR on 10/18 under d1280's fallback, because the W3 session that owned it is no longer live.

  • Gates at this head:
    • gh pr checks exits 0, including the Aggregate impact run.
    • Mergeable; the hub marked it ready.
    • No CHANGES_REQUESTED review.
  • Reviews: independent GPT-6.1 Sol reviews.
    • W3's chain of 12 ended with re-review 11 approving 1191ac5a4, with no findings.
    • The hub's delta review approved this head with no findings (Subfleet 20261018-173414-uk2335-review). It covered both merges of main (62a0117f9, then 7ad002f54):
      • every historical receipt is preserved, 49 in all, and main's VAT fixture is kept in its own schema;
      • independent AST and runtime closure passes for all five totals (26/27/17/55/54), with war_widows_pension added under Give War Widow's Pension its own CPI-uprated variable ahead of freezing BSP #2382;
      • all ten minimums reproduce under the PR's rule (a fifth of the smallest yearly effect, rounded down to two significant figures);
      • a fresh seven-year calibration on certified uk-data 1.58.1 reproduces all eight receipts;
      • the VAT move (centre £34.0bn → £27.6bn, minimum £6.1bn → £5.0bn) matches Split VAT grossing into household share and survey coverage #2187's divisor change from 0.38 to 0.7 × 0.67;
      • the workflow is byte-identical to the approved version: pull_request only, no forks, minimal permissions, and only aggregate JSON, the comment and the summary leave the job.
      • Tests: 192 focused and 33 pinned-data tests pass.
  • d1601 A (Max, 2026-10-17): exact base-to-head national changes are published, as in policyengine-api, and level, reform and decile protections keep their rules. privacy.py's executable code is unchanged; only its text records the ruling.
  • Impact: tooling and test data only. The model code equals main, so no figure moves.
  • Known findings, stated rather than counted:
  • Follow-up: Compare two datasets on one model in the aggregate-impact tool #2425, the data-axis PR stacked on this one, now merges main and goes to the hub.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Report each pull request's effect on national aggregates on the certified dataset

1 participant