Repository navigation
Report each pull request's effect on national aggregates on the certified dataset - #2335
Conversation
The report measures one model checkout on the pinned certified dataset over 2025-26 to 2031-32: receipts by tax head, spending and caseloads by benefit, poverty by measure and group, inequality, own-decile incomes and the standard reform costings. It checks them against a fiscal partition of the model's totals (policyengine_uk/data/fiscal_partition.yaml): - closure: each total's components are exactly the partition's heads and declared double counts; - accounting identities, household by household; - a counted-once check, which perturbs each component by £1 and requires each tax total to move by exactly £1 per unit, and each spending total by at most £1. compare then sets a base measurement against a head one. It fails on a crash, a different dataset, non-finite figures or a check the base passes and the head breaks. It flags moves above the proposed thresholds, and renders a pull request comment. The partition's tax heads follow the #2159 work's specification, encoded for main before that fix: corporate_sdlt is a declared double count inside expected_sdlt. Spending heads cover gov_spending and household_benefits. Test data: conftest pins enhanced_frs_2024_25 from uk-data 1.58.0 by Hugging Face commit and SHA-256 (tests/certified_dataset.yaml), verifies the bytes before data tests run, and adds a mandatory-data mode (POLICYENGINE_UK_REQUIRE_TEST_DATA). In that mode, data tests fail rather than skip without data, and run only on the pinned dataset. The reform-cost fixtures are rebased onto 1.58.0, keeping each earlier value as a receipt with the model and data that produced it. Refs #2334. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
.github/workflows/aggregate-impact.yaml measures the pull request's base and head on the same runner image. Each side syncs its own uv.lock and runs the head's copy of the tool against its own model, on the pinned dataset whose bytes the tool verifies. The report job then compares the two, writes the job summary and one updated comment, and fails on structural failures. Fork and Dependabot runs, which cannot read the private dataset, record a skip instead. Refs #2334. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sections with no change collapse into one line, an unchanged decile table into one sentence, and the counted-once summary lists only real violations (a tax total moving by other than £1 per unit, or a spending total by more than £1), not the means-test offsets every benefit shows. Refs #2334. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Refs #2334. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The aggregate impact report (#2335) adds gov_spending, household_benefits and spending_heads to fiscal_partition.yaml. The tax tests now check the three tax totals only, require that spending heads borrow no tax head id or variable and that no spending head names a tax total, and apply dropped_when to the tax totals alone. Checked against #2335's file with these tax sections merged in: this file's tests and #2335's partition and guard tests pass. The vat head records that 76 households have negative VAT on policyengine-uk-data 1.58.0 (consumption or the full-rate expenditure share is negative in the data; policyengine-uk-data issue 565). Refs #2159 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Refs #2334. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…lookup - Set POLICYENGINE_UK_REQUIRE_TEST_DATA=1 (W3's conftest switch, #2335) wherever require-data is on: main pushes, merge queue groups and releases. A test that needs the dataset then fails with its own message instead of skipping. Unused until #2335 lands. - A manual dispatch of the main workflow always tests; the merge queue lookup applies only to pushes, and a failed lookup parses to zero. - Document the re-run path's residual and where docs-only pull requests build the documentation; the workflow-shape test covers the release workflow's require-data and the lookup. From the GPT-6.1 Sol delta review of 96e9cec (APPROVE, non-blocking). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The standard reforms move into the tool (standard_reforms.yaml), so a vendored copy carries them: the eight reform-cost fixtures, plus a 10% rise in the Universal Credit standard allowance and in the State Pension. Each names the variable it must move, the direction and a minimum. A reform that moves its target by less, by nothing or the wrong way in any year fails responsive:<id>, so a reform the model silently ignores cannot pass as a small number. A relative reform is built from each side's own baseline parameters: that model year's value times the factor, for every year of the horizon. On uk-data 1.58.0 all ten respond in every year, 2025-26 to 2031-32. The UC standard allowance rise adds £3.0bn to £3.4bn of Universal Credit, and the State Pension rise £12.4bn to £14.4bn. Refs #2334. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The certified dataset is licensed survey microdata, and only aggregates may be published. The workflow uploaded each side's measurement as an artifact, including the household-level arrays the decile comparison uses, and cached the dataset in the Actions cache. Both are removed: - base and head are now measured in one job, so the household detail never travels between jobs; only the report directory (the two result.json files, the comment and the summary) is uploaded; - the dataset is downloaded on each run and never cached; - measure writes household detail under private/, apart from result.json; - check details give counts and weighted totals, not one household's value. test_aggregate_impact_workflow.py fails if the workflow uploads anything but the report, uses the Actions cache, copies anything private into the report, or splits the measurement across jobs. Refs #2334. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The aggregate impact report (#2335) adds gov_spending, household_benefits and spending_heads to fiscal_partition.yaml. The tax tests now check the three tax totals only, require that spending heads borrow no tax head id or variable and that no spending head names a tax total, and apply dropped_when to the tax totals alone. Checked against #2335's file with these tax sections merged in: this file's tests and #2335's partition and guard tests pass. The vat head records that 76 households have negative VAT on policyengine-uk-data 1.58.0 (consumption or the full-rate expenditure share is negative in the data; policyengine-uk-data issue 565). Refs #2159 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Aggregate impact on certified data: passBase Checks: 0 broken by this change, 1 failing on both sides, 0 fixed. Moves flagged for review: 0 (thresholds in Notes
Moves flagged for reviewNone. Spending by benefit and service
Caseloads
Household net income by decileNo household's net income changes in any year. No change in any year: receipts by tax head; government balance, each component counted once; model totals; components the model counts twice; household income totals; population; poverty rates; people in poverty; inequality; mean household net income in each side's own deciles; standard reform costings (change in gov_balance); standard reforms: change in each reform's target. Checks failing on both base and head (1)
Counted-once checkEach of 84 components was raised by £1 per unit in 2026-27, and every head of its totals was read household by household: its own head must rise by exactly £1, no other tax head may move, no other benefit may rise, and the model's own total must move by what its heads moved. A head that breaks a rule is measured again twice: with every other head held at its baseline value, and with every other head held at its value after the first £1 while the component rises again. It fails where either still shows the response. 0 failed; 0 were skipped.
Provenance
Dataset: Receipts by tax head: head levels
Spending by benefit and service: head levels
Caseloads: head levels
Government balance, each component counted once: head levels
Model totals: head levels
Components the model counts twice: head levels
Household income totals: head levels
Population: head levels
Poverty rates: head levels
People in poverty: head levels
Inequality: head levels
Mean household net income in each side's own deciles: head levels
Standard reform costings (change in gov_balance): head levels
Standard reforms: change in each reform's target: head levels
|
Counted-once check. It judged each total's net movement, which a means-test fall, or the eligibility-jump exclusion, could use to hide a double count. It now reads each head's response household by household: - a component's own head must rise by exactly £1; - no other tax head may move; - no other benefit may rise in any household (rises and falls are not netted); - the model's own total must move by what its heads moved, plus one for a declared double count. That last rule catches a total whose formula adds a zero component twice. A head that declares a read may respond, but not one for one. The snapshot of supplied inputs is taken after formula results are discarded, so an earlier result is never frozen as an input. Comparison. - A missing or crashed base now fails. - A check failing in a year where the base passes is a regression, even if it already failed in another year. - A check the base ran that is missing from the head fails (or is noted, when the partition changed), and the head must have run the core checks. - A figure on one side only is shown as added or removed, never as no change. - A non-finite head figure fails directly. - Households must match one to one. Measurement. - Finiteness is checked after the reform costings. - A rebuilt baseline must match in every year. - A skipped check is recorded as skipped. - metrics_version covers the effective spec, horizon and reforms. - Geographic exclusivity guards are checked on the data. Test data. - Data tests are identified by their marker, so the data-free tests under microsimulation/ run without credentials. - The setup hook runs before pytest's skip handling, so a skip marker cannot turn missing data into a skip. - The latest-data smoke test is a named exception to the pin. The partition's notes now describe main before the #2159 fix correctly. Refs #2334. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Qualification compares a candidate's report with the previous pair's. With only a status per check, a check that fails in more years than before looks unchanged; failing_scopes lets it be told apart. Refs #2334. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The tool digest was read from disk when a measurement finished, so editing the tool during a run stamped the result with code that had not produced it. It is now taken when the module loads. Refs #2334. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Two fixes from this PR's W3 report change the lists
Neither PR touches |
…hanges Merges origin/main (1a31ef1). - conftest: main's --shard selection and shard header now sit in the same hooks as the data plan, instead of two definitions of each hook. The pin is found through the installed package, so CI's tests that copy conftest into a scratch directory still load it. - Pin: the certified pair is now policyengine-uk 2.130.1 with uk-data 1.58.1 (policyengine.py bundle uk-6.4.3): enhanced_frs_2024_25.h5 at Hugging Face commit 2526edbb, sha256 056e80fd. The reform-cost fixtures are recomputed on it with this model (2.132.6), each new value added as a receipt. - Partition: Scottish Child Payment, Healthy Start and the Two Child Limit Payment are now in gov_spending (#2366), Statutory Paternity Pay is a new spending head (#2365), and the Pension Age Winter Heating Payment is in household_benefits (#2047). Closure caught each one. - The SSP/SMP Universal Credit interaction is fixed on main (#2365), so the certified test no longer expects it to fail. - The workflow test moves to code_health/ and skips in an installed wheel, which has no .github, as the other CI-script tests do. - The dataset is projected to 2032 (#2376), so 2031-32 is no longer extrapolated differently; the README says so. Refs #2334. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Re-review fixes: - The counted-once guard judges each household: its own head rises by exactly what was added, no other tax head moves, no other benefit rises, and the model total moves by its heads. Readers and zero-baseline benefits use a two-point slope test, so a passport (slope 0) is told apart from containment (slope 1). - A head that breaks a rule, or a reader with a linear response that is not one for one, is measured again with every other head held at baseline. Responses carried only by another head are reported as mediated, not failed. Incapacity Benefit raises Universal Credit only through the Carer's Allowance overlap, which reflects a missing UC income source (#2424), not a double count. - Components that are non-zero only in zero-weight households are skipped. - Check details suppress cells below 10 records; the geography guard nets declared double counts out of their containers. - Closure, counted-once and responsiveness checks are scoped per problem and per year, so a comparison sees new failures, not just "still fails". - The tool digest is taken at load; the definitions digest covers parsed spec fields; household detail is written under private/. - The workflow posts a fallback comment when the comparison crashes and always replaces an earlier comment. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Each standard reform's minimum is now a fifth of its smallest yearly effect over 2025-26 to 2031-32 on uk-data 1.58.1, measured with policyengine-uk 2.130.1 on core 3.33.0 (the certified model) and with 2.132.6 on core 3.33.2; every target change was identical between the two. Minimums are rounded down to two significant figures, and parse_reforms refuses a minimum that does not follow from its recorded smallest_effect. A new `calibrate` command recomputes both from fresh measurements. The personal allowance minimum falls from £1bn to £0.35bn: its smallest year, 2031-32, is worth £1.78bn, so the old minimum was 56% of the effect. VAT's rises from £4bn to £6.1bn. The flag thresholds are kept and their basis recorded: flags only order the comment, every change is listed, and runs are deterministic. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- A head total or caseload built from fewer than 10 records is published as suppressed (privacy.py): one recipient's total and caseload give the award and the weight. On certified 1.58.1 this hides Disabled Students' Allowance. Compare, render and the W4 long format carry the flag; a suppressed figure is neither non-finite nor flagged. - Counted-once messages, problem counts and shares, mediated counts and perturbed-unit counts keep no figure under 10 in the published result. - The direct-effect step also holds the other heads at their values after the first point while the component moves to the second, so a head gated by a passport on another head (h = c * (b > 0)) keeps its slope and fails. Mediated records list the other heads that moved, not a proven route; the documented limit covers falls that outweigh a rise. - The PR comment is replaced on any failed run, even before the comparison. - Wording: the half alternative for reform minimums, the single-minimum weakness, the VAT receipts ratio by data version, and the 1% reform flag threshold. On certified 1.58.1 (2026-27) the guard gives the same verdicts: 314 pass, 15 warn, 1 fail (sign:vat, pre-existing). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sol review of #2425: a head from fewer than 10 records was suppressed, but a total containing it, less the published heads, gave it back (one rare £123.45 benefit with weight 2 recovered as £246.90). Each published total that contains a suppressed head in a year (model totals, balance aggregates, gov_balance and the household income totals) is now rounded to the nearest £1bn, or suppressed if the head is worth £0.5bn or more. Totals without one stay exact; the W4 long format flags rounded rows. On certified 1.58.1 Disabled Students' Allowance is suppressed in every year, so spending, benefit and balance totals are published to the nearest £1bn. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Decile rows publish no household weight or base income, round changes to what the comment shows, and publish no change where fewer than 10 weighted households change. In a year with a suppressed head, decile mean incomes are rounded to £1,000. - A rounded total's exact change is read from each side's private detail and published only when no suppressed head inside it changed; otherwise it is shown as withheld and not flagged. A suppressed head never shows a change. - Records, perturbed units and problem households count only records with weight. - A failing determinism check gives no amounts. - A component perturbed in fewer than 10 units publishes no responses, coefficients or shares. - The comment describes the second hold, and a head-only suppressed figure shows as suppressed. - The VAT receipts comment says only the 62% is a same-model ratio. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ostics (Sol re-review 4) - A protected total's exact change is withheld when any suppressed head inside it, or the total itself, is suppressed on one side only. - Identity and overlap diagnostics count only households with weight. - Decile relative changes are published to 0.1%, so the all row can't pin the base income total to within £0.5bn. - Decile changes that round to £0 are described as such, not as no change. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… re-review 5) Rounding a total that contains a suppressed head let repeated reports narrow the head down: each rounded level less the exact published heads bounds it. Every total containing a suppressed head is now published without it that year, with the heads it leaves out and their signs under excludes; no published figure depends on the head at all. - Signs follow each total: model and balance totals add their heads, gov_balance subtracts spending heads, household net income adds benefits and subtracts taxes. Market income holds no head. The HBAI total resets negative incomes, so it is recomputed household by household before the reset, or suppressed if the model can't support that. - Decile incomes leave the suppressed heads out per household, so decile levels and changes carry none of them; the decile rounding is gone. - The comparison aligns sides that leave out different heads using published values only, so the private hidden.json and private_delta are gone. The comment marks such totals with a dagger, and the W4 format lists the excluded heads. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…view 6) - A total is published without a suppressed head only when every identity behind it holds that year; otherwise it is suppressed, since a formula that scales its components would let total less head reveal the head. Decile incomes follow the same rule. - The HBAI total is rebuilt only where its component lists reproduce the model's income household by household (council tax abolished, say). - Poverty and inequality are measured on HBAI income: in a year when a suppressed head is part of it, they are suppressed too. - A standard reform that moves a suppressed head in a year publishes neither its costing nor its target change for that year, and its responsiveness detail gives no amount. - The comparison lines up different exclusions only for additive totals, and withholds decile rows when the two sides' incomes leave out different heads. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Totals holding a suppressed head or double count are rebuilt household by household from their other components, never as the model's total less the suppressed one, so no tolerance or scaling formula leaves a residual. - Declared double counts get the same small-cell suppression as heads. - Identity checks and the HBAI reproduction check leave out households that hold a suppressed figure, so their verdicts and details can't depend on it. - Deciles, poverty and the Gini are withheld while HBAI income holds one. - Reform figures for a year are withheld when the reform changes any record of a suppressed component, leaves a new small cell, or changes gov_balance or its target in 1 to 9 households. - Withheld comparisons render as "withheld", not "no change"; decile notes show when every year is withheld. - privacy.py states the threat model and the d1601 residual. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… after the model loaded Merges aggregate-impact-report at 0924449 (components rebuilt from their published parts, double counts protected, reform figures withheld from small cells, withheld comparisons shown as such). Conflicts were the decile branch in compare(), the empty-deciles text in render, and the appended tests; both sides are kept. Sol re-review 7 of #2425, P3 6: `python -m policyengine_uk.utils.aggregate_impact` imports policyengine_uk, which builds the model and caches its parameters before the command can fingerprint its files. `measure` now records `model_loaded_before_fingerprint`; the data axis refuses a side where it is true or missing, and the model axis warns. The README's data-axis commands use a copy of the tool, as CI does. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- The sign check of a head from fewer than 10 records isn't run that year, and its caseload is suppressed with it: either would say whether its few records are negative. - A count of fewer than 10 non-finite values is given as "fewer than 10". - Overlap guards leave out households holding a suppressed figure, as the identities do. - The published guard drops a few-unit double count's coefficient, and the double_count_present detail comes from it. - A comparison withheld in every year says why instead of "household detail missing". - privacy.py says "no published total includes it" and names the deliberate exceptions (formula responses, the counted-once verdict, no dominance rule). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Conflict in compare(): the withheld years are now worked out before the axis branch, and the data axis keeps its own note. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…view 9) The published guard drops a few-unit double count's coefficient, and the comment formatted it as a number. It now says it isn't shown, and the test renders the stripped guard. privacy.py names the double-count-present verdict alongside the counted-once one. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- The certified household-detail test treats `excludes` as metadata, not a
per-household column.
- `excludes` is set through a helper that rebinds the record's map. Main's
cached-array scan read `x.setdefault("excludes", {})[year] = ...` as a write
into a simulation array: a call whose first argument is a string looks
like `entity("variable", period)`.
- The W4 long format gains `exclude_signs` ({metric id: +1 or -1}) beside
`excludes`, so additive totals with different exclusions can be lined up
from published figures.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Max ruled d1601: A (2026-10-17 13:23Z, recorded in cos; relayed by the UK hub).
This unblocks this PR for its 10/19 live date, and #2425 after it. When it's ready, hand it to the UK hub with a |
|
The UK hub is adopting this PR (d1280 fallback: the W3 session that owned it is no longer live). Fleet ops asked that it land before 10/19.
|
|
Adoption handoff at Merged main Validation:
Open items for the hub:
|
|
Round 2 handoff at Merged main
The PR body now records current-main receipts, minimums, causes and tests. Independent hub review and hosted CI remain pending; the full repository suite and optional local main/head comparison were not run. d1601 A remains unchanged: exact changes are published and level protection stays.
|
|
Landed by the UK hub through the merge queue, pinned to head
|
Fixes #2334
This is workstream W3 of the Autumn Budget 2026 CI plan. Max ruled cos d1601 A on 2026-10-17: publish exact base-to-head changes, including changes between model versions, as policyengine-api does; level protection stays. The PR remains a draft pending the UK hub’s independent review and CI. GPT-6.1 Sol approved the pre-adoption head,
1191ac5a4(delta review 11, no findings), after re-reviews 7–10.What it adds
An aggregate-impact report. The module is
policyengine_uk.utils.aggregate_impact, and the workflow is.github/workflows/aggregate-impact.yaml. On every PR it measures the base and the head on the same certified dataset, on the same runner image. Each side syncs its ownuv.lockand runs the PR head's copy of the tool. The report covers every year from 2025-26 to 2031-32:Base and head are measured in one job. The dataset is licensed survey microdata, so only aggregates leave that job; a test enforces it (see "Data protection" below).
It posts one comment and updates it on later runs, including after a failed run. It fails on:
Moves above the flag thresholds are listed, not failed.
A fiscal partition,
policyengine_uk/data/fiscal_partition.yaml.corporate_sdltis a declared double count insideexpected_sdlt.gov_spendingandhousehold_benefits, 55 heads, following main's lists after Count statutory sick, maternity and paternity pay as Universal Credit earned income #2365, Count Scottish Child Payment, Healthy Start and the Two Child Limit Payment in government spending #2366, Count the Pension Age Winter Heating Payment in household and HBAI income #2047 and Give War Widow's Pension its own CPI-uprated variable ahead of freezing BSP #2382.Immutable test data and a mandatory-data mode, in
tests/conftest.py.tests/certified_dataset.yamlpins uk-data 1.58.1'senhanced_frs_2024_25.h5by Hugging Face commit2526edbb(sha256056e80fd…). policyengine.py bundle uk-6.4.3 certified that artifact for model 2.130.1.POLICYENGINE_UK_REQUIRE_TEST_DATA=1, every data test fails in setup when the dataset is unavailable, and runs only on the pinned file. The one named exception istest_latest_data_smoke.py, which exists to read the latest data; it is reported in the pytest header.Reform fixtures rebased onto 1.58.1.
reforms_config.yamlkeeps every earlier value as a receipt, with the model and data that produced it: the unpinned values, 1.40.3 and 1.58.0 at 2.126.8, and 1.58.1 at 2.132.6. The 1.40.3 receipts show how much of each move is the data.Invariants the report enforces
Closure. For each of
gov_tax,household_tax,pre_budget_change_household_tax,gov_spendingandhousehold_benefits, the model's own component list is exactly the partition's heads plus its declared double counts, each listed once. Each head's declared entity is its variable's entity.Identities, household by household.
dropped_when(council tax abolition).gov_balanceequalsgov_tax − gov_spending.Counted once, household by household. Each component is raised by £1 per unit, and every head of its totals is read in every household. A two-point slope test separates a passport (slope 0) from containment (slope 1) for heads allowed to respond. The rules:
A head that breaks a rule, or that the slope test screens, is measured again twice:
It fails where either measurement still shows the response, so a containment gated by another head (a passport) is caught too. A response that both measurements clear is reported as mediated. The other heads that moved in those households are listed as candidates, not as a proven route. For example, Incapacity Benefit lowers Carer's Allowance through the overlapping-benefit rules, and Universal Credit counts Carer's Allowance as income but, wrongly (Count Incapacity Benefit and SDA as Universal Credit unearned income (UC transitional regulations reg. 25) #2424), not Incapacity Benefit.
Further rules:
Documented limits:
Bounds and finiteness. Weights are finite and non-negative. Poverty rates lie in [0, 1]. Every figure is finite, reform costings included. Heads declared
sign: nonnegativeare never negative.Determinism. A rebuilt unreformed simulation reproduces the baseline
gov_balanceexactly, in every year.Reform responsiveness. Each standard reform names the variable it must move, the direction and a minimum (
standard_reforms.yaml). A reform that moves its target by less, by nothing or the wrong way failsresponsive:<id>for that year, so a reform the model silently ignores cannot pass as a small number.Comparison.
Thresholds adopted under cos d1344
d1344 delegates these calls to the workstream, with an independent review agreeing. The full basis is in the files.
standard_reforms.yaml). Each is a fifth of the reform's smallest yearly effect, 2025-26 to 2031-32, on the certified pair, rounded down to two significant figures.parse_reformsrefuses a minimum that doesn't follow from the recordedsmallest_effect, andpython -m policyengine_uk.utils.aggregate_impact calibraterecomputes both.Initial calibration before adoption (
1191ac5a4; models 2.130.1–2.132.6 on uk-data 1.58.1). This table records the original minimums; the merged-model changes appear in the adoption update below.Flag thresholds (
report_spec.yaml), a judgement recorded with alternatives:Flags only decide which moves lead the comment. Every changed figure is listed whether or not it is flagged. When the comment would exceed GitHub's limit, sections are left out, and those figures are in the report artifact.
Data protection
An earlier version of this workflow (commits 073e808 to 8d3e482) uploaded each side's measurement as an Actions artifact, including household-level arrays, and cached the dataset in the Actions cache. Those artifacts and the cache were deleted on 11 October, and
fd94415removes both:both sides are measured in one job, and only the report directory is uploaded (two
result.jsonfiles of aggregates, the comment and the summary);the dataset is never cached;
household detail is written under
private/, which is deleted after the comparison;small cells (
privacy.py, which also states the threat model and its deliberate exceptions). A head total, caseload or declared double count built from 1 to 9 records is published as suppressed, and its years are listed. Its raw value never enters the record. A suppressed head's caseload is suppressed with it, since a caseload of zero would show that none of its records is positive.A total holding a suppressed figure is rebuilt that year from its other components, household by household. It is never the model's total less the suppressed figure, and the components it leaves out are recorded with their signs under
excludes. Earlier versions subtracted the figure and gated that on the identities holding. Sol's re-review 7 showed that a £1 identity tolerance, together with a benefit-uprating setting, still left a residual from which the figure could be recovered. A rebuilt total is the published parts' sum exactly, so no formula or tolerance matters. What each total is rebuilt from:gov_balance: the rebuiltgov_taxless the rebuiltgov_spending.household_benefits, less the rebuilthousehold_taxand pension contributions (its own adds and subtracts). It is suppressed if those can't be read.Checks whose verdict could depend on a suppressed figure leave its holders out, or aren't run.
Poverty, inequality and deciles are withheld in a year when HBAI income holds a suppressed figure. Households are ranked into deciles on HBAI income, so which decile a holder falls into would depend on the figure. The comparison says so for that year.
A standard reform's costing and target change are withheld for a year (published as suppressed, and the responsiveness detail gives no amount) when the reform:
gov_balancein 1 to 9 households, or its target variable in 1 to 9 records, by more than half a penny.Re-review 7's cases fail without these rules and pass with them: two suppressed pensions whose changes cancel, a £4 pension raised 10%, and an allowance rise that leaves one taxpayer.
Comparisons. Where the two sides leave out different components, additive totals are lined up by taking the union out of both, from published values only. The change is then the published parts' change, and the total is marked †. A change that can't be shown is rendered "withheld", never "no change": a mean or the clipped HBAI total with different exclusions, or a figure suppressed on either side. Decile rows are withheld when the two incomes leave out different components. When every year is withheld, the decile section says why.
Under d1601 A (Max, 2026-10-17), exact national changes between base and head are published as policyengine-api does, including changes between model versions or datasets. This includes a change from 1 to 9 changed records of a total with 10 or more. Level protection stays; decile rows and reform figures retain their existing small-change withholding. This ruling required text updates only: the report already behaves this way.
Guard details give a small count as "fewer than 10", and the published result drops small unit counts, household counts and shares.
comparetreats a suppressed cell as neither non-finite nor flagged. The W4 long format carriessuppressedand, for each total, theexcludeslist with each entry's sign inexclude_signs.On certified 1.58.1 at
1191ac5a4, Disabled Students' Allowance's total and caseload are suppressed in all seven years (14 of 1,484 rows in the W4 format). Sixteen figures are rebuilt without it in every year (112 cells):gov_spending,household_benefitsandgov_balance;Against
f8dfecd4f's subtraction, these moved only by the identity tolerance that subtraction carried: decile means by under £0.002 per household, and the national totals by £600 to £6,700. Every value is identical between092444913and1191ac5a4. DSA isn't part of HBAI income, so the HBAI total, market income, poverty and Gini are published as measured, and the decile ranking is unaffected. No standard reform touches it or leaves a small cell, so every costing is published. The only failing check issign:vat, which fails on main too. DSA's own sign check is skipped. The measurement took 391s. The W4 fixture isbudget-metrics-uk-data-1.58.1-pe-uk-1191ac5a4.json.An earlier comment on this PR and six earlier report artifacts published Disabled Students' Allowance figures built from fewer than 10 records, before suppression existed. The comment's current text suppresses them. Under Max's standing rule (d932), old revisions and artifacts holding survey small cells need no action.
test_aggregate_impact_workflow.pyfails if the workflow uploads anything else, uses the cache, or splits the measurement across jobs. It also fails if the comment step stops running after a failure: the step writes a FAIL comment when the comparison wrote none, so an earlier passing comment can't stand.Tests
measure_yearon a stub simulation, including every check's status and detail and the decile incomes. Adding negative awards to this test found the caseload leak above;compare, the comment and the W4 format;gov_taxless receipts gives nothing), cancelling pension changes, a £4 pension, the allowance rise that leaves one taxpayer, reform costings from 3, 9 and 10 changed households, abolished council tax of £1,200, £0.50 and £0.01, and an identity whose only failure is a holder;Microsimulation.calculate(...).sum().microsimulationmarker):test_aggregate_impact_certified.pyruns 2026-27 with the counted-once check on every tax component and five benefits, and the two relative reforms.Pre-adoption validation at
1191ac5a4:pytest policyengine_uk/tests/aggregate_impact policyengine_uk/tests/code_health/test_aggregate_impact_workflow.py -m "not microsimulation" -n 4: 192 passed without credentials, at1191ac5a4.1191ac5a4(above).test_aggregate_impact_certified.pyin mandatory mode on the 1.58.1 pin: 9 passed at1191ac5a4.test_aggregate_impact_certified.pywithPOLICYENGINE_UK_REQUIRE_TEST_DATA=1on the 1.58.1 pin: 9 passed.microsimulationtests passed on the 1.58.1 pin in mandatory mode.sign:vat, pre-existing).The full Python suite was not run locally; CI runs it.
Findings recorded before adoption at
1191ac5a4corporate_sdltis counted twice ingov_tax,household_taxandpre_budget_change_household_tax(Corporate SDLT is counted twice in household and government tax totals #2159, fixed in Count corporate SDLT once and let SDLT abolition remove all of it #2235, which is open). The partition declares it, so it passes as a known double count.sign:vat, failing on both sides; policyengine-uk-data#565).gov_spending(Scottish Child Payment and Healthy Start are household benefits but missing from government spending #2338, by Count Scottish Child Payment, Healthy Start and the Two Child Limit Payment in government spending #2366).Not in this PR
POLICYENGINE_UK_REQUIRE_TEST_DATAin the existing workflows.axiom: n/a: CI, test-data and reporting infrastructure; no policy change
🤖 Generated with Claude Code
Round 2 adoption update — newer main merged and recalibrated
Current head:
a6f736468a752ddf8a63f3735f261b24e603b041. Merged main7ad002f549680a9ca024c896532697d532a3a92a(policyengine-uk 2.135.7) in7717438bffbd5aa45fff464466b5d043e473052b, on top of round-one head7ac83a4cdf4ee72812bdc7157ae4935bd4599578, preserving the author and original history. Thereforms_config.yamlconflict is resolved. The PR remains a draft pending the UK hub's independent review and hosted CI; no merge or readiness action is requested. #2425 remains unchanged.d1601 A remains as ruled by Max on 2026-10-17: publish exact base-to-head changes, including changes between model versions or datasets, as policyengine-api does; level protection stays. No national-change protection was added.
Historical round-one evidence only: at
7ac83a4cdf4ee72812bdc7157ae4935bd4599578on model 2.135.6, the focused/workflow suite passed 192 tests and mandatory pinned-data suite passed 33 tests. Its full seven-year invariant harness completed, with the State Pension / Universal Credit counted-once finding in 2029-30 and the existing negative-VAT finding retained. Round one's minimum changes were higher rate £710m → £700m and additional rate £550m → £560m. Those tests and findings are historical evidence; the fresh results below describe the newer merged tree.Closure and partition. Runtime registry closure and structure checks pass for all five totals:
gov_tax26,household_tax27,pre_budget_change_household_tax17,gov_spending55,household_benefits54. Mainf1a01e4d54f13bdd1847bde52cffca7422873c3f(#2382, merged at50063e1da71495ebaf7aa767f049000ead3cc1b3) addswar_widows_pensionto both spending lists. The partition adds that one person-level spending head, giving 55 spending heads. VAT #2187 changes grossing formulas but no component list. Tax heads and the existingcorporate_sdltdeclared double count stay under #2235/#2159.Reform receipts and centres. Verified the certified uk-data 1.58.1 bytes first: SHA256
056e80fd8e9151d78f432a25cde863340baee81304c6216a37fb575ce8563784. Recalibrated with model 2.135.7, Core 3.33.2, on the same pin. Preserved all 41 historical receipt mappings, including all round-one receipts and main's complete incoming VAT mapping in its original schema (plus its source commit), then appended eight new 2029gov_balancereceipts in the round-oneimpact/data/modelformat. The main VAT mapping is historical evidence whose dataset/model are not inferred. Refreshed 1 expected-impact centre(s); parameters and tolerances stay.The official
python -m policyengine_uk.utils.aggregate_impact calibrate .job/calibration-measurement/result.jsoncommand passed and all ten outputs match the applied calibration. Only 1 responsiveness minimum(s) change from round one under the existing rule: one fifth of the smallest directed yearly target effect over 2025-26 to 2031-32, rounded down to two significant figures. Each changed minimum and its storedsmallest_effectwere updated together; all other historical calibration fields stay.vat_standard_plus_2ppSmallest yearly effects moving by more than 10% since 2.135.6:
vat_standard_plus_2pp: smallest yearly effect £30,863,018,576 → £25,006,282,297 (-18.98%). VAT grossing divisor changed from 0.38 to household share 0.7 times survey coverage 0.67 (=0.469). Holding the consumption base fixed, VAT reform effects scale by 0.38/0.469. Cause: main98ab77093b0f48e4cb50677c2c9403a5533631f9(Split VAT grossing into household share and survey coverage #2187). This attribution is an inference from the formula and measured ratio.Fresh local validation on the merged model:
pytest policyengine_uk/tests/aggregate_impact policyengine_uk/tests/code_health/test_aggregate_impact_workflow.py -m 'not microsimulation' -n 4 --dist worksteal -o xfail_strict=true: 192 passed, 0 failed, 0 errors, 0 skipped (192 distinct nodes).POLICYENGINE_UK_REQUIRE_TEST_DATA=1 pytest policyengine_uk/tests/aggregate_impact/test_aggregate_impact_certified.py policyengine_uk/tests/microsimulation policyengine_uk/tests/test_capital_gains_tax_components.py -m microsimulation -n 0 -o xfail_strict=true: 33 passed, 0 failed, 0 errors, 0 skipped (33 distinct nodes).state_pension. The 2029 counted-once command exits 1 for its recorded failure; no guard pass is claimed.The round-one State Pension / Universal Credit finding is confirmed unchanged. The public failure signature and common-component statuses match round one; the affected small-cell count remains protected. Attribution to main issue #1947 remains provisional. Added components:
war_widows_pensionpass. Declared double counts pass: True; reset reproduces the baseline: True.make formatand lint passed before the calibration commit;uvx ruff format --check .passed before pushing.The calibration measurement separately records 1 failing ancillary check(s) and 17 warning(s). Failing IDs:
sign:vat. Negative VAT remains the existing negative-consumption/data finding uk-data #565.The adoption adds receipt/calibration and partition metadata changes without editing model policy formulas or parameters beyond the merged main. All suites and microsimulations ran in the foreground, with no more than four pytest workers and one microsimulation at a time. The complete repository Python suite was not run locally. Hosted CI supplies the per-PR base/head report; independent review belongs to the hub. No all-seven-year counted-once pass is claimed.