Skip to content

Preserve modifier inputs across direct callbacks and measurements - #2673

Merged
vahid-ahmadi merged 6 commits into
mainfrom
fix-scenario-modifier-inputs-2666
Oct 26, 2026
Merged

vahid-ahmadi merged 6 commits into
mainfrom
fix-scenario-modifier-inputs-2666

Conversation

@juaristi22

@juaristi22 juaristi22 commented Oct 25, 2026 •

Copy link
Copy Markdown
Collaborator

Fixes #2666
Fixes #2667

A modifier can read an output and replace it with set_input during construction, through Scenario.apply, or directly on a built simulation. Later reads, dynamics, marginal rates and exports retain the surviving explicit inputs visible to that simulation, including input-only variables absent at construction, overwrites and explicit zero. Calculated intermediates remain live; stores on another clone or branch do not become parent inputs.

reset_calculations now retains every surviving explicit input, including ordinary post-construction set_input calls outside any modifier. A reset alone no longer undoes those overrides. To restore a formula or default, call delete_arrays(variable, period), then reset or recalculate. Every top-level set_input on a built simulation queues dependent-cache cleanup before its next outermost read, including simulations with no modifier or scenario. The guide and changelog disclose both general caller-visible changes.

The constructor and apply_reform preserve their existing success-only own-write naming and input-list isolation. Shared consumers inspect actual stored-input metadata. Formula-owned writes to a child branch while any ancestor is calculating use Core's selective invalidation, preserving unrelated computed values. Pension Credit and the UC housing calculation therefore share their two earnings-tax counterfactuals in either calculation order. Idle user-branch edits still refresh outputs to the cold-construction benchmark. Clone calculation depth and separate baselines retain their existing lifecycle.

All three marginal-rate formulas take fresh scratch names distinct from existing registrations and the current branch. Identity-guarded finally cleanup detaches only the branch each formula owns, including on failure. Existing named user branches keep their objects and inputs. The CGT step, masks, held responses and all tax arithmetic remain unchanged. After a successful construction-time modifier, the completed cleanup clears its pending flag so the first read does not drop computed values twice; callback or cleanup failure retains invalidation for a subsequent retry. Default construction now also drains pending framework input invalidation before cloning its baseline, then clears the flag only after successful cleanup. The default baseline therefore receives coherent explicit inputs and cold calculations rather than copying pending work for another first-read drain. Independent Scenario baselines and user callbacks keep their existing order. This does not suppress framework writes, change the active-ancestor rule or discard queued external edits.

The declared minimum Core version is now 3.33.0, matching the public drop_computed_arrays API introduced in its tagged changelog. The lock's resolved Core remains 3.33.2; other dependency versions, sources and hashes are unchanged. The guide names this minimum and explains the explicit cache boundary after direct parameter-tree updates.

Merged policyengine.py #718, which closes #717, supplies the wrapper boundaries. Its actual merge ba50a5f8104a906a6080d01af2b5883a27b3f108 uses _apply_step to clear computed arrays in finally after the Policy callback, late Dynamic parameter application and Dynamic callback, preserving their order. These boundaries are included in published wrapper 6.11.0, tag commit b4158177d8e4d52657cb04b2ec022044688aa841. This country model needs no wrapper dependency or copy of orchestration code.

Historical public real-wrapper checks recorded in #717 use two country versions. The released-version checks use UK 2.132.6/Core 3.33.3: a local wrapper prototype restores a specified £1,000 annual Child Benefit increase and passes 6 supported cases. Separate historical supplemental candidate measurements use native source corresponding to the earlier published commit 422885e993a6381be711397cb8dfb2be6496dd4e with Core 3.33.3. On that candidate source the prototype replaces a stale zero response with the negative cold-matching response (approximately −£27.48 annual employment income) and passes 6 supplemental integration cases. These are public synthetic household checks; they are not measurements of the final native commit or a population estimate.

UK 2.132.6 has a separate named-baseline-branch limitation: its nonzero labour-supply control fails even cold. The historical candidate comparison does not attribute every intervening native baseline change to this PR. Wrapper 6.11.0 contains #718, but its uk, models and dev extras still pin policyengine-uk==2.132.6 and policyengine-core==3.33.3. The remaining shipping bridge is to merge and release this native UK repair, then advance the wrapper UK pin or installed country model to a suitable native release; a change to the pinned extras needs its own wrapper packaging release. The earlier 6.10.2 installed metadata identifies historical test environments and does not describe the latest wrapper release.

The current formatted candidate is bb05334596c07d206113caab17d9bdc68b64726d, based on actual main 2b0b74f6185a69b8b6ded4eb028cf9240bbb9a79, which includes the independently reviewed Housing Benefit special-disregard update. Its source project version is UK 2.162.0; the frozen native Python 3.13.14/Core 3.33.2 environment still reports installed UK metadata 2.161.2. No environment sync or other dependency change was made.

Historical R4 evidence remains 1,021 distinct native passes/one existing allowed credential skip, using a latest-per-ID ledger rather than repeated-result sums. The three new default-baseline tests first failed causally on unchanged 37cb2390: pending work crossed the clone boundary, first reads ran two additional actual drains, and an injected terminal-drain failure did not raise because that drain was absent. After the narrow constructor fix, the affected seven-module suite passed 148 cases with no failures, errors or skips on the previous 112f71bd base. Those are explicitly previous-base executions; no whole-suite claim is made for this final candidate.

Fresh validation on this exact bb053345/2b0b74f6 source passed 37 Python cases (313.614 seconds) and 44 Housing Benefit YAML cases (5.683 seconds). The Python selection covers all 29 affected HB/CTR properties and 8 constructor, PC and after-load controls. A separate 8-case Core 3.33.0 gate passed (31.089 seconds), and 6 actual merged #718 integration cases passed on Core 3.33.3 (43.862 seconds). All four selections had zero failures, errors, skips or xfails and executed every selected ID once per environment. These elapsed values describe CI execution, not the performance comparison. The fresh selections are reported separately from the historical 148-case suite and 1,021-pass ledger; they are not summed into a new full-suite claim. The historical allowlisted skip is policyengine_uk/tests/test_scenario_baseline_parameters.py::test_before_load_spending_scenario_constructs_on_the_dataset.

All fresh gates use Python 3.13.14 and import the same final UK source project 2.162.0. Installed editable UK metadata remains 2.161.2 in the native environment, 2.161.1 in the minimum-Core environment and 2.158.2 in the wrapper environment. The wrapper environment reports installed metadata 6.10.2 but executes the immutable actual merged #718 source ba50a5f8, with its model module verified by hash; this does not identify a wrapper release. Earlier R4 compatibility executions (17 cases on Core 3.33.0, the real public constructor/reset/drop/MTR probe, six actual merged-wrapper cases and the public NI EPG probe) remain historical evidence and are not presented as fresh executions of the R5 change. The current published wrapper release is 6.11.0, as described above.

The new seven-case RED result was five causal failures and two passing failure controls. Four failures expose CGT collision/exception ownership, and one exposes duplicate after-load cleanup. Original 30 Scenario cases remain unchanged. Existing read-only cache, held response, static marginal-rate, tax/WFP inequality, tolerance and pinned property examples remain intact. The byte-unchanged policyengine_uk/tests/test_pension_credit_tax_branch_reuse.py::test_full_calculation_reuses_two_earnings_tax_counterfactuals retains its exact two-call assertion in both orders.

The new constructor regressions are policyengine_uk/tests/test_scenario_modifier_inputs.py::test_default_baseline_clones_after_framework_input_drain, policyengine_uk/tests/test_scenario_modifier_inputs.py::test_default_constructor_first_reads_do_not_repeat_framework_drain, and policyengine_uk/tests/test_scenario_modifier_inputs.py::test_default_baseline_drain_failure_retains_pending_inputs. They retain positive cold-reference PC/HNI, explicit inputs, the unchanged two-counterfactual contract and failure recovery.

The final public synthetic benchmark compares immutable archives of actual base 2b0b74f6 and candidate bb053345, with identical parameter-file/directory topology and unchanged natural GC settings. P1 reads Pension Credit before household net income; P2 reads household net income first; M exercises the marginal-rate and held-response fixture. Each snapshot/case has one warmup, three unprofiled repeats and one profiled observation. Thirty distinct primary constructions, two held controls and six constructor/query profile pairs completed across two sessions, with no valid case repeated. Every direct economic and held output difference was zero, and positive/cold-reference checks passed.

Candidate/base median cost ratios are:

Public fixture Constructor CPU Query CPU Matched constructor + query CPU Matched constructor + query wall
P1: Pension Credit first 1.073633 1.004210 1.059243 1.064710
P2: Household net income first 1.045180 0.939565 0.976757 0.976443
M: Marginal rates/held responses 1.003336 0.961550 0.986241 0.990488

The matched CPU medians in seconds (minimum–maximum of the three repeats) are P1 base 1.483190 (1.473230–1.682783) versus candidate 1.571059 (1.552682–1.652038); P2 base 1.553876 (1.486268–1.604590) versus candidate 1.517760 (1.447520–1.777766); and M base 4.308435 (3.848448–4.500409) versus candidate 4.249156 (4.248524–4.291988). These are medians of each matched construction-plus-query observation, not sums of separate medians. End-to-end CPU ratios including ordinary release are 1.059139, 0.976767 and 0.986223 respectively. P1 retains a measured 5.9% total CPU increase; the observed P2/M medians are lower in these three-repeat public fixtures. This is not a zero-overhead claim or a population timing estimate.

Both Pension Credit cases preserve exactly two tax counterfactuals, three clones/branches and identical Core calculation counts in each base/candidate pair. Their profiled query full-cache drops are zero, while candidate construction performs eight drops. M preserves its five counters and 22 clones/branches, but its profiled query performs three cleanup-helper drops versus two on the base, with 16 during candidate construction. The zero-query-drain result therefore applies only to P1/P2.

The initial benchmark session 78047 remains an overall FAIL: its 20,000-event telemetry cap overflowed only in the untimed baseline-M held-query GC tail, after its timed observations and economic comparator completed. The retained evidence includes P1/P2 and baseline M. A separate authorized candidate-M-only session 58600 passed in 34.674123 seconds, completing only the missing five primary constructions and one held control. All timed GC records are complete; the untimed baseline-M held-query GC tail remains incomplete. The completion recorder appended 20,042 events below its 100,000 cap. Default thresholds [2000, 10, 10], enabled state and prior callbacks were restored. This combined evidence is not represented as an uninterrupted full-run PASS, and no GC-causality claim or causal timing subtraction is made. The inherited aggregate workflow allows 120 minutes; a completed certified published-head report and reviewer acceptance remain separate readiness gates.

The changed guide passed one HTML build with notebook execution disabled; its source bytes remain unchanged. The single R5 formatter preserved all 1,794 reviewed CPython 3.13.14 ASTs, with only byte formatting in the Scenario test file. The subsequent conflict-free rebase preserved all 13 nonmetadata PR path bytes and all 10 nonmetadata Housing Benefit update path bytes, with the exact country-version/Core-floor union in the two manifests. The full tracked source aggregate is 2d983d36e505e3026b6d95bf54c4a4428a8fa831255039ffff443bed88488150. Full published-head CI, including the completed certified aggregate report and review acceptance, remains a separate readiness requirement.

Reproducer: policyengine_uk/tests/test_direct_modifier_inputs.py::test_direct_callback_response_matches_parameter_benchmark
Reproducer: policyengine_uk/tests/test_direct_modifier_inputs.py::test_direct_writes_survive_reset_marginal_rates_and_export
Reproducer: policyengine_uk/tests/test_direct_modifier_inputs.py::test_internal_nested_branch_keeps_unrelated_tax_counterfactual_cache
Reproducer: policyengine_uk/tests/test_direct_modifier_inputs.py::test_idle_nested_user_branch_edit_matches_cold_parameter_benchmark
Reproducer: policyengine_uk/tests/test_scenario_modifier_inputs.py::test_modifier_recalculates_cached_marginal_rate_branches
Reproducer: policyengine_uk/tests/code_health/test_cached_arrays_not_written_in_place.py::test_branching_runs_with_read_only_cache
Reproducer: policyengine_uk/tests/test_capital_gains_response_default.py::test_pay_rise_branches_hold_the_realisation_response
Reproducer: policyengine_uk/tests/test_property_income_properties.py::test_choice_is_never_worse_than_either_route
Reproducer: policyengine_uk/tests/test_scenario_modifier_inputs.py::test_capital_gains_rate_preserves_preoccupied_user_branch
Reproducer: policyengine_uk/tests/test_scenario_modifier_inputs.py::test_capital_gains_rate_failure_preserves_user_branch_and_detaches_owned_scratch
Reproducer: policyengine_uk/tests/test_scenario_modifier_inputs.py::test_after_load_modifier_clears_calculated_values_once
Reproducer: policyengine_uk/tests/test_scenario_modifier_inputs.py::test_after_load_cleanup_failure_keeps_pending_write

Reproducer: policyengine_uk/tests/test_reform_inputs_survive_reset.py::test_a_reform_names_no_input_it_did_not_store

axiom: n/a: simulation input retention, cache invalidation and marginal-rate scratch-branch lifecycle; no statutory tax or benefit rule changes.

@github-actions

github-actions Bot commented Oct 25, 2026 •

Copy link
Copy Markdown
Contributor

Aggregate impact on certified data: pass

Base 2b0b74f618 against head 736dfb53a2, both on enhanced_frs_2024_25.h5 (policyengine-uk-data 1.58.1, sha256 056e80fd8e91), fiscal years 2025-26 to 2031-32. Each year is a full microsimulation run on each side.

Checks: 0 broken by this change, 1 failing on both sides, 0 fixed. Moves flagged for review: 0 (thresholds in report_spec.yaml; a flag does not fail the job).

Workflow run and full report

Notes

  • In the years a figure from fewer than 10 records is suppressed (spending.disabled_students_allowance), every total containing it, and the decile incomes, are rebuilt from their other components, so no published total includes it. Those totals are marked †. A change is withheld where the two sides leave out different figures and can't be lined up.

Moves flagged for review

None.

Spending by benefit and service

Measure 2025-26 2026-27 2027-28 2028-29 2029-30 2030-31 2031-32
Disabled Students' Allowance suppressed suppressed suppressed suppressed suppressed suppressed suppressed

Caseloads

Measure 2025-26 2026-27 2027-28 2028-29 2029-30 2030-31 2031-32
Disabled Students' Allowance suppressed suppressed suppressed suppressed suppressed suppressed suppressed

Standard reform costings (change in gov_balance)

Measure 2025-26 2026-27 2027-28 2028-29 2029-30 2030-31 2031-32
Raise additional rate by 3pp 0 0 0 0 0 suppressed suppressed

Standard reforms: change in each reform's target

Measure 2025-26 2026-27 2027-28 2028-29 2029-30 2030-31 2031-32
Raise additional rate by 3pp: change in income_tax 0 0 0 0 0 suppressed suppressed

Household net income by decile

No household's net income changes in any year.

No change in any year: receipts by tax head; government balance, each component counted once; model totals; household income totals; population; poverty rates; people in poverty; inequality; mean household net income in each side's own deciles.

Checks failing on both base and head (1)
  • VAT excluding private school fees is never negative (sign:vat): 2025-26: 76 units below zero, a weighted £-53,889,156 in total; 2026-27: 76 units below zero, a weighted £-55,420,387 in total; 2027-28: 76 units below zero, a weighted £-56,618,038 in total
Counted-once check

Each of 88 components was raised by £1 per unit in 2026-27, and every head of its totals was read household by household: its own head must rise by exactly £1, no other tax head may move, no other benefit may rise, and the model's own total must move by what its heads moved. A head that breaks a rule is measured again twice: with every other head held at its baseline value, and with every other head held at its value after the first £1 while the component rises again. It fails where either still shows the response. 0 failed; 0 were skipped.

  • widowed_parents_allowance: pension_credit responds only through other heads in fewer than 10 households (other heads that moved there: carers_allowance; gov_spending; not a failure)
  • widowed_parents_allowance: pension_credit responds only through other heads in fewer than 10 households (other heads that moved there: carers_allowance; household_benefits; not a failure)
Provenance
Side Commit Runtime Environment hash Partition hash Time
base main 2b0b74f618 uk 2.162.0 (files ab542f2982d0), core 3.33.2, py 3.13.17 aarch64 8c5522632932 9a54a111ed04 1735.29s
head #2673 merged into main 736dfb53a2 uk 2.162.0 (files e4ac6473eaca), core 3.33.2, py 3.13.17 aarch64 8c5522632932 9a54a111ed04 1747.65s

Dataset: hf://policyengine/policyengine-uk-data-private/enhanced_frs_2024_25.h5@2526edbbb687a3b54dbb11272e5389e4b62d2ac2, sha256 056e80fd8e9151d78f432a25cde863340baee81304c6216a37fb575ce8563784 (verified on both sides), policyengine-uk-data 1.58.1.

Receipts by tax head: head levels
Measure (head) 2025-26 2026-27 2027-28 2028-29 2029-30 2030-31 2031-32
Income tax £298.70bn £318.31bn £335.60bn £350.52bn £365.79bn £383.53bn £401.03bn
National Insurance (employee and self-employed) £52.37bn £54.71bn £56.48bn £58.15bn £60.22bn £62.11bn £64.44bn
National Insurance (employer) £150.59bn £156.99bn £161.74bn £166.13bn £173.24bn £178.59bn £185.29bn
Employer NI change borne by capital £0.00bn £0.00bn £0.00bn £0.00bn £0.00bn £0.00bn £0.00bn
Employer NI change borne by consumers £0.00bn £0.00bn £0.00bn £0.00bn £0.00bn £0.00bn £0.00bn
Capital gains tax £12.61bn £13.05bn £13.53bn £14.02bn £14.52bn £15.05bn £15.58bn
Student loan repayments £7.79bn £8.09bn £8.31bn £8.51bn £8.72bn £8.97bn £9.28bn
VAT excluding private school fees £249.37bn £255.28bn £261.81bn £267.87bn £274.12bn £280.56bn £286.81bn
Change in VAT from its baseline £0.00bn £0.00bn £0.00bn £0.00bn £0.00bn £0.00bn £0.00bn
VAT on private school fees £1.95bn £2.00bn £2.05bn £2.10bn £2.21bn £2.21bn £2.31bn
Change in private school VAT from its baseline £0.00bn £0.00bn £0.00bn £0.00bn £0.00bn £0.00bn £0.00bn
Fuel duties £24.53bn £24.64bn £26.64bn £26.69bn £26.42bn £25.76bn £26.35bn
Alcohol duties £0.00bn £0.00bn £0.00bn £0.00bn £0.00bn £0.00bn £0.00bn
Tobacco duties £0.00bn £0.00bn £0.00bn £0.00bn £0.00bn £0.00bn £0.00bn
Vehicle excise duties £0.00bn £0.00bn £0.00bn £0.00bn £0.00bn £0.00bn £0.00bn
Betting and gaming duties £0.00bn £0.00bn £0.00bn £0.00bn £0.00bn £0.00bn £0.00bn
TV licence fee £4.26bn £4.41bn £4.42bn £4.44bn £4.45bn £4.47bn £4.48bn
Council tax £49.21bn £51.88bn £54.97bn £58.29bn £61.55bn £64.92bn £68.49bn
High value council tax surcharge £0.00bn £0.00bn £0.00bn £0.48bn £0.49bn £0.50bn £0.51bn
Northern Ireland domestic rates £0.55bn £0.58bn £0.60bn £0.63bn £0.66bn £0.69bn £0.72bn
Business rates £34.68bn £38.04bn £38.88bn £39.76bn £43.07bn £43.13bn £44.57bn
Stamp duty land tax on the household's own purchases £6.90bn £7.39bn £7.81bn £8.21bn £8.62bn £9.02bn £9.42bn
Stamp duty land tax paid by companies £4.81bn £5.98bn £6.73bn £7.46bn £8.21bn £8.87bn £9.16bn
Land transaction tax (Wales) £0.07bn £0.08bn £0.09bn £0.09bn £0.10bn £0.10bn £0.11bn
Land and buildings transaction tax (Scotland) £0.31bn £0.32bn £0.34bn £0.35bn £0.37bn £0.38bn £0.40bn
Unspecified tax change borne by capital £0.00bn £0.00bn £0.00bn £0.00bn £0.00bn £0.00bn £0.00bn
Unspecified tax change borne by consumers £0.00bn £0.00bn £0.00bn £0.00bn £0.00bn £0.00bn £0.00bn
Wealth tax £0.00bn £0.00bn £0.00bn £0.00bn £0.00bn £0.00bn £0.00bn
Wealth tax excluding the main residence £0.00bn £0.00bn £0.00bn £0.00bn £0.00bn £0.00bn £0.00bn
Land value tax £0.00bn £0.00bn £0.00bn £0.00bn £0.00bn £0.00bn £0.00bn
Carbon tax £0.00bn £0.00bn £0.00bn £0.00bn £0.00bn £0.00bn £0.00bn
Spending by benefit and service: head levels
Measure (head) 2025-26 2026-27 2027-28 2028-29 2029-30 2030-31 2031-32
State Pension £121.64bn £126.07bn £127.08bn £128.87bn £132.52bn £136.31bn £140.08bn
Pension Credit £6.66bn £7.04bn £7.32bn £7.46bn £7.60bn £7.77bn £7.94bn
Winter Fuel Payment £1.86bn £1.86bn £1.80bn £1.78bn £1.79bn £1.79bn £1.80bn
Pension Age Winter Heating Payment (Scotland) £0.19bn £0.20bn £0.20bn £0.21bn £0.21bn £0.22bn £0.22bn
Universal Credit £75.21bn £80.38bn £82.93bn £84.23bn £85.58bn £86.87bn £87.97bn
Housing Benefit £8.55bn £8.89bn £9.17bn £9.31bn £9.58bn £9.84bn £10.10bn
Council Tax Reduction £1.98bn £2.02bn £2.07bn £2.11bn £2.16bn £2.21bn £2.25bn
Employment and Support Allowance (income-related) £0.79bn £0.81bn £0.83bn £0.85bn £0.86bn £0.89bn £0.90bn
Employment and Support Allowance (contribution-based) £5.20bn £5.34bn £5.46bn £5.59bn £5.72bn £5.85bn £5.99bn
Jobseeker's Allowance (income-based) £0.00bn £0.00bn £0.00bn £0.00bn £0.00bn £0.00bn £0.00bn
Jobseeker's Allowance (contribution-based) £0.29bn £0.30bn £0.30bn £0.31bn £0.32bn £0.32bn £0.33bn
Income Support £0.12bn £0.00bn £0.00bn £0.00bn £0.00bn £0.00bn £0.00bn
Working Tax Credit £0.00bn £0.00bn £0.00bn £0.00bn £0.00bn £0.00bn £0.00bn
Child Tax Credit £0.00bn £0.00bn £0.00bn £0.00bn £0.00bn £0.00bn £0.00bn
Incapacity Benefit £0.00bn £0.00bn £0.00bn £0.00bn £0.00bn £0.00bn £0.00bn
Personal Independence Payment £26.94bn £28.06bn £29.13bn £29.83bn £30.54bn £31.26bn £31.96bn
Disability Living Allowance £7.49bn £7.80bn £8.10bn £8.29bn £8.49bn £8.69bn £8.88bn
Attendance Allowance £8.68bn £9.04bn £9.38bn £9.61bn £9.83bn £10.07bn £10.29bn
Carer's Allowance £4.48bn £4.67bn £4.84bn £4.96bn £5.08bn £5.19bn £5.31bn
Carer Support Payment (Scotland) £0.31bn £0.32bn £0.33bn £0.34bn £0.35bn £0.36bn £0.37bn
Carer's Allowance Supplement (Scotland) £0.00bn £0.04bn £0.04bn £0.04bn £0.04bn £0.04bn £0.04bn
Severe Disablement Allowance £0.03bn £0.03bn £0.03bn £0.03bn £0.03bn £0.03bn £0.03bn
Industrial Injuries Disablement Benefit £0.50bn £0.52bn £0.53bn £0.54bn £0.55bn £0.57bn £0.58bn
Armed Forces Compensation Scheme £0.61bn £0.62bn £0.64bn £0.65bn £0.67bn £0.68bn £0.70bn
War Widow's or Widower's Pension £0.00bn £0.00bn £0.00bn £0.00bn £0.00bn £0.00bn £0.00bn
Child Benefit £16.16bn £16.83bn £17.43bn £17.85bn £18.26bn £18.69bn £19.10bn
Scottish Child Payment £0.49bn £0.52bn £0.54bn £0.55bn £0.56bn £0.57bn £0.56bn
Two Child Limit Payment (Scotland, counterfactual) £0.00bn £0.00bn £0.00bn £0.00bn £0.00bn £0.00bn £0.00bn
Healthy Start vouchers £0.05bn £0.05bn £0.05bn £0.05bn £0.05bn £0.05bn £0.05bn
Maternity Allowance £0.00bn £0.00bn £0.00bn £0.00bn £0.00bn £0.00bn £0.00bn
Statutory Maternity Pay £3.30bn £3.39bn £3.47bn £3.55bn £3.63bn £3.72bn £3.80bn
Statutory Paternity Pay £0.00bn £0.00bn £0.00bn £0.00bn £0.00bn £0.00bn £0.00bn
Statutory Sick Pay £1.12bn £1.15bn £1.18bn £1.21bn £1.24bn £1.27bn £1.29bn
Sure Start Maternity Grant £0.01bn £0.01bn £0.01bn £0.01bn £0.01bn £0.01bn £0.01bn
Bereavement Support Payment £0.67bn £0.69bn £0.70bn £0.72bn £0.73bn £0.75bn £0.77bn
Widowed Parent's Allowance £0.00bn £0.00bn £0.00bn £0.00bn £0.00bn £0.00bn £0.00bn
Tax-Free Childcare £0.55bn £0.56bn £0.57bn £0.58bn £0.59bn £0.61bn £0.59bn
Extended childcare entitlement £2.99bn £3.50bn £3.60bn £3.67bn £3.77bn £3.85bn £3.94bn
Universal childcare entitlement £2.05bn £2.23bn £2.28bn £2.34bn £2.40bn £2.47bn £2.52bn
Targeted childcare entitlement £0.50bn £0.51bn £0.52bn £0.53bn £0.54bn £0.56bn £0.52bn
Childcare Offer for Wales £0.26bn £0.28bn £0.28bn £0.29bn £0.30bn £0.30bn £0.31bn
Care to Learn £0.00bn £0.00bn £0.00bn £0.00bn £0.00bn £0.00bn £0.00bn
Childcare Grant £0.00bn £0.00bn £0.00bn £0.00bn £0.00bn £0.00bn £0.00bn
Parents' Learning Allowance £0.20bn £0.20bn £0.20bn £0.20bn £0.20bn £0.20bn £0.20bn
Adult Dependants' Grant £0.06bn £0.06bn £0.04bn £0.04bn £0.04bn £0.04bn £0.04bn
Travel Grant £0.00bn £0.00bn £0.00bn £0.00bn £0.00bn £0.00bn £0.00bn
Disabled Students' Allowance suppressed suppressed suppressed suppressed suppressed suppressed suppressed
16 to 19 Bursary Fund £0.00bn £0.00bn £0.00bn £0.00bn £0.00bn £0.00bn £0.00bn
NHS spending £203.83bn £204.54bn £205.13bn £205.79bn £206.49bn £207.23bn £207.73bn
State education spending £0.02bn £0.02bn £0.02bn £0.02bn £0.02bn £0.02bn £0.02bn
Transport subsidy spending £25.05bn £25.13bn £25.21bn £25.29bn £25.37bn £25.46bn £25.53bn
Cost-of-living support payment £0.00bn £0.00bn £0.00bn £0.00bn £0.00bn £0.00bn £0.00bn
Energy Bills Rebate £0.00bn £0.00bn £0.00bn £0.00bn £0.00bn £0.00bn £0.00bn
Energy Price Guarantee subsidy £0.00bn £0.00bn £0.00bn £0.00bn £0.00bn £0.00bn £0.00bn
Basic income £0.00bn £0.00bn £0.00bn £0.00bn £0.00bn £0.00bn £0.00bn
Disability basic income £0.00bn £0.00bn £0.00bn £0.00bn £0.00bn £0.00bn £0.00bn
Other public spending budget change £0.00bn £0.00bn £0.00bn £0.00bn £0.00bn £0.00bn £0.00bn
Caseloads: head levels
Measure (head) 2025-26 2026-27 2027-28 2028-29 2029-30 2030-31 2031-32
State Pension 11,261k 11,158k 10,849k 10,746k 10,783k 10,822k 10,848k
Pension Credit 1,485k 1,483k 1,480k 1,461k 1,462k 1,464k 1,460k
Winter Fuel Payment 7,842k 7,793k 7,494k 7,407k 7,432k 7,459k 7,477k
Pension Age Winter Heating Payment (Scotland) 803k 803k 803k 800k 803k 806k 808k
Universal Credit 6,334k 6,341k 6,366k 6,347k 6,345k 6,333k 6,298k
Housing Benefit 1,249k 1,247k 1,250k 1,234k 1,238k 1,240k 1,239k
Council Tax Reduction 3,175k 3,198k 3,209k 3,229k 3,244k 3,257k 3,264k
Employment and Support Allowance (income-related) 71k 71k 71k 71k 71k 72k 72k
Employment and Support Allowance (contribution-based) 719k 721k 723k 726k 728k 731k 732k
Jobseeker's Allowance (income-based) 0k 0k 0k 0k 0k 0k 0k
Jobseeker's Allowance (contribution-based) 67k 67k 67k 68k 68k 68k 68k
Income Support 25k 0k 0k 0k 0k 0k 0k
Working Tax Credit 0k 0k 0k 0k 0k 0k 0k
Child Tax Credit 0k 0k 0k 0k 0k 0k 0k
Incapacity Benefit 0k 0k 0k 0k 0k 0k 0k
Personal Independence Payment 3,566k 3,579k 3,589k 3,601k 3,613k 3,626k 3,635k
Disability Living Allowance 1,291k 1,295k 1,299k 1,303k 1,308k 1,312k 1,316k
Attendance Allowance 1,694k 1,700k 1,705k 1,711k 1,716k 1,723k 1,727k
Carer's Allowance 1,034k 1,038k 1,041k 1,044k 1,048k 1,051k 1,053k
Carer Support Payment (Scotland) 71k 71k 72k 72k 72k 72k 73k
Carer's Allowance Supplement (Scotland) 0k 71k 72k 72k 72k 72k 73k
Severe Disablement Allowance 4k 4k 4k 4k 4k 4k 4k
Industrial Injuries Disablement Benefit 137k 137k 138k 138k 139k 139k 140k
Armed Forces Compensation Scheme 93k 93k 94k 94k 94k 95k 95k
War Widow's or Widower's Pension 0k 0k 0k 0k 0k 0k 0k
Child Benefit 8,102k 8,128k 8,142k 8,165k 8,191k 8,216k 8,232k
Scottish Child Payment 350k 352k 353k 357k 354k 355k 342k
Two Child Limit Payment (Scotland, counterfactual) 0k 0k 0k 0k 0k 0k 0k
Healthy Start vouchers 199k 200k 201k 201k 202k 203k 203k
Maternity Allowance 0k 0k 0k 0k 0k 0k 0k
Statutory Maternity Pay 332k 334k 335k 336k 337k 338k 339k
Statutory Paternity Pay 0k 0k 0k 0k 0k 0k 0k
Statutory Sick Pay 181k 182k 182k 183k 183k 184k 185k
Sure Start Maternity Grant 26k 26k 26k 26k 26k 27k 27k
Bereavement Support Payment 98k 98k 98k 99k 99k 99k 100k
Widowed Parent's Allowance 0k 0k 0k 0k 0k 0k 0k
Tax-Free Childcare 1,014k 1,024k 1,024k 1,027k 1,027k 1,031k 1,013k
Extended childcare entitlement 796k 801k 806k 809k 815k 819k 820k
Universal childcare entitlement 1,071k 1,074k 1,077k 1,080k 1,086k 1,096k 1,097k
Targeted childcare entitlement 165k 156k 154k 155k 155k 156k 144k
Childcare Offer for Wales 39k 39k 39k 39k 39k 39k 39k
Care to Learn 0k 0k 0k 0k 0k 0k 0k
Childcare Grant 0k 0k 0k 0k 0k 0k 0k
Parents' Learning Allowance 98k 98k 99k 99k 99k 100k 100k
Adult Dependants' Grant 18k 18k 12k 12k 12k 13k 13k
Travel Grant 0k 0k 0k 0k 0k 0k 0k
Disabled Students' Allowance suppressed suppressed suppressed suppressed suppressed suppressed suppressed
16 to 19 Bursary Fund 0k 0k 0k 0k 0k 0k 0k
Cost-of-living support payment 0k 0k 0k 0k 0k 0k 0k
Energy Bills Rebate 0k 0k 0k 0k 0k 0k 0k
Energy Price Guarantee subsidy 0k 0k 0k 0k 0k 0k 0k
Basic income 0k 0k 0k 0k 0k 0k 0k
Disability basic income 0k 0k 0k 0k 0k 0k 0k
Government balance, each component counted once: head levels
Measure (head) 2025-26 2026-27 2027-28 2028-29 2029-30 2030-31 2031-32
Government receipts, each component once £898.70bn £941.74bn £979.99bn £1,013.71bn £1,052.76bn £1,088.87bn £1,128.96bn
Government spending £528.87bn † £543.67bn † £551.43bn † £557.71bn † £566.13bn † £574.77bn † £582.74bn †
Government balance, each receipt once £369.83bn † £398.07bn † £428.56bn † £456.00bn † £486.63bn † £514.10bn † £546.22bn †
Benefits in household_benefits but not in gov_spending £0.00bn £0.00bn £0.00bn £0.00bn £0.00bn £0.00bn £0.00bn
Model totals: head levels
Measure (head) 2025-26 2026-27 2027-28 2028-29 2029-30 2030-31 2031-32
gov_tax £898.70bn £941.74bn £979.99bn £1,013.71bn £1,052.76bn £1,088.87bn £1,128.96bn
household_tax £496.79bn £527.48bn £554.39bn £577.61bn £603.19bn £627.52bn £654.55bn
pre_budget_change_household_tax £489.00bn £519.39bn £546.08bn £568.63bn £593.98bn £618.05bn £644.75bn
gov_spending £528.87bn † £543.67bn † £551.43bn † £557.71bn † £566.13bn † £574.77bn † £582.74bn †
household_benefits £528.87bn † £543.67bn † £551.43bn † £557.71bn † £566.13bn † £574.77bn † £582.74bn †
gov_balance £369.83bn † £398.07bn † £428.56bn † £456.00bn † £486.63bn † £514.10bn † £546.22bn †
Household income totals: head levels
Measure (head) 2025-26 2026-27 2027-28 2028-29 2029-30 2030-31 2031-32
Household market income £1,651.49bn £1,713.03bn £1,764.05bn £1,812.97bn £1,862.19bn £1,918.39bn £1,985.28bn
Household net income £1,644.46bn † £1,688.65bn † £1,719.42bn † £1,750.39bn † £1,781.36bn † £1,820.67bn † £1,866.89bn †
Household net income (HBAI definition) £1,457.33bn £1,504.44bn £1,537.45bn £1,569.13bn £1,602.35bn £1,640.08bn £1,686.54bn
Population: head levels
Measure (head) 2025-26 2026-27 2027-28 2028-29 2029-30 2030-31 2031-32
People 69,224k 69,466k 69,668k 69,891k 70,128k 70,381k 70,550k
Benefit units 35,100k 35,223k 35,325k 35,438k 35,558k 35,686k 35,772k
Households 31,037k 31,145k 31,236k 31,336k 31,442k 31,555k 31,631k
Poverty rates: head levels
Measure (head) 2025-26 2026-27 2027-28 2028-29 2029-30 2030-31 2031-32
Absolute poverty, before housing costs: everyone 16.0% 15.1% 15.5% 15.6% 15.7% 15.8% 15.8%
Absolute poverty, before housing costs: children 24.0% 21.2% 21.5% 21.5% 21.7% 21.9% 21.8%
Absolute poverty, before housing costs: working-age adults 13.8% 13.5% 14.0% 14.1% 14.2% 14.2% 14.3%
Absolute poverty, before housing costs: pensioners 13.7% 13.2% 13.1% 13.4% 13.5% 13.5% 13.5%
Absolute poverty, after housing costs: everyone 21.5% 20.8% 21.3% 21.5% 21.8% 21.9% 22.2%
Absolute poverty, after housing costs: children 33.1% 30.7% 31.1% 31.1% 31.4% 31.7% 32.3%
Absolute poverty, after housing costs: working-age adults 19.8% 19.6% 20.3% 20.5% 20.7% 20.9% 21.1%
Absolute poverty, after housing costs: pensioners 12.7% 12.4% 12.3% 12.5% 12.9% 12.9% 13.0%
Relative poverty, before housing costs: everyone 17.2% 16.4% 16.5% 16.4% 16.3% 16.4% 16.6%
Relative poverty, before housing costs: children 26.2% 23.5% 23.4% 23.1% 22.8% 23.0% 23.3%
Relative poverty, before housing costs: working-age adults 14.6% 14.4% 14.7% 14.7% 14.6% 14.7% 14.9%
Relative poverty, before housing costs: pensioners 15.0% 14.8% 14.1% 14.1% 14.0% 14.0% 14.2%
Relative poverty, after housing costs: everyone 21.6% 21.0% 21.2% 21.2% 21.3% 21.4% 21.8%
Relative poverty, after housing costs: children 33.3% 31.2% 31.0% 30.7% 31.0% 31.0% 31.5%
Relative poverty, after housing costs: working-age adults 19.9% 19.8% 20.2% 20.3% 20.5% 20.5% 20.7%
Relative poverty, after housing costs: pensioners 12.9% 12.4% 12.2% 11.9% 11.8% 11.9% 12.8%
People in poverty: head levels
Measure (head) 2025-26 2026-27 2027-28 2028-29 2029-30 2030-31 2031-32
Absolute poverty, before housing costs: everyone 11,047k 10,485k 10,771k 10,894k 11,001k 11,087k 11,136k
Absolute poverty, before housing costs: children 3,579k 3,184k 3,231k 3,241k 3,284k 3,320k 3,324k
Absolute poverty, before housing costs: working-age adults 5,844k 5,751k 6,036k 6,136k 6,188k 6,228k 6,267k
Absolute poverty, before housing costs: pensioners 1,625k 1,550k 1,504k 1,517k 1,530k 1,539k 1,545k
Absolute poverty, after housing costs: everyone 14,858k 14,424k 14,825k 15,015k 15,271k 15,438k 15,663k
Absolute poverty, after housing costs: children 4,944k 4,610k 4,671k 4,688k 4,757k 4,821k 4,919k
Absolute poverty, after housing costs: working-age adults 8,410k 8,360k 8,748k 8,915k 9,052k 9,141k 9,257k
Absolute poverty, after housing costs: pensioners 1,504k 1,454k 1,405k 1,412k 1,461k 1,475k 1,487k
Relative poverty, before housing costs: everyone 11,894k 11,401k 11,466k 11,487k 11,424k 11,524k 11,707k
Relative poverty, before housing costs: children 3,917k 3,523k 3,519k 3,491k 3,451k 3,491k 3,549k
Relative poverty, before housing costs: working-age adults 6,200k 6,134k 6,337k 6,400k 6,386k 6,438k 6,537k
Relative poverty, before housing costs: pensioners 1,776k 1,744k 1,611k 1,597k 1,587k 1,595k 1,621k
Relative poverty, after housing costs: everyone 14,968k 14,591k 14,771k 14,823k 14,961k 15,048k 15,362k
Relative poverty, after housing costs: children 4,977k 4,676k 4,659k 4,634k 4,691k 4,712k 4,793k
Relative poverty, after housing costs: working-age adults 8,454k 8,451k 8,719k 8,845k 8,924k 8,983k 9,106k
Relative poverty, after housing costs: pensioners 1,537k 1,464k 1,393k 1,344k 1,346k 1,354k 1,463k
Inequality: head levels
Measure (head) 2025-26 2026-27 2027-28 2028-29 2029-30 2030-31 2031-32
Gini coefficient, equivalised HBAI income before housing costs 0.3086 0.3059 0.3066 0.3072 0.3076 0.3082 0.3096
Mean household net income in each side's own deciles: head levels
Measure (head) 2025-26 2026-27 2027-28 2028-29 2029-30 2030-31 2031-32
Decile 1: mean household_net_income per household £17,778 † £17,871 † £17,830 † £17,889 † £18,127 † £18,315 † £18,443 †
Decile 2: mean household_net_income per household £27,960 † £28,862 † £28,623 † £28,879 † £29,157 † £29,627 † £30,135 †
Decile 3: mean household_net_income per household £34,382 † £35,487 † £36,016 † £36,491 † £36,755 † £37,306 † £38,128 †
Decile 4: mean household_net_income per household £38,748 † £40,687 † £41,893 † £42,410 † £43,111 † £43,924 † £44,180 †
Decile 5: mean household_net_income per household £43,805 † £44,625 † £45,606 † £46,287 † £47,380 † £48,284 † £49,578 †
Decile 6: mean household_net_income per household £48,199 † £49,518 † £50,739 † £51,510 † £52,185 † £53,080 † £54,241 †
Decile 7: mean household_net_income per household £56,235 † £57,200 † £57,769 † £58,292 † £59,031 † £60,103 † £61,824 †
Decile 8: mean household_net_income per household £69,163 † £70,582 † £71,819 † £73,070 † £74,098 † £75,624 † £77,424 †
Decile 9: mean household_net_income per household £79,493 † £81,439 † £82,842 † £84,150 † £85,534 † £87,059 † £89,936 †
Decile 10: mean household_net_income per household £117,577 † £120,291 † £121,928 † £124,157 † £126,037 † £128,332 † £131,101 †
Standard reform costings (change in gov_balance): head levels
Measure (head) 2025-26 2026-27 2027-28 2028-29 2029-30 2030-31 2031-32
Raise basic rate by 1pp £6.47bn £6.76bn £6.67bn £6.86bn £7.06bn £7.35bn £7.55bn
Raise higher rate by 1pp £3.69bn £3.98bn £4.03bn £4.29bn £4.46bn £4.78bn £4.91bn
Raise personal allowance by ~800GBP/year £-3.82bn £-3.94bn £-4.01bn £-4.09bn £-4.18bn £-4.18bn £-2.00bn
Raise child benefit by 25GBP/week per additional child £-1.99bn £-1.75bn £-1.56bn £-1.46bn £-1.35bn £-1.24bn £-1.12bn
Reduce Universal Credit taper rate to 20% £-20.39bn £-20.96bn £-21.38bn £-21.57bn £-21.87bn £-22.17bn £-22.60bn
Raise Class 1 main employee NICs rate to 10% £11.02bn £11.42bn £11.77bn £12.07bn £12.37bn £12.75bn £13.07bn
Raise VAT standard rate by 2pp £24.89bn £25.55bn £26.14bn £26.75bn £27.39bn £28.03bn £28.66bn
Raise additional rate by 3pp £2.96bn £3.21bn £3.26bn £3.46bn £3.65bn suppressed suppressed
Raise the Universal Credit standard allowance by 10% £-3.02bn £-3.15bn £-3.26bn £-3.30bn £-3.34bn £-3.39bn £-3.44bn
Raise the State Pension by 10% £-7.79bn £-8.02bn £-8.01bn £-8.12bn £-8.43bn £-8.54bn £-8.87bn
Standard reforms: change in each reform's target: head levels
Measure (head) 2025-26 2026-27 2027-28 2028-29 2029-30 2030-31 2031-32
Raise basic rate by 1pp: change in income_tax £6.61bn £6.91bn £6.83bn £7.03bn £7.24bn £7.47bn £7.73bn
Raise higher rate by 1pp: change in income_tax £3.69bn £3.99bn £4.04bn £4.25bn £4.47bn £4.74bn £4.93bn
Raise personal allowance by ~800GBP/year: change in income_tax £-3.95bn £-4.08bn £-4.17bn £-4.24bn £-4.34bn £-4.41bn £-1.76bn
Raise child benefit by 25GBP/week per additional child: change in child_benefit £2.33bn £2.14bn £1.95bn £1.84bn £1.73bn £1.62bn £1.50bn
Reduce Universal Credit taper rate to 20%: change in universal_credit £20.39bn £20.95bn £21.37bn £21.62bn £21.87bn £22.22bn £22.61bn
Raise Class 1 main employee NICs rate to 10%: change in national_insurance £11.28bn £11.74bn £12.06bn £12.37bn £12.69bn £13.03bn £13.49bn
Raise VAT standard rate by 2pp: change in vat £24.69bn £25.35bn £25.94bn £26.54bn £27.17bn £27.81bn £28.43bn
Raise additional rate by 3pp: change in income_tax £2.96bn £3.21bn £3.26bn £3.46bn £3.66bn suppressed suppressed
Raise the Universal Credit standard allowance by 10%: change in universal_credit £3.02bn £3.15bn £3.26bn £3.32bn £3.39bn £3.44bn £3.48bn
Raise the State Pension by 10%: change in state_pension £10.68bn £11.18bn £11.31bn £11.55bn £12.00bn £12.44bn £12.94bn

@vahid-ahmadi

Copy link
Copy Markdown
Collaborator

Round 1 at 422885e99. Verdict: not ready to leave draft. The fix does what #2666 and #2667 asked, on both routes, and every one of our reproductions now gives the right answer. But CI has 7 failures at this head, all caused by the change, and one of them is a real change in results, not just a test that needs updating. The head also makes Microcosm runs about a quarter slower on CI (point 4).

Earlier points from our issue comments

New findings

  1. must: CI fails at this head (7 cases in 4 modules). I re-ran the four modules locally (core 3.33.12): all pass on main, and the same 7 cases fail at the head.

    • test_capital_gains_response_default.py::test_pay_rise_branches_hold_the_realisation_response and code_health/test_cached_arrays_not_written_in_place.py::test_branching_runs_with_read_only_cache[marginal_rates] read simulation.branches["adult_1_pay_rise"] after the rate is calculated. The new finally in marginal_tax_rate.py deletes that branch, so they hit a KeyError / empty set. If deleting the branch is what you want, these two tests need another way to reach the branch (for example a monkeypatched get_branch). I found nothing else in the PolicyEngine org that reads these branch names.
    • test_pension_credit_tax_branch_reuse.py::test_full_calculation_reuses_two_earnings_tax_counterfactuals (both cases, pension_credit and household_net_income): two extra counterfactual runs from uc_non_dep_pension_credit. I think the cause is clone() setting _cgt_calculation_depth = 0 (simulation.py:617). So a formula that makes a branch and calls set_input on it counts as "external" (:586-589). The branch then runs drop_calculated_values on its next read and throws away the values it inherited, which were meant to be reused. This is a cost increase, not a wrong result, but it hits every branch-based formula (the PC counterfactuals, uses_property_allowance, tax_credits_applicable_income, uc_non_dep_receives_pension_credit and the marginal-rate branches). One fix: treat a set_input as external only when no calculation is running on the simulation or its parent, for example by passing the parent's depth into the clone or by checking the tracer stack.
    • test_property_income_properties.py (3 cases): these are a change in results. The tests share a module-scoped simulation (pensioner_sim, :440). They set uses_property_allowance with set_input to switch a route, and rely on reset_calculations in set_pensioner_inputs (:499) to bring back the formula for the next example. At the head the reset keeps the override, so the route from an earlier example stays in place. A plain check shows the new contract: set child_benefit to formula + £1,000 on a built Simulation, call reset_calculations(), and you get £1,455.83 on main but £2,455.83 at the head.
  2. must: say plainly that reset_calculations has changed, and decide if that is wanted. This is the change behind the property-income failures. On main, reset_calculations undid any set_input made after the build (except what apply_reform named). At the head it keeps every input still stored. The PR weakened two existing assertions in test_reform_inputs_survive_reset.py to allow this (:210-229, :407-419). I should own our part here: in Simulation.apply_dynamics deletes inputs that a scenario's simulation modifier set with set_input #2666 point 4 we suggested making reset_calculations read _user_input_keys, and this is what that leads to. It is a reasonable contract for modifiers. But any caller that uses reset_calculations to undo an override now gets the override back. Inside the repo that covers these tests, and it could cover callers elsewhere. Either keep the new contract and fix the tests (using delete_arrays) and the changelog (which now says only "keep inputs stored by scenario modifiers"), or limit the wider retention to inputs set during a modifier or callback and leave the old reset alone.

  3. should: every top-level set_input on any built simulation now refreshes on the next read. That includes simulations with no modifier. It is correct (on main the values went stale), but it changes results for any script that edits a built simulation and then reads values it had already calculated. The two built-simulation set_input sites I found in policyengine-uk-data (targets/build_loss_matrix.py:332-341 and datasets/childcare/takeup_rate.py:51-77) set inputs before any dependent read, so they are not affected. Please say this in the changelog, not only "scenario modifiers and direct callbacks".

  4. must: the aggregate-impact report has not run on this PR, and the head is slower. Both runs (40189740365 and 40195966546) were cancelled at the 60-minute limit while measuring the head. In run 40195966546 the base took 1,820 s. On the same runner the head took 1,051 s to reach the point the base reached in 766 s (1.37x), and then about 117 s per step against 102 s (1.15x). So I expect the head to need about 2,250 s, about a quarter more than the base. The PR does not change uv.lock or pyproject.toml, so both sides ran with the same packages and the extra time comes from the code. Other PRs today pass with base and head at about 1,700 s each, so the limit is tight anyway, but this PR adds real time. My guess is the cause is point 1's second bullet (branches dropping inherited values), but I have not profiled it. Locally, with certified Microcosm 2024-25 and a £13,000 personal allowance reform, base and head give the same gov_balance (£256.543bn), household_net_income (£1,880.177bn), mean marginal_tax_rate and pip_reform totals. So I found no change in results without modifiers, but my laptop was too loaded for those timings to mean much. Please find and fix the slowdown, then get one run of this report to finish (a longer limit for one run would do) before the PR leaves draft.

    On the slowdown seen in our policyengine.py #718 review (a 99 s Microcosm build and a 104 s cold calculation, against 13.6 s and 3.0 s): that compared 2.132.6 with this head, 26 minor releases apart, so most of it is not from this PR. Against the merge base (2.158.3) I see about a quarter more time on CI, as above. On a synthetic one-adult, one-child household it is 2 to 10% more (build plus apply_dynamics 3.69 s against 3.52 s, marginal_tax_rate plus household_net_income 3.50 s against 3.21 s; two runs each, one process at a time). Where the rest of the gap from 2.132.6 comes from is still open.

  5. should: version floors. The new docs tell callers to use simulation.drop_computed_arrays() (docs/book/usage/scenarios.md), and policyengine.py Marriage allowance fix #718 calls it after each modifier step. That method comes from policyengine-core, not from this PR: it is in core 3.33.0 and later but not in 3.32.27, and pyproject.toml:27 still allows policyengine-core>=3.32.27 (the lock has 3.33.2). Raising the floor to >=3.33.0 would make the docs true for every install. For order: Marriage allowance fix #718 needs only core ≥3.33.0 to work, but it fixes the late-elasticity labour-supply response only on a policyengine-uk release that includes this PR, as Marriage allowance fix #718's own changelog says. So this PR should be released first, and Marriage allowance fix #718 should then pin to that version.

  6. nit: the capital-gains rate branch. marginal_tax_rate_on_capital_gains.py still uses the fixed name adult_{i}_cg_rise with a bare del at the end. It has no collision-safe name and no finally, unlike the two earnings rates. A user branch of that name would be reused and then deleted, and an exception would leave the branch behind.

  7. nit: after the after-data-load modifier, simulation.py:442-444 calls drop_calculated_values(self), but _external_input_change_pending is still True, so the first read drops everything a second time. Clearing the flag there avoids that.

What I checked and found fine

  • Private core methods: _stores_user_input and _get_visible_branch_names are already used on main (utils/input_dataframe_export.py:39,62), so this adds no new private-API dependency. Both exist in core 3.32.27.
  • Aliasing: retained_input_variables returns names only. Branch arrays are still core's lazy per-branch copies (get_branch docstring, core 3.33.12), and deleting the scratch branches frees memory that main kept.
  • Thread safety: _input_key_logs is keyed by id(simulation) and cleared in finally, the same as _reform_logs before.

What I ran: our scripts-iss2666 reproductions (repro2666.py, repro2667.py, the direct-route, input-only and late-elasticity scripts) at main and at the head. I also ran the PR's three test files on both, the 4 modules CI flagged on both, and the full CI logs for run 40195966541 (10,252 collected, 7 failed). I used fresh venvs with policyengine-core 3.33.12, while CI uses 3.33.2 from the lock. I read simulation.py, measurement_branches.py and the three rate formulas in full. I read the new test files in part, and docs/book/usage/scenarios.md only for the new section. For timing I compared CI job times for this PR with five other PR runs today, and timed the synthetic household at the head and at main, one after the other. I did not run the YAML tests (CI: pass) or profile where the extra time goes.

🤖 Generated with Claude Code

@juaristi22
juaristi22 force-pushed the fix-scenario-modifier-inputs-2666 branch from 422885e to bb05334 Compare October 25, 2026 20:04
@juaristi22

Copy link
Copy Markdown
Collaborator Author

Changes responding to the findings in the review at 422885e99:

  1. The seven CI failures are addressed without relaxing policy checks: tests capture the owned temporary earnings branches; nested formula writes preserve unrelated inherited caches and the exact two-call Pension Credit benchmark; reusable property fixtures delete only their own previous forced route. The historical R4 ledger is 1,021 native passes/one existing credential skip. After the additional default-baseline repair, the affected 148-case suite passed on 112f. On exact final bb053345/2b0, 37 Python cases and 44 HB YAML cases now pass with zero failures, errors, skips or xfails. These are bounded fresh selections, not a repeated full-suite claim or a sum with historical counts. Published-head remote CI is still required after the guarded push.
  2. Kept the explicit-input contract and documented it plainly. reset_calculations retains every remaining explicit input, including ordinary post-construction overrides. delete_arrays(variable, period), followed by reset/recalculation, removes an override and restores the formula/default.
  3. The guide and changelog now state that every top-level set_input on a built simulation refreshes dependents before the next outermost read, including callers with no modifier. Internal formula/framework writes keep selective invalidation.
  4. The ancestry guard fixes the original extra PC counterfactual calls. A later matched scalar trace proved pending framework invalidation was copied into the default baseline and drained again on its first read. R5 drains actual pending work successfully before that clone, preserving explicit inputs and the two-call counterfactual contract. Three causal regression failures became GREEN within the 148-case old-base affected suite. The completed public benchmark compares immutable actual 2b0 and final bb053345 archives with matched parameter topology and unchanged natural GC settings. Across three unprofiled repeats per snapshot/case, candidate/base median constructor-plus-query CPU ratios are P1 1.059243, P2 0.976757 and M 0.986241; query CPU ratios are 1.004210/0.939565/0.961550 and constructor CPU ratios 1.073633/1.045180/1.003336. Thus P1 retains a measured 5.9% total CPU increase. The paired economic/held differences are zero and Core/counter/clone/branch counts match; only the two PC queries have zero full-cache drops. M still has three query cleanup drops versus base two. The 30 primary constructions/two held controls/six profile pairs combine retained session 78047 and missing-candidate-M-only session 58600, with no valid case repeated. Original 78047 remains FAIL because its telemetry cap truncated only the untimed baseline-M held-query GC tail; all timed observations/GC data and the held economic comparison are valid. Completion 58600 passed, recording 20,042 natural-GC events under its 100,000 cap, with GC defaults restored. No uninterrupted-run PASS, zero-overhead, GC-causality or synthetic-to-population claim is made. The inherited workflow allows 120 minutes, but a completed certified report on the published head and reviewer acceptance remain readiness gates.
  5. Raised the declared Core floor to 3.33.0, where the official tagged changelog introduces drop_computed_arrays; resolved Core 3.33.2 and all other dependency versions/sources/hashes remain unchanged. Earlier Core 3.33.0 passed 17 R4 cases plus the public probe; those remain historical. The final constructor change passes a fresh 8-case minimum-Core gate; six separate actual merged Marriage allowance fix #718/Core 3.33.3 integration cases also pass, each selection with zero failures, errors, skips or xfails. Separate gates do not inflate native counts. Merged wrapper #718 closes Abolishing Marriage Allowance has no impact #717 and is included in released wrapper 6.11.0. Its UK/models/dev extras still pin UK 2.132.6 and Core 3.33.3. The remaining bridge is an appropriate native UK release containing this repair, then a wrapper UK-pin or installed-model update; updated pinned extras need a subsequent wrapper packaging release.
  6. CGT marginal rates now use the same fresh-name and identity-guarded finally cleanup as earnings rates, preserving occupied user branch objects/inputs and detaching only owned scratch on failure. The seven new cases include positive cold-rate and exception controls; CGT arithmetic is unchanged.
  7. Successful after-load modifier cleanup clears the pending flag after the actual drop completes, avoiding duplicate first-read cleanup. The default-baseline clone now likewise follows a successful real framework drain, and injected drain failure keeps pending inputs and prevents cloning. Callback/cleanup failure and independent Scenario baseline order remain intact.

Final native candidate is bb05334596c07d206113caab17d9bdc68b64726d on actual main 2b0b74f6, source UK 2.162.0 with frozen installed metadata 2.161.2/Core 3.33.2/Python 3.13.14. The current-base 37 Python/44 YAML and separate compatibility 8 Core330/6 actual718 selections all pass. Their CI runtimes are 313.614, 5.683, 31.089 and 43.862 seconds respectively; those are not benchmark timings. All import final UK source 2.162.0, while installed UK metadata remains native 2.161.2/minimum-Core 2.161.1/wrapper 2.158.2. The wrapper gate executes immutable merged #718 ba50a5f8 despite installed wrapper metadata 6.10.2; published wrapper 6.11.0 contains #718. The measured performance tradeoffs and telemetry limitation are disclosed above and in the description; full published-head CI/report and reviewer acceptance remain open. Full published-head CI/report and reviewer acceptance remain required. No repeated readiness/review nudge or national parity claim.

@juaristi22
juaristi22 marked this pull request as ready for review October 25, 2026 21:16
@vahid-ahmadi

Copy link
Copy Markdown
Collaborator

Round 2 at bb0533459. Against main 2b0b74f61 (the merge base). Thanks for the rework; every round 1 point is now dealt with, and I got one of them wrong (point 4, see below).

Round 1 findings

  1. must, CI failures (7 cases): fixed. The two branch tests now capture the owned scratch branches through a patched get_branch and check they are detached afterwards (code_health/test_cached_arrays_not_written_in_place.py:553-582, test_capital_gains_response_default.py:466-503). The Pension Credit counterfactual count is back to two: set_input is only "external" when no ancestor is calculating (simulation.py:593-600), and pending work is drained before the default baseline is cloned (simulation.py:431-433). The property fixtures delete only their own forced route before the reset (test_property_income_properties.py:105-107, :341-342, :504-505). All four modules pass locally.
  2. must, the reset_calculations contract: fixed, by keeping it. You kept the wider retention and now say so plainly in the changelog (changelog.d/fix-scenario-modifier-inputs-2666.fixed.md:1) and the guide (docs/book/usage/scenarios.md, new section), with delete_arrays as the way to undo an override. That was one of the two routes round 1 offered, so this closes it. I also checked who it could hit. Inside the repo, every dynamics call (dynamics/participation.py:371,388,395, progression.py:66,86, labour_supply.py:341) resets and then sets, so nothing changes. In the org, the other callers I found (free-childcare-reform, housing-policy-dashboard, uk-childcare-100k-threshold, scotland-income-tax-reform) either reset then set, or set variables that are stored in Microcosm (would_claim_tfc, childcare_expenses, rent), which main already kept. So I found no caller whose results change.
  3. should, every top-level set_input now refreshes: fixed. Changelog line 3 and the guide now say this applies with no modifier too.
  4. must, aggregate report and slowdown: fixed. Run 40229074134 finished: base 1,735.29 s, head 1,747.65 s for the same 220 figures (+0.7%), and no figure changed. One correction on our side: "about a quarter slower" in round 1 was an extrapolation from a cancelled run at 422885e99, not a measurement. I cannot now tell how much of it was real at that head and how much the ancestry guard in ace3990bd removed. Either way, the head is no slower on CI now. Local timings are below.
  5. should, version floor: fixed. pyproject.toml:27 is now policyengine-core>=3.33.0; the lock still resolves 3.33.2.
  6. nit, capital-gains branch: fixed. Same fresh name and identity-checked finally as the earnings rates (marginal_tax_rate_on_capital_gains.py:48-71).
  7. nit, double drop after the modifier: fixed. The flag is cleared after the drop (simulation.py:446-449), and test_default_constructor_first_reads_do_not_repeat_framework_drain covers it.

New findings

  1. nit: calculate clears the flag before the drop. At simulation.py:1451-1453 the flag goes to False and then drop_calculated_values runs. The other two sites (:431-433, :447-449) drop first and clear after, and test_default_baseline_drain_failure_retains_pending_inputs tests that order for the baseline. If the drop raised here, the stale values would stay with no retry. Swapping the two lines would make all three the same.
  2. nit: the guide says "before its next outermost read". Only calculate (and what goes through it, such as calculate_add and entity calls) drains the queue. A direct get_array or holder read straight after a set_input still sees the old cached value. "Before the next calculate" would be more exact.

Nothing at must or should.

Timings (base 2b0b74f61 against head bb0533459)

Step (Microcosm 2024-25, £13,000 personal allowance, 2026) Base Head
Build 192.9 s 178.4 s
gov_balance 190.6 s 178.7 s
marginal_tax_rate 165.8 s 159.3 s
Build with reform_pip_phase_in 193.2 s 181.0 s
gov_balance under it 184.7 s 178.5 s

Every result is identical on both sides: gov_balance £253.958bn, household_net_income £1,880.163bn, mean marginal_tax_rate 0.193006, PIP in 2029 £26.394bn, gov_balance under the PIP scenario £260.480bn. Head was 3 to 8% faster on each step, which I read as noise: one run each, on a busy laptop. On the synthetic household (bench_syn.py, two runs each) base gave 3.83 to 5.76 s for build plus apply_dynamics and 4.59 to 5.45 s for marginal_tax_rate plus household_net_income; head gave 4.50 to 4.55 s and 4.94 to 4.99 s. Taken with CI's +0.7%, I see no slowdown left. bench2.py makes no set_input after the build, so it does not test the new refresh path; the synthetic scripts and the reproductions below do.

What I ran: fresh worktrees and fresh uv sync --frozen venvs for base and head (policyengine-core 3.33.2, Python 3.13, the CI lock). At the head: the two new test files, test_reform_inputs_survive_reset.py, the four modules from round 1 (test_capital_gains_response_default.py, code_health/test_cached_arrays_not_written_in_place.py, test_pension_credit_tax_branch_reuse.py, test_property_income_properties.py) and test_measurement_branches.py, test_marginal_rates_hold_behavioural_responses.py, test_labour_supply_couples.py, test_marginal_rate_precision.py: 273 passed. Our round 1 scripts reset_contract.py, bench_syn.py and bench2.py, base then head, one process at a time. reset_contract.py gives £1,455.83 on base and £2,455.83 at the head, the documented new contract. Our #2666 / #2667 reproductions at the head (repro2666.py, repro2667.py, direct_fix.py, input_only.py) give the same answers as at 422885e99: MTR 0.676 and a response of −£34.24 in all four #2667 arms, retention 0.324 on both sides in #2666, and would_claim_tfc and childcare_expenses kept through a reset. I read the full diff of simulation.py, measurement_branches.py, the three rate formulas, the four edited test files and the new guide section, and the aggregate-impact log and report.

What I did not do: re-run the new tests on base (round 1 showed 38 of 66 failing on main), run the YAML tests (CI: pass), or read every case in test_scenario_modifier_inputs.py and test_direct_modifier_inputs.py (I read their test names only). The two nits above can go in a follow-up.

Verdict: ready to merge. CI is green (34 pass, 1 skipping is Publish), and the aggregate report passes with no change in any figure.

🤖 Generated with Claude Code

@vahid-ahmadi vahid-ahmadi left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving at bb05334; see the comment above.

@vahid-ahmadi
vahid-ahmadi added this pull request to the merge queue Oct 25, 2026
Merged via the queue into main with commit 235c0ef Oct 26, 2026
36 checks passed
@vahid-ahmadi
vahid-ahmadi deleted the fix-scenario-modifier-inputs-2666 branch October 26, 2026 00:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

2 participants