Repository navigation
Evaluate claims_all_entitled_benefits per benefit unit - #1831
Conversation
The formula summed its seven reported-benefit columns across the whole simulation (.sum() < 1), so in microdata the flag was False for every benefit unit whenever any benefit unit reported any of them, and would_claim_council_tax_reduction collapsed to FRS reporters only. Housing benefit, income support and the tax credits are unaffected: their own already-claiming eligibility gates carry the reporters-only behaviour independently of this flag. Refs PolicyEngine/policyengine-uk-data#448. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…path Review findings on the previous commit: the docstrings, test module docstring and changelog fragment said "reports any" where the formula tests the combined reported amount < £1, and the test named for single-benefit-unit semantics actually built two benefit units. Both now match the code: documentation states the threshold, and a genuine one-benefit-unit situation covers the calculator path. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
|
Ruling (Max, 2026-10-12, cos d1390): reporters only. This PR's per-benefit-unit fix lands with council tax reduction kept explicitly to families who report it, until a calibrated |
|
Ready for hub handoff at Tests: focused Python/property 9 passed; paired national CTR YAML 6 passed; formatting and lint pass. Two real enhanced-FRS simulations and all source/dataset/identity/flag checks pass. New-head CI run 38335603604 is in progress (YAML/lint/evidence/docs/Python 3.12 smoke pass; four Python shards running, Python 3.11/3.13/3.14 smokes queued as of 02:32Z). Previous head's complete full suite passed. Impact: CTR amounts and recipients exactly unchanged in UK/England/Scotland/Wales. HB +£749.855384m, +180,511 weighted recipient benefit units; household net income +£749.855374m. All 670 HB changes are new awards to nonreporters whose corrected family flag turns true. Other 21 measured payments are exactly unchanged on every record. Full dataset: 54,142 nonreporting units without a CTR data flag receive zero CTR; zero per-family flag mismatches. Dataset SHA256 Please dispatch the independent GPT-6.1 Sol review at Progress, reproducible aggregate evidence, |
|
Independent review (GPT-6.1 Sol, UK hub): REQUEST_CHANGES at VERDICT: REQUEST_CHANGES
The per-benefit-unit fix itself is correct. In claims_all_entitled_benefits.py:23, removing CTR satisfies d1390. would_claim_council_tax_reduction.py:16 defaults solely to reporters; its comment names d1390 and links uk-data#448. The aggregates show identical amounts and recipient status on every record, with zero CTR for all 54,142 non-reporting units without an override. Tests cover separate/shared households and all seven reports in test_claims_all_entitled_benefits.py:71. The property at line 202 checks zero claim flag and all three CTR outputs; ctr_reporters_only.yaml:1 pairs entitled non-reporters’ zero defaults with positive explicit-flag awards. No assertions were weakened or new skips/xfails introduced. The microsimulation test remains present and was explicitly deselected locally. Impact accounting reconciles: the live body includes both full SHAs and dataset hash; its country tables match Read-only static review; no tests or microsimulation run independently. |
vahid-ahmadi
left a comment
vahid-ahmadi
left a comment
There was a problem hiding this comment.
Needs changes before merge. The per-benefit-unit fix is right, but the new CTR default turns Council Tax Reduction off for every household entered without reports, which is every calculator household.
@MaxGhenis, I reviewed this at head 4f1cf4700.
-
must: calculator households lose all CTR (
variables/gov/local_authorities/council_tax_reduction/would_claim_council_tax_reduction.py:16-19).-
On main, a household with no reports has
claims_all_entitled_benefitstrue, sowould_claim_council_tax_reductionis true and CTR is paid. With this PR the default iscouncil_tax_benefit_reported > 0alone, so a household with no reports gets £0 unless the caller sets the flag. -
Probe, 2026, single adult, private rent £8,000, council tax £1,800, no earnings, no roles or reports. I ran the branch, then the same tree with the two changed variable files restored from main
1d3abecef:Household CTR main CTR branch Net income change 70, North West £1,800.00 £0 −£1,800.00 70, Wales £1,800.00 £0 −£1,800.00 30, Scotland £633.96 £0 −£633.96 -
On the enhanced FRS this changes nothing, because main's simulation-wide
.sum()already madeclaims_all_entitled_benefitsfalse for every family. The body's own figure shows it: 54,276 family flags change with the per-unit fix. That is why the survey CTR table is identical. The cost lands on households entered by hand instead. -
The PR adds 193
would_claim_council_tax_reduction: truefixture lines to keep existing expectations. That is the same effect, seen in the fixtures. -
The d1390 comment is about survey take-up. It doesn't say calculator households should stop getting CTR, and the body and changelog don't mention it.
-
Fix: keep
claims_all_entitled_benefits | reported_ctras the formula default, and get reporters-only on survey runs from the data side. That could be awould_claim_council_tax_reductioncolumn in the dataset (the end state in uk-data#448) or an interim column set fromcouncil_tax_benefit_reported. This is not a one-line revert. With the per-unit fix, restoring the OR on its own would turn on CTR for those 54,276 survey families, as happened with HB. So the OR can only come back together with a uk-data release that supplies the column, sequenced with this PR like the HB ruling. -
If reporters-only for every input is really what is meant, it needs its own explicit ruling and a line in the changelog. The app and API would then have to send the flag.
-
-
should: hold the merge until the Housing Benefit ruling is in and applied. The +£749.9m HB from 670 new non-reporter awards is still open as d1455 in the 13 October comment. Whichever way it goes,
housing-benefit.ipynbandgov/dwp/housing_benefit/takeup.yamlstill say enhanced FRS take-up follows reported claims, and need to match the ruling. Calculator households are not affected here, because a single benefit unit's sum was already per unit. -
nit: the docstring says "the seven means-tested benefits the formula checks" (
claims_all_entitled_benefits.py:9-17). The list matches the seven_reportedvariables at lines 27-33. It might be worth saying that CTR, Pension Credit and Child Benefit are not among them, so a family reporting only Pension Credit still counts as claiming everything. -
nit: main has moved 115 commits since your last merge (
1d3abecef).git merge-treeagainst current main (714b1438c) is clean. CI at this head is green on every job (YAML, the four Python shards, lint, fix evidence, docs and the smoke imports).
Repo process: the .fixed.md fragment fits. Reproducer: and axiom: lines are present, and Fix evidence passes. No file overlaps with #1820, #1855, #1864, #2184, #2187, #2189, #2193 or #2382.
What I checked
- Read the full body, the three comments and the whole diff against
1d3abecef. - Ran the 14 changed YAML files: 207 cases passed. Ran the 11 changed pytest files with
-m 'not microsimulation': 54 passed, 1 deselected. I did not run the enhanced FRS microsimulation test, because I have no access to the data on this machine. - Restored main's
.sum()inclaims_all_entitled_benefits.py.TestPerBenefitUnit::test_one_reporter_does_not_switch_off_the_other_benefit_unitthen fails, so the reproducer catches the bug. - Ran the household probe in finding 1 on the branch and on main's two variable files.
ruff checkandruff format --checkon the 13 changed Python files: clean.- Take-up is a modelling assumption, not a statutory rule, so I had no legislation to check it against. I did not re-derive the survey figures in the body.
🤖 Generated with Claude Code
|
@vahid-ahmadi, addressed each point at
Validation:
The exact per-benefit-unit reproducer fails with main's 🤖 Generated with Claude Code |
Aggregate impact on certified data: passBase Checks: 0 broken by this change, 1 failing on both sides, 0 fixed. Moves flagged for review: 0 (thresholds in Notes
Moves flagged for reviewNone. Spending by benefit and service
Caseloads
Household net income by decileNo household's net income changes in any year. No change in any year: receipts by tax head; government balance, each component counted once; model totals; components the model counts twice; household income totals; population; poverty rates; people in poverty; inequality; mean household net income in each side's own deciles; standard reform costings (change in gov_balance); standard reforms: change in each reform's target. Checks failing on both base and head (1)
Counted-once checkEach of 84 components was raised by £1 per unit in 2026-27, and every head of its totals was read household by household: its own head must rise by exactly £1, no other tax head may move, no other benefit may rise, and the model's own total must move by what its heads moved. A head that breaks a rule is measured again twice: with every other head held at its baseline value, and with every other head held at its value after the first £1 while the component rises again. It fails where either still shows the response. 0 failed; 0 were skipped.
Provenance
Dataset: Receipts by tax head: head levels
Spending by benefit and service: head levels
Caseloads: head levels
Government balance, each component counted once: head levels
Model totals: head levels
Components the model counts twice: head levels
Household income totals: head levels
Population: head levels
Poverty rates: head levels
People in poverty: head levels
Inequality: head levels
Mean household net income in each side's own deciles: head levels
Standard reform costings (change in gov_balance): head levels
Standard reforms: change in each reform's target: head levels
|
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
claims_all_entitled_benefits is now false for every family on data-built simulations, and is still tested per family in hand-entered situations. Main summed the seven reported benefits over the whole simulation, so on survey data the flag was false everywhere. Tested per family on data, every non-reporting survey family turned true. would_claim_CTC, _WTC and _IS read the flag, and tax credits pre-minimum is gated only by take-up, so tax credits leaked into CTR and HB means tests, tax credit awards and the 2024 Winter Fuel Payment benefit test (review r6). The gate sits in the flag, so every reader keeps main's survey value, and a dataset-supplied flag still overrides it. Tests: review r6's MOCK reproducer, which fails on 94d5ebe; a property test that on data every take-up flag reading the flag equals the family's own report, and every downstream output equals explicit reported-only take-up; the microdata test now checks main's survey value. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Clean merge. The only file both sides changed is test_benefit_cap_specified_benefits_properties.py: this branch's capped_family claim flag and main's three LCWRA wrapper tests touch separate hunks, and both are kept. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Both properties now suppress Hypothesis's too_slow check, as most of the repo's property tests do. On this host at load 170-240, one draw stalled for 4.9 s in the CTR property and one for 11.8 s in the tax credit property, between simulations. The check allows 30 s over the first ten inputs. Every example and every other check is kept. Round 7 saw the CTR property fail once in a parallel batch, and no traceback was printed. That run's cause cannot be recovered, so this removes one measured candidate rather than a confirmed one. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Hand-off from the UK hub build job (round 7b): head Pushed
CI at the head: all 17 checks pass, including Python shards 1–5, YAML and Fix evidence. Aggregate impact against main Impact for 2024–2026 (round 7, certified 1.58.1): all 32 vectors are byte-identical to main in each year. In 2024, Reproducer: this round re-ran it against a Flaky test (
The cause of round 7's single failure can't be recovered, because its traceback was never printed. With Other checks: Open for the hub:
The PR body is updated. Evidence is in the job's |
|
Merged by the UK hub (session local_8e19f9d5, Opus 5.5) as
🤖 Generated with Claude Code |
2 participants
Fixes #2393
Evaluate
claims_all_entitled_benefitsindependently for each benefit unit in hand-entered situations, keep main's result on data-built simulations, and keep calculator take-up and survey reporters-only defaults for Council Tax Reduction (CTR) and Housing Benefit (HB).Root cause
Main's formula is
add(benunit, period, SEVEN).sum() < 1, where SEVEN is the seven reported means-tested benefits (Child Tax Credit, Working Tax Credit, Universal Credit, HB, income-based JSA, Income Support and income-related ESA). The.sum()totals those reports over the whole simulation and broadcasts one value to every benefit unit:This PR first replaced the simulation-wide
.sum()with a per-family test everywhere. On data-built simulations, every non-reporting survey family then turned true: 54,276 benefit units on the Enhanced FRS.would_claim_CTC,would_claim_WTCandwould_claim_ISarereported | claims_all_entitled_benefits, andchild_tax_credit_pre_minimumandworking_tax_credit_pre_minimumare gated by take-up only. So review r6 found tax credits leaking on 2024 survey data:tax_creditsrose from £156m to £6,151m;2025 and 2026 matched main only because tax credits stop paying on 5 April 2025 (
gov.dwp.tax_credits.active).Fix
claims_all_entitled_benefitsreturns false for every family whenbuilt_from_data(simulation), and the per-family test otherwise. The flag has five readers:would_claim_CTC,_WTC,_IS,would_claim_housing_benefitandwould_claim_council_tax_reduction. All five get main's survey value from the flag itself, so no reader needs its own gate for it:claims_all_entitled_benefitscolumn in a dataset still overrides the formula, as on main.The alternative, gating the three legacy
would_claim_*formulas, would leave the published flag itself at 54,276 true values on survey data.Behaviour and rulings
claims_all_entitled_benefits | positive_reportrule per benefit unit. Vahid's three 2026 calculator probes receive £1,800, £1,800 and £633.96 CTR without reports or take-up inputs.Intended differences from main on data-built simulations
gov.dwp.income_support.active) still pay, and only if it reports none of the seven but is entitled.wrapper.policybuilds one-row datasets for 2025 and later. Those contract cases have not been re-read against this condition.Data-source detection
built_from_datacovers UKSimulation,Microsimulation, core simulations over the UK system, clones, branches and UKbuild_from_situationrebuilds. All 20 supported routes are pinned with synthetic pytest coverage.Known limitation: in core 3.33.2, directly calling
SimulationBuilder.build_from_dicton an existing data-built object leaves stale source flags and the old dataset, with no reliable situation marker. The four unsupported overwrite routes (core overwrite, its clone, its new branch, and UK overwrite via the core builder) stay misclassified as data: 20 of 24 routes are correct. core#627, which is open, is the right place to fix them. Use UK's supportedbuild_from_situation. The helper and the dataset guidance document this. No population-size, ID-type or weight heuristic was introduced.Tests
Reproducer for r6's leak:
test_data_built_nonreporters_get_no_tax_creditsis r6's MOCK probe. A non-reporting lone parent (child 5, £12,000 earnings, North West private renter) and a non-reporting working pensioner sit beside a family that reports Child Tax Credit and CTR, in 2024. On data, every flag is false and only the reporter claims Child Tax Credit. The lone parent gets no tax credits and no tax credit award, and is not on a Winter Fuel Payment relevant benefit. Entered by hand, the same families keep the per-family flag[True, True, False]. It fails on94d5ebe32(the data-built flag is[True, True, False]) and passes at the head.Property:
test_data_built_takeup_equals_reported_onlycovers MOCK data in 2024 and 2026, for one to three families with any subset of the seven reports and CTR. On data:claims_all_entitled_benefitsequals the family's own report;It fails on
94d5ebe32and passes at the head.Microdata:
test_survey_takeup_follows_reports_in_microdatanow asserts main's survey value: the flag is false for every family in 2024 and 2026, and CTC, WTC and IS take-up equal reports. It runs only with the certified dataset; round 7 ran it on certified 1.58.1, where it passes at the fix and fails on94d5ebe32.Earlier tests in this PR (unchanged):
Hypothesis timing (
928a82d7d): both take-up properties now suppress onlyHealthCheck.too_slow, as 123 of the repo's other 150 Hypothesis test files do. Their examples and every other check are unchanged. On a saturated host (load 170–240), one input draw stalled for 4.9 s in the CTR property and one for 11.8 s in the tax credit property, between simulations. The check allows 30 s over the first ten inputs. The CTR property failed once in a parallel batch during round 7, with no traceback printed. That run's cause can't be recovered, so this removes a measured candidate, not a confirmed cause.Impact on certified Enhanced FRS 1.58.1
Round 7 ran 2024, 2025 and 2026 default-law
Microsimulations, sequentially and aggregates only. It compared main60b09a4e5with the fixf56cfc50c, using the same locked environment and the dataset read in place (SHA-256056e80fd8e9151d78f432a25cde863340baee81304c6216a37fb575ce8563784, matchingcertified_dataset.yaml).All 32 vectors are byte-identical (SHA-256) to main in each of the three years. The vectors are the union of r6's lists plus three:
would_claim_*;council_tax_reduction(£m)housing_benefit(£m)tax_credits(£m)tax_credit_award(£m)child_tax_credit(£m)working_tax_credit(£m)income_support(£m)winter_fuel_payment(£m)pension_age_winter_heating_payment(£m)household_net_income(£m)claims_all_entitled_benefitstrue (benefit units, unweighted)would_claim_CTC/_WTC/_IStrue (unweighted)would_claim_housing_benefit/_council_tax_reductiontrue (unweighted)Sums use household or benefit-unit weights as the variable's entity requires, in float64. r6 measured 2024
tax_creditsat £6,150.836m on94d5ebe32; the fix returns it to main's £155.975m. Main's 2024 hashes equal r6's main hashes.origin/main9d81a5dfahas since been merged (97bc6b874, clean). None of its commits changesclaims_all_entitled_benefitsor its five readers, and928a82d7dchanges test settings only, so the impact was not rerun. On a dataset where some family reports one of the seven and no column supplies the flag, main's flag is false for every family. Main's CTR and HB take-up,flag | report, is then the report, which is exactly this PR's data-built rule. So the identity holds by construction, and it was measured above. Absolute totals at later main commits can differ on both sides alike.Disclosures from review r6
legacy_award_readers/income_support_take_up_and_person_awards.yaml, the non-reporting lodger's benefit unit now carriesclaims_all_entitled_benefits: false; this was not disclosed before. In that hand-entered case another member reports Income Support. On main, that report switched the flag off for every family. Under the per-family rule the lodger's flag would be true, and so would itswould_claim_IS(reported | claims_all_entitled_benefits), which the case expects to be false. The input keeps the case on its subject, capital sharing.would_claim_council_tax_reduction: trueinputs and 6claims_all_entitled_benefits: trueinputs (12 and 9 lines in total, counting output assertions), all in the newctr_takeup.yaml(its explicit-override and calculator cases), plus the lodger input above.Validation
At
97bc6b874(main merged) and928a82d7d:uvx ruff format --check .passes on 1,920 files.97bc6b874, all 17 runnable example tests pass (-n 4), and both properties pass serially (CTR 32 inputs, tax credits 12). At928a82d7d, the tax credit property passes serially (12 inputs); the CTR property passed all 23 inputs it reached before a 500 s guard (no failure).test_disable_simulated_benefits.pyandtest_tax_credits_pension_credit_passport_properties.pyshow 48 passed and none failed before the guard.The host ran at load 70–240 throughout, so suites ran in timed slices. Run-level results for the flaky-test check are in the hand-off comment. CI at the head runs the full suite.
Independent final-head review and CI remain for the hub; this build job does not review or merge its own work.
Reproducer:
policyengine_uk/tests/test_claims_all_entitled_benefits.py::test_data_built_nonreporters_get_no_tax_credits,policyengine_uk/tests/test_claims_all_entitled_benefits.py::test_data_built_takeup_equals_reported_only,policyengine_uk/tests/test_claims_all_entitled_benefits.py::TestPerBenefitUnit::test_one_reporter_does_not_switch_off_the_other_benefit_unit,policyengine_uk/tests/test_claims_all_entitled_benefits.py::test_ctr_is_zero_for_every_nonreporter_without_an_explicit_claim_flag,policyengine_uk/tests/test_claims_all_entitled_benefits.py::test_mock_data_takeup_is_reporters_onlyaxiom: n/a: survey take-up assumptions and vectorisation, with no statutory policy rule change.
🤖 Generated with Claude Code