From 6d72d727c5327f1f9846693ca2fd13f273850521 Mon Sep 17 00:00:00 2001 From: Anthony Volk <14987227+anth-volk@users.noreply.github.com> Date: Fri, 9 Oct 2026 18:11:54 +0400 Subject: [PATCH 01/12] Apply Housing Benefit accommodation disregards within the first fiscal year --- changelog.d/hb-accommodation-dates.fixed.md | 1 + .../couple/minors.yaml | 10 +- .../couple/older.yaml | 10 +- .../couple/younger.yaml | 10 +- .../lone_parent/older.yaml | 10 +- .../lone_parent/younger.yaml | 10 +- .../single/older.yaml | 10 +- .../single/younger.yaml | 10 +- ..._or_temporary_accommodation_disregard.yaml | 484 +++++++++++++++++- ...est_housing_benefit_accommodation_dates.py | 222 ++++++++ ...ing_benefit_applicable_income_disregard.py | 139 ++--- ...ed_or_temporary_accommodation_disregard.py | 133 +++-- .../housing_benefit_entitlement.py | 55 +- 13 files changed, 947 insertions(+), 157 deletions(-) create mode 100644 changelog.d/hb-accommodation-dates.fixed.md create mode 100644 policyengine_uk/tests/test_housing_benefit_accommodation_dates.py diff --git a/changelog.d/hb-accommodation-dates.fixed.md b/changelog.d/hb-accommodation-dates.fixed.md new file mode 100644 index 0000000000..dc4921837d --- /dev/null +++ b/changelog.d/hb-accommodation-dates.fixed.md @@ -0,0 +1 @@ +- Applied Housing Benefit accommodation earnings disregards from their statutory commencement within the first fiscal year, calculating awards before combining date segments. diff --git a/policyengine_uk/parameters/gov/dwp/housing_benefit/means_test/income_disregard/specified_or_temporary_accommodation/couple/minors.yaml b/policyengine_uk/parameters/gov/dwp/housing_benefit/means_test/income_disregard/specified_or_temporary_accommodation/couple/minors.yaml index b44d66a347..06d3a3ab8a 100644 --- a/policyengine_uk/parameters/gov/dwp/housing_benefit/means_test/income_disregard/specified_or_temporary_accommodation/couple/minors.yaml +++ b/policyengine_uk/parameters/gov/dwp/housing_benefit/means_test/income_disregard/specified_or_temporary_accommodation/couple/minors.yaml @@ -3,15 +3,13 @@ values: # Schedule 4 paragraph 18 (NI: Schedule 5 paragraph 18) has effect from 5 # October 2026: SI 2026/753 reg. 2(4), as substituted by SI 2026/978 reg. # 2(2), and NI SR 2026/157 reg. 2(4). There was no such disregard before. - # Neither instrument uprates the sums. The model reads each fiscal year at - # 30 April, so 2026 (2026-27) has no paragraph 18 disregard although it - # applied from 5 October 2026 (183 of the year's 365 days); 2027 onwards - # has the full amounts. fiscal_year_blend is not used: half of an amount - # for the whole year is not the full amount for half the year, because - # the disregard is capped at net earnings and counts in the £17.10 test. + # Neither instrument uprates the sums. Preserve the legal date so awards + # can be calculated separately before/after commencement, including the + # earnings cap and additional-disregard test, then combined over the year. 2006-03-06: 0 2026-10-05: 97.33 metadata: + preserve_calendar_dates: true period: week unit: currency-GBP label: Housing benefit specified or temporary accommodation earnings disregard, couple both under 18 diff --git a/policyengine_uk/parameters/gov/dwp/housing_benefit/means_test/income_disregard/specified_or_temporary_accommodation/couple/older.yaml b/policyengine_uk/parameters/gov/dwp/housing_benefit/means_test/income_disregard/specified_or_temporary_accommodation/couple/older.yaml index 93d44c2c49..cc5cde75ce 100644 --- a/policyengine_uk/parameters/gov/dwp/housing_benefit/means_test/income_disregard/specified_or_temporary_accommodation/couple/older.yaml +++ b/policyengine_uk/parameters/gov/dwp/housing_benefit/means_test/income_disregard/specified_or_temporary_accommodation/couple/older.yaml @@ -3,15 +3,13 @@ values: # Schedule 4 paragraph 18 (NI: Schedule 5 paragraph 18) has effect from 5 # October 2026: SI 2026/753 reg. 2(4), as substituted by SI 2026/978 reg. # 2(2), and NI SR 2026/157 reg. 2(4). There was no such disregard before. - # Neither instrument uprates the sums. The model reads each fiscal year at - # 30 April, so 2026 (2026-27) has no paragraph 18 disregard although it - # applied from 5 October 2026 (183 of the year's 365 days); 2027 onwards - # has the full amounts. fiscal_year_blend is not used: half of an amount - # for the whole year is not the full amount for half the year, because - # the disregard is capped at net earnings and counts in the £17.10 test. + # Neither instrument uprates the sums. Preserve the legal date so awards + # can be calculated separately before/after commencement, including the + # earnings cap and additional-disregard test, then combined over the year. 2006-03-06: 0 2026-10-05: 119.7 metadata: + preserve_calendar_dates: true period: week unit: currency-GBP label: Housing benefit specified or temporary accommodation earnings disregard, couple with one member 25 or over diff --git a/policyengine_uk/parameters/gov/dwp/housing_benefit/means_test/income_disregard/specified_or_temporary_accommodation/couple/younger.yaml b/policyengine_uk/parameters/gov/dwp/housing_benefit/means_test/income_disregard/specified_or_temporary_accommodation/couple/younger.yaml index 0e156837f2..e17570f62f 100644 --- a/policyengine_uk/parameters/gov/dwp/housing_benefit/means_test/income_disregard/specified_or_temporary_accommodation/couple/younger.yaml +++ b/policyengine_uk/parameters/gov/dwp/housing_benefit/means_test/income_disregard/specified_or_temporary_accommodation/couple/younger.yaml @@ -3,15 +3,13 @@ values: # Schedule 4 paragraph 18 (NI: Schedule 5 paragraph 18) has effect from 5 # October 2026: SI 2026/753 reg. 2(4), as substituted by SI 2026/978 reg. # 2(2), and NI SR 2026/157 reg. 2(4). There was no such disregard before. - # Neither instrument uprates the sums. The model reads each fiscal year at - # 30 April, so 2026 (2026-27) has no paragraph 18 disregard although it - # applied from 5 October 2026 (183 of the year's 365 days); 2027 onwards - # has the full amounts. fiscal_year_blend is not used: half of an amount - # for the whole year is not the full amount for half the year, because - # the disregard is capped at net earnings and counts in the £17.10 test. + # Neither instrument uprates the sums. Preserve the legal date so awards + # can be calculated separately before/after commencement, including the + # earnings cap and additional-disregard test, then combined over the year. 2006-03-06: 0 2026-10-05: 61.53 metadata: + preserve_calendar_dates: true period: week unit: currency-GBP label: Housing benefit specified or temporary accommodation earnings disregard, couple both under 25, one 18 or over diff --git a/policyengine_uk/parameters/gov/dwp/housing_benefit/means_test/income_disregard/specified_or_temporary_accommodation/lone_parent/older.yaml b/policyengine_uk/parameters/gov/dwp/housing_benefit/means_test/income_disregard/specified_or_temporary_accommodation/lone_parent/older.yaml index a038a6223f..584388350a 100644 --- a/policyengine_uk/parameters/gov/dwp/housing_benefit/means_test/income_disregard/specified_or_temporary_accommodation/lone_parent/older.yaml +++ b/policyengine_uk/parameters/gov/dwp/housing_benefit/means_test/income_disregard/specified_or_temporary_accommodation/lone_parent/older.yaml @@ -3,15 +3,13 @@ values: # Schedule 4 paragraph 18 (NI: Schedule 5 paragraph 18) has effect from 5 # October 2026: SI 2026/753 reg. 2(4), as substituted by SI 2026/978 reg. # 2(2), and NI SR 2026/157 reg. 2(4). There was no such disregard before. - # Neither instrument uprates the sums. The model reads each fiscal year at - # 30 April, so 2026 (2026-27) has no paragraph 18 disregard although it - # applied from 5 October 2026 (183 of the year's 365 days); 2027 onwards - # has the full amounts. fiscal_year_blend is not used: half of an amount - # for the whole year is not the full amount for half the year, because - # the disregard is capped at net earnings and counts in the £17.10 test. + # Neither instrument uprates the sums. Preserve the legal date so awards + # can be calculated separately before/after commencement, including the + # earnings cap and additional-disregard test, then combined over the year. 2006-03-06: 0 2026-10-05: 77.73 metadata: + preserve_calendar_dates: true period: week unit: currency-GBP label: Housing benefit specified or temporary accommodation earnings disregard, lone parent aged 25 or over diff --git a/policyengine_uk/parameters/gov/dwp/housing_benefit/means_test/income_disregard/specified_or_temporary_accommodation/lone_parent/younger.yaml b/policyengine_uk/parameters/gov/dwp/housing_benefit/means_test/income_disregard/specified_or_temporary_accommodation/lone_parent/younger.yaml index 4875f03194..4a858af7ec 100644 --- a/policyengine_uk/parameters/gov/dwp/housing_benefit/means_test/income_disregard/specified_or_temporary_accommodation/lone_parent/younger.yaml +++ b/policyengine_uk/parameters/gov/dwp/housing_benefit/means_test/income_disregard/specified_or_temporary_accommodation/lone_parent/younger.yaml @@ -3,15 +3,13 @@ values: # Schedule 4 paragraph 18 (NI: Schedule 5 paragraph 18) has effect from 5 # October 2026: SI 2026/753 reg. 2(4), as substituted by SI 2026/978 reg. # 2(2), and NI SR 2026/157 reg. 2(4). There was no such disregard before. - # Neither instrument uprates the sums. The model reads each fiscal year at - # 30 April, so 2026 (2026-27) has no paragraph 18 disregard although it - # applied from 5 October 2026 (183 of the year's 365 days); 2027 onwards - # has the full amounts. fiscal_year_blend is not used: half of an amount - # for the whole year is not the full amount for half the year, because - # the disregard is capped at net earnings and counts in the £17.10 test. + # Neither instrument uprates the sums. Preserve the legal date so awards + # can be calculated separately before/after commencement, including the + # earnings cap and additional-disregard test, then combined over the year. 2006-03-06: 0 2026-10-05: 61.41 metadata: + preserve_calendar_dates: true period: week unit: currency-GBP label: Housing benefit specified or temporary accommodation earnings disregard, lone parent under 25 diff --git a/policyengine_uk/parameters/gov/dwp/housing_benefit/means_test/income_disregard/specified_or_temporary_accommodation/single/older.yaml b/policyengine_uk/parameters/gov/dwp/housing_benefit/means_test/income_disregard/specified_or_temporary_accommodation/single/older.yaml index 63bc144f20..62a3ddf015 100644 --- a/policyengine_uk/parameters/gov/dwp/housing_benefit/means_test/income_disregard/specified_or_temporary_accommodation/single/older.yaml +++ b/policyengine_uk/parameters/gov/dwp/housing_benefit/means_test/income_disregard/specified_or_temporary_accommodation/single/older.yaml @@ -3,15 +3,13 @@ values: # Schedule 4 paragraph 18 (NI: Schedule 5 paragraph 18) has effect from 5 # October 2026: SI 2026/753 reg. 2(4), as substituted by SI 2026/978 reg. # 2(2), and NI SR 2026/157 reg. 2(4). There was no such disregard before. - # Neither instrument uprates the sums. The model reads each fiscal year at - # 30 April, so 2026 (2026-27) has no paragraph 18 disregard although it - # applied from 5 October 2026 (183 of the year's 365 days); 2027 onwards - # has the full amounts. fiscal_year_blend is not used: half of an amount - # for the whole year is not the full amount for half the year, because - # the disregard is capped at net earnings and counts in the £17.10 test. + # Neither instrument uprates the sums. Preserve the legal date so awards + # can be calculated separately before/after commencement, including the + # earnings cap and additional-disregard test, then combined over the year. 2006-03-06: 0 2026-10-05: 77.73 metadata: + preserve_calendar_dates: true period: week unit: currency-GBP label: Housing benefit specified or temporary accommodation earnings disregard, single claimant aged 25 or over diff --git a/policyengine_uk/parameters/gov/dwp/housing_benefit/means_test/income_disregard/specified_or_temporary_accommodation/single/younger.yaml b/policyengine_uk/parameters/gov/dwp/housing_benefit/means_test/income_disregard/specified_or_temporary_accommodation/single/younger.yaml index 10cfa31c01..99591d498e 100644 --- a/policyengine_uk/parameters/gov/dwp/housing_benefit/means_test/income_disregard/specified_or_temporary_accommodation/single/younger.yaml +++ b/policyengine_uk/parameters/gov/dwp/housing_benefit/means_test/income_disregard/specified_or_temporary_accommodation/single/younger.yaml @@ -3,15 +3,13 @@ values: # Schedule 4 paragraph 18 (NI: Schedule 5 paragraph 18) has effect from 5 # October 2026: SI 2026/753 reg. 2(4), as substituted by SI 2026/978 reg. # 2(2), and NI SR 2026/157 reg. 2(4). There was no such disregard before. - # Neither instrument uprates the sums. The model reads each fiscal year at - # 30 April, so 2026 (2026-27) has no paragraph 18 disregard although it - # applied from 5 October 2026 (183 of the year's 365 days); 2027 onwards - # has the full amounts. fiscal_year_blend is not used: half of an amount - # for the whole year is not the full amount for half the year, because - # the disregard is capped at net earnings and counts in the £17.10 test. + # Neither instrument uprates the sums. Preserve the legal date so awards + # can be calculated separately before/after commencement, including the + # earnings cap and additional-disregard test, then combined over the year. 2006-03-06: 0 2026-10-05: 61.41 metadata: + preserve_calendar_dates: true period: week unit: currency-GBP label: Housing benefit specified or temporary accommodation earnings disregard, single claimant under 25 diff --git a/policyengine_uk/tests/policy/baseline/finance/benefit/family/housing_benefit/applicable_income/housing_benefit_specified_or_temporary_accommodation_disregard.yaml b/policyengine_uk/tests/policy/baseline/finance/benefit/family/housing_benefit/applicable_income/housing_benefit_specified_or_temporary_accommodation_disregard.yaml index d6027e8a89..31287d5870 100644 --- a/policyengine_uk/tests/policy/baseline/finance/benefit/family/housing_benefit/applicable_income/housing_benefit_specified_or_temporary_accommodation_disregard.yaml +++ b/policyengine_uk/tests/policy/baseline/finance/benefit/family/housing_benefit/applicable_income/housing_benefit_specified_or_temporary_accommodation_disregard.yaml @@ -20,8 +20,9 @@ # 5,061.16; £61.53 → 3,199.56; £119.70 → 6,224.40; £5 → 260; £10 → 520; £25 # → 1,300; £17.10 → 889.20. # -# The model reads each fiscal year at 30 April: 2027 (2027-28) has para 18 in -# full; 2026 (2026-27) reads 30 April 2026, before 5 October, so it has none. +# The accommodation amount retains its legal date: 2026-27 has 182 days +# before 5 October and 183 days afterwards. Completed awards are combined +# after earnings caps, childcare, the income threshold and taper. # Earnings are below the income tax personal allowance and the National # Insurance primary threshold (both £12,570), so net earnings equal gross # earnings, and the hours are below every £17.10 work condition unless a @@ -401,7 +402,7 @@ housing_benefit_specified_or_temporary_accommodation_disregard: 0 housing_benefit_applicable_income_disregard: 260 -- name: In 2026 the model reads 30 April 2026, before para 18 came into force +- name: The 2026 fiscal year includes the 183 days from 5 October period: 2026 absolute_error_margin: 0.01 input: @@ -415,11 +416,9 @@ members: [person] in_specified_or_temporary_accommodation: true output: - # KNOWN LIMITATION, intended: para 18 applies from 5 October 2026, for 183 - # of the 365 days of 2026-27, but the model reads the year's law at 30 - # April, so 2026-27 has no para 18 disregard. - housing_benefit_specified_or_temporary_accommodation_disregard: 0 - housing_benefit_applicable_income_disregard: 260 # £5 × 52 + # Full £77.73 × 52 annual equivalent for 183/365 of the year. + housing_benefit_specified_or_temporary_accommodation_disregard: 2026.51693 + housing_benefit_applicable_income_disregard: 2286.51693 - name: In 2025 there is no para 18 disregard period: 2025 @@ -680,9 +679,9 @@ age: 30 employment_income: 7_800 # £150 a week weekly_hours: 16 - childcare_expenses: 2_080 # £40 a week child: age: 5 + housing_benefit_qualifying_childcare_charges: 2_080 # pre-assessed £40/week benunits: benunit: would_claim_uc: false # not on Universal Credit, which passports the means test @@ -781,3 +780,470 @@ # 10,400 − 0.65 × (10,140 − 4,968.60) = 10,400 − 3,361.41; 2,627.27 less # than above, which is 0.65 × 4,041.96. housing_benefit_entitlement: 7_038.59 + +# First-year award regressions: GB SI 2026/753 reg 1 and SI 2026/978 reg 2; +# NI SR 2026/157 regs 1/2; HB reg 71 and NI reg 69. Household facts stay +# constant; statutory weekly amounts × 52 apply within each date segment. + +- name: GB and NI working-age awards include the first year while pension-age awards do not + period: 2026 + absolute_error_margin: 0.01 + input: + age: [30,30,30,30,30,30,30,30] + is_claimant_or_partner: [true,true,true,true,true,true,true,true] + is_lone_parent: [false,false,false,false,false,false,false,false] + is_couple: [false,false,false,false,false,false,false,false] + employment_income: [5000,5000,5000,5000,5000,5000,5000,5000] + income_tax: [0,0,0,0,0,0,0,0] + national_insurance: [0,0,0,0,0,0,0,0] + pension_contributions: [0,0,0,0,0,0,0,0] + housing_benefit_net_earnings: [5000,5000,5000,5000,5000,5000,5000,5000] + housing_benefit_special_earnings_disregard: [260,260,260,260,260,260,260,260] + housing_benefit_permitted_work_disregard: [0,0,0,0,0,0,0,0] + housing_benefit_pension_age_regulations_apply: [false,false,false,false,true,true,true,true] + housing_benefit_on_passporting_benefit: [false,false,false,false,false,false,false,false] + housing_benefit_tariff_income: [0,0,0,0,0,0,0,0] + guarantee_credit: [0,0,0,0,0,0,0,0] + income_support: [0,0,0,0,0,0,0,0] + jsa_income: [0,0,0,0,0,0,0,0] + esa_income: [0,0,0,0,0,0,0,0] + tax_credits: [0,0,0,0,0,0,0,0] + housing_benefit_applicable_income_childcare_element: [0,0,0,0,0,0,0,0] + meets_housing_benefit_additional_earnings_disregard_conditions: [false,false,false,false,false,false,false,false] + housing_benefit_applicable_amount: [0,0,0,0,0,0,0,0] + in_specified_or_temporary_accommodation: [true,true,true,true,true,true,true,true] + benunit_rent: [10400,10400,10400,10400,10400,10400,10400,10400] + LHA_eligible: [false,false,false,false,false,false,false,false] + housing_benefit_meals_deduction: [0,0,0,0,0,0,0,0] + housing_benefit_non_dep_deductions: [0,0,0,0,0,0,0,0] + country: ["ENGLAND","WALES","SCOTLAND","NORTHERN_IRELAND","ENGLAND","WALES","SCOTLAND","NORTHERN_IRELAND"] + output: + # Before: 10,400 - 65% × 4,740 = 7,319; after: 10,400 - 65% × 698.04 = 9,946.274. + housing_benefit_entitlement: [8636.236005479452,8636.236005479452,8636.236005479452,8636.236005479452,7319,7319,7319,7319] + +- name: Low earnings cross the income threshold so completed awards must be combined + period: 2026 + absolute_error_margin: 0.01 + input: + age: 30 + is_claimant_or_partner: true + is_lone_parent: false + is_couple: false + employment_income: 1000 + income_tax: 0 + national_insurance: 0 + pension_contributions: 0 + housing_benefit_net_earnings: 1000 + housing_benefit_special_earnings_disregard: 260 + housing_benefit_permitted_work_disregard: 0 + housing_benefit_pension_age_regulations_apply: false + housing_benefit_on_passporting_benefit: false + housing_benefit_tariff_income: 0 + guarantee_credit: 0 + income_support: 0 + jsa_income: 0 + esa_income: 0 + tax_credits: 0 + housing_benefit_applicable_income_childcare_element: 0 + meets_housing_benefit_additional_earnings_disregard_conditions: false + housing_benefit_applicable_amount: 500 + in_specified_or_temporary_accommodation: true + benunit_rent: 10400 + LHA_eligible: false + housing_benefit_meals_deduction: 0 + housing_benefit_non_dep_deductions: 0 + output: + # Before: 10,400 - 65% × (740 - 500) = 10,244; after: full eligible rent. + housing_benefit_applicable_income_disregard: 631.013698630137 + housing_benefit_entitlement: 10322.213698630138 + +- name: The zero-award floor applies before annual integration + period: 2026 + absolute_error_margin: 0.01 + input: + age: 30 + is_claimant_or_partner: true + is_lone_parent: false + is_couple: false + employment_income: 7000 + income_tax: 0 + national_insurance: 0 + pension_contributions: 0 + housing_benefit_net_earnings: 7000 + housing_benefit_special_earnings_disregard: 260 + housing_benefit_permitted_work_disregard: 0 + housing_benefit_pension_age_regulations_apply: false + housing_benefit_on_passporting_benefit: false + housing_benefit_tariff_income: 0 + guarantee_credit: 0 + income_support: 0 + jsa_income: 0 + esa_income: 0 + tax_credits: 0 + housing_benefit_applicable_income_childcare_element: 0 + meets_housing_benefit_additional_earnings_disregard_conditions: false + housing_benefit_applicable_amount: 0 + in_specified_or_temporary_accommodation: true + benunit_rent: 2000 + LHA_eligible: false + housing_benefit_meals_deduction: 0 + housing_benefit_non_dep_deductions: 0 + output: + # Before: max(0, 2,000 - 65% × 6,740) = 0; after: 2,000 - 65% × 2,698.04 = 246.274. + housing_benefit_entitlement: 123.47436164383558 + +- name: Zero earnings cannot create an accommodation deduction + period: 2026 + absolute_error_margin: 0.01 + input: + age: 30 + is_claimant_or_partner: true + is_lone_parent: false + is_couple: false + employment_income: 0 + income_tax: 0 + national_insurance: 0 + pension_contributions: 0 + housing_benefit_net_earnings: 0 + housing_benefit_special_earnings_disregard: 0 + housing_benefit_permitted_work_disregard: 0 + housing_benefit_pension_age_regulations_apply: false + housing_benefit_on_passporting_benefit: false + housing_benefit_tariff_income: 0 + guarantee_credit: 0 + income_support: 0 + jsa_income: 0 + esa_income: 0 + tax_credits: 0 + housing_benefit_applicable_income_childcare_element: 0 + meets_housing_benefit_additional_earnings_disregard_conditions: false + housing_benefit_applicable_amount: 0 + in_specified_or_temporary_accommodation: true + benunit_rent: 10400 + LHA_eligible: false + housing_benefit_meals_deduction: 0 + housing_benefit_non_dep_deductions: 0 + output: + # No earner and no net earnings; income is nil and the award cannot exceed eligible rent. + housing_benefit_specified_or_temporary_accommodation_disregard: 0 + housing_benefit_applicable_income_disregard: 0 + housing_benefit_entitlement: 10400 + +- name: ENGLAND qualifying childcare uses the remaining earnings in each date segment + period: 2026 + absolute_error_margin: 0.01 + input: + people: + parent: + age: 30 + is_claimant_or_partner: true + employment_income: 5000 + income_tax: 0 + national_insurance: 0 + pension_contributions: 0 + child: + age: 5 + is_claimant_or_partner: false + is_child_or_young_person_for_legacy_benefits: true + housing_benefit_qualifying_childcare_charges: 1000 + benunits: + family: + members: [parent, child] + housing_benefit_net_earnings: 5000 + housing_benefit_special_earnings_disregard: 1300 + housing_benefit_permitted_work_disregard: 0 + housing_benefit_pension_age_regulations_apply: false + housing_benefit_on_passporting_benefit: false + housing_benefit_tariff_income: 0 + guarantee_credit: 0 + income_support: 0 + jsa_income: 0 + esa_income: 0 + tax_credits: 0 + working_tax_credit: 0 + child_tax_credit: 0 + meets_housing_benefit_additional_earnings_disregard_conditions: false + housing_benefit_applicable_amount: 0 + in_specified_or_temporary_accommodation: true + benunit_rent: 10400 + LHA_eligible: false + housing_benefit_meals_deduction: 0 + housing_benefit_non_dep_deductions: 0 + households: + household: + members: [parent, child] + country: ENGLAND + output: + # Before: £1,300 standard + £1,000 childcare gives £2,700 income. + # After: accommodation takes all remaining £3,700 earnings, leaving + # no childcare deduction and no assessable income. HB reg 27/71; + # NI regs 25/69: (8,645 × 182 + 10,400 × 183) / 365. + housing_benefit_entitlement: 9524.904109589042 + +- name: NORTHERN_IRELAND qualifying childcare uses the remaining earnings in each date segment + period: 2026 + absolute_error_margin: 0.01 + input: + people: + parent: + age: 30 + is_claimant_or_partner: true + employment_income: 5000 + income_tax: 0 + national_insurance: 0 + pension_contributions: 0 + child: + age: 5 + is_claimant_or_partner: false + is_child_or_young_person_for_legacy_benefits: true + housing_benefit_qualifying_childcare_charges: 1000 + benunits: + family: + members: [parent, child] + housing_benefit_net_earnings: 5000 + housing_benefit_special_earnings_disregard: 1300 + housing_benefit_permitted_work_disregard: 0 + housing_benefit_pension_age_regulations_apply: false + housing_benefit_on_passporting_benefit: false + housing_benefit_tariff_income: 0 + guarantee_credit: 0 + income_support: 0 + jsa_income: 0 + esa_income: 0 + tax_credits: 0 + working_tax_credit: 0 + child_tax_credit: 0 + meets_housing_benefit_additional_earnings_disregard_conditions: false + housing_benefit_applicable_amount: 0 + in_specified_or_temporary_accommodation: true + benunit_rent: 10400 + LHA_eligible: false + housing_benefit_meals_deduction: 0 + housing_benefit_non_dep_deductions: 0 + households: + household: + members: [parent, child] + country: NORTHERN_IRELAND + output: + # Before: £1,300 standard + £1,000 childcare gives £2,700 income. + # After: accommodation takes all remaining £3,700 earnings, leaving + # no childcare deduction and no assessable income. HB reg 27/71; + # NI regs 25/69: (8,645 × 182 + 10,400 × 183) / 365. + housing_benefit_entitlement: 9524.904109589042 + + +- name: An explicit annual applicable-income input is not recalculated + period: 2026 + absolute_error_margin: 0.01 + input: + age: 30 + is_claimant_or_partner: true + is_lone_parent: false + is_couple: false + employment_income: 5000 + income_tax: 0 + national_insurance: 0 + pension_contributions: 0 + housing_benefit_net_earnings: 5000 + housing_benefit_special_earnings_disregard: 260 + housing_benefit_permitted_work_disregard: 0 + housing_benefit_pension_age_regulations_apply: false + housing_benefit_on_passporting_benefit: false + housing_benefit_tariff_income: 0 + guarantee_credit: 0 + income_support: 0 + jsa_income: 0 + esa_income: 0 + tax_credits: 0 + housing_benefit_applicable_income_childcare_element: 0 + meets_housing_benefit_additional_earnings_disregard_conditions: false + housing_benefit_applicable_amount: 0 + in_specified_or_temporary_accommodation: true + benunit_rent: 10400 + LHA_eligible: false + housing_benefit_meals_deduction: 0 + housing_benefit_non_dep_deductions: 0 + housing_benefit_applicable_income: 1000 + output: + # Keep supplied income: 10,400 - 65% × 1,000 = 9,750 in both segments. + housing_benefit_entitlement: 9750 + +- name: An explicit annual zero disregard is not overwritten by segmentation + period: 2026 + absolute_error_margin: 0.01 + input: + age: 30 + is_claimant_or_partner: true + is_lone_parent: false + is_couple: false + employment_income: 5000 + income_tax: 0 + national_insurance: 0 + pension_contributions: 0 + housing_benefit_net_earnings: 5000 + housing_benefit_special_earnings_disregard: 260 + housing_benefit_permitted_work_disregard: 0 + housing_benefit_pension_age_regulations_apply: false + housing_benefit_on_passporting_benefit: false + housing_benefit_tariff_income: 0 + guarantee_credit: 0 + income_support: 0 + jsa_income: 0 + esa_income: 0 + tax_credits: 0 + housing_benefit_applicable_income_childcare_element: 0 + meets_housing_benefit_additional_earnings_disregard_conditions: false + housing_benefit_applicable_amount: 0 + in_specified_or_temporary_accommodation: true + benunit_rent: 10400 + LHA_eligible: false + housing_benefit_meals_deduction: 0 + housing_benefit_non_dep_deductions: 0 + housing_benefit_applicable_income_disregard: 0 + output: + # Keep supplied disregard: 10,400 - 65% × 5,000 = 7,150 in both segments. + housing_benefit_entitlement: 7150 + +- name: An explicit annual childcare deduction is retained inside each award calculation + period: 2026 + absolute_error_margin: 0.01 + input: + age: 30 + is_claimant_or_partner: true + is_lone_parent: false + is_couple: false + employment_income: 5000 + income_tax: 0 + national_insurance: 0 + pension_contributions: 0 + housing_benefit_net_earnings: 5000 + housing_benefit_special_earnings_disregard: 260 + housing_benefit_permitted_work_disregard: 0 + housing_benefit_pension_age_regulations_apply: false + housing_benefit_on_passporting_benefit: false + housing_benefit_tariff_income: 0 + guarantee_credit: 0 + income_support: 0 + jsa_income: 0 + esa_income: 0 + tax_credits: 0 + housing_benefit_applicable_income_childcare_element: 1000 + meets_housing_benefit_additional_earnings_disregard_conditions: false + housing_benefit_applicable_amount: 0 + in_specified_or_temporary_accommodation: true + benunit_rent: 10400 + LHA_eligible: false + housing_benefit_meals_deduction: 0 + housing_benefit_non_dep_deductions: 0 + output: + # Before income: 3,740; after income is floored at zero after the supplied £1,000 childcare deduction. + housing_benefit_entitlement: 9187.830136986302 + +- name: The additional disregard earnings test uses the full accommodation amount + period: 2026 + absolute_error_margin: 0.01 + input: + age: 30 + is_claimant_or_partner: true + is_lone_parent: false + is_couple: false + employment_income: 5000 + income_tax: 0 + national_insurance: 0 + pension_contributions: 0 + housing_benefit_net_earnings: 5000 + housing_benefit_special_earnings_disregard: 260 + housing_benefit_permitted_work_disregard: 0 + housing_benefit_pension_age_regulations_apply: false + housing_benefit_on_passporting_benefit: false + housing_benefit_tariff_income: 0 + guarantee_credit: 0 + income_support: 0 + jsa_income: 0 + esa_income: 0 + tax_credits: 0 + housing_benefit_applicable_income_childcare_element: 0 + meets_housing_benefit_additional_earnings_disregard_conditions: true + housing_benefit_applicable_amount: 0 + in_specified_or_temporary_accommodation: true + benunit_rent: 10400 + LHA_eligible: false + housing_benefit_meals_deduction: 0 + housing_benefit_non_dep_deductions: 0 + output: + # Before: £260 + £889.20 qualifies. After: £5,000 is below £260 + £4,041.96 + £889.20. + housing_benefit_applicable_income_disregard: 2729.8988493150687 + housing_benefit_entitlement: 8924.434252054794 + +- name: A supplied zero accommodation amount preserves its override + period: 2026 + absolute_error_margin: 0.01 + input: + age: 30 + is_claimant_or_partner: true + is_lone_parent: false + is_couple: false + employment_income: 5000 + income_tax: 0 + national_insurance: 0 + pension_contributions: 0 + housing_benefit_net_earnings: 5000 + housing_benefit_special_earnings_disregard: 260 + housing_benefit_permitted_work_disregard: 0 + housing_benefit_pension_age_regulations_apply: false + housing_benefit_on_passporting_benefit: false + housing_benefit_tariff_income: 0 + guarantee_credit: 0 + income_support: 0 + jsa_income: 0 + esa_income: 0 + tax_credits: 0 + housing_benefit_applicable_income_childcare_element: 0 + meets_housing_benefit_additional_earnings_disregard_conditions: false + housing_benefit_applicable_amount: 0 + in_specified_or_temporary_accommodation: true + benunit_rent: 10400 + LHA_eligible: false + housing_benefit_meals_deduction: 0 + housing_benefit_non_dep_deductions: 0 + housing_benefit_specified_or_temporary_accommodation_disregard: 0 + output: + # The supplied full-year zero leaves only the standard £260 disregard. + housing_benefit_applicable_income_disregard: 260 + housing_benefit_entitlement: 7319 + +- name: Working-age passporting remains effective across the date boundary + period: 2026 + absolute_error_margin: 0.01 + input: + age: 30 + is_claimant_or_partner: true + is_lone_parent: false + is_couple: false + employment_income: 5000 + income_tax: 0 + national_insurance: 0 + pension_contributions: 0 + housing_benefit_net_earnings: 5000 + housing_benefit_special_earnings_disregard: 260 + housing_benefit_permitted_work_disregard: 0 + housing_benefit_pension_age_regulations_apply: false + housing_benefit_on_passporting_benefit: true + housing_benefit_tariff_income: 0 + guarantee_credit: 0 + income_support: 0 + jsa_income: 0 + esa_income: 0 + tax_credits: 0 + housing_benefit_applicable_income_childcare_element: 0 + meets_housing_benefit_additional_earnings_disregard_conditions: false + housing_benefit_applicable_amount: 0 + in_specified_or_temporary_accommodation: true + benunit_rent: 10400 + LHA_eligible: false + housing_benefit_meals_deduction: 0 + housing_benefit_non_dep_deductions: 0 + output: + # The whole income remains disregarded; no tapered assessment is introduced. + housing_benefit_applicable_income_disregard: 5000 + housing_benefit_entitlement: 10400 diff --git a/policyengine_uk/tests/test_housing_benefit_accommodation_dates.py b/policyengine_uk/tests/test_housing_benefit_accommodation_dates.py new file mode 100644 index 0000000000..1c941b01a7 --- /dev/null +++ b/policyengine_uk/tests/test_housing_benefit_accommodation_dates.py @@ -0,0 +1,222 @@ +"""Statutory dates and reform compatibility for HB accommodation disregards.""" + +from datetime import date + +import pytest + +from policyengine_uk import CountryTaxBenefitSystem, Simulation +from policyengine_uk.model_api import Variable, BenUnit, YEAR + + +PREFIX = ( + "gov.dwp.housing_benefit.means_test.income_disregard." + "specified_or_temporary_accommodation" +) + + +@pytest.fixture(scope="module") +def system(): + return CountryTaxBenefitSystem() + + +def node(parameters): + return parameters.gov.dwp.housing_benefit.means_test.income_disregard.specified_or_temporary_accommodation + + +@pytest.mark.parametrize( + "family,age,amount", + [ + ("single", "younger", 61.41), + ("single", "older", 77.73), + ("lone_parent", "younger", 61.41), + ("lone_parent", "older", 77.73), + ("couple", "minors", 97.33), + ("couple", "younger", 61.53), + ("couple", "older", 119.70), + ], +) +def test_processed_amounts_keep_5_october_commencement(system, family, age, amount): + """GB SI 2026/753 and NI SR 2026/157 commence on 5 October, not January.""" + leaf = node(system.parameters).children[family].children[age] + assert leaf("2026-10-04") == 0 + assert leaf("2026-10-05") == amount + + +def situation(years=(2026,)): + return { + "people": { + "person1": { + "age": {year: 30 for year in years}, + "employment_income": {year: 1_000 for year in years}, + "income_tax": {year: 0 for year in years}, + "national_insurance": {year: 0 for year in years}, + "pension_contributions": {year: 0 for year in years}, + } + }, + "benunits": { + "family": { + "members": ["person1"], + "in_specified_or_temporary_accommodation": { + year: True for year in years + }, + "housing_benefit_net_earnings": {year: 1_000 for year in years}, + "housing_benefit_special_earnings_disregard": { + year: 260 for year in years + }, + "housing_benefit_applicable_amount": {year: 500 for year in years}, + "housing_benefit_on_passporting_benefit": { + year: False for year in years + }, + "housing_benefit_pension_age_regulations_apply": { + year: False for year in years + }, + "housing_benefit_tariff_income": {year: 0 for year in years}, + "housing_benefit_applicable_income_childcare_element": { + year: 0 for year in years + }, + "meets_housing_benefit_additional_earnings_disregard_conditions": { + year: False for year in years + }, + "benunit_rent": {year: 10_400 for year in years}, + "housing_benefit_meals_deduction": {year: 0 for year in years}, + "housing_benefit_non_dep_deductions": {year: 0 for year in years}, + "LHA_eligible": {year: False for year in years}, + "income_support": {year: 0 for year in years}, + "jsa_income": {year: 0 for year in years}, + "esa_income": {year: 0 for year in years}, + "tax_credits": {year: 0 for year in years}, + } + }, + "households": {"household": {"members": ["person1"]}}, + } + + +def test_first_year_combines_completed_awards(): + """Reg 71: average the awards, not the income before its £500 threshold.""" + simulation = Simulation(situation=situation()) + before = 10_400 - 0.65 * (1_000 - 260 - 500) + after = 10_400 + days_before = (date(2026, 10, 5) - date(2026, 4, 6)).days + days_after = (date(2027, 4, 6) - date(2026, 10, 5)).days + expected = (before * days_before + after * days_after) / 365 + assert simulation.calculate("housing_benefit_entitlement", 2026)[ + 0 + ] == pytest.approx(expected, abs=0.01) + + +def test_bare_year_zero_reform_has_fiscal_year_boundaries(): + """A 2026 reform must not leak into the following 2027-28 fiscal year.""" + simulation = Simulation( + situation=situation((2025, 2026, 2027)), + reform={f"{PREFIX}.single.older": {"2026": 0}}, + ) + amounts = [ + simulation.calculate( + "housing_benefit_specified_or_temporary_accommodation_disregard", year + )[0] + for year in (2025, 2026, 2027) + ] + assert amounts == pytest.approx([0, 0, 77.73 * 52], abs=0.01) + + +def test_dated_zero_reform_is_not_overwritten(): + simulation = Simulation( + situation=situation(), + reform={f"{PREFIX}.single.older": {"2026-10-05.2027-04-05": 0}}, + ) + assert simulation.calculate("housing_benefit_entitlement", 2026)[ + 0 + ] == pytest.approx(10_400 - 0.65 * (1_000 - 260 - 500), abs=0.01) + + +def test_later_supplied_income_keeps_its_value(): + simulation = Simulation(situation=situation()) + simulation.set_input("housing_benefit_applicable_income", 2026, [2_000]) + assert simulation.calculate("housing_benefit_entitlement", 2026)[ + 0 + ] == pytest.approx(10_400 - 0.65 * (2_000 - 500), abs=0.01) + + +def test_single_day_zero_reform_changes_only_that_day(): + simulation = Simulation( + situation=situation(), + reform={f"{PREFIX}.single.older": {"2026-10-05": 0}}, + ) + before = 10_400 - 0.65 * (1_000 - 260 - 500) + assert simulation.calculate("housing_benefit_entitlement", 2026)[ + 0 + ] == pytest.approx((before * 183 + 10_400 * 182) / 365, abs=0.01) + + +def test_only_supplied_period_overrides_recalculation(): + data = situation((2025, 2026)) + data["benunits"]["family"]["housing_benefit_applicable_income"] = {2025: 2_000} + simulation = Simulation(situation=data) + before = 10_400 - 0.65 * (1_000 - 260 - 500) + assert simulation.calculate("housing_benefit_entitlement", 2026)[ + 0 + ] == pytest.approx((before * 182 + 10_400 * 183) / 365, abs=0.01) + + +def test_replaced_income_formula_is_not_bypassed(): + class housing_benefit_applicable_income(Variable): + entity = BenUnit + definition_period = YEAR + value_type = float + + def formula(benunit, period, parameters): + return benunit.filled_array(2_000) + + simulation = Simulation(situation=situation()) + simulation.tax_benefit_system.update_variable(housing_benefit_applicable_income) + assert simulation.calculate("housing_benefit_entitlement", 2026)[ + 0 + ] == pytest.approx(10_400 - 0.65 * (2_000 - 500), abs=0.01) + + +def test_neutralized_income_is_not_recalculated(): + simulation = Simulation(situation=situation()) + simulation.tax_benefit_system.neutralize_variable( + "housing_benefit_applicable_income" + ) + assert simulation.calculate("housing_benefit_entitlement", 2026)[0] == 10_400 + + +def test_replaced_disregard_formula_is_not_bypassed(): + class housing_benefit_applicable_income_disregard(Variable): + entity = BenUnit + definition_period = YEAR + value_type = float + + def formula(benunit, period, parameters): + return benunit.filled_array(100) + + simulation = Simulation(situation=situation()) + simulation.tax_benefit_system.update_variable( + housing_benefit_applicable_income_disregard + ) + assert simulation.calculate("housing_benefit_entitlement", 2026)[ + 0 + ] == pytest.approx(10_400 - 0.65 * (1_000 - 100 - 500), abs=0.01) + + +def test_calculation_does_not_reparent_or_edit_parameter_leaves(): + simulation = Simulation(situation=situation()) + schedule = node(simulation.tax_benefit_system.parameters) + parents = { + leaf.name: leaf.parent + for family in ("single", "lone_parent", "couple") + for leaf in schedule.children[family].children.values() + } + values = { + leaf.name: [(value.instant_str, value.value) for value in leaf.values_list] + for family in ("single", "lone_parent", "couple") + for leaf in schedule.children[family].children.values() + } + simulation.calculate("housing_benefit_entitlement", 2026) + for family in ("single", "lone_parent", "couple"): + for leaf in schedule.children[family].children.values(): + assert leaf.parent is parents[leaf.name] + assert [ + (value.instant_str, value.value) for value in leaf.values_list + ] == values[leaf.name] diff --git a/policyengine_uk/variables/gov/dwp/housing_benefit/applicable_income/housing_benefit_applicable_income_disregard.py b/policyengine_uk/variables/gov/dwp/housing_benefit/applicable_income/housing_benefit_applicable_income_disregard.py index 8b12285381..b5d5ad0c89 100644 --- a/policyengine_uk/variables/gov/dwp/housing_benefit/applicable_income/housing_benefit_applicable_income_disregard.py +++ b/policyengine_uk/variables/gov/dwp/housing_benefit/applicable_income/housing_benefit_applicable_income_disregard.py @@ -1,4 +1,73 @@ from policyengine_uk.model_api import * +from policyengine_uk.variables.gov.dwp.housing_benefit.applicable_income.housing_benefit_applicable_income_childcare_element import ( + hb_capped_childcare_charges, + housing_benefit_applicable_income_childcare_element, +) +from policyengine_uk.variables.gov.dwp.housing_benefit.applicable_income.housing_benefit_childcare_earnings_limit import ( + hb_childcare_earnings_limit, + hb_standard_earnings_disregard, + housing_benefit_childcare_earnings_limit, +) +from policyengine_uk.variables.gov.dwp.housing_benefit.applicable_income.housing_benefit_specified_or_temporary_accommodation_disregard import ( + hb_accommodation_segments, + hb_uses_annual_override, +) + + +def hb_segment_deductions(benunit, period, parameters, accommodation): + """Apply childcare/earnings limits to one full accommodation amount.""" + net = max_(benunit("housing_benefit_net_earnings", period), 0) + standard = hb_standard_earnings_disregard(benunit, period) + accommodation = min_(max_(accommodation, 0), max_(net - standard, 0)) + childcare_name = "housing_benefit_applicable_income_childcare_element" + if hb_uses_annual_override( + benunit, + period, + childcare_name, + housing_benefit_applicable_income_childcare_element.formula, + ): + childcare = benunit(childcare_name, period) + else: + limit_name = "housing_benefit_childcare_earnings_limit" + if hb_uses_annual_override( + benunit, + period, + limit_name, + housing_benefit_childcare_earnings_limit.formula, + ): + childcare_limit = benunit(limit_name, period) + else: + childcare_limit = hb_childcare_earnings_limit( + benunit, period, accommodation + ) + childcare = min_( + hb_capped_childcare_charges(benunit, period, parameters), childcare_limit + ) + name = "housing_benefit_applicable_income_disregard" + if hb_uses_annual_override( + benunit, period, name, housing_benefit_applicable_income_disregard.formula + ): + return benunit(name, period), childcare + p = parameters(period).gov.dwp.housing_benefit.means_test.income_disregard + additional_amount = p.worker * WEEKS_IN_YEAR + # Compare in pence, including equality at a float32 earnings boundary. + covers_additional = np.round(net.astype(float), 2) >= np.round( + (standard + accommodation + childcare + additional_amount).astype(float), 2 + ) + additional = where( + benunit( + "meets_housing_benefit_additional_earnings_disregard_conditions", period + ) + & covers_additional, + additional_amount, + 0, + ) + disregard = where( + benunit("housing_benefit_on_passporting_benefit", period), + net, + standard + accommodation + additional, + ) + return disregard, childcare class housing_benefit_applicable_income_disregard(Variable): @@ -26,6 +95,10 @@ class housing_benefit_applicable_income_disregard(Variable): "(housing_benefit_permitted_work_disregard) replaces the ordinary " "standard amount, retaining a higher lone-parent disregard. It does " "not replace the separate accommodation or additional disregard." + " The annual result combines the capped deductions calculated with " + "the full accommodation amount in each effective-date segment. " + "Entitlement applies its income threshold and taper within those " + "segments rather than using an averaged disregard as a weekly rule." ) definition_period = YEAR unit = GBP @@ -45,67 +118,9 @@ class housing_benefit_applicable_income_disregard(Variable): ) def formula(benunit, period, parameters): - p = parameters(period).gov.dwp.housing_benefit.means_test.income_disregard - net_earnings = benunit("housing_benefit_net_earnings", period) - # Working age Sch 4 paras 4, 7 and 10; pension age Sch 4 paras 2 and 7. - standard = benunit("housing_benefit_special_earnings_disregard", period) - # Working-age para 10A(2) / pension-age para 5A(2): replace the - # ordinary amount, except for a higher lone-parent amount. The input - # already applies the statutory work and partner-allocation limits. - permitted = max_(benunit("housing_benefit_permitted_work_disregard", period), 0) - standard = where( - permitted > 0, - min_( - net_earnings, - max_(permitted, benunit("is_lone_parent", period) * standard), - ), - standard, - ) - # Working age Sch 4 para 18 (NI Sch 5 para 18), from 5 October 2026: - # specified or temporary accommodation. It applies alongside paras 3 - # to 10A, none of which excludes it, and takes the earnings they - # leave. - accommodation = min_( - benunit( - "housing_benefit_specified_or_temporary_accommodation_disregard", - period, - ), - net_earnings - standard, - ) - # Working age Sch 4 para 17; pension age Sch 4 para 9. - additional_amount = p.worker * WEEKS_IN_YEAR - childcare = benunit( - "housing_benefit_applicable_income_childcare_element", period - ) - # "Equal or exceed", compared in pence: net earnings are float32, so - # exactly £1,149.20 is held as £1,149.19995, and an exact comparison - # with the total fails at equality. From 5 October 2026 the total - # includes the para 18 amount (para 17(3)(a), "paragraphs 3 to 10A - # and 18"); it is zero under the pension-age para 9(3)(a). - covers_additional = np.round(net_earnings.astype(float), 2) >= np.round( - (standard + accommodation + childcare + additional_amount).astype(float), - 2, - ) - additional = where( - benunit( - "meets_housing_benefit_additional_earnings_disregard_conditions", period + return sum( + hb_segment_deductions(benunit, period, parameters, accommodation)[0] * share + for accommodation, share in hb_accommodation_segments( + benunit, period, parameters ) - & covers_additional, - additional_amount, - 0, - ) - # Working age Sch 4 para 12 (NI: SR 2006/405 Sch 5 para 12): "Where a - # claimant is on universal credit, income support, an income-based - # jobseeker's allowance or an income-related employment and support - # allowance, his earnings." The working-age Regulations apply at any - # age where the claimant or partner is on one of these benefits (SI - # 2006/213 reg 5(1)(b); SI 2006/214 reg 5(2)), so there is no age - # condition. - on_passporting_benefit = benunit( - "housing_benefit_on_passporting_benefit", period - ) - return where( - on_passporting_benefit, - net_earnings, - standard + accommodation + additional, ) diff --git a/policyengine_uk/variables/gov/dwp/housing_benefit/applicable_income/housing_benefit_specified_or_temporary_accommodation_disregard.py b/policyengine_uk/variables/gov/dwp/housing_benefit/applicable_income/housing_benefit_specified_or_temporary_accommodation_disregard.py index 401d1c6f55..1218ad5f38 100644 --- a/policyengine_uk/variables/gov/dwp/housing_benefit/applicable_income/housing_benefit_specified_or_temporary_accommodation_disregard.py +++ b/policyengine_uk/variables/gov/dwp/housing_benefit/applicable_income/housing_benefit_specified_or_temporary_accommodation_disregard.py @@ -1,4 +1,89 @@ from policyengine_uk.model_api import * +from policyengine_core.parameters import ParameterNode +from policyengine_uk.utils.excise import fiscal_year_segments + + +def hb_uses_annual_override(benunit, period, variable_name, default_formula): + """Keep supplied annual values and replacement formulas out of recalculation.""" + simulation = benunit.simulation + variable = simulation.tax_benefit_system.get_variable(variable_name) + formula = variable.get_formula(period) + if ( + variable.is_neutralized + or formula is None + or formula.__code__ != default_formula.__code__ + ): + return True + # Core records set_input separately from computed caches. Reading those + # records avoids treating a cached formula result as an explicit override. + branch = simulation + visible = {"default"} + while branch is not None: + visible.add(branch.branch_name) + branch = getattr(branch, "parent_branch", None) + holder = benunit.get_holder(variable_name) + return any( + name == variable_name + and input_period == period + and branch_name in visible + and holder._get_array_from_storage(input_period, branch_name) is not None + for name, branch_name, input_period in getattr( + simulation, "_user_input_keys", () + ) + ) + + +def hb_accommodation_segments(benunit, period, parameters): + """Full annual-equivalent accommodation amounts and fiscal-year weights.""" + name = "housing_benefit_specified_or_temporary_accommodation_disregard" + if hb_uses_annual_override( + benunit, + period, + name, + housing_benefit_specified_or_temporary_accommodation_disregard.formula, + ): + yield benunit(name, period), 1 + return + raw = benunit.simulation.tax_benefit_system.parameters + accommodation = raw.gov.dwp.housing_benefit.means_test.income_disregard.specified_or_temporary_accommodation + schedule = ParameterNode("hb_accommodation_schedule", data={}) + for family in ("single", "lone_parent", "couple"): + for age, leaf in accommodation.children[family].children.items(): + # add_child sets the parent: clone leaves, never modify the model + # tree or replace values supplied by a parameter reform. + schedule.add_child(f"{family}_{age}", leaf.clone()) + p = parameters( + period + ).gov.dwp.housing_benefit.means_test.income_disregard.specified_or_temporary_accommodation + person = benunit.members + earner = ( + (person("employment_income", period) > 0) + | (person("self_employment_income", period) != 0) + | (person("employment_benefits", period) > 0) + ) + has_earner = benunit.any(person("is_claimant_or_partner", period) & earner) + eldest = benunit("eldest_claimant_or_partner_age", period) + older = eldest >= p.age_threshold.older + applies = ( + benunit("in_specified_or_temporary_accommodation", period) + & has_earner + & ~benunit("housing_benefit_pension_age_regulations_apply", period) + ) + for rates, share in fiscal_year_segments(schedule, period.start.year): + couple_amount = select( + [older, eldest >= p.age_threshold.younger], + [rates.couple_older, rates.couple_younger], + default=rates.couple_minors, + ) + weekly = select( + [benunit("is_lone_parent", period), benunit("is_couple", period)], + [ + where(older, rates.lone_parent_older, rates.lone_parent_younger), + couple_amount, + ], + default=where(older, rates.single_older, rates.single_younger), + ) + yield where(applies, weekly * WEEKS_IN_YEAR, 0), share class housing_benefit_specified_or_temporary_accommodation_disregard(Variable): @@ -17,8 +102,10 @@ class housing_benefit_specified_or_temporary_accommodation_disregard(Variable): "from the partner's. This is the full amount; " "housing_benefit_applicable_income_disregard adds it to the other " "disregards and caps the total at net earnings. The pension-age " - "Regulations have no such disregard. The model reads each fiscal year " - "at 30 April, so it is zero for 2026-27 and in full from 2027-28." + "Regulations have no such disregard. This annual output combines " + "the full amounts over their effective-date segments. The earnings " + "limits and award are calculated separately within each segment, " + "with annual earnings and household circumstances held constant." ) definition_period = YEAR unit = GBP @@ -31,43 +118,7 @@ class housing_benefit_specified_or_temporary_accommodation_disregard(Variable): ) def formula(benunit, period, parameters): - p = parameters( - period - ).gov.dwp.housing_benefit.means_test.income_disregard.specified_or_temporary_accommodation - person = benunit.members - # Para 18(1)(b): the claimant or partner is an employed earner or a - # self-employed earner. Anyone with earnings is, including statutory - # sick or maternity pay from a continuing employment and a - # self-employment loss. The amount is capped at net earnings in - # housing_benefit_applicable_income_disregard, so a family without - # net earnings gets nothing either way. - earner = ( - (person("employment_income", period) > 0) - | (person("self_employment_income", period) != 0) - | (person("employment_benefits", period) > 0) - ) - has_earner = benunit.any(person("is_claimant_or_partner", period) & earner) - # Para 18(2): the claimant's age, or the elder member's for a couple. - eldest = benunit("eldest_claimant_or_partner_age", period) - older = eldest >= p.age_threshold.older - couple_amount = select( - [older, eldest >= p.age_threshold.younger], - [p.couple.older, p.couple.younger], - default=p.couple.minors, - ) - weekly_amount = select( - [benunit("is_lone_parent", period), benunit("is_couple", period)], - [ - where(older, p.lone_parent.older, p.lone_parent.younger), - couple_amount, - ], - default=where(older, p.single.older, p.single.younger), - ) - # Para 18(1)(a), and only in the working-age Regulations (SI 2006/213; - # NI SR 2006/405): SI 2006/214 and SR 2006/406 are not amended. - applies = ( - benunit("in_specified_or_temporary_accommodation", period) - & has_earner - & ~benunit("housing_benefit_pension_age_regulations_apply", period) + return sum( + amount * share + for amount, share in hb_accommodation_segments(benunit, period, parameters) ) - return where(applies, weekly_amount * WEEKS_IN_YEAR, 0) diff --git a/policyengine_uk/variables/gov/dwp/housing_benefit/entitlement/housing_benefit_entitlement.py b/policyengine_uk/variables/gov/dwp/housing_benefit/entitlement/housing_benefit_entitlement.py index 553deed221..84690e73aa 100644 --- a/policyengine_uk/variables/gov/dwp/housing_benefit/entitlement/housing_benefit_entitlement.py +++ b/policyengine_uk/variables/gov/dwp/housing_benefit/entitlement/housing_benefit_entitlement.py @@ -1,4 +1,15 @@ from policyengine_uk.model_api import * +from policyengine_uk.variables.gov.dwp.housing_benefit.applicable_income.housing_benefit_applicable_income import ( + hb_income_before_deductions, + housing_benefit_applicable_income, +) +from policyengine_uk.variables.gov.dwp.housing_benefit.applicable_income.housing_benefit_applicable_income_disregard import ( + hb_segment_deductions, +) +from policyengine_uk.variables.gov.dwp.housing_benefit.applicable_income.housing_benefit_specified_or_temporary_accommodation_disregard import ( + hb_accommodation_segments, + hb_uses_annual_override, +) class housing_benefit_entitlement(Variable): @@ -9,7 +20,14 @@ class housing_benefit_entitlement(Variable): "income over the applicable amount. Where the Local Housing Allowance " "applies, the eligible rent is the maximum rent (LHA), which is the " "lower of the LHA rate and the rent, so the cap applies before the " - "taper." + "taper. When accommodation earnings disregards change within the " + "fiscal year, calculate income floors, childcare/earnings limits, " + "the income threshold, taper and zero-award floor separately in " + "each segment, then combine completed awards. Annual earnings, rent, " + "ages and household status remain constant. Other parameters keep " + "the model's annual convention. Supplied annual income/deductions " + "and replacement formulas retain their normal override semantics. " + "Annual income intermediates are not period-specific assessments." ) entity = BenUnit definition_period = YEAR @@ -34,6 +52,9 @@ class housing_benefit_entitlement(Variable): "https://www.legislation.gov.uk/nisr/2006/406/regulation/14D", "https://www.legislation.gov.uk/nisr/2006/406/regulation/48", "https://www.legislation.gov.uk/nisr/2006/406/regulation/49", + "https://www.legislation.gov.uk/uksi/2026/753/regulation/1", + "https://www.legislation.gov.uk/uksi/2026/978/regulation/2", + "https://www.legislation.gov.uk/nisr/2026/157/regulation/2", ) def formula(benunit, period, parameters): @@ -65,5 +86,33 @@ def formula(benunit, period, parameters): withdrawal_rate = parameters( period ).gov.dwp.housing_benefit.means_test.withdrawal_rate - taper = max_(0, income - applicable_amount) * withdrawal_rate - return max_(0, maximum_housing_benefit - taper) + segments = list(hb_accommodation_segments(benunit, period, parameters)) + if len(segments) == 1 or hb_uses_annual_override( + benunit, + period, + "housing_benefit_applicable_income", + housing_benefit_applicable_income.formula, + ): + taper = max_(0, income - applicable_amount) * withdrawal_rate + return max_(0, maximum_housing_benefit - taper) + income_before_deductions = hb_income_before_deductions( + benunit, period, parameters + ) + retain_annual_income = benunit( + "housing_benefit_pension_age_regulations_apply", period + ) | benunit("housing_benefit_on_passporting_benefit", period) + annual_award = benunit.empty_array() + for accommodation, share in segments: + disregard, childcare = hb_segment_deductions( + benunit, period, parameters, accommodation + ) + segment_income = where( + retain_annual_income, + income, + max_(0, income_before_deductions - disregard - childcare), + ) + taper = max_(0, segment_income - applicable_amount) * withdrawal_rate + annual_award = annual_award + share * max_( + 0, maximum_housing_benefit - taper + ) + return annual_award From de48e391989960e56c8454ec5aea9693e299dec5 Mon Sep 17 00:00:00 2001 From: Anthony Volk <14987227+anth-volk@users.noreply.github.com> Date: Fri, 9 Oct 2026 21:38:47 +0400 Subject: [PATCH 02/12] Use childcare_expenses in accommodation date regressions --- ...ecified_or_temporary_accommodation_disregard.yaml | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/policyengine_uk/tests/policy/baseline/finance/benefit/family/housing_benefit/applicable_income/housing_benefit_specified_or_temporary_accommodation_disregard.yaml b/policyengine_uk/tests/policy/baseline/finance/benefit/family/housing_benefit/applicable_income/housing_benefit_specified_or_temporary_accommodation_disregard.yaml index 31287d5870..21aad0ceb0 100644 --- a/policyengine_uk/tests/policy/baseline/finance/benefit/family/housing_benefit/applicable_income/housing_benefit_specified_or_temporary_accommodation_disregard.yaml +++ b/policyengine_uk/tests/policy/baseline/finance/benefit/family/housing_benefit/applicable_income/housing_benefit_specified_or_temporary_accommodation_disregard.yaml @@ -681,7 +681,7 @@ weekly_hours: 16 child: age: 5 - housing_benefit_qualifying_childcare_charges: 2_080 # pre-assessed £40/week + childcare_expenses: 2_080 # £40/week; detailed qualifying-care rules deferred benunits: benunit: would_claim_uc: false # not on Universal Credit, which passports the means test @@ -929,7 +929,7 @@ housing_benefit_applicable_income_disregard: 0 housing_benefit_entitlement: 10400 -- name: ENGLAND qualifying childcare uses the remaining earnings in each date segment +- name: ENGLAND childcare_expenses uses the remaining earnings in each date segment period: 2026 absolute_error_margin: 0.01 input: @@ -945,7 +945,7 @@ age: 5 is_claimant_or_partner: false is_child_or_young_person_for_legacy_benefits: true - housing_benefit_qualifying_childcare_charges: 1000 + childcare_expenses: 1000 benunits: family: members: [parent, child] @@ -977,10 +977,10 @@ # Before: £1,300 standard + £1,000 childcare gives £2,700 income. # After: accommodation takes all remaining £3,700 earnings, leaving # no childcare deduction and no assessable income. HB reg 27/71; - # NI regs 25/69: (8,645 × 182 + 10,400 × 183) / 365. + # NI regs 24/69: (8,645 × 182 + 10,400 × 183) / 365. housing_benefit_entitlement: 9524.904109589042 -- name: NORTHERN_IRELAND qualifying childcare uses the remaining earnings in each date segment +- name: NORTHERN_IRELAND childcare_expenses uses the remaining earnings in each date segment period: 2026 absolute_error_margin: 0.01 input: @@ -996,7 +996,7 @@ age: 5 is_claimant_or_partner: false is_child_or_young_person_for_legacy_benefits: true - housing_benefit_qualifying_childcare_charges: 1000 + childcare_expenses: 1000 benunits: family: members: [parent, child] From 218c03bb74677ba661ff97f9dea2de1790ffe3cc Mon Sep 17 00:00:00 2001 From: Max Ghenis Date: Thu, 29 Oct 2026 01:14:32 -0400 Subject: [PATCH 03/12] Test HB accommodation date invariants and document the additional-disregard cliff --- ..._benefit_accommodation_dates_properties.py | 331 ++++++++++++++++++ 1 file changed, 331 insertions(+) create mode 100644 policyengine_uk/tests/test_housing_benefit_accommodation_dates_properties.py diff --git a/policyengine_uk/tests/test_housing_benefit_accommodation_dates_properties.py b/policyengine_uk/tests/test_housing_benefit_accommodation_dates_properties.py new file mode 100644 index 0000000000..508a2fbefd --- /dev/null +++ b/policyengine_uk/tests/test_housing_benefit_accommodation_dates_properties.py @@ -0,0 +1,331 @@ +"""Properties of fiscal-year Housing Benefit accommodation date integration. + +D1. No change, no segments: when none of the seven monetary parameters + changes, one full-year segment gives the direct unsegmented award. +D2. Convex combination: the annual award is between the segment awards and + equals their independently calculated elapsed-day-weighted mean. +D3. Bounds: with nonnegative eligible rent and non-dependant deductions, + entitlement is between zero and eligible rent. +D4. Monotonicity: raising accommodation never lowers the annual award. + Intended violation: the additional-disregard earnings test can switch + off. At net earnings of 1,149.20, standard 260 and no childcare, raising + later accommodation from zero to one penny removes the 889.20 additional + disregard. The annual award falls from 10,400 to 10,110.22. The strict + xfail retains this smallest positive penny increment; a separate property + verifies monotonicity with the additional disregard disabled. +D5. Day count: 2026-27 has 182 days before 5 October and 183 from that + date, totalling 365; inspect production segments, not just a calendar. + +Fixture-scale populations hold annual incomes, rent and circumstances +constant. The direct oracle implements the earnings cap, residual childcare +limit, additional earnings test, income floor, threshold and award floor +independently of the production helpers. It uses calendar dates independently +of the production weights. All seven rates, family shapes, negative net +earnings, permitted work, childcare and tax credits are exercised. +""" + +from datetime import date + +import numpy as np +import pytest +from hypothesis import HealthCheck, example, given, settings +from hypothesis import strategies as st + +from policyengine_core.periods import period as make_period +from policyengine_core.reforms import Reform +from policyengine_core.simulations import Simulation +from policyengine_uk.system import system +from policyengine_uk.variables.gov.dwp.housing_benefit.applicable_income.housing_benefit_specified_or_temporary_accommodation_disregard import ( + hb_accommodation_segments, +) + + +PREFIX = ( + "gov.dwp.housing_benefit.means_test.income_disregard." + "specified_or_temporary_accommodation" +) +RATES = { + ("single", "younger"): 61.41, + ("single", "older"): 77.73, + ("lone_parent", "younger"): 61.41, + ("lone_parent", "older"): 77.73, + ("couple", "minors"): 97.33, + ("couple", "younger"): 61.53, + ("couple", "older"): 119.70, +} +AGES = {"minors": 17, "younger": 23, "older": 30} +TOLERANCE = 0.01 +PROPERTY_SETTINGS = settings( + max_examples=5, + deadline=None, + derandomize=True, + suppress_health_check=[HealthCheck.too_slow, HealthCheck.data_too_large], +) + + +def case(**overrides): + result = dict( + family="single", + age="older", + net=1_000, + other=0, + special=False, + permitted=0, + childcare=0, + credits=0, + worker=False, + rent=10_400, + applicable=500, + non_dep=0, + ) + result.update(overrides) + return result + + +@st.composite +def families(draw): + family, age = draw(st.sampled_from(list(RATES))) + return case( + family=family, + age=age, + net=draw(st.integers(-1_000, 30_000)), + other=draw(st.integers(0, 20_000)), + special=draw(st.booleans()), + permitted=draw(st.sampled_from([0, 520, 1_040, 1_300, 9_542])), + childcare=draw(st.integers(0, 20_000)) if family == "lone_parent" else 0, + credits=draw(st.integers(0, 5_000)), + worker=draw(st.booleans()), + rent=draw(st.integers(0, 30_000)), + applicable=draw(st.integers(0, 15_000)), + non_dep=draw(st.integers(0, 5_000)), + ) + + +EDGE_CASES = [case(family=family, age=age) for family, age in RATES] +EDGE_CASES += [ + case(net=-100, rent=0), + case(net=30_000, rent=100), + case(family="lone_parent", net=10_000, childcare=20_000, credits=500), + case(net=1_149.20, worker=True), + case(net=1_149.21, worker=True, permitted=520), +] +POPULATIONS = st.lists(families(), min_size=1, max_size=24) + + +def calculate(units, year=2026, rates=None): + people, benunits, households = {}, {}, {} + for i, unit in enumerate(units): + adults = 2 if unit["family"] == "couple" else 1 + children = int(unit["family"] == "lone_parent") + members = [] + for j in range(adults + children): + name = f"p{i}_{j}" + gross = max(1, unit["net"]) if j == 0 else 0 + people[name] = { + "age": {year: AGES[unit["age"]] if j < adults else 8}, + "is_claimant_or_partner": {year: j < adults}, + "employment_income": {year: gross}, + "income_tax_before_winter_fuel_payment_charge": { + year: gross - unit["net"] if j == 0 else 0 + }, + "national_insurance": {year: 0}, + "pension_contributions": {year: 0}, + "private_pension_income": {year: unit["other"] if j == 0 else 0}, + "childcare_expenses": {year: unit["childcare"] if j == 0 else 0}, + } + members.append(name) + inputs = { + "housing_benefit_net_earnings": unit["net"], + "housing_benefit_special_earnings_disregard_eligible": unit["special"], + "housing_benefit_permitted_work_disregard": unit["permitted"], + "meets_housing_benefit_additional_earnings_disregard_conditions": unit[ + "worker" + ], + "housing_benefit_applicable_amount": unit["applicable"], + "benunit_rent": unit["rent"], + "housing_benefit_non_dep_deductions": unit["non_dep"], + "housing_benefit_meals_deduction": 0, + "housing_benefit_tariff_income": 0, + "LHA_eligible": False, + "in_specified_or_temporary_accommodation": True, + "housing_benefit_on_passporting_benefit": False, + "housing_benefit_pension_age_regulations_apply": False, + "working_tax_credit": unit["credits"], + "child_tax_credit": 0, + "tax_credits": unit["credits"], + "income_support": 0, + "jsa_income": 0, + "esa_income": 0, + } + benunits[f"b{i}"] = { + "members": members, + **{name: {year: value} for name, value in inputs.items()}, + } + households[f"h{i}"] = {"members": members} + situation = {"people": people, "benunits": benunits, "households": households} + if rates is None: + sim = Simulation(tax_benefit_system=system, situation=situation) + else: + reform = Reform.from_dict( + { + f"{PREFIX}.{family}.{age}": {"2026-10-05.2027-04-05": value} + for (family, age), value in rates.items() + } + ) + sim = Simulation(tax_benefit_system=reform(system), situation=situation) + return sim, np.asarray( + sim.calculate("housing_benefit_entitlement", year), dtype=float + ) + + +def direct_award(unit, weekly, year=2026): + """Independent complete award for a full, unweighted weekly rate.""" + p = system.parameters(year).gov.dwp.housing_benefit.means_test + standard_rates = p.income_disregard + net = max(0, unit["net"]) + lone_parent = unit["family"] == "lone_parent" + selected = min( + net, + 52 + * ( + standard_rates.lone_parent + if lone_parent + else standard_rates.special + if unit["special"] + else standard_rates.couple + if unit["family"] == "couple" + else standard_rates.single + ), + ) + standard = ( + min(net, max(unit["permitted"], selected if lone_parent else 0)) + if unit["permitted"] > 0 + else selected + ) + accommodation = min(max(0, weekly * 52), max(0, net - standard)) + cap = system.parameters( + year + ).gov.dwp.tax_credits.working_tax_credit.elements.childcare_1 + charges = min(unit["childcare"], cap * 52) if lone_parent else 0 + childcare = min(charges, max(0, net - standard - accommodation) + unit["credits"]) + additional = standard_rates.worker * 52 + if not unit["worker"] or round(net * 100) < round( + (standard + accommodation + childcare + additional) * 100 + ): + additional = 0 + income = max( + 0, + unit["net"] + + unit["other"] + + unit["credits"] + - standard + - accommodation + - childcare + - additional, + ) + return max( + 0, + unit["rent"] + - unit["non_dep"] + - p.withdrawal_rate * max(0, income - unit["applicable"]), + ) + + +def segments(sim, year): + return list( + hb_accommodation_segments( + sim.benunit, make_period(year), sim.tax_benefit_system.parameters + ) + ) + + +@PROPERTY_SETTINGS +@given(units=POPULATIONS) +@example(units=EDGE_CASES) +def test_d1_no_change_no_segments(units): + sim, awards = calculate(units, year=2027) + schedule = sim.tax_benefit_system.parameters.gov.dwp.housing_benefit.means_test.income_disregard.specified_or_temporary_accommodation + for family, age in RATES: + leaf = schedule.children[family].children[age] + assert not any( + "2027-04-06" < v.instant_str < "2028-04-06" for v in leaf.values_list + ) + actual_segments = segments(sim, 2027) + assert len(actual_segments) == 1 + assert actual_segments[0][1] == 1 + expected = [direct_award(u, RATES[u["family"], u["age"]], 2027) for u in units] + np.testing.assert_allclose(awards, expected, atol=TOLERANCE, rtol=0) + + +@PROPERTY_SETTINGS +@given(units=POPULATIONS) +@example(units=EDGE_CASES) +def test_d2_convex_combination(units): + _, awards = calculate(units) + before_days = (date(2026, 10, 5) - date(2026, 4, 6)).days + after_days = (date(2027, 4, 6) - date(2026, 10, 5)).days + before = np.array([direct_award(u, 0) for u in units]) + after = np.array([direct_award(u, RATES[u["family"], u["age"]]) for u in units]) + expected = (before * before_days + after * after_days) / (before_days + after_days) + np.testing.assert_allclose(awards, expected, atol=TOLERANCE, rtol=0) + assert np.all(awards >= np.minimum(before, after) - TOLERANCE) + assert np.all(awards <= np.maximum(before, after) + TOLERANCE) + + +@PROPERTY_SETTINGS +@given(units=POPULATIONS) +@example(units=EDGE_CASES) +def test_d3_bounds(units): + _, awards = calculate(units) + assert np.all(awards >= 0) + assert np.all(awards <= np.array([u["rent"] for u in units]) + TOLERANCE) + + +@PROPERTY_SETTINGS +@given(units=POPULATIONS, bump=st.integers(1, 100_000)) +@example(units=EDGE_CASES, bump=100) +def test_d4_monotonicity_without_additional_disregard(units, bump): + units = [{**u, "worker": False} for u in units] + _, before = calculate(units) + _, after = calculate( + units, rates={key: value + bump / 100 for key, value in RATES.items()} + ) + assert np.all(after >= before - TOLERANCE) + + +@pytest.mark.xfail( + strict=True, + raises=AssertionError, + reason="D4 intended violation: accommodation can switch off the additional disregard", +) +@PROPERTY_SETTINGS +@given(extra=st.integers(0, 10_000), bump=st.integers(1, 100)) +@example(extra=0, bump=1) +def test_d4_unrestricted_monotonicity(extra, bump): + # Shrink an additional-disregard boundary witness independently of the + # current statutory accommodation amount, by reforming the later rate. + unit = case(net=1_149.20 + extra, worker=True, applicable=0) + low = {key: extra / 52 for key in RATES} + high = {key: (extra + bump / 100) / 52 for key in RATES} + _, before = calculate([unit], rates=low) + _, after = calculate([unit], rates=high) + assert after[0] >= before[0] - TOLERANCE, (extra, bump, before[0], after[0]) + + +@PROPERTY_SETTINGS +@given(units=POPULATIONS) +@example(units=EDGE_CASES) +def test_d5_day_count(units): + sim, _ = calculate(units) + actual_segments = segments(sim, 2026) + assert len(actual_segments) == 2 + days = [share * 365 for _, share in actual_segments] + assert days == pytest.approx([182, 183], abs=1e-10) + assert sum(days) == pytest.approx(365, abs=1e-10) + np.testing.assert_array_equal(actual_segments[0][0], np.zeros(len(units))) + np.testing.assert_allclose( + actual_segments[1][0], + [52 * RATES[u["family"], u["age"]] for u in units], + atol=TOLERANCE, + rtol=0, + ) From bda6163b8d04b2e6042bb82b580458f7ecbc2bea Mon Sep 17 00:00:00 2001 From: Max Ghenis Date: Thu, 29 Oct 2026 03:07:22 -0400 Subject: [PATCH 04/12] Pin the additional-disregard cliff with a passing regression, not an xfail MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit CI's Test / Result job fails any xfail without an allowlist entry, and the allowlist is for known bugs. The D4 counterexample is the law, not a bug: HB Regs 2006 Sch. 4 para. 17(1) and (3), as SI 2026/978 reg. 2 amends them, apply the additional disregard only where net earnings reach the standard, accommodation, childcare and additional amounts together, so a penny of accommodation can remove all £889.20 of it. Main shows the same cliff in 2027-28, with no date segments. The new test pins the case: £10,400 with no accommodation, £10,110.22 with a penny, the later segment's award from the independent oracle, and the 182/183-day combination. The restricted monotonicity property stays. Co-Authored-By: Claude Opus 5.5 --- ..._benefit_accommodation_dates_properties.py | 53 ++++++++++--------- 1 file changed, 29 insertions(+), 24 deletions(-) diff --git a/policyengine_uk/tests/test_housing_benefit_accommodation_dates_properties.py b/policyengine_uk/tests/test_housing_benefit_accommodation_dates_properties.py index 508a2fbefd..283823a670 100644 --- a/policyengine_uk/tests/test_housing_benefit_accommodation_dates_properties.py +++ b/policyengine_uk/tests/test_housing_benefit_accommodation_dates_properties.py @@ -6,13 +6,15 @@ equals their independently calculated elapsed-day-weighted mean. D3. Bounds: with nonnegative eligible rent and non-dependant deductions, entitlement is between zero and eligible rent. -D4. Monotonicity: raising accommodation never lowers the annual award. - Intended violation: the additional-disregard earnings test can switch - off. At net earnings of 1,149.20, standard 260 and no childcare, raising - later accommodation from zero to one penny removes the 889.20 additional - disregard. The annual award falls from 10,400 to 10,110.22. The strict - xfail retains this smallest positive penny increment; a separate property - verifies monotonicity with the additional disregard disabled. +D4. Monotonicity: without the additional disregard, raising accommodation + never lowers the annual award. With it, the award can fall, as the law + intends: the additional disregard applies only where net earnings reach + the standard, accommodation, childcare and additional amounts together + (HB Regs 2006 Sch. 4 para. 17(1) and (3), as SI 2026/978 reg. 2 amends + them; NI Sch. 5 para. 17 and SR 2026/157 reg. 2). At net earnings of + 1,149.20, standard 260 and no childcare, raising later accommodation from + zero to one penny removes the 889.20 additional disregard, and the annual + award falls from 10,400 to 10,110.22. A regression pins that case. D5. Day count: 2026-27 has 182 days before 5 October and 183 from that date, totalling 365; inspect production segments, not just a calendar. @@ -293,23 +295,26 @@ def test_d4_monotonicity_without_additional_disregard(units, bump): assert np.all(after >= before - TOLERANCE) -@pytest.mark.xfail( - strict=True, - raises=AssertionError, - reason="D4 intended violation: accommodation can switch off the additional disregard", -) -@PROPERTY_SETTINGS -@given(extra=st.integers(0, 10_000), bump=st.integers(1, 100)) -@example(extra=0, bump=1) -def test_d4_unrestricted_monotonicity(extra, bump): - # Shrink an additional-disregard boundary witness independently of the - # current statutory accommodation amount, by reforming the later rate. - unit = case(net=1_149.20 + extra, worker=True, applicable=0) - low = {key: extra / 52 for key in RATES} - high = {key: (extra + bump / 100) / 52 for key in RATES} - _, before = calculate([unit], rates=low) - _, after = calculate([unit], rates=high) - assert after[0] >= before[0] - TOLERANCE, (extra, bump, before[0], after[0]) +def test_d4_a_penny_of_accommodation_can_switch_off_the_additional_disregard(): + # Net earnings of 1,149.20 equal the 260 standard plus the 889.20 + # additional disregard, so the additional disregard applies with no + # accommodation amount and stops applying with a penny of it. The later + # rate is reformed to a penny a year so that only the threshold moves. + unit = case(net=1_149.20, worker=True, applicable=0) + _, before = calculate([unit], rates={key: 0 for key in RATES}) + _, after = calculate([unit], rates={key: 0.01 / 52 for key in RATES}) + assert before[0] == pytest.approx(10_400, abs=TOLERANCE) + assert after[0] == pytest.approx(10_110.22, abs=TOLERANCE) + # The later segment loses the whole additional disregard: its award is + # 10,400 less the 65% taper on 1,149.20 - 260 - 0.01 = 889.19 of income. + later = direct_award(unit, 0.01 / 52) + assert later == pytest.approx(10_400 - 0.65 * 889.19, abs=TOLERANCE) + before_days = (date(2026, 10, 5) - date(2026, 4, 6)).days + after_days = (date(2027, 4, 6) - date(2026, 10, 5)).days + expected = (direct_award(unit, 0) * before_days + later * after_days) / ( + before_days + after_days + ) + assert after[0] == pytest.approx(expected, abs=TOLERANCE) @PROPERTY_SETTINGS From b8e40c3a1c945169d65e9e5ec74e18dc0ab9bb1f Mon Sep 17 00:00:00 2001 From: Max Ghenis Date: Thu, 29 Oct 2026 04:30:25 -0400 Subject: [PATCH 05/12] Read visible branches through the reviewed supplied-inputs helper main's test_every_walk_up_to_an_ancestor_simulation_is_reviewed lists each function that walks parent_branch. hb_uses_annual_override walked it itself to collect branch names; _visible_branch_names returns the same set (this branch, its ancestors, default), so the check reads the same storage keys. Co-Authored-By: Claude Opus 5.5 --- ...nefit_specified_or_temporary_accommodation_disregard.py | 7 ++----- 1 file changed, 2 insertions(+), 5 deletions(-) diff --git a/policyengine_uk/variables/gov/dwp/housing_benefit/applicable_income/housing_benefit_specified_or_temporary_accommodation_disregard.py b/policyengine_uk/variables/gov/dwp/housing_benefit/applicable_income/housing_benefit_specified_or_temporary_accommodation_disregard.py index 1218ad5f38..d10c7d0b1a 100644 --- a/policyengine_uk/variables/gov/dwp/housing_benefit/applicable_income/housing_benefit_specified_or_temporary_accommodation_disregard.py +++ b/policyengine_uk/variables/gov/dwp/housing_benefit/applicable_income/housing_benefit_specified_or_temporary_accommodation_disregard.py @@ -1,6 +1,7 @@ from policyengine_uk.model_api import * from policyengine_core.parameters import ParameterNode from policyengine_uk.utils.excise import fiscal_year_segments +from policyengine_uk.utils.supplied_inputs import _visible_branch_names def hb_uses_annual_override(benunit, period, variable_name, default_formula): @@ -16,11 +17,7 @@ def hb_uses_annual_override(benunit, period, variable_name, default_formula): return True # Core records set_input separately from computed caches. Reading those # records avoids treating a cached formula result as an explicit override. - branch = simulation - visible = {"default"} - while branch is not None: - visible.add(branch.branch_name) - branch = getattr(branch, "parent_branch", None) + visible = set(_visible_branch_names(simulation)) holder = benunit.get_holder(variable_name) return any( name == variable_name From 5dc13b793b1b81b56ed068d41766104e2e5a1d75 Mon Sep 17 00:00:00 2001 From: Max Ghenis Date: Thu, 29 Oct 2026 05:07:55 -0400 Subject: [PATCH 06/12] Keep HB override provenance in step with population rebuilds MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit A Sol review of b488ed1cc found that build_from_populations keeps the replaced holders' input records (policyengine-core#605). Once the applicable-income formula refilled the period, hb_uses_annual_override read the stale record as a supplied value and skipped the date segments: £10,400 instead of the fresh £10,322.21. The five variables the check asks about join SUPPLIED_INPUT_VARIABLES, so Simulation.calculate forgets such a record before the formula refills the period. The check moves to a shared stored_as_input helper, which counts a value stored as an input, supplied or the model's own (set_model_value). supplied_input would drop a scenario's own value: the variable would read £2,000 while the 2026-27 award ignored it (£10,322.21, not £9,425). Co-Authored-By: Claude Opus 5.5 --- ...est_housing_benefit_accommodation_dates.py | 21 ++++++++++++++++++ policyengine_uk/utils/supplied_inputs.py | 21 ++++++++++++++++++ ...ed_or_temporary_accommodation_disregard.py | 22 +++++++------------ 3 files changed, 50 insertions(+), 14 deletions(-) diff --git a/policyengine_uk/tests/test_housing_benefit_accommodation_dates.py b/policyengine_uk/tests/test_housing_benefit_accommodation_dates.py index 1c941b01a7..df657eae2a 100644 --- a/policyengine_uk/tests/test_housing_benefit_accommodation_dates.py +++ b/policyengine_uk/tests/test_housing_benefit_accommodation_dates.py @@ -6,6 +6,7 @@ from policyengine_uk import CountryTaxBenefitSystem, Simulation from policyengine_uk.model_api import Variable, BenUnit, YEAR +from policyengine_uk.utils.supplied_inputs import set_model_value PREFIX = ( @@ -158,6 +159,26 @@ def test_only_supplied_period_overrides_recalculation(): ] == pytest.approx((before * 182 + 10_400 * 183) / 365, abs=0.01) +def test_rebuilt_populations_leave_no_supplied_income_behind(): + data = situation() + data["benunits"]["family"]["housing_benefit_applicable_income"] = {2026: 2_000} + simulation = Simulation(situation=data) + simulation.build_from_populations(Simulation(situation=situation()).populations) + simulation.calculate("housing_benefit_applicable_income", 2026) + before = 10_400 - 0.65 * (1_000 - 260 - 500) + assert simulation.calculate("housing_benefit_entitlement", 2026)[ + 0 + ] == pytest.approx((before * 182 + 10_400 * 183) / 365, abs=0.01) + + +def test_scenario_adjusted_income_keeps_its_value(): + simulation = Simulation(situation=situation()) + set_model_value(simulation, "housing_benefit_applicable_income", 2026, [2_000]) + assert simulation.calculate("housing_benefit_entitlement", 2026)[ + 0 + ] == pytest.approx(10_400 - 0.65 * (2_000 - 500), abs=0.01) + + def test_replaced_income_formula_is_not_bypassed(): class housing_benefit_applicable_income(Variable): entity = BenUnit diff --git a/policyengine_uk/utils/supplied_inputs.py b/policyengine_uk/utils/supplied_inputs.py index 70e7b0edf3..12ad3ebf0e 100644 --- a/policyengine_uk/utils/supplied_inputs.py +++ b/policyengine_uk/utils/supplied_inputs.py @@ -56,6 +56,11 @@ "extended_childcare_entitlement_term_share", "extended_childcare_entitlement_total_per_child", "gas_consumption_kwh", + "housing_benefit_applicable_income", + "housing_benefit_applicable_income_childcare_element", + "housing_benefit_applicable_income_disregard", + "housing_benefit_childcare_earnings_limit", + "housing_benefit_specified_or_temporary_accommodation_disregard", "in_HE", "other_local_bus_trips", "pip_dl_category", @@ -167,6 +172,22 @@ def supplied_by(population, variable_name: str, period: Period) -> bool: ) +def stored_as_input(population, variable_name: str, period: Period) -> bool: + """Whether a value set as an input for ``period``, supplied or the + model's own (``set_model_value``), is stored for ``variable_name`` on + this branch or one it was made from. A value the engine carried over from + an earlier period, or cached after a calculation, is not.""" + _check_registered(variable_name) + simulation = population.simulation + input_keys = getattr(simulation, "_user_input_keys", None) or () + holder = population.get_holder(variable_name) + return any( + (variable_name, branch_name, period) in input_keys + and holder._get_array_from_storage(period, branch_name) is not None + for branch_name in _visible_branch_names(simulation) + ) + + def supplied_input( population, variable_name: str, period: Period ) -> Optional[np.ndarray]: diff --git a/policyengine_uk/variables/gov/dwp/housing_benefit/applicable_income/housing_benefit_specified_or_temporary_accommodation_disregard.py b/policyengine_uk/variables/gov/dwp/housing_benefit/applicable_income/housing_benefit_specified_or_temporary_accommodation_disregard.py index d10c7d0b1a..4808207ca5 100644 --- a/policyengine_uk/variables/gov/dwp/housing_benefit/applicable_income/housing_benefit_specified_or_temporary_accommodation_disregard.py +++ b/policyengine_uk/variables/gov/dwp/housing_benefit/applicable_income/housing_benefit_specified_or_temporary_accommodation_disregard.py @@ -1,7 +1,7 @@ from policyengine_uk.model_api import * from policyengine_core.parameters import ParameterNode from policyengine_uk.utils.excise import fiscal_year_segments -from policyengine_uk.utils.supplied_inputs import _visible_branch_names +from policyengine_uk.utils.supplied_inputs import stored_as_input def hb_uses_annual_override(benunit, period, variable_name, default_formula): @@ -15,19 +15,13 @@ def hb_uses_annual_override(benunit, period, variable_name, default_formula): or formula.__code__ != default_formula.__code__ ): return True - # Core records set_input separately from computed caches. Reading those - # records avoids treating a cached formula result as an explicit override. - visible = set(_visible_branch_names(simulation)) - holder = benunit.get_holder(variable_name) - return any( - name == variable_name - and input_period == period - and branch_name in visible - and holder._get_array_from_storage(input_period, branch_name) is not None - for name, branch_name, input_period in getattr( - simulation, "_user_input_keys", () - ) - ) + # A value set as an input for this period overrides recalculation, and so + # does a scenario's own value (set_model_value), which the variable then + # reads; a cached formula result does not. SUPPLIED_INPUT_VARIABLES lists + # each name passed here, so Simulation.calculate forgets an input record + # that a population rebuild left without its value before the formula + # refills the period. + return stored_as_input(benunit, variable_name, period) def hb_accommodation_segments(benunit, period, parameters): From d5c96d95a07f240af4638a94a7c2ada3edf02a21 Mon Sep 17 00:00:00 2001 From: Max Ghenis Date: Thu, 29 Oct 2026 06:25:42 -0400 Subject: [PATCH 07/12] Read input provenance from the holder's own simulation MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit stored_as_input checked this simulation's input record, so after build_from_populations a stale record could call a donor's calculated applicable income supplied (award £10,400, not £10,322.21), and a value the donor had supplied could count as calculated (£10,322.21, not the annual taper's £8,775). It now asks the holder, which answers from the record of the simulation it was built in, so moved populations keep their own provenance. set_model_value values still count. Co-Authored-By: Claude Opus 5.5 --- ...est_housing_benefit_accommodation_dates.py | 33 +++++++++++++++++++ policyengine_uk/utils/supplied_inputs.py | 21 +++++++----- ...ed_or_temporary_accommodation_disregard.py | 9 +++-- 3 files changed, 49 insertions(+), 14 deletions(-) diff --git a/policyengine_uk/tests/test_housing_benefit_accommodation_dates.py b/policyengine_uk/tests/test_housing_benefit_accommodation_dates.py index df657eae2a..8193580669 100644 --- a/policyengine_uk/tests/test_housing_benefit_accommodation_dates.py +++ b/policyengine_uk/tests/test_housing_benefit_accommodation_dates.py @@ -171,6 +171,39 @@ def test_rebuilt_populations_leave_no_supplied_income_behind(): ] == pytest.approx((before * 182 + 10_400 * 183) / 365, abs=0.01) +@pytest.mark.parametrize( + "variable", + [ + "housing_benefit_applicable_income", + "housing_benefit_applicable_income_disregard", + "housing_benefit_specified_or_temporary_accommodation_disregard", + ], +) +def test_rebuilt_populations_keep_a_calculated_value_calculated(variable): + # The populations come from a simulation that has calculated the value; + # this simulation's own record still says it was supplied. + data = situation() + data["benunits"]["family"][variable] = {2026: 2_000} + simulation = Simulation(situation=data) + donor = Simulation(situation=situation()) + donor.calculate(variable, 2026) + simulation.build_from_populations(donor.populations) + before = 10_400 - 0.65 * (1_000 - 260 - 500) + assert simulation.calculate("housing_benefit_entitlement", 2026)[ + 0 + ] == pytest.approx((before * 182 + 10_400 * 183) / 365, abs=0.01) + + +def test_rebuilt_populations_keep_a_supplied_income(): + data = situation() + data["benunits"]["family"]["housing_benefit_applicable_income"] = {2026: 3_000} + simulation = Simulation(situation=situation()) + simulation.build_from_populations(Simulation(situation=data).populations) + assert simulation.calculate("housing_benefit_entitlement", 2026)[ + 0 + ] == pytest.approx(10_400 - 0.65 * (3_000 - 500), abs=0.01) + + def test_scenario_adjusted_income_keeps_its_value(): simulation = Simulation(situation=situation()) set_model_value(simulation, "housing_benefit_applicable_income", 2026, [2_000]) diff --git a/policyengine_uk/utils/supplied_inputs.py b/policyengine_uk/utils/supplied_inputs.py index 12ad3ebf0e..1eb4719056 100644 --- a/policyengine_uk/utils/supplied_inputs.py +++ b/policyengine_uk/utils/supplied_inputs.py @@ -173,18 +173,21 @@ def supplied_by(population, variable_name: str, period: Period) -> bool: def stored_as_input(population, variable_name: str, period: Period) -> bool: - """Whether a value set as an input for ``period``, supplied or the - model's own (``set_model_value``), is stored for ``variable_name`` on - this branch or one it was made from. A value the engine carried over from - an earlier period, or cached after a calculation, is not.""" + """Whether the value stored for ``variable_name`` and ``period``, on this + branch or one it was made from, was set as an input: supplied, or the + model's own (``set_model_value``). A value the engine carried over from + an earlier period, or cached after a calculation, is not. + + The holder answers from the input record of the simulation it was built + in. Populations taken from another simulation (``build_from_populations``) + keep their own record, so a value the other simulation calculated stays + calculated, and one supplied there stays supplied, whatever this + simulation's record says.""" _check_registered(variable_name) - simulation = population.simulation - input_keys = getattr(simulation, "_user_input_keys", None) or () holder = population.get_holder(variable_name) return any( - (variable_name, branch_name, period) in input_keys - and holder._get_array_from_storage(period, branch_name) is not None - for branch_name in _visible_branch_names(simulation) + holder._is_input(period, branch_name) + for branch_name in _visible_branch_names(population.simulation) ) diff --git a/policyengine_uk/variables/gov/dwp/housing_benefit/applicable_income/housing_benefit_specified_or_temporary_accommodation_disregard.py b/policyengine_uk/variables/gov/dwp/housing_benefit/applicable_income/housing_benefit_specified_or_temporary_accommodation_disregard.py index 4808207ca5..214dcc52bb 100644 --- a/policyengine_uk/variables/gov/dwp/housing_benefit/applicable_income/housing_benefit_specified_or_temporary_accommodation_disregard.py +++ b/policyengine_uk/variables/gov/dwp/housing_benefit/applicable_income/housing_benefit_specified_or_temporary_accommodation_disregard.py @@ -5,7 +5,7 @@ def hb_uses_annual_override(benunit, period, variable_name, default_formula): - """Keep supplied annual values and replacement formulas out of recalculation.""" + """Keep annual inputs and replacement formulas out of recalculation.""" simulation = benunit.simulation variable = simulation.tax_benefit_system.get_variable(variable_name) formula = variable.get_formula(period) @@ -17,10 +17,9 @@ def hb_uses_annual_override(benunit, period, variable_name, default_formula): return True # A value set as an input for this period overrides recalculation, and so # does a scenario's own value (set_model_value), which the variable then - # reads; a cached formula result does not. SUPPLIED_INPUT_VARIABLES lists - # each name passed here, so Simulation.calculate forgets an input record - # that a population rebuild left without its value before the formula - # refills the period. + # reads; a cached formula result does not, even in populations taken from + # another simulation. SUPPLIED_INPUT_VARIABLES lists each name passed + # here, as stored_as_input and set_model_value require. return stored_as_input(benunit, variable_name, period) From 6e090ed0000a6ddce7fd5e645cf0c14313a3ba95 Mon Sep 17 00:00:00 2001 From: Max Ghenis Date: Thu, 29 Oct 2026 08:04:15 -0400 Subject: [PATCH 08/12] Move input records with the holders of rebuilt populations Simulation.build_from_populations points each population at the new simulation, but core leaves each holder bound to the simulation that made it and keeps the new simulation's record of the values the rebuild replaced (policyengine-core#605). A clone or branch copies the simulation's record and binds its copied holders to itself, so it read the replaced populations' record: a supplied income of 3,000 from another simulation awarded 8,775 in the rebuilt simulation but 10,322.21 in its clone or branch. An input set on the rebuilt simulation went into the other simulation's record, and supplied_input missed the other simulation's inputs (#2747). adopt_population_records binds each holder from another simulation to this one, keeps that simulation's input and model-value records for the values the holder still stores, drops the values it calculated or carried over, and forgets this simulation's records of values the rebuild removed. build_from_populations also empties the fast cache, which formulas read. A Hypothesis property checks that the rebuilt simulation, its clone and its branch give the provenance and award of a fresh copy of the other simulation. It found two cases that also fail at d5c96d95a, now pinned as examples: a donor's income cached before a scenario set its disregard (10,244 against 10,075), and a receiver's fast-cache income from before the rebuild (10,400 against 9,042.15). Fixes #2747 Co-Authored-By: Claude Opus 5.5 --- .../rebuilt-population-records.fixed.md | 1 + policyengine_uk/simulation.py | 14 +- ...est_housing_benefit_accommodation_dates.py | 175 +++++++++++++++++- policyengine_uk/tests/test_input_records.py | 61 +++++- policyengine_uk/utils/supplied_inputs.py | 101 +++++++--- ...ed_or_temporary_accommodation_disregard.py | 7 +- 6 files changed, 328 insertions(+), 31 deletions(-) create mode 100644 changelog.d/rebuilt-population-records.fixed.md diff --git a/changelog.d/rebuilt-population-records.fixed.md b/changelog.d/rebuilt-population-records.fixed.md new file mode 100644 index 0000000000..d1baf414e1 --- /dev/null +++ b/changelog.d/rebuilt-population-records.fixed.md @@ -0,0 +1 @@ +- Fixed `Simulation.build_from_populations` with another simulation's populations: the simulation, its clones and its branches now treat the values that simulation supplied as supplied, record inputs set afterwards, and calculate again the values that simulation calculated or this one cached for the replaced populations. diff --git a/policyengine_uk/simulation.py b/policyengine_uk/simulation.py index dd1e49c53a..c3df87de02 100644 --- a/policyengine_uk/simulation.py +++ b/policyengine_uk/simulation.py @@ -83,10 +83,10 @@ ) from policyengine_uk.utils.supplied_inputs import ( SUPPLIED_INPUT_VARIABLES, + adopt_population_records, copy_model_values, drop_missing_model_values, drop_missing_supplied_inputs, - forget_model_values, forget_replaced_model_values, model_value_input_records, ) @@ -592,10 +592,14 @@ def delete_arrays(self, variable: str, period: Period = None) -> None: def build_from_populations(self, populations) -> None: super().build_from_populations(populations) - # The new populations hold none of the values set_model_value stored, - # so an equal value a rebuild stores (from a situation or a frame) is - # supplied. - forget_model_values(self) + # Formulas read the fast cache, so drop what it holds for the replaced + # populations, as _start_new_population does. + if getattr(self, "_fast_cache", None) is not None: + self._fast_cache = {} + # Keep the records of inputs and model values (set_model_value) of + # the values the new populations hold, and only those: an equal value + # a rebuild stores later (from a situation or a frame) is supplied. + adopt_population_records(self) def set_input(self, variable_name: str, period: Period, value) -> None: # What is set here is supplied, even a value equal to the model's own, diff --git a/policyengine_uk/tests/test_housing_benefit_accommodation_dates.py b/policyengine_uk/tests/test_housing_benefit_accommodation_dates.py index 8193580669..d29b7bcd29 100644 --- a/policyengine_uk/tests/test_housing_benefit_accommodation_dates.py +++ b/policyengine_uk/tests/test_housing_benefit_accommodation_dates.py @@ -3,10 +3,19 @@ from datetime import date import pytest +from hypothesis import HealthCheck, example, given, settings +from hypothesis import strategies as st + +from policyengine_core.periods import period from policyengine_uk import CountryTaxBenefitSystem, Simulation from policyengine_uk.model_api import Variable, BenUnit, YEAR -from policyengine_uk.utils.supplied_inputs import set_model_value +from policyengine_uk.utils.aggregate_impact.guard import snapshot_inputs +from policyengine_uk.utils.supplied_inputs import ( + set_model_value, + stored_as_input, + supplied_input, +) PREFIX = ( @@ -212,6 +221,170 @@ def test_scenario_adjusted_income_keeps_its_value(): ] == pytest.approx(10_400 - 0.65 * (2_000 - 500), abs=0.01) +def snapshotted(simulation): + # The aggregate-impact guard discards formula results before it captures + # the inputs, so it must tell them apart from rebuilt populations' inputs. + snapshot_inputs(simulation, ["housing_benefit_applicable_income"]) + return simulation + + +# A clone or branch copies its simulation's records and binds the holders it +# copies to itself, so it must find the records of rebuilt populations there. +REBUILT_AND_COPIES = pytest.mark.parametrize( + "copy", + [ + lambda simulation: simulation, + lambda simulation: simulation.clone(), + lambda simulation: simulation.get_branch("after_rebuild"), + snapshotted, + ], + ids=["rebuilt", "clone", "branch", "snapshot"], +) + + +@REBUILT_AND_COPIES +def test_copies_of_rebuilt_populations_keep_a_supplied_income(copy): + data = situation() + data["benunits"]["family"]["housing_benefit_applicable_income"] = {2026: 3_000} + simulation = Simulation(situation=situation()) + simulation.build_from_populations(Simulation(situation=data).populations) + copied = copy(simulation) + assert copied.calculate("housing_benefit_entitlement", 2026)[0] == pytest.approx( + 10_400 - 0.65 * (3_000 - 500), abs=0.01 + ) + benunit = copied.populations["benunit"] + assert supplied_input( + benunit, "housing_benefit_applicable_income", period(2026) + ).tolist() == [3_000] + + +@REBUILT_AND_COPIES +def test_copies_of_rebuilt_populations_keep_a_calculated_income_calculated(copy): + data = situation() + data["benunits"]["family"]["housing_benefit_applicable_income"] = {2026: 2_000} + simulation = Simulation(situation=data) + donor = Simulation(situation=situation()) + donor.calculate("housing_benefit_applicable_income", 2026) + simulation.build_from_populations(donor.populations) + copied = copy(simulation) + before = 10_400 - 0.65 * (1_000 - 260 - 500) + assert copied.calculate("housing_benefit_entitlement", 2026)[0] == pytest.approx( + (before * 182 + 10_400 * 183) / 365, abs=0.01 + ) + benunit = copied.populations["benunit"] + assert ( + supplied_input(benunit, "housing_benefit_applicable_income", period(2026)) + is None + ) + + +@REBUILT_AND_COPIES +def test_copies_of_rebuilt_populations_keep_a_scenario_adjusted_income(copy): + donor = Simulation(situation=situation()) + set_model_value(donor, "housing_benefit_applicable_income", 2026, [2_000]) + simulation = Simulation(situation=situation()) + simulation.build_from_populations(donor.populations) + copied = copy(simulation) + assert copied.calculate("housing_benefit_entitlement", 2026)[0] == pytest.approx( + 10_400 - 0.65 * (2_000 - 500), abs=0.01 + ) + # Still the model's value, not a supplied one. + benunit = copied.populations["benunit"] + assert ( + supplied_input(benunit, "housing_benefit_applicable_income", period(2026)) + is None + ) + + +def test_rebuilt_populations_replace_values_calculated_before(): + # Formulas read the fast cache, which held the income calculated here. + simulation = Simulation(situation=situation()) + simulation.calculate("housing_benefit_applicable_income", 2026) + data = situation() + data["benunits"]["family"]["housing_benefit_applicable_income"] = {2026: 3_000} + simulation.build_from_populations(Simulation(situation=data).populations) + assert simulation.calculate("housing_benefit_entitlement", 2026)[ + 0 + ] == pytest.approx(10_400 - 0.65 * (3_000 - 500), abs=0.01) + + +# How a simulation comes to hold each override variable's 2026 value. +OVERRIDE_VARIABLES = ( + "housing_benefit_applicable_income", + "housing_benefit_applicable_income_disregard", + "housing_benefit_specified_or_temporary_accommodation_disregard", +) +SIMULATION_STATES = st.tuples( + *( + st.tuples( + st.sampled_from(["absent", "supplied", "model", "calculated"]), + st.integers(0, 4_000), + ) + for _ in OVERRIDE_VARIABLES + ) +) + + +def simulation_in(state): + data = situation() + for name, (origin, amount) in zip(OVERRIDE_VARIABLES, state): + if origin == "supplied": + data["benunits"]["family"][name] = {2026: amount} + simulation = Simulation(situation=data) + for name, (origin, amount) in zip(OVERRIDE_VARIABLES, state): + if origin == "model": + set_model_value(simulation, name, 2026, [amount]) + elif origin == "calculated": + simulation.calculate(name, 2026) + return simulation + + +def provenance_and_award(simulation): + benunit = simulation.populations["benunit"] + provenance = [] + for name in OVERRIDE_VARIABLES: + supplied = supplied_input(benunit, name, period(2026)) + provenance.append( + ( + stored_as_input(benunit, name, period(2026)), + None if supplied is None else supplied.tolist(), + ) + ) + award = simulation.calculate("housing_benefit_entitlement", 2026).tolist() + return provenance, award + + +@settings( + max_examples=5, + deadline=None, + derandomize=True, + suppress_health_check=[HealthCheck.too_slow], +) +@given(donor_state=SIMULATION_STATES, receiver_state=SIMULATION_STATES) +# The other simulation cached income before a scenario set its disregard. +@example( + donor_state=(("calculated", 868), ("model", 0), ("supplied", 0)), + receiver_state=(("absent", 0),) * 3, +) +# This simulation calculated income before taking supplied income. +@example( + donor_state=(("supplied", 2_589), ("absent", 0), ("absent", 0)), + receiver_state=(("calculated", 0), ("absent", 0), ("absent", 0)), +) +def test_rebuilt_populations_answer_as_the_simulation_that_made_them( + donor_state, receiver_state +): + """R1: whatever this simulation held before, populations taken from + another simulation give the provenance and award of a fresh copy of that + simulation, in this simulation and in its clones and branches.""" + expected = provenance_and_award(simulation_in(donor_state)) + simulation = simulation_in(receiver_state) + simulation.build_from_populations(simulation_in(donor_state).populations) + copies = [simulation, simulation.clone(), simulation.get_branch("after_rebuild")] + for copied in copies: + assert provenance_and_award(copied) == expected + + def test_replaced_income_formula_is_not_bypassed(): class housing_benefit_applicable_income(Variable): entity = BenUnit diff --git a/policyengine_uk/tests/test_input_records.py b/policyengine_uk/tests/test_input_records.py index e80fa327dc..4bb646b3c4 100644 --- a/policyengine_uk/tests/test_input_records.py +++ b/policyengine_uk/tests/test_input_records.py @@ -5,12 +5,16 @@ populations keeps the record of inputs the rebuild drops (policyengine-core#605). ``Simulation.delete_arrays`` forgets such a record, so a value the engine later carries into that period is not exported as an -input. +input. ``Simulation.build_from_populations`` keeps only the records of values +the new populations hold, and takes the records of populations from another +simulation along with them. """ import pytest +from policyengine_core.periods import period from policyengine_uk import Simulation +from policyengine_uk.utils.supplied_inputs import supplied_input pytestmark = pytest.mark.usefixtures("cloned_uk_tax_benefit_system") @@ -30,3 +34,58 @@ def test_deleting_after_a_rebuild_does_not_export_a_carried_over_input(): # The engine carries 2025's hours into 2026; they are not an input. assert sim.calculate("hours_worked", 2026).tolist() == [1_000] assert sim.to_input_dict()["hours_worked"] == {"2025": [1_000]} + + +def test_rebuilt_populations_export_their_own_inputs(): + sim = Simulation(situation=situation({2025: 500})) + sim.build_from_populations( + Simulation(situation=situation({2026: 1_000})).populations + ) + assert sim.to_input_dict()["hours_worked"] == {"2026": [1_000]} + + +def test_rebuilt_populations_record_an_input_set_later(): + # The holder comes from another simulation; the input is set here. + sim = Simulation(situation=situation({2025: 500})) + sim.build_from_populations( + Simulation(situation=situation({2025: 1_000})).populations + ) + sim.set_input("hours_worked", 2026, [700]) + assert sim.to_input_dict()["hours_worked"] == {"2025": [1_000], "2026": [700]} + assert sim.clone().to_input_dict()["hours_worked"] == { + "2025": [1_000], + "2026": [700], + } + + +def income_situation(adjusted_net_income=None): + person = {"age": {2026: 40}, "employment_income": {2026: 30_000}} + if adjusted_net_income is not None: + person["adjusted_net_income"] = {2026: adjusted_net_income} + return { + "people": {"person": person}, + "benunits": {"benunit": {"members": ["person"]}}, + "households": {"household": {"members": ["person"]}}, + } + + +def test_rebuilt_populations_keep_an_input_supplied_to_their_simulation(): + sim = Simulation(situation=income_situation()) + donor = Simulation(situation=income_situation(50_000)) + sim.build_from_populations(donor.populations) + person = sim.populations["person"] + assert supplied_input(person, "adjusted_net_income", period(2026)).tolist() == [ + 50_000 + ] + # Net income reads the supplied adjusted net income, as the variable does. + assert sim.calculate("net_income", 2026).tolist() == [50_000] + + +def test_rebuilt_populations_keep_a_value_their_simulation_calculated(): + sim = Simulation(situation=income_situation(50_000)) + donor = Simulation(situation=income_situation()) + donor.calculate("adjusted_net_income", 2026) + sim.build_from_populations(donor.populations) + person = sim.populations["person"] + assert supplied_input(person, "adjusted_net_income", period(2026)) is None + assert sim.calculate("net_income", 2026).tolist() == [30_000] diff --git a/policyengine_uk/utils/supplied_inputs.py b/policyengine_uk/utils/supplied_inputs.py index 1eb4719056..00c9c3a361 100644 --- a/policyengine_uk/utils/supplied_inputs.py +++ b/policyengine_uk/utils/supplied_inputs.py @@ -10,23 +10,26 @@ clone gets its own copy of the record. The helpers also require a stored value, so missing arrays do not count as supplied inputs. -``build_from_situation`` installs fresh populations, and so fresh holders, -through ``Simulation.build_from_populations``, which keeps the record +``Simulation.build_from_populations`` installs new populations, and so new +holders, but core keeps the record of the values they replace (policyengine-core#605). The engine could then carry an earlier value into a period whose input the rebuild dropped, and the stale record would pass it off -as supplied. So ``Simulation.calculate`` forgets the records of a variable in -``SUPPLIED_INPUT_VARIABLES`` whose stored value is gone before the engine can -refill it, and the helpers only answer for those variables. -``Simulation.delete_arrays`` forgets such records too, for any variable it -deletes. +as supplied. So ``Simulation.build_from_populations`` keeps only the records of +values the new holders store, taking those of holders from another simulation +along with them (``adopt_population_records``). ``Simulation.calculate`` +forgets the records of a variable in ``SUPPLIED_INPUT_VARIABLES`` whose stored +value is gone before the engine can refill it, and the helpers only answer for +those variables. ``Simulation.delete_arrays`` forgets such records too, for any +variable it deletes. A scenario that stores a value the model calculated (``set_model_value``) records it as the model's, and the helpers do not return it. The record of such a value lasts only while it is the value stored: ``Simulation`` forgets it when ``set_input`` stores a value for its period, whatever the value (an input set for a longer or shorter period that stores nothing there leaves -it), when ``delete_arrays`` removes it, and when the populations are -rebuilt, and a clone or branch gets its own copy of the record. +it), when ``delete_arrays`` removes it, and when a rebuild installs +populations that do not hold it, and a clone or branch gets its own copy of +the record. """ from typing import List, Optional @@ -178,11 +181,13 @@ def stored_as_input(population, variable_name: str, period: Period) -> bool: model's own (``set_model_value``). A value the engine carried over from an earlier period, or cached after a calculation, is not. - The holder answers from the input record of the simulation it was built - in. Populations taken from another simulation (``build_from_populations``) - keep their own record, so a value the other simulation calculated stays - calculated, and one supplied there stays supplied, whatever this - simulation's record says.""" + The holder answers from the input record of the simulation bound to it, + which also tells a tier's input from a cache in the other tier. + ``Simulation.build_from_populations`` binds holders taken from another + simulation to this one with their records (``adopt_population_records``), + so a value the other simulation calculated stays calculated, and one + supplied there stays supplied, in this simulation and its clones and + branches.""" _check_registered(variable_name) holder = population.get_holder(variable_name) return any( @@ -244,9 +249,9 @@ def set_model_value(simulation, variable_name: str, period, value) -> None: ``supplied_input_periods`` do not return the stored value while it is the one stored here. A later ``Simulation.set_input`` for the period is supplied, whatever its value, and so is any value stored after - ``delete_arrays`` removes this one or the populations are rebuilt (see - ``forget_replaced_model_values``, ``drop_missing_model_values`` and - ``forget_model_values``). If the value was supplied before (a scenario + ``delete_arrays`` removes this one or a rebuild installs populations that + do not hold it (see ``forget_replaced_model_values``, + ``drop_missing_model_values`` and ``adopt_population_records``). If the value was supplied before (a scenario adjusting a user's entry), the new value stays supplied.""" _check_registered(variable_name) period = periods.period(period) @@ -344,10 +349,64 @@ def drop_missing_model_values(simulation, variable_name: str) -> None: del model_values[key] -def forget_model_values(simulation) -> None: - """Forget every model value: rebuilt populations hold none of them.""" - if getattr(simulation, "_model_values", None): - simulation._model_values = {} +def adopt_population_records(simulation) -> None: + """Bind each holder of the populations ``build_from_populations`` just + installed to ``simulation``, and keep its records of inputs and model + values for those holders' values only. + + Core points each population at the simulation, but leaves each holder + bound to the simulation that made it and keeps this simulation's record + of the values it replaced (policyengine-core#605). The other simulation's + records then answer for the holder, and a deletion changes them instead + of this simulation's. A clone or branch copies this simulation's records + and binds the holders it copies to itself (``Simulation.clone``, + ``Holder.clone``), so the records must move with the holders. A holder from another simulation brings each input and + model value that simulation recorded and the holder still stores, and + drops every other value: the other simulation may have calculated or + carried it over before an input changed, or under another tax-benefit + system, so this simulation calculates it again. A holder this simulation + made keeps this simulation's records. Records of values the new + populations do not hold go, as do the values. A holder from no + simulation, or from one with no input record (core's legacy mode, which + no ``Simulation`` constructor leaves), keeps its binding and values.""" + input_keys = getattr(simulation, "_user_input_keys", None) + own_model_values = getattr(simulation, "_model_values", None) or {} + kept_keys, model_values = set(), {} + for population in simulation.populations.values(): + for name, holder in population._holders.items(): + source = holder.simulation + if source is simulation: + kept_keys.update(key for key in input_keys or () if key[0] == name) + model_values.update( + (key, value) + for key, value in own_model_values.items() + if key[0] == name + ) + continue + source_keys = getattr(source, "_user_input_keys", None) + if source_keys is None: + continue + # Ask while the holder still reads its source's record. + kept_keys.update( + key + for key in source_keys + if key[0] == name and holder._is_input(key[2], key[1]) + ) + source_model_values = getattr(source, "_model_values", None) or {} + model_values.update( + (key, value) + for key, value in source_model_values.items() + if key[0] == name + and _is_model_value( + source, key, holder._get_array_from_storage(key[2], key[1]) + ) + ) + holder._drop_computed() + holder.simulation = simulation + if input_keys is not None: + # A clone may share the record with its original, so replace it. + simulation._user_input_keys = kept_keys + simulation._model_values = model_values def copy_model_values(simulation, clone) -> None: diff --git a/policyengine_uk/variables/gov/dwp/housing_benefit/applicable_income/housing_benefit_specified_or_temporary_accommodation_disregard.py b/policyengine_uk/variables/gov/dwp/housing_benefit/applicable_income/housing_benefit_specified_or_temporary_accommodation_disregard.py index 214dcc52bb..7f29a3b4cc 100644 --- a/policyengine_uk/variables/gov/dwp/housing_benefit/applicable_income/housing_benefit_specified_or_temporary_accommodation_disregard.py +++ b/policyengine_uk/variables/gov/dwp/housing_benefit/applicable_income/housing_benefit_specified_or_temporary_accommodation_disregard.py @@ -15,11 +15,12 @@ def hb_uses_annual_override(benunit, period, variable_name, default_formula): or formula.__code__ != default_formula.__code__ ): return True - # A value set as an input for this period overrides recalculation, and so + # An input the caller set for this period overrides recalculation, and so # does a scenario's own value (set_model_value), which the variable then # reads; a cached formula result does not, even in populations taken from - # another simulation. SUPPLIED_INPUT_VARIABLES lists each name passed - # here, as stored_as_input and set_model_value require. + # another simulation or in their clones and branches. + # SUPPLIED_INPUT_VARIABLES lists each name passed here, as stored_as_input + # and set_model_value require. return stored_as_input(benunit, variable_name, period) From 371c220938ffc9b831dc33255a124a1b8ed290fd Mon Sep 17 00:00:00 2001 From: Max Ghenis Date: Thu, 29 Oct 2026 13:35:29 -0400 Subject: [PATCH 09/12] Copy another simulation's populations at a rebuild instead of sharing them MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 6e090ed00 rebound each holder taken from another simulation to the rebuilt one. The two simulations still shared those holders, so the rebinding left the other simulation with holders bound to the rebuilt one. The sixth review found three failures: - A holder from a simulation that stores values on disk kept its disk setting. The rebuilt simulation has no memory configuration, so the next input it stored crashed in core's Holder._set. - The input-record filter asked only the first storage tier. A cache in memory hid an input on disk, so the rebuilt simulation lost the input: £10,322.21 instead of £8,775 with a supplied £3,000. - An input set on the other simulation went into the rebuilt simulation's record. Neither simulation noted the change, so both kept a stale weekly_hours of 10 instead of 20. Simulation.build_from_populations now takes a copy of each population and holder another simulation made, as Simulation.clone copies them (copy_other_populations in utils/supplied_inputs.py), and leaves the other simulation's populations, values and records as they were. Neither simulation's later changes reach the other. Populations no simulation has taken yet, the only ones the model's own builders pass, go in as before. Before copying, the helper reads what the other simulation recorded of each holder: the inputs that survive in either storage tier, and the model values (set_model_value) still stored. keep_population_records then keeps those records and this simulation's own records for the holders it made, drops the other values the copies hold, and gives each copy the storage settings this simulation gives its own holders. New tests in test_input_records.py cover: - a later input on populations from a simulation that stores on disk; - storage on disk when this simulation stores there; - an input on disk under a cache in memory; - the other simulation keeping its own values and inputs. In the Housing Benefit tests, two simulations rebuilt from the same populations now answer independently. The R1 property's docstring now names the states it generates. Fixes #2747 Co-Authored-By: Claude Opus 5.5 --- policyengine_uk/simulation.py | 8 +- ...est_housing_benefit_accommodation_dates.py | 26 ++- policyengine_uk/tests/test_input_records.py | 62 +++++- policyengine_uk/utils/supplied_inputs.py | 208 +++++++++++++----- 4 files changed, 238 insertions(+), 66 deletions(-) diff --git a/policyengine_uk/simulation.py b/policyengine_uk/simulation.py index c3df87de02..b9692e87bf 100644 --- a/policyengine_uk/simulation.py +++ b/policyengine_uk/simulation.py @@ -83,11 +83,12 @@ ) from policyengine_uk.utils.supplied_inputs import ( SUPPLIED_INPUT_VARIABLES, - adopt_population_records, copy_model_values, + copy_other_populations, drop_missing_model_values, drop_missing_supplied_inputs, forget_replaced_model_values, + keep_population_records, model_value_input_records, ) from policyengine_uk.reforms import create_structural_reforms_from_parameters @@ -591,6 +592,9 @@ def delete_arrays(self, variable: str, period: Period = None) -> None: del records[recorded_period] def build_from_populations(self, populations) -> None: + # Take a copy of what another simulation made, so that neither changes + # the other's values or records. + populations, adopted = copy_other_populations(self, populations) super().build_from_populations(populations) # Formulas read the fast cache, so drop what it holds for the replaced # populations, as _start_new_population does. @@ -599,7 +603,7 @@ def build_from_populations(self, populations) -> None: # Keep the records of inputs and model values (set_model_value) of # the values the new populations hold, and only those: an equal value # a rebuild stores later (from a situation or a frame) is supplied. - adopt_population_records(self) + keep_population_records(self, adopted) def set_input(self, variable_name: str, period: Period, value) -> None: # What is set here is supplied, even a value equal to the model's own, diff --git a/policyengine_uk/tests/test_housing_benefit_accommodation_dates.py b/policyengine_uk/tests/test_housing_benefit_accommodation_dates.py index d29b7bcd29..b21f051e5d 100644 --- a/policyengine_uk/tests/test_housing_benefit_accommodation_dates.py +++ b/policyengine_uk/tests/test_housing_benefit_accommodation_dates.py @@ -296,6 +296,24 @@ def test_copies_of_rebuilt_populations_keep_a_scenario_adjusted_income(copy): ) +def test_simulations_rebuilt_from_the_same_populations_answer_alone(): + data = situation() + data["benunits"]["family"]["housing_benefit_applicable_income"] = {2026: 3_000} + donor = Simulation(situation=data) + first = Simulation(situation=situation()) + second = Simulation(situation=situation()) + first.build_from_populations(donor.populations) + second.build_from_populations(donor.populations) + award = pytest.approx(10_400 - 0.65 * (3_000 - 500), abs=0.01) + assert second.calculate("housing_benefit_entitlement", 2026)[0] == award + first.set_input("housing_benefit_applicable_income", 2026, [4_000]) + assert first.calculate("housing_benefit_entitlement", 2026)[0] == pytest.approx( + 10_400 - 0.65 * (4_000 - 500), abs=0.01 + ) + assert second.calculate("housing_benefit_entitlement", 2026)[0] == award + assert donor.calculate("housing_benefit_entitlement", 2026)[0] == award + + def test_rebuilt_populations_replace_values_calculated_before(): # Formulas read the fast cache, which held the income calculated here. simulation = Simulation(situation=situation()) @@ -374,9 +392,11 @@ def provenance_and_award(simulation): def test_rebuilt_populations_answer_as_the_simulation_that_made_them( donor_state, receiver_state ): - """R1: whatever this simulation held before, populations taken from - another simulation give the provenance and award of a fresh copy of that - simulation, in this simulation and in its clones and branches.""" + """R1: for the states generated here (each override variable absent, + supplied, set by a scenario or calculated for 2026, on the default + branch), whichever of them this simulation held before, populations taken + from another simulation give the provenance and award of a fresh copy of + that simulation, in this simulation and in its clones and branches.""" expected = provenance_and_award(simulation_in(donor_state)) simulation = simulation_in(receiver_state) simulation.build_from_populations(simulation_in(donor_state).populations) diff --git a/policyengine_uk/tests/test_input_records.py b/policyengine_uk/tests/test_input_records.py index 4bb646b3c4..ac51552cea 100644 --- a/policyengine_uk/tests/test_input_records.py +++ b/policyengine_uk/tests/test_input_records.py @@ -6,11 +6,14 @@ (policyengine-core#605). ``Simulation.delete_arrays`` forgets such a record, so a value the engine later carries into that period is not exported as an input. ``Simulation.build_from_populations`` keeps only the records of values -the new populations hold, and takes the records of populations from another -simulation along with them. +the new populations hold. It copies populations from another simulation, with +that simulation's records of the inputs they hold, so the two simulations do +not share holders. """ +import numpy as np import pytest +from policyengine_core.experimental.memory_config import MemoryConfig from policyengine_core.periods import period from policyengine_uk import Simulation @@ -89,3 +92,58 @@ def test_rebuilt_populations_keep_a_value_their_simulation_calculated(): person = sim.populations["person"] assert supplied_input(person, "adjusted_net_income", period(2026)) is None assert sim.calculate("net_income", 2026).tolist() == [30_000] + + +def disk_backed(hours): + # With no memory to spare, a simulation stores its values on disk. + sim = Simulation(situation=situation(hours)) + sim.memory_config = MemoryConfig(max_memory_occupation=0) + sim.build_from_situation(situation(hours)) + return sim + + +def test_rebuilt_populations_take_inputs_from_a_simulation_storing_on_disk(): + # This simulation has no memory configuration. + sim = Simulation(situation=situation({2025: 500})) + sim.build_from_populations(disk_backed({2025: 1_000}).populations) + sim.set_input("hours_worked", 2026, [700]) + assert sim.calculate("hours_worked", 2026).tolist() == [700] + assert sim.to_input_dict()["hours_worked"] == {"2025": [1_000], "2026": [700]} + + +def test_rebuilt_populations_store_values_where_this_simulation_does(): + sim = disk_backed({2025: 500}) + sim.build_from_populations( + Simulation(situation=situation({2025: 1_000})).populations + ) + sim.set_input("hours_worked", 2026, [700]) + disk = sim.get_holder("hours_worked")._disk_storage + assert disk.get(period(2026)).tolist() == [700] + + +def test_rebuilt_populations_keep_an_input_a_cache_hides(): + # A value cached in memory hides the input stored on disk. + donor = disk_backed({2025: 1_000}) + donor.memory_config = MemoryConfig(max_memory_occupation=1) + donor.get_holder("hours_worked").put_in_cache(np.array([999.0]), period(2025)) + sim = Simulation(situation=situation({2025: 500})) + sim.build_from_populations(donor.populations) + assert sim.to_input_dict()["hours_worked"] == {"2025": [1_000]} + assert sim.calculate("hours_worked", 2025).tolist() == [1_000] + + +def test_rebuilt_populations_leave_their_simulation_as_it_was(): + donor = Simulation(situation=situation({2025: 520})) + sim = Simulation(situation=situation({2025: 100})) + sim.build_from_populations(donor.populations) + assert donor.calculate("weekly_hours", 2025).tolist() == [10] + assert sim.calculate("weekly_hours", 2025).tolist() == [10] + # Each recalculates from its own inputs; neither's changes reach the other. + donor.set_input("hours_worked", 2025, [1_040]) + assert donor.calculate("weekly_hours", 2025).tolist() == [20] + assert sim.calculate("weekly_hours", 2025).tolist() == [10] + sim.set_input("hours_worked", 2025, [260]) + assert sim.calculate("weekly_hours", 2025).tolist() == [5] + assert donor.calculate("weekly_hours", 2025).tolist() == [20] + assert donor.to_input_dict()["hours_worked"] == {"2025": [1_040]} + assert sim.to_input_dict()["hours_worked"] == {"2025": [260]} diff --git a/policyengine_uk/utils/supplied_inputs.py b/policyengine_uk/utils/supplied_inputs.py index 00c9c3a361..e1fac3e782 100644 --- a/policyengine_uk/utils/supplied_inputs.py +++ b/policyengine_uk/utils/supplied_inputs.py @@ -15,8 +15,9 @@ (policyengine-core#605). The engine could then carry an earlier value into a period whose input the rebuild dropped, and the stale record would pass it off as supplied. So ``Simulation.build_from_populations`` keeps only the records of -values the new holders store, taking those of holders from another simulation -along with them (``adopt_population_records``). ``Simulation.calculate`` +values the new holders store. It copies populations from another simulation, +with that simulation's records of the inputs they hold, rather than share them +(``copy_other_populations``, ``keep_population_records``). ``Simulation.calculate`` forgets the records of a variable in ``SUPPLIED_INPUT_VARIABLES`` whose stored value is gone before the engine can refill it, and the helpers only answer for those variables. ``Simulation.delete_arrays`` forgets such records too, for any @@ -36,6 +37,7 @@ import numpy as np from policyengine_core import periods +from policyengine_core.holders.helpers import _is_input as _stored_as_input_in from policyengine_core.periods import Period @@ -183,8 +185,8 @@ def stored_as_input(population, variable_name: str, period: Period) -> bool: The holder answers from the input record of the simulation bound to it, which also tells a tier's input from a cache in the other tier. - ``Simulation.build_from_populations`` binds holders taken from another - simulation to this one with their records (``adopt_population_records``), + ``Simulation.build_from_populations`` copies holders from another + simulation into this one with their records (``copy_other_populations``), so a value the other simulation calculated stays calculated, and one supplied there stays supplied, in this simulation and its clones and branches.""" @@ -251,8 +253,9 @@ def set_model_value(simulation, variable_name: str, period, value) -> None: supplied, whatever its value, and so is any value stored after ``delete_arrays`` removes this one or a rebuild installs populations that do not hold it (see ``forget_replaced_model_values``, - ``drop_missing_model_values`` and ``adopt_population_records``). If the value was supplied before (a scenario - adjusting a user's entry), the new value stays supplied.""" + ``drop_missing_model_values`` and ``keep_population_records``). If the + value was supplied before (a scenario adjusting a user's entry), the new + value stays supplied.""" _check_registered(variable_name) period = periods.period(period) population = simulation.get_variable_population(variable_name) @@ -349,64 +352,151 @@ def drop_missing_model_values(simulation, variable_name: str) -> None: del model_values[key] -def adopt_population_records(simulation) -> None: - """Bind each holder of the populations ``build_from_populations`` just - installed to ``simulation``, and keep its records of inputs and model - values for those holders' values only. - - Core points each population at the simulation, but leaves each holder - bound to the simulation that made it and keeps this simulation's record - of the values it replaced (policyengine-core#605). The other simulation's - records then answer for the holder, and a deletion changes them instead - of this simulation's. A clone or branch copies this simulation's records - and binds the holders it copies to itself (``Simulation.clone``, - ``Holder.clone``), so the records must move with the holders. A holder from another simulation brings each input and - model value that simulation recorded and the holder still stores, and - drops every other value: the other simulation may have calculated or - carried it over before an input changed, or under another tax-benefit - system, so this simulation calculates it again. A holder this simulation - made keeps this simulation's records. Records of values the new - populations do not hold go, as do the values. A holder from no - simulation, or from one with no input record (core's legacy mode, which - no ``Simulation`` constructor leaves), keeps its binding and values.""" +def _made_elsewhere(simulation, owner) -> bool: + return owner is not None and owner is not simulation + + +def _records_of(holder): + """What the simulation bound to ``holder`` records of the values it holds: + the input keys whose input survives in either storage tier, the model + values (``set_model_value``) still stored, and whether that simulation + keeps no input record (core's legacy mode, which no ``Simulation`` + constructor leaves, where every stored value counts as an input).""" + name = holder.variable.name + source = holder.simulation + source_keys = getattr(source, "_user_input_keys", None) + if source_keys is None: + keys = { + (name, branch_name, holder._storage_period(period)) + for branch_name, period in holder.get_known_branch_periods() + } + return keys, {}, True + # Ask each tier: a cache in memory can hide an input on disk. + keys = { + key + for key in source_keys + if key[0] == name + and any( + _stored_as_input_in(holder, key[2], key[1], tier) + for tier in ("memory", "disk") + ) + } + source_model_values = getattr(source, "_model_values", None) or {} + model_values = { + key: value + for key, value in source_model_values.items() + if key[0] == name + and _is_model_value(source, key, holder._get_array_from_storage(key[2], key[1])) + } + return keys, model_values, False + + +def copy_other_populations(simulation, populations): + """Return the populations ``simulation`` builds from: ``populations``, + with a copy bound to ``simulation`` of each population and holder another + simulation made. Also return, for each copied holder, what that + simulation recorded of the inputs and model values it holds. + + Core installs the populations it is given and points each at the new + simulation, but leaves each holder bound to the simulation that made it, + and keeps the new simulation's record of the values they replace + (policyengine-core#605). Two live simulations would then share holders, + and the holder's records and storage settings would be those of one of + them: an input either stores goes into that one's record, and the other + neither records it nor recalculates what depends on it. So each + simulation gets its own copy, as ``Simulation.clone`` makes, and the + other simulation keeps its populations, values and records. A copy holds + what the original held when the copy was made, and later changes to + either do not reach the other. Populations no simulation has taken yet + (from ``instantiate_entities``) are installed as they are.""" + person_key = simulation.tax_benefit_system.person_entity.key + copy_all = _made_elsewhere(simulation, populations[person_key].simulation) + copied = { + key + for key, population in populations.items() + if copy_all or _made_elsewhere(simulation, population.simulation) + } + # Read each record while the holder is bound to the simulation that + # made it, before copying. + records = { + (key, name): _records_of(holder) + for key, population in populations.items() + for name, holder in population._holders.items() + if key in copied or _made_elsewhere(simulation, holder.simulation) + } + result = dict(populations) + if person_key in copied: + result[person_key] = populations[person_key].clone( + simulation, share_arrays=True + ) + for key in copied - {person_key}: + result[key] = populations[key].clone( + simulation, result[person_key], share_arrays=True + ) + adopted = [] + for (key, name), record in records.items(): + population = result[key] + holder = population._holders[name] + if key not in copied: + holder = holder.clone(population, share_arrays=True) + population._holders[name] = holder + holder.simulation = simulation + adopted.append((holder, *record)) + return result, adopted + + +def _match_storage(holder, simulation) -> None: + """Give a copied holder the storage settings ``simulation`` gives the + holders it makes (``Holder.__init__``), keeping what it stores: a holder + copied from a simulation with another ``memory_config`` would otherwise + read this simulation's missing one when it stores a value.""" + config = getattr(simulation, "memory_config", None) + name = holder.variable.name + on_disk = bool(config) and name not in config.priority_variables + if on_disk and holder._disk_storage is None: + holder._disk_storage = holder.create_disk_storage() + holder._on_disk_storable = on_disk + holder._do_not_store = bool(config) and name in config.variables_to_drop + + +def keep_population_records(simulation, adopted) -> None: + """Keep ``simulation``'s records of inputs and model values for the values + its populations hold, and only those, after ``copy_other_populations``. + + A copied holder brings each input and model value the simulation that + made it recorded and it still stores, and drops every other value: that + simulation may have calculated or carried it over before an input + changed, or under another tax-benefit system, so this simulation + calculates it again. A holder this simulation made keeps this + simulation's records. Records of values the new populations do not hold + go, as do the values. A holder copied from a simulation with no input + record (core's legacy mode, where every stored value counts as an input) + keeps every value it stores, with a record of each.""" + adopted_ids = {id(holder) for holder, *_ in adopted} + own_names = { + name + for population in simulation.populations.values() + for name, holder in population._holders.items() + if holder.simulation is simulation and id(holder) not in adopted_ids + } input_keys = getattr(simulation, "_user_input_keys", None) own_model_values = getattr(simulation, "_model_values", None) or {} - kept_keys, model_values = set(), {} - for population in simulation.populations.values(): - for name, holder in population._holders.items(): - source = holder.simulation - if source is simulation: - kept_keys.update(key for key in input_keys or () if key[0] == name) - model_values.update( - (key, value) - for key, value in own_model_values.items() - if key[0] == name - ) - continue - source_keys = getattr(source, "_user_input_keys", None) - if source_keys is None: - continue - # Ask while the holder still reads its source's record. - kept_keys.update( - key - for key in source_keys - if key[0] == name and holder._is_input(key[2], key[1]) - ) - source_model_values = getattr(source, "_model_values", None) or {} - model_values.update( - (key, value) - for key, value in source_model_values.items() - if key[0] == name - and _is_model_value( - source, key, holder._get_array_from_storage(key[2], key[1]) - ) - ) - holder._drop_computed() - holder.simulation = simulation + kept_keys = {key for key in input_keys or () if key[0] in own_names} + model_values = { + key: value for key, value in own_model_values.items() if key[0] in own_names + } + for holder, keys, values, legacy in adopted: + kept_keys.update(keys) + model_values.update(values) + # Install new containers rather than edit the old ones, so that nothing + # still holding an old one sees the change. if input_keys is not None: - # A clone may share the record with its original, so replace it. simulation._user_input_keys = kept_keys simulation._model_values = model_values + for holder, keys, values, legacy in adopted: + if not legacy: + holder._drop_computed() + _match_storage(holder, simulation) def copy_model_values(simulation, clone) -> None: From 3cfab0639ccdc998148f4b81747a33aa96897efc Mon Sep 17 00:00:00 2001 From: Max Ghenis Date: Thu, 29 Oct 2026 15:13:29 -0400 Subject: [PATCH 10/12] Test that rebuilt populations forget the inputs they replace The copy at 371c22093 keeps this simulation's input records only for the holders it made, but no test failed when it kept them all: the readers that also check storage hid the stale record. supplied_input does not check storage marks, so a value cached in the replaced period read back as the replaced input. The new test sets a cache of 123 for an adjusted net income the rebuild replaced and checks that supplied_input returns None. It fails on main (daec84d5b), where supplied_input returns the cache, and passes at 6e090ed00 and 371c22093. The keep_replaced_records mutation, which survived the hub's mutation run, now fails it. Co-Authored-By: Claude Opus 5.5 --- policyengine_uk/tests/test_input_records.py | 14 ++++++++++++++ 1 file changed, 14 insertions(+) diff --git a/policyengine_uk/tests/test_input_records.py b/policyengine_uk/tests/test_input_records.py index ac51552cea..e98b4a5fbe 100644 --- a/policyengine_uk/tests/test_input_records.py +++ b/policyengine_uk/tests/test_input_records.py @@ -147,3 +147,17 @@ def test_rebuilt_populations_leave_their_simulation_as_it_was(): assert donor.calculate("weekly_hours", 2025).tolist() == [20] assert donor.to_input_dict()["hours_worked"] == {"2025": [1_040]} assert sim.to_input_dict()["hours_worked"] == {"2025": [260]} + + +def test_rebuilt_populations_forget_the_inputs_they_replace(): + data = situation({2025: 500}) + data["people"]["person"]["adjusted_net_income"] = {2026: 50_000} + sim = Simulation(situation=data) + sim.build_from_populations( + Simulation(situation=situation({2025: 1_000})).populations + ) + # A value cached in the period is not the input the rebuild replaced. + holder = sim.get_holder("adjusted_net_income") + holder.put_in_cache(np.array([123.0]), period(2026)) + person = sim.populations["person"] + assert supplied_input(person, "adjusted_net_income", period(2026)) is None From 8cac7e91f95cd7274a5c4c7654fb88d6e8491fb5 Mon Sep 17 00:00:00 2001 From: Max Ghenis Date: Thu, 29 Oct 2026 22:17:17 -0400 Subject: [PATCH 11/12] Keep a model value's record when a cache hides it in the other tier MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The seventh review found that copying another simulation's holders dropped a model value (set_model_value) stored on disk under a value cached in memory. _records_of compared the recorded model value with what storage returned first, the cache, so the copy kept the input key but not the model marker. _drop_computed then removed the cache, and the disk value read back as a supplied input. In the review's example, adjusted net income of £50,000 set by the model on disk under a cache of 999 gave a supplied_input of 50,000 and net income of £50,000 in the rebuilt simulation, its clone and its branch, against the original simulation's None and £30,000. _records_of now compares each model value with the stored input, read through core's get_stored_input, which asks each tier whether it holds the input. The review's script now gives None and £30,000 in all three copies, and the aggregate-guard snapshot keeps the model origin. test_rebuilt_populations_keep_a_model_value_a_cache_hides pins the case. It fails at 3cfab0639 and passes here. The copy_other_populations docstring now says that the copy shares each in-memory array with the original until it first reads it, so writing in place into the original's array before then changes what the copy reads. Co-Authored-By: Claude Opus 5.5 --- changelog.d/rebuilt-population-records.fixed.md | 2 +- policyengine_uk/tests/test_input_records.py | 17 ++++++++++++++++- policyengine_uk/utils/supplied_inputs.py | 15 +++++++++++---- 3 files changed, 28 insertions(+), 6 deletions(-) diff --git a/changelog.d/rebuilt-population-records.fixed.md b/changelog.d/rebuilt-population-records.fixed.md index d1baf414e1..21c4dc18d8 100644 --- a/changelog.d/rebuilt-population-records.fixed.md +++ b/changelog.d/rebuilt-population-records.fixed.md @@ -1 +1 @@ -- Fixed `Simulation.build_from_populations` with another simulation's populations: the simulation, its clones and its branches now treat the values that simulation supplied as supplied, record inputs set afterwards, and calculate again the values that simulation calculated or this one cached for the replaced populations. +- Fixed `Simulation.build_from_populations` with another simulation's populations: the simulation, its clones and its branches now treat the values that simulation supplied as supplied and the values the model set itself as the model's, record inputs set afterwards, and calculate again the values that simulation calculated or this one cached for the replaced populations. diff --git a/policyengine_uk/tests/test_input_records.py b/policyengine_uk/tests/test_input_records.py index e98b4a5fbe..a29f1d61fb 100644 --- a/policyengine_uk/tests/test_input_records.py +++ b/policyengine_uk/tests/test_input_records.py @@ -17,7 +17,7 @@ from policyengine_core.periods import period from policyengine_uk import Simulation -from policyengine_uk.utils.supplied_inputs import supplied_input +from policyengine_uk.utils.supplied_inputs import set_model_value, supplied_input pytestmark = pytest.mark.usefixtures("cloned_uk_tax_benefit_system") @@ -132,6 +132,21 @@ def test_rebuilt_populations_keep_an_input_a_cache_hides(): assert sim.calculate("hours_worked", 2025).tolist() == [1_000] +def test_rebuilt_populations_keep_a_model_value_a_cache_hides(): + # The model's own value, stored on disk, under a value cached in memory. + donor = disk_backed({2025: 1_000}) + set_model_value(donor, "adjusted_net_income", period(2026), [50_000]) + donor.memory_config = MemoryConfig(max_memory_occupation=1) + holder = donor.get_holder("adjusted_net_income") + holder.put_in_cache(np.array([999.0]), period(2026)) + sim = Simulation(situation=situation({2025: 500})) + sim.build_from_populations(donor.populations) + for copy in (sim, sim.clone(), sim.get_branch("copy")): + assert copy.calculate("adjusted_net_income", 2026).tolist() == [50_000] + person = copy.populations["person"] + assert supplied_input(person, "adjusted_net_income", period(2026)) is None + + def test_rebuilt_populations_leave_their_simulation_as_it_was(): donor = Simulation(situation=situation({2025: 520})) sim = Simulation(situation=situation({2025: 100})) diff --git a/policyengine_uk/utils/supplied_inputs.py b/policyengine_uk/utils/supplied_inputs.py index e1fac3e782..c6f81f512b 100644 --- a/policyengine_uk/utils/supplied_inputs.py +++ b/policyengine_uk/utils/supplied_inputs.py @@ -38,6 +38,7 @@ import numpy as np from policyengine_core import periods from policyengine_core.holders.helpers import _is_input as _stored_as_input_in +from policyengine_core.holders.helpers import get_stored_input from policyengine_core.periods import Period @@ -381,12 +382,14 @@ def _records_of(holder): for tier in ("memory", "disk") ) } + # Compare each model value with the stored input, not with what storage + # returns first, which can be that cache. source_model_values = getattr(source, "_model_values", None) or {} model_values = { key: value for key, value in source_model_values.items() if key[0] == name - and _is_model_value(source, key, holder._get_array_from_storage(key[2], key[1])) + and _is_model_value(source, key, get_stored_input(holder, key[2], key[1])) } return keys, model_values, False @@ -406,9 +409,13 @@ def copy_other_populations(simulation, populations): neither records it nor recalculates what depends on it. So each simulation gets its own copy, as ``Simulation.clone`` makes, and the other simulation keeps its populations, values and records. A copy holds - what the original held when the copy was made, and later changes to - either do not reach the other. Populations no simulation has taken yet - (from ``instantiate_entities``) are installed as they are.""" + what the original held when the copy was made, and inputs either stores + later do not reach the other. The copy shares each in-memory array with + the original until it first reads it (``share_arrays``, see core's + ``InMemoryStorage.clone``), so writing in place into the original's + array before then changes what the copy reads. Populations no simulation + has taken yet (from ``instantiate_entities``) are installed as they + are.""" person_key = simulation.tax_benefit_system.person_entity.key copy_all = _made_elsewhere(simulation, populations[person_key].simulation) copied = { From dd79d3fb5dfcd7300be865486d91825e651c6266 Mon Sep 17 00:00:00 2001 From: Max Ghenis Date: Thu, 29 Oct 2026 23:09:29 -0400 Subject: [PATCH 12/12] Start the changelog fragments with their sentences Towncrier adds each entry's bullet itself, so a fragment that starts with '- ' renders as '- - ...' in CHANGELOG.md. The Changelog fragments check warned on both at 8cac7e91f, and the eighth review asked for the fix. Co-Authored-By: Claude Opus 5.5 --- changelog.d/hb-accommodation-dates.fixed.md | 2 +- changelog.d/rebuilt-population-records.fixed.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/changelog.d/hb-accommodation-dates.fixed.md b/changelog.d/hb-accommodation-dates.fixed.md index dc4921837d..925209cb6c 100644 --- a/changelog.d/hb-accommodation-dates.fixed.md +++ b/changelog.d/hb-accommodation-dates.fixed.md @@ -1 +1 @@ -- Applied Housing Benefit accommodation earnings disregards from their statutory commencement within the first fiscal year, calculating awards before combining date segments. +Applied Housing Benefit accommodation earnings disregards from their statutory commencement within the first fiscal year, calculating awards before combining date segments. diff --git a/changelog.d/rebuilt-population-records.fixed.md b/changelog.d/rebuilt-population-records.fixed.md index 21c4dc18d8..7d7858e665 100644 --- a/changelog.d/rebuilt-population-records.fixed.md +++ b/changelog.d/rebuilt-population-records.fixed.md @@ -1 +1 @@ -- Fixed `Simulation.build_from_populations` with another simulation's populations: the simulation, its clones and its branches now treat the values that simulation supplied as supplied and the values the model set itself as the model's, record inputs set afterwards, and calculate again the values that simulation calculated or this one cached for the replaced populations. +Fixed `Simulation.build_from_populations` with another simulation's populations: the simulation, its clones and its branches now treat the values that simulation supplied as supplied and the values the model set itself as the model's, record inputs set afterwards, and calculate again the values that simulation calculated or this one cached for the replaced populations.