Reset negative HBAI household income to zero before housing costs, as HBAI does

changelog.d/hbai-negative-income-reset.fixed.md

@@ -0,0 +1 @@
+- HBAI household net income before housing costs now resets a negative figure to zero, as HBAI does; income after housing costs, deciles, poverty flags and poverty gaps follow from the reset figure. Poverty headcounts do not change while both person-weighted BHC and AHC median incomes are positive (as on the enhanced FRS); poverty gaps fall and income totals rise by the sum of the negative incomes. Households whose income is reset now count in the bottom income decile instead of being left out with decile -1, as long as households at zero hold at most a tenth of people, so bottom-decile figures move. The figure before the reset is the new `hbai_household_net_income_before_reset_to_zero`, which the labour supply responses use, so they are unchanged.

docs/book/validation/hbai.md

@@ -6,7 +6,8 @@ PolicyEngine UK includes variables that match to the Households Below Average In
 
 These are:
 
-- `hbai_household_net_income`: Household net income, before housing costs
+- `hbai_household_net_income`: Household net income, before housing costs, with a negative figure reset to zero
+- `hbai_household_net_income_before_reset_to_zero`: The same income before a negative figure is reset to zero
 - `hbai_household_net_income_ahc`: Household net income, after housing costs
 - `equiv_hbai_household_net_income`: Equivalised household net income, before housing costs (BHC)
 - `equiv_hbai_household_net_income_ahc`: Equivalised household net income, after housing costs (AHC)
@@ -44,7 +45,12 @@ The table below shows how the HBAI income components map to PolicyEngine variabl
 | Structural insurance premiums (for owner occupiers) | `structural_insurance_payments` |
 | Ground rent and service charges |`housing_service_charges`|
 
-`hbai_household_net_income_ahc` deducts all five housing-cost rows above from `hbai_household_net_income`. Housing costs can exceed income, leaving negative AHC income. The equivalised AHC income and AHC poverty measures use that result; household income deciles use BHC income.
+## Negative incomes
+
+HBAI resets a negative income before housing costs to zero: "Negative incomes BHC are reset to zero, but negative AHC incomes calculated from the adjusted BHC incomes are possible. Where incomes have been adjusted to zero BHC, income AHC is derived from the adjusted BHC income." ([HBAI Background Information and Methodology report FYE 2025](https://www.gov.uk/government/statistics/households-below-average-income-for-financial-years-ending-1995-to-2025/households-below-average-income-background-information-and-methodology-report-fye-2025#negative-incomes)).
+
+PolicyEngine does the same. `hbai_household_net_income` is `hbai_household_net_income_before_reset_to_zero` reset to zero when negative, and `hbai_household_net_income_ahc` subtracts all five housing costs listed above from the reset figure, so it can be negative. The equivalised incomes, `household_income_decile`, the poverty flags and the poverty gaps all use the reset figure. A household whose income was negative is below every absolute poverty line, and its before-housing-costs poverty gap is the whole line. It is below the relative line while the median person's income is positive; if half or more of people had BHC income at or below zero, the reset would make the BHC median, and so the BHC relative line, zero. Households with equal incomes share the rank at the top of their group, so households at zero are in the bottom decile while they hold at most a tenth of people. On the enhanced FRS under 1% of people have an income at or below zero before the reset, so both conditions hold. The labour supply responses use the figure before the reset, which is the household's actual budget constraint.
+`hbai_household_net_income_ahc` deducts all five housing-cost rows above from the reset `hbai_household_net_income`. Ground rent and service charges (`housing_service_charges`) are the fifth cost. Housing costs can exceed income, leaving negative AHC income. The equivalised AHC income and AHC poverty measures use that result; household income deciles use BHC income.
 
 In FRS-based datasets, `housing_service_charges` contains the annualised property charges recorded in `CHRGAMT1`–`CHRGAMT9`: ground rent, chief rent, service charges, regular maintenance, caravan site rent, factoring charges, other regular charges and combined charges. `CHRGAMT2`, historically feu duty, is unused in the FRS 2024–25 dictionary. The combined-charge question is an alternative to recording individual amounts. Water and sewerage and rent are separate inputs; calculator users should supply each cost once, with service charges additional to the other housing-cost inputs. See the [FRS 2024–25 question instructions](https://doc.ukdataservice.ac.uk/doc/9563/mrdoc/pdf/9563_frs_2024-25_question_instructions_final.pdf), property charges section (pages 98–99), and its [variable dictionaries](https://doc.ukdataservice.ac.uk/doc/9563/mrdoc/ukda_data_dictionaries.zip).
 

policyengine_uk/dynamics/labour_supply.py

@@ -96,7 +96,7 @@ class LabourSupplyResponseData(BaseModel):
 
 def apply_labour_supply_responses(
     sim: Simulation,
-    target_variable: str = "hbai_household_net_income",
+    target_variable: str = "hbai_household_net_income_before_reset_to_zero",
     input_variable: str = "employment_income",
     year: int = 2025,
     count_adults: int = 2,
@@ -110,7 +110,11 @@ def apply_labour_supply_responses(
 
     Args:
         sim: PolicyEngine simulation object (should have baseline attribute)
-        target_variable: Variable that drives labour supply decisions
+        target_variable: Variable that drives labour supply decisions. The
+            default is HBAI net income before HBAI resets a negative income
+            to zero: the household's actual budget constraint. The reset is
+            a statistical convention; measured on the reset figure, a pay
+            rise for a household below zero would look fully withdrawn.
         input_variable: Variable representing labour supply (typically employment_income)
         year: Year for calculation
         count_adults: Number of adults to calculate responses for
@@ -194,7 +198,7 @@ def apply_labour_supply_responses(
 
 def apply_progression_responses(
     sim: Simulation,
-    target_variable: str = "hbai_household_net_income",
+    target_variable: str = "hbai_household_net_income_before_reset_to_zero",
     input_variable: str = "employment_income",
     year: int = 2025,
     count_adults: int = 2,
@@ -208,7 +212,11 @@ def apply_progression_responses(
 
     Args:
         sim: PolicyEngine simulation object (should have baseline attribute)
-        target_variable: Variable that drives labour supply decisions
+        target_variable: Variable that drives labour supply decisions. The
+            default is HBAI net income before HBAI resets a negative income
+            to zero: the household's actual budget constraint. The reset is
+            a statistical convention; measured on the reset figure, a pay
+            rise for a household below zero would look fully withdrawn.
         input_variable: Variable representing labour supply (typically employment_income)
         year: Year for calculation
         count_adults: Number of adults to calculate responses for

policyengine_uk/dynamics/progression.py

@@ -16,7 +16,7 @@
 
 def calculate_derivative(
     sim: Simulation,
-    target_variable: str = "hbai_household_net_income",
+    target_variable: str = "hbai_household_net_income_before_reset_to_zero",
     input_variable: str = "employment_income",
     year: int = 2025,
     count_adults: int = 2,
@@ -30,7 +30,9 @@ def calculate_derivative(
 
     Args:
         sim: PolicyEngine simulation object
-        target_variable: Variable to measure changes in (typically household_net_income)
+        target_variable: Variable to measure changes in (by default HBAI net
+            income before a negative income is reset to zero, so that the
+            rate reflects the household's actual budget constraint)
         input_variable: Variable to change (typically employment_income)
         year: Year for calculation
         count_adults: Number of adults to apply changes to
@@ -89,7 +91,7 @@ def calculate_derivative(
 
 def calculate_relative_income_change(
     sim: Simulation,
-    target_variable: str = "hbai_household_net_income",
+    target_variable: str = "hbai_household_net_income_before_reset_to_zero",
     year: int = 2025,
 ) -> pd.DataFrame:
     """Calculate relative change in income between baseline and scenario.
@@ -135,7 +137,7 @@ def calculate_relative_income_change(
 
 def calculate_derivative_change(
     sim: Simulation,
-    target_variable: str = "hbai_household_net_income",
+    target_variable: str = "hbai_household_net_income_before_reset_to_zero",
     input_variable: str = "employment_income",
     year: int = 2025,
     count_adults: int = 2,

policyengine_uk/tests/policy/baseline/household/income/hbai_negative_income.yaml

@@ -0,0 +1,191 @@
+# HBAI resets a negative household income before housing costs to zero and
+# derives income after housing costs from the reset figure, which can be
+# negative: "Negative incomes BHC are reset to zero, but negative AHC incomes
+# calculated from the adjusted BHC incomes are possible. Where incomes have
+# been adjusted to zero BHC, income AHC is derived from the adjusted BHC
+# income." (HBAI Background Information and Methodology report FYE 2025,
+# "Negative incomes").
+#
+# The cases use 2024 (FYE 2025), where the absolute poverty lines are HBAI's
+# published FYE 2025 lines: for a single adult 15,040.08 a year before housing
+# costs and 11,276.52 after (see poverty/hbai_family_type_lines.yaml).
+# Universal Credit is not claimed, so the household's only income is what the
+# case gives it.
+
+- name: Deductions above income leave a negative figure that is reset to zero
+  period: 2024
+  absolute_error_margin: 0.1
+  input:
+    people:
+      adult:
+        age: 40
+        maintenance_expenses: 5_000
+    benunits:
+      benunit:
+        members: [adult]
+        would_claim_uc: false
+    households:
+      household:
+        members: [adult]
+        council_tax: 1_500
+        rent: 6_000
+        housing_service_charges: 300
+        tenure_type: RENT_PRIVATELY
+  output:
+    hbai_household_net_income_before_reset_to_zero: -6_500
+    hbai_household_net_income: 0
+    real_hbai_household_net_income: 0
+    equiv_hbai_household_net_income: 0
+    # Income after housing costs is reset BHC less rent and service charges.
+    hbai_household_net_income_ahc: -6_300
+    # Divided by the 0.58 AHC equivalence factor for a single adult.
+    equiv_hbai_household_net_income_ahc: -10_862.07
+    # The before-housing-costs gap is the whole line, never more.
+    poverty_gap_bhc: 15_040.08
+    poverty_gap: 15_040.08
+    deep_poverty_gap: 7_520.04
+    # The after-housing-costs gap is the line plus the negative income.
+    poverty_gap_ahc: 17_576.52
+    in_poverty_bhc: true
+    in_poverty_ahc: true
+    in_deep_poverty_bhc: true
+    in_deep_poverty_ahc: true
+    in_poverty: true
+    in_deep_poverty: true
+
+- name: An owner-occupier's housing costs come off the reset income
+  period: 2024
+  absolute_error_margin: 0.1
+  input:
+    people:
+      adult:
+        age: 40
+        maintenance_expenses: 3_000
+    benunits:
+      benunit:
+        members: [adult]
+        would_claim_uc: false
+    households:
+      household:
+        members: [adult]
+        council_tax: 1_000
+        tenure_type: OWNED_WITH_MORTGAGE
+        mortgage_interest_repayment: 2_000
+        water_and_sewerage_charges: 400
+        structural_insurance_payments: 100
+        housing_service_charges: 600
+  output:
+    hbai_household_net_income_before_reset_to_zero: -4_000
+    hbai_household_net_income: 0
+    hbai_household_net_income_ahc: -3_100
+
+- name: A negative figure given directly is reset; housing costs still come off
+  period: 2024
+  absolute_error_margin: 0.1
+  input:
+    people:
+      adult:
+        age: 40
+    benunits:
+      benunit:
+        members: [adult]
+    households:
+      household:
+        members: [adult]
+        hbai_household_net_income_before_reset_to_zero: -25_000
+        rent: 4_800
+        housing_service_charges: 200
+        tenure_type: RENT_PRIVATELY
+  output:
+    hbai_household_net_income: 0
+    hbai_household_net_income_ahc: -5_000
+    poverty_gap_bhc: 15_040.08
+    poverty_gap_ahc: 16_276.52
+
+- name: A figure of exactly zero stays zero
+  period: 2024
+  absolute_error_margin: 0.1
+  input:
+    people:
+      adult:
+        age: 40
+    benunits:
+      benunit:
+        members: [adult]
+    households:
+      household:
+        members: [adult]
+        hbai_household_net_income_before_reset_to_zero: 0
+  output:
+    hbai_household_net_income: 0
+    hbai_household_net_income_ahc: 0
+
+- name: A positive income is unchanged by the reset
+  period: 2024
+  absolute_error_margin: 0.1
+  input:
+    people:
+      adult:
+        age: 40
+    benunits:
+      benunit:
+        members: [adult]
+    households:
+      household:
+        members: [adult]
+        hbai_household_net_income_before_reset_to_zero: 12_345
+        rent: 5_000
+        housing_service_charges: 450
+        tenure_type: RENT_PRIVATELY
+  output:
+    hbai_household_net_income: 12_345
+    hbai_household_net_income_ahc: 6_895
+    # 12,345 is below the 15,040.08 line: a gap of 2,695.08.
+    poverty_gap_bhc: 2_695.08
+
+- name: Earnings above deductions give the same income with or without the reset
+  period: 2024
+  absolute_error_margin: 0.1
+  input:
+    people:
+      adult:
+        age: 40
+        employment_income: 20_000
+    benunits:
+      benunit:
+        members: [adult]
+        would_claim_uc: false
+    households:
+      household:
+        members: [adult]
+        council_tax: 1_500
+  output:
+    # 20,000 less Income Tax of 20% x (20,000 - 12,570) = 1,486, Class 1 NICs
+    # of 8% x (20,000 - 12,570) = 594.40 and Council Tax of 1,500.
+    hbai_household_net_income_before_reset_to_zero: 16_419.60
+    hbai_household_net_income: 16_419.60
+
+- name: With Council Tax abolished, the reset applies to income without it
+  period: 2024
+  absolute_error_margin: 0.1
+  input:
+    gov.contrib.abolish_council_tax: true
+    people:
+      adult:
+        age: 40
+        maintenance_expenses: 3_000
+    benunits:
+      benunit:
+        members: [adult]
+        would_claim_uc: false
+    households:
+      household:
+        members: [adult]
+        council_tax: 1_500
+        rent: 2_000
+        tenure_type: RENT_PRIVATELY
+  output:
+    # Maintenance paid only: the reform drops Council Tax from the deductions.
+    hbai_household_net_income_before_reset_to_zero: -3_000
+    hbai_household_net_income: 0
+    hbai_household_net_income_ahc: -2_000

policyengine_uk/tests/test_hbai_housing_costs_properties.py

@@ -1,8 +1,10 @@
 """HBAI AHC deducts every housing cost once, including service charges.
 
-BHC is supplied directly to isolate the housing-cost contract from the
-calculation and zero floor of income before housing costs. Each vector
-population contains a household with zero BHC and positive service charges.
+The generated properties supply BHC directly to isolate the housing-cost
+contract from the calculation and zero floor of income before housing costs.
+Each vector population contains a household with zero BHC and positive service
+charges. A separate differential case computes BHC to ensure service charges
+are deducted only after housing costs.
 """
 
 import numpy as np
@@ -154,3 +156,57 @@ def test_extra_service_charges_reduce_ahc_by_the_same_amount(
     np.testing.assert_allclose(
         before_ahc - after_ahc, extra_input, rtol=1e-6, atol=0.03
     )
+
+
+@pytest.mark.parametrize("year", [2024, 2025, 2026])
+def test_service_charges_do_not_change_computed_bhc(year):
+    # Positive computed BHC exposes a misplaced BHC deduction that the zero
+    # floor could hide. Neither BHC variable is supplied as an input.
+    simulation = Simulation(
+        situation={
+            "people": {
+                "person": {
+                    "age": {year: 40},
+                    "employment_income": {year: 60_000},
+                }
+            },
+            "benunits": {
+                "benunit": {
+                    "members": ["person"],
+                    "would_claim_uc": {year: False},
+                }
+            },
+            "households": {
+                "household": {
+                    "members": ["person"],
+                    "tenure_type": {year: "OWNED_OUTRIGHT"},
+                    **{
+                        name: {year: 1_200 if name == "housing_service_charges" else 0}
+                        for name in HOUSING_COSTS
+                    },
+                }
+            },
+        }
+    )
+    bhc_variables = (
+        "hbai_household_net_income_before_reset_to_zero",
+        "hbai_household_net_income",
+    )
+    before_bhc = {
+        name: np.asarray(simulation.calculate(name, year)).copy()
+        for name in bhc_variables
+    }
+    before_ahc = np.asarray(
+        simulation.calculate("hbai_household_net_income_ahc", year)
+    ).copy()
+    assert all(np.all(income > 2_400) for income in before_bhc.values())
+
+    simulation.set_input("housing_service_charges", year, [2_400])
+    simulation.reset_calculations()
+
+    for name in bhc_variables:
+        np.testing.assert_array_equal(
+            simulation.calculate(name, year), before_bhc[name]
+        )
+    after_ahc = np.asarray(simulation.calculate("hbai_household_net_income_ahc", year))
+    np.testing.assert_allclose(before_ahc - after_ahc, 1_200, rtol=0, atol=0.01)