Derive property_wealth from its uprated components instead of saving the WAS total

changelog.d/property-wealth-from-components.fixed.md

@@ -0,0 +1 @@
+Derive worldwide gross property wealth from six uprated inputs covering all seven WAS terms; second homes and buy-to-let use the combined residential value and secured debt from #501. Keep the WAS total only for conditioning, remove inherited saved totals, and retain overseas property and other real estate separately. Requires a companion model stacked on policyengine-uk#1974 and #1969 for complete wealth, UK land and benefit-capital routing; domestic purchase proxies include buy-to-let through the combined residential input. Wealth, land, decile and reform outputs can change; calibration controls and objective terms are unchanged, but losses and fitted weights can change. Regional HPI still adjusts only main residence. Existing published datasets are not migrated.

docs/imputations.md

@@ -42,7 +42,7 @@ Imputes household wealth components using demographic and income predictors. The
 | Variable | WAS round 8 source | Description |
 |----------|--------------------|-------------|
 | `owned_land` | `DVLUKValR8_sum` | Value of land in the UK |
-| `property_wealth` | `DVPropertyR8` | Gross value of all property, main residence included |
+| `property_wealth` | `DVPropertyR8` | Gross value of all property; sequential conditioning draw only, never saved |
 | `private_pension_wealth` | `totalpenr8_aggr` − `dvvaldbt_scaper8_aggr` | Private pension wealth other than current-employment defined benefit rights: defined contribution pots, retained rights, pensions in payment |
 | `directly_held_shares` | `DVFShUKVR8_aggr` + `DVFESHARESR8_aggr` | UK shares and employee shares and options, held outside ISAs and pooled funds |
 | `unit_and_investment_trusts` | `DVFCollVR8_aggr` | Unit trusts and investment trusts (one survey question) |
@@ -53,6 +53,8 @@ Imputes household wealth components using demographic and income predictors. The
 | `main_residence_value` | `DVHValueR8` | Value of main home |
 | `other_residential_property_value` | `DVHseValR8_sum` + `DVBltValR8_sum` | Second homes and buy-to-let property |
 | `non_residential_property_value` | `DVBlDValR8_sum` | Other buildings, such as shops and garages |
+| `overseas_property_value` | `DVLOSValR8_sum` | Overseas land and property |
+| `other_property_value` | `DVOprValR8_sum` | Other real estate, unspecified geography/type |
 | `savings` | `DVSaValR8_aggr` | Savings account balances (cash ISAs excluded) |
 | `num_vehicles` | `vcarnr8` | Number of vehicles owned |
 | `student_loan_balance` | `Tot_LosR8_aggr` − `Tot_los_exc_SLCR8_aggr` | Student Loans Company balance, allocated to people |
@@ -61,7 +63,7 @@ Imputes household wealth components using demographic and income predictors. The
 | `non_residential_property_secured_debt` | `DVBldDebtR8_sum` | Mortgages and loans secured on other buildings |
 | `owned_land_secured_debt` | `DVLUKDebtR8_sum` | Loans secured on UK land |
 
-`corporate_wealth` is the exact sum of its three components on every household; private pension wealth is not part of it, because pension rights are disregarded capital in every means test. A secured debt is zero wherever the household holds none of the asset it is secured on. Land and property overseas (`DVLOSValR8_sum`), other real estate (`DVOPrValR8_sum`) and overseas shares (`DVFShOSVR8_aggr`) are not exported separately; `property_wealth` and the financial wealth totals include them.
+`corporate_wealth` is the exact sum of its three components on every household; private pension wealth is not part of it, because pension rights are disregarded capital in every means test. A secured debt is zero wherever the household holds none of the asset it is secured on. The seven gross WAS property terms are retained in six saved inputs: second homes and buy-to-let share one residential input. `property_wealth` remains a conditioning draw only; the companion model calculates the total from those six uprated inputs. Overseas shares (`DVFShOSVR8_aggr`) remain included only in the financial wealth totals.
 
 Each target draws its quantile with its own seed. microimpute otherwise uses one seed for every target, so a household drawn high for one asset is drawn high for every later target conditioned on it, which makes sparse targets such as the debt secured on land near-certain for the households that hold the asset.
 

policyengine_uk_data/datasets/imputations/regional_property_uprating.py

@@ -3,13 +3,11 @@
 
 The WAS QRF imputation compresses regional property values — London is
 undervalued by ~41% and the North East overvalued by ~39% relative to
-UK HPI Dec 2025.  This module rescales main_residence_value (and
-property_wealth proportionally) so that regional means match official
-average house prices, following the same pattern as
-``_calibrate_energy_to_need`` in ``consumption.py``.
+UK HPI Dec 2025.  This module rescales main_residence_value so that
+regional means match official average house prices, following the same
+pattern as ``_calibrate_energy_to_need`` in ``consumption.py``.
 """
 
-import numpy as np
 import pandas as pd
 from policyengine_uk.data import UKSingleYearDataset
 
@@ -32,8 +30,9 @@ def _calibrate_property_to_hpi(household: pd.DataFrame) -> pd.DataFrame:
     applies it multiplicatively.  This preserves within-region
     distributional shape while anchoring the level to admin data.
 
-    property_wealth is adjusted by the same factor so that the land-to-
-    property ratio stays consistent.
+    property_wealth is not stored: policyengine-uk sums it from
+    main_residence_value and the other property components, so it picks up
+    this rescaling without being scaled here.
     """
     hpi_prices = _load_regional_house_prices()
     region = household["region"].values
@@ -56,7 +55,6 @@ def _calibrate_property_to_hpi(household: pd.DataFrame) -> pd.DataFrame:
 
         factor = hpi_price / imputed_mean
         household.loc[owners_mask, "main_residence_value"] *= factor
-        household.loc[owners_mask, "property_wealth"] *= factor
 
     return household
 
@@ -77,7 +75,6 @@ def uprate_property_by_region(
     household = _calibrate_property_to_hpi(household)
 
     dataset.household["main_residence_value"] = household["main_residence_value"].values
-    dataset.household["property_wealth"] = household["property_wealth"].values
 
     dataset.validate()
     return dataset

policyengine_uk_data/datasets/imputations/wealth.py

@@ -68,6 +68,9 @@
     "other_residential_property_secured_debt",
     "non_residential_property_secured_debt",
     "owned_land_secured_debt",
+    # Append new outputs: retain #501's chain order and quantile seeds.
+    "overseas_property_value",
+    "other_property_value",
 ]
 
 # corporate_wealth is not imputed: it is the sum of its imputed components,
@@ -108,6 +111,22 @@
     "student_loan_balance": ("Tot_LosR8_aggr", "-Tot_los_exc_SLCR8_aggr"),
 }
 
+# Seven WAS source terms become six model inputs: #501 combines second
+# homes and buy-to-let. Keep direct UK land and overseas property distinct.
+PROPERTY_COMPONENTS = (
+    "main_residence_value",
+    "other_residential_property_value",
+    "non_residential_property_value",
+    "owned_land",
+    "overseas_property_value",
+    "other_property_value",
+)
+
+# Keep the source total as a sequential predictor, but let the companion
+# model sum the six uprated inputs, covering all seven WAS source terms.
+# Independent predictions need not reconcile to the conditioning draw.
+CONDITIONING_ONLY_VARIABLES = ("property_wealth",)
+
 WAS_RENAMES = {
     "R8xshhwgt": "household_weight",
     # Components for estimating land holdings.
@@ -143,7 +162,8 @@
     "DVPriRntR8": "is_renter",  # {1, 2} TODO: Get codebook values.
     "CTAmtR8": "council_tax",
     # Other columns for reference.
-    "DVLOSValR8_sum": "non_uk_land",
+    # Overseas land AND property (including dwellings/time-shares), not land only.
+    "DVLOSValR8_sum": "overseas_property_value",
     "HFINWNTR8_Sum": "net_financial_wealth",
     "HFINWR8_SUM": "gross_financial_wealth",
     "TotalWlthR8": "wealth",
@@ -156,6 +176,7 @@
     "DVBLtDebtR8_sum": "buy_to_let_debt",
     "DVBlDValR8_sum": "non_residential_property_value",
     "DVBldDebtR8_sum": "non_residential_property_secured_debt",
+    "DVOprValR8_sum": "other_property_value",
     "DVTotinc_bhcR8": "household_net_income",
     "DVSaValR8_aggr": "savings",
     "vcarnr8": "num_vehicles",
@@ -202,6 +223,29 @@ def generate_was_table(was: pd.DataFrame):
 
     was = was.rename(columns=RENAMES).fillna(0)[list(RENAMES.values())]
 
+    # WAS gross property fields use -9 (not applicable) and -8 (unknown/refused).
+    # Treat these as missing, like NaN. Gross asset values cannot be negative;
+    # reject undocumented codes without exposing any household records.
+    # Validate the seven raw terms before combining residential values: a
+    # missing/negative buy-to-let value must not reduce or hide in second homes.
+    gross_columns = [
+        "property_wealth",
+        "main_residence_value",
+        "second_homes_value",
+        "buy_to_let_value",
+        "non_residential_property_value",
+        "owned_land",
+        "overseas_property_value",
+        "other_property_value",
+    ]
+    was[gross_columns] = was[gross_columns].replace({-9: 0, -8: 0})
+    negative = was[gross_columns].lt(0).sum()
+    if negative.any():
+        raise ValueError(
+            "Unexpected negative WAS gross property values (household counts): "
+            + ", ".join(f"{name}={count}" for name, count in negative.items() if count)
+        )
+
     was["is_renting"] = was["is_renter"] == 1
 
     # Private pension wealth other than current-employment defined benefit
@@ -228,6 +272,7 @@ def generate_was_table(was: pd.DataFrame):
     "impute_variables": tuple(IMPUTE_VARIABLES),
     "derived_targets": tuple(DERIVED_TARGETS.items()),
     "quantile_draw_seeds": tuple(QUANTILE_DRAW_SEEDS.items()),
+    "property_mapping_version": 2,
 }
 
 
@@ -361,6 +406,135 @@ def use_independent_quantile_draws(model: QRF) -> QRF:
     return model
 
 
+def store_wealth_predictions(
+    dataset: UKSingleYearDataset, predictions: pd.DataFrame
+) -> UKSingleYearDataset:
+    """Write imputed wealth to the dataset, skipping conditioning-only outputs.
+
+    `student_loan_balance` is allocated to people; every other saved output is
+    a household column.
+    """
+    # Also remove an inherited saved total when augmenting an existing dataset.
+    dataset.household.drop(
+        columns=list(CONDITIONING_ONLY_VARIABLES), errors="ignore", inplace=True
+    )
+    for column in predictions.columns:
+        if column in CONDITIONING_ONLY_VARIABLES:
+            continue
+        if column == "student_loan_balance":
+            dataset.person[column] = _allocate_student_loan_balance_to_people(
+                household_balances=predictions[column].clip(lower=0),
+                person=dataset.person,
+            )
+            continue
+        dataset.household[column] = predictions[column].values
+    return dataset
+
+
+def require_property_component_model():
+    """Reject a model that would silently omit the newly retained categories.
+
+    The companion model changes, including the UK-only land formula, are
+    specified in docs/property-wealth-model-changes.md. They must land before
+    building a release with this mapping.
+    """
+    from policyengine_uk.system import system
+
+    new_components = {
+        "overseas_property_value",
+        "other_property_value",
+    }
+    missing = new_components - system.variables.keys()
+    if missing:
+        raise ValueError(
+            "Property imputation requires the companion policyengine-uk change "
+            "in docs/property-wealth-model-changes.md; missing variables: "
+            + ", ".join(sorted(missing))
+        )
+    property_adds = sorted(system.variables["property_wealth"].adds or ())
+    residential_adds = sorted(system.variables["residential_property_value"].adds or ())
+    if property_adds != sorted(
+        [
+            "residential_property_value",
+            "non_residential_property_value",
+            "owned_land",
+            "overseas_property_value",
+            "other_property_value",
+        ]
+    ) or residential_adds != sorted(
+        [
+            "main_residence_value",
+            "other_residential_property_value",
+        ]
+    ):
+        raise ValueError(
+            "policyengine-uk property aggregates must include all seven WAS "
+            "categories; apply docs/property-wealth-model-changes.md"
+        )
+    required_index = "gov.economic_assumptions.indices.obr.per_capita.gdp"
+    if any(
+        system.variables[name].uprating != required_index
+        for name in PROPERTY_COMPONENTS
+    ):
+        raise ValueError(
+            "Every WAS property component requires the GDP-per-capita model uprating index"
+        )
+
+    # The UK dataset loader materialises yearly inputs using this registry;
+    # variable attributes alone do not uprate those explicitly supplied arrays.
+    from pathlib import Path
+    import yaml
+    from policyengine_uk.data import economic_assumptions
+
+    with open(
+        Path(economic_assumptions.__file__).parent / "uprating_indices.yaml"
+    ) as registry_file:
+        registry = yaml.safe_load(registry_file)
+    registered = [name for names in registry.values() for name in names]
+    gdp_registered = registry.get(
+        "gov.economic_assumptions.yoy_growth.obr.per_capita.gdp", ()
+    )
+    if any(
+        registered.count(name) != 1 or name not in gdp_registered
+        for name in PROPERTY_COMPONENTS
+    ):
+        raise ValueError(
+            "Every WAS property component requires UK dataset uprating registration"
+        )
+
+    # Exercise the installed formula directly: adding the inputs and aggregate
+    # without changing land estimation would still double-count UK land and
+    # tax overseas assets. These are synthetic formula probes, not survey data.
+    from policyengine_core.periods import period
+    from policyengine_uk_data.datasets.frs_release import CURRENT_FRS_RELEASE
+
+    probe_period = period(CURRENT_FRS_RELEASE.base_year)
+    regions = system.variables["region"].possible_values
+    inputs = {
+        "region": regions.encode(np.array(["LONDON", "LONDON", "LONDON"])),
+        "property_wealth": np.array([100.0, 100.0, 150.0]),
+        "owned_land": np.array([100.0, 0.0, 20.0]),
+        "overseas_property_value": np.array([0.0, 100.0, 30.0]),
+        **{
+            name: np.zeros(3)
+            for name in PROPERTY_COMPONENTS
+            if name not in ("owned_land", "overseas_property_value")
+        },
+        "main_residence_value": np.array([0.0, 0.0, 100.0]),
+    }
+    formula = system.variables["household_land_value"].get_formula(probe_period)
+    land = formula(lambda name, _: inputs[name], probe_period, system.parameters)
+    regional = system.parameters(
+        probe_period
+    ).household.wealth.land.intensity.property_wealth_by_region
+    expected_land = [100.0, 0.0, 100.0 * regional.LONDON + 20.0]
+    if not np.allclose(land, expected_land):
+        raise ValueError(
+            "policyengine-uk land formula must count UK land once and exclude "
+            "overseas property; apply docs/property-wealth-model-changes.md"
+        )
+
+
 def save_imputation_models():
     """
     Train and save wealth imputation model.
@@ -422,6 +596,7 @@ def impute_wealth(dataset: UKSingleYearDataset) -> UKSingleYearDataset:
     """
     dataset = dataset.copy()
 
+    require_property_component_model()
     model = create_wealth_model()
     sim = Microsimulation(dataset=dataset)
     predictors = model.input_columns
@@ -432,16 +607,7 @@ def impute_wealth(dataset: UKSingleYearDataset) -> UKSingleYearDataset:
         "NORTHERN_IRELAND", "WALES"
     )  # WAS doesn't sample NI -> put NI households in Wales (closest aggregate)
     output_df = derive_wealth_outputs(model.predict(input_df))
-
-    for column in output_df.columns:
-        if column == "student_loan_balance":
-            dataset.person[column] = _allocate_student_loan_balance_to_people(
-                household_balances=output_df[column].clip(lower=0),
-                person=dataset.person,
-            )
-            continue
-        dataset.household[column] = output_df[column].values
-
+    dataset = store_wealth_predictions(dataset, output_df)
     dataset.validate()
 
     return dataset

policyengine_uk_data/storage/uprating_factors.csv

@@ -56,7 +56,9 @@ mortgage_interest_repayment,1.0,1.0,1.262,1.874,2.288,2.599,2.927,3.167,3.3,3.45
 net_financial_wealth,1.0,1.0,1.092,1.147,1.19,1.223,1.258,1.297,1.34,1.384,1.384,1.384,1.384,1.384,1.384
 non_residential_property_value,1.0,1.0,1.092,1.147,1.19,1.223,1.258,1.297,1.34,1.384,1.384,1.384,1.384,1.384,1.384
 other_investment_income,1.0,1.0,1.092,1.147,1.19,1.223,1.258,1.297,1.34,1.384,1.384,1.384,1.384,1.384,1.384
+other_property_value,1.0,1.0,1.092,1.147,1.19,1.223,1.258,1.297,1.34,1.384,1.384,1.384,1.384,1.384,1.384
 other_residential_property_value,1.0,1.0,1.092,1.147,1.19,1.223,1.258,1.297,1.34,1.384,1.384,1.384,1.384,1.384,1.384
+overseas_property_value,1.0,1.0,1.092,1.147,1.19,1.223,1.258,1.297,1.34,1.384,1.384,1.384,1.384,1.384,1.384
 owned_land,1.0,1.0,1.092,1.147,1.19,1.223,1.258,1.297,1.34,1.384,1.384,1.384,1.384,1.384,1.384
 pension_credit_reported,1.0,1.04,1.144,1.209,1.237,1.277,1.301,1.327,1.353,1.38,1.38,1.38,1.38,1.38,1.38
 pension_credit_reported_capital,1.0,1.0,1.102,1.161,1.204,1.256,1.293,1.335,1.376,1.417,1.462,1.508,1.555,1.604,1.655

policyengine_uk_data/storage/uprating_growth_factors.csv

@@ -56,7 +56,9 @@ mortgage_interest_repayment,0,0.0,0.262,0.485,0.221,0.136,0.126,0.082,0.042,0.04
 net_financial_wealth,0,0.0,0.092,0.05,0.037,0.028,0.029,0.031,0.033,0.033,0.0,0.0,0.0,0.0,0.0
 non_residential_property_value,0,0.0,0.092,0.05,0.037,0.028,0.029,0.031,0.033,0.033,0.0,0.0,0.0,0.0,0.0
 other_investment_income,0,0.0,0.092,0.05,0.037,0.028,0.029,0.031,0.033,0.033,0.0,0.0,0.0,0.0,0.0
+other_property_value,0,0.0,0.092,0.05,0.037,0.028,0.029,0.031,0.033,0.033,0.0,0.0,0.0,0.0,0.0
 other_residential_property_value,0,0.0,0.092,0.05,0.037,0.028,0.029,0.031,0.033,0.033,0.0,0.0,0.0,0.0,0.0
+overseas_property_value,0,0.0,0.092,0.05,0.037,0.028,0.029,0.031,0.033,0.033,0.0,0.0,0.0,0.0,0.0
 owned_land,0,0.0,0.092,0.05,0.037,0.028,0.029,0.031,0.033,0.033,0.0,0.0,0.0,0.0,0.0
 pension_credit_reported,0,0.04,0.1,0.057,0.023,0.032,0.019,0.02,0.02,0.02,0.0,0.0,0.0,0.0,0.0
 pension_credit_reported_capital,0,0.0,0.102,0.054,0.037,0.043,0.029,0.032,0.031,0.03,0.032,0.031,0.031,0.032,0.032