Lock policyengine-uk 2.145.0, follow its bus fares and bind thin target rows

changelog.d/1095-uk-bind-thin-target-rows.changed.md

@@ -0,0 +1 @@
+A UK target row is no longer left out of the calibration because few source households carry it (María's rule of 2026-10-21): the support floor now drops only rows that no household carries, the benefit-cap amount bands join its scope, and the per-row exclusions that rested on thin support alone retire (the benefit-cap amount bands and the HMRC £500k to £1m dividend and property-income-count rows). A bound national, regional or BRMA row that fewer than ten source households carry keeps its model figures in the build, as BRMA rows' do (constituency and local-authority rows keep the area-support rule): the diagnostics, the national gate evidence, the certifier, the national role and the Orrery summary withhold them, the national fit gate decides on the published rows and counts the withheld ones, and no run forwards an epoch's loss. A published row that, with the other published rows, would give back a combination of withheld rows resting on fewer than ten source households is withheld beside them (the next-smallest band, not the total), so no difference of published figures isolates fewer than ten cases. The A/B that retired the thirteen entries is in `experiments/1095-uk-thin-row-binding-receipts.md`.

changelog.d/1331-uk-bus-fare-receipts-engine-window.changed.md

@@ -0,0 +1 @@
+The 2025 England and London bus fare receipt rows now move from 2024-25 on policyengine-uk's own window (María's ruling of 2026-10-21, replacing the calendar-year basis of 2026-09-10): each area's BUS0415a quarter-end index is day-weighted over 6 April to 5 April, as the engine reads `gov.dft.bus.local_bus_fares_index` (PolicyEngine/policyengine-uk#2546), and England's receipts move as London's plus England outside London's. The bus parity receipt ties each area's factor to the engine's index ratio, and the engine test holds the tie within 0.1% at the locked engine (microcosm#1331).

changelog.d/1331-uk-bus-single-fare-share-cap-years.changed.md

@@ -0,0 +1 @@
+The NTS bus stage reads the single-ticket share from the years under England's £2 fare cap (2023 and 2024), as nine months of the FRS 2024-25 base year are (its last three sit under the £3 cap, which the extract does not observe), and its ticket receipt adds fare-mix lines for England outside London: the share of point-of-use boardings above £2 before the cap and their mean fare, and the share paid at £2 under it, each withheld below 10 boardings or 10 households. The receipt now weights boardings by the full trip weight, as the stage's share does (microcosm#1331).

changelog.d/uk-policyengine-uk-2-145.changed.md

@@ -0,0 +1 @@
+The UK lock moves from policyengine-uk 2.134.4 to 2.145.0, the release that carries pe-uk#2536 and pe-uk#2546 (policyengine-core stays at 3.33.3): the uprating pins' stamp, the concept coverage (302 to 311 inputs), the coverage manifest, the H2 fixture, the cited versions and `APPROVED_UV_LOCK_SHA256` re-derive, and the concept mapping's capital reason follows 2.136.1's complete-override rule. The build leaves `defined_benefit_pension_share`, the pre-2015 salary-sacrifice input, the PIP points inputs and `local_bus_uncapped_single_fare` at their defaults, and the stage texts or the PR say why.

changelog.d/uk-uc-capital-sentinel-receipt.changed.md

@@ -0,0 +1 @@
+The UC capital stage follows policyengine-uk 2.136.1, which values property apart and shares it across a household's claimants and partners: its texts and engine tests say so. A benefit unit left on the unavailable sentinel beside a unit that records the household's property, for UC or Pension Credit, is the one case the engine counts twice: the stage's receipt counts such units, and the UC capital gate now refuses any. The stage's texts also note that both recorded figures project to later years by GDP per head, while from 2.139.1 the engine projects other residential property by house prices.

changelog.d/uk-war-widows-pension-split.fixed.md

@@ -0,0 +1 @@
+The FRS spine writes War Widow's or Widower's Pension (FRS benefit code 9) to policyengine-uk's own `war_widows_pension_reported` input, added in 2.135.7 (pe-uk#2382), instead of adding it to `bsp_reported` (#1222): the pension rises with CPI, while Bereavement Support Payment's rates are frozen. An SPI-channel row keeps its FRS twin's value of the new column, as it does for `bsp_reported`, and the column follows `bsp_reported` through the graph, the SPI stages, the benefit coherence gate and the H2 fixture.

docs/evidence/uk-thin-rows-1095/scripts/ab_binding.py

@@ -0,0 +1,168 @@
+"""A/B of the thin-row binding on one spine (microcosm#1095 U3, #1333).
+
+Usage: ab_binding.py <control-run-dir> <treatment-run-dir> <spine-h5>
+
+Both runs calibrate the same spine; the treatment binds the rows that María's rule of
+2026-10-21 brought back (thin support alone leaves no row out). Every figure is recomputed
+from each run's own target-support sidecars (the compiled matrix, the targets, the design and
+final weights), so a row whose published figures are withheld is still measured here.
+
+Prints, for each run: the share of rows within 10%, the mean absolute relative error and the
+weight anatomy (Kish ESS, the share of design mass whose weight was stretched 9.5 times or
+more, and the max over median of the positive weights), all over the rows bound in BOTH runs;
+the shared rows that cross the 10% line either way; and, per family of newly bound rows, the
+fit of each row in both runs (the control's weights read through the treatment's matrix row),
+its carriers' support band and how far the binding stretched their weights.
+
+Its output describes single rows and small groups of licensed records, so it stays with the
+licensed spine, outside the repository; experiments/1095-uk-thin-row-binding-receipts.md
+gives what may be published.
+"""
+
+from __future__ import annotations
+
+import sys
+from collections import defaultdict
+from pathlib import Path
+
+import numpy as np
+import pandas as pd
+from scipy import sparse
+
+STRETCH = 9.5
+
+
+def band(n: int) -> str:
+    return "<10" if n < 10 else ("10-29" if n < 30 else "30+")
+
+
+def family(name: str) -> str:
+    if name.startswith("dwp.benefit_cap."):
+        return "benefit-cap amount bands"
+    if "uc_payment_dist" in name or "payment_distribution" in name:
+        return "UC payment bands"
+    if name.startswith("hmrc/"):
+        return "HMRC 500k-1m cells"
+    return "other"
+
+
+def load(run: Path) -> dict:
+    matrix = sparse.csr_matrix(sparse.load_npz(run / "target_support_matrix.npz"))
+    vectors = np.load(run / "target_support_vectors.npz", allow_pickle=True)
+    names = [str(n) for n in vectors["names"]]
+    return {
+        "matrix": matrix,
+        "names": names,
+        "index": {n: i for i, n in enumerate(names)},
+        "target": np.asarray(vectors["target_vector"], dtype=float),
+        "design": np.asarray(vectors["design_weights"], dtype=float),
+        "final": np.asarray(vectors["final_weights"], dtype=float),
+        "ids": np.asarray(vectors["household_ids"])
+        if "household_ids" in vectors
+        else None,
+    }
+
+
+def errors(run: dict, weights: np.ndarray) -> np.ndarray:
+    estimate = run["matrix"] @ weights
+    with np.errstate(divide="ignore", invalid="ignore"):
+        return (estimate - run["target"]) / run["target"]
+
+
+def anatomy(run: dict) -> dict:
+    final, design = run["final"], run["design"]
+    positive = final > 0
+    ratio = np.divide(final, design, out=np.zeros_like(final), where=design > 0)
+    return {
+        "ess": float(final.sum() ** 2 / np.square(final).sum()),
+        "stretched_mass": float(design[ratio >= STRETCH].sum() / design.sum()),
+        "max_over_median": float(final[positive].max() / np.median(final[positive])),
+    }
+
+
+def main() -> None:
+    control, treatment = load(Path(sys.argv[1])), load(Path(sys.argv[2]))
+    if control["ids"] is not None and treatment["ids"] is not None:
+        assert np.array_equal(control["ids"], treatment["ids"]), (
+            "the runs order households differently"
+        )
+    with pd.HDFStore(sys.argv[3], mode="r") as store:
+        household = store.select(
+            "household", columns=["household_id", "source_household_id"]
+        )
+    source = (
+        household.set_index("household_id")["source_household_id"]
+        .reindex(treatment["ids"])
+        .to_numpy()
+        if treatment["ids"] is not None
+        else None
+    )
+    shared = sorted(set(control["names"]) & set(treatment["names"]))
+    added = sorted(set(treatment["names"]) - set(control["names"]))
+    dropped = sorted(set(control["names"]) - set(treatment["names"]))
+    for label, run in (("control", control), ("treatment", treatment)):
+        rel = errors(run, run["final"])
+        picked = np.asarray([rel[run["index"][n]] for n in shared])
+        a = anatomy(run)
+        print(
+            f"== {label}: {len(run['names'])} rows; over the {len(shared)} shared rows "
+            f"{np.mean(np.abs(picked) <= 0.10):.4f} within 10%, mean |error| {np.nanmean(np.abs(picked)):.4f}; "
+            f"ESS {a['ess']:.0f}; design mass stretched {STRETCH}x+ {a['stretched_mass']:.4f}; "
+            f"LOCAL max/median {a['max_over_median']:.1f}"
+        )
+    print(
+        f"rows bound in the treatment only: {len(added)}; in the control only: {dropped}"
+    )
+    rel_c = errors(control, control["final"])
+    rel_t = errors(treatment, treatment["final"])
+    out_of, into = [], []
+    for name in shared:
+        c, t = rel_c[control["index"][name]], rel_t[treatment["index"][name]]
+        if abs(c) <= 0.10 < abs(t):
+            out_of.append(name)
+        if abs(t) <= 0.10 < abs(c):
+            into.append(name)
+    print(f"shared rows leaving 10% under the binding: {len(out_of)} {out_of}")
+    print(f"shared rows entering 10% under the binding: {len(into)} {into}")
+    # Newly bound rows: the control's weights read through the treatment's matrix row.
+    by_family: dict[str, list[str]] = defaultdict(list)
+    for name in added:
+        by_family[family(name)].append(name)
+    ratio_c = np.divide(
+        control["final"],
+        control["design"],
+        out=np.zeros_like(control["final"]),
+        where=control["design"] > 0,
+    )
+    ratio_t = np.divide(
+        treatment["final"],
+        treatment["design"],
+        out=np.zeros_like(treatment["final"]),
+        where=treatment["design"] > 0,
+    )
+    for fam, names in sorted(by_family.items()):
+        print(f"-- {fam}: {len(names)} rows (LOCAL ONLY below)")
+        carriers_all = np.zeros(len(treatment["final"]), dtype=bool)
+        for name in names:
+            row = treatment["matrix"][treatment["index"][name]]
+            cols = row.indices[row.data != 0]
+            carriers_all[cols] = True
+            target = treatment["target"][treatment["index"][name]]
+            est_c = float(np.ravel(row @ control["final"])[0])
+            est_t = float(np.ravel(row @ treatment["final"])[0])
+            support = band(len(pd.unique(source[cols]))) if source is not None else "?"
+            print(
+                f"   {name}: support {support}; error unbound {est_c / target - 1:+.3f} -> bound {est_t / target - 1:+.3f}; "
+                f"carriers stretched {STRETCH}x+ {np.mean(ratio_c[cols] >= STRETCH) if len(cols) else 0:.2f} -> {np.mean(ratio_t[cols] >= STRETCH) if len(cols) else 0:.2f}"
+            )
+        design = treatment["design"]
+        mass = design[carriers_all].sum()
+        print(
+            f"   family carriers' design mass stretched {STRETCH}x+: control "
+            f"{design[carriers_all & (ratio_c >= STRETCH)].sum() / mass if mass else 0:.3f}, treatment "
+            f"{design[carriers_all & (ratio_t >= STRETCH)].sum() / mass if mass else 0:.3f}"
+        )
+
+
+if __name__ == "__main__":
+    main()

experiments/1095-uk-thin-row-binding-receipts.md

@@ -0,0 +1,46 @@
+# microcosm#1095 U3: binding thin target rows, receipts for #1333
+
+This receipt gives no row's model figure, and no count or weighted figure that rests on fewer than ten source households. Figures from the SPI channel stay out of it. Publisher figures appear as published.
+
+## The question
+
+María's rule of 2026-10-21: a target row is not left out because few source households carry it. Only a measured distortion of the calibration keeps it out. Thirteen per-row entries in `uk/calibration_measure_exclusions.json` rested on thin support, and #1333 retires them:
+
+- the eleven benefit-cap amount bands, held out after #882: on its arms the bands sat far outside their bound, and the solve moved weight onto cloned households;
+- the HMRC £500k to £1m dividend and property-income-count cells, held out after #807: fitting them distorted neighbouring cells.
+
+The support floor now drops only the rows that no source household carries, which no weights can fit. For each retired entry the question is whether binding it distorts the calibration now.
+
+## Method
+
+- One spine, built on #1333's head before its second review (policyengine-uk 2.145.0's release commit, whose package files the PyPI wheel matches), was calibrated twice with `UK_REVIEW_DATE=2026-10-21`:
+  - as pushed (the treatment);
+  - with the thin-row commit reverted (the control), so the two runs differ only in the binding.
+- Both runs pass all seven terminal gates.
+- `docs/evidence/uk-thin-rows-1095/scripts/ab_binding.py` recomputes fit and weight anatomy from each run's own target-support sidecars (`target_support_matrix.npz`, `target_support_vectors.npz`). So it measures a withheld row too. Its output stays with the licensed spine, outside the repository.
+- A newly bound row is read through the treatment's compiled row, at each run's weights.
+
+## Results
+
+- **Rows.** The control binds 1,216 rows and the treatment 1,244. The 28 new rows are 16 UC payment-distribution bands, 10 benefit-cap amount bands and the two HMRC cells. Of these, 22 rest on fewer than ten source households, so their figures are withheld. The floor drops the bands that no household carries.
+- **Shared rows.** Over the 1,216 rows bound in both runs, 99.0% are within 10% of their target in the control and 98.9% in the treatment. The mean absolute relative error is 0.6% in both. One shared row moves just past 10%. It is SPI-derived, so this receipt does not name it. No shared row moves the other way.
+- **Weights.** The effective sample size is about 6,900 in both runs. The share of design weight stretched 9.5 times or more is 0.8% in both. The weight-ratio gate passes in both.
+- **New rows.** Each newly bound row whose figures are published reaches its target. In the two DWP families, binding does not raise the share of the carriers' design weight that is stretched 9.5 times or more. The HMRC cells' carriers are SPI-derived, so their figures stay out of this receipt.
+
+No retired entry goes back.
+
+## Rows given back by difference (Vahid's second review, item 10)
+
+A withheld row's figure can come back as a family total less its published bands, or through any other linear difference of published figures. The build now withholds a published row beside the thin ones while such a difference isolates fewer than ten source households (`uk_withheld_row_sets` in `uk_runtime/diagnostics.py`).
+
+On the treatment run's compiled rows and weights:
+
+- before the fix, the published rows gave back seven of the 38 withheld rows exactly, each on its own;
+- with the fix, the build withholds eight more published rows. Each is a row that one of those differences needs, chosen with the fewest carriers. Most are the next band beside a thin one;
+- an independent check then finds no combination of withheld rows that the published rows give back and that rests on fewer than ten source households. The combinations that are still given back, such as the withheld bands of a family together, each rest on ten or more.
+
+This receipt names none of these rows. Naming a row withheld beside another would show that its neighbour is thin.
+
+## To repeat on a later spine
+
+Calibrate the spine twice, with and without the thin rows bound, then run the script on the two runs and the spine. Compare the shared rows' share within 10%, the effective sample size and the stretched share with the figures above.