From ad482e62aeca0e57dc95a9fca23d53dcadc0aa6b Mon Sep 17 00:00:00 2001 From: Vahid Ahmadi Date: Wed, 27 May 2026 12:16:30 +0200 Subject: [PATCH 1/4] Document Ubuntu vs macOS reproducibility diagnosis (#1209) #1209 reported PolicyEngine UK producing slightly different aggregates on ubuntu-latest vs macos-latest CI. The triage narrowed it to dict-iteration-order differences in apply_uprating's tree walk (closed fix attempt PR #1223). A series of follow-on PRs since rebuilt the affected uprating code path: - #1254 added multi-year datasets and separated uprating into a discrete pipeline step; - PRs #1543 / #1544 / commit 13a4ac72 rebuilt uprate_rent which was the primary visible site of divergence; - #1523 skipped a redundant dataset copy in apply_uprating. CI runs on ubuntu-latest only, so there's no live cross-OS test to prove the issue is fully resolved. Persist the diagnosis as a known- issue page so that, if reproducibility regression resurfaces, the next person doesn't have to re-derive the root-cause shape. Includes a sketched minimal regression test for reintroducing deterministic ordering once a cross-OS CI matrix is restored. --- changelog.d/1209.md | 1 + docs/book/validation/ubuntu-vs-macos.md | 103 ++++++++++++++++++++++++ 2 files changed, 104 insertions(+) create mode 100644 changelog.d/1209.md create mode 100644 docs/book/validation/ubuntu-vs-macos.md diff --git a/changelog.d/1209.md b/changelog.d/1209.md new file mode 100644 index 0000000000..bf3a29370c --- /dev/null +++ b/changelog.d/1209.md @@ -0,0 +1 @@ +- Document the Ubuntu vs macOS reproducibility issue at `docs/book/validation/ubuntu-vs-macos.md`, recording the diagnosed dict-iteration-order root cause in `apply_uprating`, the series of follow-on PRs that rebuilt the affected uprating path (#1254, #1523, #1543, #1544), and a minimal regression-test sketch for use if the divergence resurfaces. diff --git a/docs/book/validation/ubuntu-vs-macos.md b/docs/book/validation/ubuntu-vs-macos.md new file mode 100644 index 0000000000..8cf47798bc --- /dev/null +++ b/docs/book/validation/ubuntu-vs-macos.md @@ -0,0 +1,103 @@ +# Ubuntu vs macOS reproducibility + +```{note} +Issue [#1209](https://github.com/PolicyEngine/policyengine-uk/issues/1209) +reported that PolicyEngine UK produced different aggregates on +`ubuntu-latest` vs `macos-latest` CI for the same dataset and the +same `policyengine-uk` version. The triage narrowed the divergence to +**parameter uprating**: a limited subset of parameters were uprated on +macOS but not on Ubuntu. This page records the diagnosis and the +current status so that, if a similar regression surfaces, the next +person doesn't have to re-derive the root-cause shape. +``` + +## Symptom + +Running the test suite or a microsimulation on the same dataset (with +the same `policyengine-uk-data` artifact) against the same +`policyengine-uk` version produces slightly different aggregates +across OS: + +- Affected: a limited subset of parameters (the dumps attached to the + original issue showed values diverged only at parameters that should + have been uprated by the OBR indices). +- Not affected: dataset content (verified identical between runs), + formulas, or rate parameters that were directly authored in YAML. + +The order of magnitude was small (a fraction of a percent on weighted +aggregates) but reproducibility-breaking. + +## Diagnosed root cause + +The triage in +[`fix-ubuntu-macos-parameter-uprating`](https://github.com/PolicyEngine/policyengine-uk/pull/1223) +narrowed it to **dictionary iteration order in the uprating step**: +when `apply_uprating` walked the parameter tree, the order in which +nested parameters were visited could differ between platforms in +specific cases, and floating-point accumulation differences meant +some values landed at slightly different rounded levels. + +## Current status + +A series of follow-on changes have since landed that interact with +this code path: + +- [PR #1254](https://github.com/PolicyEngine/policyengine-uk/pull/1254) + added multi-year datasets and separated uprating into a discrete + pipeline step. +- [PR #1543](https://github.com/PolicyEngine/policyengine-uk/pull/1543) / + [#1544](https://github.com/PolicyEngine/policyengine-uk/pull/1544) / + the [`uprate_rent` StringArray-conversion fix](https://github.com/PolicyEngine/policyengine-uk/commit/13a4ac72) + rebuilt the rent uprating path that was the primary visible site of + the divergence in #1209's dumps. +- [PR #1523](https://github.com/PolicyEngine/policyengine-uk/pull/1523) + skipped a redundant dataset copy in `apply_uprating`. + +The CI runs on `ubuntu-latest` only, so a cross-OS regression test was +not added when the original fixes landed. **If reproducibility across +OS becomes important for a downstream consumer (PolicyEngine.org +production, an external academic user), the next step is**: + +1. Restore the cross-OS CI matrix described in the original issue (or + carry the diagnostic shadow repo at + [`nikhilwoodruff/policyengine-uk-private`](https://github.com/nikhilwoodruff/policyengine-uk-private) + as a documented external check). +2. Run the same dataset through the build on `ubuntu-latest` and + `macos-latest`; compare the per-parameter dumps. If any divergence + reappears, the same iteration-order class of bug is the prime + suspect — re-instrument `apply_uprating` to sort children + deterministically. + +## How to add a deterministic-order regression test + +A minimal in-repo test that would catch a reintroduction: + +```python +def test_apply_uprating_deterministic_order(): + """Two passes of apply_uprating on the same tree should produce + bit-identical results regardless of dict insertion order.""" + from policyengine_uk import Microsimulation + + sim1 = Microsimulation() + sim2 = Microsimulation() + for variable in ["state_pension", "child_benefit", "universal_credit"]: + # Both Microsimulation instances build from the same data, + # uprate independently, and should converge to identical + # aggregates after the uprating step. + assert sim1.calculate(variable, 2025).sum() == sim2.calculate( + variable, 2025 + ).sum() +``` + +If this test passes on Ubuntu but a stronger cross-OS comparison fails, +the divergence is somewhere outside the iteration-order-via-dict +hypothesis and the analysis needs to start from the per-parameter +dumps again. + +## References + +- Tracking issue: [#1209](https://github.com/PolicyEngine/policyengine-uk/issues/1209). +- Original triage: [#1205](https://github.com/PolicyEngine/policyengine-uk/pull/1205). +- Closed fix attempt: [#1223](https://github.com/PolicyEngine/policyengine-uk/pull/1223). +- Follow-on uprating refactors: #1254, #1523, #1543, #1544. +- Diagnostic shadow repo: [`nikhilwoodruff/policyengine-uk-private`](https://github.com/nikhilwoodruff/policyengine-uk-private) (private; carries both OS runners). From 9ac113b2847b28b8924c89e967ba0683358e9fc8 Mon Sep 17 00:00:00 2001 From: vahid-ahmadi Date: Thu, 8 Oct 2026 15:52:22 +0100 Subject: [PATCH 2/4] Update the Ubuntu vs macOS page with the cause and its resolution Correct the root cause to the import-time growth-factor table (#1223), record that #1254 removed it, and add the October 2026 check that the parameter tree is byte-identical on macOS and Linux. Replace the regression-test sketch, which needed a default dataset and compared two runs on one OS, with a checksum check. Type the changelog fragment. Co-Authored-By: Claude Opus 5.5 --- changelog.d/1209.added.md | 1 + changelog.d/1209.md | 1 - docs/book/validation/ubuntu-vs-macos.md | 173 ++++++++++++------------ 3 files changed, 84 insertions(+), 91 deletions(-) create mode 100644 changelog.d/1209.added.md delete mode 100644 changelog.d/1209.md diff --git a/changelog.d/1209.added.md b/changelog.d/1209.added.md new file mode 100644 index 0000000000..cf2da497f8 --- /dev/null +++ b/changelog.d/1209.added.md @@ -0,0 +1 @@ +Documented the Ubuntu vs macOS reproducibility issue (#1209) at `docs/book/validation/ubuntu-vs-macos.md`: the import-time uprating cause, how it was resolved, and the October 2026 check that the parameter tree is byte-identical across macOS and Linux. diff --git a/changelog.d/1209.md b/changelog.d/1209.md deleted file mode 100644 index bf3a29370c..0000000000 --- a/changelog.d/1209.md +++ /dev/null @@ -1 +0,0 @@ -- Document the Ubuntu vs macOS reproducibility issue at `docs/book/validation/ubuntu-vs-macos.md`, recording the diagnosed dict-iteration-order root cause in `apply_uprating`, the series of follow-on PRs that rebuilt the affected uprating path (#1254, #1523, #1543, #1544), and a minimal regression-test sketch for use if the divergence resurfaces. diff --git a/docs/book/validation/ubuntu-vs-macos.md b/docs/book/validation/ubuntu-vs-macos.md index 8cf47798bc..03708c0e32 100644 --- a/docs/book/validation/ubuntu-vs-macos.md +++ b/docs/book/validation/ubuntu-vs-macos.md @@ -3,101 +3,94 @@ ```{note} Issue [#1209](https://github.com/PolicyEngine/policyengine-uk/issues/1209) reported that PolicyEngine UK produced different aggregates on -`ubuntu-latest` vs `macos-latest` CI for the same dataset and the -same `policyengine-uk` version. The triage narrowed the divergence to -**parameter uprating**: a limited subset of parameters were uprated on -macOS but not on Ubuntu. This page records the diagnosis and the -current status so that, if a similar regression surfaces, the next -person doesn't have to re-derive the root-cause shape. +`ubuntu-latest` and `macos-latest` for the same dataset and the same +`policyengine-uk` version. The cause was in parameter uprating. As of +October 2026 the parameter tree is byte-identical across macOS and +Linux, and CI has moved back to Linux. This page records the diagnosis, +the fix and how to check for a regression. ``` ## Symptom -Running the test suite or a microsimulation on the same dataset (with -the same `policyengine-uk-data` artifact) against the same -`policyengine-uk` version produces slightly different aggregates -across OS: - -- Affected: a limited subset of parameters (the dumps attached to the - original issue showed values diverged only at parameters that should - have been uprated by the OBR indices). -- Not affected: dataset content (verified identical between runs), - formulas, or rate parameters that were directly authored in YAML. - -The order of magnitude was small (a fraction of a percent on weighted -aggregates) but reproducibility-breaking. - -## Diagnosed root cause - -The triage in -[`fix-ubuntu-macos-parameter-uprating`](https://github.com/PolicyEngine/policyengine-uk/pull/1223) -narrowed it to **dictionary iteration order in the uprating step**: -when `apply_uprating` walked the parameter tree, the order in which -nested parameters were visited could differ between platforms in -specific cases, and floating-point accumulation differences meant -some values landed at slightly different rounded levels. - -## Current status - -A series of follow-on changes have since landed that interact with -this code path: - -- [PR #1254](https://github.com/PolicyEngine/policyengine-uk/pull/1254) - added multi-year datasets and separated uprating into a discrete - pipeline step. -- [PR #1543](https://github.com/PolicyEngine/policyengine-uk/pull/1543) / - [#1544](https://github.com/PolicyEngine/policyengine-uk/pull/1544) / - the [`uprate_rent` StringArray-conversion fix](https://github.com/PolicyEngine/policyengine-uk/commit/13a4ac72) - rebuilt the rent uprating path that was the primary visible site of - the divergence in #1209's dumps. -- [PR #1523](https://github.com/PolicyEngine/policyengine-uk/pull/1523) - skipped a redundant dataset copy in `apply_uprating`. - -The CI runs on `ubuntu-latest` only, so a cross-OS regression test was -not added when the original fixes landed. **If reproducibility across -OS becomes important for a downstream consumer (PolicyEngine.org -production, an external academic user), the next step is**: - -1. Restore the cross-OS CI matrix described in the original issue (or - carry the diagnostic shadow repo at - [`nikhilwoodruff/policyengine-uk-private`](https://github.com/nikhilwoodruff/policyengine-uk-private) - as a documented external check). -2. Run the same dataset through the build on `ubuntu-latest` and - `macos-latest`; compare the per-parameter dumps. If any divergence - reappears, the same iteration-order class of bug is the prime - suspect — re-instrument `apply_uprating` to sort children - deterministically. - -## How to add a deterministic-order regression test - -A minimal in-repo test that would catch a reintroduction: - -```python -def test_apply_uprating_deterministic_order(): - """Two passes of apply_uprating on the same tree should produce - bit-identical results regardless of dict insertion order.""" - from policyengine_uk import Microsimulation - - sim1 = Microsimulation() - sim2 = Microsimulation() - for variable in ["state_pension", "child_benefit", "universal_credit"]: - # Both Microsimulation instances build from the same data, - # uprate independently, and should converge to identical - # aggregates after the uprating step. - assert sim1.calculate(variable, 2025).sum() == sim2.calculate( - variable, 2025 - ).sum() -``` - -If this test passes on Ubuntu but a stronger cross-OS comparison fails, -the divergence is somewhere outside the iteration-order-via-dict -hypothesis and the analysis needs to start from the per-parameter -dumps again. +The divergence was first spotted in +[#1205](https://github.com/PolicyEngine/policyengine-uk/pull/1205), +whose results differed between the two runners. A diagnostic +repository that ran the same code on both runners and stored the full +entity dataset as artifacts showed that: + +- the input data were identical on both operating systems; +- a limited subset of parameters was uprated on macOS but not on + Ubuntu (the parameter-tree dumps are attached to + [#1209](https://github.com/PolicyEngine/policyengine-uk/issues/1209#issuecomment-3062418736)). + +The effect on weighted aggregates was small, but it broke +reproducibility. + +## Cause + +[#1223](https://github.com/PolicyEngine/policyengine-uk/pull/1223) +(closed without merging) traced it to an **import-time ordering +problem**: the uprating growth-factor table +(`BASELINE_GROWFACTORS = create_policyengine_uprating_factors_table()`) +was computed when its module was imported, before the parameter tree +had been fully processed. Whether the tree was ready by then depended +on the order in which modules and parameter files were loaded, and that +order differed between the two operating systems. + +## How it was resolved + +- [#1092](https://github.com/PolicyEngine/policyengine-uk/pull/1092) + and [#1207](https://github.com/PolicyEngine/policyengine-uk/pull/1207) + ran the Test job on `macos-latest` in 2025, so that CI matched the + platform that gave the expected results. +- [#1254](https://github.com/PolicyEngine/policyengine-uk/pull/1254) + (multi-year datasets, with uprating as a separate step) removed the + import-time `BASELINE_GROWFACTORS` table. Uprating is now applied by + `apply_uprating` in + [`policyengine_uk/data/economic_assumptions.py`](https://github.com/PolicyEngine/policyengine-uk/blob/main/policyengine_uk/data/economic_assumptions.py). +- Later fixes to the same path: + [#1523](https://github.com/PolicyEngine/policyengine-uk/pull/1523) + (no redundant dataset copy in `apply_uprating`), + [#1543](https://github.com/PolicyEngine/policyengine-uk/pull/1543) and + [#1544](https://github.com/PolicyEngine/policyengine-uk/pull/1544) + (`uprate_rent` lookups). +- policyengine-core now sorts each directory listing when it loads the + parameter tree, so the load order no longer depends on the file + system. + +On 3 October 2026 the Test job moved from macOS to `ubuntu-24.04-arm` +(commit +[`67c63861`](https://github.com/PolicyEngine/policyengine-uk/commit/67c638613)). +That change verified, on main at `3c48247e`, that: + +- the full parameter tree (2,706 parameter objects and 43,666 dated + values, including the baseline copies) is byte-identical on macOS + arm64, `ubuntu-latest` and `ubuntu-24.04-arm`; +- the YAML tests pass on both Linux runners; +- every pytest outcome matches the macOS run of the same code, test by + test. + +#1209 can be closed on that evidence. + +## Checking for a regression + +CI now runs the Test job on Linux arm64 and the smoke imports on x86 +`ubuntu-latest`. There is no macOS job. If an OS difference is +suspected again: + +1. Dump the processed parameter tree on each platform (every parameter + and every dated value, including `parameters.baseline`) and compare + checksums. This is the check the October 2026 move relied on. +2. If the trees differ, look first at anything that runs at import time + or depends on file or dictionary order. +3. If the trees match but results differ, compare the uprated dataset + variables next. #1209 showed the input data themselves were the same + on both systems. ## References - Tracking issue: [#1209](https://github.com/PolicyEngine/policyengine-uk/issues/1209). -- Original triage: [#1205](https://github.com/PolicyEngine/policyengine-uk/pull/1205). -- Closed fix attempt: [#1223](https://github.com/PolicyEngine/policyengine-uk/pull/1223). -- Follow-on uprating refactors: #1254, #1523, #1543, #1544. -- Diagnostic shadow repo: [`nikhilwoodruff/policyengine-uk-private`](https://github.com/nikhilwoodruff/policyengine-uk-private) (private; carries both OS runners). +- Where the difference was first seen: [#1205](https://github.com/PolicyEngine/policyengine-uk/pull/1205). +- Diagnosis: [#1223](https://github.com/PolicyEngine/policyengine-uk/pull/1223) (closed). +- CI moved to macOS: #1092, #1207; moved back to Linux: [`67c63861`](https://github.com/PolicyEngine/policyengine-uk/commit/67c638613). +- Uprating rebuild and follow-ups: #1254, #1523, #1543, #1544. From 3613e853f423eb9dd4d8e255b10763ef655013d1 Mon Sep 17 00:00:00 2001 From: vahid-ahmadi Date: Mon, 12 Oct 2026 11:05:47 +0200 Subject: [PATCH 3/4] Describe the current CI layout on the Ubuntu vs macOS page The Test job is now the sharded test suite on ubuntu-24.04-arm. Co-Authored-By: Claude Opus 5.5 --- docs/book/validation/ubuntu-vs-macos.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/docs/book/validation/ubuntu-vs-macos.md b/docs/book/validation/ubuntu-vs-macos.md index 03708c0e32..8f2d867ad2 100644 --- a/docs/book/validation/ubuntu-vs-macos.md +++ b/docs/book/validation/ubuntu-vs-macos.md @@ -74,7 +74,9 @@ That change verified, on main at `3c48247e`, that: ## Checking for a regression -CI now runs the Test job on Linux arm64 and the smoke imports on x86 +CI now runs the test suite (`.github/workflows/test-suite.yaml`: the +YAML tests and the Python tests, split over four shards) on Linux arm64 +(`ubuntu-24.04-arm`), and the install and smoke-import checks on x86 `ubuntu-latest`. There is no macOS job. If an OS difference is suspected again: From 4583587e20157a384d6fa031bb19f647f5db75f5 Mon Sep 17 00:00:00 2001 From: Max Ghenis Date: Fri, 23 Oct 2026 02:14:05 -0400 Subject: [PATCH 4/4] Correct the Ubuntu vs macOS cause and CI history The page reached main through #2386 and its changelog line shipped in 2.143.0, so this drops the released .added fragment and corrects main's copy: - Cause: the page presented #1223's import-time BASELINE_GROWFACTORS theory, but at a1832dc77 that table only read parameters and, with print_diff=False, returned the stored CSV. The dumps on #1209 differ in exactly 82 DWP parameters, all uprated through gov.benefit_uprating_cpi (itself chained from OBR CPI), main tree only; core then loaded directories with unsorted os.listdir. Labelled as an inference, with core#490/#491. - #1092 added macOS alongside Ubuntu; #1205 and #1207 made it macOS-only. - #1205 was the second case after #1091. - apply_uprating is dataset uprating; parameter uprating is core's. - Python tests now run on five shards (c0c010d49). - No source sizes the effect on aggregates. Co-Authored-By: Claude Opus 5.5 --- .../1209-ubuntu-macos-corrections.changed.md | 1 + changelog.d/1209.added.md | 1 - docs/book/validation/ubuntu-vs-macos.md | 104 +++++++++++------- 3 files changed, 68 insertions(+), 38 deletions(-) create mode 100644 changelog.d/1209-ubuntu-macos-corrections.changed.md delete mode 100644 changelog.d/1209.added.md diff --git a/changelog.d/1209-ubuntu-macos-corrections.changed.md b/changelog.d/1209-ubuntu-macos-corrections.changed.md new file mode 100644 index 0000000000..91b7f4114e --- /dev/null +++ b/changelog.d/1209-ubuntu-macos-corrections.changed.md @@ -0,0 +1 @@ +Corrected the Ubuntu vs macOS reproducibility page (docs/book/validation/ubuntu-vs-macos.md): the parameter dumps on #1209 point to order-dependent chained uprating, not the import-time growth-factor table, #1092 added macOS alongside Ubuntu, and the Python tests now run on five shards. diff --git a/changelog.d/1209.added.md b/changelog.d/1209.added.md deleted file mode 100644 index cf2da497f8..0000000000 --- a/changelog.d/1209.added.md +++ /dev/null @@ -1 +0,0 @@ -Documented the Ubuntu vs macOS reproducibility issue (#1209) at `docs/book/validation/ubuntu-vs-macos.md`: the import-time uprating cause, how it was resolved, and the October 2026 check that the parameter tree is byte-identical across macOS and Linux. diff --git a/docs/book/validation/ubuntu-vs-macos.md b/docs/book/validation/ubuntu-vs-macos.md index 8f2d867ad2..d3a8b4e2cd 100644 --- a/docs/book/validation/ubuntu-vs-macos.md +++ b/docs/book/validation/ubuntu-vs-macos.md @@ -2,65 +2,93 @@ ```{note} Issue [#1209](https://github.com/PolicyEngine/policyengine-uk/issues/1209) -reported that PolicyEngine UK produced different aggregates on -`ubuntu-latest` and `macos-latest` for the same dataset and the same -`policyengine-uk` version. The cause was in parameter uprating. As of -October 2026 the parameter tree is byte-identical across macOS and -Linux, and CI has moved back to Linux. This page records the diagnosis, -the fix and how to check for a regression. +reported that PolicyEngine UK gave different results on `ubuntu-latest` +and `macos-latest` for the same dataset and the same `policyengine-uk` +version. The cause was in parameter uprating. As of October 2026 the +parameter tree is byte-identical across macOS and Linux, and CI has moved +back to Linux. This page records the diagnosis, the fix and how to check +for a regression. ``` ## Symptom -The divergence was first spotted in +This divergence was spotted in [#1205](https://github.com/PolicyEngine/policyengine-uk/pull/1205), -whose results differed between the two runners. A diagnostic -repository that ran the same code on both runners and stored the full -entity dataset as artifacts showed that: +whose results differed between the two runners. It was the second such +case: [#1091](https://github.com/PolicyEngine/policyengine-uk/issues/1091) +(April 2025) had found benefit parameters that uprated on macOS but not +on Ubuntu, and [#1092](https://github.com/PolicyEngine/policyengine-uk/pull/1092) +fixed that one. A diagnostic repository that ran the same code on both +runners and stored the full entity dataset as artifacts showed that: - the input data were identical on both operating systems; - a limited subset of parameters was uprated on macOS but not on - Ubuntu (the parameter-tree dumps are attached to - [#1209](https://github.com/PolicyEngine/policyengine-uk/issues/1209#issuecomment-3062418736)). + Ubuntu. Parameter-tree dumps attached to a + [later comment on #1209](https://github.com/PolicyEngine/policyengine-uk/issues/1209#issuecomment-3062418736) + show 82 Department for Work and Pensions parameters, all uprated + through `gov.benefit_uprating_cpi`, that were uprated on macOS but + kept their set values on Ubuntu. -The effect on weighted aggregates was small, but it broke +No record gives the size of the effect on aggregates, but it broke reproducibility. ## Cause +The dumps show where the trees differed: the 82 parameters differ only +in the main tree, while their `parameters.baseline` copies match. At the +time `gov.benefit_uprating_cpi` held only a 2005 value and was itself +uprated from `gov.obr.consumer_price_index`, so these were chained +upratings (a parameter uprated from an index that is itself uprated), the +case #1092 had found in April 2025. policyengine-core then loaded each +parameter directory in unsorted `os.listdir` order, which can differ +between file systems, and uprating depended on the order in which it +reached the parameters, so a parameter reached before its uprating index +had been extended got no new values. +[policyengine-core#490](https://github.com/PolicyEngine/policyengine-core/issues/490) +later described the same failure in policyengine-us. The order the 2025 +runners actually listed the files in was never recorded, so this is +inferred from the code and the dumps rather than traced. + [#1223](https://github.com/PolicyEngine/policyengine-uk/pull/1223) -(closed without merging) traced it to an **import-time ordering -problem**: the uprating growth-factor table -(`BASELINE_GROWFACTORS = create_policyengine_uprating_factors_table()`) -was computed when its module was imported, before the parameter tree -had been fully processed. Whether the tree was ready by then depended -on the order in which modules and parameter files were loaded, and that -order differed between the two operating systems. +(closed without merging) proposed a different cause: that the uprating +growth-factor table (`BASELINE_GROWFACTORS`) was computed at import time, +before the system was initialised. That table only read parameter values +and, as called (`print_diff=False`), returned the stored +`uprating_growth_factors.csv`, so it could not change the parameter tree. ## How it was resolved - [#1092](https://github.com/PolicyEngine/policyengine-uk/pull/1092) - and [#1207](https://github.com/PolicyEngine/policyengine-uk/pull/1207) - ran the Test job on `macos-latest` in 2025, so that CI matched the - platform that gave the expected results. + (April 2025) added `macos-latest` alongside `ubuntu-latest` to the + pull-request Test job and fixed the first case, #1091. When #1209 + appeared, [#1205](https://github.com/PolicyEngine/policyengine-uk/pull/1205) + dropped `ubuntu-latest` from that job to unblock itself, and + [#1207](https://github.com/PolicyEngine/policyengine-uk/pull/1207) + moved the main-branch Test job to `macos-latest` (both July 2025). - [#1254](https://github.com/PolicyEngine/policyengine-uk/pull/1254) (multi-year datasets, with uprating as a separate step) removed the - import-time `BASELINE_GROWFACTORS` table. Uprating is now applied by - `apply_uprating` in + import-time `BASELINE_GROWFACTORS` table that #1223 had suspected. + Dataset uprating is now applied by `apply_uprating` in [`policyengine_uk/data/economic_assumptions.py`](https://github.com/PolicyEngine/policyengine-uk/blob/main/policyengine_uk/data/economic_assumptions.py). -- Later fixes to the same path: + Parameter uprating is policyengine-core's `uprate_parameters`, called + from `policyengine_uk/tax_benefit_system.py`. +- Later fixes to the dataset uprating path: [#1523](https://github.com/PolicyEngine/policyengine-uk/pull/1523) (no redundant dataset copy in `apply_uprating`), [#1543](https://github.com/PolicyEngine/policyengine-uk/pull/1543) and [#1544](https://github.com/PolicyEngine/policyengine-uk/pull/1544) (`uprate_rent` lookups). - policyengine-core now sorts each directory listing when it loads the - parameter tree, so the load order no longer depends on the file - system. + parameter tree, and uprates each parameter after the index it is + uprated from + ([policyengine-core#491](https://github.com/PolicyEngine/policyengine-core/pull/491), + May 2026), so neither load order nor the file system can change + uprated values. On 3 October 2026 the Test job moved from macOS to `ubuntu-24.04-arm` (commit -[`67c63861`](https://github.com/PolicyEngine/policyengine-uk/commit/67c638613)). +[`67c63861`](https://github.com/PolicyEngine/policyengine-uk/commit/67c638613), +in [#2115](https://github.com/PolicyEngine/policyengine-uk/pull/2115)). That change verified, on main at `3c48247e`, that: - the full parameter tree (2,706 parameter objects and 43,666 dated @@ -75,16 +103,17 @@ That change verified, on main at `3c48247e`, that: ## Checking for a regression CI now runs the test suite (`.github/workflows/test-suite.yaml`: the -YAML tests and the Python tests, split over four shards) on Linux arm64 -(`ubuntu-24.04-arm`), and the install and smoke-import checks on x86 -`ubuntu-latest`. There is no macOS job. If an OS difference is +YAML tests in one job and the Python tests split over five shards) on +Linux arm64 (`ubuntu-24.04-arm`), and the install and smoke-import checks +on x86 `ubuntu-latest`. There is no macOS job. If an OS difference is suspected again: 1. Dump the processed parameter tree on each platform (every parameter and every dated value, including `parameters.baseline`) and compare checksums. This is the check the October 2026 move relied on. -2. If the trees differ, look first at anything that runs at import time - or depends on file or dictionary order. +2. If the trees differ, look first at chained uprating (a parameter + uprated from an index that is itself uprated) and at anything else + that depends on file or dictionary order. 3. If the trees match but results differ, compare the uprated dataset variables next. #1209 showed the input data themselves were the same on both systems. @@ -92,7 +121,8 @@ suspected again: ## References - Tracking issue: [#1209](https://github.com/PolicyEngine/policyengine-uk/issues/1209). -- Where the difference was first seen: [#1205](https://github.com/PolicyEngine/policyengine-uk/pull/1205). -- Diagnosis: [#1223](https://github.com/PolicyEngine/policyengine-uk/pull/1223) (closed). -- CI moved to macOS: #1092, #1207; moved back to Linux: [`67c63861`](https://github.com/PolicyEngine/policyengine-uk/commit/67c638613). +- Where this difference was seen: [#1205](https://github.com/PolicyEngine/policyengine-uk/pull/1205); the earlier case: [#1091](https://github.com/PolicyEngine/policyengine-uk/issues/1091), fixed by #1092. +- A proposed cause that did not hold: [#1223](https://github.com/PolicyEngine/policyengine-uk/pull/1223) (closed without merging). +- Order-independent uprating: [policyengine-core#490](https://github.com/PolicyEngine/policyengine-core/issues/490), [#491](https://github.com/PolicyEngine/policyengine-core/pull/491). +- CI moved to macOS: #1092 (macOS alongside Ubuntu), #1205 and #1207 (macOS only); moved back to Linux: [`67c63861`](https://github.com/PolicyEngine/policyengine-uk/commit/67c638613) (#2115). - Uprating rebuild and follow-ups: #1254, #1523, #1543, #1544.