diff --git a/changelog.d/1209-ubuntu-macos-corrections.changed.md b/changelog.d/1209-ubuntu-macos-corrections.changed.md new file mode 100644 index 0000000000..91b7f4114e --- /dev/null +++ b/changelog.d/1209-ubuntu-macos-corrections.changed.md @@ -0,0 +1 @@ +Corrected the Ubuntu vs macOS reproducibility page (docs/book/validation/ubuntu-vs-macos.md): the parameter dumps on #1209 point to order-dependent chained uprating, not the import-time growth-factor table, #1092 added macOS alongside Ubuntu, and the Python tests now run on five shards. diff --git a/docs/book/validation/ubuntu-vs-macos.md b/docs/book/validation/ubuntu-vs-macos.md index 8f2d867ad2..d3a8b4e2cd 100644 --- a/docs/book/validation/ubuntu-vs-macos.md +++ b/docs/book/validation/ubuntu-vs-macos.md @@ -2,65 +2,93 @@ ```{note} Issue [#1209](https://github.com/PolicyEngine/policyengine-uk/issues/1209) -reported that PolicyEngine UK produced different aggregates on -`ubuntu-latest` and `macos-latest` for the same dataset and the same -`policyengine-uk` version. The cause was in parameter uprating. As of -October 2026 the parameter tree is byte-identical across macOS and -Linux, and CI has moved back to Linux. This page records the diagnosis, -the fix and how to check for a regression. +reported that PolicyEngine UK gave different results on `ubuntu-latest` +and `macos-latest` for the same dataset and the same `policyengine-uk` +version. The cause was in parameter uprating. As of October 2026 the +parameter tree is byte-identical across macOS and Linux, and CI has moved +back to Linux. This page records the diagnosis, the fix and how to check +for a regression. ``` ## Symptom -The divergence was first spotted in +This divergence was spotted in [#1205](https://github.com/PolicyEngine/policyengine-uk/pull/1205), -whose results differed between the two runners. A diagnostic -repository that ran the same code on both runners and stored the full -entity dataset as artifacts showed that: +whose results differed between the two runners. It was the second such +case: [#1091](https://github.com/PolicyEngine/policyengine-uk/issues/1091) +(April 2025) had found benefit parameters that uprated on macOS but not +on Ubuntu, and [#1092](https://github.com/PolicyEngine/policyengine-uk/pull/1092) +fixed that one. A diagnostic repository that ran the same code on both +runners and stored the full entity dataset as artifacts showed that: - the input data were identical on both operating systems; - a limited subset of parameters was uprated on macOS but not on - Ubuntu (the parameter-tree dumps are attached to - [#1209](https://github.com/PolicyEngine/policyengine-uk/issues/1209#issuecomment-3062418736)). + Ubuntu. Parameter-tree dumps attached to a + [later comment on #1209](https://github.com/PolicyEngine/policyengine-uk/issues/1209#issuecomment-3062418736) + show 82 Department for Work and Pensions parameters, all uprated + through `gov.benefit_uprating_cpi`, that were uprated on macOS but + kept their set values on Ubuntu. -The effect on weighted aggregates was small, but it broke +No record gives the size of the effect on aggregates, but it broke reproducibility. ## Cause +The dumps show where the trees differed: the 82 parameters differ only +in the main tree, while their `parameters.baseline` copies match. At the +time `gov.benefit_uprating_cpi` held only a 2005 value and was itself +uprated from `gov.obr.consumer_price_index`, so these were chained +upratings (a parameter uprated from an index that is itself uprated), the +case #1092 had found in April 2025. policyengine-core then loaded each +parameter directory in unsorted `os.listdir` order, which can differ +between file systems, and uprating depended on the order in which it +reached the parameters, so a parameter reached before its uprating index +had been extended got no new values. +[policyengine-core#490](https://github.com/PolicyEngine/policyengine-core/issues/490) +later described the same failure in policyengine-us. The order the 2025 +runners actually listed the files in was never recorded, so this is +inferred from the code and the dumps rather than traced. + [#1223](https://github.com/PolicyEngine/policyengine-uk/pull/1223) -(closed without merging) traced it to an **import-time ordering -problem**: the uprating growth-factor table -(`BASELINE_GROWFACTORS = create_policyengine_uprating_factors_table()`) -was computed when its module was imported, before the parameter tree -had been fully processed. Whether the tree was ready by then depended -on the order in which modules and parameter files were loaded, and that -order differed between the two operating systems. +(closed without merging) proposed a different cause: that the uprating +growth-factor table (`BASELINE_GROWFACTORS`) was computed at import time, +before the system was initialised. That table only read parameter values +and, as called (`print_diff=False`), returned the stored +`uprating_growth_factors.csv`, so it could not change the parameter tree. ## How it was resolved - [#1092](https://github.com/PolicyEngine/policyengine-uk/pull/1092) - and [#1207](https://github.com/PolicyEngine/policyengine-uk/pull/1207) - ran the Test job on `macos-latest` in 2025, so that CI matched the - platform that gave the expected results. + (April 2025) added `macos-latest` alongside `ubuntu-latest` to the + pull-request Test job and fixed the first case, #1091. When #1209 + appeared, [#1205](https://github.com/PolicyEngine/policyengine-uk/pull/1205) + dropped `ubuntu-latest` from that job to unblock itself, and + [#1207](https://github.com/PolicyEngine/policyengine-uk/pull/1207) + moved the main-branch Test job to `macos-latest` (both July 2025). - [#1254](https://github.com/PolicyEngine/policyengine-uk/pull/1254) (multi-year datasets, with uprating as a separate step) removed the - import-time `BASELINE_GROWFACTORS` table. Uprating is now applied by - `apply_uprating` in + import-time `BASELINE_GROWFACTORS` table that #1223 had suspected. + Dataset uprating is now applied by `apply_uprating` in [`policyengine_uk/data/economic_assumptions.py`](https://github.com/PolicyEngine/policyengine-uk/blob/main/policyengine_uk/data/economic_assumptions.py). -- Later fixes to the same path: + Parameter uprating is policyengine-core's `uprate_parameters`, called + from `policyengine_uk/tax_benefit_system.py`. +- Later fixes to the dataset uprating path: [#1523](https://github.com/PolicyEngine/policyengine-uk/pull/1523) (no redundant dataset copy in `apply_uprating`), [#1543](https://github.com/PolicyEngine/policyengine-uk/pull/1543) and [#1544](https://github.com/PolicyEngine/policyengine-uk/pull/1544) (`uprate_rent` lookups). - policyengine-core now sorts each directory listing when it loads the - parameter tree, so the load order no longer depends on the file - system. + parameter tree, and uprates each parameter after the index it is + uprated from + ([policyengine-core#491](https://github.com/PolicyEngine/policyengine-core/pull/491), + May 2026), so neither load order nor the file system can change + uprated values. On 3 October 2026 the Test job moved from macOS to `ubuntu-24.04-arm` (commit -[`67c63861`](https://github.com/PolicyEngine/policyengine-uk/commit/67c638613)). +[`67c63861`](https://github.com/PolicyEngine/policyengine-uk/commit/67c638613), +in [#2115](https://github.com/PolicyEngine/policyengine-uk/pull/2115)). That change verified, on main at `3c48247e`, that: - the full parameter tree (2,706 parameter objects and 43,666 dated @@ -75,16 +103,17 @@ That change verified, on main at `3c48247e`, that: ## Checking for a regression CI now runs the test suite (`.github/workflows/test-suite.yaml`: the -YAML tests and the Python tests, split over four shards) on Linux arm64 -(`ubuntu-24.04-arm`), and the install and smoke-import checks on x86 -`ubuntu-latest`. There is no macOS job. If an OS difference is +YAML tests in one job and the Python tests split over five shards) on +Linux arm64 (`ubuntu-24.04-arm`), and the install and smoke-import checks +on x86 `ubuntu-latest`. There is no macOS job. If an OS difference is suspected again: 1. Dump the processed parameter tree on each platform (every parameter and every dated value, including `parameters.baseline`) and compare checksums. This is the check the October 2026 move relied on. -2. If the trees differ, look first at anything that runs at import time - or depends on file or dictionary order. +2. If the trees differ, look first at chained uprating (a parameter + uprated from an index that is itself uprated) and at anything else + that depends on file or dictionary order. 3. If the trees match but results differ, compare the uprated dataset variables next. #1209 showed the input data themselves were the same on both systems. @@ -92,7 +121,8 @@ suspected again: ## References - Tracking issue: [#1209](https://github.com/PolicyEngine/policyengine-uk/issues/1209). -- Where the difference was first seen: [#1205](https://github.com/PolicyEngine/policyengine-uk/pull/1205). -- Diagnosis: [#1223](https://github.com/PolicyEngine/policyengine-uk/pull/1223) (closed). -- CI moved to macOS: #1092, #1207; moved back to Linux: [`67c63861`](https://github.com/PolicyEngine/policyengine-uk/commit/67c638613). +- Where this difference was seen: [#1205](https://github.com/PolicyEngine/policyengine-uk/pull/1205); the earlier case: [#1091](https://github.com/PolicyEngine/policyengine-uk/issues/1091), fixed by #1092. +- A proposed cause that did not hold: [#1223](https://github.com/PolicyEngine/policyengine-uk/pull/1223) (closed without merging). +- Order-independent uprating: [policyengine-core#490](https://github.com/PolicyEngine/policyengine-core/issues/490), [#491](https://github.com/PolicyEngine/policyengine-core/pull/491). +- CI moved to macOS: #1092 (macOS alongside Ubuntu), #1205 and #1207 (macOS only); moved back to Linux: [`67c63861`](https://github.com/PolicyEngine/policyengine-uk/commit/67c638613) (#2115). - Uprating rebuild and follow-ups: #1254, #1523, #1543, #1544.