Skip to content

Correct the Ubuntu vs macOS reproducibility page (#1209) - #1730

Merged
vahid-ahmadi merged 5 commits into
mainfrom
vahid-ahmadi/issue-1209-ubuntu-macos-diagnosis
Oct 24, 2026
Merged

Correct the Ubuntu vs macOS reproducibility page (#1209)#1730
vahid-ahmadi merged 5 commits into
mainfrom
vahid-ahmadi/issue-1209-ubuntu-macos-diagnosis

Conversation

@vahid-ahmadi

@vahid-ahmadi vahid-ahmadi commented May 27, 2026 •
edited
Loading

Copy link
Copy Markdown
Collaborator

Corrects the Ubuntu vs macOS reproducibility diagnosis page (docs/book/validation/ubuntu-vs-macos.md), written for #1209.

The page itself is already on main: it shipped in #2386 (merged 21 October 2026), and its original .added changelog entry was released in 2.143.0. This branch has merged main, dropped the released fragment and now carries only corrections to main's copy of the page.

What this PR corrects (from its changelog fragment):

Corrected the Ubuntu vs macOS reproducibility page (docs/book/validation/ubuntu-vs-macos.md): the parameter dumps on #1209 point to order-dependent chained uprating, not the import-time growth-factor table, #1092 added macOS alongside Ubuntu, and the Python tests now run on five shards.

The claim-by-claim review behind these corrections, with sources, is in the "Ready for hub review" comment on this PR.

The corrected page notes that #1209 can be closed (the parameter tree is byte-identical across macOS and Linux and CI is back on Linux); the issue itself is left for its author to close.

  • Changelog fragment: changelog.d/1209-ubuntu-macos-corrections.changed.md (changed).
  • Docs and changelog only; no code changes.

🤖 Generated with Claude Code

@vahid-ahmadi
vahid-ahmadi force-pushed the vahid-ahmadi/issue-1209-ubuntu-macos-diagnosis branch from 9520af9 to 4810381 Compare October 8, 2026 14:53
vahid-ahmadi and others added 3 commits October 12, 2026 10:59
@vahid-ahmadi
Document Ubuntu vs macOS reproducibility diagnosis (#1209)
ad482e6 
#1209 reported PolicyEngine UK producing slightly different aggregates
on ubuntu-latest vs macos-latest CI. The triage narrowed it to
dict-iteration-order differences in apply_uprating's tree walk
(closed fix attempt PR #1223).

A series of follow-on PRs since rebuilt the affected uprating code
path:

- #1254 added multi-year datasets and separated uprating into a
  discrete pipeline step;
- PRs #1543 / #1544 / commit 13a4ac7 rebuilt uprate_rent which was
  the primary visible site of divergence;
- #1523 skipped a redundant dataset copy in apply_uprating.

CI runs on ubuntu-latest only, so there's no live cross-OS test to
prove the issue is fully resolved. Persist the diagnosis as a known-
issue page so that, if reproducibility regression resurfaces, the
next person doesn't have to re-derive the root-cause shape.

Includes a sketched minimal regression test for reintroducing
deterministic ordering once a cross-OS CI matrix is restored.
@vahid-ahmadi @claude
Update the Ubuntu vs macOS page with the cause and its resolution
9ac113b 
Correct the root cause to the import-time growth-factor table (#1223), record that #1254 removed it, and add the October 2026 check that the parameter tree is byte-identical on macOS and Linux. Replace the regression-test sketch, which needed a default dataset and compared two runs on one OS, with a checksum check. Type the changelog fragment.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@vahid-ahmadi @claude
Describe the current CI layout on the Ubuntu vs macOS page
3613e85 
The Test job is now the sharded test suite on ubuntu-24.04-arm.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@vahid-ahmadi
vahid-ahmadi force-pushed the vahid-ahmadi/issue-1209-ubuntu-macos-diagnosis branch from 4810381 to 3613e85 Compare October 12, 2026 09:05
@vahid-ahmadi vahid-ahmadi mentioned this pull request Oct 12, 2026
Merged
MaxGhenis and others added 2 commits October 23, 2026 02:13
@MaxGhenis @claude
Merge main into the Ubuntu vs macOS page branch
1f8cde2 
Main already carries this page (#2386) and released its changelog line in 2.143.0.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@MaxGhenis @claude
Correct the Ubuntu vs macOS cause and CI history
4583587 
The page reached main through #2386 and its changelog line shipped in
2.143.0, so this drops the released .added fragment and corrects main's
copy:

- Cause: the page presented #1223's import-time BASELINE_GROWFACTORS
  theory, but at a1832dc that table only read parameters and, with
  print_diff=False, returned the stored CSV. The dumps on #1209 differ
  in exactly 82 DWP parameters, all uprated through
  gov.benefit_uprating_cpi (itself chained from OBR CPI), main tree only;
  core then loaded directories with unsorted os.listdir. Labelled as an
  inference, with core#490/#491.
- #1092 added macOS alongside Ubuntu; #1205 and #1207 made it macOS-only.
- #1205 was the second case after #1091.
- apply_uprating is dataset uprating; parameter uprating is core's.
- Python tests now run on five shards (c0c010d).
- No source sizes the effect on aggregates.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@MaxGhenis

MaxGhenis commented Oct 23, 2026

Copy link
Copy Markdown
Collaborator

UK hub (Claude build session, for Max)
Ready for hub review

Head: 4583587e20157a384d6fa031bb19f647f5db75f5

Read this first. The page is already on main, word for word, through #2386 (d50ee85a9, merged 2026-10-21). Its changelog line shipped in 2.143.0 (CHANGELOG.md:198).

This branch now:

  • merges main (1f8cde250);
  • deletes changelog.d/1209.added.md;
  • corrects main's copy;
  • adds changelog.d/1209-ubuntu-macos-corrections.changed.md.

Against main, the diff is the page and the fragment. Suggested title on landing: "Correct the Ubuntu vs macOS reproducibility page".

Claims checked

Claim Verdict Source
#1209 reported different results on the two runners; the diagnostic repo found identical input data true ("results", not "aggregates") #1209 body
A subset of parameters was uprated on macOS but not on Ubuntu (comment 3062418736) true. I diffed the two attached dumps: 82 paths differ, all gov.dwp.*, none in baseline.*, 668 extra dated values on macOS (e.g. gov.dwp.pip.mobility.standard 2026: 29.19 vs 29.757). All 82 take uprating: gov.benefit_uprating_cpi at a1832dc77 Linux_parameters.txt, Darwin_parameters.txt
"first spotted in #1205" fixed: second case, after #1091 (April 2025, fixed by #1092) #1091, #1092
"effect on weighted aggregates was small" cut: no source sizes it #1205, #1209, #1223 threads
Cause: #1223's import-time BASELINE_GROWFACTORS table false as the cause. At a1832dc77, create_policyengine_uprating_factors_table only reads parameters and, as called with print_diff=False (economic_assumptions.py:185-186), returns pd.read_csv(file_path) (:71) a1832dc77:policyengine_uk/data/economic_assumptions.py:15-71
Replacement cause order-dependent chained uprating, labelled on the page as inferred, not traced. gov.benefit_uprating_cpi held only 2005-01-01: 100 with uprating: gov.obr.consumer_price_index (a1832dc77:.../benefit_uprating_cpi.yaml). Core loaded directories unsorted (policyengine-core@49b4c7c3:policyengine_core/parameters/parameter_node.py:94 os.listdir). policyengine-core#490 describes the same order dependence core#490
#1092 and #1207 "ran the Test job on macos-latest" fixed: #1092 made the PR job a matrix [ubuntu-latest, macos-latest]; #1205 dropped Ubuntu; #1207 moved code_changes.yaml to macos-latest PR file diffs
#1254 removed BASELINE_GROWFACTORS true git grep on main finds only this page
apply_uprating in economic_assumptions.py true, but it is dataset uprating; parameter uprating is core's uprate_parameters policyengine_uk/data/economic_assumptions.py:247; tax_benefit_system.py:14-15,212
#1523, #1543, #1544 true merged PRs
Core now sorts directory listings true; also orders uprating by dependency (core#491, merged 2026-05-14) core master parameter_node.py:92 sorted(os.listdir(...)); uprate_parameters.py:37,198
67c63861 on 3 October 2026; byte-identical tree (2,706 objects, 43,666 values) on 3c48247e; YAML and pytest parity true as recorded in the commit message (in #2115). The review agent re-hashed the two Linux artifacts of run 37120279129: identical sha256 79abbf29… git show 67c638613
"split over four shards" outdated: five test-suite.yaml:141 shard: [1, 2, 3, 4, 5] (since c0c010d49)
Smoke import on ubuntu-latest; no macOS job true pr_code_changes.yaml:79-84; git grep -i macos .github/workflows (comments only)
Regression step 2 "anything that runs at import time" reworded to chained uprating and order dependence as above

Corrections: 8.

Uncertain

Full ledger: ~/reviews/uk-hub/jobs/docs-c2/b/1730-ledger.md.

@MaxGhenis MaxGhenis mentioned this pull request Oct 23, 2026
Closed
3 of 5 tasks
@vahid-ahmadi vahid-ahmadi changed the title Document Ubuntu vs macOS reproducibility diagnosis (#1209) Correct the Ubuntu vs macOS reproducibility page (#1209) Oct 23, 2026
@vahid-ahmadi
vahid-ahmadi added this pull request to the merge queue Oct 23, 2026
Merged via the queue into main with commit bc8bdf7 Oct 24, 2026
10 checks passed
@vahid-ahmadi
vahid-ahmadi deleted the vahid-ahmadi/issue-1209-ubuntu-macos-diagnosis branch October 24, 2026 10:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@vahid-ahmadi @MaxGhenis