Migrate the TAXSIM benchmark and dashboard to the certified Populace US build by MaxGhenis · Pull Request #1216 · PolicyEngine/policyengine-taxsim

MaxGhenis · 2026-09-24T03:10:40Z

Moves the TAXSIM-vs-PolicyEngine benchmark and its dashboard from the archived Enhanced CPS to the Populace US build that policyengine.py certifies. The Enhanced CPS stays on the dashboard as a secondary dataset, because the Populace build lacks its high-income tail (check 1).

All rates below are PolicyEngine's own comparison of its emulator against NBER's taxsimtest binary on identical inputs. NBER and Dan Feenberg have not endorsed them as error statistics. The dashboard, the release notes and every summary JSON (metadata.comparisonNote) say so.

Stacked on #1204 (Alabama) and #1205 (otheritem routing), plus main. Both are merged into this branch, so the diff includes them until they land. Merge them first. The benchmark results below include both.

Data build and model versions

Build: populace-us-2024-spm-20260915. This is the US release in policyengine.py's bundle manifest on main. I checked it through the GitHub API on 2026-09-23 and again at 2026-09-24 03:10 UTC; main was at 4dc5959, and the manifest's last change was on 2026-09-22.

Source: hf://policyengine/populace-us, file populace_us_2024.h5, pinned to Hugging Face commit 8ab57ffc2ca41d8af631ff62ebbce95e966299f3.
sha256 6496cc4393d4d3c6574f76eca231de5898c803b9067645591fd5c4d3e65aee84.

I did not use the newer tag populace-us-2024-spm-receipts-20260923:

It is not in the policyengine.py manifest.
Its default dataset is a different file (populace_us_2024_receipt_qualified).
Its release manifest declares policyengine-us ==2.2.1 with no certification record.

Version conflict, resolved by splitting the two jobs.

Reading the H5. The build is certified for policyengine-us 2.2.1 and policyengine-core 3.32.5; the release manifest's publisher claim is >=2.0.0,<2.3. scripts/convert_h5_to_taxsim.py reads the certified build only in that environment, pinned in scripts/populace-convert-requirements.txt. On that default path it refuses any other version and verifies the H5's sha256. The --h5 and --dataset options exist for other files and skip both checks.
Running the benchmark. The benchmark runs the emulator in the existing dashboard environment (policyengine-us 2.6.17, core 3.32.6), outside the certified range.
Why that is safe. The emulator never loads the H5. It builds its own households from the TAXSIM rows (TaxsimMicrosimDataset in policyengine_runner.py). The loader, role-variable and dataset-pin compatibility that the certification covers therefore never comes into play at 2.6.17.
Remaining risk. The deductions are computed under 2.2.1 rules; roles and filing status come from the build itself (see below). Both engines receive the same values, but they could differ from what 2.6.17 would compute. I did not run a sensitivity analysis. I diffed the two installed packages. The formula files behind every derived input are byte-identical: deductible_mortgage_interest, the medical, charitable and casualty deductions, rent and tax_unit_childcare_expenses. The only difference under parameters/gov/irs/deductions/itemized is the itemized-deduction limitation, which no input uses. AGI, which sets the medical and charitable floors, can still differ between versions.

Covering 2021–2025. This follows the eCPS benchmark:

The 79,729 records, one per tax unit, are 2024 data.
The same nominal records are scored under each year's law, with only year changed.
Deductions are PolicyEngine's 2024 amounts. There is no uprating.

Input definitions

The converter follows the taxsimtest documentation, because taxsimtest is the benchmark binary. Every definition is in the converter's docstring. Differences from the eCPS inputs:

Itemized deductions are populated (check 2).
Incomes are the primary taxpayer's and spouse's. Dependents' own income is left out; dependents who would file their own returns are not represented.
Expenses are summed per tax unit. The old converter looked up rent, property tax and mortgage by household and gave the full amount to every tax unit in the household.
intrec adds non-qualified dividends, which the TAXSIM docs allow in intrec for years after 2003.
pensions adds IRA and other retirement-account distributions, matching TAXSIM's definition of the field (taxable pensions plus IRA distributions).
transfers is veterans' benefits, workers' compensation and child support received. The emulator maps it to general_assistance (Map TAXSIM transfers to general_assistance (#455) #1052).
nonprop stays 0. The emulator maps it to no PolicyEngine variable, so a value would be dropped on the PolicyEngine side only. This is a follow-up.
Infants are coded 1, as the TAXSIM docs instruct; the eCPS coded 2,114 dependent ages as 0. Dependent ages are listed youngest first.
Roles and filing status come from the build.
- Populace stores each person's tax_unit_role_input (HEAD, SPOUSE, DEPENDENT) and each unit's filing_status_input.
- policyengine-us 2.2.1 has no variable that reads them, and 2.6.17 has none either. is_tax_unit_head is the oldest adult and is_tax_unit_spouse the next-oldest.
- Left alone, that rule makes 1,161 of the build's dependents spouses, which turns those returns joint. It also gives 13,217 units a different or missing head: 12,965 different, and 252 units, mostly lone teenagers, with no head at all.
- The converter pins PolicyEngine's role and filing_status variables to the build's values before calculating anything. All 79,729 units are kept. The 35,592 joint returns match the build's JOINT count exactly.
- The independent review of this PR caught the age-rule problem.
Filing status codes. Joint returns are mstat 2 and all others mstat 1. That collapses 8,554 heads of household (TAXSIM infers the status from dependents), 876 married filing separately and 554 surviving spouses. The emulator's Microsimulation path treats mstat 6 as a two-person unit (has_spouse = np.isin(mstat, [2, 6])), so I did not use it. This is a follow-up.

Check 1: top-income coverage (Populace lacks the tail)

Counts are unweighted TAXSIM records. "Total income" is the sum of the 16 TAXSIM income fields. That definition reproduces the eCPS counts in the brief exactly, and tests/test_populace_inputs.py pins both columns.

Records with	eCPS (111,347)	Populace (79,729)
Total income of $1M or more	11,454	1,291
Total income of $10M or more	6,867	3
Largest total income	$348.0M	$14.7M
Long-term capital gains ≠ 0	34,552	8,514
S-corp/partnership income ≠ 0	26,162	4,569

The Populace build has essentially no top tail. A separate session has been opened for it: "Fix Microcosm top-income tail (no returns above $10M AGI)". This matters for how the headline is read. On the eCPS, the emulator does worst exactly there: for records with $1M+ total income, federal agreement within 1% is 29–37% (2021–2025), against 89–91% overall. On Populace, federal agreement within 1% for its 1,291 records with $1M+ total income is 77–79% (2021–2025). The Populace headline rates therefore exclude most of the records where the emulator disagrees most, and they should not be compared one-for-one with the eCPS rates.

The eCPS is therefore kept, not dropped:

The dashboard has a dataset toggle, with Populace by default and "Enhanced CPS (archived)" one click away. ?dataset=ecps links straight to the eCPS view.
The eCPS view was regenerated with the same code in this PR, so Alabama is scored as state 1.
The eCPS view has its own release.
Populace's caveat on the dashboard says the tail is untested and that its rates are not comparable to the eCPS rates.

Check 2: itemized deductions are now tested

In the eCPS inputs, proptax, mortgage and otheritem are zero in every record. In the Populace inputs, the nonzero counts are:

Field	Populace records ≠ 0	Definition used (taxsimtest docs, items 21, 22 and 24)
`proptax`	17,216	Real estate taxes
`mortgage`	38,744	Non-AMT-preference deductions as net deductible amounts: deductible mortgage interest + medical above the 7.5% floor + charitable after AGI limits + casualty losses
`otheritem`	14,108	AMT-preference deductions; for 2021–2025, investment interest (TAXSIM lists "interest other than own residence" here)

This builds on #1205 and does not conflict with it. #1205 routes mortgage and otheritem into PolicyEngine's deductible_mortgage_interest, and it is merged into this branch. From that session's taxsimtest sweeps:

The binary applies no floor to mortgage, which is why the converter supplies net deductible amounts.
A negative mortgage makes taxsimtest stop the whole run (exit 1), so the converter rejects negative values.
The split between mortgage and otheritem changes state itemized deductions in AR, DE, IA (2020–22), KS, ME, MS, NC and WI (2023+), and can move federal tax through the state tax. The Route TAXSIM otheritem into itemized deductions alongside mortgage #1205 sweep covered those state-years for its test inputs.
Example from the second review, a 2024 Delaware single filer: moving $18,000 from mortgage to otheritem raised state tax by $929.63 and lowered federal tax by $111.56.

Why otheritem holds only investment interest for 2021–2025. TAXSIM's other otheritem components are empty or not PolicyEngine inputs in these years:

Miscellaneous deductions are disallowed after 2017 (IRC §67(h), quoted below).
The preference share of medical expenses is zero, because Form 6251 has no medical adjustment. The full 2024 Part I is extracted below and has no medical line; page 1 of the 2021, 2022, 2023 and 2025 forms has no "medical" either (pdftotext -layout, grep).
Other state and local taxes have no PolicyEngine input.
Local income tax is a PolicyEngine output, not an input.

Extracted evidence (pdftotext -layout; Cornell LII HTML), fetched 2026-09-23

IRC §67(h), "Suspension for taxable years beginning after 2017" (26 U.S.C. §67, Cornell LII; amended by Pub. L. 119–21, §70110):
Notwithstanding subsection (a), no miscellaneous itemized deduction shall be allowed for any taxable year beginning after December 31, 2017.

Form 6251 (2024), page 1, Part I "Alternative Minimum Taxable Income" (the complete Part I; there is no medical-expense line):
1 Enter the amount from Form 1040 or 1040-SR, line 15, if more than zero. [...]
2a If filing Schedule A (Form 1040), enter the taxes from Schedule A, line 7; otherwise, enter the amount from Form 1040 or 1040-SR, line 12
b Tax refund from Schedule 1 (Form 1040), line 1 or line 8z
c Investment interest expense (difference between regular tax and AMT)
d Depletion (difference between regular tax and AMT)
e Net operating loss deduction from Schedule 1 (Form 1040), line 8a. Enter as a positive amount
f Alternative tax net operating loss deduction
g Interest from specified private activity bonds exempt from the regular tax
h Qualified small business stock, see instructions
i Exercise of incentive stock options (excess of AMT income over regular tax income)
j Estates and trusts (amount from Schedule K-1 (Form 1041), box 12, code A)
k Disposition of property (difference between AMT and regular tax gain or loss)
l Depreciation on assets placed in service after 1986 (difference between regular tax and AMT)
m Passive activities (difference between AMT and regular tax income or loss)
n Loss limitations (difference between AMT and regular tax income or loss)
o Circulation costs (difference between regular tax and AMT)
p Long-term contracts (difference between AMT and regular tax income)
q Mining costs (difference between regular tax and AMT)
r Research and experimental costs (difference between regular tax and AMT)
s Income from certain installment sales before January 1, 1987
t Intangible drilling costs preference
3 Other adjustments, including income-based related adjustments
4 Alternative minimum taxable income. Combine lines 1 through 3. [...]

For contrast, the 2017 Form 6251 line 2 reads "Reserved for future use". Line 1's elided text and line 4's MFS threshold are omitted above. Part I is otherwise complete.

Check 3: every state, including Alabama

Populace. All 51 SOI states are present; Alabama (SOI 1) has 1,411 records, and the smallest state has 654. The converter raises on an unknown FIPS code instead of writing 0, and check_source (Code Alabama as TAXSIM state 1 and stop reporting state 0 as Texas #1204) rejects state 0 before any model runs.
Tests. test_populace_inputs_cover_every_state_including_alabama pins the committed inputs. DatasetTests.test_every_dataset_source_is_complete checks both datasets. Code Alabama as TAXSIM state 1 and stop reporting state 0 as Texas #1204's tests/test_state_codes.py passes against the new converter.
eCPS. The regenerated data has an AL row (1,155 records). TX drops to 7,182, matching Code Alabama as TAXSIM state 1 and stop reporting state 0 as Texas #1204's own refresh.

Check 4: releases, metadata and before/after rates

Releases. The release notes are generated from the provenance files. They carry the not-endorsed label, the dataset caveat, the build and hashes, the environment and the rates table.

Populace: populace-comparison-run-35958762943 (run 35958762943, commit 0d9befa).
eCPS: ecps-comparison-run-35950220750 (run 35950220750, commit 480361f).

The Populace workflow staged its own release. The eCPS workflow's stage-release step got HTTP 403: Resource not accessible by integration from gh release create, including on a retry; the same step succeeded before and after. I staged that draft from the run's own full-data-ecps-* artifacts, with the notes the workflow generates, and checked every CSV's sha256 against its provenance outputSha256. The notes say this. Both releases are now published, and the dashboard's "Full data" buttons point at them.

2025 has ten excluded records. The August Linux taxsimtest build crashes with SIGFPE in mttax24_, its Montana routine, on ten Populace records for 2025. Each record crashes on its own, and one is simply a single filer aged 23 with $25,000 of wages (taxsimid 54307). The macOS build and the Linux build under amd64 emulation both compute these records.

Rather than lose the year, the refresh now bisects a crashing batch, scores every other record, and lists each excluded taxsimid and routine in the summary (metadata.taxsimCrashes), the provenance, the release notes and on the dashboard.
2025 Populace rates therefore cover 79,719 records: taxsimid 53820, 53823, 53824, 53833, 53923, 53924, 54172, 54307, 54380 and 54382 are excluded. No other year or dataset excludes anything.
This is the same kind of failure as September Linux TAXSIM crashes in Maryland routine for 2024–2025 ECPS inputs #1214 (Maryland, September build). An upstream report with these records is a follow-up.

Summary JSON metadata (dashboard/public/data/<dataset>/<year>/summary_<year>.json → metadata):

dataset: id, label, caveat, source file and sha256, records. For Populace it adds buildId, hfRepo, hfRevision, hfCommit, h5File, h5Sha256, dataYear, certifiedBy, conversionModel (pe-us 2.2.1 / core 3.32.5) and converterSha256.
policyengineUsVersion 2.6.17, policyengineCoreVersion.
taxsimtestBuild cd2026081819, taxsimBinarySha256 00a321d2…a89b.
emulatorCommit, scriptSha256, requirementsSha256, batchSize, peakWorkerRssMiB, comparisonNote.

Which code produced each run.

The eCPS run is at 480361f and the Populace run at 0d9befa.
Between them, policyengine_taxsim/ changed only in Code Alabama as TAXSIM state 1 and stop reporting state 0 as Texas #1204's review commit (7d75045). I read that diff: it touches validation of missing or invalid state codes, report formatting and docstrings. Every input here has a valid state from 1 to 51, so both runs use the same calculation code.
Both runs use the August taxsimtest build. Update bundled macOS and Linux TAXSIM test binaries to the October 27, 2026 build #1206, the September binaries, is not in this branch. Its Linux build crashes in Maryland for 2024–2025 (September Linux TAXSIM crashes in Maryland routine for 2024–2025 ECPS inputs #1214; workaround drafted in Recover dashboard refresh with explicit Georgia and Maryland TAXSIM fallbacks #1215). If Update bundled macOS and Linux TAXSIM test binaries to the October 27, 2026 build #1206 lands, the whole benchmark should be rerun on one binary.

Before and after, by year.

"Live" is what the dashboard shows today: the eCPS refresh of 2026-09-21, with Alabama coded 0 and reported as TX.
"eCPS" is the same population rerun with this PR's code; the Alabama fix is the main change.
"Populace" is the new population.
"1%" means within 1% of the gross-income proxy, and "$15" means within $15.

Year	Records (live / eCPS / Populace)	Fed $15 (live)	State $15 (live)	Fed 1% (live)	State 1% (live)	Fed $15 (eCPS)	State $15 (eCPS)	Fed 1% (eCPS)	State 1% (eCPS)	Fed $15 (Pop.)	State $15 (Pop.)	Fed 1% (Pop.)	State 1% (Pop.)
2021	111,347 / 111,347 / 79,729	78.8	71.9	89.3	88.9	78.8	71.5	89.3	88.9	83.2	65.3	96.9	88.6
2022	111,347 / 111,347 / 79,729	81.1	74.0	89.8	88.7	81.1	73.7	89.8	88.7	86.0	64.4	98.1	87.7
2023	111,347 / 111,347 / 79,729	81.1	81.3	89.8	93.2	81.1	81.0	89.8	93.2	86.8	74.3	98.1	93.2
2024	111,347 / 111,347 / 79,729	83.8	83.6	90.9	95.7	83.8	83.3	90.9	95.7	88.4	75.1	96.4	94.3
2025	111,347 / 111,347 / 79,719	83.0	87.3	90.9	96.1	83.0	87.0	90.9	96.2	84.8	78.8	96.4	94.8

The federal difference is the top tail, not better emulation. Without its 11,454 records with $1M+ total income, the eCPS scores 96.3–97.1% federal agreement within 1% (2021–2025). Populace scores 96.4–98.1%.

State agreement within $15 is lower on Populace. In most of the states that moved, the drop sits in records with itemized-deduction inputs, which the eCPS never had (see the table below).

Subgroups (Populace), from the full CSVs, using the same match rules as the summaries:

year                     group     n  fed15  st15  fed1   st1
 2021                       all 79729   83.2  65.3  96.9  88.6
 2021        itemized input > 0 38941   74.4  63.2  95.2  89.6
 2021 TAXSIM itemizes (v17 > 0) 10499   32.0  59.7  92.1  91.7
 2021             otheritem > 0 14108   53.8  55.1  93.8  90.5
 2021       total income >= $1M  1291    3.8  62.1  77.2  86.9
 2021      total income >= $10M     3    0.0  66.7   0.0 100.0
 2022                       all 79729   86.0  64.4  98.1  87.7
 2022        itemized input > 0 38941   78.1  61.6  96.7  88.6
 2022 TAXSIM itemizes (v17 > 0) 10011   34.5  53.1  92.0  89.7
 2022             otheritem > 0 14108   58.4  52.2  94.1  89.2
 2022       total income >= $1M  1291    2.8  68.3  77.2  89.5
 2022      total income >= $10M     3    0.0 100.0   0.0 100.0
 2023                       all 79729   86.8  74.3  98.1  93.2
 2023        itemized input > 0 38941   79.5  71.2  96.7  93.6
 2023 TAXSIM itemizes (v17 > 0)  9031   34.7  61.3  91.6  91.7
 2023             otheritem > 0 14108   60.8  61.0  94.0  92.7
 2023       total income >= $1M  1291    4.3  74.1  77.2  92.4
 2023      total income >= $10M     3    0.0  66.7   0.0 100.0
 2024                       all 79729   88.4  75.1  96.4  94.3
 2024        itemized input > 0 38941   80.7  70.6  94.0  94.0
 2024 TAXSIM itemizes (v17 > 0)  8349   39.8  57.9  86.8  90.8
 2024             otheritem > 0 14108   65.4  57.3  89.6  91.9
 2024       total income >= $1M  1291   48.6  68.7  78.5  93.8
 2024      total income >= $10M     3    0.0 100.0   0.0 100.0
 2025                       all 79719   84.8  78.8  96.4  94.8
 2025        itemized input > 0 38940   74.4  73.3  94.1  94.4
 2025 TAXSIM itemizes (v17 > 0)  9337   16.3  56.2  87.0  90.8
 2025             otheritem > 0 14108   50.7  59.0  89.4  92.0
 2025       total income >= $1M  1290   47.1  68.8  78.4  93.9
 2025      total income >= $10M     3    0.0  66.7   0.0 100.0

Subgroups (eCPS), for comparison. Records with $1M+ total income are where federal agreement is lowest:

year                     group      n  fed15  st15  fed1  st1
 2021                       all 111347   78.8  71.5  89.3 88.9
 2021 TAXSIM itemizes (v17 > 0)     96    4.2  10.4  49.0 97.9
 2021       total income >= $1M  11454    0.7  56.2  28.8 81.8
 2021      total income >= $10M   6867    0.0  54.1  30.5 81.7
 2022                       all 111347   81.1  73.7  89.8 88.7
 2022 TAXSIM itemizes (v17 > 0)    172   40.7   7.6  64.5 76.2
 2022       total income >= $1M  11454    0.7  67.5  28.9 89.0
 2022      total income >= $10M   6867    0.0  66.4  30.5 89.3
 2023                       all 111347   81.1  81.0  89.8 93.2
 2023 TAXSIM itemizes (v17 > 0)    150   39.3   6.7  63.3 57.3
 2023       total income >= $1M  11454    0.7  74.3  28.9 93.4
 2023      total income >= $10M   6867    0.0  73.6  30.5 93.6
 2024                       all 111347   83.8  83.3  90.9 95.7
 2024 TAXSIM itemizes (v17 > 0)    268   17.9   3.4  54.5 73.1
 2024       total income >= $1M  11454    1.4  70.5  36.9 94.5
 2024      total income >= $10M   6867    0.0  69.4  38.4 94.4
 2025                       all 111347   83.0  87.0  90.9 96.2
 2025 TAXSIM itemizes (v17 > 0)    900   15.1  45.0  73.1 87.8
 2025       total income >= $1M  11454    1.3  75.0  36.9 95.8
 2025      total income >= $10M   6867    0.0  74.2  38.4 96.0

Ten states whose rates moved most, eCPS → Populace (same code). States are ranked by the mean, over 2021–2025, of the largest absolute change among the four rates. Values shown are for 2024.
Ranked by the mean over 2021-2025 of the largest absolute change among the four rates. Values are 2024: eCPS -> Populace (live dashboard in brackets; the live dashboard counted Alabama under TX).

State	Mean max move	Records 2024 (eCPS / Pop.)	Fed $15	State $15	Fed 1%	State 1%
AL	17.2 pp	1,155 / 1,411	71.2 -> 88.6 [–]	66.1 -> 60.3 [–]	82.8 -> 96.8 [–]	99.7 -> 97.2 [–]
KY	15.4 pp	1,156 / 1,036	86.3 -> 91.4 [86.3]	96.4 -> 78.7 [96.4]	91.4 -> 96.7 [91.4]	100.0 -> 98.0 [100.0]
DE	15.4 pp	1,300 / 1,591	84.6 -> 89.1 [84.6]	91.1 -> 72.4 [91.1]	90.6 -> 96.7 [90.6]	99.4 -> 98.0 [99.4]
CA	15.0 pp	10,178 / 3,876	83.1 -> 82.9 [83.1]	97.1 -> 78.0 [97.1]	91.0 -> 95.7 [91.0]	98.3 -> 97.6 [98.3]
KS	14.4 pp	1,378 / 1,211	83.5 -> 87.0 [83.5]	98.7 -> 80.4 [98.7]	91.2 -> 96.4 [91.2]	99.5 -> 98.1 [99.5]
MS	13.6 pp	2,006 / 1,243	85.8 -> 91.2 [85.8]	98.7 -> 85.0 [98.7]	92.0 -> 97.7 [92.0]	99.7 -> 99.2 [99.7]
NJ	12.9 pp	2,158 / 2,407	81.9 -> 87.7 [81.9]	94.2 -> 79.1 [94.2]	91.0 -> 96.6 [91.0]	98.5 -> 97.7 [98.5]
DC	12.6 pp	1,998 / 673	84.6 -> 75.5 [84.6]	81.2 -> 59.4 [81.2]	93.4 -> 94.7 [93.4]	92.0 -> 89.7 [92.0]
WI	12.6 pp	1,592 / 1,588	83.1 -> 87.9 [83.1]	89.1 -> 78.1 [89.1]	90.6 -> 96.6 [90.6]	98.4 -> 98.0 [98.4]
CT	12.4 pp	1,216 / 1,294	82.1 -> 88.8 [82.1]	78.1 -> 63.8 [78.1]	89.6 -> 97.7 [89.6]	98.5 -> 98.1 [98.5]

What drives most of the state moves (2024, Populace). State agreement within $15 split by whether the record has an itemized-deduction input. This is observed agreement, not a diagnosis. In most of these states the drop sits in the records the eCPS never tested; CT and OR do not follow the pattern.

state  records  state $15, all  n with itemized input  state $15, with  n without  state $15, without
   AL     1411         60.3                    692          44.1        719             75.9
   KY     1036         78.7                    479          64.7        557             90.7
   DE     1591         72.4                    758          54.9        833             88.4
   CA     3876         78.0                   1948          58.4       1928             97.9
   KS     1211         80.4                    633          64.1        578             98.3
   MS     1243         85.0                    589          70.1        654             98.3
   NJ     2407         79.1                   1138          66.6       1269             90.2
   DC      673         59.4                    355          45.4        318             75.2
   WI     1588         78.1                    812          60.6        776             96.4
   CT     1294         63.8                    653          63.7        641             64.0
  ALL    79729         75.1                  38941          70.6      40788             79.4

Merge order, conflicts and follow-ups

Merge order. Merge Route TAXSIM otheritem into itemized deductions alongside mortgage #1205 and Code Alabama as TAXSIM state 1 and stop reporting state 0 as Texas #1204 first; both are merged into this branch. Code Alabama as TAXSIM state 1 and stop reporting state 0 as Texas #1204 now also rewrites dashboard/public/data/<year>/. This PR moves that data to data/ecps/<year>/ and supersedes it with the same-code eCPS rerun.
Expected conflict. Recover dashboard refresh with explicit Georgia and Maryland TAXSIM fallbacks #1215 (draft, Maryland fallback) edits scripts/refresh_dashboard.py and the refresh workflow. Whichever lands second has to reconcile with DATASETS, --dataset and per-dataset batch sizes.
Follow-ups filed as tasks:
- the emulator drops nonprop;
- the Microsimulation path treats mstat 6 as a two-person unit;
- policyengine-us ignores Populace's tax_unit_role_input and filing_status_input. Every policyengine.py simulation on this build re-infers roles by age.
- diagnose federal gaps among TAXSIM itemizers.
Upstream report. The Montana 2025 crash (records above) still needs to be reported to NBER.
Top-tail fix. When the Microcosm top-income fix ships as a certified build, update POPULACE_BUILD, regenerate the inputs and rerun. The eCPS view can then retire.

Test plan

tests/test_populace_inputs.py, run locally. Converter behavior on synthetic sims:
- filers' incomes and per-unit expenses;
- itemized definitions and dependent ages;
- pinning the source roles (the reviewer's 50-year-old head with a 20-year-old student dependent);
- negative mortgage rejected, uncertified model refused.
Committed-file checks:
- provenance sha256, build, model and converter hash;
- all 51 states including Alabama;
- itemized coverage;
- the documented coverage table;
- the eCPS coverage counts from the brief.
tests/test_dashboard_refresh.py: dataset registry completeness, provenance validation, deterministic state-stratified sample, release notes (label, caveat, hashes, rates, rejection of smoke runs).
tests/test_state_codes.py (Code Alabama as TAXSIM state 1 and stop reporting state 0 as Texas #1204) against the new converter, in the pinned 2.6.17 refresh environment.
Refresh smoke tests and full 2021–2025 runs for both datasets on Actions.
bun run build of the dashboard. Checked the static export in a browser: the eCPS view shows the toggle, caveat, not-endorsed note and hero provenance. The dashboard vitest suite is already broken on main (tests import a nonexistent ../LandingPage); Fix dashboard vitest suite and run it in CI #1160 fixes it.
Two independent reviews (Subfleet review lanes). The second rebuilt all 79,729 Populace records from the H5 and the build's roles with zero mismatches. Findings fixed in this branch:
- source roles;
- Populace batch size;
- a stale-load race on dataset switch;
- a missing dataset showing as 0%;
- placeholder release tags;
- release notes accepting smoke output;
- legacy H5 files in the role reader;
- legacy result CSVs in the loader;
- the converter's hash on Windows;
- a year with no scored records;
- the eCPS caveat wording.
CI matrix (lint, changelog, tests on Ubuntu, macOS and Windows with Python 3.10 and 3.11) on the final head; confirmed before the PR was marked ready.

🤖 Generated with Claude Code

TAXSIM `otheritem` was accepted but silently dropped on the 2021+ PolicyEngine path (#484). The bundled taxsimtest binary treats `otheritem` and `mortgage` identically on the federal return: each is deducted in full on Schedule A (no AGI floor, outside the SALT cap) and neither is added back to AMT income, despite the documentation calling `otheritem` an AMT preference. Sum both into PolicyEngine's `deductible_mortgage_interest`, the input with exactly that treatment. Tests run the bundled binary and the emulator on paired records that move the same dollars between the two columns (2021-2026, SALT-capped and AMT-sensitive cases, a California itemizer). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

The eCPS comparison inputs (cps_households.csv) were built in 4ccfead by vectorized_validation.py, whose FIPS-to-SOI table had no entry for FIPS 1 and fell back to mapping.get(fips, 0). All 1,155 Alabama tax units (taxsimid 88052-89206) therefore had state 0, which TAXSIM reads as "no state tax": neither engine computed Alabama income tax, every one counted as a state match, and get_state_code(0) defaulted to "TX", so the dashboard's TX row held 8,337 = 7,182 Texas + 1,155 Alabama households and Alabama never appeared. Re-running the converter on every candidate eCPS H5 revision confirms all state-0 rows have state_fips 1. - Recode those 1,155 rows 0 -> 1 (no other byte changes). - convert_h5_to_taxsim.py raises on an unknown FIPS instead of using 0. - refresh_dashboard.py reads cps_households.csv, sets the year per run, and rejects state 0, any invalid code, or a missing state before any model runs; it no longer deletes its source. - utils: get_state_code(0) returns None (reporting), get_state_label(0) is TAXSIM's "State not specified", and the Texas proxy lives only in get_calculation_state_code/get_calculation_fips. Output echoes the input state (0 stays 0, as TAXSIM does); invalid codes raise instead of being simulated as California under a TX label; state_breakdown no longer merges state 0 into TX. The R wrapper stops on unknown codes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…ates Pin it with a strict xfail on a Kansas record, correct the AMT-income rule (v26 also nets the QBI deduction), cite Dan Feenberg's #484 description of otheritem as Schedule A line 16, and scope the CE claim to federal tax. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Refresh run 35859275514 on 2b69146: same pinned environment (policyengine-us 2.6.17, policyengine-core 3.32.6), TAXSIM binary and flags as the Sept 21 data, with inputs from the corrected cps_households.csv. Alabama now has its own row (1,155 households) and Texas drops from 8,337 to its 7,182 real households. No non-Alabama household changed any match flag; the only output difference outside Alabama is one PolicyEngine fiitax that moved by at most $0.03. Full CSVs are in the draft release full-ecps-comparison-run-35859275514, which must be published for the dashboard's download links to work. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

The Enhanced CPS behind the benchmark was archived on 2026-07-02. This converts the Populace US 2024 build that policyengine.py certifies (populace-us-2024-spm-20260915) into TAXSIM inputs and makes it the dashboard's default population, keeping the Enhanced CPS as a secondary dataset because Populace lacks its high-income tail. - scripts/convert_h5_to_taxsim.py: vectorized rewrite. Downloads the H5 at its pinned Hugging Face commit, verifies sha256, refuses any model but the certified policyengine-us 2.2.1 / core 3.32.5, and writes populace_households.csv plus provenance JSON. Incomes are the filers'; expenses are summed per tax unit (no household double counting). proptax, mortgage and otheritem follow the taxsimtest definitions, populated for the first time. Units with no filer are dropped. - scripts/refresh_dashboard.py: DATASETS registry and --dataset flag; dataset provenance, taxsimtest build and a not-endorsed note in every summary; deterministic Populace drill-down sample; release notes generated from provenance files. - Refresh workflow takes a dataset input; artifacts and release tags are named by dataset. - Dashboard: dataset toggle (Populace default, ?dataset=ecps), data under public/data/<dataset>/<year>, dataset caveats and the comparison note beside the rates; landing, docs and web-runner copy updated. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…tion

vercel · 2026-09-24T03:10:46Z

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated
policyengine-taxsim	Ready	Preview	Sep 24, 2026 6:47am UTC

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…rate-benchmark-to-populace # Conflicts: # dashboard/public/config-data.json # dashboard/src/constants/index.js

The first 5,000-record Populace worker for 2021 exceeded the 5 GiB budget; its records populate more inputs than the eCPS rows. Batch size now defaults per dataset (DATASETS batchSize); the eCPS keeps 5,000. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

- validate_state_number treats pd.NA and NaT (nullable dtypes) as a missing state, i.e. state 0, as it already did None, blank and NaN; error messages show the exact value (51.0000001, not 51). - The R wrapper sets NA states to 0 before handing the frame to Python, since reticulate passes an integer NA as its raw sentinel by default. - check_source rejects non-integer, blank and NaN state codes (it used to truncate 1.5 to 1) and names the first offending taxsimid. - detailed_report widens its State column for the "State not specified" row and lists it after the real states. - Tests: comparator state_code is blank for state 0; api._validate_csv rejects 52, -1 and 0.5 and accepts 0 and blank; nullable missing state runs as state 0; -1 is tested separately (TAXSIM expands it, the emulator does not support it). - Docs and comments: TAXSIM stops at the first invalid code (it does not reject -1); Alabama was coded 0 until September 23, 2026; the source lacks Alabama's CPS-income half (d317b05); sources are no longer deleted. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

- Converter: pin PolicyEngine's is_tax_unit_head/spouse/dependent and filing_status to the build's tax_unit_role_input and filing_status_input before calculating. policyengine-us has no variable that reads them and takes the two oldest adults as head and spouse, so adult dependents (5,343 in the build) became spouses and 252 lone teenage heads became filer-less units. - Dashboard: ignore superseded dataset loads, so a slower earlier load cannot replace the selected dataset; treat a dataset with no summary or records as missing instead of 0% agreement. - Release notes refuse partial (smoke) provenance. - Point the Populace downloads at the rerun's release tag. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…rate-benchmark-to-populace # Conflicts: # docs/dashboard-refresh.md # scripts/refresh_dashboard.py

…turns) All 79,729 tax units are kept: with tax_unit_role_input pinned, every unit has a head, and the 35,592 joint returns match the build's JOINT filing status exactly. Counts in the dashboard docs, web runner and tests updated. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Same emulator commit as the Populace run's inputs (480361f: #1204, #1205, main), August taxsimtest build. Alabama is scored as state 1. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Windows CI checked the CSVs out with CRLF endings, so their sha256 no longer matched the provenance. Mark the hashed input CSVs -text, normalize line endings when hashing the converter, and read text files as UTF-8. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

The August Linux taxsimtest build died with SIGFPE in mttax24_ (taxsim.f:20897, Montana 2024+) on a 2,000-record Populace batch for 2025; the same batch runs cleanly on the macOS build and under amd64 emulation, like the Maryland crash in #1214. When the binary dies from a signal, the worker now reruns the batch in halves until each crash is pinned to one record, scores every other record, and records each excluded taxsimid and routine in the part checkpoint, the summary and provenance metadata (taxsimCrashes), and the release notes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

- Converter: return no source roles for variable-centric (legacy) H5 files instead of failing; inputs regenerated byte-identically. - Keep the converter script LF on every platform so its recorded hash holds on Windows regenerations. - Refuse to summarize a year in which no record could be scored. - Dashboard: treat legacy result CSVs as data, keep the dataset toggle on the error screen, and word the eCPS caveat precisely (its three deduction inputs are zero; TAXSIM still itemizes some records). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Five years of summaries and drill-down samples for the default dataset. 2025 scores 79,719 of 79,729 records: ten Montana records crash the August Linux taxsimtest build (SIGFPE in mttax24_) and are listed in the summary metadata. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

MaxGhenis and others added 7 commits September 22, 2026 23:49

Merge #1205 (otheritem itemized routing) for the Populace benchmark run

73f48f8

Merge main (Maine rent-utilities flag, #1128) into the Populace migra…

480361f

…tion

MaxGhenis and others added 2 commits September 23, 2026 23:13

Point the dashboard's dataset downloads at the refresh runs' releases

d1a1764

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Merge remote-tracking branch 'origin/fix-alabama-state-zero' into mig…

dd16bfd

…rate-benchmark-to-populace # Conflicts: # dashboard/public/config-data.json # dashboard/src/constants/index.js

vercel Bot deployed to Preview September 24, 2026 03:14

View deployment

vercel Bot deployed to Preview September 24, 2026 03:16

View deployment

MaxGhenis and others added 4 commits September 23, 2026 23:33

Merge remote-tracking branch 'origin/fix-alabama-state-zero' into mig…

a8d7aee

…rate-benchmark-to-populace # Conflicts: # docs/dashboard-refresh.md # scripts/refresh_dashboard.py

vercel Bot deployed to Preview September 24, 2026 03:41

View deployment

Refresh the Enhanced CPS dashboard data (run 35950220750)

5d3cc9d

Same emulator commit as the Populace run's inputs (480361f: #1204, #1205, main), August taxsimtest build. Alabama is scored as state 1. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

vercel Bot deployed to Preview September 24, 2026 04:05

View deployment

State each dataset's caveat in its summaries and release notes

f796f54

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

vercel Bot deployed to Preview September 24, 2026 04:19

View deployment

vercel Bot deployed to Preview September 24, 2026 04:20

View deployment

vercel Bot deployed to Preview September 24, 2026 04:53

View deployment

vercel Bot deployed to Preview September 24, 2026 05:10

View deployment

MaxGhenis and others added 2 commits September 24, 2026 02:45

Say on the dashboard which records taxsimtest could not score

1969b6c

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

vercel Bot deployed to Preview September 24, 2026 06:47

View deployment

MaxGhenis marked this pull request as ready for review September 24, 2026 07:20

Navigation Menu

Repository navigation

Migrate the TAXSIM benchmark and dashboard to the certified Populace US build - #1216

Migrate the TAXSIM benchmark and dashboard to the certified Populace US build#1216
MaxGhenis wants to merge 21 commits into
mainfrom
migrate-benchmark-to-populace

MaxGhenis commented Sep 24, 2026 •

edited

Loading

Uh oh!

vercel Bot commented Sep 24, 2026 •

edited

Loading

Uh oh!

This branch was successfully deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

MaxGhenis commented Sep 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Data build and model versions

Input definitions

Check 1: top-income coverage (Populace lacks the tail)

Check 2: itemized deductions are now tested

Check 3: every state, including Alabama

Check 4: releases, metadata and before/after rates

Merge order, conflicts and follow-ups

Test plan

Uh oh!

vercel Bot commented Sep 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

This branch was successfully deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

MaxGhenis commented Sep 24, 2026 •

edited

Loading

vercel Bot commented Sep 24, 2026 •

edited

Loading