Skip to content

Add ACS 2020–2024 county income source packages - #386

Merged
MaxGhenis merged 5 commits into
mainfrom
acs-5yr-county-income-2024
Oct 24, 2026
Merged

MaxGhenis merged 5 commits into
mainfrom
acs-5yr-county-income-2024

Conversation

@MaxGhenis

@MaxGhenis MaxGhenis commented Oct 24, 2026 •

Copy link
Copy Markdown
Contributor

Adds seven Census ACS 2020–2024 5-year county source packages, with publisher estimates and margins of error kept as separate facts. The 2024 Census API dataset exists, so no 2023 fallback was needed.

Status: implementation and validation complete; retained as a draft for maintainer review. Three commits pin the publisher artifacts, add exact source-column bound validation, and add the generated packages and county tests.

The supplied 2026-10-24 publisher JSON files were copied unchanged, with SHA-256 pins in seven data manifests and FETCH-MANIFEST.json. No county responses were re-fetched. Source URLs contain no API credentials. R2 fields are content-addressed locators; this PR does not assert that bytes were uploaded.

Table Source counties, excluding PR Numeric estimate cells Numeric MOE cells
B01003 3,144 3,144 130
B11001 3,144 9,432 9,432
B19001 3,144 53,448 53,448
B19013 3,144 3,143 3,143
B19025 3,144 3,144 3,144
B19051 3,144 9,432 9,432
B19061 3,144 3,144 3,144

All seven publisher responses contain 3,222 county rows including Puerto Rico. Full source rows preserve those responses; facts exclude state 72. County GEO_ID and Census NAME are preserved. B11001 includes total, family, and nonfamily households. B19001 includes its total and all 16 brackets, with income bounds in fact constraints and package metadata; the first and last brackets have unbounded endpoints.

Publisher special cases:

  • B01003 has 3,014 MOE codes -555555555 with annotation ***** (MOE not appropriate for a controlled estimate). They remain in source rows/cells and unavailable-cell metadata, and are not replaced with zero.
  • B19013 suppresses De Baca County, New Mexico (35011): estimate -666666666 / annotation -, MOE -222222222 / annotation **. Those cells are preserved, with no numeric estimate or MOE fact invented. Therefore the table covers 3,144 source counties and 3,143 numeric counties.
  • No table failed ingestion. There are 166,760 numeric facts and 3,016 preserved unavailable cells.

Verified properties:

  • Each source table covers all 3,144 county-equivalents in the 50 states and DC; Puerto Rico is absent from facts.
  • Connecticut uses exactly its nine planning regions 09110–09190, with no legacy county codes.
  • Each county’s 16 B19001 brackets sum to its B19001 total.
  • B19051 with earnings plus no earnings equals its total; B11001 family plus nonfamily equals its total.
  • Every emitted numeric estimate/MOE is nonnegative and equals its exact publisher cell, with row/cell lineage and pinned artifact provenance.
  • Facts retain end-year 2024 identity plus explicit 2020-01-01–2024-12-31 survey reference coverage, vintage acs_5_year_2020_2024, and publisher 2024-dollar income definitions.
  • No aging, imputation, cross-source reconciliation, consumer target selection, or solver construction is introduced.

The generator reuses the existing full JSON-table parser. The small acceptance-validator change recognizes exact typed lower/upper bounds declared on guarded wide-table columns, matching existing row-bound evidence. It does not change schemas, fact values, identities, or periods. MOE concepts and statistic constraints distinguish them from estimates; metadata explicitly prohibits adding MOEs across categories/counties.

axiom: n/a: source data package

Approved Chronicle agent role: ledger-source-ingestor.

Deterministic checks and tests:

  • python -m chronicle.sources.census_acs5_county --check: all seven packages reproduce byte-for-byte.
  • County tests: 61 passed, serial, including source-package/core/consumer/agent acceptance, exhaustive numeric cell preservation, county inventories, identities, bounds, suppression, unsupported years, and malformed-source rejection.
  • Source-suite tests: 80 passed.
  • Bundle regressions: 24 passed; the full merged-bundle test skipped locally because unrelated R2-only artifacts are unavailable.
  • Broader seven-file run with -n 3: 260 passed, one unrelated LIHEAP PDF cell-count failure with newer unpinned pypdf 6.21.2. Both LIHEAP cases passed when rerun with the repository’s frozen dependencies (pypdf 6.11.0); no unrelated code was changed.
  • Repository-wide Ruff check passed using the lockfile version 0.14.10; diff checks passed.

LLM judge verdicts:

  • ledger-source-fidelity: PASS. Independent exhaustive reconstruction matched all 166,760 numeric declarations and all 3,016 unavailable encodings to publisher bytes, hashes, names/GEOIDs, bounds, and period/vintage.
  • ledger-boundary: PASS. Independent review includes the source-column bound validator; no computed or consumer-owned values are introduced.
  • ledger-contract: n/a; no contract/schema changes.

MaxGhenis and others added 5 commits October 23, 2026 21:18
…for 2023

source_package_count counts built packages only; the seven ACS county
packages have no 2023 artifact and land in skipped_sources (17 -> 24).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The ledger-source-fidelity and ledger-boundary judges passed at cc7d6f2
with two findings, both applied here:

- Puerto Rico's 78 municipios are now emitted like every other county row
  (3,222 rows: 3,144 non-PR counties plus 78), matching acs_s0101_state_2024
  and acs_s2201_district_2024. Scope selection belongs to consumers.
  PR's 76 sentinel B01003 MOEs (-555555555 with *****) go to
  unavailable_cells and never become facts.
- B19013's margin of error now carries aggregation "median" like its
  estimate.

All seven source_package.yaml files were regenerated with the generator;
--check passes twice and the raw JSON SHA-256 pins are unchanged. Facts:
166,760 -> 170,896.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@MaxGhenis

Copy link
Copy Markdown
Contributor Author

Judges at 517ee9d: ledger-source-fidelity PASS, ledger-boundary PASS (Subfleet run 20261023-232551-lwi-j2-chr386). Their one should-fix was that the nine mutation tests couldn't create temp directories in the judge sandbox. I reran tests/test_chronicle_census_acs5_county.py locally with a writable TMPDIR: 65 passed, including all nine mutation tests.

@MaxGhenis
MaxGhenis marked this pull request as ready for review October 24, 2026 05:21
@MaxGhenis
MaxGhenis merged commit b2b5d52 into main Oct 24, 2026
3 checks passed
@MaxGhenis
MaxGhenis deleted the acs-5yr-county-income-2024 branch October 24, 2026 06:03
MaxGhenis added a commit that referenced this pull request Oct 24, 2026
…ort sources

The one conflict is in tests/test_chronicle_bundle.py's source-package
pins. This branch adds built packages, so it keeps source_package_count
at 415. #386's seven county packages have no 2023 artifact and are
skipped, so main's skipped_source_count of 24 and its skipped-source
list are taken. The diff from the reviewed head 49809b6 is exactly
#386's diff from 200bf6d.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
MaxGhenis added a commit that referenced this pull request Oct 24, 2026
The package adds 565 government facts at tax_year:2025, one per existing
commune key, so geography_count is unchanged and commune:11002 goes from
9 to 10. It adds one source, one source table and one built package.
SPF spells nine communes unlike Statbel, and the package keeps SPF's
wording, so nine commune geography-name warnings are added (123 -> 132).

The deltas were measured with Chronicle's own bundle functions. A replay
of the full bundle test against #385's CI bundle, with #386's seven
skipped county packages and this package added, passes, and main's
unchanged test fails against it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
MaxGhenis added a commit that referenced this pull request Oct 24, 2026
…ages

The two packages add 34,881 person facts at country:BE: 34,629 quarterly
daily-wage-class cells from 2023 Q1 to 2026 Q1, and 252 annual key
figures for 2020-2025. They add two sources, two source tables and two
built packages. They add no new geography, period or entity key and no
warning.

This stacks on #387's pins. The deltas were measured with Chronicle's
own bundle functions and confirmed against #388's CI bundle on the old
base. A replay of the full bundle test against #385's CI bundle, with
#386's skipped packages, #387's package and these two packages added,
passes, and #387's test fails against it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
MaxGhenis added a commit that referenced this pull request Oct 24, 2026
The package adds 35 calendar-2025 facts: 19 person, 2 benefit_unit
premium payments and 14 government expenditure. It adds one
statistical_scope geography, the German-speaking Community, plus one
source, one source table and one built package. It adds no warning.

This stacks on #388's pins. The deltas were measured with Chronicle's
own bundle functions and confirmed against #389's CI bundle on the old
base. A replay of the full bundle test against #385's CI bundle, with
#386's skipped packages and #387's, #388's and this package added,
passes, and #388's test fails against it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
anth-volk added a commit that referenced this pull request Oct 26, 2026
…n-value

Brings in main at 272c987: FSA portfolio and ACS S1501 (#347), the certifi
lock (#378), NZ QES earnings and PREFU 2026 (#382), UK excise facts (#384),
Statbel commune population (#383), Belgian exchange-rate and pension sources
(#385), ACS 5-year county income (#386), SPF Finances commune surcharge
(#387), RSZ wage classes (#388), Ostbelgien Kindergeld (#389) and GSGB 2025
gambling participation (#391).

One file conflicted, tests/test_chronicle_bundle.py, in the merged-bundle
warning pins:

- warning_count: main's 133 is 91 geography-name warnings, 24 groupby value
  label warnings, 17 cross-package value label warnings and one duplicate
  semantic key. This branch turns the value label warnings into errors, so
  the pin is 92 (91 + 1). Both sides' comments are kept, and the #376 comment
  now says it removes 41 (17 + 24).
- The warning-code Counter keeps main's 91 geography-name warnings and drops
  the two value label codes, as this branch does.
- The list of cross-package value label warnings (main's, now with freq=A)
  is replaced by this branch's assertion that no value label error or
  warning remains.

Every other count pin is main's: this branch adds no fact.

The 92 holds only once freq=A and GSGB's national_lottery_draws_in_person
each have one label, which this merge does not choose.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
anth-volk added a commit that referenced this pull request Oct 26, 2026
After merging main (dece5e9), 3,238 dimension values are emitted by two or
more default-bundle packages and worded one way, and the register did not
list them. Each takes its one wording, as the register's rule says; no
existing entry changes or goes:

- census_acs.county: 3,222 county codes from the seven ACS 5-year county
  packages (#386);
- statistic estimate and margin_of_error (ACS county packages, ONS families
  and households);
- NZ QES and PREFU earnings_component, earnings_denominator,
  earnings_sector, earnings_series_reference and seasonal_adjustment (#382);
- UK published_unit (gbp_million, million_sticks, thousand_kilograms) and
  accounting_basis cash (#384);
- age_band age_25_to_34 and age_65_to_74 (GSGB 2025 and HMRC CGT by age);
- time 2025 (the two 2025 Eurostat packages).

The register now holds 8,062 values in 155 dimensions (4,824 in 147
before). BUILD_YEARS adds the 14 new packages with no 2023 artifact (the
eight ACS packages at 2024, the six FSA portfolio packages at 2026), and
the register records dece5e9 as the commit its packages were built at.

Four values worded more than one way still need a chosen label, so the
script cannot yet reproduce this file and --static-check reports two
declarations the register contradicts: freq=A, income_range=all,
gambling_commission.gsgb_activity=national_lottery_draws_in_person and
salary_sacrifice_threshold_position=all.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant