Repository navigation
Conversation
…1 state cells (#297) The fiscal-refresh build can join each person's CPS ASEC A_HGA from the pinned Census person member (exact income year and PERIDNUM, with the Census household, line and age checked) and classify it into the seven ACS S1501 population-25-and-over classes as cps_educational_attainment. The stored-input register flags the column as auxiliary: carried for conditioning and calibration, not consumed by rules. The US target compiler binds the S1501 state 25+ detail cells as person counts of the column, and the tool's materializer counts them. The pinned Chronicle feed has no S1501 facts, so nothing binds until PolicyEngine/chronicle#347 merges and the feed is re-pinned. experiments/modal_297.py runs the tests and the receipt on Modal; only it reads the draft package, from the fork. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…Z-e2621e231) Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part of #297 (the student debt pilot, Project 2: PolicyEngine/us-student-debt#1, ROI task us-student-debt#5).
Summary
The US fiscal-refresh build can now carry educational attainment as an auxiliary person column,
cps_educational_attainment, beside the raw CensusA_HGAit is classified from. The US target compiler binds the ACS 1-year S1501 state cells for the population 25 and over as person counts of that column, and the tool's materializer counts them. Nothing binds in a release yet: the pinned Chronicle feed has no S1501 facts, and they arrive with PolicyEngine/chronicle#347.--cps-educational-attainment(opt-in) joins each person to the pinned official CPS ASEC person member on (income year,PERIDNUM) and checks the Census household (source_household_id=PH_SEQ),P_SEQ,A_LINENOandA_AGE. It never predicts or defaults a value, and a PUF support clone takes its source person's code.--asec-attainment-source INCOME_YEAR=PATHsupplies local archives; unmapped years are fetched and verified against the same pins (spm_role_source.ASEC_SPM_ROLE_SOURCES). A signal gate refuses missing or unknown values, a universe mismatch, and implausible below-diploma or bachelor's-or-higher shares.PERIDNUM).Classes and code mapping
Seven classes, the S1501 population-25-and-over categories (Chronicle value ids reused), plus
not_in_universe. Merging the first two gives the six classes the issue asks for (US_EDUCATIONAL_ATTAINMENT_SIX_CLASSES).A_HGASCHLless_than_9th_grade9th_to_12th_grade_no_diplomahigh_school_graduate_includes_equivalencysome_college_no_degreeassociates_degree(occupational 41 + academic 42)bachelors_degreegraduate_or_professional_degreenot_in_universeSources checked code for code:
A_HGA, "Item 18h - Educational attainment": the ASEC public-use data dictionaries for survey years 2023-2026,https://www2.census.gov/programs-surveys/cps/datasets/<YYYY>/march/asec<YYYY>_ddl_pub_full.pdf(PDF pages 22, 23, 24, 28). All four list 0 and 31-46 identically. In the three pinned members,A_HGA == 0exactly whenA_AGE < 15.SCHL:https://www2.census.gov/programs-surveys/acs/tech_docs/pums/data_dict/PUMS_Data_Dictionary_2024.txt.S1501_C01_006E-013E(25 and over) and001E-005E(18 to 24).The
SCHLmap is defined and tested, but the ACS-local lane does not readSCHLfrom PUMS yet (out of scope here).ROI classes (
US_DEGREE_ROI_CLASSES): two-year degree =associates_degree, four-year degree =bachelors_degree, each against thehigh_school_graduate_includes_equivalencybaseline.Column choice
A Microcosm auxiliary column, not a policyengine-us input: the analysis joins it on
person_id, and no policyengine-us change is needed. Recorded on #297: #297 (comment).For the published default
populace-us-2024-spm-20260915, the receipt run also wrote aperson_id, A_HGA, cps_educational_attainmentsidecar (166,321 rows, sha25634994413e255524f16ab9d985bd8ca8470deab1149b3ee395d2ee307c9d7e62b). It is in the run's Modal volume (microcosm-297-cache,results/20261022T171517Z-e2621e231/) and is not committed, so the ROI work can start before a new build exists.Provenance flag
microcosm.data.stored_inputs.US_AUXILIARY_STORED_COLUMNSis a new register of auxiliary columns, carried for conditioning and calibration and not consumed by rules. It is merged intoUS_STORED_NON_VARIABLE_COLUMNS, so the stored-input gates accept the column. Its reason starts with "Auxiliary:" and names its producer, and the register tests bind it toeducational_attainment.US_CPS_EDUCATIONAL_ATTAINMENT_COLUMN. The stage's receipt (us_cps_educational_attainment.jsonin the build artifacts) recordscolumn_role: auxiliary.Targets and compile path
fiscal_targets._acs_attainment_reference_from_factadmitscensus_acs.acs1_<year>.s1501.state_attainment_25_plus.<st>detail rows for the 50 states and DC. The family iscensus_educational_attainment, the materializerpopulation_attainment, and the person count is taken over the cell's half-open age band. It leaves the universe totals unbound, because PEP binds population by state and age. It also leaves the 18-to-24 universe unbound (the ACS counts dormitory residents where they study), and Puerto Rico. A changed category, age band or measure raises instead of binding._population_attainment_household_valuescounts the cell, andpopulation_attainmentjoins the materializers that accept a restated age bound. A frame without the column refuses to materialize these targets rather than binding zeros.fiscal_target_references.json's CD-to-state reconciliation rule matches oneducational_attainment_categorytoo, since one state's seven rows differ only in class.census_acs.educational_attainment.State totals vs S1501 (receipt)
experiments/297-cps-educational-attainment-receipt.md(JSON:experiments/receipts/297-cps-educational-attainment-s1501.json). The stage ran on the published default at its calibrated weights; the S1501 facts were compiled through the real compiler (357 targets, 0 mismatches against the raw Census response) and materialized with the tool's own code.National, 25 and over:
less_than_9th_gradein small states. A standalone per-state entropy calibration to the cells, which is not the release solve, fits exactly. It keeps at least 82.6% of each state's Kish ESS, but its largest weight ratio is 131 (LA). Merging the two below-diploma classes thickens those cells.What waits on chronicle#347
This PR waits only on PolicyEngine/chronicle#347 merging and the next Chronicle release, which carries the S1501 package
census/acs_s1501_state_2024. Its raw file is already on R2 at #347's head81e3f986.us/chronicle_feed.json,chronicle_feed_scope.json) adds the S1501 pairs, and the parity manifest regenerates and declarescensus_acs.acs1_2024compiled. A release that compiles those rows must run with--cps-educational-attainment.Everything else here works without it. Temporary draft input: only
experiments/modal_297.pyreads the draft package, from the fork (DTrim99/chronicle@81e3f986, labelled TEMPORARY in the file). The facts there are content-identical to a local build at4448a7cand at81e3f98. No lockfile, feed pin or package resource points at the fork.SCF education-debt imputation (#1073 / PR #1246), note only
#1073's hold-out already tried SCF
educas a predictor.build_plus_educationraised household rank correlation from 0.168 to 0.207 but worsened the share-with-debt error from -0.30 pp to -0.92 pp. This column supplies the host side: map SCFeducand the host reference person's class to the same seven classes. The share error argues for adding it alongside the FSA borrower-count calibration (#1074), not as a bare predictor. #1246 is unchanged.Modal runs (
experiments/modal_297.py, appmicrocosm-297-experiments, volumemicrocosm-297-cache)20261022T171517Z-e2621e231(fc-01M5J9R996YS9K51QVQQAFTVZ1), at this branch's head: tests (the new module, fiscal targets, the stored-input register, the Chronicle feed, labels, the data shard's stored-input tests, the builder's attainment/age/ledger-filter subset,ci_test_plan.py verify), then the engine-contract register, spec-bundle and bundle-contract tests, then the receipt. All three passed (tests 420 s, contract 489 s, receipt 123 s; peak 4.6 GB). The receipt is committed in the follow-up commit.20261022T160713Z-d25305d52(fc-01M5J5VMMRPHA6206FSPZBFQPC): the probe of the host and the Census members that set the identity checks. The host'sPH_SEQis globalized, andsource_household_idcarries Census's. Every hostPERIDNUMis found in its year's member.Notes
+ 1here,+ 4there) and the producer-bound set. Both are additive.🤖 Generated with Claude Code