US_CDC_PRAMS : Import Automation - #2205
Open
kartik-s21 wants to merge 69 commits into
Open
kartik-s21 wants to merge 69 commits into
kartik-s21 wants to merge 69 commits into
Conversation
Code fix unenergy
…s, and fix pipeline scripts
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Code Review
This pull request establishes import automation for the US CDC Pregnancy Risk Assessment Monitoring System (PRAMS) dataset. The changes introduce a manifest, validation configurations, and golden datasets, while refactoring the download and processing scripts to use robust HTTP sessions and modern pandas methods. Unit tests have also been updated to use class-level setup and teardown. The review feedback suggests making the confidence interval string-splitting logic more robust against variable spacing and improving exception handling and caching during ZIP file downloads.
…tures
- Address CRA review findings for US CDC PRAMS import:
* Drop suppressed data ('.' or '(.-.)') instead of converting to '0.0'
* Fix CI regex to handle percentages >= 10.0% and prevent parenthesis truncation
* Use named regex capturing groups for confidence interval column extraction
* Add ScalingFactor=100 to lower and upper confidence interval limits
* Update provenance and download URLs to canonical active endpoint
* Implement atomic file downloads and artifact publishing (.tmp + os.replace)
* Standardize logging with absl.logging.fatal
* Add State and National test coverage in test fixtures and streamline datasets
* Document Java 8+ prerequisite and automation cadence in README.md
…tigial 2018 lines, and test actual csv data
…nicode dashes in CI regex, safe boolean masking for Postpartum, validate unmapped SVs, add deduplication, clean up download logic, and document refresh procedure
kartik-s21
requested review from
balit-raibot,
pulkeet-wq and
rohitkumarbhagat
September 22, 2026 23:06
…, fix deletion check description, add validation_config.json and resource_limits to manifest, resolve exception contracts in helper functions, reformat long lines <= 100 chars, and enforce POSIX EOF newlines
…2016-2022) - Transition from legacy 2016-2020 PDF parser (tabula-py) to pure Python openpyxl Excel ingestion. - Dynamically discover and download official consolidated multi-year PRAMS Excel workbooks from the CDC landing page. - Extend dataset coverage through observation year 2022 (45,524 total observations), preserving 2021 production data without regression. - Update manifest.json, validation_config.json, and Google3 metadata to reflect the 2016-2022 observation date range. - Replace PDF fixtures with lightweight Excel test fixture (< 50 KB) and verify hermetic unit tests pass in 0.16s.
…rsing, unmapped SV logging fix, dynamic date freshness, and dead dependency cleanup
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Modernizes and automates the
US_CDC_PRAMSimport pipeline to ingest official consolidated multi-year Excel workbooks, expanding observation coverage from 2016 through 2022 (45,524 observations) across all 50 states, NYC, Puerto Rico, and aggregated sites.This replaces the legacy Java/Tabula PDF scraper with a pure Python
openpyxlpipeline, eliminates external JVM runtime dependencies, resolves historical Confidence Interval scaling factor discrepancies (ScalingFactor: 100.0), establishes automated dynamic workbook discovery for future years, and implements robust header validation and comma-safe numeric parsing.Key Changes
1. Pipeline Modernization (
scripts/cdc_prams/process.py)openpyxl), expanding date coverage from 2016–2020 to 2016–2022._validate_sheet_headers()to verify row 5 sub-headers (Denominator/Sample Size,Weighted %,Lower 95% CI,Upper 95% CI) before ingestion, preventing silent data corruption if CDC layout shifts._parse_float()to strip string commas (e.g.,"1,200"), whitespace, and handle missing markers (na,-,*) cleanly with debug logging..map(), ensuring error logs and exceptions display actual variable names instead of[nan].ScalingFactor: 100.0across percentages and Confidence Interval lower/upper limits.PRAMS.csv,PRAMS.mcf,PRAMS.tmcf) to.tmpfiles and moves them atomically withos.replace().2. Download Stage (
scripts/cdc_prams/download_input_files.py&download.py)discover_excel_url(), dynamically discovers the latest multi-year workbook from the CDC landing page and logs explicit errors upon mismatch or network failures rather than silently falling back._get_session()withurllib3.util.retry.Retry(5 retries with exponential backoff on 429, 500, 502, 503, 504) and shared across download scripts.3. Configuration & Validation (
validation_config.json&manifest.json)check_all_statvars_freshnessto dynamically evaluate againstCURRENT_DATE(CAST(SUBSTRING(CAST(MaxDate AS VARCHAR), 1, 4) AS INT) >= (EXTRACT(YEAR FROM CURRENT_DATE) - 4) AND MaxDate >= '2022')."manifest.json"to"source_files"inmanifest.json."US CDC Pregnancy Risk Assessment Monitoring System (PRAMS) MCH indicators across states, territories, and cities.".tabula-pyfromimport-automation/executor/requirements.txtand dead constant_YEAR/ unused imports fromconstants.py.Test Environment & Latest Cloud Batch Job Run
2026_09_24T21_09_14_424492_07_00Artifact Links
Validation Suite Results
check_all_statvars_freshnesscheck_max_date_consistentcheck_empty_importcheck_lint_error_countcheck_missing_refs_countcheck_deleted_records_percentscalingFactor: 100.0harmonization (0 real data loss)Execution Performance
Linked Artifacts