Add new Commerce EDA Poverty import

[shvngisingh]

[PR #2178] Commerce EDA Poverty: Automated Persistent Poverty County Import

1. Overview

This PR introduces an automated, end-to-end import pipeline for the Commerce EDA Persistent Poverty Counties (PPC) dataset (statvar_imports/commerce_eda_poverty/), decoupling the import from legacy manual Google Cloud Storage staging and transitioning to an autonomous, web-sourced ingestion pipeline.

The dataset provides benchmarked county-level poverty percentage rates published by the U.S. Economic Development Administration (EDA) across statutory periods:

• 1990: 1990 Decennial Census (poverty_rate_1990)
• 2000: 2000 Decennial Census (poverty_rate_2000)
• 2020: 2020 Island Areas Decennial Census (poverty_rate_2020 for territories 60, 66, 69, 78)
• 2021: 2017–2021 ACS 5-Year Estimates (poverty_rate_2021 for all 50 states, DC, and Puerto Rico)

---

2. Key Architecture & Pipeline Components

• Automated Web Scraper & Downloader (download_poverty.py / download.py):
  • Downloads the official EDA_FY23_PPCs.xlsx workbook directly from EDA (https://www.eda.gov/sites/default/files/2023-03/EDA_FY23_PPCs.xlsx).
  • Automatic fallback to Wayback Machine archive mirror if primary web server / bot protection blocks automated requests.
  • Supports --input_file for local workbook or CSV ingestion.
  • Automatically extracts the Underlying_Data worksheet (3,241 raw county rows) into input_files/Poverty.csv and stages output/Poverty_original.csv.
• Preprocessing & Cleaning (process_poverty.py):
  • Dynamically resolves source files (Excel or CSV) and header rows.
  • Standardizes geographic identifiers to 5-digit county FIPS (geoId/XXXXX) and 2-digit island territory codes (geoId/XX).
  • Filters out state summary aggregates (rejecting XX000 FIPS).
  • Partitions recent rates into 2020 (island territories) and 2021 (states, DC, PR) based on survey metadata.
  • Validates percentage rates within $[0.0, 100.0]$ and outputs output/Poverty_cleaned.csv.
• StatVar Processing (stat_var_processor.py):
  • Maps to the canonical Data Commons statistical variable:
    Count_Person_BelowPovertyLevelInThePast12Months_AsFractionOf_Count_Person
  • Generates 9,689 observation nodes across 3,232 places into output/Poverty_output.csv and output/Poverty_output.tmcf.
• Automated Schedule (manifest.json):
  • Configured for weekly automated execution (cron_schedule: "30 05 * * 1").

---

3. Dataset & Statistical Variable Details

Attribute	Details
Statistical Variable	dcid:Count_Person_BelowPovertyLevelInThePast12Months_AsFractionOf_Count_Person
Population Type	dcid:Person
Measured Property	dcid:count
Measurement Denominator	dcid:Count_Person
Poverty Status	dcid:BelowPovertyLevelInThePast12Months
Unit / Scaling	Percent / 100
Geographic Coverage	3,232 places (50 states, DC, Puerto Rico, American Samoa, Guam, Northern Mariana Islands, US Virgin Islands)
Total Observations	9,689 SVObs
Value Range	0.0% – 83.3%

---

4. Validation Configuration (validation_config.json)

The validation config enforces dataset integrity, geographic completeness, and bounds:

1. check_percent_min_value & check_percent_max_value: Verifies poverty percentages are within $[0.0, 100.0]%$.
2. check_num_places_count (NUM_PLACES_COUNT): Asserts county coverage is between 3,100 and 3,250 places (safeguard against unintended data truncation).
3. check_num_observations_check (NUM_OBSERVATIONS_CHECK): Asserts total observation count across benchmark years is between 9,000 and 10,000.
4. check_max_date_consistent: Ensures MaxDate is uniform across places.
5. check_date_span_sql (DuckDB SQL_VALIDATOR): Asserts MinDate = 1990 and MaxDate BETWEEN 2021 AND 2026.
6. check_missing_refs_count: Enforces 0 unresolved entity or schema references.
7. check_lint_error_count: Enforces 0 MCF/TMCF lint errors.
8. check_deleted_records_percent: Allows at most 0.1% deleted records across refreshes.

---

5. Review Comments Addressed

• Full Geographic Coverage: Fully restored dataset to 3,232 counties across all benchmark periods, removing hardcoded sampling and adding hermetic test fixtures (test_data/Poverty_input.csv, test_data/Poverty_expected_output.csv).
• Decoupled from GCS: Built standalone downloader with retry and mirror failover; eliminated hard dependency on manual GCS staging.
• Survey Year Partitioning: Separated 2020 Island Area Census estimates from 2021 ACS 5-Year state estimates.
• Cleaned PVMap & Direct Schematization: Mapped variableMeasured directly to dcs:Count_Person_BelowPovertyLevelInThePast12Months_AsFractionOf_Count_Person in poverty_pvmap.csv.
• Code Health & Linting: Resolved all lint issues (unused imports, line wrapping, docstrings); process_poverty.py rated 9.80/10 with 0 warnings.

---

6. Verification & Testing

• Unit Tests: 22/22 unit tests pass in 2.2s (./run_tests.sh -p statvar_imports/commerce_eda_poverty):
  • Download retry logic, empty body error handling, mirror failover, and local file ingestion.
  • GEOID cleaning, survey year validation, edge cases, out-of-bounds sanitation, and full fixture processing.
• Data Commons Import Linter / GenMCF:
  • 0 fatal, 0 error(s)
  • 9,689 rows processed, 87,201 property-value pairs validated, 9,689 nodes resolved.

[gemini-code-assist]

Code Review

This pull request introduces a new data importer for the Commerce EDA Poverty dataset, which tracks Persistent Poverty Counties. It includes metadata, mapping files, generated outputs, and a preprocessing script (process_poverty.py). The review feedback identifies two key issues: a hardcoded absolute project path in the preprocessing script that will cause failures in other environments, and an incorrect place type (AdministrativeArea1 instead of County) in the commented-out metadata configuration.

[saanikaaa]

If this is new import, we should have a CL as well to bring textproto. Mcf and dataset info about this import