added validation_config.json WorldDevelopmentIndicators

[niveditasing]

New GCS Flag:

• Introduced --historical_gcs_path (defaulting to gs://unresolved_mcf/world_bank/wdi/deleted_rows_07_2026.csv) to specify the GCS location of deleted historical records.

Merge & Deduplication Logic (worldbank.py):

• Loads the GCS-hosted historical deleted dataset using retry_call with exponential backoff (3 tries) and fails fast via logging.fatal if the file cannot be read.
• Concatenates fresh World Bank data and deleted historical data, normalizing key columns and deduplicating across composite keys (StatisticalVariable, ISO3166Alpha3, Year, observationPeriod, unit, measurementMethod, scalingFactor) while prioritizing fresh observations (keep='first').
• Ensures output/ directory initialization at the start of process() and adds unit tests in worldbank_test.py (test_merge_historical_data).

Import Validation (validation_config.json & manifest.json):

• Configured DELETED_RECORDS_PERCENT threshold at 0.1%.
• Added check_max_date_freshness (SQL_VALIDATOR) to verify per-StatVar MaxDate freshness (latest_year >= EXTRACT(YEAR FROM CURRENT_DATE) - 3 for all active StatVars).

Artifacts & Testing Results:

• Historical GCS data: https://storage.mtls.cloud.google.com/unresolved_mcf/world_bank/wdi/deleted_rows_07_2026.csv
• Differ summary: https://storage.mtls.cloud.google.com/unresolved_mcf/world_bank/wdi/differ_summary.json
• Validation output: https://storage.mtls.cloud.google.com/datcom-import-test/scripts/world_bank/wdi/WorldDevelopmentIndicators/2026_09_21T20_53_48_038376_07_00/input0/validation/validation_output.csv

[gemini-code-assist]

Code Review

This pull request introduces a new validation configuration file, validation_config.json, for the World Bank WDI dataset. It defines rules for checking deleted record counts and performing golden file comparisons. The review feedback highlights path inconsistencies for golden files and a missing directory prefix for input files, which could prevent the validation tool from locating the necessary data.

[balit-raibot]

@gemini-code-assist review

[gemini-code-assist]

Code Review

This pull request introduces import validation for the World Bank WDI dataset by adding a validation configuration, updating the manifest, and providing golden data files. Feedback includes a missing file reference for 'golden_WorldBank.csv', a path mismatch for the summary report, and a suggestion to move golden files to a local directory for better self-containment. Additionally, the documentation header in the README should be generalized to describe the process rather than the PR, and the threshold for the deleted records percentage may need adjustment to reflect the intended strictness.

[pulkeet-wq]

LGTM

[pulkeet-wq]

LGTM

[SandeepTuniki]

Thanks for the changes! I've left a couple of review suggestions inline below.

Also, please update the PR description so it matches the final implementation (it currently still mentions the older gs://resolved_mcf/.../deleted_historical_data_06_2026.csv path, a graceful fallback instead of logging.fatal, and golden checks).

[SandeepTuniki]

Hi @niveditasing, you comments say the changes are made, but I don't see new commits since my review. Did you forget to push the changes? Also, the PR description still shows the old text.

[niveditasing]

Hi @niveditasing, you comments say the changes are made, but I don't see new commits since my review. Did you forget to push the changes? Also, the PR description still shows the old text.

Sorry for the inconvenience; I forgot to push the changes earlier. I have resolved the PR comments and pushed the updates now.

[SandeepTuniki]

Thanks for the changes!

Note: I updated the PR description for now, but please keep it up to date for future PRs.