Skip to content

Report which parameters a policy sets that the current model no longer has - #4039

Draft
anth-volk wants to merge 5 commits into
v2plan/1k-142-copy-reused-policiesfrom
v2plan/1k-143-report-missing-parameters
Draft

anth-volk wants to merge 5 commits into
v2plan/1k-142-copy-reused-policiesfrom
v2plan/1k-143-report-missing-parameters

Conversation

@anth-volk

Copy link
Copy Markdown
Collaborator

Fixes #4038

Stacked on #4036, which adds the shared check (parameters_missing_from_model) used here.

Summary

Policy reads now say which parameters a policy sets that the current tax-benefit model no longer has, as missing_parameters: the parameter paths, sorted; [] when the model has them all; null when they cannot be checked. The field is added next to the existing ones; nothing else in the responses changes.

Which routes carry it, and why

Route Why Checked against null when
GET /{country}/policy/{policy_id} The app's newer interface reads every saved reform's changes through this route today. The leaf parameters of the country model the Public API runs (PolicyEngineCountry.parameter_names). The stored policy is not a parameter-path object, or the country has no loaded model.
GET /v2/policies/{policy_id} The route that interface reads a saved reform's policy through after it moves to the v2 routes. The catalog of the running PolicyEngine.py version. That catalog is not published yet.
GET /v2/policies, POST /v2/policies They return the same PolicyItem, so the field is always there. The same catalog. The same.

Both checks use parameters_missing_from_model, the comparison the dual-write copy applies (#4036). The v1 route reads the loaded country model rather than the Supabase catalog: v1 reads stay on Cloud SQL, as the Stage 10 runbook requires, a Supabase outage or its five-second timeouts cannot slow a v1 read, and the loaded model is the one a v1 calculation runs. For one release the two sets are the same: .py builds its catalog from the same country package version's leaf parameters (get_descendants() filtered to parameters), and catalog extraction checks the package version against the bundle manifest. A v1 policy reported here is therefore one a first copy under the running release refuses as missing_parameters. Computing the set takes about 40 ms once per process for the 102,530 US parameters.

Other changes:

  • read_parameters_by_name reads names in batches of 1,000, so a page of policies on older catalog versions cannot bind more values than PostgreSQL allows (review round 1).
  • The v2 policy service returns CheckedPolicyRead and CheckedPolicyPage, which only the missing-parameter check builds, so mypy rejects a route result that skipped it (review round 2).
  • select_catalog raises MetadataCatalogNotPublishedError (a MetadataCatalogUnavailableError) when the running version's catalog is absent. The v2 routes return null only for that case and still return 503 when the catalog cannot be queried; the copy's catalog_absent and catalog_query_failed reasons now come from the type, not from the chained exception.
  • The migration contract registry pins result.item.missing_parameters on the v2 policy routes; the generated migration contracts are refreshed.
  • The OpenAPI spec, the changelog and a new "Missing parameters" section of the Stage 10 runbook describe the field.

Migration notes

  • Route groups covered: policy (v1 GET /{country}/policy/{policy_id}; native v2 /v2/policies).
  • What stays on the Flask/v1 path: the v1 route is still Flask and still reads only Cloud SQL.
  • User-visible API contract change, additive only: one new field on each of these responses. Existing callers that ignore unknown fields are unaffected.

Environment variables

None. The change reads no environment variables and touches no workflow, image or infrastructure.

What this PR waits for

#4036 (the shared check and the dual-write fix) merges first. The app change that shows these parameters and blocks running such policies (PolicyEngine/policyengine-app-v2, linked from the tracking issue) has an effect only once this PR is deployed.

Testing

Each commit was written test-first; the failing run came before the code change.

  • 28febfb63 (v2 routes): 12 tests failed first ('PolicyRead' object has no attribute 'missing_parameters', HTTP 500 from the routes, and the PolicyItem required fields); then 26 passed in the two files.
  • 4eaac4a2c (v1 route): the service tests failed at import (policy_parameter_service did not exist) and 3 route tests failed (policy_routes has no policy_parameter_service); then they passed. Against the loaded US model (policyengine-us 2.2.1): gov.dol.minimum_wage and gov.irs.credits.ctc.amount.base[0].amount exist, the node gov.irs.credits.ctc.amount does not count as a parameter, and gov.irs.income.bracket.thresholds.1.WIDOW (renamed SURVIVING_SPOUSE) is reported missing.
  • b8cc4c0c2 (review round 1: batches): against the unbatched read, the test failed with one statement holding 2,500 names; then 5 passed.
  • 761050053 (review round 1: not-published catalog type, contract registry, runbook): a catalog query failure was turned into null instead of raising (DID NOT RAISE), and the copy tests failed at import; then they passed.
  • e29bd6e4c (review round 2: CheckedPolicyRead): the new tests failed at import (CheckedPolicyRead and CheckedPolicyPage did not exist); then 64 passed in the v2 policy and copy test files. Round 2 found no bugs.
  • On every commit: ruff format --check ., ruff check ., uv run --frozen --extra dev mypy, python scripts/run_quality_guards.py, the Phase 10 policy test group, and python scripts/export_migration_contracts.py.
  • After the last commit: the whole tests/unit and tests/contract suites (3,400 passed).
  • Not run locally: the MySQL and PostgreSQL integration suites (no disposable databases were available). GitHub Actions runs on this branch are cancelled at the user's request.

Inline suppression added: # noqa: PLC0415 on the in-function import of policyengine_api.country in loaded_model_parameter_names. Importing that module loads every country model, so a module-level import would load them whenever the routes are imported; the module cannot be added to banned-module-level-imports because metadata_service imports it at module level. household_calculation_service uses the same suppression for the same reason.

🤖 Generated with Claude Code

anth-volk and others added 5 commits October 26, 2026 17:25
Every policy returned by GET /v2/policies/{policy_id}, GET /v2/policies
and POST /v2/policies now carries missing_parameters: the parameter
paths it sets that the catalog of the running PolicyEngine.py version
no longer has, sorted. It is empty for a policy bound to the running
catalog version, and null when that catalog is not published, since
nothing can then be checked. The field is required in the response
model, so it is always present.

V2PolicyService checks the policies after reading them
(with_missing_parameters): one catalog selection and one read of the
running version's parameters by name, only for policies bound to an
older version. The comparison is parameters_missing_from_model, the
check the dual-write copy uses.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
v3 reads a saved reform's changes through GET /{country}/policy/{id},
so that response now also carries missing_parameters: the parameter
paths the policy sets that the country model the Public API runs no
longer has, sorted; empty when it has them all; null when the stored
policy is not a parameter-path object or the country has no loaded
model. Existing fields are unchanged.

PolicyParameterService compares the paths with the loaded model's leaf
parameters (PolicyEngineCountry.parameter_names, computed once per
process) using parameters_missing_from_model, the comparison the v2
copy uses. It reads nothing from Supabase, so v1 reads stay on Cloud
SQL and a Supabase outage cannot slow them. The v2 catalog of a release
holds the same leaf parameters as that release's model, so a policy
reported here is one the copy refuses as missing_parameters.

The OpenAPI spec, the changelog fragment and the Stage 10 runbook
describe the field on both the v1 and the v2 routes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… the bind limit

Review found that GET /v2/policies could now fail with 503: the missing
parameter check reads every distinct parameter name of a page's
policies on older catalog versions in one IN filter, and a page of up
to 500 policies with up to 1,000 values each can bind more than the
65,535 values a PostgreSQL statement allows. read_parameters_by_name
now reads names in batches of 1,000. The copy, which reads at most
1,000 names, runs one statement as before.

The recording session used in unit tests gains all(), which this read
calls.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… by type

with_missing_parameters returned null for an absent catalog and
re-raised a failed catalog query by checking whether the exception's
cause was a SQLAlchemyError, which depended on how select_catalog
chained it, and no test covered the failure branch. select_catalog now
raises MetadataCatalogNotPublishedError, a MetadataCatalogUnavailableError,
for an absent catalog: the v2 routes return null for that type only,
and the copy's catalog_absent and catalog_query_failed reasons come
from the type too. Existing handlers of MetadataCatalogUnavailableError
are unchanged.

Also from review:
- the migration contract registry pins result.item.missing_parameters on
  the v2 policy create and detail routes, and the generated migration
  contracts are refreshed;
- a test covers create_policy filling in the field;
- the Stage 10 runbook no longer says every policy reported as missing
  is refused by the copy: a copy made before a release removed the
  parameter is kept.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
PolicyRead.missing_parameters defaulted to None, which is also what the
response documents for an unpublished catalog, so a future read that
skipped the check would have told clients the catalog is unpublished.
The field moves to CheckedPolicyRead (and pages of them to
CheckedPolicyPage), which only with_missing_parameters builds; the v2
policy service, its route protocol and the response models now take
those types, so mypy rejects a route result that was never checked.

Tests also cover the field in the GET /v2/policies response, and the
catalog-query-failure case again chains the database error, as
select_catalog raises it, so the test still shows that neither message
reaches the copy log.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

This branch was successfully deployed

1 active deployment
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant