Repository navigation
Split catalog_selection into a validation module and a database-reads module - #4179
Merged
Merged
Conversation
…ailing) Add structure tests for policyengine_api/data/v2/catalog: the validation module imports no SQLAlchemy, SQLModel or database module (and the table models only under TYPE_CHECKING), the reads module holds select_catalog, and no code imports the former catalog_selection module. Point every test import at catalog_selection_validation. These fail until the split lands: the new modules do not exist yet. Refs #4176 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
policyengine_api/data/v2/catalog/catalog_selection.py mixed catalog version validation, its error types and the SelectedCatalog result type with the one catalog query. Every v2 validator, transformation and type that needed the version validation or the error types therefore imported SQLAlchemy and SQLModel. - catalog_selection_validation.py holds the validation, constants, error types and SelectedCatalog. It imports no SQLAlchemy, SQLModel or database module, and the table models only under TYPE_CHECKING. - catalog_selection_reads.py holds select_catalog. - Every import of the former module now names one of the two. With the table models' own SQLAlchemy imports set aside, no v2 validator, transformation or type module reaches sqlalchemy any more (23 did before). Fixes #4176 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## master #4179 +/- ##
===========================================
+ Coverage 53.80% 88.29% +34.48%
===========================================
Files 281 293 +12
Lines 16310 17510 +1200
Branches 2639 2876 +237
===========================================
+ Hits 8776 15460 +6684
+ Misses 6803 1337 -5466
+ Partials 731 713 -18 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
…ode (failing) The static check read only catalog_selection_validation.py itself, so it missed that importing the module first runs the catalog package's __init__.py, which imports extraction and through it SQLModel, SQLAlchemy and every v2 table model. - Import the module in a fresh interpreter and list the database modules then loaded; proof fixtures show a mixed module is rejected and a database-free one accepted. - Resolve relative imports and record "from package import name" as package.name, so imports such as "from policyengine_api.data.v2 import models" are seen. - Search the repository's text files for the former module's name, which also catches mock.patch targets and docs. test_importing_the_validation_module_loads_no_database_code fails until the catalog package stops importing extraction when it loads. Refs #4176 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Importing any module in policyengine_api.data.v2.catalog first ran the package's __init__.py, which imported extraction and, through it, SQLModel, SQLAlchemy and every v2 table model. So importing catalog_selection_validation still loaded the database code it no longer imports itself. No code used the names __init__.py re-exported, so it is now only a docstring. Importing a v2 validators, transformations or types module now loads SQLAlchemy in 3 of 39 cases, each of which imports the table models itself; on master 23 did. Register the checks as rule api.catalog.selection-validation-no-database, with proof fixtures, and cite it in v2-code-organization.md. Refs #4176 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- The rule's proof now runs the static import check on the fixtures as well as the fresh-interpreter check. - A new test pins which imports count as TYPE_CHECKING-only: those under "if TYPE_CHECKING:" or "if typing.TYPE_CHECKING:", and no other "if". - The search for the former module's name reads every tracked file (git ls-files) except release notes, instead of a list of directories and suffixes. - The rule's scope adds the files the runtime check depends on: the parent package __init__.py files and supported_countries.py. - v2-code-organization.md describes which connectors call select_catalog instead of stating an unchecked constraint. Refs #4176 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… (failing) The search matches the bare word catalog_selection, so a later variable or sentence using it would fail the test as if the removed module were still referenced. The new test fails until the pattern matches only module references: a dotted module path, the file name or an import. Refs #4176 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Match a dotted module path ending in .catalog_selection, the file name catalog_selection.py, or "import catalog_selection", so the bare word stays usable for variables and prose. Also list test_the_reads_module_holds_the_catalog_query under rule api.catalog.selection-validation-no-database, whose statement says the catalog query lives in catalog_selection_reads. Refs #4176 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…alog-split # Conflicts: # docs/generated/rules.md
This branch was previously deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #4176
Summary
policyengine_api/data/v2/catalog/catalog_selection.pyheld catalog version validation, its error types and theSelectedCatalogresult type together with the one catalog query,select_catalog. Any v2 validator, transformation or type that needed the version validation or the error types therefore loaded SQLAlchemy and SQLModel, mostly throughpolicyengine_api/query_parameters.py.catalog_selection_validation.py:validate_policyengine_version,SUPPORTED_V2_COUNTRY_IDS,MAXIMUM_POLICYENGINE_VERSION_LENGTH, the five error types andSelectedCatalog. It imports no SQLAlchemy, SQLModel or database module, and the table models only underTYPE_CHECKING, forSelectedCatalog's field annotations.catalog_selection_reads.py:select_catalog, unchanged. Only the policies and metadata read connectors import it.catalog_selection.pyis removed. All 15 application modules and 11 test modules that imported it now import one of the two.policyengine_api/data/v2/catalog/__init__.pyis now only a docstring. It used to importextraction, so importing any module in the package, including the new validation module, loaded SQLModel, SQLAlchemy and every table model. No code used the names it re-exported.api.catalog.selection-validation-no-database, enforced by three pytest checks in theTestjob (the static and fresh-interpreter import checks, and that the reads module holds the query), with proof fixtures.v2-code-organization.mdcites it, anddocs/generated/rules.mdandAGENTS.mdare regenerated.No behaviour changes. One side effect:
SelectedCatalog's field annotations name the table models, which are now imported only underTYPE_CHECKING, sotyping.get_type_hints(SelectedCatalog)raisesNameError. Nothing reads them at runtime; it would matter only ifSelectedCatalogbecame a Pydantic field or a FastAPI parameter.Measured effect
Importing each of the 39 v2
validators,transformationsandtypesmodules in a fresh interpreter:master, 23 loadsqlalchemy;comparison_runs.transformations,identity.transformations,identity.validators), each because it imports the table models at runtime itself, whichv2-code-organization.mdallows.In the static import graph (Import Linter 2.15), with the table models' own SQLAlchemy imports set aside, 23 of those modules reached
sqlalchemythroughcatalog_selectionbefore this change and none do after it. This is step 2.5 of the agent-proof code-quality plan and prepares the Import Linter contracts in the next PR. Import Linter does not count the implicit import of a parent package's__init__.py, which is why the runtime check above is a separate test.Open pull requests this conflicts with
#4108 and #4092 change
catalog_selection.py. Their change (derivingSUPPORTED_V2_COUNTRY_IDSfromSUPPORTED_COUNTRY_IDS) is already onmaster. Whichever merges second resolves the conflict by importing fromcatalog_selection_validation(orcatalog_selection_readsforselect_catalog). The registry,docs/generated/rules.mdandAGENTS.mdchanges may also conflict as text with other open rule PRs; regenerate withpython -m scripts.rules render.Testing
Tests that show a defect came first, each in its own commit that failed against the code before it:
tests/unit/v2/test_catalog_selection_modules.pychecks that the validation module imports no database code and the table models only underTYPE_CHECKING; that the reads module definesselect_catalogand the validation modulevalidate_policyengine_version; and that no module imports the former one.test_metadata_service.py::test_read_modules_import_no_policyengine_or_v1_metadata_sourcenow checks both new modules, and the 11 test modules that imported the former module importcatalog_selection_validation.test_importing_the_validation_module_loads_no_database_codeimports the module in a fresh interpreter and lists the database modules then loaded. It failed with 173 modules loaded, starting withpolicyengine_api.data.v2.models, until 7c3aba2 emptied the package__init__.py. The same commit added the proof test (fresh-interpreter check only), made the static import helper resolve relative imports andfrom package import submodule, and added a search for the former module's name.test_the_name_search_matches_module_references_onlyfailed because the search matched the bare wordcatalog_selection; d8c5c31 narrowed it to module references.Test coverage added after review, with no failing commit because it checks test helpers rather than application code (95bab05): the proof test also runs the static check on the fixtures,
test_only_imports_under_type_checking_count_as_annotation_onlypins which imports count asTYPE_CHECKING-only, and the name search reads every tracked file except release notes.After the changes, run by node ID or by file:
test_catalog_selection_modules.py, the updated isolation test andtests/unit/test_supported_countries.py::test_every_country_list_follows_a_country_added_to_the_one_listpass;test_supported_countries.py, the one test that uses the import was run (above).ruff format,ruff check,mypy,python scripts/run_quality_guards.pyandpython -m scripts.rules guard --base-ref origin/masterpass.🤖 Generated with Claude Code