Skip to content

Assign every v2 route and service file to a layer and enforce the v2 layer rules - #4191

Draft
anth-volk wants to merge 9 commits into
v2-architecture-import-linterfrom
v2-architecture-layer-check
Draft

anth-volk wants to merge 9 commits into
v2-architecture-import-linterfrom
v2-architecture-layer-check

Conversation

@anth-volk

@anth-volk anth-volk commented Nov 4, 2026 •

Copy link
Copy Markdown
Collaborator

Fixes #4190

Summary

Adds the v2-layers quality guard, which gives every API v2 route and service file a layer and checks each layer's imports and calls. It turns three doc-only v2 rules into checked ones and adds three rules. Step 2.6 of the agent-proof code-quality plan (its AST part). Stacked on #4184 (Import Linter), which builds on #4179 (merged).

What the guard checks

scripts/rules/v2_layers.py uses the standard library only, like the rest of the rules tool. scripts/run_quality_guards.py runs it in the quality-guards job of pr.yml.

Rule Checks Status
api.v2.files-assigned-to-layers (new) Every .py file under policyengine_api/services/v2 and policyengine_api/fastapi_routes/v2 gets a layer from its file name or from rules/v2-layer-manifest.toml. A file with neither fails. enforced
api.v2.routes-no-sessions-or-sql (was doc-only) No module under fastapi_routes/v2 imports SQLAlchemy, SQLModel, a connector, a database_session module, the v1 ORM module policyengine_api.data.orm or policyengine_api.data.v2.database. None calls a SQLAlchemy or SQLModel function (except its exception classes), any method on a session (including a database_session container's read() or transaction()) or a session factory. ratcheted, 4 tolerated
api.v2.services-no-sql (was doc-only) services.py modules import nothing from SQLAlchemy or SQLModel but the Session type (for annotations) and sqlalchemy.exc (to raise typed failures). They call no SQLAlchemy or SQLModel function but its exception classes and session factories, which catches and_, or_, func and col even through import sqlalchemy.exc. Services may open sessions, which the rule doesn't forbid. They also call no data-access session method (exec, execute, add, get, get_one, the bulk methods, connection, delete and so on), and import no FastAPI or Starlette. enforced, 0 today
api.v2.connectors-never-commit (was doc-only) Connector modules, and a database_connectors/__init__.py, don't call commit, rollback, close, begin, begin_nested, reset or invalidate on a session, open none (no session factory, injected session_factory, or database_session container read()/transaction()), and import no v1 ORM, v2 database or database_session module. enforced, 0 today
api.v2.session-modules-no-data-access (new) database_session.py modules sit directly in their resource package and call no data-access session method (add, get, merge, exec, execute and so on) or SQL builder, and import no connector. They open, commit, roll back and close sessions. enforced, 0 today
api.v2.pure-modules-no-session-calls (new) Validators, transformations, types, process-state modules and package __init__ files:
  • import nothing from SQLAlchemy or SQLModel but the Session type;
  • import no connector, session, v1 ORM or v2 database module;
  • call no method on a session, no SQLAlchemy or SQLModel function and no session factory.
enforced, 0 today

The fourth doc-only rule, api.v2.connectors-no-cross-calls, became ratcheted in #4184 through Import Linter's independence contract.

Layers by file name:

  • services/v2/<resource>/: services.py, validators.py, transformations.py, types.py and database_session.py. A database_session.py must sit directly in its resource package, where api.v2.session-modules-no-queries (pytest) also checks it.
  • services/v2/<resource>/database_connectors/: <creates|reads|updates|deletes>[_name].py.
  • fastapi_routes/v2/: routes.py and <name>_routes.py, plus request_models.py and response_models.py.
  • __init__.py anywhere.

Manifest (9 files with other names). It lives under the code-owned rules/ directory, and each entry gives a layer and a reason:

  • http-support: fastapi_routes/v2/errors.py, user_auth.py, user_resources.py, and metadata/{common,query_dependencies,response_cache,search_file_response}.py;
  • http-model: households/document_models.py;
  • process-state: services/v2/metadata/selected_catalogs.py.

The manifest may give a file under services/v2 only service, validator, transformation, types or process-state, and a file under fastapi_routes/v2 only route, http-model or http-support, so an entry can't switch a file's checks off: each of those layers has the full checks of its layer. Every entry needs a reason. The guard also fails on an entry for a missing file, or for a file whose name already gives it a layer.

Attribute-aware calls.

  • A method call counts as a session call only when its receiver is named session or <name>_session. The .session of a request or websocket (Starlette's session data: request, req, websocket, ws or *_request, also as self.request) and HTTP or client sessions (http_session, requests_session, _client_session) don't count; request_session and request_context.session do. So session.add, self._session.exec and database_session.transaction() count; set.add, cache.get, sessions_router.get and session_ids.add don't. Routes and the pure layers may call no method on a session at all, so a method left off a list can't slip through there.
  • In routes, services and the pure layers, any call to a name from SQLAlchemy or SQLModel counts, except its exception classes, and so does <Model>.__table__.<builder>(). Connectors may build SQL. A session factory counts for any call into policyengine_api.data.orm or policyengine_api.data.v2.database (get_v1_session_factory, build_v2_session_factory, the engines, …), for SQLAlchemy's Session, sessionmaker and create_engine, and for an injected *session_factory (session_factory(), self._session_factory(), session_factory.begin()). A local helper named select doesn't count.
  • Relative imports and from package import submodule are resolved. Imports made only under TYPE_CHECKING count (decided 2026-11-04).

Tolerated (baseline rules/baselines/v2-layers-routes.txt, 4 entries, one per occurrence). countries/routes.py, households/routes.py, policies/routes.py and user_resources.py import sqlalchemy.exc to catch SQLAlchemyError. I counted these as violations rather than allowing exception imports in routes; allowing them would mean changing the rule's statement. The fix is for the services to raise typed failures. A new finding fails, and so does a baseline entry that no longer occurs.

Limits.

  • A session object bound to a name without session in it isn't recognised.
  • An alias of a model's table (TABLE = Policy.__table__) isn't recognised.
  • A call made through a name re-exported by another module isn't resolved.
  • Services aren't checked for ending transactions themselves, because the rule's statement doesn't forbid it.

Import Linter (#4184) covers the import side transitively.

Open pull requests that add v2 files

I scanned each open PR's head with this guard. Once this merges, each PR must give every new file under services/v2 or fastapi_routes/v2 a layer, by its name or a manifest entry. The baseline can't grow, so each PR must also clear its own new findings.

No other open PR adds files under these directories.

Docs and registry

CI and environment variables

  • No workflow changes. The guard runs in the existing "Run quality guards" step, with plain python and no project dependencies. Its proofs run in the existing "Test the rules registry tool" step.
  • The guard reads no environment variable. It runs git ls-files, through the file-length guard's helper, to list tracked files.

Testing

Tests first, in 4ba6e71, which failed because scripts/rules/v2_layers.py didn't exist yet. The review-round tests failed before their fixes: 5328792 before 834b127, d0a49ed before d18175c, and f7a58d2 before 5af955d.

  • scripts/rules/tests/test_v2_layers.py (36 tests):
    • the six proofs: each bad tree gives exactly its expected_findings and each good tree none;
    • assignment by name and by manifest;
    • receiver-aware session calls (set.add not counted);
    • builders counted only when imported from SQLAlchemy or SQLModel;
    • imports under TYPE_CHECKING counted;
    • package and relative imports resolved;
    • connectors may add and flush but not end a transaction;
    • baseline: listed, unlisted and stale findings;
    • the unassigned-file message;
    • manifest validation;
    • every file in the repository has a layer;
    • every layer under services/v2 has its calls checked, and a connector package __init__ counts as a connector;
    • the repository's own session factories;
    • services' SQLAlchemy and SQLModel imports;
    • names that only contain the word session;
    • one baseline line per occurrence;
    • manifest layers per directory, and the required reason;
    • routes and pure layers calling any session method;
    • the full service and connector method lists;
    • manifest files getting their layer's checks;
    • nested session modules;
    • SQLAlchemy calls through import sqlalchemy.exc;
    • web and client sessions;
    • __table__ builders;
    • injected session factories and containers;
    • request and websocket session data by owner.
  • Expectations changed in implementing commits, each because of an intended change:
    • 4ba6e71 → 6ae4136: from .. import database_session (not ...) names the resource's session module.
    • 834b127: services now also report imports sqlalchemy for import sqlalchemy as sa.
    • d18175c: pure modules and the pure proof also report their sqlmodel.select import. (The services test's and_, or_, func.lower and col call findings were added in the failing commit d0a49ed.)

After the change:

  • the 36 tests pass on Python 3.12 and 3.14;
  • python scripts/run_quality_guards.py passes, including v2-layers;
  • python -m scripts.rules guard --base-ref origin/master passes;
  • ruff format --check ., ruff check . and mypy pass.

🤖 Generated with Claude Code

@anth-volk
anth-volk added this pull request to stack #4185 November 4, 2026 16:10
@anth-volk anth-volk changed the title v2 architecture layer check Assign every v2 route and service file to a layer and enforce the v2 layer rules Nov 4, 2026
@codecov

codecov Bot commented Nov 4, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
⚠️ Please upload report for BASE (v2-architecture-import-linter@367b2a5). Learn more about missing BASE report.

Additional details and impacted files
@@                       Coverage Diff                        @@
##             v2-architecture-import-linter    #4191   +/-   ##
================================================================
  Coverage                                 ?   88.29%           
================================================================
  Files                                    ?      293           
  Lines                                    ?    17510           
  Branches                                 ?     2876           
================================================================
  Hits                                     ?    15460           
  Misses                                   ?     1337           
  Partials                                 ?      713           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

anth-volk and others added 7 commits November 4, 2026 20:56
scripts/rules/tests/test_v2_layers.py specifies a quality guard over every
.py file under services/v2 and fastapi_routes/v2:

- each file is assigned a layer by its name or by
  rules/v2-layer-manifest.toml, and an unassigned file fails;
- route modules import no SQLAlchemy, SQLModel, connector, session or v2
  database module and call no SQL builder or session method;
- services build no SQL, call no data-access session method and import no
  FastAPI or Starlette;
- connectors don't open, commit, roll back or close a session.

Calls count only on a receiver named like a session (session.add, not
set.add), and SQL builders only when imported from SQLAlchemy or SQLModel.
Imports under TYPE_CHECKING count. Proof fixture trees for the four rules
include a route that imports a connector and a file with no layer.

These fail until scripts/rules/v2_layers.py exists.

Refs #4190

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Add the v2-layers quality guard (scripts/rules/v2_layers.py, standard
library only), run by scripts/run_quality_guards.py in the quality-guards
job.

- It discovers every .py file under services/v2 and fastapi_routes/v2 and
  takes its layer from its file name, or from rules/v2-layer-manifest.toml
  for the 9 files with other names. A file with neither fails: new rule
  api.v2.files-assigned-to-layers.
- api.v2.routes-no-sessions-or-sql is ratcheted. Its baseline lists the 4
  route modules that import sqlalchemy.exc to catch SQLAlchemyError.
- api.v2.services-no-sql and api.v2.connectors-never-commit are enforced,
  with no findings today.
- Session calls count only on a receiver named like a session, and SQL
  builders and session factories only when imported from SQLAlchemy,
  SQLModel or the v2 database module. Imports under TYPE_CHECKING count.

Each rule has proof fixture trees, and v2-code-organization.md cites each
rule id. The test fix resolves "from .. import database_session" to the
resource's own session module, as Python does.

Fixes #4190

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Validators, transformations, types, process-state modules and package
  __init__ files touch no session: new rule
  api.v2.pure-modules-no-session-calls, with proof fixtures. A
  database_connectors/__init__.py counts as a connector.
- The repository's own session factories count: the v1 ones in
  policyengine_api.data.orm, build_v2_session_factory, the engines, and a
  database_session container's transaction() in a route.
- Services import nothing from SQLAlchemy or SQLModel but the Session type
  and sqlalchemy.exc, so and_, or_, func and col count, as does a select()
  on a model's __table__.
- A name that only contains "session" (sessions_router, session_ids,
  request.session) is not a session.
- Each occurrence of a finding needs its own baseline entry, and a
  duplicated entry is stale.
- The manifest gives only the layers of the file's directory, never
  "package", and every entry needs a reason.

Refs #4190

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- New rule api.v2.pure-modules-no-session-calls: validators,
  transformations, types, process-state modules and package __init__
  files call no session method, SQL builder or session factory. A
  database_connectors/__init__.py is checked as a connector.
- Session factories are any call into policyengine_api.data.orm or
  policyengine_api.data.v2.database, or SQLAlchemy's Session,
  sessionmaker and create_engine. Routes and connectors may not import
  the v1 ORM module, and a route may not call a database_session
  container's read() or transaction().
- Services import nothing from SQLAlchemy or SQLModel but the Session
  type and sqlalchemy.exc, so and_, or_, func and col are caught, and a
  builder called on a model's __table__ counts.
- A session receiver is named session or <name>_session, and
  request.session and http_session are not sessions, so sessions_router
  and session_ids are not flagged.
- Each occurrence of a finding needs its own baseline line, and a
  duplicated line is reported as no longer applying.
- The manifest may give only the layers of the file's directory, and
  every entry needs a reason.
- v2-code-organization.md has a "File layers" section, which the files
  rule's rationale now links to.

The real code still has exactly the 4 tolerated sqlalchemy.exc imports.

Refs #4190

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Refs #4190

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Routes and the pure layers may call no session method at all, so
  get_one, bulk_save_objects, connection and expunge count; services'
  and connectors' lists gain get_one, the bulk methods, reset and
  invalidate.
- A manifest file in a pure or HTTP layer gets that layer's import and
  call checks, including session factories.
- A database_session.py below a resource package has no layer.
- Every call into SQLAlchemy or SQLModel but its exception classes
  counts, so "import sqlalchemy.exc" doesn't open sqlalchemy.and_.
- request.session on any request object, and client or requests
  sessions, are not database sessions.
- Only a builder called on <Model>.__table__ counts, once.

Refs #4190

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Routes and the pure layers may call no method on a session, so a
  method missing from a list can't slip through. Services' and
  connectors' lists gain get_one, the bulk methods, connection,
  expunge, reset and invalidate.
- The pure layers also get import checks (nothing from SQLAlchemy or
  SQLModel but the Session type; no connector, session, v1 ORM or v2
  database module), so a file the manifest assigns to one is checked as
  fully as a named one.
- A database_session.py below a resource package has no layer, so it
  can't escape the session-module rule.
- Every call into SQLAlchemy or SQLModel counts in routes, services and
  the pure layers, except to its exception classes, so
  "import sqlalchemy.exc" no longer opens sqlalchemy.and_.
- A request's .session and client or requests sessions are not database
  sessions; only <Model>.__table__.<builder>() counts as a table builder.
- The pure rule's scope leaves out the connector package __init__ files,
  which are checked as connectors. The docs list what the guard
  recognises and what it doesn't.

Refs #4190

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@anth-volk
anth-volk force-pushed the v2-architecture-layer-check branch from 80e0f9d to d18175c Compare November 4, 2026 16:57
anth-volk and others added 2 commits November 4, 2026 21:25
- An injected session factory opens a session: session_factory.begin()
  and self._session_factory() count in connectors and routes, and so does
  a database_session container's transaction() in a connector.
- New rule api.v2.session-modules-no-data-access: a database_session
  module opens, commits, rolls back and closes sessions, but adds, gets,
  merges or executes nothing and builds no SQL. Proof fixtures included.
- A request's .session is recognised by its owner (req, websocket,
  *_request), so request_context.session and request_session are
  database sessions.

Refs #4190

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Injected session factories open sessions: session_factory(),
  self._session_factory() and session_factory.begin() count in routes,
  connectors and the pure layers, and a connector may not call a
  database_session container's read() or transaction().
- New rule api.v2.session-modules-no-data-access: database_session
  modules call no data-access session method or SQL builder and import
  no connector. Today's modules only open, commit, roll back and close.
- A request's .session is recognised by its owner (req, request,
  websocket, ws, *_request), so request_context.session and
  request_session are database sessions, and requests_session is not.
- The docs say services may open sessions, which their rule doesn't
  forbid.

The real code still has exactly the 4 tolerated sqlalchemy.exc imports.

Refs #4190

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

This branch was successfully deployed

1 active (outdated) deployment
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Assign every v2 route and service file to a layer and enforce the doc-only v2 layer rules

1 participant