Skip to content

FND-3879: add pyatlan_v9 creators for the 2 Dataplex types - #1103

Merged
cmgrote merged 2 commits into
mainfrom
FND-3879
Oct 21, 2026
Merged

cmgrote merged 2 commits into
mainfrom
FND-3879

Conversation

@cmgrote

@cmgrote cmgrote commented Oct 21, 2026

Copy link
Copy Markdown
Collaborator

✨ Description

Adds creator() for the 2 Dataplex types in pyatlan_v9, following the structure of FND-3848 (Cognos).

GCPDataplexAspectType.creator(
    resource_name="projects/acme-prod/locations/global/aspectTypes/pii",
    connection_qualified_name="default/dataplex/1700000000",
)
# qualified_name == "default/dataplex/1700000000/aspect-type/projects/acme-prod/locations/global/aspectTypes/pii", name == "pii"

BigqueryRoutine.creator(
    name="clean_orders", definition="CREATE PROCEDURE …",
    schema_qualified_name="default/bigquery/1700000000/acme-prod/sales",
)
# qualified_name == "default/bigquery/1700000000/acme-prod/sales/_routines_/clean_orders"
  • GCPDataplexAspectType: exact app parity. The qualifiedName is the one atlan-dataplex-app's aspect_type_qn() emits. The creator takes the full GCP resource name, validates its projects/*/locations/*/aspectTypes/* shape, and derives name, project and location from it, as the app does.
  • BigqueryRoutine uses the legacy _routines_ shape ({schemaQN}/_routines_/{name}). Existing routine assets carry it, and atlan-dataplex-app aspect-sync parses it.
    • Current atlan-bigquery-app crawls (ware-1973) emit Function / Procedure instead; the docstring says so.
    • The census template {conn}/{project}/{name} is a fallback bug in the Dataplex app (dataplex_enricher.py:173), not the real shape.
    • Required inputs are name, definition and schema qn. The creator derives the schema/database/connection attributes and sets the atlan_schema relationship, mirroring Procedure.creator.
  • Typed updater / trim_to_required on both types. Blank inputs raise (validate_required_fields).
  • Adds AtlanConnectorType.DATAPLEX with category SAAS. Connection.creator needs it. The app declares no category; SAAS matches the typedef's Catalog/SaaS model location. ⚠️ Reviewer: confirm the category.
  • Overlays are in pyatlan_v9/model/assets/_overlays/{gcp_dataplex_aspect_type,bigquery_routine}.py. The 2 classes are regenerated from the models PR branch with the same flags as sync-python-sdk.yaml.
  • ⚠️ The BigqueryRoutine._QUALIFIED_NAME_PATTERN changes from the generic 4-segment pattern to ^.+/_routines_/[^/]+$, so validate() now rejects routine qns without _routines_.

Linear: FND-3879
Paired models PR: https://github.com/atlanhq/models/pull/2102

Merge order

  1. This PR first.
  2. models#2102 second.

The sync bot regenerates with overlays from atlan-python main, so in this order its PR is a no-op for these 2 types. In the other order it would regenerate them without the creators.


🧩 Type of change

  • 🚀 New feature (non-breaking change that adds functionality)

✅ How has this been tested?

  • tests_v9/unit/model/gcp_dataplex_aspect_type_test.py and bigquery_routine_test.py (31 tests) cover:

    • the literal app qualifiedName and validate(for_creation=True);
    • None/blank inputs and a malformed resource name / schema qn;
    • derived vs explicit parent attributes;
    • the create alias, the updater and trim_to_required.
  • tests_v9/integration/dataplex_asset_test.py creates:

    • a Dataplex connection with an aspect type;
    • a BigQuery connection → database → schema → routine.

    It retrieves each one, asserts the qualifiedName, attributes, connector and atlan_schema link, then purges. It passed locally against a test tenant (2 passed).

  • pytest tests_v9/unit: 9258 passed, 5 skipped.

  • Generator fidelity: regenerating from the models branch with these overlays reproduces the committed classes exactly, and pre-commit leaves them unchanged.


📋 Checklist

  • My code follows the project’s style guidelines
  • I’ve performed a self-review of my code
  • There are no new warnings from my changes
  • I’ve added tests to cover my changes
  • All new and existing tests pass locally

🤖 Generated with Claude Code

https://claude.ai/code/session_017CGtFm63WJtCpeBjakhwhx

cmgrote and others added 2 commits October 21, 2026 14:06
…on test

Adds AtlanConnectorType.DATAPLEX (needed by Connection.creator) and regenerates
the two classes from the rebased models branch.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017CGtFm63WJtCpeBjakhwhx
@cmgrote cmgrote added the run_pyatlan_v9_integration_tests To trigger pyatlan v9 (msgspec-based) integration tests suite (name the workflow checks) label Oct 21, 2026
@cmgrote cmgrote closed this Oct 21, 2026
@cmgrote cmgrote reopened this Oct 21, 2026
@cmgrote
cmgrote merged commit 373a950 into main Oct 21, 2026
129 of 294 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

run_pyatlan_v9_integration_tests To trigger pyatlan v9 (msgspec-based) integration tests suite (name the workflow checks)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant