Iceberg/Unity Catalog ATTACH support and SQL pushdown generalization - #86
Merged
adsharma merged 2 commits intoSep 19, 2026
Merged
Conversation
Iceberg and Unity Catalog are SQL engines, so besides LOAD FROM they now support ATTACH as graph-queryable databases, reusing the shared DuckDB-backed catalog machinery: - iceberg: new IcebergStorageExtension (DBTYPE ICEBERG). ATTACH '<warehouse>' AS <alias> attaches the REST catalog inside the embedded DuckDB instance under the attached alias; connection options come from the iceberg_* settings. IcebergConnector::connect() honors the attach path/alias instead of ignoring them, with server-independent validation (missing warehouse/endpoint, bad SCHEMA/SKIP_UNSUPPORTED_TABLE). - unity_catalog: fix DuckDBCatalog using the attach path instead of the attached alias as its catalog name ( broke ATTACH ... AS <alias> whenever alias != path); LOAD-first/INSTALL-fallback for the uc_catalog DuckDB extension so pre-installed copies work offline without a repository origin conflict; drop duplicated delta install. - shared DuckDB catalog: init() collects tables across all result chunks; AttachedDuckDBDatabase::getTableColumnNames() accepts qualified catalog[.schema].table references and scopes information_schema accordingly (unqualified fallback preserved). Tests: new iceberg_attach.test (runnable ATTACH validation + SKIPped live REST-catalog flow); unity_catalog.test gains a SKIPped aliased-attach and pushdown case (both need live servers, per repo convention).
adsharma
force-pushed
the
feat/iceberg-unity-attach-pushdown
branch
from
September 19, 2026 01:13
fe8faf0 to
f5e1ca5
Compare
…tic pushdown test - attached_duckdb_database.h: narrow the getTableColumnNames fallback to most-to-least specific filters (catalog+schema, schema-only, then unqualified last resort) with comments documenting the 2-part/3-part convention shared with the push-down optimizer; mark header helpers inline instead of static. - iceberg_connector.cpp: validate configuration before creating the embedded DuckDB instance instead of duplicating the setup block in both branches; behavior unchanged. - iceberg storage CMakeLists: drop duckdb_scan.cpp, already provided by the delta_connector static lib linked into the extension. - README: document that an empty ATTACH path falls back to the iceberg_warehouse option. - duckdb_rel.test: new hermetic DuckDBAttachAliasJoinPushdown case covering attach-under-alias, catalog init, column resolution and join pushdown with property projections (no external server needed).
adsharma
added a commit
to adsharma/ladybug
that referenced
this pull request
Sep 19, 2026
ForeignJoinPushDownOptimizer no longer assumes DuckDB-file-catalog shapes: - Column lookup passes the qualified catalog.schema.table reference through to the attached database (which scopes information_schema when it can), with an unqualified fallback so attached databases from older extension builds keep working. - Rel-table join columns prefer the src/dst naming convention (the same convention createForeignRelTable uses) instead of blindly taking the first two columns, falling back to ordinal position. SQL catalogs such as Iceberg REST or Unity Catalog do not guarantee endpoint columns come first. - ATTACH error hint now also suggests loading the iceberg/uc_catalog extensions. Backward compatible in both directions: works with old extension builds (unqualified fallback) and old core builds are unaffected by the extension-side changes. Companion extensions PR: LadybugDB/extensions#86.
adsharma
added a commit
to LadybugDB/ladybug
that referenced
this pull request
Sep 19, 2026
ForeignJoinPushDownOptimizer no longer assumes DuckDB-file-catalog shapes: - Column lookup passes the qualified catalog.schema.table reference through to the attached database (which scopes information_schema when it can), with an unqualified fallback so attached databases from older extension builds keep working. - Rel-table join columns prefer the src/dst naming convention (the same convention createForeignRelTable uses) instead of blindly taking the first two columns, falling back to ordinal position. SQL catalogs such as Iceberg REST or Unity Catalog do not guarantee endpoint columns come first. - ATTACH error hint now also suggests loading the iceberg/uc_catalog extensions. Backward compatible in both directions: works with old extension builds (unqualified fallback) and old core builds are unaffected by the extension-side changes. Companion extensions PR: LadybugDB/extensions#86.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Iceberg and Unity Catalog are SQL engines, so besides LOAD FROM they now support ATTACH as graph-queryable databases, reusing the shared DuckDB-backed catalog machinery (DuckDBCatalog + AttachedDuckDBDatabase).
Iceberg (new ATTACH support)
IcebergStorageExtension(DBTYPE ICEBERG):ATTACH '<warehouse>' AS <alias>attaches the REST catalog inside the embedded DuckDB instance under the attached alias; endpoint/auth come from theiceberg_*options; namespace viaSCHEMAattach option (defaultdefault).IcebergConnector::connect()honors the attach path (warehouse) and alias instead of ignoring them;LOAD FROMbehavior (fixediceberg_catalogalias, file scans) is unchanged.SCHEMA/SKIP_UNSUPPORTED_TABLEproduce clear errors.Unity Catalog (fixes)
DuckDBCatalogis now built with the attached alias (not the path) as its catalog name, soATTACH ... AS <alias>works when alias != path; generated SQL and information_schema lookups reference the alias.uc_catalog(e.g. from the stable repo) is reused offline instead of failing on acore_nightlyorigin conflict; also drops a duplicatedinstall delta.Shared pushdown generalization
DuckDBCatalog::init()collects tables across all result chunks (large catalogs no longer lose tables past the first chunk).AttachedDuckDBDatabase::getTableColumnNames()accepts qualifiedcatalog[.schema].tablereferences and scopesinformation_schemaaccordingly, with unqualified fallback.Tests
iceberg/test/test_files/iceberg_attach.test: runnable ATTACH-validation case plus aSKIPped live REST-catalog flow (ATTACH, LOAD FROM, MATCH, CREATE REL + EXPLAIN pushdown, DETACH).unity_catalog.test: newSKIPped aliased-attach, MATCH and pushdown case (needs a UC server at 127.0.0.1:8080, per repo convention).Companion main-repo PR generalizes
ForeignJoinPushDownOptimizer(qualified lookups with unqualified fallback, src/dst join-column detection) and extends the ATTACH error hint. The two PRs are independently mergeable in either order.