Skip to content

feat(iceberg): re-pin a view's Iceberg tables each time a query names it - #2784

Draft
NoahKusaba wants to merge 4 commits into
apache:mainfrom
NoahKusaba:feat/iceberg-repin-views
Draft

NoahKusaba wants to merge 4 commits into
apache:mainfrom
NoahKusaba:feat/iceberg-repin-views

Conversation

@NoahKusaba

@NoahKusaba NoahKusaba commented Nov 6, 2026 •

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

Closes #2726.
Closes #2882.

Follow-up to #2638, split out at andygrove's request so the plan rewriting gets its own review.

Rationale for this change

#2638 pins each catalog-backed Iceberg table to one version when a query looks it up. DataFusion stores a view as the plan it was created with, so a view over an Iceberg table keeps reading the version from when the view was created. In single-process DataFusion, the same view reads new commits on every run.

What changes are included in this PR?

When a query names a view, the pinning schema provider re-pins the view's stored plan: every scan of a pinned catalog-backed table, including scans inside subqueries, is replaced with the version its catalog serves now. Each distinct table is loaded once, and the loads run concurrently. PinnedTable keeps the IcebergCatalogTableProvider it was pinned from (never sent between nodes), so it can be pinned again.

A view is any provider DataFusion inlines, one whose get_logical_plan() returns a plan: a ViewTable from CREATE VIEW, a DataFrame's into_view or into_temporary_view, or a custom view. The lookup returns a RepinnedView that wraps the original and replaces only its plan, so the view keeps its table type, schema, definition, constraints and column defaults.

A lookup never fails because a table can't be loaded, for example because it was dropped from its catalog or the catalog is unreachable. The PinnedTable holds the load error instead (Result<Table, Arc<DataFusionError>>) and still knows the catalog table it came from. A query fails with the error only when it reads the table: register_iceberg adds an analyzer rule, RequireLoadedTables, that fails a query reading such a table with its load error while the query is planned, before the client encodes it. So:

  • DESCRIBE, DROP VIEW, CREATE OR REPLACE VIEW and DROP TABLE still work, since they look the table or view up without reading it. Before, DROP TABLE on a registered table its catalog couldn't load failed (Iceberg: DROP TABLE fails on a registered table its catalog can't load #2882).
  • A view is always inlined, even while a table it reads is gone, so a view created over it stores a plan that still names the table, and reads it once it loads again.
  • Reading a table that's gone, directly, through a view or in a subquery, with or without a filter, fails with the iceberg error and its cause intact, rather than with the Internal error the client wraps encoding failures in (Client hides plan-encoding errors behind an Internal error #2885).

This matches DataFusion, where IcebergCatalogTableProvider loads its table in scan(). The analyzer rule is a stopgap with a TODO to remove it once #2881 lands: tables would then be loaded when a query runs, and a table that can't be loaded fails the query there.

It re-pins the stored plan rather than re-planning the view's SQL. SchemaProvider::table gets no session state to plan with, and a view created from a DataFrame has no SQL. Nested views need no recursion, since DataFusion inlines a view into any plan built over it.

It also adds a TODO on ComposedLogicalExtensionCodec's error handling in ballista-core, to be fixed as part of upstreaming it to DataFusion: it joins its codecs' errors into one string, so the cause of an encoding failure can't be reached through the error chain.

Limitations:

  • A reused DataFrame or Flight SQL prepared statement still reads the version from when it was planned. Neither looks its tables up again.
  • A query that reads a table both through a view and directly looks it up twice, so a commit landing between the two lookups could give them different versions.

Are these changes tested?

Yes, with standalone Ballista over a memory catalog:

  • should_read_a_views_latest_version creates a view over a table and a second view over the first with a subquery on the table, commits a row, and checks that both views return it.
  • should_read_a_dataframe_views_latest_version does the same for into_view and into_temporary_view, and checks that each keeps its table type.
  • should_read_a_view_created_while_its_table_was_gone creates a view over a view while the table is gone, checks that reading it fails, then brings the table back and checks that it reads the latest version.
  • should_fail_to_read_a_table_that_is_gone checks the error from reading the table directly, through a view, and in a subquery, with and without a filter.
  • should_describe_drop_and_replace_while_a_table_is_gone checks that DESCRIBE, DROP VIEW, CREATE OR REPLACE VIEW and DROP TABLE still work.

Each error is compared in full against the error the catalog returns when loading the table, so it doesn't depend on DataFusion's or iceberg's wording.

Are there any user-facing changes?

Yes: views over Iceberg tables now read the latest version, and a table or view that can't be loaded fails only when read, so it can still be described, dropped or replaced. The docs' limitation for views is narrowed to reused DataFrames and prepared statements.

Follow-ups

@andygrove andygrove left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @NoahKusaba, this is a really nice follow-up. Views reading the latest version brings us in line with what DataFusion does on its own. Keeping DESCRIBE, DROP VIEW and CREATE OR REPLACE VIEW working when a table is gone is a thoughtful touch. The re-pinning itself looks right to me. Each distinct table loads once, subqueries are covered, and each scan keeps its projected schema. One question about views built on top of a broken view, and a small test suggestion, both inline.

Comment thread integrations/iceberg/src/catalog.rs Resolved
Comment thread integrations/iceberg/tests/scan.rs Resolved
@NoahKusaba

Copy link
Copy Markdown
Contributor Author

test failure is fake, apparently the CI can't download apache rat

@andygrove andygrove left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the update @NoahKusaba. Both earlier points are addressed nicely. Holding the load error in PinnedTable is a neat way to keep a view over a missing table inlined, and it fixes #2882 too. I traced the re-pinning through and it looks right to me. DataFusion inlines views when the plan is built, so nested views do pick up the latest version. One concern about the error a user sees for a missing table, and two small things, all inline.

let Some(provider) = table.downcast_ref::<IcebergCatalogTableProvider>() else {
return Ok(Some(table));
if let Some(provider) = table.downcast_ref::<IcebergCatalogTableProvider>() {
let table = load(provider).await;

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One thing I noticed while tracing the error path. A plain SELECT * FROM t on a table that was dropped used to fail at lookup with a clean No such table. Now it fails when the client encodes the plan, so it comes back as an Internal error with the "likely a bug in DataFusion" text from distributed_query.rs. That's probably the most common way users will hit a missing table, so it feels like a step back for them. Would you be up for landing #2885 before or with this one? It would also be nice to have a test that reads t directly while it's gone, alongside the view cases.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed, that's the common case. #2885 wouldn't reach it yet, though. ballista-iceberg pins ballista-core at a DataFusion 55 commit until datafusion-iceberg supports 56, and the composed codec would still flatten the cause.

So register_iceberg now adds an analyzer rule, RequireLoadedTables. It fails a query that reads a table its catalog couldn't load, with the load error, while the query is planned. Direct reads, view reads and subqueries, with or without a filter, now all fail before anything is encoded:

require_loaded_iceberg_tables
caused by
IcebergError(TableNotFound)
caused by
Execution error: TableNotFound => No such table: …

DataFusion doesn't analyze DDL or DESCRIBE, so dropping, replacing and describing still work while a table is gone, and #2882 stays fixed.

The rule is a stopgap with a TODO to remove it once #2881 lands, since a table that can't be loaded will then fail the query where it's loaded. should_fail_to_read_a_table_that_is_gone now reads t directly, through a view and in a subquery, each with and without a filter.

Comment thread integrations/iceberg/tests/scan.rs Resolved
Comment thread integrations/iceberg/src/catalog.rs Resolved
@NoahKusaba
NoahKusaba force-pushed the feat/iceberg-repin-views branch from e5b7f95 to 02b9a17 Compare November 20, 2026 20:56
NoahKusaba and others added 2 commits November 20, 2026 16:43
register_iceberg adds an analyzer rule that returns a pinned table's load error before the client encodes the plan, so direct, view and subquery reads, with or without a filter, fail with the iceberg error rather than an Internal encoding error. Temporary until tables are pinned when a query runs (apache#2881).
@NoahKusaba
NoahKusaba requested a review from andygrove November 20, 2026 21:46
@NoahKusaba
NoahKusaba marked this pull request as draft November 22, 2026 17:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation Improvements or additions to documentation

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Iceberg: DROP TABLE fails on a registered table its catalog can't load Iceberg: re-plan views from their SQL so they read the latest table version

2 participants