Skip to content

Core: Add table-location to REST table responses - #18783

Draft
dhruvarya-db wants to merge 4 commits into
apache:mainfrom
dhruvarya-db:rest-v4-table-location-upstream
Draft

dhruvarya-db wants to merge 4 commits into
apache:mainfrom
dhruvarya-db:rest-v4-table-location-upstream

Conversation

@dhruvarya-db

@dhruvarya-db dhruvarya-db commented Nov 4, 2026 •

Copy link
Copy Markdown
Contributor

This PR builds on #18782, and its first commit contains the changes from #18782. To review only the changes in this PR, use https://github.com/apache/iceberg/pull/18783/changes/3c43da7bce1816a74a4a4a7828160dedbb40903d..a7fbb38d7287eac0fd4cfded6e7197df7829fd9e

Why

In format version 4, the location field in table metadata is optional. Paths in the metadata can also be relative to the table location. A client must know the table location to find these files and to write new files. The format spec says that the catalog must supply the location when the metadata does not include it. This change is outlined in the REST spec PR for format version 4, #17642. It adds a table-location field for this purpose to load, commit, and unregister responses.

What changes

The Java REST client reads table-location from load, commit, and unregister responses. If the field is present, the client uses it as the table location, instead of the location field in the metadata. The client does not send this location back to the server in later commits, so the catalog keeps control of the location. The client keeps the response as it was received, so the same response gives the same JSON again.

The client rejects a table-location that is not an absolute path with a URI scheme. It also rejects a version 4 response that has no location at all. Without this check, the error occurs later in a write, and the cause is hard to find.

The server code in CatalogHandlers sends table-location only for version 4 tables. Responses for versions 1 to 3 do not change.

One result can be unexpected. A create transaction can start from a staged response that gets its location from table-location. The create commit then sends that location to the server. This is necessary, because the create commit is the step that saves the location of the new table.

Testing

New coverage

  • TestLoadTableResponseParser and TestUnregisterTableResponseParser have new tests. They check JSON with and without table-location, and that table-location overrides location.
  • The same tests check that the client records no metadata change, and that it rejects a relative location or no location.
  • TestCatalogHandlersTableLocation is a new suite. It checks that the server sends table-location only for version 4 tables.
  • TestRESTCatalogFormatV4 is a new end-to-end suite. It loads a version 4 table without location over HTTP and appends data to it.
  • This test checks that the client uses the location from the catalog, and that the commit does not send the location back. It fails if the server stops sending table-location.

Coverage from the current suites

The current parser tests and REST suites use tables with format versions 1 to 3. They pass with no change, which shows that responses for these versions are the same as before.


AI Disclosure

  • Model: Claude Opus 5.5
  • Platform/Tool: Claude Code
  • Human Oversight: [under self-review]
  • Prompt Summary: Carry a catalog-supplied table location on REST table responses and apply it to the client's table metadata for format version 4 tables.

durovo and others added 4 commits November 4, 2026 01:48
Base commit with the changes from apache#18782, squashed into one commit.

Co-authored-by: Isaac <no-reply@databricks.com>
Format version 4 makes the table location optional in metadata and allows
relative paths, so the REST spec proposal for v4 has the catalog supply the
location as table-location on load, commit and unregister responses. Read and
write the field, apply it to the client's metadata without recording a change,
and send it from CatalogHandlers for format version 4 tables only.

Co-authored-by: Isaac <no-reply@databricks.com>
Serve a format version 4 table whose metadata omits location and supplies it as
table-location, and check that the client uses it, appends succeed, and the
commit does not send the location back as SetLocation.

Co-authored-by: Isaac <no-reply@databricks.com>
…tests

CatalogHandlers.loadTable now rejects format version 4 tables, so the
table-location test loads tables at every format version through loadTableV2.

Co-authored-by: Isaac <no-reply@databricks.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants