Skip to content

Core: Load tables through the v2 REST endpoint - #18782

Open
dhruvarya-db wants to merge 6 commits into
apache:mainfrom
dhruvarya-db:rest-v2-load-endpoint-client
Open

dhruvarya-db wants to merge 6 commits into
apache:mainfrom
dhruvarya-db:rest-v2-load-endpoint-client

Conversation

@dhruvarya-db

@dhruvarya-db dhruvarya-db commented Nov 4, 2026 •

Copy link
Copy Markdown
Contributor

Why

Add a new v2 load endpoint so that clients can load format version 4 tables. This change is outlined in the REST spec PR for format version 4, #17642. Only the v2 endpoint can return a version 4 table, so clients that know only versions 1 to 3 never get one.

What changes

The Java REST client now uses the v2 endpoint to load and refresh a table when the server lists v2 in its configuration response. If the server does not list v2, the client uses v1 as before. Commits always use v1. Servers that do not know about v2 see no change, and the default list of endpoints in the client does not change.

If the server lists v2, the client does not need the v1 load endpoint. Refresh uses the same load path as the first load. RESTSessionCatalog sets this load path on the table operations after the newTableOps hooks return them. Subclasses that override these hooks keep working with no change. If a subclass makes the operations with its own table path, refresh continues to use that path.

CatalogHandlers.loadTable ( v1 load handler ): rejects a version 4 table with the new UnsupportedFormatVersionException, and the error message tells the user to use v2. This exception is a subclass of BadRequestException, so the server sends HTTP 400. CatalogHandlers.loadTableV2: returns tables at all format versions. A server that uses CatalogHandlers gets the v1 rule with no more changes. If such a server uses loadTable for version 4 tables now, it must use loadTableV2 for the v2 endpoint.

The Java client changes this error back to UnsupportedFormatVersionException, so code can find it without a check of the message. An old client that does not know the type gets BadRequestException with the same message.

The reference test server uses these two handlers and lists v2. A v1 load of a version 4 table gets HTTP 400.

Testing

New coverage

  • TestRESTCatalogV2LoadTable is a new suite. It checks that the client loads and refreshes tables with format versions 1 to 4 through v2, and that commits use v1.
  • The same suite checks that a client that uses only v1 gets UnsupportedFormatVersionException for a version 4 table.
  • It also checks a server that lists only v2 and a server that lists only v1. It checks that refresh uses the v2 path, and that operations with a custom path continue to refresh from that path. It also checks that a subclass that overrides the newTableOps hooks is still called and refreshes through v2.
  • TestCatalogHandlersLoadTable is a new suite. It checks that loadTable returns tables with format versions 1 to 3 and rejects version 4, and that loadTableV2 returns all versions.
  • TestErrorHandlers has new tests. They check that the client gets UnsupportedFormatVersionException for this error type, and BadRequestException for other HTTP 400 errors.
  • TestResourcePaths and TestEndpoint check the new path and the new endpoint.
  • The v2 tests fail if the client uses v1 instead of v2. The rejection tests fail if the server does not do the check.

Coverage from the current suites

The reference test server now lists v2, so the current REST suites use v2 for each load. These suites are TestRESTCatalog, TestFreshnessAwareLoading, TestReferencedByQueryParam, and the REST compatibility kit in open-api. TestRESTCatalog and TestFreshnessAwareLoading override the newTableOps hooks with the signatures from before this PR, and they pass with no change.


AI Disclosure

  • Model: Claude Opus 5.5
  • Platform/Tool: Claude Code
  • Human Oversight: [Manual review of PR. Reworded parts of the description manually]
  • Prompt Summary: Load and refresh REST tables through the v2 load endpoint when the server advertises it, falling back to v1, and make the REST test server serve format version 4 tables only through v2.

durovo and others added 4 commits November 4, 2026 00:02
The v4 REST proposal (apache#17642) adds GET /v2/{prefix}/namespaces/{namespace}/tables/{table}
so that format version 4 tables are never returned to clients that only use v1.
Load and refresh tables through v2 when the server advertises it and fall back
to v1 otherwise. The test server serves the v2 route but does not advertise it
yet.

Co-authored-by: Isaac <no-reply@databricks.com>
The REST spec proposal for format version 4 says v1 load must not return v4
tables. Advertise the v2 load endpoint from the test server by default and
reject v1 loads of format version 4 tables with a 400 that points at v2.
Update load-path assertions in existing tests from v1 to v2.

Co-authored-by: Isaac <no-reply@databricks.com>
The REST spec proposal for format version 4 says the v1 load endpoint must not
return v4 tables. Enforce this in CatalogHandlers.loadTable, which servers use
for v1 loads, and add CatalogHandlers.loadTableV2 for the v2 endpoint. The
reference test server now uses both handlers instead of its own check.

Co-authored-by: Isaac <no-reply@databricks.com>
…ables

A v1 load of a format version 4 table now fails with a dedicated type, a
subclass of BadRequestException, so clients can tell it apart from a malformed
request. The table error handler maps the type back to the exception on the
client; clients that do not know the type still get BadRequestException.

Co-authored-by: Isaac <no-reply@databricks.com>
@dhruvarya-db
dhruvarya-db marked this pull request as draft November 4, 2026 00:16
Refresh derived the v2 load path from the table's v1 path, so a custom path
from a subclass fell back to v1 and failed on servers that list only v2. Have
RESTSessionCatalog pass the load path it already chooses to the table
operations through new newTableOps overloads, and deprecate the current
overloads. This removes the v1-to-v2 path mapping in ResourcePaths.

Co-authored-by: Isaac <no-reply@databricks.com>
@dhruvarya-db
dhruvarya-db marked this pull request as ready for review November 4, 2026 00:42
Adding newTableOps overloads meant that subclasses overriding the existing
hooks were no longer called. Keep calling the existing hooks and set the load
path on the returned table operations instead. Operations that a subclass
built with its own path keep refreshing from that path. This removes the new
overloads, the deprecations and the revapi entries.

Co-authored-by: Isaac <no-reply@databricks.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants