Repository navigation
Core: Load tables through the v2 REST endpoint - #18782
Open
dhruvarya-db wants to merge 6 commits into
Open
dhruvarya-db wants to merge 6 commits into
dhruvarya-db wants to merge 6 commits into
Conversation
The v4 REST proposal (apache#17642) adds GET /v2/{prefix}/namespaces/{namespace}/tables/{table} so that format version 4 tables are never returned to clients that only use v1. Load and refresh tables through v2 when the server advertises it and fall back to v1 otherwise. The test server serves the v2 route but does not advertise it yet. Co-authored-by: Isaac <no-reply@databricks.com>
The REST spec proposal for format version 4 says v1 load must not return v4 tables. Advertise the v2 load endpoint from the test server by default and reject v1 loads of format version 4 tables with a 400 that points at v2. Update load-path assertions in existing tests from v1 to v2. Co-authored-by: Isaac <no-reply@databricks.com>
The REST spec proposal for format version 4 says the v1 load endpoint must not return v4 tables. Enforce this in CatalogHandlers.loadTable, which servers use for v1 loads, and add CatalogHandlers.loadTableV2 for the v2 endpoint. The reference test server now uses both handlers instead of its own check. Co-authored-by: Isaac <no-reply@databricks.com>
…ables A v1 load of a format version 4 table now fails with a dedicated type, a subclass of BadRequestException, so clients can tell it apart from a malformed request. The table error handler maps the type back to the exception on the client; clients that do not know the type still get BadRequestException. Co-authored-by: Isaac <no-reply@databricks.com>
dhruvarya-db
marked this pull request as draft
November 4, 2026 00:16
Refresh derived the v2 load path from the table's v1 path, so a custom path from a subclass fell back to v1 and failed on servers that list only v2. Have RESTSessionCatalog pass the load path it already chooses to the table operations through new newTableOps overloads, and deprecate the current overloads. This removes the v1-to-v2 path mapping in ResourcePaths. Co-authored-by: Isaac <no-reply@databricks.com>
dhruvarya-db
marked this pull request as ready for review
November 4, 2026 00:42
Adding newTableOps overloads meant that subclasses overriding the existing hooks were no longer called. Keep calling the existing hooks and set the load path on the returned table operations instead. Operations that a subclass built with its own path keep refreshing from that path. This removes the new overloads, the deprecations and the revapi entries. Co-authored-by: Isaac <no-reply@databricks.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Add a new v2 load endpoint so that clients can load format version 4 tables. This change is outlined in the REST spec PR for format version 4, #17642. Only the v2 endpoint can return a version 4 table, so clients that know only versions 1 to 3 never get one.
What changes
The Java REST client now uses the v2 endpoint to load and refresh a table when the server lists v2 in its configuration response. If the server does not list v2, the client uses v1 as before. Commits always use v1. Servers that do not know about v2 see no change, and the default list of endpoints in the client does not change.
If the server lists v2, the client does not need the v1 load endpoint. Refresh uses the same load path as the first load.
RESTSessionCatalogsets this load path on the table operations after thenewTableOpshooks return them. Subclasses that override these hooks keep working with no change. If a subclass makes the operations with its own table path, refresh continues to use that path.CatalogHandlers.loadTable( v1 load handler ): rejects a version 4 table with the newUnsupportedFormatVersionException, and the error message tells the user to use v2. This exception is a subclass ofBadRequestException, so the server sends HTTP 400.CatalogHandlers.loadTableV2: returns tables at all format versions. A server that usesCatalogHandlersgets the v1 rule with no more changes. If such a server usesloadTablefor version 4 tables now, it must useloadTableV2for the v2 endpoint.The Java client changes this error back to
UnsupportedFormatVersionException, so code can find it without a check of the message. An old client that does not know the type getsBadRequestExceptionwith the same message.The reference test server uses these two handlers and lists v2. A v1 load of a version 4 table gets HTTP 400.
Testing
New coverage
TestRESTCatalogV2LoadTableis a new suite. It checks that the client loads and refreshes tables with format versions 1 to 4 through v2, and that commits use v1.UnsupportedFormatVersionExceptionfor a version 4 table.TestCatalogHandlersLoadTableis a new suite. It checks thatloadTablereturns tables with format versions 1 to 3 and rejects version 4, and thatloadTableV2returns all versions.TestErrorHandlershas new tests. They check that the client getsUnsupportedFormatVersionExceptionfor this error type, andBadRequestExceptionfor other HTTP 400 errors.TestResourcePathsandTestEndpointcheck the new path and the new endpoint.Coverage from the current suites
The reference test server now lists v2, so the current REST suites use v2 for each load. These suites are
TestRESTCatalog,TestFreshnessAwareLoading,TestReferencedByQueryParam, and the REST compatibility kit inopen-api. TestRESTCatalog and TestFreshnessAwareLoading override the newTableOps hooks with the signatures from before this PR, and they pass with no change.AI Disclosure