Repository navigation
Core: Add table-location to REST table responses - #18783
Draft
dhruvarya-db wants to merge 4 commits into
Draft
dhruvarya-db wants to merge 4 commits into
dhruvarya-db wants to merge 4 commits into
Conversation
Base commit with the changes from apache#18782, squashed into one commit. Co-authored-by: Isaac <no-reply@databricks.com>
Format version 4 makes the table location optional in metadata and allows relative paths, so the REST spec proposal for v4 has the catalog supply the location as table-location on load, commit and unregister responses. Read and write the field, apply it to the client's metadata without recording a change, and send it from CatalogHandlers for format version 4 tables only. Co-authored-by: Isaac <no-reply@databricks.com>
Serve a format version 4 table whose metadata omits location and supplies it as table-location, and check that the client uses it, appends succeed, and the commit does not send the location back as SetLocation. Co-authored-by: Isaac <no-reply@databricks.com>
…tests CatalogHandlers.loadTable now rejects format version 4 tables, so the table-location test loads tables at every format version through loadTableV2. Co-authored-by: Isaac <no-reply@databricks.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR builds on #18782, and its first commit contains the changes from #18782. To review only the changes in this PR, use https://github.com/apache/iceberg/pull/18783/changes/3c43da7bce1816a74a4a4a7828160dedbb40903d..a7fbb38d7287eac0fd4cfded6e7197df7829fd9e
Why
In format version 4, the
locationfield in table metadata is optional. Paths in the metadata can also be relative to the table location. A client must know the table location to find these files and to write new files. The format spec says that the catalog must supply the location when the metadata does not include it. This change is outlined in the REST spec PR for format version 4, #17642. It adds atable-locationfield for this purpose to load, commit, and unregister responses.What changes
The Java REST client reads
table-locationfrom load, commit, and unregister responses. If the field is present, the client uses it as the table location, instead of thelocationfield in the metadata. The client does not send this location back to the server in later commits, so the catalog keeps control of the location. The client keeps the response as it was received, so the same response gives the same JSON again.The client rejects a
table-locationthat is not an absolute path with a URI scheme. It also rejects a version 4 response that has no location at all. Without this check, the error occurs later in a write, and the cause is hard to find.The server code in
CatalogHandlerssendstable-locationonly for version 4 tables. Responses for versions 1 to 3 do not change.One result can be unexpected. A create transaction can start from a staged response that gets its location from
table-location. The create commit then sends that location to the server. This is necessary, because the create commit is the step that saves the location of the new table.Testing
New coverage
TestLoadTableResponseParserandTestUnregisterTableResponseParserhave new tests. They check JSON with and withouttable-location, and thattable-locationoverrideslocation.TestCatalogHandlersTableLocationis a new suite. It checks that the server sendstable-locationonly for version 4 tables.TestRESTCatalogFormatV4is a new end-to-end suite. It loads a version 4 table withoutlocationover HTTP and appends data to it.table-location.Coverage from the current suites
The current parser tests and REST suites use tables with format versions 1 to 3. They pass with no change, which shows that responses for these versions are the same as before.
AI Disclosure