Skip to content

OpenAPI: IRC spec changes for v4 (relax responses in place) - #17642

Open
dhruvarya-db wants to merge 9 commits into
apache:rest-v4-changesfrom
dhruvarya-db:rest-v4-relax-in-place
Open

dhruvarya-db wants to merge 9 commits into
apache:rest-v4-changesfrom
dhruvarya-db:rest-v4-relax-in-place

Conversation

@dhruvarya-db

@dhruvarya-db dhruvarya-db commented Aug 13, 2026 •

Copy link
Copy Markdown
Contributor

Alternative to #17543.

Same goal — let the REST catalog carry table format version 4 — but this version changes the existing response models in place instead of introducing V2 copies of them, and it does not bump the API version. The only new endpoint is GET /v2/{prefix}/namespaces/{namespace}/tables/{table}, which returns the same LoadTableResponse as v1. Format version 4 tables are loaded only through it, so a client that does not support format version 4 is never sent one.

Changes:

  • loadTableV2 (GET /v2/.../tables/{table}): new endpoint that may return tables at any format version, including format version 4. It shares its parameters and response model with v1 loadTable.
  • loadTable (GET /v1/.../tables/{table}): servers must not return format version 4 tables from it. Such a request fails with a 422 whose error type is UnsupportedTableFormatVersionException.
  • The If-None-Match and snapshots parameters are defined once in components/parameters and shared by both load endpoints.
  • Snapshot: add root-manifest and make manifest-list optional, with a oneOf requiring exactly one of the two. Format versions 1-3 use manifest-list; version 4 uses root-manifest.
  • TableMetadata: raise the format-version maximum to 4.
  • TableLocation: new shared schema for the table's base location, used to resolve relative paths in v4 metadata. LoadTableResult, UnregisterTableResult, and CommitTableResponse gain an optional table-location field that references it.
  • CommitTableResponse: metadata-location is no longer required (absent when the table has no client-visible metadata location). LoadTableResult already treated it as optional; its description now also covers that case.

This is a proposal to drive discussion; v4 is still in development.

@@ -3851,11 +3872,35 @@ components:

LoadTableResult:

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In this approach, I think we still need a GET v2/../tables/{table} endpoint. V4 tables should only be loaded through the new endpoint to avoid the possibility of returning a V4 table to an older client.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was relying on the fact that at least the java implementation (TableMetadataParser.java) checks format-version before parsing any of the other fields. But it is understandable that other implementations might not follow the same approach. I have forked a new GET v2/../tables/{table} endpoint as per your suggestion. I did not create a new TableMetadataV2 though (doing so will land us with many duplicated classes and endpoints again).

@amogh-jahagirdar amogh-jahagirdar Aug 19, 2026 •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah we shouldn't make assumptions on how other parser implementations work. Compatibility/expected behavior from arbitrary clients largely needs to be reasoned backwards from the shape of the structure not implementation details on how that structure may be interpreted.

I think adding the endpoint is the most reliable way, we almost certainly don't need a new in-memory representaation of TableMetadata.

@github-actions

Copy link
Copy Markdown

This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@iceberg.apache.org list. Thank you for your contributions.

@github-actions github-actions Bot added the stale label Sep 19, 2026
@github-actions

Copy link
Copy Markdown

This pull request has been closed due to lack of activity. This is not a judgement on the merit of the PR in any way. It is just a way of keeping the PR queue manageable. If you think that is incorrect, or the pull request requires review, you can revive the PR at any time.

Comment thread open-api/rest-catalog-open-api.yaml Outdated
Comment on lines +1042 to +1044
This endpoint must not return a format version 4 table.
A server must not serve a format version 4 table through this endpoint, so that a client that does not support format version 4 is never handed a response it cannot read.
Format version 4 tables are loaded through the `/v2` endpoint instead.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The first two sentences also say the same thing, and the text doesn't say what a server should return instead (400?).

Can we remove these lines?

Suggested change
This endpoint must not return a format version 4 table.
A server must not serve a format version 4 table through this endpoint, so that a client that does not support format version 4 is never handed a response it cannot read.
Format version 4 tables are loaded through the `/v2` endpoint instead.
Servers MUST NOT return format version 4 tables from this endpoint.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done. Added a short line describing the expected error (400: UnsupportedTableFormatVersionError)

Comment thread open-api/rest-catalog-open-api.yaml Outdated
Comment on lines +1324 to +1355


The response contains both configuration and table metadata. The configuration, if non-empty is used
as additional configuration for the table that overrides catalog configuration. For example, this
configuration may change the FileIO implementation to be used for the table.


The response also contains the table's full metadata, matching the table metadata JSON file.


The catalog configuration may contain credentials that should be used for subsequent requests for the
table. The configuration key "token" is used to pass an access token to be used as a bearer token
for table requests. Otherwise, a token may be passed using a RFC 8693 token type as a configuration
key. For example, "urn:ietf:params:oauth:token-type:jwt=<JWT-token>".
parameters:
- $ref: '#/components/parameters/data-access'
- name: If-None-Match
in: header
description:
An optional header that allows the server to return 304 (Not Modified) if the metadata
is current. The content is the value of the ETag received in a CreateTableResponse,
LoadTableResponse or CommitTableResponse.
required: false
schema:
type: string
- in: query
name: snapshots
description:
The snapshots to return in the body of the metadata via the `snapshots` field. Setting
the value to `all` would return the full set of snapshots currently valid for the table.
Setting the value to `refs` would load all snapshots referenced by branches or tags.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: if this endpoint stays, the If-None-Match and snapshots parameters and the description block are copied from v1 and will drift as soon as someone edits the v1 text. Can we define the parameters once in components/parameters, $ref them from both load operations, and point to the v1 description instead of repeating it?

This should be a compatible change even if we touch the older v1 endpoints.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Makes sense. I also did the same for table-location

@danielcweeks

Copy link
Copy Markdown
Contributor

@dhruvarya-db can you change this to point to the rest-v4-changes branch?

durovo and others added 3 commits November 12, 2026 00:02
Applies the format-version-4 changes to the existing table response schemas in place, without introducing versioned schemas or endpoints and without bumping the API version. Existing clients keep using the same endpoints; a client that only understands format versions 1-3 uses the `format-version` in the returned metadata to detect a v4 table and stop reading, as the spec already requires.

- Snapshot: add `root-manifest` and make `manifest-list` optional, with a oneOf requiring exactly one of the two. Format versions 1-3 use `manifest-list` and version 4 uses `root-manifest`.
- TableMetadata: raise the format-version maximum to 4; document that `location` is optional in v4.
- LoadTableResult / UnregisterTableResult: add `table-location` for resolving relative paths in v4 metadata.
- CommitTableResponse: add `table-location`; `metadata-location` is no longer required (absent for catalog-managed tables).

make lint and make generate both pass.
Adds GET /v2/{prefix}/namespaces/{namespace}/tables/{table}, which returns the same LoadTableResponse as the existing v1 endpoint. Format version 4 tables are served only through this endpoint, and the v1 loadTable description now states that it must not return a v4 table, so a client that does not support format version 4 is never handed a v4 response through the existing path.

The response model is unchanged and shared between the two endpoints; only the path differs, and no model or API version is bumped.

make lint and make generate both pass.
Defines the If-None-Match and snapshots parameters once in components/parameters and references them from both load operations, so the two cannot drift. The v2 loadTable description now refers to the v1 endpoint instead of repeating it, and the v1 guard is reduced to a single sentence.

Co-authored-by: Isaac <no-reply@databricks.com>
@dhruvarya-db
dhruvarya-db changed the base branch from main to rest-v4-changes November 12, 2026 00:04
@dhruvarya-db
dhruvarya-db force-pushed the rest-v4-relax-in-place branch from ab96647 to c5cf410 Compare November 12, 2026 00:04
durovo and others added 3 commits November 12, 2026 00:33
…adTable for v4 tables

The v1 loadTable endpoint must not return format version 4 tables. This documents what a server returns instead: a 400 with an UnsupportedTableFormatVersionException error, so a client can tell that the table needs the /v2 endpoint rather than seeing a generic bad request.

Co-authored-by: Isaac <no-reply@databricks.com>
…generic

The example no longer names a specific format version or endpoint, matching the other error examples, so it can be reused by other endpoints and version combinations.

Co-authored-by: Isaac <no-reply@databricks.com>
LoadTableResult, UnregisterTableResult, and CommitTableResponse each repeated the table-location definition, including its rules on absolute paths, when it must be present, and precedence over metadata.location. They now reference a single TableLocation schema, following PageToken, so the rules cannot drift between responses. The JSON shape is unchanged.

Co-authored-by: Isaac <no-reply@databricks.com>
@dhruvarya-db

Copy link
Copy Markdown
Contributor Author

@dhruvarya-db can you change this to point to the rest-v4-changes branch?

@danielcweeks Thanks for the review. The PR has been retargeted and your comments have been addressed.

Comment thread open-api/rest-catalog-open-api.py Outdated


The `metadata-location` field is optional.
It is absent when the metadata is staged but not committed, as in a create transaction, and when the table has no client-visible metadata location, as for a catalog-managed table where the catalog is the source of truth for table state and no metadata pointer need exist.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

First time I see this formalized. Happy to see it!

@gaborkaszab gaborkaszab left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

General question: do we want to also add ColumnFile list into DataFile with this PR?

@@ -3015,12 +3085,21 @@ components:
type: string

Snapshot:

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just heads-up, there might be new encryption related fields in Snapshot: #18503

@dhruvarya-db

Copy link
Copy Markdown
Contributor Author

@gaborkaszab

General question: do we want to also add ColumnFile list into DataFile with this PR?

I feel that it might be better to do that in a followup to keep the scope of this PR contained. I imagine that we will need to create v2 variants of the scan planning endpoints. A table could start out as a v1 table when the client first loads it but it might get upgraded with column files attached by the time we get to server-side scan planning.

Comment thread open-api/rest-catalog-open-api.yaml Outdated
description: Location of the snapshot's manifest list file
description:
Location of the snapshot's manifest list file.
Used for format versions 1-3 and must be absent for format version 4, which uses `root-manifest` instead.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

format version 4 and higher? Just a bit of future proofing

Comment thread open-api/rest-catalog-open-api.yaml Outdated
type: string
description:
Location of the snapshot's root manifest.
Required for format version 4 and must be absent for format versions 1-3.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same as above 4 and higher?

Comment thread open-api/rest-catalog-open-api.yaml Outdated
type: string
description:
Location of the committed table metadata file.
Absent when the table has no client-visible metadata location, as for a catalog-managed table where the catalog is the source of truth for table state.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we should avoid the "catalog-managed" table jargon, all tables are catalog managed in iceberg

Comment thread open-api/rest-catalog-open-api.yaml Outdated
400:
$ref: '#/components/responses/BadRequestErrorResponse'
description:
Bad Request - The request is malformed, or the table is at a format version that this endpoint does not serve.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For the unsupported case should we be using 406? That seems ot better fit the "I know what you're trying to do but I can't support it" use case we're going for.

Comment thread open-api/rest-catalog-open-api.py Outdated
The table metadata JSON is returned in the `metadata` field. The corresponding file location of table metadata should be returned in the `metadata-location` field, unless the metadata is not yet committed. For example, a create transaction may return metadata that is staged but not committed.
Clients can check whether metadata has changed by comparing metadata locations after the table has been created.

Clients must not require this field to be present, and must not use it to bypass the catalog for reads or commits.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we need to be a bit more careful with wording here " Clients must not require this field to be present, and must not use it to bypass the catalog for reads or commits.". It's being made optional in this PR, which is fine but the whole requirement here may falsely cause folks to think their client implementation is incompliant with the spec when it's not. Also it's catalog responsibility anyways to vend scoped credentials so the whole "bypas the catalog for reads or commits" also feels unneccessary to say.

The `table-location` field carries the table's base location.
Format version 4 allows location fields in metadata to be relative, and such paths must be resolved against the table location.
Format version 4 also makes `metadata.location` optional, so a table may have metadata that contains relative paths and omits `location`.
Servers must populate `table-location` for any such table, because it cannot be read otherwise.

@amogh-jahagirdar amogh-jahagirdar Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

cc @danielcweeks to double check this but I think table-location should be required whenever metadata.location is absent, not just when there are relative paths (that's how i'm interpreting "such table") . Even if every path in the metadata is absolute, the table spec requires a table location to be provided whenever location is omitted

Comment thread open-api/rest-catalog-open-api.yaml Outdated
Comment on lines +1094 to +1095
Servers MUST NOT return format version 4 tables from this endpoint.
A request to load a format version 4 table fails with a 400 `UnsupportedTableFormatVersionException`; clients should load it through the `/v2` endpoint instead.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
Servers MUST NOT return format version 4 tables from this endpoint.
A request to load a format version 4 table fails with a 400 `UnsupportedTableFormatVersionException`; clients should load it through the `/v2` endpoint instead.
Servers MUST NOT return format version 4 tables from this endpoint.
A request to load a format version 4 table fails with a 400 `UnsupportedTableFormatVersionException`; clients must load v4 tables through the `/v2` endpoint.

Comment thread open-api/rest-catalog-open-api.yaml Outdated
Comment on lines +1132 to +1141
400:
$ref: '#/components/responses/BadRequestErrorResponse'
description:
Bad Request - The request is malformed, or the table is at a format version that this endpoint does not serve.
content:
application/json:
schema:
$ref: '#/components/schemas/IcebergErrorResponse'
examples:
UnsupportedTableFormatVersion:
$ref: '#/components/examples/UnsupportedTableFormatVersionError'

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Similar to above, I think 422 is probably more accurate.

Comment thread open-api/rest-catalog-open-api.yaml Outdated
Comment on lines +4366 to +4368
description:
The table's base location, used to resolve relative paths in metadata.
Must be an absolute path with a URI scheme when present, and must be present when the returned metadata contains relative paths and omits `location`, because the metadata cannot be resolved otherwise.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: I don't think presence rules belong on a string schema. Whether the field is present depends on the response that contains it, and "must be present when..." is already in the LoadTableResult description, so now the rule is stated twice and the copies can drift. Can we limit this to what the value is ("The table's base location, used to resolve relative paths in metadata. Must be an absolute location with a URI scheme.") and keep the presence and precedence rules in the containing response descriptions?

Comment thread open-api/rest-catalog-open-api.yaml Outdated
summary: The table's format version is not served by this endpoint
value: {
"error": {
"message": "The given table's format version is not supported by this endpoint",

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: this message doesn't tell the user what to do. Can we follow the usual "Cannot X because Y" form and include the remedy? For example: "Cannot load format version 4 table through /v1: use the /v2 loadTable endpoint".

Comment thread open-api/rest-catalog-open-api.yaml Outdated


Servers MUST NOT return format version 4 tables from this endpoint.
A request to load a format version 4 table fails with a 400 `UnsupportedTableFormatVersionException`; clients should load it through the `/v2` endpoint instead.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think 422 is more accurate as a return code. This is not a malformed request, but a semantically incorrect request.

Comment thread open-api/rest-catalog-open-api.yaml Outdated
Comment on lines +1372 to +1373
Behaves exactly like the v1 `loadTable` endpoint, except that it may also return format version 4 tables.
Clients that support format version 4 should load tables through this endpoint.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we need to clean up this description. We plan on changing the behavior from v1 (e.g. read restrictions must be handled if seen), so saying it's just like v1 is not accurate.

Comment thread open-api/rest-catalog-open-api.yaml Outdated
Comment on lines +2339 to +2340
is current. The content is the value of the ETag received in a CreateTableResponse,
LoadTableResponse or CommitTableResponse.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
is current. The content is the value of the ETag received in a CreateTableResponse,
LoadTableResponse or CommitTableResponse.
is current.

We're reusing this component, so we shouldn't duplicate where it's valid here in the description.

Comment thread open-api/rest-catalog-open-api.yaml Outdated
description: Location of the snapshot's manifest list file
description:
Location of the snapshot's manifest list file.
Used for format versions 1-3 and must be absent for format version 4, which uses `root-manifest` instead.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is not an accurate statement since v4 tables can reference older snapshots.

Comment thread open-api/rest-catalog-open-api.yaml Outdated
type: string
description:
Location of the snapshot's root manifest.
Required for format version 4 and must be absent for format versions 1-3.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same here, statement is inaccurate since v4 tables can still reference older manifests.

Comment thread open-api/rest-catalog-open-api.yaml Outdated
type: string
description:
The table's base location.
Required through format version 3, where it may be a path without a URI scheme; readers prepend a scheme for consistency with v4 absolute paths.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I believe even v3 required a scheme, please double check with table spec, so we aren't duplicating (and possibly making unnecessary statements).

durovo and others added 3 commits November 17, 2026 23:58
The v1 loadTable endpoint now rejects format version 4 tables with a 406 UnsupportedTableFormatVersionException, matching how the spec uses 406 for operations the server will not perform. Its 400 response is the shared BadRequestErrorResponse again.

Snapshot now describes root-manifest as used by format versions 4 and higher, and the CommitTableResponse metadata-location description is shortened.

Co-authored-by: Isaac <no-reply@databricks.com>
…ent rule

Replaces the rule that clients must not require metadata-location or use it to bypass the catalog with the upstream guidance that, when present, clients can compare metadata locations to detect changes.

Co-authored-by: Isaac <no-reply@databricks.com>
- v1 loadTable rejects format version 4 tables with a 422 UnsupportedTableFormatVersionException, and clients must use the /v2 endpoint. The example error names the cause and the remedy.
- TableLocation describes only the value. The presence and precedence rules stay in LoadTableResult, and CommitTableResponse and UnregisterTableResult refer to them.
- TableMetadata.location keeps the upstream definition instead of repeating the table spec.
- loadTableV2 has its own description instead of claiming to behave exactly like v1.
- The snapshots default is stated on each load endpoint, and the shared If-None-Match description no longer lists response types.
- Remove the catalog-managed wording.

Co-authored-by: Isaac <no-reply@databricks.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants