Skip to content

REST: Allow disabling remote signing with s3.remote-signing-enabled - #4134

Open
Ankitraj-builds wants to merge 2 commits into
apache:mainfrom
Ankitraj-builds:fix/rest-allow-disabling-remote-signing
Open

Ankitraj-builds wants to merge 2 commits into
apache:mainfrom
Ankitraj-builds:fix/rest-allow-disabling-remote-signing

Conversation

@Ankitraj-builds

@Ankitraj-builds Ankitraj-builds commented Oct 21, 2026 •

Copy link
Copy Markdown
Contributor

Closes #3888

Rationale for this change

REST catalogs like Lakekeeper return remote signing settings (s3.signer, s3.signer.uri) in the table response config. Since #3956 the FileIO property order is table metadata < catalog config < server config < vended credentials, so the server's signer always wins. A client that already has its own S3 credentials cannot turn remote signing off, and every S3 request pays an extra round trip to the signer.

This adds a catalog property, s3.remote-signing-enabled. When the catalog config sets it to false, RestCatalog._load_file_io() removes s3.signer from the merged FileIO properties, so remote signing is off even if the server config sets a signer. When it is not set or true, nothing changes. This covers Table, StagedTable and scan planning FileIOs.

Based on review feedback, this replaces the first version, which used an empty s3.signer as the opt-out.

Are these changes tested?

Yes, in tests/catalog/test_rest.py:

  • test_load_table_remote_signing_disabled_in_catalog_config: the server returns s3.signer, the catalog sets s3.remote-signing-enabled=false, and the table FileIO has no s3.signer. This test fails on main.
  • test_load_table_server_signer_kept_unless_disabled: with the property not set, or set to true, the server signer is kept.

All 177 tests in tests/catalog/test_rest.py pass locally, and lint passes.

Are there any user-facing changes?

Yes. New REST catalog property s3.remote-signing-enabled (default true), documented in configuration.md. Nothing changes for users who do not set it.

Comment on lines +3487 to +3499
metadata_location = "s3://warehouse/database/table/metadata/00001.metadata.json"
rest_mock.get(
f"{TEST_URI}v1/namespaces/fokko/tables/table",
json={
"metadata-location": metadata_location,
"metadata": example_table_metadata_with_snapshot_v1,
"config": {"s3.signer": "S3V4RestSigner", "s3.signer.uri": "http://signer"},
},
status_code=200,
request_headers=TEST_HEADERS,
)
catalog = RestCatalog("rest", uri=TEST_URI, token=TEST_TOKEN, **{"s3.signer": ""})
table = catalog.load_table(("fokko", "table"))

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This seems akward to me. The catalog should pass valid information. @c-thiel what is Lakekeeper doing here?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @Fokko. This was on the first version, which used an empty s3.signer. That is replaced now: users set s3.remote-signing-enabled=false instead (rambleraptor suggested this), so an empty value has no special meaning anymore.

The use case from #3888: some users already have their own S3 credentials, but the REST catalog sends s3.signer to every client. They then can’t turn signing off, and every S3 request makes an extra trip to the signer. Java uses the same s3.remote-signing-enabled property, and it’s also listed in #4140.

Happy to change the approach if you’d prefer something else.

Comment thread pyiceberg/catalog/rest/__init__.py Resolved

@rambleraptor rambleraptor left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've got a quick question.

@Ankitraj-builds Ankitraj-builds changed the title REST: Allow disabling remote signing with an empty s3.signer REST: Allow disabling remote signing with s3.remote-signing-enabled Oct 22, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Allow disabling remote signing in REST Catalog

3 participants