Skip to content

REST catalog: bound Idempotency-Key retries to the advertised idempotency-key-lifetime #4428

Open

Description

@HotSushi

Feature Request / Improvement

Follow-up to #4290 and #4310 cc: @YuangGao @Fokko

#4310 sends Idempotency-Key on mutations and retries keyed POSTs under rest.client.*, but it doesn't enforce the lifetime the server advertises in GET /v1/config. The PR describes this as intentional because urllib3 retries are bounded. They aren't bounded by the key lifetime, though, and a retry outside that window can apply a mutation twice.

The REST catalog spec says:

Clients SHOULD NOT reuse an Idempotency-Key after this window elapses.

Java enforces this in ExponentialHttpRequestRetryStrategy (apache/iceberg#18070, building on apache/iceberg#17947).

Why it matters

An overloaded server responds with 503 and Retry-After. urllib3 honors Retry-After by default (respect_retry_after_header=True) and sleeps for the full value. backoff_max caps backoff only, not Retry-After.

Example with idempotency-key-lifetime: PT5M:

  1. The client sends a commit. The server applies it, but the response is lost (timeout, proxy 502, dropped connection).
  2. The retry gets 503 with Retry-After: 600.
  3. urllib3 sleeps 600s and retries with the same key.
  4. The server has evicted the key, treats the request as new, and applies the mutation again.

The result is a duplicate commit and corrupted table state, which is exactly what the key is supposed to prevent.

Backoff alone can also exceed short windows. For example, max-retries=5, backoff-factor=10 and a 30s socket timeout add up to about 450s of retrying.

Current behavior

  • In pyiceberg/catalog/rest/__init__.py, ConfigResponse.idempotency_key_lifetime is read as a string and only checked for presence.
  • _create_connection_adapter(..., retry_post=True) adds POST to allowed_methods. Nothing parses the lifetime or compares it with elapsed time.
  • PT0S or a negative duration still turns on keyed retries. In Java, a non-positive lifetime means keyed requests are never retried.
  • A malformed value is accepted silently.

Requested behavior

  1. Parse idempotency-key-lifetime as an ISO-8601 duration, e.g. with pydantic.TypeAdapter(timedelta).
    • Malformed: log a warning and disable the bound. The key is still sent, matching Java's HTTPClient.parseKeyLifetime.
    • Zero or negative: never retry keyed requests.
  2. Record when the first attempt of a keyed request started.
  3. Before each retry of a keyed request, decline the retry if elapsed + next_sleep >= lifetime. next_sleep is the Retry-After value when present, otherwise the backoff time including jitter. When a retry is declined, return the last response so _handle_non_200_response raises the typed error, as it does today when retries run out.
  4. Leave non-keyed requests and idempotent methods (GET, HEAD, PUT, DELETE) without a key as they are.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions