Skip to content

fix: keep last_updated_ms from preceding the latest metadata-log entry - #4273

Merged
Fokko merged 1 commit into
apache:mainfrom
YuangGao:fix/last-updated-ms-not-before-metadata-log
Nov 13, 2026
Merged

Fokko merged 1 commit into
apache:mainfrom
YuangGao:fix/last-updated-ms-not-before-metadata-log

Conversation

@YuangGao

@YuangGao YuangGao commented Nov 3, 2026

Copy link
Copy Markdown
Contributor

Closes #4189

Rationale for this change

  • A staged snapshot sets last_updated_ms to its own timestamp. If another commit lands before the transaction commits, the new metadata ends up older than its latest metadata-log entry, and Java rejects it once the gap exceeds a minute.
  • update_table_metadata now bumps last_updated_ms to the current time whenever it doesn't come after the base metadata's, not only when they're equal.

Are these changes tested?

  • test_init.py — a snapshot older than the base metadata no longer sets last_updated_ms before the latest metadata-log entry.
  • test_update_metadata_with_multiple_updates used a snapshot timestamp older than its base metadata, so it now uses a later one.

Are there any user-facing changes?

  • No.

@Fokko Fokko left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @YuangGao! I reproduced the bug from #4189 on main (last_updated_ms lands 61s before the latest metadata-log entry, exactly what Java's TableMetadata parser rejects) and confirmed this PR fixes it. I also compared against the Java implementation, see the inline comment.

if metadata_location:
new_metadata = _update_table_metadata_log(new_metadata, metadata_location, base_metadata.last_updated_ms)
if base_metadata.last_updated_ms == new_metadata.last_updated_ms:
if new_metadata.last_updated_ms <= base_metadata.last_updated_ms:

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I compared this against Java's TableMetadata.Builder (core/src/main/java/org/apache/iceberg/TableMetadata.java). Java never inherits the snapshot timestamp into last-updated-ms at all: addSnapshot() does not touch it, and build() stamps System.currentTimeMillis() (lines 1559-1561). PyIceberg inherits the snapshot timestamp in AddSnapshotUpdate because snapshots are staged eagerly (in Java the snapshot is produced at commit time), which is the root cause of #4189. This change restores the invariant Java enforces at lines 394-403 (last-updated-ms within one minute after the latest metadata-log entry), so from Java's perspective the output is accepted again.

Comment thread tests/table/test_init.py
new_metadata = update_table_metadata(
table_v2.metadata, (AddSnapshotUpdate(snapshot=stale_snapshot),), False, table_v2.metadata_location
)
assert new_metadata.last_updated_ms > new_metadata.metadata_log[-1].timestamp_ms

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: nice regression test. The issue's repro also covers the SetSnapshotRefUpdate path (the snapshot log gets stamped with the stale snapshot timestamp). I probed that separately: Java tolerates it (snapshot-log entries are validated with the same one-minute tolerance, and the stamping actually matches Java's timeOfChange behavior for added snapshots), so this is fine as is. Just mentioning in case you want a variant of this test with a SetSnapshotRefUpdate to main for completeness.

@Fokko
Fokko added this pull request to the merge queue Nov 13, 2026
Merged via the queue into apache:main with commit 9eb8395 Nov 13, 2026
20 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

A long transaction committed past a concurrent tag writes metadata Java rejects

2 participants