Repository navigation
Core: ensure snapshot timestamp is monotonically increasing for V4 tables #16293
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from all commits
29970b9
9d3744e
cead2ea
c454906
c12134e
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -35,6 +35,7 @@ | |
| import com.github.benmanes.caffeine.cache.LoadingCache; | ||
| import java.io.IOException; | ||
| import java.math.RoundingMode; | ||
| import java.time.Clock; | ||
| import java.util.Arrays; | ||
| import java.util.Collection; | ||
| import java.util.List; | ||
|
|
@@ -119,6 +120,7 @@ public void accept(String file) { | |
| private SnapshotAncestryValidator snapshotAncestryValidator = | ||
| SnapshotAncestryValidator.NON_VALIDATING; | ||
|
|
||
| private Clock clock = Clock.systemUTC(); | ||
| private ExecutorService workerPool; | ||
| private ExecutorService writePool; | ||
| private int writePoolParallelism = ThreadPools.WORKER_THREAD_POOL_SIZE; | ||
|
|
@@ -358,7 +360,7 @@ public Snapshot apply() { | |
| sequenceNumber, | ||
| snapshotId(), | ||
| parentSnapshotId, | ||
| System.currentTimeMillis(), | ||
| snapshotTimestampMillis(parentSnapshot), | ||
| operation(), | ||
| summary(base), | ||
| base.currentSchemaId(), | ||
|
|
@@ -683,6 +685,20 @@ protected ManifestReader<DeleteFile> newDeleteManifestReader(ManifestFile manife | |
| return ManifestFiles.readDeleteManifest(manifest, ops.io(), ops.current().specsById()); | ||
| } | ||
|
|
||
| @VisibleForTesting | ||
| void setClock(Clock newClock) { | ||
| this.clock = newClock; | ||
| } | ||
|
|
||
| private long snapshotTimestampMillis(Snapshot parentSnapshot) { | ||
| long now = clock.millis(); | ||
| if (parentSnapshot != null) { | ||
| return Math.max(now, parentSnapshot.timestampMillis() + 1); | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Any reason we can't just produce the monotonic timestamp for older format versions as well? Sure, it's not a spec level guarantee for those and we shouldn't validate on read that it's monotonic for older format versions, but from an implementation perspective I think we can just do it going forward?
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Hmm do you think we should do this monotonic production for older formats? or just change the validation?
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I think it's both. I'm less opinionated on the monotinc production for older formats, it just seems to eliminate the portion of the if condition and is still spec compliant, and produces more "expected" timestamps etc. I think validaiton needs to be updated to make sure that any v4 snapshots are monotonic. So both I think, one makes sure that we produce reaosnable values in the Java implementation, and the other is a read-side validation (though since the comparison is done relative to the new v4 snapshot which we guarnateed that its monotonic, maybe it doesn't matter here either).
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
I am good with this. The reference implementation can change to monotonic timestamp for older format versions. we just don't validate it.
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Done in the latest commit: Java always stamps |
||
| } | ||
|
|
||
| return now; | ||
| } | ||
|
|
||
| protected long snapshotId() { | ||
| if (snapshotId == null) { | ||
| synchronized (this) { | ||
|
|
||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -59,6 +59,7 @@ | |
| static final int MIN_FORMAT_VERSION_ROW_LINEAGE = 3; | ||
| static final int MIN_FORMAT_VERSION_PARQUET_MANIFESTS = 4; | ||
| static final int MIN_FORMAT_VERSION_OPTIONAL_LOCATION = 4; | ||
| static final int MIN_FORMAT_VERSION_MONOTONIC_TIMESTAMPS = 4; | ||
| static final int INITIAL_SPEC_ID = 0; | ||
| static final int INITIAL_SORT_ORDER_ID = 1; | ||
| static final int INITIAL_SCHEMA_ID = 0; | ||
|
|
@@ -378,15 +379,6 @@ | |
| } | ||
| last = logEntry; | ||
| } | ||
|
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Dropped the These are different clocks: The metadata-log checks remain unchanged; both sides are wall-clock |
||
| if (last != null) { | ||
| Preconditions.checkArgument( | ||
| // commits can happen concurrently from different machines. | ||
| // A tolerance helps us avoid failure for small clock skew | ||
| lastUpdatedMillis - last.timestampMillis() >= -ONE_MINUTE, | ||
| "Invalid update timestamp %s: before last snapshot log entry at %s", | ||
| lastUpdatedMillis, | ||
| last.timestampMillis()); | ||
| } | ||
|
|
||
| MetadataLogEntry previous = null; | ||
| for (MetadataLogEntry metadataEntry : previousFiles) { | ||
|
|
@@ -1270,6 +1262,18 @@ | |
| snapshot.sequenceNumber(), | ||
| lastSequenceNumber); | ||
|
|
||
| if (formatVersion >= MIN_FORMAT_VERSION_MONOTONIC_TIMESTAMPS && snapshot.parentId() != null) { | ||
| Snapshot parent = snapshotsById.get(snapshot.parentId()); | ||
| if (parent != null) { | ||
| ValidationException.check( | ||
| snapshot.timestampMillis() > parent.timestampMillis(), | ||
| "Invalid snapshot timestamp %s: not after parent snapshot %s at %s", | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Sorry for the delayed review, maybe in a follow on we can include the snapshot ID in the error message, maybe something like
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. parent snapshot id was already included in the error msg. the conflicting committing/child snapshot id wasn't really useful, as it wasn't committed. we can't inspect it from table history. |
||
| snapshot.timestampMillis(), | ||
| snapshot.parentId(), | ||
| parent.timestampMillis()); | ||
| } | ||
| } | ||
|
|
||
| this.lastSequenceNumber = snapshot.sequenceNumber(); | ||
| snapshots.add(snapshot); | ||
| snapshotsById.put(snapshot.snapshotId(), snapshot); | ||
|
|
@@ -1842,11 +1846,11 @@ | |
| Set<Long> addedSnapshotIds = Sets.newHashSet(); | ||
| Set<Long> intermediateSnapshotIds = Sets.newHashSet(); | ||
| for (MetadataUpdate update : changes) { | ||
| if (update instanceof MetadataUpdate.AddSnapshot) { | ||
|
Check warning on line 1849 in core/src/main/java/org/apache/iceberg/TableMetadata.java
|
||
| // adds must always come before set current snapshot | ||
| MetadataUpdate.AddSnapshot addSnapshot = (MetadataUpdate.AddSnapshot) update; | ||
| addedSnapshotIds.add(addSnapshot.snapshot().snapshotId()); | ||
| } else if (update instanceof MetadataUpdate.SetSnapshotRef) { | ||
|
Check warning on line 1853 in core/src/main/java/org/apache/iceberg/TableMetadata.java
|
||
| MetadataUpdate.SetSnapshotRef setRef = (MetadataUpdate.SetSnapshotRef) update; | ||
| long snapshotId = setRef.snapshotId(); | ||
| if (addedSnapshotIds.contains(snapshotId) | ||
|
|
||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -27,9 +27,13 @@ | |
| import java.io.File; | ||
| import java.io.IOException; | ||
| import java.nio.file.Paths; | ||
| import java.time.Clock; | ||
| import java.time.Instant; | ||
| import java.time.ZoneOffset; | ||
| import java.util.List; | ||
| import java.util.concurrent.ExecutorService; | ||
| import java.util.concurrent.Executors; | ||
| import java.util.concurrent.TimeUnit; | ||
| import java.util.stream.Collectors; | ||
| import javax.annotation.Nonnull; | ||
| import org.apache.iceberg.exceptions.ValidationException; | ||
|
|
@@ -276,4 +280,127 @@ public void testWriteManifestsWithInvalidParallelismThrows() { | |
| executor.shutdownNow(); | ||
| } | ||
| } | ||
|
|
||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Maybe some transaction tests which produce multipel snapshots?
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Added |
||
| @TestTemplate | ||
| public void snapshotTimestampsAreMonotonicallyIncreasing() { | ||
| table.newFastAppend().appendFile(FILE_A).commit(); | ||
| Snapshot first = table.currentSnapshot(); | ||
|
|
||
| table.newFastAppend().appendFile(FILE_B).commit(); | ||
| Snapshot second = table.currentSnapshot(); | ||
| assertThat(second.timestampMillis()).isGreaterThan(first.timestampMillis()); | ||
|
|
||
| table.newFastAppend().appendFile(FILE_C).commit(); | ||
| Snapshot third = table.currentSnapshot(); | ||
| assertThat(third.timestampMillis()).isGreaterThan(second.timestampMillis()); | ||
| } | ||
|
|
||
| @TestTemplate | ||
| public void lamportClockFastForwardsDriftedClock() { | ||
| table.newFastAppend().appendFile(FILE_A).commit(); | ||
| long firstTs = table.currentSnapshot().timestampMillis(); | ||
|
|
||
| long driftedTime = firstTs - TimeUnit.MINUTES.toMillis(2); | ||
| Clock driftedClock = Clock.fixed(Instant.ofEpochMilli(driftedTime), ZoneOffset.UTC); | ||
| AppendFiles append = table.newFastAppend().appendFile(FILE_B); | ||
| ((SnapshotProducer<?>) append).setClock(driftedClock); | ||
| append.commit(); | ||
|
|
||
| assertThat(table.currentSnapshot().timestampMillis()).isGreaterThan(firstTs); | ||
| } | ||
|
|
||
| @TestTemplate | ||
| public void transactionCommitsProduceMonotonicTimestamps() { | ||
| table.newFastAppend().appendFile(FILE_A).commit(); | ||
| Snapshot base = table.currentSnapshot(); | ||
|
|
||
| Transaction txn = table.newTransaction(); | ||
| txn.newFastAppend().appendFile(FILE_B).commit(); | ||
| Snapshot first = txn.table().currentSnapshot(); | ||
| txn.newFastAppend().appendFile(FILE_C).commit(); | ||
| Snapshot second = txn.table().currentSnapshot(); | ||
| txn.commitTransaction(); | ||
|
|
||
| assertThat(first.parentId()).isEqualTo(base.snapshotId()); | ||
| assertThat(first.timestampMillis()).isGreaterThan(base.timestampMillis()); | ||
| assertThat(second.parentId()).isEqualTo(first.snapshotId()); | ||
| assertThat(second.timestampMillis()).isGreaterThan(first.timestampMillis()); | ||
| assertThat(table.currentSnapshot().snapshotId()).isEqualTo(second.snapshotId()); | ||
| } | ||
|
|
||
| @TestTemplate | ||
| public void transactionRetryFastForwardsTimestampsPastConflictingSnapshot() { | ||
| table.updateProperties().set(TableProperties.COMMIT_NUM_RETRIES, "1").commit(); | ||
|
|
||
| table.newFastAppend().appendFile(FILE_A).commit(); | ||
| long baseTs = table.currentSnapshot().timestampMillis(); | ||
| Transaction txn = table.newTransaction(); | ||
|
|
||
| long firstCommitTs = baseTs + 1; | ||
| Clock firstClock = Clock.fixed(Instant.ofEpochMilli(firstCommitTs), ZoneOffset.UTC); | ||
| AppendFiles firstAppend = txn.newFastAppend().appendFile(FILE_B); | ||
| ((SnapshotProducer<?>) firstAppend).setClock(firstClock); | ||
| firstAppend.commit(); | ||
|
|
||
| long secondCommitTs = firstCommitTs + 1; | ||
| Clock secondClock = Clock.fixed(Instant.ofEpochMilli(secondCommitTs), ZoneOffset.UTC); | ||
| AppendFiles secondAppend = txn.newFastAppend().appendFile(FILE_C); | ||
| ((SnapshotProducer<?>) secondAppend).setClock(secondClock); | ||
| secondAppend.commit(); | ||
|
|
||
| long conflictTs = baseTs + 1_000_000L; | ||
| AppendFiles conflictAppend = table.newFastAppend().appendFile(FILE_D); | ||
| ((SnapshotProducer<?>) conflictAppend) | ||
| .setClock(Clock.fixed(Instant.ofEpochMilli(conflictTs), ZoneOffset.UTC)); | ||
| conflictAppend.commit(); | ||
| Snapshot conflict = table.currentSnapshot(); | ||
| assertThat(conflict.timestampMillis()).isEqualTo(conflictTs); | ||
| assertThat(txn.table().currentSnapshot().timestampMillis()).isLessThan(conflictTs); | ||
|
|
||
| txn.commitTransaction(); | ||
|
|
||
| Snapshot retriedSecond = table.currentSnapshot(); | ||
| Snapshot retriedFirst = table.snapshot(retriedSecond.parentId()); | ||
| assertThat(retriedFirst.parentId()).isEqualTo(conflict.snapshotId()); | ||
| assertThat(retriedFirst.timestampMillis()).isGreaterThan(conflict.timestampMillis()); | ||
| assertThat(retriedSecond.timestampMillis()).isGreaterThan(retriedFirst.timestampMillis()); | ||
| } | ||
|
|
||
| @TestTemplate | ||
| public void monotonicityIsScopedToTargetBranch() { | ||
| table.newFastAppend().appendFile(FILE_A).commit(); | ||
| long mainParentTs = table.currentSnapshot().timestampMillis(); | ||
|
|
||
| // Create a branch pointing at the current main head. | ||
| String branchName = "test-branch"; | ||
| table.manageSnapshots().createBranch(branchName).commit(); | ||
|
|
||
| // Commit to the branch with a clock far in the future so the branch head's | ||
| // timestamp-ms is much higher than main's head timestamp-ms. | ||
| long branchFutureTs = mainParentTs + 1_000_000L; | ||
| Clock branchClock = Clock.fixed(Instant.ofEpochMilli(branchFutureTs), ZoneOffset.UTC); | ||
| AppendFiles branchAppend = table.newFastAppend().appendFile(FILE_B).toBranch(branchName); | ||
| ((SnapshotProducer<?>) branchAppend).setClock(branchClock); | ||
| branchAppend.commit(); | ||
| long branchTs = table.snapshot(branchName).timestampMillis(); | ||
| assertThat(branchTs) | ||
| .as("branch commit should adopt the simulated far-future wall-clock value") | ||
| .isEqualTo(branchFutureTs); | ||
|
|
||
| // Commit to main with a clock just slightly after mainParentTs but far below branchTs. | ||
| // The spec requires monotonicity "on the same branch", so main's new timestamp must be | ||
| // constrained only by main's previous head, not by the unrelated future timestamp on | ||
| // the other branch. | ||
| long mainNewTs = mainParentTs + 5; | ||
| Clock mainClock = Clock.fixed(Instant.ofEpochMilli(mainNewTs), ZoneOffset.UTC); | ||
| AppendFiles mainAppend = table.newFastAppend().appendFile(FILE_C); | ||
| ((SnapshotProducer<?>) mainAppend).setClock(mainClock); | ||
| mainAppend.commit(); | ||
|
|
||
| long actualMainTs = table.currentSnapshot().timestampMillis(); | ||
| assertThat(actualMainTs) | ||
| .as("Main's monotonicity must be relative to main's parent, not the branch head") | ||
| .isEqualTo(mainNewTs) | ||
| .isLessThan(branchTs); | ||
| } | ||
| } | ||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I feel a little weird that we have the algorithm here, but the validation in the commit still allows for backwards drift. I would feel a little bit better if we changed the enforcement validation there as well if the format version > 4.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Specifically this
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think I somewhat agreee with this. I think the enforcement validation should change but I think we should make sure that comparison is done with lastUpdatedMillis reflecting the actually assigned snapshotTimestampMillis. In essence if parent snapshot was committed with 12:00:00.000, and the current clock is running slow and is at 11:59:57 , and the new snapshot is set to 12:00:01 based on the current logic, we should make sure this comparison is actually done against 12:00:01
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@RussellSpitzer The quoted snippet compares
TableMetadata.lastUpdatedMillis(wall clock) to the snapshot log, not parent→child snapshottimestamp-ms. v4 now validates parent→child inaddSnapshot. I dropped the check you quoted rather than tightening it: snapshot-log entries for new snapshots carry Lamporttimestamp-msand are no longer comparable tolast-updated-ms.