Repository navigation
Conversation
d7d5df7 to
03b47f5
Compare
| private Long firstRowId = null; | ||
| private byte[] deletedPositions = null; | ||
| private byte[] replacedPositions = null; | ||
| private Long snapshotTimestampMicros = null; |
There was a problem hiding this comment.
Note that inherit(long, long) below doesn't fill snapshotTimestampMicros yet. That's intentional: this PR is limited to the schema, interface, and struct changes.
Read inheritance comes in the next stacked PR. There, inherit takes the manifest's snapshot timestamp as a third argument and fills it under the same condition as the data sequence number, and the root manifest writer persists the commit timestamp for added and modified entries.
There was a problem hiding this comment.
The spec in #18526 says that when writing any manifest, a new (ADDED) entry "must have a null snapshot_timestamp". It doesn't make an exception for the root manifest. But your plan above says the root manifest writer will write the commit time for new entries. For sequence numbers, the spec only asks for null in leaf manifests, so the root writer can fill them in. Should the timestamp rule allow the same, or should the root writer leave it null?
| 10_000L, // firstRowId | ||
| null, // deletedPositions | ||
| null, // replacedPositions | ||
| null), // snapshotTimestampMicros |
There was a problem hiding this comment.
null because timestamp inheritance on read isn't in this PR (the sequence number and first row ID here are inherited). The next stacked PR changes this to the manifest's timestamp. Same for the added entry in mdvFilter.
| dataFile( | ||
| new TrackingStruct( | ||
| EntryStatus.ADDED, SNAPSHOT_ID, 5L, 5L, null, 5_000L, null, null), | ||
| EntryStatus.ADDED, SNAPSHOT_ID, 5L, 5L, null, 5_000L, null, null, null), |
There was a problem hiding this comment.
null because timestamp inheritance on read isn't in this PR. The next stacked PR changes this to the manifest's timestamp.
| dataFileWithColumnFiles( | ||
| new TrackingStruct( | ||
| EntryStatus.MODIFIED, SNAPSHOT_ID, 5L, 3L, 34L, 5_000L, null, null), | ||
| EntryStatus.MODIFIED, SNAPSHOT_ID, 5L, 3L, 34L, 5_000L, null, null, null), |
There was a problem hiding this comment.
null because timestamp inheritance on read isn't in this PR. Like the data sequence number, the timestamp is reset when column files are updated, so the next stacked PR changes this to the manifest's timestamp.
| dataFile( | ||
| new TrackingStruct( | ||
| EntryStatus.ADDED, SNAPSHOT_ID, 5L, 5L, null, 10_000L, null, null), | ||
| EntryStatus.ADDED, SNAPSHOT_ID, 5L, 5L, null, 10_000L, null, null, null), |
There was a problem hiding this comment.
null because timestamp inheritance on read isn't in this PR. The next stacked PR changes these to the manifest's timestamp.
| dataFile( | ||
| new TrackingStruct( | ||
| EntryStatus.ADDED, SNAPSHOT_ID, 5L, 5L, 1234567L, 5_000L, null, null), | ||
| EntryStatus.ADDED, SNAPSHOT_ID, 5L, 5L, 1234567L, 5_000L, null, null, SNAPSHOT_TS), |
There was a problem hiding this comment.
Unrelated to this PR, but noticed while filling in timestamps: withModifiedSnapshotId here is an ADDED entry with a non-null modified snapshot ID. Since #18984, TrackingBuilder.dvUpdated() no longer sets a modified snapshot ID on added entries, so this fixture describes a state the builder can't produce. The test still passes because it only checks that the modified snapshot ID isn't inherited. A carried-forward EXISTING entry that was modified earlier would be a realistic replacement. I left it alone to keep this PR focused, and can follow up separately.
| dataFile( | ||
| new TrackingStruct(EntryStatus.ADDED, SNAPSHOT_ID, 5L, 5L, null, null, null, null), | ||
| new TrackingStruct( | ||
| EntryStatus.ADDED, SNAPSHOT_ID, 5L, 5L, null, null, null, null, SNAPSHOT_TS), |
There was a problem hiding this comment.
Note that withoutFirstRowId isn't a realistic production state: an added entry wouldn't have a explicit file sequence number and a snapshot timestamp without a first row ID. In practice the file sequence number, first row ID, and snapshot timestamp are either all assigned by the root writer at commit time or all inherited together on read. The fixture keeps the other fields explicit on purpose, so first row ID is the only value this test exercises.
| null, // no change bitmaps | ||
| null)); | ||
| null, | ||
| null)); // snapshot timestamp |
There was a problem hiding this comment.
null even though FILE_A has a timestamp: append starts fresh from TrackingBuilder.added, and the root writer doesn't write the commit timestamp in this PR. The next stacked PR changes this to COMMIT_SNAPSHOT_TS. Same for the other append tests in this file.
| return javaClass.cast(getInternal(pos)); | ||
| } | ||
|
|
||
| private Object get(int pos) { |
There was a problem hiding this comment.
The tracking wrapper is split into leaf and root getInternal implementations because the two write schemas no longer share a prefix: leaf tracking omits deleted_positions and replaced_positions, so the appended snapshot_timestamp is at position 6 in leaf manifests and 8 in root manifests.
The shared switch only worked before because the leaf tracking schema happened to be a prefix of the root schema. One switch per schema doesn't rely on that, which should be more robust as fields are added, and it follows the same pattern as LeafTrackedFileWrapper and RootTrackedFileWrapper.
| * <p>Returns null if the snapshot timestamp is unknown. This may happen for files in tables with | ||
| * format version below 4. | ||
| */ | ||
| default Long snapshotTimestampMicros() { |
There was a problem hiding this comment.
I feel like a better name for the content file API would be lastUpdatedTimestampMicros? Feels weird to reference snapshot at this abstraction
There was a problem hiding this comment.
the last updated timestamp is a row lineage concept tied to row id and last updated sequence number.
A data file is immutable. Hence I thought lastUpdatedTimestampMicros may not be accurate for ContentFile interface.
There was a problem hiding this comment.
I think its correct to do last updated since we do say it is "added or changed" in the javadoc? This is the lastUpdatedTimestampMicros of this particular file right?
There was a problem hiding this comment.
I was mainly thinking about along the line of "the timestamp of the snapshot that added or last modified the file".
But I can see your points now from the file perspective (like the last update time in filesystem). it is the time when the file was added to or last updated in the table. I can change it.
| } | ||
|
|
||
| /** Returns the microsecond timestamp of the snapshot that added the manifest file. */ | ||
| default Long snapshotTimestampMicros() { |
There was a problem hiding this comment.
should this one just be a long? For a manifest file wouldn't we always know this (regardless of inheritance)?
There was a problem hiding this comment.
For v4 tables (green or upgraded), their manifest files (committed before or after the upgrade) would always have it via inheritance.
for v1-v3 tables, their manifest files won't have this. Are you thinking about wiring in the snapshot timestamp inheritance for v1-v3 tables too? and don't expose the metadata column for v1-v3 tables?
There was a problem hiding this comment.
Ah yeah sorry I was thinking from a pure v4 mindset, for the older v1-v3 manifests you're right they won't have this and we can't derive those because the snapshots for when those were written may have been expired. We could fill in a sentinel value (unix epoch) but that seems worse and error prone to interpretation when later materialized (compared to just leveraging the snapshot post v4 upgrade). So I think I'd leave it as is then
03b47f5 to
b946034
Compare
| Types.NestedField.optional( | ||
| 8, | ||
| "snapshot_timestamp", | ||
| Types.TimestampType.withZone(), |
There was a problem hiding this comment.
yes, see the spec discussion here: #18526 (comment)
| this.modifiedSnapshotId = null; | ||
| this.deletedPositions = null; | ||
| this.replacedPositions = null; | ||
| this.snapshotTimestampMicros = null; |
There was a problem hiding this comment.
When a file is added with an older sequence number (appendAtSeq, used by compaction), it keeps the old sequence number, but its timestamp is null here, so it gets the new commit's time. The two values then disagree: the rows look old by sequence number but new by timestamp. Should these files keep an older timestamp too? #18526 requires null for all ADDED entries, so this would need a spec change as well.
There was a problem hiding this comment.
Rewrites must copy non-null _last_updated_sequence_number and _last_updated_timestamp forward in data files, so compacted rows don't inherit either value, and ADDED entries can stay null as #18526 specifies.
| private Long firstRowId = null; | ||
| private byte[] deletedPositions = null; | ||
| private byte[] replacedPositions = null; | ||
| private Long snapshotTimestampMicros = null; |
There was a problem hiding this comment.
The spec in #18526 says that when writing any manifest, a new (ADDED) entry "must have a null snapshot_timestamp". It doesn't make an exception for the root manifest. But your plan above says the root manifest writer will write the commit time for new entries. For sequence numbers, the spec only asks for null in leaf manifests, so the root writer can fill them in. Should the timestamp rule allow the same, or should the root writer leave it null?
…p metadata column V4 row lineage needs a last-updated timestamp that is inherited the same way as the last-updated sequence number. This adds the snapshot_timestamp tracking field to the root and leaf manifest schemas, the root manifest snapshot timestamp, the _last_updated_timestamp metadata column, and the ContentFile and ManifestFile accessors. Read-side inheritance and commit-side assignment follow in later changes. Generated-by: Cursor Co-authored-by: Cursor <cursoragent@cursor.com>
…mpMicros The file-level API describes when the file was added or last changed in the table, so avoid referencing snapshots at this abstraction. Generated-by: Cursor Co-authored-by: Cursor <cursoragent@cursor.com>
…ons test Generated-by: Cursor Co-authored-by: Cursor <cursoragent@cursor.com>
c0d5f07 to
f7db956
Compare
V4 row lineage needs a last-updated timestamp that is inherited the same way as the last-updated sequence number. This adds the snapshot_timestamp tracking field to the root and leaf manifest schemas, the root manifest snapshot timestamp, the _last_updated_timestamp metadata column, and the ContentFile and ManifestFile accessors.
This is the step 1 of the 6-step implementation plan as documented in the issue #17619.
Test plan:
snapshot_timestampat its own position in each schema (6 in leaf, 8 in root).TrackingBuilderkeeps the source timestamp for EXISTING, DELETED and REPLACED entries, and resets it to null on a column-file update.TrackingStruct, the file and manifest adapters, andRootManifestFileexpose the timestamp.RootManifestFileconverts the snapshot's milliseconds to microseconds._last_updated_timestampgets its own stats ID range.nullin this PR.Generated-by: Cursor