Skip to content

Core, API: add snapshot timestamp tracking and _last_updated_timestamp metadata column - #19019

Open
stevenzwu wants to merge 3 commits into
apache:mainfrom
stevenzwu:last-updated-timestamp-fields
Open

stevenzwu wants to merge 3 commits into
apache:mainfrom
stevenzwu:last-updated-timestamp-fields

Conversation

@stevenzwu

@stevenzwu stevenzwu commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

V4 row lineage needs a last-updated timestamp that is inherited the same way as the last-updated sequence number. This adds the snapshot_timestamp tracking field to the root and leaf manifest schemas, the root manifest snapshot timestamp, the _last_updated_timestamp metadata column, and the ContentFile and ManifestFile accessors.

This is the step 1 of the 6-step implementation plan as documented in the issue #17619.

Test plan:

  • Leaf and root manifests write and read back snapshot_timestamp at its own position in each schema (6 in leaf, 8 in root).
  • TrackingBuilder keeps the source timestamp for EXISTING, DELETED and REPLACED entries, and resets it to null on a column-file update.
  • TrackingStruct, the file and manifest adapters, and RootManifestFile expose the timestamp. RootManifestFile converts the snapshot's milliseconds to microseconds.
  • _last_updated_timestamp gets its own stats ID range.
  • Out of scope: inheritance is in step 2, so timestamps that would be inherited read back as null in this PR.

Generated-by: Cursor

@stevenzwu
stevenzwu force-pushed the last-updated-timestamp-fields branch 2 times, most recently from d7d5df7 to 03b47f5 Compare November 18, 2026 06:29
private Long firstRowId = null;
private byte[] deletedPositions = null;
private byte[] replacedPositions = null;
private Long snapshotTimestampMicros = null;

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note that inherit(long, long) below doesn't fill snapshotTimestampMicros yet. That's intentional: this PR is limited to the schema, interface, and struct changes.

Read inheritance comes in the next stacked PR. There, inherit takes the manifest's snapshot timestamp as a third argument and fills it under the same condition as the data sequence number, and the root manifest writer persists the commit timestamp for added and modified entries.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The spec in #18526 says that when writing any manifest, a new (ADDED) entry "must have a null snapshot_timestamp". It doesn't make an exception for the root manifest. But your plan above says the root manifest writer will write the commit time for new entries. For sequence numbers, the spec only asks for null in leaf manifests, so the root writer can fill them in. Should the timestamp rule allow the same, or should the root writer leave it null?

10_000L, // firstRowId
null, // deletedPositions
null, // replacedPositions
null), // snapshotTimestampMicros

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

null because timestamp inheritance on read isn't in this PR (the sequence number and first row ID here are inherited). The next stacked PR changes this to the manifest's timestamp. Same for the added entry in mdvFilter.

dataFile(
new TrackingStruct(
EntryStatus.ADDED, SNAPSHOT_ID, 5L, 5L, null, 5_000L, null, null),
EntryStatus.ADDED, SNAPSHOT_ID, 5L, 5L, null, 5_000L, null, null, null),

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

null because timestamp inheritance on read isn't in this PR. The next stacked PR changes this to the manifest's timestamp.

dataFileWithColumnFiles(
new TrackingStruct(
EntryStatus.MODIFIED, SNAPSHOT_ID, 5L, 3L, 34L, 5_000L, null, null),
EntryStatus.MODIFIED, SNAPSHOT_ID, 5L, 3L, 34L, 5_000L, null, null, null),

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

null because timestamp inheritance on read isn't in this PR. Like the data sequence number, the timestamp is reset when column files are updated, so the next stacked PR changes this to the manifest's timestamp.

dataFile(
new TrackingStruct(
EntryStatus.ADDED, SNAPSHOT_ID, 5L, 5L, null, 10_000L, null, null),
EntryStatus.ADDED, SNAPSHOT_ID, 5L, 5L, null, 10_000L, null, null, null),

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

null because timestamp inheritance on read isn't in this PR. The next stacked PR changes these to the manifest's timestamp.

dataFile(
new TrackingStruct(
EntryStatus.ADDED, SNAPSHOT_ID, 5L, 5L, 1234567L, 5_000L, null, null),
EntryStatus.ADDED, SNAPSHOT_ID, 5L, 5L, 1234567L, 5_000L, null, null, SNAPSHOT_TS),

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unrelated to this PR, but noticed while filling in timestamps: withModifiedSnapshotId here is an ADDED entry with a non-null modified snapshot ID. Since #18984, TrackingBuilder.dvUpdated() no longer sets a modified snapshot ID on added entries, so this fixture describes a state the builder can't produce. The test still passes because it only checks that the modified snapshot ID isn't inherited. A carried-forward EXISTING entry that was modified earlier would be a realistic replacement. I left it alone to keep this PR focused, and can follow up separately.

dataFile(
new TrackingStruct(EntryStatus.ADDED, SNAPSHOT_ID, 5L, 5L, null, null, null, null),
new TrackingStruct(
EntryStatus.ADDED, SNAPSHOT_ID, 5L, 5L, null, null, null, null, SNAPSHOT_TS),

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note that withoutFirstRowId isn't a realistic production state: an added entry wouldn't have a explicit file sequence number and a snapshot timestamp without a first row ID. In practice the file sequence number, first row ID, and snapshot timestamp are either all assigned by the root writer at commit time or all inherited together on read. The fixture keeps the other fields explicit on purpose, so first row ID is the only value this test exercises.

null, // no change bitmaps
null));
null,
null)); // snapshot timestamp

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

null even though FILE_A has a timestamp: append starts fresh from TrackingBuilder.added, and the root writer doesn't write the commit timestamp in this PR. The next stacked PR changes this to COMMIT_SNAPSHOT_TS. Same for the other append tests in this file.

return javaClass.cast(getInternal(pos));
}

private Object get(int pos) {

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The tracking wrapper is split into leaf and root getInternal implementations because the two write schemas no longer share a prefix: leaf tracking omits deleted_positions and replaced_positions, so the appended snapshot_timestamp is at position 6 in leaf manifests and 8 in root manifests.

The shared switch only worked before because the leaf tracking schema happened to be a prefix of the root schema. One switch per schema doesn't rely on that, which should be more robust as fields are added, and it follows the same pattern as LeafTrackedFileWrapper and RootTrackedFileWrapper.

* <p>Returns null if the snapshot timestamp is unknown. This may happen for files in tables with
* format version below 4.
*/
default Long snapshotTimestampMicros() {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I feel like a better name for the content file API would be lastUpdatedTimestampMicros? Feels weird to reference snapshot at this abstraction

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the last updated timestamp is a row lineage concept tied to row id and last updated sequence number.

A data file is immutable. Hence I thought lastUpdatedTimestampMicros may not be accurate for ContentFile interface.

@RussellSpitzer RussellSpitzer Nov 19, 2026 •

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think its correct to do last updated since we do say it is "added or changed" in the javadoc? This is the lastUpdatedTimestampMicros of this particular file right?

@stevenzwu stevenzwu Nov 19, 2026 •

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was mainly thinking about along the line of "the timestamp of the snapshot that added or last modified the file".

But I can see your points now from the file perspective (like the last update time in filesystem). it is the time when the file was added to or last updated in the table. I can change it.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

renamed

}

/** Returns the microsecond timestamp of the snapshot that added the manifest file. */
default Long snapshotTimestampMicros() {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should this one just be a long? For a manifest file wouldn't we always know this (regardless of inheritance)?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For v4 tables (green or upgraded), their manifest files (committed before or after the upgrade) would always have it via inheritance.

for v1-v3 tables, their manifest files won't have this. Are you thinking about wiring in the snapshot timestamp inheritance for v1-v3 tables too? and don't expose the metadata column for v1-v3 tables?

@amogh-jahagirdar amogh-jahagirdar Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah yeah sorry I was thinking from a pure v4 mindset, for the older v1-v3 manifests you're right they won't have this and we can't derive those because the snapshots for when those were written may have been expired. We could fill in a sentinel value (unix epoch) but that seems worse and error prone to interpretation when later materialized (compared to just leveraging the snapshot post v4 upgrade). So I think I'd leave it as is then

Types.NestedField.optional(
8,
"snapshot_timestamp",
Types.TimestampType.withZone(),

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

withZone?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes, see the spec discussion here: #18526 (comment)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

k 🤷

this.modifiedSnapshotId = null;
this.deletedPositions = null;
this.replacedPositions = null;
this.snapshotTimestampMicros = null;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When a file is added with an older sequence number (appendAtSeq, used by compaction), it keeps the old sequence number, but its timestamp is null here, so it gets the new commit's time. The two values then disagree: the rows look old by sequence number but new by timestamp. Should these files keep an older timestamp too? #18526 requires null for all ADDED entries, so this would need a spec change as well.

@stevenzwu stevenzwu Nov 21, 2026 •

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Rewrites must copy non-null _last_updated_sequence_number and _last_updated_timestamp forward in data files, so compacted rows don't inherit either value, and ADDED entries can stay null as #18526 specifies.

private Long firstRowId = null;
private byte[] deletedPositions = null;
private byte[] replacedPositions = null;
private Long snapshotTimestampMicros = null;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The spec in #18526 says that when writing any manifest, a new (ADDED) entry "must have a null snapshot_timestamp". It doesn't make an exception for the root manifest. But your plan above says the root manifest writer will write the commit time for new entries. For sequence numbers, the spec only asks for null in leaf manifests, so the root writer can fill them in. Should the timestamp rule allow the same, or should the root writer leave it null?

stevenzwu and others added 3 commits November 20, 2026 20:53
…p metadata column

V4 row lineage needs a last-updated timestamp that is inherited the same way as the last-updated sequence number. This adds the snapshot_timestamp tracking field to the root and leaf manifest schemas, the root manifest snapshot timestamp, the _last_updated_timestamp metadata column, and the ContentFile and ManifestFile accessors. Read-side inheritance and commit-side assignment follow in later changes.

Generated-by: Cursor
Co-authored-by: Cursor <cursoragent@cursor.com>
…mpMicros

The file-level API describes when the file was added or last changed in
the table, so avoid referencing snapshots at this abstraction.

Generated-by: Cursor
Co-authored-by: Cursor <cursoragent@cursor.com>
…ons test

Generated-by: Cursor
Co-authored-by: Cursor <cursoragent@cursor.com>
@stevenzwu
stevenzwu force-pushed the last-updated-timestamp-fields branch from c0d5f07 to f7db956 Compare November 21, 2026 04:55

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants