Skip to content

Core: Implement minimal v4 writes in MergingSnapshotProducer - #19030

Merged
rdblue merged 14 commits into
apache:mainfrom
rdblue:v4-snapshot-writes
Nov 20, 2026
Merged

rdblue merged 14 commits into
apache:mainfrom
rdblue:v4-snapshot-writes

Conversation

@rdblue

@rdblue rdblue commented Nov 19, 2026 •

Copy link
Copy Markdown
Contributor

This is a minimal set of changes to get v4 writes working with SnapshotProducer and MergingSnapshotProducer. Root manifests are managed and written by RootManifestProducer, which is roughly the v4 equivalent of MergingSnapshotProducer. This implements simple appends that write new data files to the root file. This does not write leaf files.

SnapshotProducer is refactored to have a method for writing a manifest list (writeManifestList) and one for writing a root metadata file (writeRootManifest). Nearly everything else in SnapshotProducer is reused, although the state used for file cleanup (tested in failure and recovery tests).

MergingSnapshotProducer is updated to internally use RootManifestProducer for v4 writes. Initially, I thought that it would be easier to replace MergingSnapshotProducer with a v4 version, but in the end that strategy would require a lot of churn with little benefit, so I decided to add a delegate within the class instead. We should be able to follow up and build a similar abstraction around the v3 behavior as well so that MergingSnapshotProducer selects its implementation instead of doing work directly.

Test plan:

  • Port relevant tests from TestMergeAppend
    • This required adding a version of TestBase for adaptive metadata
  • Test metadata file cleanup for failure and retry cases (does not include leaf manifests that are not yet supported)
  • Test that appended manifests, deleted data files, deleted delete files, and new delete files are rejected
  • Test concurrent append
  • Test v3 to v4 upgrade succeeds and preserves v3 manifests

Note that there are many test cases that are not ported because the supported appends are very simple. The plan is to get a basic write working and then expand functionality and tests at the same time.

@github-actions github-actions Bot added the core label Nov 19, 2026
@rdblue
rdblue marked this pull request as draft November 19, 2026 01:18
@rdblue
rdblue force-pushed the v4-snapshot-writes branch from 7be2943 to 2519371 Compare November 19, 2026 23:58
@github-actions github-actions Bot added the API label Nov 19, 2026
@rdblue
rdblue force-pushed the v4-snapshot-writes branch from 2519371 to a249e69 Compare November 19, 2026 23:59
@rdblue
rdblue force-pushed the v4-snapshot-writes branch from a249e69 to 5247faf Compare November 20, 2026 00:08
@rdblue
rdblue marked this pull request as ready for review November 20, 2026 00:09
DeleteFile[] deletes = NO_DELETES;
if (dataFile.deletionVector() != null) {
TrackedFile tracked = TrackedFileAdapters.asTrackedFile(dataFile);
TrackedFile tracked = TrackedFileAdapters.asTrackedFileUnchecked(dataFile);

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I added asTrackedFile(DataFile, Schema) to the adapters and also renamed this because it doesn't check that the DataFile is a wrapped TrackedFile.

*/
protected void validateAddedDataFiles(
TableMetadata base, Long startingSnapshotId, PartitionSet partitionSet, Snapshot parent) {
Long parentId = parent != null ? parent.snapshotId() : null;

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Validations for v4 metadata are being added in parallel (see #18699). Until those are ready, I've updated the validations to fail if there are any commits after the starting snapshot ID. The checks run as before for v3 and earlier.

* @param snapshot snapshot to apply the changes to; will be the new snapshot's parent
* @return a manifest list for the new snapshot.
*/
protected abstract List<ManifestFile> apply(TableMetadata metadataToUpdate, Snapshot snapshot);

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This method is still used for all v1-v3 tables and writeRootManifest is used for v4 and later.

}

List<ManifestFile> manifests = apply(base, parentSnapshot);
protected NewSnapshotMetadata writeManifestList(Snapshot parent, long sequenceNumber) {

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This method was factored out of apply() and is responsible for writing a v1-v3 manifest list, then returning the metadata for it to build a new snapshot.

This is directly equivalent to the writeRootManifest method that is used for v4, but that one has to be implemented by MergingSnapshotProducer.

if (!saved.manifestListLocation().equals(manifestList)) {
deleteFile(manifestList);
// also clean up unused root files created by multiple attempts
for (String rootLocation : rootToManifestLocations.keySet()) {

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Besides changes to apply, the only other change is to handle metadata file cleanup. This is done the same way, except that manifestsByManifestList is now used for all versions and renamed to rootToManifestLocations. The root location (manifest list or root manifest) points to a set of manifest locations that it contains.

The root file that gets committed is used to fetch the manifest paths it contains and those are passed to cleanUncommitted.

}
}

private static Set<String> readManifestLocations(

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In some cases, cleanup doesn't have an entry in rootToManifestLocations and the root must be read to recover the paths. This implements that read in place of snapshot.allManifests(io).

* @param io a {@link FileIO}
* @return a list of manifest locations tracked by the root manifest
*/
static List<String> readManifestLocations(ManifestFile root, FileIO io) {

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added to produce manifest paths from a root manifest for SnapshotProducer.readManifestLocations.

import org.junit.jupiter.api.BeforeEach;
import org.junit.jupiter.api.io.TempDir;

public class AdaptiveMetadataTestBase {

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is equivalent to TestBase, but is for v4 (adaptive metadata tree) testes. This is going to grow as more write cases are implemented and tests are converted from v3 TestBase.

import org.junit.jupiter.params.ParameterizedTest;
import org.junit.jupiter.params.provider.FieldSource;

public class TestAppend extends AdaptiveMetadataTestBase {

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These are tests ported from TestMergeAppend. I removed tests that did not apply:

  • Tests for manifest merge behavior and configuration
  • Tests that multiple partition specs produce multiple manifests
  • Tests of threadpool usage
  • Tests for appendManifest (not yet supported)
  • Tests for appends that produce multiple manifests (not yet supported)
  • Tests for manifest counters (not yet supported)

Tests that were ported over:

  • Append cases
  • Multiple partition specs in a write
  • Failure to commit from repeated CommitFailedException (and cleanup)
  • Recovery using retries after CommitFailedException (and cleanup)

@rdblue rdblue changed the title Core: Implement v4 writes through MergingSnapshotProducer Core: Implement minimal v4 writes in MergingSnapshotProducer Nov 20, 2026
@danielcweeks
danielcweeks self-requested a review November 20, 2026 00:55
throw new UnsupportedOperationException("Cannot commit deleted delete files to v4");
}

if (rootManifestProducer == null || hasNewDataFiles) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think we're actually getting reuse here because we rebuild on retry.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think all we need to do to address this is set hasNewDataFiles = false after adding all the files

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There is not much reuse right now because the root producer doesn't create leaf manifests yet. Right now, this just reuses the commit UUID that is generated.

Comment on lines +103 to +109
validateFiles(
rootFiles,
dataSeqs(1L, 1L),
fileSeqs(1L, 1L),
ids(snapshotId, snapshotId),
locations(FILE_A, fileNewSpec),
statuses(EntryStatus.ADDED, EntryStatus.ADDED));

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This doesn't actually assert what the name indicates because we don't check which spec was written to.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There are writer tests for mixed partition specs that validate partitions are correctly stored and read. These tests are for the manifest layout that results from mixed partition specs. So the assertion here is that both files were added to a single root manifest, rather than two separate leaf manifests for different partition specs.

assertThat(latest.rootManifestLocation())
.as("Pending root manifest should be rewritten")
.isNotEqualTo(pendingRootLocation);
assertThat(defaultTable.io().newInputFile(pendingRootLocation).exists()).isFalse();

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: it would be nice to add an as() context here since this will just result in something like "expected false, but got true".

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Now that this no longer uses LocalFileIO, it can list metadata files and make assertions that are more reliable than this. I updated it to:

    assertThat(listRootManifests(defaultTable))
        .containsExactlyInAnyOrder(latest.rootManifestLocation(), snap.rootManifestLocation());

*
* @param metadataToUpdate the base table metadata to apply changes to
* @param snapshot snapshot to apply the changes to
* @param snapshot snapshot to apply the changes to; will be the new snapshot's parent

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is the comment change neccessary?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I updated this for consistency with the new method, which makes this statement. It's not strictly required if you think it should be removed.

throw new UnsupportedOperationException("Cannot commit deleted delete files to v4");
}

if (rootManifestProducer == null || hasNewDataFiles) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think all we need to do to address this is set hasNewDataFiles = false after adding all the files

@github-actions github-actions Bot removed the API label Nov 20, 2026
import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
import org.apache.iceberg.relocated.com.google.common.collect.Maps;

public class InMemoryTables {

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is equivalent to TestBase, but doesn't rely on LocalFileIO because it produces paths that do not have a scheme. This will be used for all v4 (and later) test suites.

MetadataUpdate.AddSnapshot addSnapshot =
(MetadataUpdate.AddSnapshot) body.updates().get(0);
assertThatThrownBy(
() -> table.io().newInputFile(addSnapshot.snapshot().manifestListLocation()))

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This relied on incorrect behavior in InMemoryFileIO where you could not create an InputFile unless the file existed. The correct behavior is to check InputFile#exists().

@Override
public long sequenceNumber() {
return file.tracking().dataSequenceNumber();
return file.tracking() != null ? file.tracking().dataSequenceNumber() : -1;

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This was updated to match the current behavior of ManifestFile before it is committed. Otherwise the appendManifest tests fail because tracking() throws a NullPointerException.

closed = true;
}

public void clear() {

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Used to clear files between test cases so that listing metadata files doesn't find metadata from other cases.

@danielcweeks
danielcweeks self-requested a review November 20, 2026 21:15

@danielcweeks danielcweeks left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Some minor comments, but I don't think any major blockers.

Comment on lines 82 to +84
public InputFile newInputFile(String location) {
Preconditions.checkState(!closed, "Cannot call newInputFile after calling close()");
byte[] contents = IN_MEMORY_FILES.get(location);
if (null == contents) {
throw new NotFoundException("No in-memory file found for location: %s", location);
}
return new InMemoryInputFile(location, contents);
return new InMemoryInputFile(location, this);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It looks like there might be a subtle behavior change here, but I believe we only use the InMemoryFileIO for testing purposes, so it's probably ok?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I mentioned this in a different comment, but the previous behavior violated the FileIO contract so this is a bug fix to unblock the tests. And yes, we use this for testing (and didn't have many tests depending on this behavior) so I think it's okay.

Comment on lines +2635 to +2640
assertThat(
table
.io()
.newInputFile(addSnapshot.snapshot().manifestListLocation())
.exists())
.isFalse();

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: .as("manifest list should be cleaned up: %s", location) would be nice here

@rdblue
rdblue merged commit 76e9ed1 into apache:main Nov 20, 2026
40 checks passed
@rdblue

rdblue commented Nov 20, 2026

Copy link
Copy Markdown
Contributor Author

Merged. Thanks, @danielcweeks and @amogh-jahagirdar for reviewing!


protected void cleanAll() {
for (String manifestList : manifestsByManifestList.keySet()) {
for (String manifestList : rootToManifestLocations.keySet()) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: manifestList -> rootLocation/rootFile

* @param io a {@link FileIO}
* @return a list of manifest locations tracked by the root manifest
*/
static List<String> readManifestLocations(ManifestFile root, FileIO io) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should this take a RootManifestFile instead of a ManifestFile? Passing a leaf manifest would not fail here; the filter would return an empty list, and cleanup would then treat none of the committed manifests as referenced.

String rootManifestLocation =
ops.metadataFileLocation(FileFormat.PARQUET.addExtension(filename));
OutputFile rawOutputFile = ops.io().newOutputFile(rootManifestLocation);
return ops.encryption().encrypt(rawOutputFile);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I assume encryption is out of scope for this first version, which makes sense. Should writeRootManifest fail fast when table encryption is enabled, though? Otherwise the commit succeeds with an encrypted root that can't be read back.

try (V4ManifestReader reader =
builder(root, io, new Schema(), ImmutableMap.of())
.metricsConfig(MetricsConfig.getDefault())
.select(TrackedFile.CONTENT_TYPE.name(), TrackedFile.LOCATION.name())

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: content_type is always projected along with status, which the reader uses to skip deleted entries. Selecting only TrackedFile.LOCATION.name() would be enough here.


public InMemoryInputFile(String location, byte[] contents) {
Preconditions.checkNotNull(location, "location is null");
Preconditions.checkNotNull(contents, "contents is null");

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

curious why did we drop the null check and defensive clone? For this constructor, shouldn't contents be required?

throw new UnsupportedOperationException("Cannot remove data files in v4");
} else if (deleteFilterManager.containsDeletes()) {
throw new UnsupportedOperationException("Cannot remove delete files in v4");
} else if (!appendManifests.isEmpty()) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should this also check rewrittenAppendManifests? For example, a manifest appended to a v1 table is copied into that list, and if the table is upgraded to v4 before commit, the copy isn't added to the root and is deleted after commit, so its files are silently dropped. A test for this case would be good too.


for (TrackedFile dataManifest : root.dataManifests()) {
if (dataManifest.tracking().isLive()) {
rootWriter.existing(dataManifest);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I assume upgrading v1/v2 tables directly to v4 is out of scope for now, since the test plan covers v3 → v4. But that upgrade is currently allowed.

The first append after the upgrade would fail, as v1/v2 manifests have no first_row_id. The v3 manifest list writer assigns row IDs to such manifests, but the root doesn't when carrying them forward. Should the root assign it the same way?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants