Repository navigation
Core: Implement minimal v4 writes in MergingSnapshotProducer - #19030
Conversation
7be2943 to
2519371
Compare
2519371 to
a249e69
Compare
a249e69 to
5247faf
Compare
| DeleteFile[] deletes = NO_DELETES; | ||
| if (dataFile.deletionVector() != null) { | ||
| TrackedFile tracked = TrackedFileAdapters.asTrackedFile(dataFile); | ||
| TrackedFile tracked = TrackedFileAdapters.asTrackedFileUnchecked(dataFile); |
There was a problem hiding this comment.
I added asTrackedFile(DataFile, Schema) to the adapters and also renamed this because it doesn't check that the DataFile is a wrapped TrackedFile.
| */ | ||
| protected void validateAddedDataFiles( | ||
| TableMetadata base, Long startingSnapshotId, PartitionSet partitionSet, Snapshot parent) { | ||
| Long parentId = parent != null ? parent.snapshotId() : null; |
There was a problem hiding this comment.
Validations for v4 metadata are being added in parallel (see #18699). Until those are ready, I've updated the validations to fail if there are any commits after the starting snapshot ID. The checks run as before for v3 and earlier.
| * @param snapshot snapshot to apply the changes to; will be the new snapshot's parent | ||
| * @return a manifest list for the new snapshot. | ||
| */ | ||
| protected abstract List<ManifestFile> apply(TableMetadata metadataToUpdate, Snapshot snapshot); |
There was a problem hiding this comment.
This method is still used for all v1-v3 tables and writeRootManifest is used for v4 and later.
| } | ||
|
|
||
| List<ManifestFile> manifests = apply(base, parentSnapshot); | ||
| protected NewSnapshotMetadata writeManifestList(Snapshot parent, long sequenceNumber) { |
There was a problem hiding this comment.
This method was factored out of apply() and is responsible for writing a v1-v3 manifest list, then returning the metadata for it to build a new snapshot.
This is directly equivalent to the writeRootManifest method that is used for v4, but that one has to be implemented by MergingSnapshotProducer.
| if (!saved.manifestListLocation().equals(manifestList)) { | ||
| deleteFile(manifestList); | ||
| // also clean up unused root files created by multiple attempts | ||
| for (String rootLocation : rootToManifestLocations.keySet()) { |
There was a problem hiding this comment.
Besides changes to apply, the only other change is to handle metadata file cleanup. This is done the same way, except that manifestsByManifestList is now used for all versions and renamed to rootToManifestLocations. The root location (manifest list or root manifest) points to a set of manifest locations that it contains.
The root file that gets committed is used to fetch the manifest paths it contains and those are passed to cleanUncommitted.
| } | ||
| } | ||
|
|
||
| private static Set<String> readManifestLocations( |
There was a problem hiding this comment.
In some cases, cleanup doesn't have an entry in rootToManifestLocations and the root must be read to recover the paths. This implements that read in place of snapshot.allManifests(io).
| * @param io a {@link FileIO} | ||
| * @return a list of manifest locations tracked by the root manifest | ||
| */ | ||
| static List<String> readManifestLocations(ManifestFile root, FileIO io) { |
There was a problem hiding this comment.
Added to produce manifest paths from a root manifest for SnapshotProducer.readManifestLocations.
| import org.junit.jupiter.api.BeforeEach; | ||
| import org.junit.jupiter.api.io.TempDir; | ||
|
|
||
| public class AdaptiveMetadataTestBase { |
There was a problem hiding this comment.
This is equivalent to TestBase, but is for v4 (adaptive metadata tree) testes. This is going to grow as more write cases are implemented and tests are converted from v3 TestBase.
| import org.junit.jupiter.params.ParameterizedTest; | ||
| import org.junit.jupiter.params.provider.FieldSource; | ||
|
|
||
| public class TestAppend extends AdaptiveMetadataTestBase { |
There was a problem hiding this comment.
These are tests ported from TestMergeAppend. I removed tests that did not apply:
- Tests for manifest merge behavior and configuration
- Tests that multiple partition specs produce multiple manifests
- Tests of threadpool usage
- Tests for appendManifest (not yet supported)
- Tests for appends that produce multiple manifests (not yet supported)
- Tests for manifest counters (not yet supported)
Tests that were ported over:
- Append cases
- Multiple partition specs in a write
- Failure to commit from repeated
CommitFailedException(and cleanup) - Recovery using retries after
CommitFailedException(and cleanup)
| throw new UnsupportedOperationException("Cannot commit deleted delete files to v4"); | ||
| } | ||
|
|
||
| if (rootManifestProducer == null || hasNewDataFiles) { |
There was a problem hiding this comment.
I don't think we're actually getting reuse here because we rebuild on retry.
There was a problem hiding this comment.
I think all we need to do to address this is set hasNewDataFiles = false after adding all the files
There was a problem hiding this comment.
There is not much reuse right now because the root producer doesn't create leaf manifests yet. Right now, this just reuses the commit UUID that is generated.
| validateFiles( | ||
| rootFiles, | ||
| dataSeqs(1L, 1L), | ||
| fileSeqs(1L, 1L), | ||
| ids(snapshotId, snapshotId), | ||
| locations(FILE_A, fileNewSpec), | ||
| statuses(EntryStatus.ADDED, EntryStatus.ADDED)); |
There was a problem hiding this comment.
This doesn't actually assert what the name indicates because we don't check which spec was written to.
There was a problem hiding this comment.
There are writer tests for mixed partition specs that validate partitions are correctly stored and read. These tests are for the manifest layout that results from mixed partition specs. So the assertion here is that both files were added to a single root manifest, rather than two separate leaf manifests for different partition specs.
| assertThat(latest.rootManifestLocation()) | ||
| .as("Pending root manifest should be rewritten") | ||
| .isNotEqualTo(pendingRootLocation); | ||
| assertThat(defaultTable.io().newInputFile(pendingRootLocation).exists()).isFalse(); |
There was a problem hiding this comment.
nit: it would be nice to add an as() context here since this will just result in something like "expected false, but got true".
There was a problem hiding this comment.
Now that this no longer uses LocalFileIO, it can list metadata files and make assertions that are more reliable than this. I updated it to:
assertThat(listRootManifests(defaultTable))
.containsExactlyInAnyOrder(latest.rootManifestLocation(), snap.rootManifestLocation());| * | ||
| * @param metadataToUpdate the base table metadata to apply changes to | ||
| * @param snapshot snapshot to apply the changes to | ||
| * @param snapshot snapshot to apply the changes to; will be the new snapshot's parent |
There was a problem hiding this comment.
Is the comment change neccessary?
There was a problem hiding this comment.
I updated this for consistency with the new method, which makes this statement. It's not strictly required if you think it should be removed.
| throw new UnsupportedOperationException("Cannot commit deleted delete files to v4"); | ||
| } | ||
|
|
||
| if (rootManifestProducer == null || hasNewDataFiles) { |
There was a problem hiding this comment.
I think all we need to do to address this is set hasNewDataFiles = false after adding all the files
| import org.apache.iceberg.relocated.com.google.common.base.Preconditions; | ||
| import org.apache.iceberg.relocated.com.google.common.collect.Maps; | ||
|
|
||
| public class InMemoryTables { |
There was a problem hiding this comment.
This is equivalent to TestBase, but doesn't rely on LocalFileIO because it produces paths that do not have a scheme. This will be used for all v4 (and later) test suites.
| MetadataUpdate.AddSnapshot addSnapshot = | ||
| (MetadataUpdate.AddSnapshot) body.updates().get(0); | ||
| assertThatThrownBy( | ||
| () -> table.io().newInputFile(addSnapshot.snapshot().manifestListLocation())) |
There was a problem hiding this comment.
This relied on incorrect behavior in InMemoryFileIO where you could not create an InputFile unless the file existed. The correct behavior is to check InputFile#exists().
| @Override | ||
| public long sequenceNumber() { | ||
| return file.tracking().dataSequenceNumber(); | ||
| return file.tracking() != null ? file.tracking().dataSequenceNumber() : -1; |
There was a problem hiding this comment.
This was updated to match the current behavior of ManifestFile before it is committed. Otherwise the appendManifest tests fail because tracking() throws a NullPointerException.
| closed = true; | ||
| } | ||
|
|
||
| public void clear() { |
There was a problem hiding this comment.
Used to clear files between test cases so that listing metadata files doesn't find metadata from other cases.
danielcweeks
left a comment
There was a problem hiding this comment.
Some minor comments, but I don't think any major blockers.
| public InputFile newInputFile(String location) { | ||
| Preconditions.checkState(!closed, "Cannot call newInputFile after calling close()"); | ||
| byte[] contents = IN_MEMORY_FILES.get(location); | ||
| if (null == contents) { | ||
| throw new NotFoundException("No in-memory file found for location: %s", location); | ||
| } | ||
| return new InMemoryInputFile(location, contents); | ||
| return new InMemoryInputFile(location, this); |
There was a problem hiding this comment.
It looks like there might be a subtle behavior change here, but I believe we only use the InMemoryFileIO for testing purposes, so it's probably ok?
There was a problem hiding this comment.
I mentioned this in a different comment, but the previous behavior violated the FileIO contract so this is a bug fix to unblock the tests. And yes, we use this for testing (and didn't have many tests depending on this behavior) so I think it's okay.
| assertThat( | ||
| table | ||
| .io() | ||
| .newInputFile(addSnapshot.snapshot().manifestListLocation()) | ||
| .exists()) | ||
| .isFalse(); |
There was a problem hiding this comment.
nit: .as("manifest list should be cleaned up: %s", location) would be nice here
|
Merged. Thanks, @danielcweeks and @amogh-jahagirdar for reviewing! |
|
|
||
| protected void cleanAll() { | ||
| for (String manifestList : manifestsByManifestList.keySet()) { | ||
| for (String manifestList : rootToManifestLocations.keySet()) { |
There was a problem hiding this comment.
nit: manifestList -> rootLocation/rootFile
| * @param io a {@link FileIO} | ||
| * @return a list of manifest locations tracked by the root manifest | ||
| */ | ||
| static List<String> readManifestLocations(ManifestFile root, FileIO io) { |
There was a problem hiding this comment.
Should this take a RootManifestFile instead of a ManifestFile? Passing a leaf manifest would not fail here; the filter would return an empty list, and cleanup would then treat none of the committed manifests as referenced.
| String rootManifestLocation = | ||
| ops.metadataFileLocation(FileFormat.PARQUET.addExtension(filename)); | ||
| OutputFile rawOutputFile = ops.io().newOutputFile(rootManifestLocation); | ||
| return ops.encryption().encrypt(rawOutputFile); |
There was a problem hiding this comment.
I assume encryption is out of scope for this first version, which makes sense. Should writeRootManifest fail fast when table encryption is enabled, though? Otherwise the commit succeeds with an encrypted root that can't be read back.
| try (V4ManifestReader reader = | ||
| builder(root, io, new Schema(), ImmutableMap.of()) | ||
| .metricsConfig(MetricsConfig.getDefault()) | ||
| .select(TrackedFile.CONTENT_TYPE.name(), TrackedFile.LOCATION.name()) |
There was a problem hiding this comment.
nit: content_type is always projected along with status, which the reader uses to skip deleted entries. Selecting only TrackedFile.LOCATION.name() would be enough here.
|
|
||
| public InMemoryInputFile(String location, byte[] contents) { | ||
| Preconditions.checkNotNull(location, "location is null"); | ||
| Preconditions.checkNotNull(contents, "contents is null"); |
There was a problem hiding this comment.
curious why did we drop the null check and defensive clone? For this constructor, shouldn't contents be required?
| throw new UnsupportedOperationException("Cannot remove data files in v4"); | ||
| } else if (deleteFilterManager.containsDeletes()) { | ||
| throw new UnsupportedOperationException("Cannot remove delete files in v4"); | ||
| } else if (!appendManifests.isEmpty()) { |
There was a problem hiding this comment.
Should this also check rewrittenAppendManifests? For example, a manifest appended to a v1 table is copied into that list, and if the table is upgraded to v4 before commit, the copy isn't added to the root and is deleted after commit, so its files are silently dropped. A test for this case would be good too.
|
|
||
| for (TrackedFile dataManifest : root.dataManifests()) { | ||
| if (dataManifest.tracking().isLive()) { | ||
| rootWriter.existing(dataManifest); |
There was a problem hiding this comment.
I assume upgrading v1/v2 tables directly to v4 is out of scope for now, since the test plan covers v3 → v4. But that upgrade is currently allowed.
The first append after the upgrade would fail, as v1/v2 manifests have no first_row_id. The v3 manifest list writer assigns row IDs to such manifests, but the root doesn't when carrying them forward. Should the root assign it the same way?
This is a minimal set of changes to get v4 writes working with
SnapshotProducerandMergingSnapshotProducer. Root manifests are managed and written byRootManifestProducer, which is roughly the v4 equivalent ofMergingSnapshotProducer. This implements simple appends that write new data files to the root file. This does not write leaf files.SnapshotProduceris refactored to have a method for writing a manifest list (writeManifestList) and one for writing a root metadata file (writeRootManifest). Nearly everything else inSnapshotProduceris reused, although the state used for file cleanup (tested in failure and recovery tests).MergingSnapshotProduceris updated to internally useRootManifestProducerfor v4 writes. Initially, I thought that it would be easier to replaceMergingSnapshotProducerwith a v4 version, but in the end that strategy would require a lot of churn with little benefit, so I decided to add a delegate within the class instead. We should be able to follow up and build a similar abstraction around the v3 behavior as well so thatMergingSnapshotProducerselects its implementation instead of doing work directly.Test plan:
TestMergeAppendTestBasefor adaptive metadataNote that there are many test cases that are not ported because the supported appends are very simple. The plan is to get a basic write working and then expand functionality and tests at the same time.