Repository navigation
Spark 4.2: Implement RepairTable action - #17622
Conversation
| return builder | ||
| .withFileSizeInBytes(corruptCounts ? file.fileSizeInBytes() + 4096 : file.fileSizeInBytes()) | ||
| .build(); | ||
| } |
There was a problem hiding this comment.
TODO: add test with custom stats column (not 100) and contains the iceberg property for stats column
anuragmantri
left a comment
There was a problem hiding this comment.
Thanks @rahulsmahadev. This is great and solves a repeated pain point for our customers with corrupted data.
I did an initial review and found that partition evolution causes the logic to write incorrect spec ids. Details in the comments below.
| return new ManifestWriterFactory( | ||
| sparkContext().broadcast(SerializableTableWithSize.copyOf(table)), | ||
| formatVersion, | ||
| table.spec().specId(), |
There was a problem hiding this comment.
Load manifests in L301 reads every data/delete manifest of the current snapshot regardless of partitionSpecId() and here in the writer they are bound to table.spec().specId(), which is the table's current default spec.
If there was partition evolution and we are reading older manifests, this will write incorrect partition ids to older schemas. This is the same class of bug as #666
There was a problem hiding this comment.
I ran this test and verified it is indeed an issue
@TestTemplate
public void testRepairAfterPartitionSpecEvolution() throws IOException {
Table table = createTable(PartitionSpec.unpartitioned());
appendRecords(table, records(4));
DataFile original = onlyDataFile(table);
assertThat(original.specId()).isEqualTo(0);
assertThat(original.partition().size()).isEqualTo(0);
// evolve the table to a partitioned spec; the existing manifest keeps referring to spec 0
table.updateSpec().addField("c1").commit();
table.refresh();
assertThat(table.spec().specId()).isEqualTo(1);
ManifestFile oldManifest = table.currentSnapshot().dataManifests(table.io()).get(0);
assertThat(oldManifest.partitionSpecId())
.as("the manifest written before the evolution must still be tagged with the old spec")
.isEqualTo(0);
// corrupt the stats of the entry that still belongs to the original, unpartitioned spec
corruptStats(table, oldManifest, original.location());
SparkActions.get().repairTable(table).execute();
table.refresh();
DataFile repaired = onlyDataFile(table);
assertThat(repaired.recordCount())
.as("the repair must still correct the stats")
.isEqualTo(original.recordCount());
assertThat(repaired.specId())
.as("the repaired entry must keep the spec it was originally written under")
.isEqualTo(0);
assertThat(repaired.partition().size())
.as("an unpartitioned file's partition data must still have zero fields after repair")
.isEqualTo(0);
}There was a problem hiding this comment.
Very good catch and thanks for the repro, handled it in 063328f
| ? new WriteDataManifests(writers, combinedFileType, sparkType, repaired, context) | ||
| : new WriteDeleteManifests(writers, combinedFileType, sparkType, repaired, context); | ||
|
|
||
| // preserve the entry order of the manifests being rewritten |
There was a problem hiding this comment.
I did not understand this comment. I don't think repartition(n) preserves order. Can you clarify?
There was a problem hiding this comment.
Updated the comment, you are right
szehon-ho
left a comment
There was a problem hiding this comment.
The data file path mirrors RewriteManifestsSparkAction closely and the concurrency tests are thorough. My main concern is the delete file path: two issues there look like real correctness bugs, and that path currently has no test coverage. Details inline.
Reviewed with AI assistance (Claude Opus 5 via Cursor). Findings were checked against the source by hand. The two correctness findings were derived from reading the builders and DeleteFilter, not from a reproduction.
| /** Returns the name mapping of the table, or null if the table does not define one. */ | ||
| static NameMapping nameMapping(Table table) { | ||
| String mapping = | ||
| table.properties().get(org.apache.iceberg.TableProperties.DEFAULT_NAME_MAPPING); |
There was a problem hiding this comment.
Import DEFAULT_NAME_MAPPING rather than qualifying it inline. Same for org.apache.iceberg.io.InputFile at RepairTableSparkAction L562 and L704, and java.util.function.Function at L388.
| .withFileSizeInBytes(fileSizeInBytes) | ||
| .build(); | ||
| } else { | ||
| return FileMetadata.deleteFileBuilder(spec) |
There was a problem hiding this comment.
Add .ofEqualityDeletes(...) when the file is an equality delete. FileMetadata.Builder.copy(DeleteFile) doesn't carry equalityFieldIds and build() doesn't validate them, so the rebuilt entry keeps content = EQUALITY_DELETES but writes equality_ids = null. DeleteFilter then does Sets.newHashSet(delete.equalityFieldIds()) and NPEs on the next read, with the bad entry already committed.
RewriteTablePathUtil.newEqualityDeleteEntry restores the ids explicitly for this reason.
| */ | ||
| public static final String REPAIR_COLUMN_METRICS = "repair-column-metrics"; | ||
|
|
||
| public static final boolean REPAIR_COLUMN_METRICS_DEFAULT = true; |
There was a problem hiding this comment.
Reword the javadoc above: disabling this doesn't avoid reading footers, since CheckStats calls readMetrics unconditionally and the option only skips the comparison.
Separately, consider defaulting to false, or restricting the comparison to columns present in both maps. MetricsConfig.forTable reflects the table's current schema and sort order and nothing records the config a file was written under, so true will rewrite manifests over stats that were correct when written. The clearest case is column count: once a table grows past write.metadata.metrics.max-inferred-column-defaults (100), MetricsConfig.from assigns the default mode to only the first 100 field ids and None to the rest, so recomputed metrics carry fewer bounds than entries written while the table was narrower. boundsMatch fails, the entry is flagged, and withStats writes the reduced set — the repair drops valid bounds. Setting write.metadata.metrics.default to none or counts gets there too, via the sorted-column promotion.
| } | ||
|
|
||
| @Override | ||
| public RepairTableSparkAction repairFileMetrics() { |
There was a problem hiding this comment.
Set a field here and gate the repair on it, so execute() is a no-op when no repair has been selected. RepairTable's javadoc says the repairs are selected through the configuration methods, but today the flag is unobservable and execute() repairs metrics regardless.
| return EMPTY_RESULT; | ||
| } | ||
|
|
||
| if (dryRun) { |
There was a problem hiding this comment.
Return before writeManifests on a dry run rather than writing manifests and deleting them here. Both result fields come from the verdicts, so the write isn't needed, and a failure between writing and this cleanup leaves orphan repaired-m-*.avro files in the metadata directory. If the write is deliberate as a rehearsal of the real path, a note on dryRun would help, since creating files is surprising for a dry run.
| } | ||
|
|
||
| SparkContentFile<?> newFileWrapper(Types.StructType combinedFileType, StructType sparkType) { | ||
| Types.StructType fileType = DataFile.getType(table().spec().partitionType()); |
There was a problem hiding this comment.
Pass the group's spec here, or null to skip the projection since it's unused. This projects to the current default spec, the same thing behind the bug @anuragmantri caught on the write side; it's harmless today only because CheckStats never reads partition().
| } | ||
|
|
||
| /** A manifest entry whose statistics disagree with the file it refers to. */ | ||
| public static class EntryVerdict implements Serializable { |
There was a problem hiding this comment.
Use Encoders.tuple(Encoders.STRING(), Encoders.STRING()), or select the two columns and collect Rows. Either avoids adding a public bean with setters to org.apache.iceberg.spark.actions just to satisfy Encoders.bean.
| Row fileRow = row.getStruct(4); | ||
| StructType sparkType = (StructType) fileRow.schema(); | ||
| Types.StructType combinedFileType = | ||
| DataFile.getType(Partitioning.partitionType(context.table())); |
There was a problem hiding this comment.
Hoist combinedFileType and the wrapper out of the loop, since they're identical for every row. As written each entry recomputes Partitioning.partitionType(table), rebuilds the DataFile struct type, and constructs a SparkContentFile, which runs StructProjection.create and builds a field position map. WriteManifests.call creates its wrapper once per partition.
| Metrics metrics = | ||
| RepairMetrics.readMetrics( | ||
| input, file, context.metricsConfig(file), context.nameMapping()); | ||
| return RepairMetrics.withStats(file, context.spec(file.specId()), metrics, fileSizeInBytes); |
There was a problem hiding this comment.
Preserve the existing column stats when repair-column-metrics is false. repairStats applies the full recomputed Metrics regardless, so an entry flagged only for a wrong record count or file size still gets its column stats replaced — exactly the case the option exists to protect against, since its rationale is that a table whose metrics config changed reports legitimately different column stats.
|
|
||
| @Parameters(name = "formatVersion = {0}") | ||
| public static Object[] parameters() { | ||
| return new Object[][] {new Object[] {1}, new Object[] {2}, new Object[] {3}}; |
There was a problem hiding this comment.
Add coverage for delete manifests: a position delete with wrong stats, an equality delete with wrong stats, a manifest holding both, and a v3 manifest with a DV next to a repairable entry asserting referencedDataFile, contentOffset and contentSizeInBytes survive. These cases are all data files, so WriteDeleteManifests and SparkDeleteFile ship unexercised even though doExecute iterates ManifestContent.values(). The first two would catch the issues I flagged in RepairMetrics.withStats and RepairContext.metricsConfig.
54c26be to
aaf6b9c
Compare
| String location = file.location().toString(); | ||
|
|
||
| try { | ||
| InputFile input = context.io().newInputFile(location); |
There was a problem hiding this comment.
Read file metrics through EncryptingFileIO.combine(table.io(), table.encryption()) and the DataFile/DeleteFile overloads. Opening by path bypasses key metadata, so encrypted files are read as ciphertext, caught as unreadable, and silently left unrepaired. The same change is needed in repairStats; keep the physical-size lookup on the raw input because the decrypted input reports plaintext length.
| } | ||
|
|
||
| private OutputFile newOutputFile() { | ||
| return table().io().newOutputFile(newManifestLocation()); |
There was a problem hiding this comment.
Encrypt the output with table.encryption().encrypt(rawOutputFile) and use the encrypted ManifestFiles.write overload. Otherwise a repair commits plaintext manifests with null key metadata even when table encryption is configured; those manifests contain file paths, partition values, and column bounds.
| return !countsMatch(file.columnSizes(), metrics.columnSizes()) | ||
| || !countsMatch(file.valueCounts(), metrics.valueCounts()) | ||
| || !countsMatch(file.nullValueCounts(), metrics.nullValueCounts()) | ||
| || !countsMatch(file.nanValueCounts(), metrics.nanValueCounts()) |
There was a problem hiding this comment.
Preserve metrics that cannot be reconstructed from file footers, or exclude them from this comparison. ParquetUtil.fileMetrics and OrcMetrics.fromInputFile have no writer-tracked FieldMetrics, so a correct float or double file containing NaNs recomputes an empty nanValueCounts map and may also lose the NaN-safe bounds captured by the writer. With repair-column-metrics=true, the entry is flagged and rewritten with weaker statistics. Please add no-op coverage for a file containing NaNs.
| List<ManifestFile> newManifests = Lists.newArrayList(); | ||
| long repairedCount = 0L; | ||
|
|
||
| for (ManifestContent content : ManifestContent.values()) { |
There was a problem hiding this comment.
Clean up manifests written by completed groups if a later group fails before commit. For example, if data-manifest repair writes output and delete-manifest repair then throws, execution exits before replaceManifests, which owns the current cleanup. The same leak can occur between partition-spec groups in repairTable. The existing cleanup test only injects a failure during commit, after all output has been collected.
| */ | ||
| static boolean supportsMetrics(ContentFile<?> file) { | ||
| FileFormat format = file.format(); | ||
| return format == FileFormat.PARQUET || format == FileFormat.ORC || format == FileFormat.AVRO; |
There was a problem hiding this comment.
Either support deletion vectors here or document their exclusion in the public API and Spark action Javadocs. The Puffin footer exposes the blob offset, length, referenced-data-file, and cardinality, and the input exposes the physical file length, so the DV record count, file size, and DV-specific metadata are recoverable in the normal case. RepairTable.repairFileMetrics() currently promises comparison against data and delete files, while this silently skips every DV. At minimum, document that unsupported formats are skipped and left unchanged.
There was a problem hiding this comment.
documented for now
| org.apache.iceberg.RewriteManifests rewriteManifests = table.rewriteManifests(); | ||
| deletedManifests.forEach(rewriteManifests::deleteManifest); | ||
| addedManifests.forEach(rewriteManifests::addManifest); | ||
| commit(rewriteManifests); |
There was a problem hiding this comment.
Recompute snapshot totals as part of the repair commit. RewriteManifests carries forward the previous totals, so a four-row file initially appended with record_count=104 is repaired to 4 here while total-records stays 104. Spark still uses that total for unfiltered partitioned-table scan estimates. The file-size and delete-count totals have the same issue. Please add a test that initially commits incorrect metrics and checks the repaired snapshot summary.
| InputFile input, ContentFile<?> file, MetricsConfig config, NameMapping mapping) { | ||
| switch (file.format()) { | ||
| case PARQUET: | ||
| return ParquetUtil.fileMetrics(input, config, mapping); |
There was a problem hiding this comment.
Resolve per-column metrics settings by field ID when reading files written before a column rename. For example, write a with default=none and a=full, then rename a to b. SchemaUpdate moves the override to b, but the footer still names the field a and MetricsUtil.metricsMode looks up that old name, falling back to none. With column repair enabled, this drops valid statistics despite the intended configuration staying the same. Please add a no-op test for this rename case.
| long repairedCount = verdicts.count(); | ||
| List<ManifestFile> rewritten = | ||
| manifests.stream() | ||
| .filter(manifest -> manifestsToRewrite.contains(manifest.path())) |
There was a problem hiding this comment.
Use a Set for these path lookups. manifestsToRewrite is a List, so filtering N manifests requires roughly N squared / 2 string comparisons on the driver when all are affected.
| } | ||
|
|
||
| @TestTemplate | ||
| public void testRepairIncorrectRecordCountAndFileSize() throws IOException { |
There was a problem hiding this comment.
Parameterize the file-format coverage over Parquet, ORC, and Avro. The current tests only write Parquet, so the separate ORC and Avro metrics readers are untested by this action. Include a correct-file no-op and a record-count/file-size repair for each format.
| SparkActions.get().repairTable(table).repairFileMetrics().execute(); | ||
|
|
||
| table.refresh(); | ||
| assertThat(entryLineage()) |
There was a problem hiding this comment.
Extend the v3 lineage test to multiple data files and assert that data_file.first_row_id survives the repair. entryLineage() currently checks only the snapshot ID and sequence numbers, so it would miss changed row IDs.
| assertThat(repaired.nullValueCounts()).isEqualTo(original.nullValueCounts()); | ||
| assertThat(repaired.lowerBounds()).isEqualTo(original.lowerBounds()); | ||
| assertThat(repaired.upperBounds()).isEqualTo(original.upperBounds()); | ||
| assertThat(currentRows()).containsExactlyInAnyOrderElementsOf(expectedRows); |
There was a problem hiding this comment.
Run the repair a second time with the same column-metrics option and assert zero repaired entries, no repaired manifests, and an unchanged snapshot ID. The current tests stop after the first repair, so they do not verify that repeating the action is a no-op.
There was a problem hiding this comment.
cool added an idempotenc check
Adds a Spark implementation of the RepairTable action, which repairs manifest entries whose statistics disagree with the files they refer to. The statistics of every live entry are compared against the file by reading its footer, and only the manifests that contain at least one incorrect entry are rewritten, so the cost of the commit is proportional to the number of incorrect entries rather than to the size of the table. Entries are rewritten with ManifestWriter#existing, carrying through the original snapshot id and data and file sequence numbers. This preserves the lineage of the files, and therefore which delete files apply to them, so the repair leaves the contents of the table unchanged. Repairing statistics does not change the number of live files, so the commit goes through the existing rewrite manifests validation unchanged. - RepairMetrics reads and compares statistics per format (Parquet, ORC and Avro) and rebuilds a file with corrected statistics. - Files whose statistics cannot be read are carried through unchanged and counted as incorrect but not repaired, which is what distinguishes entryStatsIncorrectCount from entryStatsRepairedCount. - The repair-column-metrics option skips the footer reads and repairs only record counts and file sizes, for tables where only those are suspect. - dryRun reports what would be repaired without committing.
Covers what happens when the table changes underneath a repair, and what is left behind when the commit does not succeed: - a concurrent append commits between planning and the repair commit: the repair succeeds and the appended records survive, since the appended data lands in a new manifest and the manifests being repaired are still present - a concurrent operation replaces the very manifest being repaired: the commit fails validation in BaseRewriteManifests#validateDeletedManifests and the table is left untouched, rather than dropping the concurrent change - a failed commit deletes the manifests the repair wrote - a commit reported as CommitStateUnknownException keeps them, as the commit may have succeeded - a dry run leaves none of them behind Note the cleanup tests assert on the manifests written by the action itself. A failed commit on a format version 1 table can also leave behind a copy of a manifest made by the core staging path in BaseRewriteManifests, which is only cleaned up after a successful commit and is not owned by this action.
Manifests of the current snapshot were loaded regardless of their partition spec, but the writer was bound to the current default spec of the table. After partition evolution, rewriting a manifest of an older spec therefore wrote the current spec id and partition type to its entries. Group the manifests by partition spec id and repair each group with a writer bound to that spec, so an entry keeps the spec it was written under. Also pass the spec specific file type to the writer instead of deriving it from the current spec of the table.
Correctness: - Equality deletes now keep their equality field ids when rebuilt. FileMetadata.Builder.copy(DeleteFile) drops them, so a repaired entry otherwise kept content EQUALITY_DELETES with null equality ids, and reading the table failed once the delete was applied. - The metrics config is resolved per file content, so a delete manifest holding both position and equality deletes no longer compares equality deletes against the position-delete config and rewrites them needlessly. - The check-side file wrapper binds to the manifest's own spec rather than the table's current spec. - When column-metrics repair is disabled, only the record count and file size of a flagged entry are corrected; the stored column stats are kept instead of being replaced with the recomputed ones. Behavior: - repairFileMetrics() now selects the repair; execute() is a no-op when no repair has been selected, as the interface describes. - repair-column-metrics defaults to false. Recomputed column stats reflect the current metrics config, which the file may not have been written under, so repairing them can overwrite correct statistics. - A dry run no longer writes manifests only to delete them; it reports what would be repaired without writing anything. Also hoist the combined file type and file wrapper out of the per-row loop in CheckStats, and replace inline-qualified references with imports.
… delete manifests The set of incorrect file paths was collected to the driver and broadcast back out. That set is unbounded, since a writer that recorded stats incorrectly usually did so for every file it wrote. Mark the entries to repair by joining the entries against the verdicts on the file path instead, so the set stays distributed. Only the set of manifests to rewrite, which is naturally small, is still collected. Verdicts are emitted as tuples rather than a bean and cached, as they are read more than once. Repartition the entries by manifest when writing so the manifest layout of the table is preserved, rather than scattered round robin by a plain repartition(n). Guard the equality field ids in RepairMetrics.withStats: an entry that is an equality delete but records no equality ids is carried through as is rather than throwing, so a malformed entry does not abort the whole repair. Add end-to-end coverage of the delete manifest path, which was previously unexercised: repair a position delete, an equality delete, and a manifest holding both, asserting the statistics are corrected and the equality field ids survive.
The two tests asserting that a failed repair leaves the table unchanged scan the table while the corruption is still present. Parquet now fetches small files eagerly using the recorded file size, so an inflated file_size_in_bytes makes the scan read past the end of the file. Corrupt only the record count in these tests, keeping the file size accurate, so the table stays readable.
Add testRepairPreservesDeletionVectorFields: a v3 table with a deletion vector and a corrupt equality delete in one manifest, asserting the DV's referencedDataFile, contentOffset, and contentSizeInBytes survive repair.
Read data-file metrics through EncryptingFileIO so encrypted files are decrypted with their key metadata rather than read as ciphertext and skipped as unreadable, and write repaired manifests through the table's encryption so they are not committed as plaintext. ParquetUtil.fileMetrics now decrypts a NativeEncryptionInputFile (mirroring Parquet.read()), which is required to read metrics from an encrypted Parquet data file. Add TestRepairTableActionEncryption covering repair on an encrypted v3 table, plus a custom column-metrics test.
…rics in RepairTable
f60e945 to
655899f
Compare
| .forEach( | ||
| (id, field) -> { | ||
| Type.TypeID typeId = field.type().typeId(); | ||
| if (typeId == Type.TypeID.FLOAT || typeId == Type.TypeID.DOUBLE) { |
There was a problem hiding this comment.
Preserve geometry bounds alongside floating-point bounds. GenericParquetWriter collects geometry bounding boxes, but ParquetMetrics.metricsFromFooter intentionally returns only counts for geometry. With repair-column-metrics=true, a correct file therefore appears corrupt, and the replacement entry loses its valid geometry bounds. Add a no-op test using a geometry file written by GenericParquetWriter.
There was a problem hiding this comment.
Done in 7e8ad05. Geometry and geography bounds are preserved
| .filter(manifest -> !deletedPaths.contains(manifest.path())) | ||
| .collect(Collectors.toList()); | ||
| manifests.addAll(addedManifests); | ||
| updateSnapshotTotals(rewriteManifests, manifests); |
There was a problem hiding this comment.
Handle failures from updateSnapshotTotals as cleanable before propagating them. Its Spark job runs during validation, before the metadata commit, but a SparkException bypasses the CleanableFailure-only cleanup below. BaseRewriteManifests also deliberately leaves caller-supplied manifests untouched, so all replacement manifests already written become orphans. Add coverage that fails the totals job after manifest writing completes.
…tals failure in RepairTable
| } finally { | ||
| writer.close(); | ||
| } | ||
|
|
||
| return writer.toManifestFiles().iterator(); |
There was a problem hiding this comment.
Clean up output manifests when this task attempt fails. If a later repairStats call throws after earlier entries have been written, finally closes the partial manifest, but toManifestFiles() is never returned to the driver. The cleanup around content/spec groups only knows about completed groups, so these files remain orphaned even if a Spark retry succeeds. Please add coverage for a failure after at least one entry has been written.
There was a problem hiding this comment.
Done, A failed write task now closes its writer and deletes every manifest it wrote before rethrowing.
| deleteFiles(Iterables.transform(addedManifests, ManifestFile::path)); | ||
| throw e; |
There was a problem hiding this comment.
Propagate failures from this totals callback as a cleanable failure so core cleanup also removes the staged copies. For v1 with snapshot-ID inheritance disabled, addManifest() has already copied the replacement manifests into rewrittenAddedManifests. This deletes only the original addedManifests; rethrowing a SparkException bypasses SnapshotProducer.cleanAll() under strict cleanup, leaving those copies behind. The failure test checks only repaired-m- files and misses them. This cleanup is safe because validation runs before metadata publication; keep the existing handling of unknown commit outcomes.
| SparkContentFile<?> newFileWrapper(Types.StructType combinedFileType, StructType sparkType) { | ||
| Types.StructType fileType = DataFile.getType(spec(specId).partitionType()); | ||
| return content == ManifestContent.DATA | ||
| ? new SparkDataFile(combinedFileType, fileType, sparkType) | ||
| : new SparkDeleteFile(combinedFileType, fileType, sparkType); |
There was a problem hiding this comment.
Load entries with their original partition tuples before projecting to the manifest's spec. After writing files under bucket(16, c1), removing that partition field and dropping c1, Partitioning.partitionType(table) and the entries metadata table omit the field, while the old spec still contains it. SparkContentFile then throws Cannot find field ... during CheckStats, aborting repair and dry runs while old files remain live. Please add coverage for this sequence; allowing missing fields alone would lose the original partition values.
…uples - Delete the manifests a write task produced when the task fails, since a failed attempt never returns them to the driver. - Rethrow a snapshot totals failure as a CleanableFailure so core cleanup also removes the staged copies made for v1 tables without snapshot ID inheritance. - Read rewritten entries from the original manifests so partition fields whose source column was dropped keep their values, and check stats without projecting the partition to the manifest's spec.
The data group writes its repaired manifest, then the delete group fails while writing its own. The test checks that the table is unchanged and that the manifest written by the data group is deleted.
| } | ||
| } | ||
|
|
||
| private void deleteFiles(Iterable<String> locations) { |
There was a problem hiding this comment.
I think we can just use CatalogUtil.deleteFiles instead of defining this again.
There was a problem hiding this comment.
Done, the action now uses CatalogUtil.deleteFiles and the private helper is gone. One difference: for a FileIO without bulk deletes it doesn't retry, while the old helper retried three times. That seems fine for best-effort cleanup of uncommitted manifests, but let me know if you'd rather keep the retries.
| (long) (1.2 * targetManifestSizeBytes)); | ||
| } | ||
|
|
||
| private <T, U> U withReusableDS(Dataset<T> ds, Function<Dataset<T>, U> func) { |
There was a problem hiding this comment.
This is the same as the one in RewriteManifestsSparkAction. Can we move it to BaseSparkAction and share it?
There was a problem hiding this comment.
Done, moved withReusableDS to BaseSparkAction. The option key and its default live there now too, and both actions' public USE_CACHING constants point at them.
| } | ||
| } | ||
|
|
||
| private static class ManifestWriterFactory implements Serializable { |
There was a problem hiding this comment.
This is nearly identical to the factory in RewriteManifestsSparkAction, apart from the manifest file prefix. Can we share one factory and pass the prefix in? Same for the outputLocation computation in the constructor.
There was a problem hiding this comment.
Done. ManifestWriterFactory is now a package-private class that both actions share, with the manifest prefix passed in, and ManifestWriterFactory.defaultOutputLocation(table) replaces the outputLocation code in both constructors.
| .collectAsList(); | ||
| } | ||
|
|
||
| private RepairContext newRepairContext(ManifestContent content) { |
There was a problem hiding this comment.
Each spec group ends up broadcasting the table twice, once for the check and once for the write. Can we broadcast it once and pass it through?
There was a problem hiding this comment.
Done, the table is broadcast once per run and passed to the check and the write of every spec group.
| repairColumnMetrics); | ||
| } | ||
|
|
||
| private List<ManifestFile> loadManifests(ManifestContent content, Snapshot snapshot) { |
There was a problem hiding this comment.
This can be a switch expression without a default, since the enum is exhaustive.
| .copy(delete) | ||
| .withMetrics(metrics) | ||
| .withFileSizeInBytes(fileSizeInBytes); | ||
| List<Integer> equalityFieldIds = delete.equalityFieldIds(); |
There was a problem hiding this comment.
These two have the same body. Can we keep one generic helper?
There was a problem hiding this comment.
Done, countsMatch and boundsMatch are now one generic mapsMatch.
| newManifests.addAll(repaired.newManifests()); | ||
| repairedCount += repaired.repairedCount(); | ||
| } | ||
| } catch (Exception e) { |
There was a problem hiding this comment.
The cleanup here and in doExecute both delete the manifests written so far, one per level. Can we collect the written manifests in one list and clean up in one place? Right now a reader has to check both catches to convince themselves nothing leaks.
There was a problem hiding this comment.
Done. doExecute now loops over the content types and spec groups itself, collects every written manifest in one list, and deletes them in its single catch.
| public Iterator<ManifestFile> call(Iterator<Row> rows) throws Exception { | ||
| Map<String, Map<String, Row>> entriesByManifest = Maps.newHashMap(); | ||
| rows.forEachRemaining( | ||
| row -> |
There was a problem hiding this comment.
These row indices only match the select in markEntriesToRepair, which is a few calls away. Can we pull the indices into constants so the contract is visible here?
There was a problem hiding this comment.
Done, the positions are named constants now (MANIFEST_INDEX and the others), and the select in markEntriesToRepair points to them.
| return format == FileFormat.PARQUET || format == FileFormat.ORC || format == FileFormat.AVRO; | ||
| } | ||
|
|
||
| static Metrics readMetrics( |
There was a problem hiding this comment.
This does more than read: the Avro, position delete and NaN/geometry branches already merge the stored stats, so what it returns is really the metrics the entry should have. Can we rename it to expectedMetrics? It reads well at both call sites, comparing stored against expected.
There was a problem hiding this comment.
Done, renamed to expectedMetrics.
…eManifests Move withReusableDS and its use-caching option to BaseSparkAction, and the manifest writer factory and default output location to a shared ManifestWriterFactory that takes the file prefix. RepairTable now uses CatalogUtil.deleteFiles, broadcasts the table once, cleans up written manifests in one place, and names the row positions it reads. RepairMetrics.readMetrics becomes expectedMetrics, and countsMatch and boundsMatch become one generic helper. Co-authored-by: Isaac <no-reply@databricks.com>
|
|
||
| List<DataFile> originals = dataFiles(table); | ||
| assertThat(originals).hasSize(2); | ||
| if (formatVersion >= 3) { |
There was a problem hiding this comment.
These assertions silently don't run for v1 and v2, so two of the three parameter values skip them entirely. Can we make this a v3-only test so the parameterization matches what's actually asserted?
There was a problem hiding this comment.
Done, it's a v3-only test now, so the row ID checks always run.
There was a problem hiding this comment.
Thanks for all the work and patience on this @rahulsmahadev. Given the size, I'd like another set of eyes on this before it goes in, cc @szehon-ho @anuragmantri @nastra.
In hindsight it probably would've been cleaner to land RepairMetrics as a separate primitive with its tests first, then handle data files only, and add deletes and encryption later. But we've been reviewing it holistically, so I don't think splitting it up at this point buys us much.
| (long) (1.2 * targetManifestSizeBytes)); | ||
| } | ||
|
|
||
| /** The outcome of repairing one group of manifests that share a partition spec. */ |
There was a problem hiding this comment.
This just restates the class name, can we drop it?
| } | ||
| } | ||
|
|
||
| /** Rebuilds the file with its expected statistics. */ |
There was a problem hiding this comment.
This just restates the method name, can we drop it?
| ? new WriteDataManifests(writers, manifests, context) | ||
| : new WriteDeleteManifests(writers, manifests, context); | ||
|
|
||
| // repartition by manifest so each manifest's entries stay together, keeping the table's layout |
There was a problem hiding this comment.
The first half of this restates the line below. Can we keep just the part about keeping the table's layout?
| "left") | ||
| .withColumn("repair", functions.col("repaired_path").isNotNull()) | ||
| .drop("repaired_path") | ||
| // WriteManifests reads these columns by position, see MANIFEST_INDEX |
There was a problem hiding this comment.
I think this pointer can go, the contract is visible from the constants themselves now.
|
|
||
| private RepairMetrics() {} | ||
|
|
||
| /** Returns the name mapping of the table, or null if the table does not define one. */ |
There was a problem hiding this comment.
This just restates the signature, can we drop it?
| import org.apache.iceberg.io.OutputFile; | ||
| import org.apache.spark.broadcast.Broadcast; | ||
|
|
||
| /** Creates the writers that Spark actions use on executors to write new manifests. */ |
There was a problem hiding this comment.
This mostly restates the class name, can we drop it?
| this.maxManifestSizeBytes = maxManifestSizeBytes; | ||
| } | ||
|
|
||
| /** Returns the location of the table's metadata files, where new manifests go by default. */ |
There was a problem hiding this comment.
This just restates the method name, can we drop it?
| protected static final Splitter COMMA_SPLITTER = Splitter.on(","); | ||
| protected static final Joiner COMMA_JOINER = Joiner.on(','); | ||
|
|
||
| // read by withReusableDS; the actions that support it expose these as USE_CACHING constants |
There was a problem hiding this comment.
This describes what the code below already shows, can we drop it?
|
@anuragmantri could you take another look when you have a chance? I've addressed Amogh's latest reviews, CI is green, and it's in the 1.13.0 milestone. |
anuragmantri
left a comment
There was a problem hiding this comment.
Thanks for working through this PR @rahulsmahadev. I did a fresh pass on the changes and they look good to me. I have a minor javadoc nit.
| * Repairs incorrect metrics of manifest entries, such as record counts, file sizes and column | ||
| * level statistics, by comparing them against the underlying data and delete files. | ||
| * | ||
| * <p>Deletion vectors (delete blobs stored in Puffin files) are not verified or repaired. Metrics |
There was a problem hiding this comment.
Two small things in javadoc:
Result.repairedManifests()returns the manifests that were replaced, not the new ones. Could theResultjavadoc say that?- This sentence mentions only NaN-safe bounds. The Spark action keeps all float, double, geometry and geography bounds as stored, as its class javadoc says. Could this say the same here?
There was a problem hiding this comment.
Thanks! Updated both: the Result javadoc now says repairedManifests() returns the manifests that were replaced by repaired copies (or would be in a dry run), and the repairFileMetrics() javadoc lists the float, double, geometry and geography bounds. I also added geography to the Spark action's class javadoc, which only mentioned floating-point and geometry bounds.
Make the lineage test v3-only, drop comments that restate the code, and clarify the RepairTable javadoc: repairedManifests() returns the replaced manifests, and the bounds of float, double, geometry and geography columns are kept as stored. Co-authored-by: Isaac <no-reply@databricks.com>
| * level statistics, by comparing them against the underlying data and delete files. | ||
| * | ||
| * <p>Deletion vectors (delete blobs stored in Puffin files) are not verified or repaired. Metrics | ||
| * that cannot be reconstructed from files are kept as stored: NaN counts and the bounds of float, |
There was a problem hiding this comment.
Could we just say that this action preserves these metrics? They can sometimes be recovered from footer statistics, such as Parquet’s optional geo bounding boxes and NaN counts, so “cannot be reconstructed from files” is too broad. I’m fine with adding support for those cases in a follow-up.
There was a problem hiding this comment.
Agreed, it now just says these are preserved, and I made the Spark action's class javadoc and the comment in RepairMetrics match. Recovering them from footer statistics when they're present sounds like a good follow-up.
| * level statistics, by comparing them against the underlying data and delete files. | ||
| * | ||
| * <p>Deletion vectors (delete blobs stored in Puffin files) are not verified or repaired. Metrics | ||
| * that cannot be reconstructed from files are kept as stored: NaN counts and the bounds of float, |
There was a problem hiding this comment.
also nit: 'kept as stored' => 'preserved' is easier to parse
Say that NaN counts and float, double, geometry and geography bounds are preserved, without claiming they can never be read from file footers. Co-authored-by: Isaac <no-reply@databricks.com>
|
I missed this was still baseline against spark 4.1. @rahulsmahadev could we baseline this against 4.2 (or just open up a new PR and this one becomes a backport up to you)? we can always backport after this but generally in the project the first PR of a new feature or fix is applied to the latest supported spark version |
…k-impl Co-authored-by: Isaac <no-reply@databricks.com>
The first version of a new feature goes to the latest supported Spark version. This moves the action, its helpers and its tests to spark/v4.2 unchanged, and applies the shared changes to BaseSparkAction, RewriteManifestsSparkAction and SparkActions to the 4.2 copies. Spark 4.1 is back to main's version and gets a backport after this lands. Co-authored-by: Isaac <no-reply@databricks.com>
|
@amogh-jahagirdar Done, I moved this PR to Spark 4.2 in place and updated the title and description. The code is unchanged apart from the move, and Spark 4.1 is back to main's version. I'll open the 4.1 backport once this lands. |
|
A note on the last line of the description: Isaac is the internal agent harness I use at work, and Claude is the model. |
szehon-ho
left a comment
There was a problem hiding this comment.
Looks good overall. I left two small comments on DV entry identity and the footer-statistics wording.
| row -> | ||
| entriesByManifest | ||
| .computeIfAbsent(row.getString(MANIFEST_INDEX), path -> Maps.newHashMap()) | ||
| .put(row.getString(FILE_PATH_INDEX), row)); |
There was a problem hiding this comment.
Could we key this by path and content range, matching DeleteFileWrapper, and carry those fields through markEntriesToRepair? This probably won’t happen in practice, but the spec doesn’t prevent different DV blobs in one Puffin file from being added in separate commits. If those entries later share a manifest, repairing another entry rewrites both DVs using whichever snapshot_id survives the path-only lookup. A regression test could check that both original snapshot IDs survive.
There was a problem hiding this comment.
Good catch. Entries are now keyed by location and content range, like DeleteFileWrapper, and markEntriesToRepair carries both fields through. The new test repairKeepsSnapshotIdsOfDeletionVectorsInOnePuffinFile commits two DVs from one Puffin file separately, merges them into one manifest next to a corrupt equality delete, and checks that every entry keeps its snapshot ID. Without the fix, the second DV came back with the first one's snapshot ID.
| } | ||
|
|
||
| // Footers don't reliably carry writer-tracked stats: NaN counts, NaN-safe bounds of float | ||
| // columns and bounding boxes of geometry and geography columns. Keep the stored bounds of those |
There was a problem hiding this comment.
Could we describe this as a limitation of the current footer metrics readers? Parquet supports optional geospatial bounding-box statistics, which the current reader doesn’t recover. Perhaps:
// Preserve NaN counts and floating-point and geospatial bounds because the
// current footer metrics readers do not fully recover these statistics.There was a problem hiding this comment.
Done, I used your wording and kept the note about dropped columns.
A Puffin file can hold several deletion vectors. Keying manifest entries by location alone could give a DV the snapshot ID of another DV in the same file when their manifest was rewritten. Entries are now keyed like DeleteFileWrapper. Also reword the comment on the preserved stats. Co-authored-by: Isaac <no-reply@databricks.com>
…k-impl Co-authored-by: Isaac <no-reply@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
|
Thank you so much @rahulsmahadev and thank you @anuragmantri @szehon-ho for reviewing the changes! |
* Spark 4.1: Backport RepairTable action (#17622) The new files are copied from Spark 4.2 as is, and the changes to BaseSparkAction, RewriteManifestsSparkAction and SparkActions are the same as in the merged Spark 4.2 commit. Co-authored-by: Isaac <no-reply@databricks.com> * Spark 4.0: Backport RepairTable action (#17622) The new files are copied from Spark 4.2 as is, and the changes to BaseSparkAction, RewriteManifestsSparkAction and SparkActions are the same as in the merged Spark 4.2 commit. Co-authored-by: Isaac <no-reply@databricks.com> * Spark 3.5: Backport RepairTable action (#17622) The new files are copied from Spark 4.2 as is, and the changes to BaseSparkAction, RewriteManifestsSparkAction and SparkActions are the same as in the merged Spark 4.2 commit. Co-authored-by: Isaac <no-reply@databricks.com> --------- Co-authored-by: Isaac <no-reply@databricks.com>
Spark implementation of the
RepairTableaction whose API was added in #17399.repairFileMetrics()compares the metrics recorded in each live manifest entry against the underlying data and delete files, and rewrites only the manifests that contain an incorrect entry. By default it checks record counts and file sizes. With therepair-column-metricsoption it also checks column sizes, value and null counts, and lower and upper bounds. NaN counts and the bounds of float, double, geometry and geography columns are preserved, and deletion vectors are not verified or repaired. Rewritten entries carry through their original snapshot id and sequence numbers, so the repair does not change which delete files apply.dryRun()reports what would be repaired without committing.RewriteManifestsSparkActionnow shares its manifest writer (ManifestWriterFactory) and itsuse-cachinghandling (BaseSparkAction.withReusableDS) with the new action. Its behavior is unchanged.This targets Spark 4.2. I'll backport it to Spark 4.1 once this lands.
Tests:
TestRepairTableAction: no-op on empty tables and correct stats, record count and file size repair, column metrics (off by default), rewriting only the affected manifests, entry lineage, partition spec evolution and a dropped partition source column, equality and position deletes (including a manifest with both) and deletion vector fields, cleanup of written manifests after a failed commit or write, keeping them when the commit state is unknown, and dry run.TestRepairMetrics: per-type metrics cases that run without Spark.TestRepairTableActionEncryption: repair on an encrypted table.This pull request and its description were written by Isaac.