Skip to content

GCP: Throw NotFoundException from GCSInputFile.getLength for missing blobs - #19048

Open
linzch3 wants to merge 3 commits into
apache:mainfrom
linzch3:gcs-input-file-missing-blob
Open

linzch3 wants to merge 3 commits into
apache:mainfrom
linzch3:gcs-input-file-missing-blob

Conversation

@linzch3

@linzch3 linzch3 commented Nov 20, 2026

Copy link
Copy Markdown

Closes #19047

Background

When a manifest list (or any other file) is deleted from GCS but still referenced by metadata, reading it fails with an NPE instead of a NotFoundException:

java.lang.NullPointerException: Cannot invoke "com.google.cloud.storage.Blob.getSize()" because the return value of "org.apache.iceberg.gcp.gcs.GCSInputFile.getBlob()" is null
	at org.apache.iceberg.gcp.gcs.GCSInputFile.getLength(GCSInputFile.java:59)
	at org.apache.iceberg.avro.AvroIterable.newFileReader(AvroIterable.java:102)

Root cause

Storage.get(BlobId) returns null for a missing object (that's also what BaseGCSFile.exists() relies on). GCSInputFile.getLength() called getBlob().getSize() without a null check. #15734 already maps 404s to NotFoundException in the input streams, but getLength() runs before any stream is opened (e.g. AvroIterable.newFileReader), so the NPE comes first.

Fix

If the blob is missing, getLength() throws NotFoundException("Location does not exist: %s", location()), the same message GCSExceptionUtil and S3InputStream use. Nothing else changes: a known length still skips the lookup.

Tests

  • New TestGcsInputFile#getLengthMissingBlob: storage.get returns null, exists() is false and getLength() throws NotFoundException with the location.
  • ./gradlew :iceberg-gcp:test and :iceberg-gcp:spotlessCheck pass locally (JDK 21). I didn't run the GCS integration tests (gcp/src/integration), they need the emulator.

Limitations

  • Only the getLength() path is covered here. Other callers of getBlob() (exists()) already handle null.
  • Core: Improve error message for missing ManifestList #16935 tried to improve the error for a missing manifest list at the core level and was closed as stale; this one fixes it where the NPE happens instead.

AI disclosure: I used an AI assistant (Claude Opus 5.5) to help write this change and the test. I reviewed the code and ran the tests myself.

Happy to adjust anything, e.g. the message wording or moving the check into BaseGCSFile, lmk.

@github-actions github-actions Bot added the GCP label Nov 20, 2026
@Override
public long getLength() {
if (blobSize == null) {
this.blobSize = getBlob().getSize();

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

GCSInputStream has the same unchecked storage.get(blobId).getSize(). Worth a follow-up?

blobSize = storage.get(blobId).getSize();

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yeah GCSInputStream has the same problem, I'll open a follow-up PR for it after this one.

@rishvin rishvin Nov 20, 2026 •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for working on this.
Wondering, that fix in GCSInputStream looks exactly the same, can it be fixed in the PR instead?

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thx @rishvin! I actually already put the GCSInputStream.readTail fix (with a test) in #19063 as the follow-up ebyhr suggested. happy to fold it into this PR and close #19063 if u prefer having both in one place, lmk.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi @linzch3, yes it will be good to have both of them in one place.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thx @rishvin, done! folded the GCSInputStream.readTail fix + test from #19063 into this PR, so both fixes live here now. closing #19063 in favor of this one.

}

@Test
public void getLengthMissingBlob() {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

https://iceberg.apache.org/contribute/#conventions-and-recommendations

  • Omit the public modifier for test classes, test methods, and lifecycle methods for newly added tests.
Suggested change
public void getLengthMissingBlob() {
void getLengthMissingBlob() {

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thx! applied the suggestion.

Co-authored-by: Yuya Ebihara <6237050+ebyhr@users.noreply.github.com>
@rishvin

rishvin commented Nov 21, 2026

Copy link
Copy Markdown

Thanks @linzch3! LGTM.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

GCSInputFile.getLength throws NullPointerException for missing blobs

3 participants