Skip to content

Native Iceberg scan drops rows from data files that store a narrower type than the column #7138

Open

Description

@rich7420

Describe the bug

Native Iceberg scans can drop rows when filtering on a column whose data files store a narrower type, for example INT data in a BIGINT column. iceberg-rust's page index skips all pages of such a file for range and IN predicates, and its row filter casts the literal to the file's type, so an out-of-range value becomes NULL. The post-scan filter cannot restore the rows.

Such files come from type promotion and from imports by add_files, migrate or snapshot, which do not check file types against the schema.

Steps to reproduce

CREATE TABLE t (id INT) USING iceberg;
INSERT INTO t VALUES (1), (2), (3);
ALTER TABLE t ALTER COLUMN id TYPE BIGINT;
INSERT INTO t VALUES (4), (5);
SELECT * FROM t WHERE id > 0;

Reproduced at 9e13649.

Expected behavior

All five rows, as Spark returns. Comet returns only 4 and 5. id < 3 and id IN (1, 2) return no rows.

Additional context

Imported files show the same behavior without any schema change. A file storing id as INT in a BIGINT column, or t as TINYINT in an INT column, lost rows for id > 0, id IN (1, 2), t < 200 and t IN (1, 200); only = returned the right rows. Reproduced on the #6847 branch, which pushes these predicates as main does.

Upstream: apache/iceberg-rust#3122. Its fix, apache/iceberg-rust#3123, was closed as stale and does not cover TINYINT or SMALLINT files.

#6847 pushes only = and null checks on columns whose type changed in the schema history, and on every BIGINT and DOUBLE column when a residual converts only in part. Tables with a name mapping push a residual only whole. Whole residuals on imported files, or on columns promoted before Iceberg 0.12, still need the upstream fix.

Activity

  1. added
    bugSomething isn't working
    priority:criticalData corruption, silent wrong results, security issues
    area:scanParquet scan / data reading
    and removed on Oct 26, 2026
  2. self-assigned this
    on Oct 26, 2026
  3. added this to the 1.3.0 milestone on Oct 26, 2026
  4. kazuyukitanimura commented on Oct 26, 2026

    @kazuyukitanimura
    Contributor

    Link to the Iceberg-rust issue?

  5. rich7420 commented on Oct 26, 2026

    @rich7420
    ContributorAuthor

    The upstream issue is apache/iceberg-rust#3122. Comet uses that reader, which can discard matching rows when filtering files whose physical type is narrower than the table type. The proposed fix, apache/iceberg-rust#3123, was closed as stale without being merged.

  6. removed their assignment
    on Nov 20, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    area:Icebergarea:scanParquet scan / data readingbugSomething isn't workingpriority:criticalData corruption, silent wrong results, security issues

    Type

    No type

    Projects

    No projects

      Milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions