Skip to content

fix(upsert): use bounded file match filter to avoid scan planning segfault - #11

Open
abnobdoss wants to merge 258 commits into
base/current-mainfrom
fix/upsert-file-match-filter
Open

fix(upsert): use bounded file match filter to avoid scan planning segfault#11
abnobdoss wants to merge 258 commits into
base/current-mainfrom
fix/upsert-file-match-filter

Conversation

@abnobdoss

@abnobdoss abnobdoss commented May 26, 2026

Copy link
Copy Markdown
Owner

Summary:
Use a bounded file match filter for the initial upsert target scan. The exact match filter remains in place for overwrite and insert filtering.

Why:
Composite-key upserts can build very large exact predicate trees before scanning target files. The bounded filter keeps initial scan predicate size tied to the number of join columns instead of the number of source keys.

Status:
Fork-only draft, separate from the native scan integration.

No external issue references.

@abnobdoss
abnobdoss force-pushed the fix/upsert-file-match-filter branch from 89dfcf6 to 3692011 Compare May 26, 2026 23:59
@abnobdoss abnobdoss changed the title perf(upsert): use bounded file match filter fix(upsert): use bounded file match filter to avoid scan planning segfault May 27, 2026
@abnobdoss
abnobdoss force-pushed the fix/upsert-file-match-filter branch from 3692011 to bbfc654 Compare May 27, 2026 00:07
dependabot Bot and others added 27 commits June 2, 2026 09:16
<!--
Thanks for opening a pull request!
-->

<!-- In the case this PR will resolve an issue, please replace
${GITHUB_ISSUE_ID} below with the actual Github issue id. -->
<!-- Closes #${GITHUB_ISSUE_ID} -->

# Rationale for this change
Alex is the RM for 0.12 release

## Are these changes tested?

## Are there any user-facing changes?

<!-- In the case of user-facing changes, please add the changelog label.
-->
# Rationale for this change

Follow-up of apache#3338
It would be preferable to return `None` instead of throwing
`StopIteration`.

Iceberg Java returns null if the specified version doesn't exist: 


https://github.com/apache/iceberg/blob/6a737004506176a902784ec30fc4d545c1ce6997/api/src/main/java/org/apache/iceberg/view/View.java#L60-L66

> `@return` a version, or null if the ID cannot be found


## Are these changes tested?

Yes

## Are there any user-facing changes?

No, the method isn't released yet. 

<!-- In the case of user-facing changes, please add the changelog label.
-->
# Rationale for this change

The partition statistics file doesn't use Puffin file:

* https://iceberg.apache.org/spec/#partition-statistics-file
> Statistics information for each unique partition tuple is stored as a
row in any of the data file format of the table (for example, Parquet or
ORC)

## Are these changes tested?

Yes

## Are there any user-facing changes?

No

<!-- In the case of user-facing changes, please add the changelog label.
-->
Closes apache#3469

# Rationale for this change

Decimal literals converted to `LongType` should use long bound sentinels
when the value is outside the long range. The existing conversion
returned integer sentinels, mismatching the requested target type.

This returns `LongAboveMax` and `LongBelowMin` for decimal-to-long
overflow.

### Relationship to Java

Java's `DecimalLiteral.to(...)` only handles `DECIMAL` and returns
`null` for other target types
([`Literals.java#L497-L505`](https://github.com/apache/iceberg/blob/main/api/src/main/java/org/apache/iceberg/expressions/Literals.java#L497-L505)),
so there is no direct decimal-to-long branch to mirror. This keeps
PyIceberg's long conversion consistent with its typed integer overflow
handling. Java's generic valueless overflow sentinels are a broader
semantic difference and are out of scope here.

## Are these changes tested?

Yes. New literal tests cover decimal values above and below the
`LongType` range.

## Are there any user-facing changes?

Yes. Decimal literal conversion to `LongType` now reports overflow with
the correct long sentinel type.

Co-authored-by: Abanoub Doss <abanoub.doss@gmail.com>
# Rationale for this change

Allow users to create a view with fewer parameters.

We can set an environment context (`engine-name` and `engine-version`)
in `summary` field by default
once apache#3441 is merged.

## Are these changes tested?

Yes

## Are there any user-facing changes?

No

<!-- In the case of user-facing changes, please add the changelog label.
-->
# Rationale for this change

- Supersedes apache#3489

## Are these changes tested?

Yes

## Are there any user-facing changes?

No

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
anxkhn and others added 29 commits August 31, 2026 14:40
* fix: reject decimal promotion that changes the scale

The DecimalType handler in promote() guarded scale equality with
`file_type.scale == file_type.scale`, which compares the file scale to
itself and is always true. As a result any decimal-to-decimal promotion
with a widening precision was accepted regardless of the scale.

Per the Iceberg spec a decimal may only be promoted when the scale is
unchanged and the precision widens (decimal(P, S) to decimal(P2, S) with
P2 > P), matching TypeUtil.isPromotionAllowed in the Java implementation.
The bug affected both paths that use promote(): on read a differing-scale
promotion built a reader at the wrong scale and reinterpreted the stored
unscaled integers (silent data corruption), and on write it let a
DataFrame column with a different scale pass the compatibility check.

Compare the file scale to the read scale instead. The identical tautology
in the test oracle masked the defect, so fix it too and add a decimal with
a different scale to the promotion matrix, plus explicit regression tests
for the schema and read paths.

Signed-off-by: Anas Khan <83116240+anxkhn@users.noreply.github.com>

* test: simplify decimal promotion coverage

Signed-off-by: Anas Khan <83116240+anxkhn@users.noreply.github.com>

---------

Signed-off-by: Anas Khan <83116240+anxkhn@users.noreply.github.com>
…pache#3651)

* test(datetime): cover invalid timestamp errors in nanos conversions

* test(datetime): assert full error messages
* Fix V3 manifest read projection

Read manifest entries and manifest lists with the latest supported schema so V3-only fields are retained while older manifests resolve missing fields to null.

* Fix legacy manifest test expectations

Exclude V3-only properties from V1 and V2 fastavro comparisons and clarify the latest read-version constant name.

* Fix V2 integration manifest expectation

Exclude V3-only properties from the reflected V2 manifest dictionary used by the REST integration test.

* Use V3 as default manifest read version
…3708)

Core: Preserve empty bounds in inspect.manifests
_to_partition_representation() had no handler for TimestampNanoType/
TimestamptzNanoType, so a datetime partition value passed through
unconverted instead of being converted to nanoseconds since epoch (the
same way TimestampType/TimestamptzType already convert to micros). The
resulting datetime object would reach DataFile.partition where an int
is expected, since TimestampNanoWriter calls write_int.

Fixes apache#3652
* Index snapshots by ID for snapshot_by_id lookups

snapshot_by_id did a linear scan over the snapshots list, and is called once
per manifest entry, making inspect.partitions() O(data_files x snapshots).

Memoize an id-to-snapshot index instead. A cached_property is not usable
here: model_copy carries __dict__ over, so a copy replacing the snapshots
would inherit a stale index. The index is tied to the list it was built from
and recomputed whenever snapshots is a different list.

* reducing comments to follow the standard

* Cache snapshot positions so in-place list mutation is picked up

Caching Snapshot instances went stale when the snapshots list was mutated
in place: `table.snapshots()` returns the live list, so replacing an entry
and passing the same list to `model_copy` did not change its identity and
the index was never rebuilt. Cache positions instead and validate on read,
falling back to a scan when they no longer line up.

Adds a regression test for in-place mutation.

* reducing comments to follow the standard

* changing approach to avoid errors and keeping the snapshot_by_id the same

* creating function to avoid duplicate code

* removing sentence from _get_snapshots_by_id

* removing sentence from _get_snapshots_by_id

---------

Co-authored-by: Matheus de Freitas Andrade <MatheusFreitas25@users.noreply.github.com>
Ruff was the only tool still configured outside pyproject.toml. The
standalone ruff.toml was added in apache#619 as a workaround for Dependabot
failing to parse the Poetry-era pyproject.toml; the project now uses
PEP 621 metadata with a setuptools backend, so that workaround no
longer applies.

Nothing referenced ruff.toml by path, and `ruff check --show-settings`
resolves to identical settings before and after the move.

Closes apache#3882
…he#3808)

`CythonBinaryDecoder.read_double` was declared `cpdef float`, which in
Cython is the C single-precision type, so every Avro double decoded by
the fast decoder was silently rounded to 32-bit precision. Values
outside the single-precision range collapse entirely: 1e308 becomes
inf and 5e-324 becomes 0.0.

`new_decoder` returns the Cython decoder whenever the extension is
built, so this is the default read path. It affects any double read
from a manifest, most visibly identity partition values on a
float/double column: writing a partition value of 429496729622.314
and reading the manifest back returns 429496729600.0.

The pure-Python `StreamingBinaryDecoder` was always correct, and
`read_float` is unaffected because a value decoded from four bytes is
already representable as a C float.


Claude-Session: https://claude.ai/code/session_01T2GWEoizz8ZGQbjatT8aZy

Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
* feat: Fuse partition and metrics filtering into manifest entry deserialization

* Fold entry_filter into fetch_manifest_entry
Bumps [tornado](https://github.com/tornadoweb/tornado) from 6.5.7 to 6.5.8.
- [Changelog](https://github.com/tornadoweb/tornado/blob/master/docs/releases.rst)
- [Commits](tornadoweb/tornado@v6.5.7...v6.5.8)

---
updated-dependencies:
- dependency-name: tornado
  dependency-version: 6.5.8
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
`_InclusiveMetricsEvaluationVisitor._may_contain_null` inverted the
condition from the Java implementation. It reported "may contain null"
when a null count was present (including a count of 0) and "cannot
contain null" when the count was absent.

Both directions are wrong. A file with a proven null count of 0 was
never pruned by `NotStartsWith`, and a file whose null count is unknown
could be pruned even though its nulls satisfy the predicate, silently
dropping matching rows.

Co-authored-by: Dylan Pulver <dylanpulver@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Bumps [mistune](https://github.com/lepture/mistune) from 3.3.2 to 3.3.3.
- [Release notes](https://github.com/lepture/mistune/releases)
- [Changelog](https://github.com/lepture/mistune/blob/main/docs/changes.rst)
- [Commits](lepture/mistune@v3.3.2...v3.3.3)

---
updated-dependencies:
- dependency-name: mistune
  dependency-version: 3.3.3
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* Infra: Require CI checks before auto-merge

* Infra: Group required CI contexts

* Infra: Add stable required CI gates

* Infra: Remove repeated workflow comments

* Infra: Enable GitHub Merge Queue

* Infra: Remove unsupported CodeQL queue gate

* Infra: Document required CI dependencies
Bumps [pydantic](https://github.com/pydantic/pydantic) from 2.13.4 to 2.13.5.
- [Release notes](https://github.com/pydantic/pydantic/releases)
- [Changelog](https://github.com/pydantic/pydantic/blob/v2.13.5/HISTORY.md)
- [Commits](pydantic/pydantic@v2.13.4...v2.13.5)

---
updated-dependencies:
- dependency-name: pydantic
  dependency-version: 2.13.5
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Bumps the codeql-action group with 2 updates: [github/codeql-action/init](https://github.com/github/codeql-action) and [github/codeql-action/analyze](https://github.com/github/codeql-action).


Updates `github/codeql-action/init` from 4.37.8 to 4.37.9
- [Release notes](https://github.com/github/codeql-action/releases)
- [Changelog](https://github.com/github/codeql-action/blob/main/CHANGELOG.md)
- [Commits](github/codeql-action@db488dd...cdf488f)

Updates `github/codeql-action/analyze` from 4.37.8 to 4.37.9
- [Release notes](https://github.com/github/codeql-action/releases)
- [Changelog](https://github.com/github/codeql-action/blob/main/CHANGELOG.md)
- [Commits](github/codeql-action@db488dd...cdf488f)

---
updated-dependencies:
- dependency-name: github/codeql-action/init
  dependency-version: 4.37.9
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: codeql-action
- dependency-name: github/codeql-action/analyze
  dependency-version: 4.37.9
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: codeql-action
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Bumps [prek](https://github.com/j178/prek) from 0.4.14 to 0.5.0.
- [Release notes](https://github.com/j178/prek/releases)
- [Changelog](https://github.com/j178/prek/blob/master/CHANGELOG.md)
- [Commits](j178/prek@v0.4.14...v0.5.0)

---
updated-dependencies:
- dependency-name: prek
  dependency-version: 0.5.0
  dependency-type: direct:development
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Bumps [mmh3](https://github.com/hajimes/mmh3) from 5.2.1 to 5.3.0.
- [Release notes](https://github.com/hajimes/mmh3/releases)
- [Changelog](https://github.com/hajimes/mmh3/blob/master/CHANGELOG.md)
- [Commits](hajimes/mmh3@v5.2.1...v5.3.0)

---
updated-dependencies:
- dependency-name: mmh3
  dependency-version: 5.3.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Bumps [huggingface-hub](https://github.com/huggingface/huggingface_hub) from 1.28.0 to 1.29.0.
- [Release notes](https://github.com/huggingface/huggingface_hub/releases)
- [Commits](huggingface/huggingface_hub@v1.28.0...v1.29.0)

---
updated-dependencies:
- dependency-name: huggingface-hub
  dependency-version: 1.29.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Bumps [polars](https://github.com/pola-rs/polars) from 1.43.2 to 1.44.1.
- [Release notes](https://github.com/pola-rs/polars/releases)
- [Commits](pola-rs/polars@py-1.43.2...py-1.44.1)

---
updated-dependencies:
- dependency-name: polars
  dependency-version: 1.44.1
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Bumps [ray](https://github.com/ray-project/ray) from 2.57.0 to 2.58.0.
- [Release notes](https://github.com/ray-project/ray/releases)
- [Commits](ray-project/ray@ray-2.57.0...ray-2.58.0)

---
updated-dependencies:
- dependency-name: ray
  dependency-version: 2.58.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Bumps [google-cloud-bigquery](https://github.com/googleapis/python-bigquery) from 3.43.0 to 3.44.0.
- [Release notes](https://github.com/googleapis/python-bigquery/releases)
- [Changelog](https://github.com/googleapis/python-bigquery/blob/main/CHANGELOG.md)
- [Commits](https://github.com/googleapis/python-bigquery/commits)

---
updated-dependencies:
- dependency-name: google-cloud-bigquery
  dependency-version: 3.44.0
  dependency-type: direct:development
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Bumps [click](https://github.com/pallets/click) from 8.4.2 to 8.5.0.
- [Release notes](https://github.com/pallets/click/releases)
- [Changelog](https://github.com/pallets/click/blob/main/CHANGES.md)
- [Commits](pallets/click@8.4.2...8.5.0)

---
updated-dependencies:
- dependency-name: click
  dependency-version: 8.5.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* Build: Bump google-auth from 2.56.3 to 2.57.0

Bumps [google-auth](https://github.com/googleapis/google-cloud-python) from 2.56.3 to 2.57.0.
- [Release notes](https://github.com/googleapis/google-cloud-python/releases)
- [Changelog](https://github.com/googleapis/google-cloud-python/blob/main/packages/google-cloud-documentai/CHANGELOG.md)
- [Commits](googleapis/google-cloud-python@google-auth-v2.56.3...google-auth-v2.57.0)

---
updated-dependencies:
- dependency-name: google-auth
  dependency-version: 2.57.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

* Build: Bump grpcio to 1.83.1 for post-quantum crypto support

google-auth 2.57.0 emits a FutureWarning on import when grpcio <
1.83.0, since it lacks post-quantum cryptography support that
google-auth will require starting October 2026. This broke test
collection for tests/catalog/test_bigquery_metastore.py under
strict warning filters.

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
)

`_load_yaml` passed `strictyaml.load(...).data` straight to
`_lowercase_dictionary_keys`. For an empty or comment-only document that
value is a `str`, not a mapping, so the call raised
`AttributeError: 'str' object has no attribute 'items'`.

`Config()` runs at import time, so commenting out the file made
`import pyiceberg.catalog` fail with an error naming neither YAML nor the
file. Return `None` instead, which the annotated return type already allows
and which the caller already handles as "keep looking".

Co-authored-by: Claude Code <noreply@anthropic.com>
Bumps [cython](https://github.com/cython/cython) from 3.2.9 to 3.3.0.
- [Release notes](https://github.com/cython/cython/releases)
- [Changelog](https://github.com/cython/cython/blob/master/CHANGES.rst)
- [Commits](cython/cython@3.2.9...3.3.0)

---
updated-dependencies:
- dependency-name: cython
  dependency-version: 3.3.0
  dependency-type: direct:development
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
@abnobdoss
abnobdoss force-pushed the fix/upsert-file-match-filter branch from bbfc654 to b44b895 Compare September 7, 2026 16:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.