fix(upsert): use bounded file match filter to avoid scan planning segfault - #11
Open
abnobdoss wants to merge 258 commits into
Open
fix(upsert): use bounded file match filter to avoid scan planning segfault#11abnobdoss wants to merge 258 commits into
abnobdoss wants to merge 258 commits into
Conversation
abnobdoss
force-pushed
the
fix/upsert-file-match-filter
branch
from
May 26, 2026 23:59
89dfcf6 to
3692011
Compare
abnobdoss
force-pushed
the
fix/upsert-file-match-filter
branch
from
May 27, 2026 00:07
3692011 to
bbfc654
Compare
<!--
Thanks for opening a pull request!
-->
<!-- In the case this PR will resolve an issue, please replace
${GITHUB_ISSUE_ID} below with the actual Github issue id. -->
<!-- Closes #${GITHUB_ISSUE_ID} -->
# Rationale for this change
Alex is the RM for 0.12 release
## Are these changes tested?
## Are there any user-facing changes?
<!-- In the case of user-facing changes, please add the changelog label.
-->
# Rationale for this change Follow-up of apache#3338 It would be preferable to return `None` instead of throwing `StopIteration`. Iceberg Java returns null if the specified version doesn't exist: https://github.com/apache/iceberg/blob/6a737004506176a902784ec30fc4d545c1ce6997/api/src/main/java/org/apache/iceberg/view/View.java#L60-L66 > `@return` a version, or null if the ID cannot be found ## Are these changes tested? Yes ## Are there any user-facing changes? No, the method isn't released yet. <!-- In the case of user-facing changes, please add the changelog label. -->
# Rationale for this change The partition statistics file doesn't use Puffin file: * https://iceberg.apache.org/spec/#partition-statistics-file > Statistics information for each unique partition tuple is stored as a row in any of the data file format of the table (for example, Parquet or ORC) ## Are these changes tested? Yes ## Are there any user-facing changes? No <!-- In the case of user-facing changes, please add the changelog label. -->
Closes apache#3469 # Rationale for this change Decimal literals converted to `LongType` should use long bound sentinels when the value is outside the long range. The existing conversion returned integer sentinels, mismatching the requested target type. This returns `LongAboveMax` and `LongBelowMin` for decimal-to-long overflow. ### Relationship to Java Java's `DecimalLiteral.to(...)` only handles `DECIMAL` and returns `null` for other target types ([`Literals.java#L497-L505`](https://github.com/apache/iceberg/blob/main/api/src/main/java/org/apache/iceberg/expressions/Literals.java#L497-L505)), so there is no direct decimal-to-long branch to mirror. This keeps PyIceberg's long conversion consistent with its typed integer overflow handling. Java's generic valueless overflow sentinels are a broader semantic difference and are out of scope here. ## Are these changes tested? Yes. New literal tests cover decimal values above and below the `LongType` range. ## Are there any user-facing changes? Yes. Decimal literal conversion to `LongType` now reports overflow with the correct long sentinel type. Co-authored-by: Abanoub Doss <abanoub.doss@gmail.com>
# Rationale for this change Allow users to create a view with fewer parameters. We can set an environment context (`engine-name` and `engine-version`) in `summary` field by default once apache#3441 is merged. ## Are these changes tested? Yes ## Are there any user-facing changes? No <!-- In the case of user-facing changes, please add the changelog label. -->
# Rationale for this change - Supersedes apache#3489 ## Are these changes tested? Yes ## Are there any user-facing changes? No Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* fix: reject decimal promotion that changes the scale The DecimalType handler in promote() guarded scale equality with `file_type.scale == file_type.scale`, which compares the file scale to itself and is always true. As a result any decimal-to-decimal promotion with a widening precision was accepted regardless of the scale. Per the Iceberg spec a decimal may only be promoted when the scale is unchanged and the precision widens (decimal(P, S) to decimal(P2, S) with P2 > P), matching TypeUtil.isPromotionAllowed in the Java implementation. The bug affected both paths that use promote(): on read a differing-scale promotion built a reader at the wrong scale and reinterpreted the stored unscaled integers (silent data corruption), and on write it let a DataFrame column with a different scale pass the compatibility check. Compare the file scale to the read scale instead. The identical tautology in the test oracle masked the defect, so fix it too and add a decimal with a different scale to the promotion matrix, plus explicit regression tests for the schema and read paths. Signed-off-by: Anas Khan <83116240+anxkhn@users.noreply.github.com> * test: simplify decimal promotion coverage Signed-off-by: Anas Khan <83116240+anxkhn@users.noreply.github.com> --------- Signed-off-by: Anas Khan <83116240+anxkhn@users.noreply.github.com>
…pache#3651) * test(datetime): cover invalid timestamp errors in nanos conversions * test(datetime): assert full error messages
* Fix V3 manifest read projection Read manifest entries and manifest lists with the latest supported schema so V3-only fields are retained while older manifests resolve missing fields to null. * Fix legacy manifest test expectations Exclude V3-only properties from V1 and V2 fastavro comparisons and clarify the latest read-version constant name. * Fix V2 integration manifest expectation Exclude V3-only properties from the reflected V2 manifest dictionary used by the REST integration test. * Use V3 as default manifest read version
…3708) Core: Preserve empty bounds in inspect.manifests
_to_partition_representation() had no handler for TimestampNanoType/ TimestamptzNanoType, so a datetime partition value passed through unconverted instead of being converted to nanoseconds since epoch (the same way TimestampType/TimestamptzType already convert to micros). The resulting datetime object would reach DataFile.partition where an int is expected, since TimestampNanoWriter calls write_int. Fixes apache#3652
* Index snapshots by ID for snapshot_by_id lookups snapshot_by_id did a linear scan over the snapshots list, and is called once per manifest entry, making inspect.partitions() O(data_files x snapshots). Memoize an id-to-snapshot index instead. A cached_property is not usable here: model_copy carries __dict__ over, so a copy replacing the snapshots would inherit a stale index. The index is tied to the list it was built from and recomputed whenever snapshots is a different list. * reducing comments to follow the standard * Cache snapshot positions so in-place list mutation is picked up Caching Snapshot instances went stale when the snapshots list was mutated in place: `table.snapshots()` returns the live list, so replacing an entry and passing the same list to `model_copy` did not change its identity and the index was never rebuilt. Cache positions instead and validate on read, falling back to a scan when they no longer line up. Adds a regression test for in-place mutation. * reducing comments to follow the standard * changing approach to avoid errors and keeping the snapshot_by_id the same * creating function to avoid duplicate code * removing sentence from _get_snapshots_by_id * removing sentence from _get_snapshots_by_id --------- Co-authored-by: Matheus de Freitas Andrade <MatheusFreitas25@users.noreply.github.com>
Ruff was the only tool still configured outside pyproject.toml. The standalone ruff.toml was added in apache#619 as a workaround for Dependabot failing to parse the Poetry-era pyproject.toml; the project now uses PEP 621 metadata with a setuptools backend, so that workaround no longer applies. Nothing referenced ruff.toml by path, and `ruff check --show-settings` resolves to identical settings before and after the move. Closes apache#3882
…he#3808) `CythonBinaryDecoder.read_double` was declared `cpdef float`, which in Cython is the C single-precision type, so every Avro double decoded by the fast decoder was silently rounded to 32-bit precision. Values outside the single-precision range collapse entirely: 1e308 becomes inf and 5e-324 becomes 0.0. `new_decoder` returns the Cython decoder whenever the extension is built, so this is the default read path. It affects any double read from a manifest, most visibly identity partition values on a float/double column: writing a partition value of 429496729622.314 and reading the manifest back returns 429496729600.0. The pure-Python `StreamingBinaryDecoder` was always correct, and `read_float` is unaffected because a value decoded from four bytes is already representable as a C float. Claude-Session: https://claude.ai/code/session_01T2GWEoizz8ZGQbjatT8aZy Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
* feat: Fuse partition and metrics filtering into manifest entry deserialization * Fold entry_filter into fetch_manifest_entry
Bumps [tornado](https://github.com/tornadoweb/tornado) from 6.5.7 to 6.5.8. - [Changelog](https://github.com/tornadoweb/tornado/blob/master/docs/releases.rst) - [Commits](tornadoweb/tornado@v6.5.7...v6.5.8) --- updated-dependencies: - dependency-name: tornado dependency-version: 6.5.8 dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
`_InclusiveMetricsEvaluationVisitor._may_contain_null` inverted the condition from the Java implementation. It reported "may contain null" when a null count was present (including a count of 0) and "cannot contain null" when the count was absent. Both directions are wrong. A file with a proven null count of 0 was never pruned by `NotStartsWith`, and a file whose null count is unknown could be pruned even though its nulls satisfy the predicate, silently dropping matching rows. Co-authored-by: Dylan Pulver <dylanpulver@users.noreply.github.com> Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Bumps [mistune](https://github.com/lepture/mistune) from 3.3.2 to 3.3.3. - [Release notes](https://github.com/lepture/mistune/releases) - [Changelog](https://github.com/lepture/mistune/blob/main/docs/changes.rst) - [Commits](lepture/mistune@v3.3.2...v3.3.3) --- updated-dependencies: - dependency-name: mistune dependency-version: 3.3.3 dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
add view catalog tests
* Infra: Require CI checks before auto-merge * Infra: Group required CI contexts * Infra: Add stable required CI gates * Infra: Remove repeated workflow comments * Infra: Enable GitHub Merge Queue * Infra: Remove unsupported CodeQL queue gate * Infra: Document required CI dependencies
Bumps [pydantic](https://github.com/pydantic/pydantic) from 2.13.4 to 2.13.5. - [Release notes](https://github.com/pydantic/pydantic/releases) - [Changelog](https://github.com/pydantic/pydantic/blob/v2.13.5/HISTORY.md) - [Commits](pydantic/pydantic@v2.13.4...v2.13.5) --- updated-dependencies: - dependency-name: pydantic dependency-version: 2.13.5 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Bumps the codeql-action group with 2 updates: [github/codeql-action/init](https://github.com/github/codeql-action) and [github/codeql-action/analyze](https://github.com/github/codeql-action). Updates `github/codeql-action/init` from 4.37.8 to 4.37.9 - [Release notes](https://github.com/github/codeql-action/releases) - [Changelog](https://github.com/github/codeql-action/blob/main/CHANGELOG.md) - [Commits](github/codeql-action@db488dd...cdf488f) Updates `github/codeql-action/analyze` from 4.37.8 to 4.37.9 - [Release notes](https://github.com/github/codeql-action/releases) - [Changelog](https://github.com/github/codeql-action/blob/main/CHANGELOG.md) - [Commits](github/codeql-action@db488dd...cdf488f) --- updated-dependencies: - dependency-name: github/codeql-action/init dependency-version: 4.37.9 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: codeql-action - dependency-name: github/codeql-action/analyze dependency-version: 4.37.9 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: codeql-action ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Bumps [prek](https://github.com/j178/prek) from 0.4.14 to 0.5.0. - [Release notes](https://github.com/j178/prek/releases) - [Changelog](https://github.com/j178/prek/blob/master/CHANGELOG.md) - [Commits](j178/prek@v0.4.14...v0.5.0) --- updated-dependencies: - dependency-name: prek dependency-version: 0.5.0 dependency-type: direct:development update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Bumps [mmh3](https://github.com/hajimes/mmh3) from 5.2.1 to 5.3.0. - [Release notes](https://github.com/hajimes/mmh3/releases) - [Changelog](https://github.com/hajimes/mmh3/blob/master/CHANGELOG.md) - [Commits](hajimes/mmh3@v5.2.1...v5.3.0) --- updated-dependencies: - dependency-name: mmh3 dependency-version: 5.3.0 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Bumps [huggingface-hub](https://github.com/huggingface/huggingface_hub) from 1.28.0 to 1.29.0. - [Release notes](https://github.com/huggingface/huggingface_hub/releases) - [Commits](huggingface/huggingface_hub@v1.28.0...v1.29.0) --- updated-dependencies: - dependency-name: huggingface-hub dependency-version: 1.29.0 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Bumps [polars](https://github.com/pola-rs/polars) from 1.43.2 to 1.44.1. - [Release notes](https://github.com/pola-rs/polars/releases) - [Commits](pola-rs/polars@py-1.43.2...py-1.44.1) --- updated-dependencies: - dependency-name: polars dependency-version: 1.44.1 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Bumps [ray](https://github.com/ray-project/ray) from 2.57.0 to 2.58.0. - [Release notes](https://github.com/ray-project/ray/releases) - [Commits](ray-project/ray@ray-2.57.0...ray-2.58.0) --- updated-dependencies: - dependency-name: ray dependency-version: 2.58.0 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Bumps [google-cloud-bigquery](https://github.com/googleapis/python-bigquery) from 3.43.0 to 3.44.0. - [Release notes](https://github.com/googleapis/python-bigquery/releases) - [Changelog](https://github.com/googleapis/python-bigquery/blob/main/CHANGELOG.md) - [Commits](https://github.com/googleapis/python-bigquery/commits) --- updated-dependencies: - dependency-name: google-cloud-bigquery dependency-version: 3.44.0 dependency-type: direct:development update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Bumps [click](https://github.com/pallets/click) from 8.4.2 to 8.5.0. - [Release notes](https://github.com/pallets/click/releases) - [Changelog](https://github.com/pallets/click/blob/main/CHANGES.md) - [Commits](pallets/click@8.4.2...8.5.0) --- updated-dependencies: - dependency-name: click dependency-version: 8.5.0 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* Build: Bump google-auth from 2.56.3 to 2.57.0 Bumps [google-auth](https://github.com/googleapis/google-cloud-python) from 2.56.3 to 2.57.0. - [Release notes](https://github.com/googleapis/google-cloud-python/releases) - [Changelog](https://github.com/googleapis/google-cloud-python/blob/main/packages/google-cloud-documentai/CHANGELOG.md) - [Commits](googleapis/google-cloud-python@google-auth-v2.56.3...google-auth-v2.57.0) --- updated-dependencies: - dependency-name: google-auth dependency-version: 2.57.0 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> * Build: Bump grpcio to 1.83.1 for post-quantum crypto support google-auth 2.57.0 emits a FutureWarning on import when grpcio < 1.83.0, since it lacks post-quantum cryptography support that google-auth will require starting October 2026. This broke test collection for tests/catalog/test_bigquery_metastore.py under strict warning filters. --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
) `_load_yaml` passed `strictyaml.load(...).data` straight to `_lowercase_dictionary_keys`. For an empty or comment-only document that value is a `str`, not a mapping, so the call raised `AttributeError: 'str' object has no attribute 'items'`. `Config()` runs at import time, so commenting out the file made `import pyiceberg.catalog` fail with an error naming neither YAML nor the file. Return `None` instead, which the annotated return type already allows and which the caller already handles as "keep looking". Co-authored-by: Claude Code <noreply@anthropic.com>
Bumps [cython](https://github.com/cython/cython) from 3.2.9 to 3.3.0. - [Release notes](https://github.com/cython/cython/releases) - [Changelog](https://github.com/cython/cython/blob/master/CHANGES.rst) - [Commits](cython/cython@3.2.9...3.3.0) --- updated-dependencies: - dependency-name: cython dependency-version: 3.3.0 dependency-type: direct:development update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
abnobdoss
force-pushed
the
fix/upsert-file-match-filter
branch
from
September 7, 2026 16:45
bbfc654 to
b44b895
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary:
Use a bounded file match filter for the initial upsert target scan. The exact match filter remains in place for overwrite and insert filtering.
Why:
Composite-key upserts can build very large exact predicate trees before scanning target files. The bounded filter keeps initial scan predicate size tied to the number of join columns instead of the number of source keys.
Status:
Fork-only draft, separate from the native scan integration.
No external issue references.