Skip to content

Core: Avoid file lookups when sizing static metadata tasks - #18467

Open
eholmer-pltr wants to merge 2 commits into
apache:mainfrom
eholmer-pltr:eholmer/static-metadata-task-size
Open

eholmer-pltr wants to merge 2 commits into
apache:mainfrom
eholmer-pltr:eholmer/static-metadata-task-size

Conversation

@eholmer-pltr

@eholmer-pltr eholmer-pltr commented Oct 12, 2026 •

Copy link
Copy Markdown

Static metadata-table tasks currently stat their backing metadata files solely to populate a synthetic file size, even though their rows are already materialized. Pass the location directly and estimate task size from the full row schema and row count, preserving file identity and useful planning statistics without that lookup. Nonempty tasks retain a positive size so their inherited row-count estimates remain correct. Supersedes #16833 and closes #18465.


AI Disclosure

Estimate static task sizes from their in-memory rows while preserving location identity and correct row-count estimates.

Generated-by: Codex
@github-actions github-actions Bot added the core label Oct 12, 2026
@github-actions

Copy link
Copy Markdown

This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@iceberg.apache.org list. Thank you for your contributions.

@github-actions github-actions Bot added the stale label Nov 12, 2026
@eholmer-pltr

eholmer-pltr commented Nov 12, 2026 •

Copy link
Copy Markdown
Author

@danielcweeks / @dhruvarya-db based on the proposal here #17642

and in particular https://github.com/apache/iceberg/pull/17642/changes#diff-ffc6f59bffb61af69bc4ef19c7b28c9fefe015b2716b35db1deb6a5a884ca4feR2019-R2020 this seems like an obvious (but trivial) blocker

@github-actions github-actions Bot removed the stale label Nov 13, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Core: Avoid unnecessary metadata-file HEAD requests when planning static metadata tables

1 participant