Skip to content

Antalya-26:6 - iceberg v3: rewriteFiles - Added logic to merge small files into bins - #2420

Draft
subkanthi wants to merge 3 commits into
antalya-26.6from
iceberg_rewrite_files_compaction
Draft

subkanthi wants to merge 3 commits into
antalya-26.6from
iceberg_rewrite_files_compaction

Conversation

@subkanthi

@subkanthi subkanthi commented Sep 23, 2026

Copy link
Copy Markdown
Collaborator

Changelog category (leave one):

  • New Feature

Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):

Implement bin-packingReWriteFiles operation that Identifies small data files below a target size threshold(defined in Settings), groups them and merges to a large file, creates a snapshot that marks the old ones as DELETED and the new larger merge file as ADDED.

Leave all other files, manifests, and history untouched

CI/CD Options

Exclude tests:

  • Fast test
  • Integration Tests
  • Stateless tests
  • Stateful tests
  • Unit tests
  • Performance tests
  • Aarch64 tests
  • All with ASAN
  • All with TSAN
  • All with MSAN
  • All with UBSAN
  • All with Coverage
  • All Regression
  • Disable CI Cache

Regression jobs to run:

  • Fast suites (mostly <1h)
  • Aggregate Functions (2h)
  • Alter (1.5h)
  • Benchmark (30m)
  • CAS (content-addressed storage; Antalya only)
  • ClickHouse Keeper (1h)
  • Iceberg (2h)
  • LDAP (1h)
  • OAuth (5m)
  • Parquet (1.5h)
  • RBAC (1.5h)
  • SSL Server (1h)
  • S3 (2h)
  • S3 Export (2h)
  • Swarms (30m)
  • Tiered Storage (2h)

@subkanthi subkanthi changed the title Added logic to merge small files into bins Antalya-26:6 - iceberg v3: rewriteFiles - Added logic to merge small files into bins Sep 23, 2026
@github-actions

github-actions Bot commented Sep 23, 2026

Copy link
Copy Markdown

Workflow [PR], commit [43c15c4]

@subkanthi subkanthi mentioned this pull request Sep 23, 2026
15 tasks

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants