Antalya 26.8: Iceberg deletion vector support (attempt #2)

[zvonand]

Changelog category (leave one):

• New Feature

Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):

Iceberg deletion vectors support (#2183 by @ianton-ru).

CI/CD Options

Exclude tests:

• Fast test
• Integration Tests
• Stateless tests
• Stateful tests
• Performance tests
• All with ASAN
• All with TSAN
• All with MSAN
• All with UBSAN
• All with Coverage
• All with Aarch64
• All Regression
• Disable CI Cache

Regression jobs to run:

• Fast suites (mostly <1h)
• Aggregate Functions (2h)
• Alter (1.5h)
• Benchmark (30m)
• ClickHouse Keeper (1h)
• Iceberg (2h)
• LDAP (1h)
• Parquet (1.5h)
• RBAC (1.5h)
• SSL Server (1h)
• S3 (2h)
• S3 Export (2h)
• Swarms (30m)
• Tiered Storage (2h)

Cherry-picked from #2183.

---

Documentation entry for user-facing changes

Goal

Add read support for Iceberg v3 Puffin deletion vectors (deletion-vector-v1) so ClickHouse applies DV bitmaps when reading Iceberg tables (local / object storage / cluster), without writing DVs.
Also expose SQL input formats Puffin / PuffinMetadata for inspecting Puffin files, and a process-global Puffin files cache for parsed DV bitmaps.

---

High-level architecture

Manifest (position deletes, content=2)
        │
        ▼
IcebergIterator ──loadDeletionVector──► Puffin footer bind + blob read
        │                                      │
        │                                      ▼
        │                              PuffinFilesCache (optional)
        ▼
IcebergDataObjectInfo.excluded_rows  (roaring bitmap of deleted positions)
        │
        ▼
StorageObjectStorageSource / DeletionVectorTransform
        │
        ├── need_only_count → cardinality via roaring rank (no Filter materialization)
        └── full read → exclude rows (DV before equality deletes)

Shared Puffin parsing / DV deserialize lives under:

Component	Role
PuffinFile	Footer parse (seekable), blob metadata, DV footer bind
PuffinDeletionVectorReader	Envelope peek, CRC, roaring deserialize, size ceilings
PuffinFilesCache	Context-global cache of cloned exclusion bitmaps
IcebergDeletionVector	Iceberg-specific load + validation vs data-file record_count
PuffinBlockInputFormat	SQL Puffin / PuffinMetadata
Iceberg path uses seekable object-storage reads. SQL formats also support a non-seekable fallback (pipes / input_format_allow_seeks = 0).

---

Feature behavior (what users get)

1. Iceberg reads honor live Puffin DVs attached as position-delete manifest entries (content = 2 / deletion vectors).
2. Non-Parquet data files with DVs are rejected (fail closed).
3. Equality deletes still work; DVs are applied before equality filters so file-local row numbers stay correct.
4. Trivial / snapshot COUNT shortcuts fail closed when any live deletes (equality, position files, or DVs) are present — do not trust poisoned snapshot summaries or naive data − deletes arithmetic.
5. Cluster / parallel read fails closed if the cluster protocol cannot carry excluded_rows or delete metadata (no silent drop of deletes).
6. SYSTEM DROP PUFFIN FILES CACHE (spaced form; underscore alias accepted) clears the cache; gated by access control.
7. Settings: use_puffin_files_cache and related server/cache size settings (see Settings / docs).

---

Safety / fail-closed decisions (intentional)

Reviewers should treat these as product decisions, not accidental omissions:

• Absolute ceilings (not FormatSettings knobs): footer payload (16 MiB), DV blob size (2 GiB, Iceberg-aligned), materialized positions, non-seekable buffer size.
• Envelope peek before allocating full DV blob; CRC after bounded read.
• Footer bind: unique blob at (content_offset, content_size) matching referenced_data_file + cardinality.
• Positions must be < data_file.record_count.
• Cache keys include storage identity, path, etag, slice, referenced data file, expected cardinality, and data-file row count.
• Cache returns clones of bitmaps so callers cannot mutate shared cache state.
• Weak / empty etags skip the cache (isEtagUsableAsCacheKey).
• COUNT / need_only_count: prefer roaring cardinality / rank; avoid building a full Filter over all file rows when only a count is needed; skip file-level count cache when excluded_rows is present.
  Explicitly out of scope / deferred (workspace rule): Poco JSON Int64 wrap of 2^63 / 2^63+1 — do not treat as a defect to fix in this PR.

---

Tests (where to look)

Unit / gtest

• Puffin: envelope, bounds, cardinality, footer bind, referenced_data_file, non-seekable buffer limit
• Cache: key (incl. storage identity), clone, weight, metrics (clear-during-load, waiter, hit-after-clear)
• Iceberg: count shortcuts, DV positions, position-delete kind presence, parquet row-deletes guard
• Parquet: need_only_count with buckets, row-group global offsets
• CacheBase / LRU: getOrSetWithOutcome*

Stateless

• Puffin happy path, allow_seeks=0, stdin pipe, error fixtures under tests/queries/0_stateless/data_puffin/
• SYSTEM DROP PUFFIN FILES CACHE parsing / privileges
  Integration
• tests/integration/test_storage_iceberg_with_spark/test_deletion_vectors.py
• MinIO fixtures under data_minio/ (dv_puffin_*), generator generate_iceberg_dv_fixture.py

---

Docs touched

• Iceberg table engine / table function
• Puffin / PuffinMetadata formats
• SYSTEM DROP PUFFIN FILES CACHE

[github-actions]

Workflow [PR], commit [4827d9c]