Skip to content

Project Antalya Roadmap 2026 #1359

Open

Description

@svb-alt

Here's the current Project Antalya roadmap for 2026. This year the focus is on direct writes to Iceberg, improved usability of the swarms and Hybrid table engine, and interoperability with regard to cloud providers and catalog types. All features are open source--there are no hold-backs.

Please suggest additional features and ideas in the comments to this issue. We also welcome contributions.

Swarms

Swarm feature tracking #2447

  • Support distributed joins in swarm queries - #1527
  • Query task distribution
    • Distribute query tasks by row group - #1237
    • Enable bucket splitting across cluster functions - #90799 (upstream)
  • Distribute INSERT … SELECT between remote-storage tables across swarm workers - #782
  • Distribute compaction across swarm workers
  • Remote swarm initiators
    • Support standalone/Kubernetes swarms with remote initiators - #756; #1289
    • Authenticate remote initiators and select the query cluster - #1576
    • Keep swarm configuration on the remote initiator - #1608
    • Reload cluster discovery settings without restarting - #2197
  • Launch swarm workers as on-demand processes - #2319
  • Drain swarm workers before scale-down with SYSTEM STOP SWARM MODE - #1014; #759
  • Choose local or swarm execution automatically based on query size - #2808; #2006
  • Discover available swarm tables and dispatch queries automatically - #2036

Tiered storage and exports

  • Export MergeTree data to Parquet and Iceberg
    • Export MergeTree parts/partitions to Parquet - replicated tables #1124; non-replicated tables #2290 (#1908)
    • Accept table functions as export destinations - #1320
    • Export materialized and ALIAS columns - #1324
    • Configure export filename patterns - #1490
    • Export MergeTree parts/partitions directly to Iceberg - #1618
    • Export all partitions in one operation - #1741
    • Split Parquet output across multiple files - #489
    • Retain partition-export history with Iceberg metadata and data file paths - #1917, #1832; #1824, #1330
  • Control export data selection and schema mapping
    • Support type conversion, schema matching, and compatible partition expressions during export - #1779; #1110, #1717
    • Allow extra source columns with positional matching - #2111
    • Match columns by name and allow optional source columns - #2220
    • Allow compatible partition expressions without repartitioning - #2074; #2352
    • Filter rows during export - #1849
  • Export expired partitions to Iceberg using TTL rules - #2442; #1793

Hybrid table engine

Hybrid tracking #2426

  • Hybrid reads and watermark management
    • Combine MergeTree and object-storage data using configurable watermarks - #1071
    • Prune Hybrid segments - #1788
    • Expose watermarks through system.hybrid_watermarks - #1789
    • Support ALIAS columns - #1272
    • Manage watermarks manually - #1659
    • Advance Hybrid watermarks automatically

Catalogs and cloud access

  • Integrate Iceberg REST catalogs - #71542 (upstream)
  • Map catalog namespaces as databases - #1337
  • Integrate the Glue catalog - reads #77257 (upstream); writes #84136 (upstream); #900
  • Integrate the BigLake catalog - #97104 (upstream)
  • Read GCS-backed tables through DataLakeCatalog - #93866 (upstream)

Iceberg tables and schema

  • Iceberg table operations
    • Create, drop, and truncate Iceberg tables - CREATE/DROP #2305, TRUNCATE #1529
    • Iceberg partition deletion
      • Support DROP PARTITION - #2533; #1046
      • Drop partitions across catalogs, mixed manifests, and evolved partition specs - #2613
    • Register existing Parquet files in an Iceberg table without rewriting them - #2753
    • Delete Iceberg data using TTL rules.
  • Read Iceberg tables with files outside the table location or on another storage backend - #1092
  • Iceberg schema evolution and types
    • Read evolved Iceberg schemas and add, drop, or modify columns - reads #69445 (upstream); writes #85769 (upstream); #816
    • Evolve complex-type schemas - #73714 (upstream)
    • Apply schema evolution through catalogs with concurrency checks - #106102 (upstream)
    • Read and write the Iceberg time type - #1761
    • Use declared column defaults in Iceberg v3 schema evolution (initial-default / write-default)
  • Back up and restore Iceberg tables

Iceberg v3

Feature tracking #2390

  • Create format-version 3 tables - #104849 (upstream)
  • Read Iceberg v3 schemas and data types
    • Read nanosecond timestamp types - #97132 (upstream)
    • Read experimental geometry/geography types - #103113 (upstream)
    • Read and write the unknown type - #2363
    • Read the Iceberg variant type - #2677
    • Read unshredded Parquet variants as Dynamic/JSON - #2666
    • Read tables with multi-column transform metadata - #2318
  • Iceberg deletion-vector reads
    • Read Iceberg v3 deletion vectors - #2183; #2002
    • Read Puffin files and deletion-vector metadata - #2179
  • Write deletion vectors for v3 mutations
  • Read and write row-lineage metadata
    • Expose row-lineage fields through system.iceberg_files - #2308
    • Write v3 manifests with row-lineage fields - #2456

Databricks and format interoperability

  • Read Databricks deletion vectors stored in .bin files - #2271
  • Read Databricks-managed and external Delta Lake/Iceberg tables through one Unity Catalog integration - #2607
  • Read Databricks Liquid-clustered tables through the native Delta Lake reader with clustering-based file skipping - #2395; #2394
  • Aggregate Apache-compatible HLL states - #2398
  • Import/export raw Apache DataSketches HLL states and select the HLL estimator - #2544
  • Persist ClickHouse aggregate-function states in Parquet/Iceberg - #2680; #2206

CAS (content-addressed storage)

CAS feature tracking #2441

  • Share MergeTree object storage across servers with deduplication, replica relinking, garbage collection, and pool inspection - #2159
  • Support MergeTree storage and data movement on CAS
    • Support Packed parts - #2579
    • Export parts and partitions from CAS disks - #2647
    • Reuse references for bulk REPLACE PARTITION, MOVE PARTITION, and ATTACH; relink merge results across replicas.
    • Reduce object-store requests when publishing CAS parts - #2794
  • Reclaim unused CAS data with parallel and distributed garbage collection
    • Delete retired blobs in parallel - #2351
    • Batch CAS garbage-collection deletes - #2534
    • Distribute GC work across multiple nodes
  • Back up and restore CAS tables using clickhouse-backup embedded mode.
  • Run CAS on Azure/POSIX storage and through encrypted disks
    • Support Azure Blob Storage
    • Support local/shared POSIX filesystems
    • Support CAS under the encrypted-disk wrapper

Performance and observability

  • Parallelize Parquet/Iceberg reads and reduce remote requests
    • Overlap reads and decoding while bounding remote I/O - #2275
    • Skip constant column chunks using statistics - #2266
    • Batch Bloom-filter reads across row groups - #2588
    • Accelerate dictionary-encoded column decoding - #2750
    • Accelerate dictionary filtering - #2735
    • Read Parquet in parallel with independent prefetch and decoding - #2235
    • Load DataLake object metadata on demand instead of issuing eager HEAD requests - #742
    • Cache REST catalog vended storage credentials - #1923
  • Prune Iceberg scans and answer partition aggregates from metadata
    • Prune manifests using partition statistics from the manifest list - #2758
    • Answer partition-column min, max, and count, including grouped/filtered queries, without reading data files - #2703
    • Use Puffin table/partition statistics for scan planning
  • Compact Iceberg manifests and data files - #2193, #2420
  • Monitor catalog requests and CAS health, and inspect Iceberg layouts
    • Expose REST, Glue, and Unity catalog request metrics - #1868
    • Expose Iceberg partition and sorting keys in system.tables - #959
    • Expose CAS GC health, request errors, and lease state through Grafana dashboards and alerts

Security

Token authentication and authorization

  • Validate tokens on the server and enforce access controls
    • Authenticate and authorize using tokens - #1078
    • Validate EC JWTs - #1596
    • Validate Entra ID tokens through OIDC/JWKS - #1784
    • Introspect OAuth tokens - #1799
    • Authenticate through Google as an identity provider - #2499
    • Map OAuth groups to settings profiles - #2664
  • Support client sign-in and token acquisition
    • Log in through browser and device-code flows - #1606
    • Acquire external JWTs on reconnect - #1809

Catalog and cloud authentication

  • Access Glue catalogs through IAM roles - #1427
  • Access Amazon S3 Tables through the Iceberg REST catalog using SigV4 - #1808
  • Access GCS through IAM/OAuth - #986
  • Authenticate to catalogs by forwarding OAuth tokens - #2329

Activity

  1. Korg95 commented on Mar 30, 2026

    @Korg95

    Hi team, wondering which quarter are we planning to implement iceberg v3 support.

  2. ylw510 commented on Aug 13, 2026

    @ylw510

    @svb-alt Hi.
    Does the roadmap support external contributors for the planned features?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions