Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
26 commits
Select commit Hold shift + click to select a range
6d7cf47
Cherry-pick of https://github.com/Altinity/ClickHouse/pull/2141 with …
zvonand Aug 6, 2026
852645a
Resolve conflicts in cherry-pick of #2141
zvonand Sep 15, 2026
c104023
Cherry-pick of https://github.com/Altinity/ClickHouse/pull/2145 with …
mkmkme Aug 11, 2026
b6b8545
Resolve conflicts in cherry-pick of #2145
zvonand Sep 15, 2026
80ee409
Cherry-pick of https://github.com/Altinity/ClickHouse/pull/2146 with …
zvonand Aug 14, 2026
c7553f2
Resolve conflicts in cherry-pick of #2146
zvonand Sep 15, 2026
fefaff6
Merge pull request #2200 from Altinity/feature/antalya-26.6/pr-1747
zvonand Aug 14, 2026
d72fb88
Cherry-pick of https://github.com/Altinity/ClickHouse/pull/2201 with …
zvonand Aug 14, 2026
6ccd444
Resolve conflicts in cherry-pick of #2201
zvonand Sep 15, 2026
df8ff97
Merge pull request #2209 from Altinity/feature/antalya-26.6/pr-1843
zvonand Aug 15, 2026
42619eb
Merge pull request #2210 from Altinity/feature/antalya-26.6/pr-1832
zvonand Aug 15, 2026
555167c
Merge pull request #2218 from Altinity/bugfix/antalya/26.6/export-inc…
mkmkme Aug 18, 2026
229cd78
Merge pull request #2229 from Altinity/feature/antalya/26.6/allow_non…
mkmkme Aug 20, 2026
f4c1fa5
Merge pull request #2253 from Altinity/feature/antalya-26.6/export-pa…
arthurpassos Aug 25, 2026
b970bee
Cherry-pick of https://github.com/Altinity/ClickHouse/pull/2249 with …
zvonand Sep 4, 2026
20cf1dc
Resolve conflicts in cherry-pick of #2249
zvonand Sep 15, 2026
19a8472
Merge pull request #2197 from Altinity/feature/antalya-26.6/cluster_d…
zvonand Sep 4, 2026
3754151
Merge pull request #2294 from Altinity/fix/antalya-26.6/export-skip-p…
zvonand Sep 10, 2026
1187302
Merge origin/antalya-26.8 into feature/antalya-26.8/auto-grp-pr-2141
zvonand Sep 17, 2026
23dc27f
Update SettingsChangesHistory.cpp
zvonand Sep 17, 2026
7b10c68
Merge origin/antalya-26.8 into feature/antalya-26.8/auto-grp-pr-2141
zvonand Sep 20, 2026
d619119
Fix settings history entries mangled while resolving the port
zvonand Sep 21, 2026
f2807ce
Fix Fast test failures of the combined 26.8 port
zvonand Sep 21, 2026
601848e
Fix the `Build (wasm_parser)` link failure on `NamedCollectionFactory`
zvonand Sep 21, 2026
28015fb
Fix `Stateless tests` and `Parser memory check` failures of the combi…
zvonand Sep 22, 2026
1f53cdb
Fix `NOT_IMPLEMENTED` on direct inserts into cluster object storage t…
zvonand Sep 23, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
The table of contents is too big for display.
Diff view
Diff view
  •  
  •  
  •  
1 change: 1 addition & 0 deletions ci/jobs/scripts/integration_tests_configs.py
Original file line number Diff line number Diff line change
Expand Up @@ -75,6 +75,7 @@ class TC:
True,
"pins azurite to fixed host port 10000 (Spark emulator mode); concurrent --dist=each workers collide on bind",
),
TC("test_export_replicated_mt_partition_to_object_storage/", True, "ZooKeeper can't handle too many parallel requests"),
TC(
"test_storage_delta/test.py",
False,
Expand Down
370 changes: 370 additions & 0 deletions docs/en/antalya/part_export.md

Large diffs are not rendered by default.

279 changes: 279 additions & 0 deletions docs/en/antalya/partition_export.md

Large diffs are not rendered by default.

73 changes: 73 additions & 0 deletions docs/en/antalya/swarm.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,73 @@
# Antalya branch

## Swarm

### Difference with upstream version

#### `storage_type` argument in object storage functions

In upstream ClickHouse, there are several table functions to read Iceberg tables from different storage backends such as `icebergLocal`, `icebergS3`, `icebergAzure`, `icebergHDFS`, cluster variants, the `iceberg` function as a synonym for `icebergS3`, and table engines like `IcebergLocal`, `IcebergS3`, `IcebergAzure`, `IcebergHDFS`.

In the Antalya branch, the `iceberg` table function and the `Iceberg` table engine unify all variants into one by using a new named argument, `storage_type`, which can be one of `local`, `s3`, `azure`, or `hdfs`.

Old syntax examples:

```sql
SELECT * FROM icebergS3('http://minio1:9000/root/table_data', 'minio', 'minio123', 'Parquet');
SELECT * FROM icebergAzureCluster('mycluster', 'http://azurite1:30000/devstoreaccount1', 'cont', '/table_data', 'devstoreaccount1', 'Eby8vdM02xNOcqFlqUwJPLlmEtlCDXJ1OUzFT50uSRZ6IFsuFq2UVErCz4I6tq/K1SZFPTOtr/KBHBeksoGMGw==', 'Parquet');
CREATE TABLE mytable ENGINE=IcebergHDFS('/table_data', 'Parquet');
```

New syntax examples:

```sql
SELECT * FROM iceberg(storage_type='s3', 'http://minio1:9000/root/table_data', 'minio', 'minio123', 'Parquet');
SELECT * FROM icebergCluster('mycluster', storage_type='azure', 'http://azurite1:30000/devstoreaccount1', 'cont', '/table_data', 'devstoreaccount1', 'Eby8vdM02xNOcqFlqUwJPLlmEtlCDXJ1OUzFT50uSRZ6IFsuFq2UVErCz4I6tq/K1SZFPTOtr/KBHBeksoGMGw==', 'Parquet');
CREATE TABLE mytable ENGINE=Iceberg('/table_data', 'Parquet', storage_type='hdfs');
```

Also, if a named collection is used to store access parameters, the field `storage_type` can be included in the same named collection:

```xml
<named_collections>
<s3>
<url>http://minio1:9001/root/</url>
<access_key_id>minio</access_key_id>
<secret_access_key>minio123</secret_access_key>
<storage_type>s3</storage_type>
</s3>
</named_collections>
```

```sql
SELECT * FROM iceberg(s3, filename='table_data');
```

By default `storage_type` is `'s3'` to maintain backward compatibility.


#### `object_storage_cluster` setting

The new setting `object_storage_cluster` controls whether a single-node or cluster variant of table functions reading from object storage (e.g., `s3`, `azure`, `iceberg`, and their cluster variants like `s3Cluster`, `azureCluster`, `icebergCluster`) is used.

Old syntax examples:

```sql
SELECT * from s3Cluster('myCluster', 'http://minio1:9001/root/data/{clickhouse,database}/*', 'minio', 'minio123', 'CSV',
'name String, value UInt32, polygon Array(Array(Tuple(Float64, Float64)))');
SELECT * FROM icebergAzureCluster('mycluster', 'http://azurite1:30000/devstoreaccount1', 'cont', '/table_data', 'devstoreaccount1', 'Eby8vdM02xNOcqFlqUwJPLlmEtlCDXJ1OUzFT50uSRZ6IFsuFq2UVErCz4I6tq/K1SZFPTOtr/KBHBeksoGMGw==', 'Parquet');
```

New syntax examples:

```sql
SELECT * from s3('http://minio1:9001/root/data/{clickhouse,database}/*', 'minio', 'minio123', 'CSV',
'name String, value UInt32, polygon Array(Array(Tuple(Float64, Float64)))')
SETTINGS object_storage_cluster='myCluster';
SELECT * FROM icebergAzure('http://azurite1:30000/devstoreaccount1', 'cont', '/table_data', 'devstoreaccount1', 'Eby8vdM02xNOcqFlqUwJPLlmEtlCDXJ1OUzFT50uSRZ6IFsuFq2UVErCz4I6tq/K1SZFPTOtr/KBHBeksoGMGw==', 'Parquet')
SETTINGS object_storage_cluster='myCluster';
```

This setting also applies to table engines and can be used with tables managed by Iceberg Catalog.

Note: The upstream ClickHouse has introduced analogous settings, such as `parallel_replicas_for_cluster_engines` and `cluster_for_parallel_replicas`. Since version 25.10, these settings work with table engines. It is possible that in the future, the `object_storage_cluster` setting will be deprecated.
56 changes: 56 additions & 0 deletions docs/en/operations/system-tables/exports.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,56 @@
---
description: 'System table containing information about in progress merge tree part exports'
keywords: ['system table', 'exports', 'merge tree', 'part']
slug: /operations/system-tables/exports
title: 'system.exports'
---

Contains information about in progress merge tree part exports

Columns:

- `source_database` ([String](/docs/en/sql-reference/data-types/string.md)) — Name of the source database.
- `source_table` ([String](/docs/en/sql-reference/data-types/string.md)) — Name of the source table.
- `destination_database` ([String](/docs/en/sql-reference/data-types/string.md)) — Name of the destination database.
- `destination_table` ([String](/docs/en/sql-reference/data-types/string.md)) — Name of the destination table.
- `create_time` ([DateTime](/docs/en/sql-reference/data-types/datetime.md)) — Date and time when the export command was received in the server.
- `part_name` ([String](/docs/en/sql-reference/data-types/string.md)) — Name of the part.
- `destination_file_path` ([String](/docs/en/sql-reference/data-types/string.md)) — File path relative to where the part is being exported to.
- `elapsed` ([Float64](/docs/en/sql-reference/data-types/float.md)) — The time elapsed (in seconds) since the export started.
- `rows_read` ([UInt64](/docs/en/sql-reference/data-types/int-uint.md)) — The number of rows read from the exported part.
- `total_rows_to_read` ([UInt64](/docs/en/sql-reference/data-types/int-uint.md)) — The total number of rows to read from the exported part.
- `total_size_bytes_compressed` ([UInt64](/docs/en/sql-reference/data-types/int-uint.md)) — The total size of the compressed data in the exported part.
- `total_size_bytes_uncompressed` ([UInt64](/docs/en/sql-reference/data-types/int-uint.md)) — The total size of the uncompressed data in the exported part.
- `bytes_read_uncompressed` ([UInt64](/docs/en/sql-reference/data-types/int-uint.md)) — The number of uncompressed bytes read from the exported part.
- `memory_usage` ([UInt64](/docs/en/sql-reference/data-types/int-uint.md)) — Current memory usage in bytes for the export operation.
- `peak_memory_usage` ([UInt64](/docs/en/sql-reference/data-types/int-uint.md)) — Peak memory usage in bytes during the export operation.

**Example**

```sql
arthur :) select * from system.exports;

SELECT *
FROM system.exports

Query id: 2026718c-d249-4208-891b-a271f1f93407

Row 1:
──────
source_database: default
source_table: source_mt_table
destination_database: default
destination_table: destination_table
create_time: 2025-11-19 09:09:11
part_name: 20251016-365_1_1_0
destination_file_path: table_root/eventDate=2025-10-16/retention=365/20251016-365_1_1_0_17B2F6CD5D3C18E787C07AE3DAF16EB1.parquet
elapsed: 2.04845441
rows_read: 1138688 -- 1.14 million
total_rows_to_read: 550961374 -- 550.96 million
total_size_bytes_compressed: 37619147120 -- 37.62 billion
total_size_bytes_uncompressed: 138166213721 -- 138.17 billion
bytes_read_uncompressed: 316892925 -- 316.89 million
memory_usage: 596006095 -- 596.01 million
peak_memory_usage: 601239033 -- 601.24 million
```

23 changes: 23 additions & 0 deletions docs/en/sql-reference/distribution-on-cluster.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
# Task distribution in *Cluster family functions

## Task distribution algorithm

Table functions such as `s3Cluster`, `azureBlobStorageCluster`, `hdsfCluster`, `icebergCluster`, and table engines like `S3`, `Azure`, `HDFS`, `Iceberg` with the setting `object_storage_cluster` distribute tasks across all cluster nodes or a subset limited by the `object_storage_max_nodes` setting. This setting limits the number of nodes involved in processing a distributed query, randomly selecting nodes for each query.

A single task corresponds to processing one source file.

For each file, one cluster node is selected as the primary node using a consistent Rendezvous Hashing algorithm. This algorithm guarantees that:
* The same node is consistently selected as primary for each file, as long as the cluster remains unchanged.
* When the cluster changes (nodes added or removed), only files assigned to those affected nodes change their primary node assignment.

This improves cache efficiency by minimizing data movement among nodes.

## `lock_object_storage_task_distribution_ms` setting

Each node begins processing files for which it is the primary node. After completing its assigned files, a node may take tasks from other nodes, either immediately or after waiting for `lock_object_storage_task_distribution_ms` milliseconds if the primary node does not request new files during that interval. The default value of `lock_object_storage_task_distribution_ms` is 500 milliseconds. This setting balances between caching efficiency and workload redistribution when nodes are imbalanced.

## `SYSTEM STOP SWARM MODE` command

If a node needs to shut down gracefully, the command `SYSTEM STOP SWARM MODE` prevents the node from receiving new tasks for *Cluster-family queries. The node finishes processing already assigned files before it can safely shut down without errors.

Receiving new tasks can be resumed with the command `SYSTEM START SWARM MODE`.
2 changes: 2 additions & 0 deletions docs/guides/oss/deployment-and-scaling/cluster-discovery.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -62,6 +62,8 @@ Traditionally, in ClickHouse, each shard and replica in the cluster needed to be

With Cluster Discovery, rather than defining each node explicitly, you simply specify a path in ZooKeeper. All nodes that register under this path in ZooKeeper will be automatically discovered and added to the cluster.

Discovery settings under `remote_servers` (including `user`, `password`, `secret`, `path`, `multicluster_root_path`, and adding or removing discovery clusters) are applied on configuration reload (for example with `SYSTEM RELOAD CONFIG`). A server restart is not required for these changes.

```xml
<remote_servers>
<cluster_name>
Expand Down
27 changes: 27 additions & 0 deletions docs/reference/engines/database-engines/datalake.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -64,6 +64,7 @@ The following settings are supported:
| `dlf_access_key_id` | Access key ID for DLF access |
| `dlf_access_key_secret` | Access key Secret for DLF access |
| `force_add_bucket` | When constructing object-storage URLs from the catalog-provided table location and `storage_endpoint`, prepend the bucket/container name even if the endpoint already contains it. Default: `false`. Set to `true` for catalogs that hand back paths without the bucket and require it to be added at the URL-construction step (Polaris-style paths). |
| `namespaces` | Comma-separated list of namespaces, implemented for catalog types: `rest`, `glue` and `unity` |

## Examples {#examples}

Expand Down Expand Up @@ -91,4 +92,30 @@ SELECT count() from database_name.table_name;
bearer token (scoped to https://storage.azure.com) instead of
`onelake_client_id`/`onelake_client_secret`. ClickHouse does not refresh the token, so the
database must be recreated after it expires.

## Namespace filter {#namespace}

By default, ClickHouse reads tables from all namespaces available in the catalog. You can limit this behavior using the `namespaces` database setting. The value should be a comma‑separated list of namespaces that are allowed to be read.

Supported catalog types are `rest`, `glue` and `unity`.

For example, if the catalog contains three namespaces - `dev`, `stage`, and `prod` - and you want to read data only from dev and stage, set:
```
namespaces='dev,stage'
```

### Nested namespaces {#namespace-nested}

The Iceberg (`rest`) catalog supports nested namespaces. The `namespaces` filter accepts the following patterns:

- `namespace` - includes tables from the specified namespace, but not from its nested namespaces.
- `namespace.nested` - includes tables from the nested namespace, but not from the parent.
- `namespace.*` - includes tables from all nested namespaces, but not from the parent.

If you need to include both a namespace and its nested namespaces, specify both explicitly. For example:
```
namespaces='namespace,namespace.*'
```

The default value is '*', which means all namespaces are included.
{/*AUTOGENERATED_END*/}
56 changes: 56 additions & 0 deletions docs/reference/engines/table-engines/integrations/iceberg.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -384,6 +384,62 @@ SETTINGS iceberg_metadata_staleness_ms=120000

**Note**: Current expectation is that metadata cache size is sufficient to hold the latest metadata snapshot in full for all active tables, if asynchronous prefetching is enabled.

## Altinity Antalya branch

### Specify storage type in arguments

Only in the Altinity Antalya branch does `Iceberg` table engine support all storage types. The storage type can be specified using the named argument `storage_type`. Supported values are `s3`, `azure`, `hdfs`, and `local`.

```sql
CREATE TABLE iceberg_table_s3
ENGINE = Iceberg(storage_type='s3', url, [, NOSIGN | access_key_id, secret_access_key, [session_token]], format, [,compression])

CREATE TABLE iceberg_table_azure
ENGINE = Iceberg(storage_type='azure', connection_string|storage_account_url, container_name, blobpath, [account_name, account_key, format, compression])

CREATE TABLE iceberg_table_hdfs
ENGINE = Iceberg(storage_type='hdfs', path_to_table, [,format] [,compression_method])

CREATE TABLE iceberg_table_local
ENGINE = Iceberg(storage_type='local', path_to_table, [,format] [,compression_method])
```

### Specify storage type in named collection

Only in Altinity Antalya branch `storage_type` can be included as part of a named collection. This allows for centralized configuration of storage settings.

```xml
<clickhouse>
<named_collections>
<iceberg_conf>
<url>http://test.s3.amazonaws.com/clickhouse-bucket/</url>
<access_key_id>test<access_key_id>
<secret_access_key>test</secret_access_key>
<format>auto</format>
<structure>auto</structure>
<storage_type>s3</storage_type>
</iceberg_conf>
</named_collections>
</clickhouse>
```

```sql
CREATE TABLE iceberg_table ENGINE=Iceberg(iceberg_conf, filename = 'test_table')
```

The default value for `storage_type` is `s3`.

### The `object_storage_cluster` setting.

Only in the Altinity Antalya branch is an alternative syntax for the `Iceberg` table engine available. This syntax allows execution on a cluster when the `object_storage_cluster` setting is non-empty and contains the cluster name.

```sql
CREATE TABLE iceberg_table_s3
ENGINE = Iceberg(storage_type='s3', url, [, NOSIGN | access_key_id, secret_access_key, [session_token]], format, [,compression]);

SELECT * FROM iceberg_table_s3 SETTINGS object_storage_cluster='cluster_simple';
```

## See also {#see-also}

- [iceberg table function](/reference/functions/table-functions/iceberg)
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -52,6 +52,20 @@ SELECT count(*) FROM azureBlobStorageCluster(

See [azureBlobStorage](/reference/functions/table-functions/azureBlobStorage#using-shared-access-signatures-sas-sas-tokens) for examples.

## Altinity Antalya branch

### `object_storage_cluster` setting.

Only in the Altinity Antalya branch, the alternative syntax for the `azureBlobStorageCluster` table function is avilable. This allows the `azureBlobStorage` function to be used with the non-empty `object_storage_cluster` setting, specifying a cluster name. This enables distributed queries over Azure Blob Storage across a ClickHouse cluster.

```sql
SELECT count(*) FROM azureBlobStorage(
'http://azurite1:10000/devstoreaccount1', 'testcontainer', 'test_cluster_count.csv', 'devstoreaccount1',
'Eby8vdM02xNOcqFlqUwJPLlmEtlCDXJ1OUzFT50uSRZ6IFsuFq2UVErCz4I6tq/K1SZFPTOtr/KBHBeksoGMGw==', 'CSV',
'auto', 'key UInt64')
SETTINGS object_storage_cluster='cluster_simple'
```

## Related {#related}

- [AzureBlobStorage engine](/reference/engines/table-engines/integrations/azureBlobStorage)
Expand Down
11 changes: 11 additions & 0 deletions docs/reference/functions/table-functions/deltalakeCluster.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -43,6 +43,17 @@ A table with the specified structure for reading data from cluster in the specif
- `_time` — Last modified time of the file. Type: `Nullable(DateTime)`. If the time is unknown, the value is `NULL`.
- `_etag` — The etag of the file. Type: `LowCardinality(String)`. If the etag is unknown, the value is `NULL`.

## Altinity Antalya branch

### `object_storage_cluster` setting.

Only in the Altinity Antalya branch alternative syntax for `deltaLakeCluster` table function is available. This allows the `deltaLake` function to be used with the non-empty `object_storage_cluster` setting, specifying a cluster name. This enables distributed queries over Delta Lake Storage across a ClickHouse cluster.

```sql
SELECT count(*) FROM deltaLake(url [,aws_access_key_id, aws_secret_access_key] [,format] [,structure] [,compression])
SETTINGS object_storage_cluster='cluster_simple'
```

## Related {#related}

- [deltaLake engine](/reference/engines/table-engines/integrations/deltalake)
Expand Down
12 changes: 12 additions & 0 deletions docs/reference/functions/table-functions/hdfsCluster.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -58,6 +58,18 @@ FROM hdfsCluster('cluster_simple', 'hdfs://hdfs1:9000/{some,another}_dir/*', 'TS
If your listing of files contains number ranges with leading zeros, use the construction with braces for each digit separately or use `?`.
</Note>

## Altinity Antalya branch

### `object_storage_cluster` setting.

Only in the Altinity Antalya branch alternative syntax for `hdfsCluster` table function is available. This allows the `hdfs` function to be used with the non-empty `object_storage_cluster` setting, specifying a cluster name. This enables distributed queries over HDFS Storage across a ClickHouse cluster.

```sql
SELECT count(*)
FROM hdfs('hdfs://hdfs1:9000/{some,another}_dir/*', 'TSV', 'name String, value UInt32')
SETTINGS object_storage_cluster='cluster_simple'
```

## Related {#related}

- [HDFS engine](/reference/engines/table-engines/integrations/hdfs)
Expand Down
12 changes: 12 additions & 0 deletions docs/reference/functions/table-functions/hudiCluster.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -42,6 +42,18 @@ A table with the specified structure for reading data from cluster in the specif
- `_time` — Last modified time of the file. Type: `Nullable(DateTime)`. If the time is unknown, the value is `NULL`.
- `_etag` — The etag of the file. Type: `LowCardinality(String)`. If the etag is unknown, the value is `NULL`.

## Altinity Antalya branch

### `object_storage_cluster` setting.

Only in the Altinity Antalya branch alternative syntax for `hudiCluster` table function is available. This allows the `hudi` function to be used with the non-empty `object_storage_cluster` setting, specifying a cluster name. This enables distributed queries over Hudi Storage across a ClickHouse cluster.

```sql
SELECT *
FROM hudi(url [,aws_access_key_id, aws_secret_access_key] [,format] [,structure] [,compression])
SETTINGS object_storage_cluster='cluster_simple'
```

## Related {#related}

- [Hudi engine](/reference/engines/table-engines/integrations/hudi)
Expand Down
Loading
Loading