diff --git a/ci/jobs/scripts/integration_tests_configs.py b/ci/jobs/scripts/integration_tests_configs.py index a8a9184c849b..70991e1ddda1 100644 --- a/ci/jobs/scripts/integration_tests_configs.py +++ b/ci/jobs/scripts/integration_tests_configs.py @@ -75,6 +75,7 @@ class TC: True, "pins azurite to fixed host port 10000 (Spark emulator mode); concurrent --dist=each workers collide on bind", ), + TC("test_export_replicated_mt_partition_to_object_storage/", True, "ZooKeeper can't handle too many parallel requests"), TC( "test_storage_delta/test.py", False, diff --git a/docs/en/antalya/part_export.md b/docs/en/antalya/part_export.md new file mode 100644 index 000000000000..5cf8e2aa8b0e --- /dev/null +++ b/docs/en/antalya/part_export.md @@ -0,0 +1,370 @@ +# ALTER TABLE EXPORT PART + +## Overview + +The `ALTER TABLE EXPORT PART` command exports individual MergeTree data parts to object storage (S3, Azure Blob Storage, etc.) or data lakes like Apache Iceberg tables (with and without catalogs), typically in Parquet format. + +**Key Characteristics:** +- **Experimental feature** - must be enabled via `allow_experimental_export_merge_tree_part` setting +- **Asynchronous** - executes in the background, returns immediately +- **Ephemeral** - no automatic retry mechanism; manual retry required on failure +- **Idempotent** - safe to re-export the same part (skips by default if file exists) +- **Preserves sort order** from the source table + +### On Apache Iceberg storage exports: + +Each MergeTree part will become a separate file (or more depending on `max_bytes` and `max_rows` settings) following the engine naming convention. Once the part has been exported, new snapshots / manifest files are generated and the data is committed using the Apache Iceberg commit mechanism. + +### On plain object storage exports: + +A commit file is shipped to the same destination directory containing all data files exported within that transaction. + +## Syntax + +```sql +ALTER TABLE [database.]table_name +EXPORT PART 'part_name' +TO TABLE [destination_database.]destination_table +SETTINGS allow_experimental_export_merge_tree_part = 1 + [, setting_name = value, ...] +``` + +## Syntax with table function + +```sql +ALTER TABLE [database.]table_name +EXPORT PART 'part_name' +TO TABLE FUNCTION s3(s3_conn, filename='table_function', partition_strategy...) +SETTINGS allow_experimental_export_merge_tree_part = 1 + [, setting_name = value, ...] +``` + +### Parameters + +- **`table_name`**: The source MergeTree table containing the part to export +- **`part_name`**: The exact name of the data part to export (e.g., `'2020_1_1_0'`, `'all_1_1_0'`) +- **`destination_table`**: The target table for the export (typically an S3, Azure, or other object storage table) + +## Requirements + +Source and destination tables must support positional schema conversion. The following differences between the two schemas are allowed: + +- **Column names** may differ between source and destination for non-partition-key columns - columns are matched by position, similar to `INSERT INTO dest SELECT * FROM src`, not by name. +- **Column types** may differ, as long as the source type is safely castable to the destination type. Set `export_merge_tree_part_allow_lossy_cast = 1` to also permit lossy casts. +- **`Tuple` element names** may differ if either the source or destination declares the tuple without named elements: an unnamed `Tuple` (e.g. `Tuple(Int32, Int32)`) is matched against the destination by element position and type only, not by name. For example, exporting from `t Tuple(Int32, Int32)` to `t Tuple(x Int32, y Int32)` is allowed as long as element types match positionally. + +The following requirements apply to the source and destination: + +1. **Column count** - source and destination must have the same number of columns by default. A mismatch in either direction throws `NUMBER_OF_COLUMNS_DOESNT_MATCH`. Set `export_merge_tree_part_schema_mismatch_mode = 'ignore_extra_source_columns_by_position'` to allow a source table with extra trailing columns; the destination having more columns than the source is still rejected in this mode. +2. **`PARTITION BY` expressions** - the whole part must land in a single destination partition. Identical expressions always satisfy this; otherwise the destination expression has to be computable from the values the source partition key pins, or be proven single-valued over the part's min/max range. The same requirement applies to the partition fields and transforms of an Apache Iceberg destination. See [Source partition key compatibility](/docs/en/antalya/partition_export.md#source-partition-key-compatibility). +3. **The position of every column backing the partition key** - it is not enough for the `PARTITION BY` expressions to be textually identical: every top-level column that provides a column or subcolumn used by the source table's partition key must have the same name at the same position in the destination table's schema. If such a column contains a named `Tuple`, its element names must also be declared in the same order (an unnamed `Tuple` on either side is exempt from this, per the allowance above). This comparison is recursive through nested tuples and through container types such as `Array` and `Map`. + + For example, `CREATE TABLE src (a Int32, b Int32) ... PARTITION BY a` and `CREATE TABLE dst (b Int32, a Int32) ... PARTITION BY a` both have the expression `PARTITION BY a`, but `a` is at position 0 in `src` and position 1 in `dst`. The export is rejected with a `BAD_ARGUMENTS` exception whose message includes `Cannot export to : partition key column 'a' is at position 0 in the source table, but the destination's column at that position is named 'b'`. + + This position check applies only to partition-key columns. A mismatch in the position of a non-partition-key column is allowed by name (see above) and is only rejected if the resulting types aren't castable. If two non-partition-key columns happen to have swapped positions but compatible types, the export succeeds and silently writes values into the wrong destination column, so keep the intended column order rather than relying on type compatibility alone. + + For `PARTITION BY t.a`, this rule applies to the top-level owning column `t`. Exporting from `t Tuple(a Int32, b Int32)` to `t Tuple(b Int32, a Int32)` is rejected, even though `a` is accessed by name. Requiring a stable layout for every partition-key owner also protects positional expressions such as `tupleElement(t, 1)` from changing their meaning after conversion. + + The same rule applies when the named tuple is nested inside a container. For example, `arr Array(Tuple(a Int32, b Int32))` and `arr Array(Tuple(b Int32, a Int32))` are incompatible when `arr` provides an input to the partition key. Likewise, tuple layouts in both the key and value types of `Map` are checked recursively. + + In this case, the export throws a `BAD_ARGUMENTS` exception whose message includes `partition key column 't' has a different Tuple element layout in the source (Tuple(a Int32, b Int32)) and destination (Tuple(b Int32, a Int32)). Tuple element names must be declared in the same order in both tables`. + + For partition expressions containing functions, the check applies to their input columns. For example, `PARTITION BY (toYYYYMM(ts), category)` requires both `ts` and `category` to have the same names at the same top-level positions in both tables. + +In case a table function is used as the destination, the schema can be omitted and it will be inferred from the source table. + +## Settings + +### `allow_experimental_export_merge_tree_part` (Required) + +- **Type**: `Bool` +- **Default**: `false` +- **Description**: Must be set to `true` to enable the experimental feature. + +### `export_merge_tree_part_overwrite_file_if_exists` (Optional) + +- **Type**: `Bool` +- **Default**: `false` +- **Description**: If set to `true`, it will overwrite the file. Otherwise, fails with exception. + +### `export_merge_tree_part_max_bytes_per_file` (Optional) + +- **Type**: `UInt64` +- **Default**: `0` +- **Description**: Maximum number of bytes to write to a single file when exporting a merge tree part. 0 means no limit. This is not a hard limit, and it highly depends on the output format granularity and input source chunk size. Using this might break idempotency, use it with care. + +### `export_merge_tree_part_max_rows_per_file` (Optional) + +- **Type**: `UInt64` +- **Default**: `0` +- **Description**: Maximum number of rows to write to a single file when exporting a merge tree part. 0 means no limit. This is not a hard limit, and it highly depends on the output format granularity and input source chunk size. Using this might break idempotency, use it with care. + +#### `export_merge_tree_part_file_already_exists_policy` (Optional) + +- **Type**: `MergeTreePartExportFileAlreadyExistsPolicy` +- **Default**: `skip` +- **Description**: Policy for handling files that already exist during export. Possible values: + - `skip` - Skip the file if it already exists + - `error` - Throw an error if the file already exists + - `overwrite` - Overwrite the file + +### `export_merge_tree_part_throw_on_pending_mutations` (Optional) + +- **Type**: `bool` +- **Default**: `true` +- **Description**: If set to true, throws if pending mutations exists for a given part. Note that by default mutations are applied to all parts, which means that if a mutation in practice would only affetct part/partition x, all the other parts/partition will throw upon export. The exception is when the `IN PARTITION` clause was used in the mutation command. Note the `IN PARTITION` clause is not properly implemented for plain MergeTree tables. + +### `export_merge_tree_part_throw_on_pending_patch_parts` (Optional) + +- **Type**: `bool` +- **Default**: `true` +- **Description**: If set to true, throws if pending patch parts exists for a given part. Note that by default mutations are applied to all parts, which means that if a mutation in practice would only affetct part/partition x, all the other parts/partition will throw upon export. The exception is when the `IN PARTITION` clause was used in the mutation command. Note the `IN PARTITION` clause is not properly implemented for plain MergeTree tables. + +### `export_merge_tree_part_filename_pattern` (Optional) + +- **Type**: `String` +- **Default**: `{part_name}_{checksum}` +- **Description**: Pattern for the filename of the exported merge tree part. The `part_name` and `checksum` are calculated and replaced on the fly. Additional macros are supported. + +### `export_merge_tree_part_allow_lossy_cast` (Optional) + +- **Type**: `Bool` +- **Default**: `false` +- **Description**: Allow `EXPORT PART`/`EXPORT PARTITION` to apply lossy (non-value-preserving) casts when the source and destination column types differ. When disabled, an export that would require a lossy cast throws instead. + + When exporting to Apache Iceberg, the partition value written to the metadata is derived from the source partition columns by casting them to the destination partition-field types and applying the destination partition transform — the same computation the exported data files use. This keeps the Iceberg metadata consistent with the data files. + + **Warning:** A lossy cast on a partition column remains semantically truncating. For example, if a table is partitioned by an `Int64` column and some partition values do not fit into a destination `Int32` partition column, both the data files and the Iceberg metadata will contain the truncated `Int32` value (they agree with each other, but the original `Int64` value is lost). Such casts require `export_merge_tree_part_allow_lossy_cast = 1`. + +### `export_merge_tree_part_schema_mismatch_mode` (Optional) + +- **Type**: `MergeTreePartExportSchemaMismatchMode` +- **Default**: `strict` +- **Description**: Controls whether `EXPORT PART`/`EXPORT PARTITION` allows a column-count mismatch between the source `MergeTree` table and the destination table. Columns are matched positionally, like `INSERT INTO dest SELECT * FROM src`. Possible values: + - `strict` - the source and destination must have the same number of columns. A mismatch in either direction throws `NUMBER_OF_COLUMNS_DOESNT_MATCH`. + - `ignore_extra_source_columns_by_position` - the source may have more columns than the destination. The extra trailing source columns (by position) are dropped and not exported. The destination having more columns than the source is still rejected in this mode. + + The extra trailing source columns are still read and evaluated (including `MATERIALIZED`/`ALIAS` columns, and any column another kept column's `ALIAS`/`MATERIALIZED` expression depends on) before being dropped, so this setting only changes which columns end up in the destination, not what is computed while reading the part. + + +## Examples + +### Basic Export to S3 + +```sql +-- Create source and destination tables +CREATE TABLE mt_table (id UInt64, year UInt16) +ENGINE = MergeTree() PARTITION BY year ORDER BY tuple(); + +CREATE TABLE s3_table (id UInt64, year UInt16) +ENGINE = S3(s3_conn, filename='data', format=Parquet, partition_strategy='hive') +PARTITION BY year; + +-- Insert and export +INSERT INTO mt_table VALUES (1, 2020), (2, 2020), (3, 2021); + +ALTER TABLE mt_table EXPORT PART '2020_1_1_0' TO TABLE s3_table +SETTINGS allow_experimental_export_merge_tree_part = 1; + +ALTER TABLE mt_table EXPORT PART '2021_2_2_0' TO TABLE s3_table +SETTINGS allow_experimental_export_merge_tree_part = 1; +``` + +### Table function export + +```sql +-- Create source and destination tables +CREATE TABLE mt_table (id UInt64, year UInt16) +ENGINE = MergeTree() PARTITION BY year ORDER BY tuple(); + +-- Insert and export +INSERT INTO mt_table VALUES (1, 2020), (2, 2020), (3, 2021); + +ALTER TABLE mt_table EXPORT PART '2020_1_1_0' TO TABLE FUNCTION s3(s3_conn, filename='table_function', format=Parquet, partition_strategy='hive') PARTITION BY year +SETTINGS allow_experimental_export_merge_tree_part = 1; +``` + +## Monitoring + +### Active Exports + +Active exports can be found in the `system.exports` table. As of now, it only shows currently executing exports. It will not show pending or finished exports. + +```sql +arthur :) select * from system.exports; + +SELECT * +FROM system.exports + +Query id: 2026718c-d249-4208-891b-a271f1f93407 + +Row 1: +────── +source_database: default +source_table: source_mt_table +destination_database: default +destination_table: destination_table +create_time: 2025-11-19 09:09:11 +part_name: 20251016-365_1_1_0 +destination_file_paths: ['table_root/eventDate=2025-10-16/retention=365/20251016-365_1_1_0_17B2F6CD5D3C18E787C07AE3DAF16EB1.1.parquet'] +elapsed: 2.04845441 +rows_read: 1138688 -- 1.14 million +total_rows_to_read: 550961374 -- 550.96 million +total_size_bytes_compressed: 37619147120 -- 37.62 billion +total_size_bytes_uncompressed: 138166213721 -- 138.17 billion +bytes_read_uncompressed: 316892925 -- 316.89 million +memory_usage: 596006095 -- 596.01 million +peak_memory_usage: 601239033 -- 601.24 million +``` + +### Export History + +You can query succeeded or failed exports in `system.part_log`. For now, it only keeps track of completion events (either success or fails). + +```sql +arthur :) select * from system.part_log where event_type='ExportPart' and table = 'replicated_source' order by event_time desc limit 1; + +SELECT * +FROM system.part_log +WHERE (event_type = 'ExportPart') AND (`table` = 'replicated_source') +ORDER BY event_time DESC +LIMIT 1 + +Query id: ae1c1cd3-c20e-4f20-8b82-ed1f6af0237f + +Row 1: +────── +hostname: arthur +query_id: +event_type: ExportPart +merge_reason: NotAMerge +merge_algorithm: Undecided +event_date: 2025-11-19 +event_time: 2025-11-19 09:08:31 +event_time_microseconds: 2025-11-19 09:08:31.974701 +duration_ms: 4 +database: default +table: replicated_source +table_uuid: 78471c67-24f4-4398-9df5-ad0a6c3daf41 +part_name: 2021_0_0_0 +partition_id: 2021 +partition: 2021 +part_type: Compact +disk_name: default +path_on_disk: +remote_file_paths ['year=2021/2021_0_0_0_78C704B133D41CB0EF64DD2A9ED3B6BA.1.parquet'] +rows: 1 +size_in_bytes: 272 +merged_from: ['2021_0_0_0'] +bytes_uncompressed: 86 +read_rows: 1 +read_bytes: 6 +peak_memory_usage: 22 +error: 0 +exception: +ProfileEvents: {} +``` + +### Profile Events + +- `PartsExports` - Successful exports +- `PartsExportFailures` - Failed exports +- `PartsExportDuplicated` - Number of part exports that failed because target already exists. +- `PartsExportTotalMilliseconds` - Total time + +### Split large files + +```sql +alter table big_table export part '2025_0_32_3' to table replicated_big_destination SETTINGS export_merge_tree_part_max_bytes_per_file=10000000, output_format_parquet_row_group_size_bytes=5000000; + +arthur :) select * from system.exports; + +SELECT * +FROM system.exports + +Query id: d78d9ce5-cfbc-4957-b7dd-bc8129811634 + +Row 1: +────── +source_database: default +source_table: big_table +destination_database: default +destination_table: replicated_big_destination +create_time: 2025-12-15 13:12:48 +part_name: 2025_0_32_3 +destination_file_paths: ['replicated_big/year=2025/2025_0_32_3_E439C23833C39C6E5104F6F4D1048BE7.1.parquet','replicated_big/year=2025/2025_0_32_3_E439C23833C39C6E5104F6F4D1048BE7.2.parquet','replicated_big/year=2025/2025_0_32_3_E439C23833C39C6E5104F6F4D1048BE7.3.parquet','replicated_big/year=2025/2025_0_32_3_E439C23833C39C6E5104F6F4D1048BE7.4.parquet'] +elapsed: 14.360427274 +rows_read: 10256384 -- 10.26 million +total_rows_to_read: 10485760 -- 10.49 million +total_size_bytes_compressed: 83779395 -- 83.78 million +total_size_bytes_uncompressed: 10611691600 -- 10.61 billion +bytes_read_uncompressed: 10440998912 -- 10.44 billion +memory_usage: 89795477 -- 89.80 million +peak_memory_usage: 107362133 -- 107.36 million + +1 row in set. Elapsed: 0.014 sec. + +arthur :) select * from system.part_log where event_type = 'ExportPart' order by event_time desc limit 1 format Vertical; + +SELECT * +FROM system.part_log +WHERE event_type = 'ExportPart' +ORDER BY event_time DESC +LIMIT 1 +FORMAT Vertical + +Query id: 95128b01-b751-4726-8e3e-320728ac6af7 + +Row 1: +────── +hostname: arthur +query_id: +event_type: ExportPart +merge_reason: NotAMerge +merge_algorithm: Undecided +event_date: 2025-12-15 +event_time: 2025-12-15 13:13:03 +event_time_microseconds: 2025-12-15 13:13:03.197492 +duration_ms: 14673 +database: default +table: big_table +table_uuid: a3eeeea0-295c-41a3-84ef-6b5463dbbe8c +part_name: 2025_0_32_3 +partition_id: 2025 +partition: 2025 +part_type: Wide +disk_name: default +path_on_disk: ./store/a3e/a3eeeea0-295c-41a3-84ef-6b5463dbbe8c/2025_0_32_3/ +remote_file_paths: ['replicated_big/year=2025/2025_0_32_3_E439C23833C39C6E5104F6F4D1048BE7.1.parquet','replicated_big/year=2025/2025_0_32_3_E439C23833C39C6E5104F6F4D1048BE7.2.parquet','replicated_big/year=2025/2025_0_32_3_E439C23833C39C6E5104F6F4D1048BE7.3.parquet','replicated_big/year=2025/2025_0_32_3_E439C23833C39C6E5104F6F4D1048BE7.4.parquet'] +rows: 10485760 -- 10.49 million +size_in_bytes: 83779395 -- 83.78 million +merged_from: ['2025_0_32_3'] +bytes_uncompressed: 10611691600 -- 10.61 billion +read_rows: 10485760 -- 10.49 million +read_bytes: 10674503680 -- 10.67 billion +peak_memory_usage: 107362133 -- 107.36 million +error: 0 +exception: +ProfileEvents: {} + +1 row in set. Elapsed: 0.044 sec. + +arthur :) select _path, formatReadableSize(_size) as _size from s3(s3_conn, filename='**', format=One); + +SELECT + _path, + formatReadableSize(_size) AS _size +FROM s3(s3_conn, filename = '**', format = One) + +Query id: c48ae709-f590-4d1b-8158-191f8d628966 + + ┌─_path────────────────────────────────────────────────────────────────────────────────┬─_size─────┐ +1. │ test/replicated_big/year=2025/2025_0_32_3_E439C23833C39C6E5104F6F4D1048BE7.1.parquet │ 17.36 MiB │ +2. │ test/replicated_big/year=2025/2025_0_32_3_E439C23833C39C6E5104F6F4D1048BE7.2.parquet │ 17.32 MiB │ +3. │ test/replicated_big/year=2025/2025_0_32_3_E439C23833C39C6E5104F6F4D1048BE7.4.parquet │ 5.04 MiB │ +4. │ test/replicated_big/year=2025/2025_0_32_3_E439C23833C39C6E5104F6F4D1048BE7.3.parquet │ 17.40 MiB │ +5. │ test/replicated_big/year=2025/commit_2025_0_32_3_E439C23833C39C6E5104F6F4D1048BE7 │ 320.00 B │ + └──────────────────────────────────────────────────────────────────────────────────────┴───────────┘ + +5 rows in set. Elapsed: 0.072 sec. +``` diff --git a/docs/en/antalya/partition_export.md b/docs/en/antalya/partition_export.md new file mode 100644 index 000000000000..334caf82f972 --- /dev/null +++ b/docs/en/antalya/partition_export.md @@ -0,0 +1,279 @@ +# ALTER TABLE EXPORT PARTITION + +## Overview + +The `ALTER TABLE EXPORT PARTITION` command exports entire partitions from Replicated*MergeTree tables to object storage (S3, Azure Blob Storage, etc.) or data lakes like Apache Iceberg tables (with and without catalogs), typically in Parquet format. This feature coordinates export part operations across all replicas using ZooKeeper. + +The set of parts that are exported is based on the list of parts the replica that received the export command sees. The other replicas will assist in the export process if they have those parts locally. Otherwise they will ignore it. + +The partition export tasks can be observed through `system.replicated_partition_exports`. The table is served from each replica's in-memory mirror, so queries do not contact ZooKeeper and are cheap to run. The mirror is refreshed on the manifest-updater poll cycle and on every status change, so a freshly written exception or terminal state may take up to one poll interval to appear. Individual part export progress can be observed as usual through `system.exports`. + +The same partition can not be exported to the same destination more than once. There are two ways to override this behavior: either by setting the `export_merge_tree_partition_force_export` setting or waiting for the task to expire. + +The export task can be killed by issuing the kill command: `KILL EXPORT PARTITION `. + +The task is persistent - it should be resumed after crashes, failures and etc. + +### On Apache Iceberg storage exports: + +Each MergeTree part will become a separate file (or more depending on `max_bytes` and `max_rows` settings) following the engine naming convention. Once all parts have been exported, new snapshots / manifest files are generated and the data is comitted using the Apache Iceberg commit mechanism. + +The manifest file produced by the commit contains a summary field `clickhouse.export-partition-transaction-id` that stores the transaction id. This field is used to implement idempotency and avoid data duplication. Some Apache Iceberg storage managers employ old manifests cleanup, ClickHouse does not. + +**IMPORTANT**: In case the storage is managed by a 3rd party application that cleans up old manifest files, it is important that the TTL of such files are greater than the timeout of export partition tasks. If it is not configured in such a way, it is possible to accidentally duplicate data in the extremely rare case a ClickHouse node is the only node working on a given export task, commits the data to Iceberg, crashes before marking the task as done and only boots up after the manifest cleanup has deleted the commit manifest. In such scenario, ClickHouse would attempt to commit those files again producing duplicates. The task timeout on ClickHouse side is controlled by the setting `export_merge_tree_partition_task_timeout_seconds`. + +The Iceberg manifest files contain statistics about the data. Exporting a merge tree partition is a non ephemeral long running task, in which nodes can be turned off and turned on. This means the stats of individual files need to be persisted somewhere in order to produce the final manifest. This is implemented through sidecars. Each data file exported will contain a "sibling" sidecar file named `_clickhouse_export_part_sidecar.avro`. ClickHouse does not clean up these files, and they can be safely deleted once the data is comitted. + +#### Source partition key compatibility + +The source partition must not be split in the destination. This is validated at schedule time through two mechanisms: + +1. Structural match: in case the source and destination are identical, the destination expression is a subset of the source expression or the destination expression can be entirely computed using only constants and the exact values guaranteed (pinned) by the source. +2. Dynamic proof: the destination expression is monotonic over the source partition min/max range. + +### On plain object storage exports: + +Each MergeTree part will become a separate file with the following name convention: `//_.`. To ensure atomicity, a commit file containing the relative paths of all exported parts is also shipped. A data file should only be considered part of the dataset if a commit file references it. The commit file will be named using the following convention: `/commit__`. + +## Syntax + +```sql +ALTER TABLE [database.]table_name +EXPORT PARTITION ID 'partition_id' +TO TABLE [destination_database.]destination_table +[SETTINGS setting_name = value, ...] +``` + +### Parameters + +- **`table_name`**: The source Replicated*MergeTree table containing the partition to export +- **`partition_id`**: The partition identifier to export (e.g., `'2020'`, `'2021'`) +- **`destination_table`**: The target table for the export (typically an S3, Azure, or other object storage table) + +## Requirements + +`EXPORT PARTITION` exports each part via the same mechanism as [`EXPORT PART`](/docs/en/antalya/part_export.md#requirements), so the source and destination tables must satisfy the same compatibility requirements. Column names may differ (columns are matched by position, not by name), and column types may differ as long as they are safely castable (or `export_merge_tree_part_allow_lossy_cast = 1` is set). Beyond that, the following requirements apply: + +1. **Column count** - source and destination must have the same number of columns by default. Set `export_merge_tree_part_schema_mismatch_mode = 'ignore_extra_source_columns_by_position'` to allow a source table with extra trailing columns; the destination having more columns than the source is still rejected in this mode. +2. **`PARTITION BY` expressions** - the whole source partition must land in a single destination partition. Identical expressions always satisfy this; otherwise the destination expression has to be computable from the values the source partition key pins, or be proven single-valued over the partition's min/max range. The same requirement applies to the partition fields and transforms of an Apache Iceberg destination. See [Source partition key compatibility](#source-partition-key-compatibility). +3. **Partition key column positions and layouts** - every top-level column that provides a column or subcolumn used by the source table's partition key must have the same name at the same position in the destination table's schema. Named `Tuple` elements within such a column must also be declared in the same order, including tuples nested inside `Array` or `Map`. This applies even if both tables' `PARTITION BY` expressions are textually identical. See [`EXPORT PART` requirements](/docs/en/antalya/part_export.md#requirements) for a worked example and the corresponding exception message. + +## Settings + +### Server Settings + +#### `allow_experimental_export_merge_tree_partition` (Required) + +- **Type**: `Bool` +- **Default**: `false` +- **Description**: Enable export replicated merge tree partition feature. It is experimental and not yet ready for production use. + +### Query Settings + +#### `export_merge_tree_partition_force_export` (Optional) + +- **Type**: `Bool` +- **Default**: `false` +- **Description**: Ignore existing partition export and overwrite the ZooKeeper entry. Allows re-exporting a partition that was already exported to the same destination. **IMPORTANT:** this is dangerous because it can lead to duplicated data, use it with caution. + +#### `export_merge_tree_partition_retry_initial_backoff_seconds` (Optional) + +- **Type**: `UInt64` +- **Default**: `5` +- **Description**: Initial delay (in seconds) before retrying a failed part export. The delay grows exponentially with the per-replica retry count (`delay = min(initial << (attempts - 1), max)`). The back-off is per-replica in-memory state: it only spaces this replica's retries out in time and never prevents another replica from attempting the same part. Retryable failures (transient memory/network/object-storage/Keeper errors) are retried until the task succeeds or `export_merge_tree_partition_task_timeout_seconds` elapses, while non-retryable failures (e.g. schema/type incompatibilities) fail the task immediately. + +#### `export_merge_tree_partition_retry_max_backoff_seconds` (Optional) + +- **Type**: `UInt64` +- **Default**: `300` +- **Description**: Maximum delay (in seconds) between retries of a failed part export. Caps the exponential growth controlled by `export_merge_tree_partition_retry_initial_backoff_seconds`. + +#### `export_merge_tree_part_file_already_exists_policy` (Optional) + +- **Type**: `MergeTreePartExportFileAlreadyExistsPolicy` +- **Default**: `skip` +- **Description**: Policy for handling files that already exist during export. Possible values: + - `skip` - Skip the file if it already exists + - `error` - Throw an error if the file already exists + - `overwrite` - Overwrite the file + +### `export_merge_tree_part_throw_on_pending_mutations` (Optional) + +- **Type**: `bool` +- **Default**: `true` +- **Description**: If set to true, throws if pending mutations exists for a given part. Note that by default mutations are applied to all parts, which means that if a mutation in practice would only affetct part/partition x, all the other parts/partition will throw upon export. The exception is when the `IN PARTITION` clause was used in the mutation command. Note the `IN PARTITION` clause is not properly implemented for plain MergeTree tables. + +### `export_merge_tree_part_throw_on_pending_patch_parts` (Optional) + +- **Type**: `bool` +- **Default**: `true` +- **Description**: If set to true, throws if pending patch parts exists for a given part. Note that by default mutations are applied to all parts, which means that if a mutation in practice would only affetct part/partition x, all the other parts/partition will throw upon export. The exception is when the `IN PARTITION` clause was used in the mutation command. Note the `IN PARTITION` clause is not properly implemented for plain MergeTree tables. + +### `export_merge_tree_part_filename_pattern` (Optional) + +- **Type**: `String` +- **Default**: `{part_name}_{checksum}` +- **Description**: Pattern for the filename of the exported merge tree part. The `part_name` and `checksum` are calculated and replaced on the fly. Additional macros are supported. + +### `export_merge_tree_partition_task_timeout_seconds` (Optional) + +- **Type**: `UInt64` +- **Default**: `3600` +- **Description**: The timeout is measured from the manifest's create_time. Set to 0 to disable the timeout. +When the timeout is exceeded the task transitions to KILLED (same terminal state as `KILL QUERY ... EXPORT PARTITION`), and a `last_exception_per_replica` entry on the replica that fires the timeout is populated with a timeout reason. + +Notes: +- Enforcement is best-effort: actual kill latency is bounded by one manifest-updater poll cycle (~30s) plus ZooKeeper watch propagation. + +### `export_merge_tree_part_allow_lossy_cast` (Optional) + +- **Type**: `Bool` +- **Default**: `false` +- **Description**: Allow `EXPORT PART`/`EXPORT PARTITION` to apply lossy (non-value-preserving) casts when the source and destination column types differ. When disabled, an export that would require a lossy cast throws instead. + + When exporting to Apache Iceberg, the partition value written to the metadata is derived from the source partition columns by casting them to the destination partition-field types and applying the destination partition transform — the same computation the exported data files use. This keeps the Iceberg metadata consistent with the data files. + + **Warning:** A lossy cast on a partition column remains semantically truncating. For example, if a table is partitioned by an `Int64` column and some partition values do not fit into a destination `Int32` partition column, both the data files and the Iceberg metadata will contain the truncated `Int32` value (they agree with each other, but the original `Int64` value is lost). Such casts require `export_merge_tree_part_allow_lossy_cast = 1`. + +### `export_merge_tree_part_schema_mismatch_mode` (Optional) + +- **Type**: `MergeTreePartExportSchemaMismatchMode` +- **Default**: `strict` +- **Description**: Controls whether `EXPORT PART`/`EXPORT PARTITION` allows a column-count mismatch between the source `MergeTree` table and the destination table. Columns are matched positionally, like `INSERT INTO dest SELECT * FROM src`. Possible values: + - `strict` - the source and destination must have the same number of columns. A mismatch in either direction throws `NUMBER_OF_COLUMNS_DOESNT_MATCH`. + - `ignore_extra_source_columns_by_position` - the source may have more columns than the destination. The extra trailing source columns (by position) are dropped and not exported. The destination having more columns than the source is still rejected in this mode. + + The extra trailing source columns are still read and evaluated (including `MATERIALIZED`/`ALIAS` columns, and any column another kept column's `ALIAS`/`MATERIALIZED` expression depends on) before being dropped, so this setting only changes which columns end up in the destination, not what is computed while reading the part. + +## Examples + +### Basic Export to S3 + +```sql +CREATE TABLE rmt_table (id UInt64, year UInt16) +ENGINE = ReplicatedMergeTree('/clickhouse/tables/{database}/rmt_table', 'replica1') +PARTITION BY year ORDER BY tuple(); + +CREATE TABLE s3_table (id UInt64, year UInt16) +ENGINE = S3(s3_conn, filename='data', format=Parquet, partition_strategy='hive') +PARTITION BY year; + +INSERT INTO rmt_table VALUES (1, 2020), (2, 2020), (3, 2020), (4, 2021); + +ALTER TABLE rmt_table EXPORT PARTITION ID '2020' TO TABLE s3_table; + +## Killing Exports + +You can cancel in-progress partition exports using the `KILL EXPORT PARTITION` command: + +```sql +KILL EXPORT PARTITION +WHERE partition_id = '2020' + AND source_table = 'rmt_table' + AND destination_table = 's3_table' +``` + +The `WHERE` clause filters exports from the `system.replicated_partition_exports` table. You can use any columns from that table in the filter. + +## Monitoring + +### Active and Completed Exports + +Monitor partition exports using the `system.replicated_partition_exports` table: + +```sql +arthur :) select * from system.replicated_partition_exports Format Vertical; + +SELECT * +FROM system.replicated_partition_exports +FORMAT Vertical + +Query id: 9efc271a-a501-44d1-834f-bc4d20156164 + +Row 1: +────── +source_database: default +source_table: replicated_source +destination_database: default +destination_table: s3_destination +create_time: 2025-11-21 18:21:51 +partition_id: 2022 +transaction_id: 9b2c1e5a-3f47-4c8e-8a1d-6f0b2d4e7c31 +query_id: 3fa3c8d3-7d6b-4f8b-9aa2-2c1f1ad0a111 +source_replica: r1 +parts: ['2022_0_0_0','2022_1_1_0','2022_2_2_0'] +parts_count: 3 +parts_to_do: 0 +status: COMPLETED +last_exception_per_replica: [] +exception_count: 0 +destination_file_paths: {'2022_0_0_0':['data/year=2022/2022_0_0_0_.parquet'],'2022_1_1_0':['data/year=2022/2022_1_1_0_.parquet'],'2022_2_2_0':['data/year=2022/2022_2_2_0_.parquet']} +committed_metadata_file: +committed_manifest_list: +committed_manifest_file: +committed_marker_file: data/commit_2022_9b2c1e5a-3f47-4c8e-8a1d-6f0b2d4e7c31 + +Row 2: +────── +source_database: default +source_table: replicated_source +destination_database: default +destination_table: iceberg_destination +create_time: 2025-11-21 18:20:35 +partition_id: 2021 +transaction_id: d0e4f7a2-8c19-4b6d-9e3a-1f5c7b2e9d40 +query_id: 1c8e0fd0-6a3a-4d6e-9bd6-bdf64adfe118 +source_replica: r2 +parts: ['2021_0_0_0'] +parts_count: 1 +parts_to_do: 0 +status: COMPLETED +last_exception_per_replica: [('r1','Code: 999. Coordination::Exception: Session expired','2021_0_0_0','2025-11-21 18:20:42',1)] +exception_count: 1 +destination_file_paths: {'2021_0_0_0':['data/year=2021/2021_0_0_0_.parquet']} +committed_metadata_file: data/metadata/v3.metadata.json +committed_manifest_list: data/metadata/snap-4029103741930112856-1-.avro +committed_manifest_file: data/metadata/-m0.avro +committed_marker_file: + +2 rows in set. Elapsed: 0.019 sec. + +arthur :) +``` + +Status values include: +- `PENDING` - Export is queued / in progress +- `COMPLETED` - Export finished successfully +- `FAILED` - Export failed +- `KILLED` - Export was cancelled + +### Exception columns + +- `last_exception_per_replica` is an `Array(Tuple(replica String, message String, part String, time DateTime, count UInt64))`. Each tuple is the most recent exception observed by a single replica plus a best-effort within-replica `count`. Replicas that have never reported an exception are omitted. +- `exception_count` is the sum of every `count` in `last_exception_per_replica`. Each replica owns its own counter, so cross-replica updates do not race; the sum is exact w.r.t. the snapshot returned. Within a single replica concurrent failing writers may under-count by one. + +### Per-part destination file paths + +- `destination_file_paths` is a `Map(String, Array(String))` keyed by source part name. Each value is the list of file paths written to the destination object storage when that part was exported (a single part can produce multiple files depending on `max_bytes` / `max_rows`). If a refresh cannot read a processed entry from ZooKeeper, the affected key holds the sentinel `` instead of silently under-counting. + +### Commit info columns + +These columns surface paths produced by the destination storage during commit, so it is possible to inspect what was written without consulting the destination directly: + +- `committed_metadata_file` — for Iceberg destinations: path of the new `vN.metadata.json` written by the commit. Empty for non-Iceberg destinations and before the commit lands. If the commit was already finished by a previous run (detected via the transaction id stored in the snapshot summary), this column carries a human-readable sentinel string instead of a path because the original committer's paths are not recoverable from inside the impl. +- `committed_manifest_list` — for Iceberg destinations: path of the manifest list file (`snap-*.avro`) referenced by the new snapshot. Empty under the same conditions as `committed_metadata_file`. +- `committed_manifest_file` — for Iceberg destinations: path of the manifest file referenced by `committed_manifest_list`. Empty under the same conditions as `committed_metadata_file`. +- `committed_marker_file` — for plain object storage destinations: path of the per-transaction commit marker file written by the destination. Empty for Iceberg destinations and for tasks that have not committed yet. + +To pick the latest exception across replicas: + +```sql +SELECT + arraySort(x -> -x.time, last_exception_per_replica)[1] AS latest_exception +FROM system.replicated_partition_exports +WHERE source_table = 'rmt_table' AND destination_table = 's3_table'; +``` + +## Related Features + +- [ALTER TABLE EXPORT PART](/docs/en/antalya/part_export.md) - Export individual parts (non-replicated) diff --git a/docs/en/antalya/swarm.md b/docs/en/antalya/swarm.md new file mode 100644 index 000000000000..a26f9de26e0a --- /dev/null +++ b/docs/en/antalya/swarm.md @@ -0,0 +1,73 @@ +# Antalya branch + +## Swarm + +### Difference with upstream version + +#### `storage_type` argument in object storage functions + +In upstream ClickHouse, there are several table functions to read Iceberg tables from different storage backends such as `icebergLocal`, `icebergS3`, `icebergAzure`, `icebergHDFS`, cluster variants, the `iceberg` function as a synonym for `icebergS3`, and table engines like `IcebergLocal`, `IcebergS3`, `IcebergAzure`, `IcebergHDFS`. + +In the Antalya branch, the `iceberg` table function and the `Iceberg` table engine unify all variants into one by using a new named argument, `storage_type`, which can be one of `local`, `s3`, `azure`, or `hdfs`. + +Old syntax examples: + +```sql +SELECT * FROM icebergS3('http://minio1:9000/root/table_data', 'minio', 'minio123', 'Parquet'); +SELECT * FROM icebergAzureCluster('mycluster', 'http://azurite1:30000/devstoreaccount1', 'cont', '/table_data', 'devstoreaccount1', 'Eby8vdM02xNOcqFlqUwJPLlmEtlCDXJ1OUzFT50uSRZ6IFsuFq2UVErCz4I6tq/K1SZFPTOtr/KBHBeksoGMGw==', 'Parquet'); +CREATE TABLE mytable ENGINE=IcebergHDFS('/table_data', 'Parquet'); +``` + +New syntax examples: + +```sql +SELECT * FROM iceberg(storage_type='s3', 'http://minio1:9000/root/table_data', 'minio', 'minio123', 'Parquet'); +SELECT * FROM icebergCluster('mycluster', storage_type='azure', 'http://azurite1:30000/devstoreaccount1', 'cont', '/table_data', 'devstoreaccount1', 'Eby8vdM02xNOcqFlqUwJPLlmEtlCDXJ1OUzFT50uSRZ6IFsuFq2UVErCz4I6tq/K1SZFPTOtr/KBHBeksoGMGw==', 'Parquet'); +CREATE TABLE mytable ENGINE=Iceberg('/table_data', 'Parquet', storage_type='hdfs'); +``` + +Also, if a named collection is used to store access parameters, the field `storage_type` can be included in the same named collection: + +```xml + + + http://minio1:9001/root/ + minio + minio123 + s3 + + +``` + +```sql +SELECT * FROM iceberg(s3, filename='table_data'); +``` + +By default `storage_type` is `'s3'` to maintain backward compatibility. + + +#### `object_storage_cluster` setting + +The new setting `object_storage_cluster` controls whether a single-node or cluster variant of table functions reading from object storage (e.g., `s3`, `azure`, `iceberg`, and their cluster variants like `s3Cluster`, `azureCluster`, `icebergCluster`) is used. + +Old syntax examples: + +```sql +SELECT * from s3Cluster('myCluster', 'http://minio1:9001/root/data/{clickhouse,database}/*', 'minio', 'minio123', 'CSV', + 'name String, value UInt32, polygon Array(Array(Tuple(Float64, Float64)))'); +SELECT * FROM icebergAzureCluster('mycluster', 'http://azurite1:30000/devstoreaccount1', 'cont', '/table_data', 'devstoreaccount1', 'Eby8vdM02xNOcqFlqUwJPLlmEtlCDXJ1OUzFT50uSRZ6IFsuFq2UVErCz4I6tq/K1SZFPTOtr/KBHBeksoGMGw==', 'Parquet'); +``` + +New syntax examples: + +```sql +SELECT * from s3('http://minio1:9001/root/data/{clickhouse,database}/*', 'minio', 'minio123', 'CSV', + 'name String, value UInt32, polygon Array(Array(Tuple(Float64, Float64)))') + SETTINGS object_storage_cluster='myCluster'; +SELECT * FROM icebergAzure('http://azurite1:30000/devstoreaccount1', 'cont', '/table_data', 'devstoreaccount1', 'Eby8vdM02xNOcqFlqUwJPLlmEtlCDXJ1OUzFT50uSRZ6IFsuFq2UVErCz4I6tq/K1SZFPTOtr/KBHBeksoGMGw==', 'Parquet') + SETTINGS object_storage_cluster='myCluster'; +``` + +This setting also applies to table engines and can be used with tables managed by Iceberg Catalog. + +Note: The upstream ClickHouse has introduced analogous settings, such as `parallel_replicas_for_cluster_engines` and `cluster_for_parallel_replicas`. Since version 25.10, these settings work with table engines. It is possible that in the future, the `object_storage_cluster` setting will be deprecated. diff --git a/docs/en/operations/system-tables/exports.md b/docs/en/operations/system-tables/exports.md new file mode 100644 index 000000000000..e26514364008 --- /dev/null +++ b/docs/en/operations/system-tables/exports.md @@ -0,0 +1,56 @@ +--- +description: 'System table containing information about in progress merge tree part exports' +keywords: ['system table', 'exports', 'merge tree', 'part'] +slug: /operations/system-tables/exports +title: 'system.exports' +--- + +Contains information about in progress merge tree part exports + +Columns: + +- `source_database` ([String](/docs/en/sql-reference/data-types/string.md)) — Name of the source database. +- `source_table` ([String](/docs/en/sql-reference/data-types/string.md)) — Name of the source table. +- `destination_database` ([String](/docs/en/sql-reference/data-types/string.md)) — Name of the destination database. +- `destination_table` ([String](/docs/en/sql-reference/data-types/string.md)) — Name of the destination table. +- `create_time` ([DateTime](/docs/en/sql-reference/data-types/datetime.md)) — Date and time when the export command was received in the server. +- `part_name` ([String](/docs/en/sql-reference/data-types/string.md)) — Name of the part. +- `destination_file_path` ([String](/docs/en/sql-reference/data-types/string.md)) — File path relative to where the part is being exported to. +- `elapsed` ([Float64](/docs/en/sql-reference/data-types/float.md)) — The time elapsed (in seconds) since the export started. +- `rows_read` ([UInt64](/docs/en/sql-reference/data-types/int-uint.md)) — The number of rows read from the exported part. +- `total_rows_to_read` ([UInt64](/docs/en/sql-reference/data-types/int-uint.md)) — The total number of rows to read from the exported part. +- `total_size_bytes_compressed` ([UInt64](/docs/en/sql-reference/data-types/int-uint.md)) — The total size of the compressed data in the exported part. +- `total_size_bytes_uncompressed` ([UInt64](/docs/en/sql-reference/data-types/int-uint.md)) — The total size of the uncompressed data in the exported part. +- `bytes_read_uncompressed` ([UInt64](/docs/en/sql-reference/data-types/int-uint.md)) — The number of uncompressed bytes read from the exported part. +- `memory_usage` ([UInt64](/docs/en/sql-reference/data-types/int-uint.md)) — Current memory usage in bytes for the export operation. +- `peak_memory_usage` ([UInt64](/docs/en/sql-reference/data-types/int-uint.md)) — Peak memory usage in bytes during the export operation. + +**Example** + +```sql +arthur :) select * from system.exports; + +SELECT * +FROM system.exports + +Query id: 2026718c-d249-4208-891b-a271f1f93407 + +Row 1: +────── +source_database: default +source_table: source_mt_table +destination_database: default +destination_table: destination_table +create_time: 2025-11-19 09:09:11 +part_name: 20251016-365_1_1_0 +destination_file_path: table_root/eventDate=2025-10-16/retention=365/20251016-365_1_1_0_17B2F6CD5D3C18E787C07AE3DAF16EB1.parquet +elapsed: 2.04845441 +rows_read: 1138688 -- 1.14 million +total_rows_to_read: 550961374 -- 550.96 million +total_size_bytes_compressed: 37619147120 -- 37.62 billion +total_size_bytes_uncompressed: 138166213721 -- 138.17 billion +bytes_read_uncompressed: 316892925 -- 316.89 million +memory_usage: 596006095 -- 596.01 million +peak_memory_usage: 601239033 -- 601.24 million +``` + diff --git a/docs/en/sql-reference/distribution-on-cluster.md b/docs/en/sql-reference/distribution-on-cluster.md new file mode 100644 index 000000000000..3a9835e23856 --- /dev/null +++ b/docs/en/sql-reference/distribution-on-cluster.md @@ -0,0 +1,23 @@ +# Task distribution in *Cluster family functions + +## Task distribution algorithm + +Table functions such as `s3Cluster`, `azureBlobStorageCluster`, `hdsfCluster`, `icebergCluster`, and table engines like `S3`, `Azure`, `HDFS`, `Iceberg` with the setting `object_storage_cluster` distribute tasks across all cluster nodes or a subset limited by the `object_storage_max_nodes` setting. This setting limits the number of nodes involved in processing a distributed query, randomly selecting nodes for each query. + +A single task corresponds to processing one source file. + +For each file, one cluster node is selected as the primary node using a consistent Rendezvous Hashing algorithm. This algorithm guarantees that: + * The same node is consistently selected as primary for each file, as long as the cluster remains unchanged. + * When the cluster changes (nodes added or removed), only files assigned to those affected nodes change their primary node assignment. + +This improves cache efficiency by minimizing data movement among nodes. + +## `lock_object_storage_task_distribution_ms` setting + +Each node begins processing files for which it is the primary node. After completing its assigned files, a node may take tasks from other nodes, either immediately or after waiting for `lock_object_storage_task_distribution_ms` milliseconds if the primary node does not request new files during that interval. The default value of `lock_object_storage_task_distribution_ms` is 500 milliseconds. This setting balances between caching efficiency and workload redistribution when nodes are imbalanced. + +## `SYSTEM STOP SWARM MODE` command + +If a node needs to shut down gracefully, the command `SYSTEM STOP SWARM MODE` prevents the node from receiving new tasks for *Cluster-family queries. The node finishes processing already assigned files before it can safely shut down without errors. + +Receiving new tasks can be resumed with the command `SYSTEM START SWARM MODE`. diff --git a/docs/guides/oss/deployment-and-scaling/cluster-discovery.mdx b/docs/guides/oss/deployment-and-scaling/cluster-discovery.mdx index 04eef1ed6262..e31df76b5e6a 100644 --- a/docs/guides/oss/deployment-and-scaling/cluster-discovery.mdx +++ b/docs/guides/oss/deployment-and-scaling/cluster-discovery.mdx @@ -62,6 +62,8 @@ Traditionally, in ClickHouse, each shard and replica in the cluster needed to be With Cluster Discovery, rather than defining each node explicitly, you simply specify a path in ZooKeeper. All nodes that register under this path in ZooKeeper will be automatically discovered and added to the cluster. +Discovery settings under `remote_servers` (including `user`, `password`, `secret`, `path`, `multicluster_root_path`, and adding or removing discovery clusters) are applied on configuration reload (for example with `SYSTEM RELOAD CONFIG`). A server restart is not required for these changes. + ```xml diff --git a/docs/reference/engines/database-engines/datalake.mdx b/docs/reference/engines/database-engines/datalake.mdx index eab5a6844d1b..07e732461ff1 100644 --- a/docs/reference/engines/database-engines/datalake.mdx +++ b/docs/reference/engines/database-engines/datalake.mdx @@ -64,6 +64,7 @@ The following settings are supported: | `dlf_access_key_id` | Access key ID for DLF access | | `dlf_access_key_secret` | Access key Secret for DLF access | | `force_add_bucket` | When constructing object-storage URLs from the catalog-provided table location and `storage_endpoint`, prepend the bucket/container name even if the endpoint already contains it. Default: `false`. Set to `true` for catalogs that hand back paths without the bucket and require it to be added at the URL-construction step (Polaris-style paths). | +| `namespaces` | Comma-separated list of namespaces, implemented for catalog types: `rest`, `glue` and `unity` | ## Examples {#examples} @@ -91,4 +92,30 @@ SELECT count() from database_name.table_name; bearer token (scoped to https://storage.azure.com) instead of `onelake_client_id`/`onelake_client_secret`. ClickHouse does not refresh the token, so the database must be recreated after it expires. + +## Namespace filter {#namespace} + +By default, ClickHouse reads tables from all namespaces available in the catalog. You can limit this behavior using the `namespaces` database setting. The value should be a comma‑separated list of namespaces that are allowed to be read. + +Supported catalog types are `rest`, `glue` and `unity`. + +For example, if the catalog contains three namespaces - `dev`, `stage`, and `prod` - and you want to read data only from dev and stage, set: +``` +namespaces='dev,stage' +``` + +### Nested namespaces {#namespace-nested} + +The Iceberg (`rest`) catalog supports nested namespaces. The `namespaces` filter accepts the following patterns: + +- `namespace` - includes tables from the specified namespace, but not from its nested namespaces. +- `namespace.nested` - includes tables from the nested namespace, but not from the parent. +- `namespace.*` - includes tables from all nested namespaces, but not from the parent. + +If you need to include both a namespace and its nested namespaces, specify both explicitly. For example: +``` +namespaces='namespace,namespace.*' +``` + +The default value is '*', which means all namespaces are included. {/*AUTOGENERATED_END*/} diff --git a/docs/reference/engines/table-engines/integrations/iceberg.mdx b/docs/reference/engines/table-engines/integrations/iceberg.mdx index 74ebe2fee57b..7a05dab165b5 100644 --- a/docs/reference/engines/table-engines/integrations/iceberg.mdx +++ b/docs/reference/engines/table-engines/integrations/iceberg.mdx @@ -384,6 +384,62 @@ SETTINGS iceberg_metadata_staleness_ms=120000 **Note**: Current expectation is that metadata cache size is sufficient to hold the latest metadata snapshot in full for all active tables, if asynchronous prefetching is enabled. +## Altinity Antalya branch + +### Specify storage type in arguments + +Only in the Altinity Antalya branch does `Iceberg` table engine support all storage types. The storage type can be specified using the named argument `storage_type`. Supported values are `s3`, `azure`, `hdfs`, and `local`. + +```sql +CREATE TABLE iceberg_table_s3 + ENGINE = Iceberg(storage_type='s3', url, [, NOSIGN | access_key_id, secret_access_key, [session_token]], format, [,compression]) + +CREATE TABLE iceberg_table_azure + ENGINE = Iceberg(storage_type='azure', connection_string|storage_account_url, container_name, blobpath, [account_name, account_key, format, compression]) + +CREATE TABLE iceberg_table_hdfs + ENGINE = Iceberg(storage_type='hdfs', path_to_table, [,format] [,compression_method]) + +CREATE TABLE iceberg_table_local + ENGINE = Iceberg(storage_type='local', path_to_table, [,format] [,compression_method]) +``` + +### Specify storage type in named collection + +Only in Altinity Antalya branch `storage_type` can be included as part of a named collection. This allows for centralized configuration of storage settings. + +```xml + + + + http://test.s3.amazonaws.com/clickhouse-bucket/ + test + test + auto + auto + s3 + + + +``` + +```sql +CREATE TABLE iceberg_table ENGINE=Iceberg(iceberg_conf, filename = 'test_table') +``` + +The default value for `storage_type` is `s3`. + +### The `object_storage_cluster` setting. + +Only in the Altinity Antalya branch is an alternative syntax for the `Iceberg` table engine available. This syntax allows execution on a cluster when the `object_storage_cluster` setting is non-empty and contains the cluster name. + +```sql +CREATE TABLE iceberg_table_s3 + ENGINE = Iceberg(storage_type='s3', url, [, NOSIGN | access_key_id, secret_access_key, [session_token]], format, [,compression]); + +SELECT * FROM iceberg_table_s3 SETTINGS object_storage_cluster='cluster_simple'; +``` + ## See also {#see-also} - [iceberg table function](/reference/functions/table-functions/iceberg) diff --git a/docs/reference/functions/table-functions/azureBlobStorageCluster.mdx b/docs/reference/functions/table-functions/azureBlobStorageCluster.mdx index 1c66ceb83f97..18763cf553d6 100644 --- a/docs/reference/functions/table-functions/azureBlobStorageCluster.mdx +++ b/docs/reference/functions/table-functions/azureBlobStorageCluster.mdx @@ -52,6 +52,20 @@ SELECT count(*) FROM azureBlobStorageCluster( See [azureBlobStorage](/reference/functions/table-functions/azureBlobStorage#using-shared-access-signatures-sas-sas-tokens) for examples. +## Altinity Antalya branch + +### `object_storage_cluster` setting. + +Only in the Altinity Antalya branch, the alternative syntax for the `azureBlobStorageCluster` table function is avilable. This allows the `azureBlobStorage` function to be used with the non-empty `object_storage_cluster` setting, specifying a cluster name. This enables distributed queries over Azure Blob Storage across a ClickHouse cluster. + +```sql +SELECT count(*) FROM azureBlobStorage( + 'http://azurite1:10000/devstoreaccount1', 'testcontainer', 'test_cluster_count.csv', 'devstoreaccount1', + 'Eby8vdM02xNOcqFlqUwJPLlmEtlCDXJ1OUzFT50uSRZ6IFsuFq2UVErCz4I6tq/K1SZFPTOtr/KBHBeksoGMGw==', 'CSV', + 'auto', 'key UInt64') +SETTINGS object_storage_cluster='cluster_simple' +``` + ## Related {#related} - [AzureBlobStorage engine](/reference/engines/table-engines/integrations/azureBlobStorage) diff --git a/docs/reference/functions/table-functions/deltalakeCluster.mdx b/docs/reference/functions/table-functions/deltalakeCluster.mdx index 3484a7cf211c..3a323ac2bc43 100644 --- a/docs/reference/functions/table-functions/deltalakeCluster.mdx +++ b/docs/reference/functions/table-functions/deltalakeCluster.mdx @@ -43,6 +43,17 @@ A table with the specified structure for reading data from cluster in the specif - `_time` — Last modified time of the file. Type: `Nullable(DateTime)`. If the time is unknown, the value is `NULL`. - `_etag` — The etag of the file. Type: `LowCardinality(String)`. If the etag is unknown, the value is `NULL`. +## Altinity Antalya branch + +### `object_storage_cluster` setting. + +Only in the Altinity Antalya branch alternative syntax for `deltaLakeCluster` table function is available. This allows the `deltaLake` function to be used with the non-empty `object_storage_cluster` setting, specifying a cluster name. This enables distributed queries over Delta Lake Storage across a ClickHouse cluster. + +```sql +SELECT count(*) FROM deltaLake(url [,aws_access_key_id, aws_secret_access_key] [,format] [,structure] [,compression]) +SETTINGS object_storage_cluster='cluster_simple' +``` + ## Related {#related} - [deltaLake engine](/reference/engines/table-engines/integrations/deltalake) diff --git a/docs/reference/functions/table-functions/hdfsCluster.mdx b/docs/reference/functions/table-functions/hdfsCluster.mdx index ea30321763a7..8c86b536b408 100644 --- a/docs/reference/functions/table-functions/hdfsCluster.mdx +++ b/docs/reference/functions/table-functions/hdfsCluster.mdx @@ -58,6 +58,18 @@ FROM hdfsCluster('cluster_simple', 'hdfs://hdfs1:9000/{some,another}_dir/*', 'TS If your listing of files contains number ranges with leading zeros, use the construction with braces for each digit separately or use `?`. +## Altinity Antalya branch + +### `object_storage_cluster` setting. + +Only in the Altinity Antalya branch alternative syntax for `hdfsCluster` table function is available. This allows the `hdfs` function to be used with the non-empty `object_storage_cluster` setting, specifying a cluster name. This enables distributed queries over HDFS Storage across a ClickHouse cluster. + +```sql +SELECT count(*) +FROM hdfs('hdfs://hdfs1:9000/{some,another}_dir/*', 'TSV', 'name String, value UInt32') +SETTINGS object_storage_cluster='cluster_simple' +``` + ## Related {#related} - [HDFS engine](/reference/engines/table-engines/integrations/hdfs) diff --git a/docs/reference/functions/table-functions/hudiCluster.mdx b/docs/reference/functions/table-functions/hudiCluster.mdx index e8f0cdf79ba5..a9dfc642ccc5 100644 --- a/docs/reference/functions/table-functions/hudiCluster.mdx +++ b/docs/reference/functions/table-functions/hudiCluster.mdx @@ -42,6 +42,18 @@ A table with the specified structure for reading data from cluster in the specif - `_time` — Last modified time of the file. Type: `Nullable(DateTime)`. If the time is unknown, the value is `NULL`. - `_etag` — The etag of the file. Type: `LowCardinality(String)`. If the etag is unknown, the value is `NULL`. +## Altinity Antalya branch + +### `object_storage_cluster` setting. + +Only in the Altinity Antalya branch alternative syntax for `hudiCluster` table function is available. This allows the `hudi` function to be used with the non-empty `object_storage_cluster` setting, specifying a cluster name. This enables distributed queries over Hudi Storage across a ClickHouse cluster. + +```sql +SELECT * +FROM hudi(url [,aws_access_key_id, aws_secret_access_key] [,format] [,structure] [,compression]) +SETTINGS object_storage_cluster='cluster_simple' +``` + ## Related {#related} - [Hudi engine](/reference/engines/table-engines/integrations/hudi) diff --git a/docs/reference/functions/table-functions/iceberg.mdx b/docs/reference/functions/table-functions/iceberg.mdx index 83384c1cedbe..25c5c9cde65d 100644 --- a/docs/reference/functions/table-functions/iceberg.mdx +++ b/docs/reference/functions/table-functions/iceberg.mdx @@ -730,6 +730,47 @@ The command returns a table with `metric_name` and `metric_value` columns showin - The `older_than` threshold protects against deleting files from in-progress writes — the default 3-day threshold provides a generous safety margin +## Altinity Antalya branch + +### Specify storage type in arguments + +Only in the Altinity Antalya branch does the `iceberg` table function support all storage types. The storage type can be specified using the named argument `storage_type`. Supported values are `s3`, `azure`, `hdfs`, and `local`. + +```sql +iceberg(storage_type='s3', url [, NOSIGN | access_key_id, secret_access_key, [session_token]] [,format] [,compression_method]) + +iceberg(storage_type='azure', connection_string|storage_account_url, container_name, blobpath, [,account_name], [,account_key] [,format] [,compression_method]) + +iceberg(storage_type='hdfs', path_to_table, [,format] [,compression_method]) + +iceberg(storage_type='local', path_to_table, [,format] [,compression_method]) +``` + +### Specify storage type in named collection + +Only in the Altinity Antalya branch can storage_type be included as part of a named collection. This allows for centralized configuration of storage settings. + +```xml + + + + http://test.s3.amazonaws.com/clickhouse-bucket/ + test + test + auto + auto + s3 + + + +``` + +```sql +iceberg(named_collection[, option=value [,..]]) +``` + +The default value for `storage_type` is `s3`. + ## See Also {#see-also} * [Iceberg engine](/reference/engines/table-engines/integrations/iceberg) diff --git a/docs/reference/functions/table-functions/icebergCluster.mdx b/docs/reference/functions/table-functions/icebergCluster.mdx index 453bda2dc1e1..4f50d2cbe508 100644 --- a/docs/reference/functions/table-functions/icebergCluster.mdx +++ b/docs/reference/functions/table-functions/icebergCluster.mdx @@ -55,6 +55,81 @@ SELECT * FROM icebergS3Cluster('cluster_simple', 'http://test.s3.amazonaws.com/c - `_time` — Last modified time of the file. Type: `Nullable(DateTime)`. If the time is unknown, the value is `NULL`. - `_etag` — The etag of the file. Type: `LowCardinality(String)`. If the etag is unknown, the value is `NULL`. +## Altinity Antalya branch + +### `icebergLocalCluster` table function + +Only in the Altinity Antalya branch, `icebergLocalCluster` designed to make distributed cluster queries when Iceberg data is stored on shared network storage mounted with a local path. The path must be identical on all replicas. + +```sql +icebergLocalCluster(cluster_name, path_to_table, [,format] [,compression_method]) +``` + +### Specify storage type in function arguments + +Only in the Altinity Antalya branch, the `icebergCluster` table function supports all storage backends. The storage backend can be specified using the named argument `storage_type`. Valid values include `s3`, `azure`, `hdfs`, and `local`. + +```sql +icebergCluster(storage_type='s3', cluster_name, url [, NOSIGN | access_key_id, secret_access_key, [session_token]] [,format] [,compression_method]) + +icebergCluster(storage_type='azure', cluster_name, connection_string|storage_account_url, container_name, blobpath, [,account_name], [,account_key] [,format] [,compression_method]) + +icebergCluster(storage_type='hdfs', cluster_name, path_to_table, [,format] [,compression_method]) + +icebergCluster(storage_type='local', cluster_name, path_to_table, [,format] [,compression_method]) +``` + +### Specify storage type in a named collection + +Only in the Altinity Antalya branch, `storage_type` can be part of a named collection. + +```xml + + + + http://test.s3.amazonaws.com/clickhouse-bucket/ + test + test + auto + auto + s3 + + + +``` + +```sql +icebergCluster(iceberg_conf[, option=value [,..]]) +``` + +The default value for `storage_type` is `s3`. + +### `object_storage_cluster` setting. + +Only in the Altinity Antalya branch, an alternative syntax for `icebergCluster` table function is available. This allows the `iceberg` function to be used with the non-empty `object_storage_cluster` setting, specifying a cluster name. This enables distributed queries over Iceberg table across a ClickHouse cluster. + +```sql +icebergS3(url [, NOSIGN | access_key_id, secret_access_key, [session_token]] [,format] [,compression_method]) SETTINGS object_storage_cluster='cluster_name' + +icebergAzure(connection_string|storage_account_url, container_name, blobpath, [,account_name], [,account_key] [,format] [,compression_method]) SETTINGS object_storage_cluster='cluster_name' + +icebergHDFS(path_to_table, [,format] [,compression_method]) SETTINGS object_storage_cluster='cluster_name' + +icebergLocal(path_to_table, [,format] [,compression_method]) SETTINGS object_storage_cluster='cluster_name' + +icebergS3(option=value [,..]) SETTINGS object_storage_cluster='cluster_name' + +iceberg(storage_type='s3', url [, NOSIGN | access_key_id, secret_access_key, [session_token]] [,format] [,compression_method]) SETTINGS object_storage_cluster='cluster_name' + +iceberg(storage_type='azure', connection_string|storage_account_url, container_name, blobpath, [,account_name], [,account_key] [,format] [,compression_method]) SETTINGS object_storage_cluster='cluster_name' + +iceberg(storage_type='hdfs', path_to_table, [,format] [,compression_method]) SETTINGS object_storage_cluster='cluster_name' + +iceberg(storage_type='local', path_to_table, [,format] [,compression_method]) SETTINGS object_storage_cluster='cluster_name' + +iceberg(iceberg_conf[, option=value [,..]]) SETTINGS object_storage_cluster='cluster_name' +``` + **See Also** - [Iceberg engine](/reference/engines/table-engines/integrations/iceberg) diff --git a/docs/reference/functions/table-functions/s3Cluster.mdx b/docs/reference/functions/table-functions/s3Cluster.mdx index 14f38222fb1c..00aeaaac20e8 100644 --- a/docs/reference/functions/table-functions/s3Cluster.mdx +++ b/docs/reference/functions/table-functions/s3Cluster.mdx @@ -89,6 +89,23 @@ Users can use the same approaches as document for the s3 function [here](/refere For details on optimizing the performance of the s3 function see [our detailed guide](/integrations/connectors/data-ingestion/AWS/performance). +## Altinity Antalya branch + +### `object_storage_cluster` setting. + +Only in the Altinity Antalya branch alternative syntax for `s3Cluster` table function is available. This allows the `s3` function to be used with the non-empty `object_storage_cluster` setting, specifying a cluster name. This enables distributed queries over S3 Storage across a ClickHouse cluster. + +```sql +SELECT * FROM s3( + 'http://minio1:9001/root/data/{clickhouse,database}/*', + 'minio', + 'ClickHouse_Minio_P@ssw0rd', + 'CSV', + 'name String, value UInt32, polygon Array(Array(Tuple(Float64, Float64)))' +) ORDER BY (name, value, polygon) +SETTINGS object_storage_cluster='cluster_simple' +``` + ## Related {#related} - [S3 engine](/reference/engines/table-engines/integrations/s3) diff --git a/docs/reference/statements/system.mdx b/docs/reference/statements/system.mdx index e2f4fde13d7c..9e083a92e529 100644 --- a/docs/reference/statements/system.mdx +++ b/docs/reference/statements/system.mdx @@ -340,6 +340,12 @@ Normally shuts down ClickHouse (like `service clickhouse-server stop` / `kill {$ Aborts ClickHouse process (like `kill -9 {$ pid_clickhouse-server}`) +## SYSTEM PRESHUTDOWN {#preshutdown} + + + +Prepare node for graceful shutdown. Unregister in autodiscovered clusters, stop accepting distributed requests to object storages (s3Cluster, icebergCluster, etc.). + ## SYSTEM INSTRUMENT {#instrument} Manages instrumentation points using LLVM's XRay feature which is available when ClickHouse is built using `ENABLE_XRAY=1`. diff --git a/programs/server/Server.cpp b/programs/server/Server.cpp index 1af6f390f008..c8fcfd0a50e8 100644 --- a/programs/server/Server.cpp +++ b/programs/server/Server.cpp @@ -101,6 +101,7 @@ #include #include #include +#include #include #include #include @@ -498,6 +499,9 @@ namespace ServerSetting extern const ServerSettingsString hdfs_libhdfs3_conf; extern const ServerSettingsString config_file; extern const ServerSettingsString users_to_ignore_early_memory_limit_check; + extern const ServerSettingsUInt64 object_storage_list_objects_cache_ttl; + extern const ServerSettingsUInt64 object_storage_list_objects_cache_size; + extern const ServerSettingsUInt64 object_storage_list_objects_cache_max_entries; } namespace ErrorCodes @@ -508,6 +512,9 @@ namespace ErrorCodes namespace FileCacheSetting { extern const FileCacheSettingsBool load_metadata_asynchronously; + extern const ServerSettingsUInt64 object_storage_list_objects_cache_size; + extern const ServerSettingsUInt64 object_storage_list_objects_cache_max_entries; + extern const ServerSettingsUInt64 object_storage_list_objects_cache_ttl; } } @@ -3344,6 +3351,8 @@ try } + global_context->startSwarmMode(); + { std::lock_guard lock(servers_lock); /// We should start interserver communications before (and more important shutdown after) tables. @@ -3540,6 +3549,10 @@ try /// try set up encryption. There are some errors in config, error will be printed and server wouldn't start. CompressionCodecEncrypted::Configuration::instance().load(config(), "encryption_codecs"); + ObjectStorageListObjectsCache::instance().setMaxSizeInBytes(server_settings[ServerSetting::object_storage_list_objects_cache_size]); + ObjectStorageListObjectsCache::instance().setMaxCount(server_settings[ServerSetting::object_storage_list_objects_cache_max_entries]); + ObjectStorageListObjectsCache::instance().setTTL(server_settings[ServerSetting::object_storage_list_objects_cache_ttl]); + auto replicas_reconnector = ReplicasReconnector::init(global_context); /// Set current database name before loading tables and databases because @@ -3913,6 +3926,8 @@ try is_cancelled = true; + global_context->stopSwarmMode(); + LOG_DEBUG(log, "Waiting for current connections to close."); size_t current_connections = 0; diff --git a/src/Access/Common/AccessType.h b/src/Access/Common/AccessType.h index 974ecadf3c80..768721c15813 100644 --- a/src/Access/Common/AccessType.h +++ b/src/Access/Common/AccessType.h @@ -220,6 +220,8 @@ enum class AccessType : uint8_t M(ALTER_REWRITE_PARTS, "REWRITE PARTS", TABLE, ALTER_TABLE) /* allows to execute ALTER REWRITE PARTS */\ M(ALTER_SETTINGS, "ALTER SETTING, ALTER MODIFY SETTING, MODIFY SETTING, RESET SETTING", TABLE, ALTER_TABLE) /* allows to execute ALTER MODIFY SETTING */\ M(ALTER_MOVE_PARTITION, "ALTER MOVE PART, MOVE PARTITION, MOVE PART", TABLE, ALTER_TABLE) \ + M(ALTER_EXPORT_PART, "ALTER EXPORT PART, EXPORT PART", TABLE, ALTER_TABLE) \ + M(ALTER_EXPORT_PARTITION, "ALTER EXPORT PARTITION, EXPORT PARTITION", TABLE, ALTER_TABLE) \ M(ALTER_FETCH_PARTITION, "ALTER FETCH PART, FETCH PARTITION", TABLE, ALTER_TABLE) \ M(ALTER_FREEZE_PARTITION, "FREEZE PARTITION, UNFREEZE", TABLE, ALTER_TABLE) \ M(ALTER_UNLOCK_SNAPSHOT, "UNLOCK SNAPSHOT", TABLE, ALTER_TABLE) \ @@ -352,6 +354,7 @@ enum class AccessType : uint8_t M(SYSTEM_DROP_SCHEMA_CACHE, "SYSTEM CLEAR SCHEMA CACHE, SYSTEM DROP SCHEMA CACHE, DROP SCHEMA CACHE", GLOBAL, SYSTEM_DROP_CACHE) \ M(SYSTEM_DROP_FORMAT_SCHEMA_CACHE, "SYSTEM CLEAR FORMAT SCHEMA CACHE, SYSTEM DROP FORMAT SCHEMA CACHE, DROP FORMAT SCHEMA CACHE", GLOBAL, SYSTEM_DROP_CACHE) \ M(SYSTEM_DROP_S3_CLIENT_CACHE, "SYSTEM CLEAR S3 CLIENT CACHE, SYSTEM DROP S3 CLIENT, DROP S3 CLIENT CACHE", GLOBAL, SYSTEM_DROP_CACHE) \ + M(SYSTEM_DROP_OBJECT_STORAGE_LIST_OBJECTS_CACHE, "SYSTEM DROP OBJECT STORAGE LIST OBJECTS CACHE", GLOBAL, SYSTEM_DROP_CACHE) \ M(SYSTEM_DROP_CACHE, "DROP CACHE", GROUP, SYSTEM) \ M(SYSTEM_RELOAD_CONFIG, "RELOAD CONFIG", GLOBAL, SYSTEM_RELOAD) \ M(SYSTEM_RELOAD_USERS, "RELOAD USERS", GLOBAL, SYSTEM_RELOAD) \ @@ -368,6 +371,7 @@ enum class AccessType : uint8_t M(SYSTEM_TTL_MERGES, "SYSTEM STOP TTL MERGES, SYSTEM START TTL MERGES, STOP TTL MERGES, START TTL MERGES", TABLE, SYSTEM) \ M(SYSTEM_FETCHES, "SYSTEM STOP FETCHES, SYSTEM START FETCHES, STOP FETCHES, START FETCHES", TABLE, SYSTEM) \ M(SYSTEM_MOVES, "SYSTEM STOP MOVES, SYSTEM START MOVES, STOP MOVES, START MOVES", TABLE, SYSTEM) \ + M(SYSTEM_SWARM, "SYSTEM STOP SWARM MODE, SYSTEM START SWARM MODE, STOP SWARM MODE, START SWARM MODE", GLOBAL, SYSTEM) \ M(SYSTEM_PULLING_REPLICATION_LOG, "SYSTEM STOP PULLING REPLICATION LOG, SYSTEM START PULLING REPLICATION LOG", TABLE, SYSTEM) \ M(SYSTEM_CLEANUP, "SYSTEM STOP CLEANUP, SYSTEM START CLEANUP", TABLE, SYSTEM) \ M(SYSTEM_VIEWS, "SYSTEM REFRESH VIEW, SYSTEM START VIEWS, SYSTEM STOP VIEWS, SYSTEM START VIEW, SYSTEM STOP VIEW, SYSTEM PAUSE VIEWS, SYSTEM PAUSE VIEW, SYSTEM CANCEL VIEW, REFRESH VIEW, START VIEWS, STOP VIEWS, START VIEW, STOP VIEW, PAUSE VIEWS, PAUSE VIEW, CANCEL VIEW", VIEW, SYSTEM_BACKGROUND) \ diff --git a/src/Analyzer/FunctionNode.cpp b/src/Analyzer/FunctionNode.cpp index ee91aed921f7..0d160b4d493d 100644 --- a/src/Analyzer/FunctionNode.cpp +++ b/src/Analyzer/FunctionNode.cpp @@ -12,6 +12,7 @@ #include #include +#include #include @@ -168,6 +169,13 @@ void FunctionNode::dumpTreeImpl(WriteBuffer & buffer, FormatState & format_state buffer << '\n' << std::string(indent + 2, ' ') << "WINDOW\n"; getWindowNode()->dumpTreeImpl(buffer, format_state, indent + 4); } + + if (!settings_changes.empty()) + { + buffer << '\n' << std::string(indent + 2, ' ') << "SETTINGS"; + for (const auto & change : settings_changes) + buffer << fmt::format(" {}={}", change.name, fieldToString(change.value)); + } } bool FunctionNode::isEqualImpl(const IQueryTreeNode & rhs, CompareOptions /*compare_options*/) const @@ -175,7 +183,7 @@ bool FunctionNode::isEqualImpl(const IQueryTreeNode & rhs, CompareOptions /*comp const auto & rhs_typed = assert_cast(rhs); if (function_name != rhs_typed.function_name || isAggregateFunction() != rhs_typed.isAggregateFunction() || isOrdinaryFunction() != rhs_typed.isOrdinaryFunction() || isWindowFunction() != rhs_typed.isWindowFunction() - || nulls_action != rhs_typed.nulls_action) + || nulls_action != rhs_typed.nulls_action || settings_changes != rhs_typed.settings_changes) return false; /// is_operator is ignored here because it affects only AST formatting @@ -207,6 +215,17 @@ void FunctionNode::updateTreeHashImpl(HashState & hash_state, CompareOptions /*c hash_state.update(isWindowFunction()); hash_state.update(nulls_action); + hash_state.update(settings_changes.size()); + for (const auto & change : settings_changes) + { + hash_state.update(change.name.size()); + hash_state.update(change.name); + + const auto & value_dump = change.value.dump(); + hash_state.update(value_dump.size()); + hash_state.update(value_dump); + } + /// is_operator is ignored here because it affects only AST formatting if (!isResolved()) @@ -228,6 +247,7 @@ QueryTreeNodePtr FunctionNode::cloneImpl() const result_function->nulls_action = nulls_action; result_function->wrap_with_nullable = wrap_with_nullable; result_function->is_operator = is_operator; + result_function->settings_changes = settings_changes; return result_function; } @@ -291,6 +311,14 @@ ASTPtr FunctionNode::toASTImpl(const ConvertToASTOptions & options) const function_ast->window_definition = window_node->toAST(new_options); } + if (!settings_changes.empty()) + { + auto settings_ast = make_intrusive(); + settings_ast->changes = settings_changes; + settings_ast->is_standalone = false; + function_ast->arguments->children.push_back(settings_ast); + } + return function_ast; } diff --git a/src/Analyzer/FunctionNode.h b/src/Analyzer/FunctionNode.h index 396991a9d860..f72f285480cc 100644 --- a/src/Analyzer/FunctionNode.h +++ b/src/Analyzer/FunctionNode.h @@ -9,6 +9,7 @@ #include #include #include +#include namespace DB { @@ -203,6 +204,18 @@ class FunctionNode final : public IQueryTreeNode wrap_with_nullable = true; } + /// Get settings changes passed to table function + const SettingsChanges & getSettingsChanges() const + { + return settings_changes; + } + + /// Set settings changes passed as last argument to table function + void setSettingsChanges(SettingsChanges settings_changes_) + { + settings_changes = std::move(settings_changes_); + } + void dumpTreeImpl(WriteBuffer & buffer, FormatState & format_state, size_t indent) const override; protected: @@ -227,6 +240,8 @@ class FunctionNode final : public IQueryTreeNode static constexpr size_t arguments_child_index = 1; static constexpr size_t window_child_index = 2; static constexpr size_t children_size = window_child_index + 1; + + SettingsChanges settings_changes; }; } diff --git a/src/Analyzer/FunctionSecretArgumentsFinderTreeNode.h b/src/Analyzer/FunctionSecretArgumentsFinderTreeNode.h index 6afdd5d3a952..2d171015192b 100644 --- a/src/Analyzer/FunctionSecretArgumentsFinderTreeNode.h +++ b/src/Analyzer/FunctionSecretArgumentsFinderTreeNode.h @@ -83,8 +83,14 @@ class FunctionTreeNodeImpl : public AbstractFunction { public: explicit ArgumentsTreeNode(const QueryTreeNodes * arguments_) : arguments(arguments_) {} - size_t size() const override { return arguments ? arguments->size() : 0; } - std::unique_ptr at(size_t n) const override { return std::make_unique(arguments->at(n).get()); } + size_t size() const override + { /// size withous skipped indexes + return arguments ? arguments->size() - skippedSize() : 0; + } + std::unique_ptr at(size_t n) const override + { /// n is relative index, some can be skipped + return std::make_unique(arguments->at(getRealIndex(n)).get()); + } private: const QueryTreeNodes * arguments = nullptr; }; diff --git a/src/Analyzer/QueryTreeBuilder.cpp b/src/Analyzer/QueryTreeBuilder.cpp index 7c0cd9ca3444..7f727724eecd 100644 --- a/src/Analyzer/QueryTreeBuilder.cpp +++ b/src/Analyzer/QueryTreeBuilder.cpp @@ -713,7 +713,12 @@ QueryTreeNodePtr QueryTreeBuilder::buildExpression(const ASTPtr & expression, co { const auto & function_arguments_list = function->arguments->as()->children; for (const auto & argument : function_arguments_list) - function_node->getArguments().getNodes().push_back(buildExpression(argument, context)); + { + if (const auto * ast_set = argument->as()) + function_node->setSettingsChanges(ast_set->changes); + else + function_node->getArguments().getNodes().push_back(buildExpression(argument, context)); + } } if (function->isWindowFunction()) diff --git a/src/Analyzer/Resolve/QueryAnalyzer.cpp b/src/Analyzer/Resolve/QueryAnalyzer.cpp index 4d11a1aa8156..7840ebc21b98 100644 --- a/src/Analyzer/Resolve/QueryAnalyzer.cpp +++ b/src/Analyzer/Resolve/QueryAnalyzer.cpp @@ -4720,6 +4720,7 @@ void QueryAnalyzer::resolveTableFunction(QueryTreeNodePtr & table_function_node, { auto table_function_node_to_resolve_typed = std::make_shared(table_function_argument_function_name); table_function_node_to_resolve_typed->getArgumentsNode() = table_function_argument_function->getArgumentsNode(); + table_function_node_to_resolve_typed->setSettingsChanges(table_function_argument_function->getSettingsChanges()); QueryTreeNodePtr table_function_node_to_resolve = std::move(table_function_node_to_resolve_typed); if (table_function_argument_function_name == "view" diff --git a/src/Analyzer/Utils.cpp b/src/Analyzer/Utils.cpp index 7344b0435ab6..5d0e5547c7f5 100644 --- a/src/Analyzer/Utils.cpp +++ b/src/Analyzer/Utils.cpp @@ -41,6 +41,7 @@ #include #include +#include #include #include @@ -57,6 +58,7 @@ #include +#include #include namespace DB @@ -1291,24 +1293,79 @@ bool hasUnknownColumn(const QueryTreeNodePtr & node, QueryTreeNodePtr table_expr return false; } -void removeExpressionsThatDoNotDependOnTableIdentifiers( +namespace +{ + +template +bool walkOrdinaryFunctions(const QueryTreeNodePtr & node, KeepFunction && keep_function) +{ + QueryTreeNodes stack = {node}; + while (!stack.empty()) + { + auto current = std::move(stack.back()); + stack.pop_back(); + if (!current) + continue; + + const auto type = current->getNodeType(); + if (type == QueryTreeNodeType::QUERY || type == QueryTreeNodeType::UNION) + return false; + + if (const auto * function = current->as()) + { + if (!function->isOrdinaryFunction()) + return false; + auto function_base = function->getFunction(); + if (!function_base || !keep_function(function_base)) + return false; + } + + for (const auto & child : current->getChildren()) + { + if (child) + stack.push_back(child); + } + } + return true; +} + +bool isSafeToDuplicateInQueryTree(const QueryTreeNodePtr & node) +{ + return walkOrdinaryFunctions( + node, + [](const FunctionBasePtr & function_base) + { + return function_base->isDeterministic() + && function_base->isDeterministicInScopeOfQuery() + && !function_base->isStateful() + && !function_base->isServerConstant() + && !functionIsDictGet(function_base->getName()) + && !functionIsJoinGet(function_base->getName()); + }); +} + +void filterConjunctions( QueryTreeNodePtr & expression, - const QueryTreeNodePtr & table_expression, + const std::function & keep, const ContextPtr & context) { auto * function = expression->as(); if (!function) + { + if (!keep(expression)) + expression = {}; return; + } if (function->getFunctionName() != "and") { - if (hasUnknownColumn(expression, table_expression)) - expression = nullptr; + if (!keep(expression)) + expression = {}; return; } std::deque conjunctions; - std::deque processing{ expression }; + std::deque processing{expression}; while (!processing.empty()) { @@ -1318,10 +1375,7 @@ void removeExpressionsThatDoNotDependOnTableIdentifiers( if (auto * function_node = node->as()) { if (function_node->getFunctionName() == "and") - std::ranges::copy( - function_node->getArguments(), - std::back_inserter(processing) - ); + std::ranges::copy(function_node->getArguments(), std::back_inserter(processing)); else conjunctions.push_back(node); } @@ -1335,7 +1389,7 @@ void removeExpressionsThatDoNotDependOnTableIdentifiers( for (const auto & node : processing) { - if (!hasUnknownColumn(node, table_expression)) + if (keep(node)) conjunctions.push_back(node); } @@ -1357,6 +1411,29 @@ void removeExpressionsThatDoNotDependOnTableIdentifiers( function->resolveAsFunction(function_impl->build(function->getArgumentColumns())); } +} + +void removeExpressionsThatDoNotDependOnTableIdentifiers( + QueryTreeNodePtr & expression, + const QueryTreeNodePtr & table_expression, + const ContextPtr & context) +{ + filterConjunctions( + expression, + [&](const QueryTreeNodePtr & node) { return !hasUnknownColumn(node, table_expression); }, + context); +} + +void removeExpressionsThatAreUnsafeToDuplicate( + QueryTreeNodePtr & expression, + const ContextPtr & context) +{ + if (!expression) + return; + + filterConjunctions(expression, isSafeToDuplicateInQueryTree, context); +} + namespace { diff --git a/src/Analyzer/Utils.h b/src/Analyzer/Utils.h index 3de98691089c..4f85e1d1fc1b 100644 --- a/src/Analyzer/Utils.h +++ b/src/Analyzer/Utils.h @@ -235,13 +235,27 @@ bool hasUnknownColumn( /** Suppose we have a table x with columns a, c, d and * a an expression like x.a > 2 AND y.b > 3 AND x.c + 1 == x.d * This method will remove the part y.b > 3 from it since it depends - * on unknown columns from a different table. + * on unknown columns from a different table. A non-function root such as + * `WHERE y.b` is dropped the same way. */ void removeExpressionsThatDoNotDependOnTableIdentifiers( QueryTreeNodePtr & expression, const QueryTreeNodePtr & replacement_table_expression, const ContextPtr & context); +/** Remove conjuncts that are unsafe to copy into another query tree (not deterministic, not + * deterministic in this query, stateful, or server-constant). Nested `and` is flattened the same + * way as `removeExpressionsThatDoNotDependOnTableIdentifiers`. Window and aggregate functions are + * also dropped. JOIN filter pushdown refuses stateful predicates via + * `ActionsDAG::hasStatefulFunctions`. + * + * The wrap `WHERE` is sent to remote cluster nodes. Node-local functions such as `hostName`, + * `dictGet`, `joinGet`, `FQDN`, and `queryID` must stay on the initiator: remotes can miss the + * dictionary, see different data, or return a different server-local value. + */ +void removeExpressionsThatAreUnsafeToDuplicate( + QueryTreeNodePtr & expression, + const ContextPtr & context); Field getFieldFromColumnForASTLiteral(const ColumnPtr & column, size_t row, const DataTypePtr & data_type); diff --git a/src/CMakeLists.txt b/src/CMakeLists.txt index f47f4f2c2bce..9857d0ac282b 100644 --- a/src/CMakeLists.txt +++ b/src/CMakeLists.txt @@ -158,6 +158,7 @@ add_headers_and_sources(dbms Storages/ObjectStorage/S3) add_headers_and_sources(dbms Storages/ObjectStorage/HDFS) add_headers_and_sources(dbms Storages/ObjectStorage/Local) add_headers_and_sources(dbms Storages/ObjectStorage/Web) +add_headers_and_sources(dbms Storages/ObjectStorage/MergeTree) add_headers_and_sources(dbms Storages/ObjectStorage/DataLakes) add_headers_and_sources(dbms Storages/ObjectStorage/DataLakes/Common) add_headers_and_sources(dbms Storages/ObjectStorage/DataLakes/Iceberg) diff --git a/src/Client/MultiplexedConnections.cpp b/src/Client/MultiplexedConnections.cpp index 851a9e81c6cf..8105dc19ac01 100644 --- a/src/Client/MultiplexedConnections.cpp +++ b/src/Client/MultiplexedConnections.cpp @@ -229,7 +229,7 @@ void MultiplexedConnections::sendQuery( void MultiplexedConnections::sendClusterFunctionReadTaskResponse(const ClusterFunctionReadTaskResponse & response) { std::lock_guard lock(cancel_mutex); - if (cancelled) + if (cancelled || !current_connection || !current_connection->isConnected()) return; current_connection->sendClusterFunctionReadTaskResponse(response); } @@ -238,7 +238,7 @@ void MultiplexedConnections::sendClusterFunctionReadTaskResponse(const ClusterFu void MultiplexedConnections::sendMergeTreeReadTaskResponse(const ParallelReadResponse & response) { std::lock_guard lock(cancel_mutex); - if (cancelled) + if (cancelled || !current_connection || !current_connection->isConnected()) return; current_connection->sendMergeTreeReadTaskResponse(response); } @@ -538,9 +538,12 @@ MultiplexedConnections::ReplicaState & MultiplexedConnections::getReplicaForRead void MultiplexedConnections::invalidateReplica(ReplicaState & state) { + Connection * old_connection = state.connection; state.connection = nullptr; state.pool_entry = IConnectionPool::Entry(); --active_connection_count; + if (current_connection == old_connection) + current_connection = nullptr; } void MultiplexedConnections::setAsyncCallback(AsyncCallback async_callback) diff --git a/src/Common/CurrentMetrics.cpp b/src/Common/CurrentMetrics.cpp index bb9bd52776cf..be7058025c05 100644 --- a/src/Common/CurrentMetrics.cpp +++ b/src/Common/CurrentMetrics.cpp @@ -12,6 +12,7 @@ M(Merge, "Number of executing background merges") \ M(MergeParts, "Number of source parts participating in current background merges") \ M(Move, "Number of currently executing moves") \ + M(Export, "Number of currently executing exports") \ M(PartMutation, "Number of mutations (ALTER DELETE/UPDATE)") \ M(ReplicatedFetch, "Number of data parts being fetched from replica") \ M(ReplicatedSend, "Number of data parts being sent to replicas") \ @@ -555,6 +556,7 @@ M(StartupScriptsExecutionState, "State of startup scripts execution: 0 = not finished, 1 = success, 2 = failure.") \ \ M(IsServerShuttingDown, "Indicates if the server is shutting down: 0 = no, 1 = yes") \ + M(IsSwarmModeEnabled, "Indicates if the swarm mode enabled or not: 0 = disabled, 1 = enabled") \ \ M(StatelessWorkerThreads, "Number of threads in the stateless worker thread pool.") \ M(StatelessWorkerThreadsActive, "Number of threads in the stateless worker thread pool running a task.") \ diff --git a/src/Common/ErrorCodes.cpp b/src/Common/ErrorCodes.cpp index da6d80d86a4a..408d299ad89d 100644 --- a/src/Common/ErrorCodes.cpp +++ b/src/Common/ErrorCodes.cpp @@ -662,6 +662,7 @@ M(780, SW_SERVER_NO_WORKERS_AVAILABLE) \ M(781, AI_PROVIDER_RESPONSE_TRUNCATED) \ M(782, AI_PROVIDER_RESPONSE_INCOMPLETE) \ + M(783, CATALOG_NAMESPACE_DISABLED) \ \ M(900, DISTRIBUTED_CACHE_ERROR) \ M(901, CANNOT_USE_DISTRIBUTED_CACHE) \ @@ -687,6 +688,9 @@ M(1013, AMBIGUOUS_HANDLER) \ M(1014, TRANSACTION_ROLLBACK_PARTIAL_FAILURE) \ M(1015, FILE_CHANGED_DURING_READ) \ + M(1016, PENDING_MUTATIONS_NOT_ALLOWED) \ + M(1017, EXPORT_PARTITION_ALREADY_EXPORTED) \ + M(1018, PARTITION_EXPORT_FAILED) \ /* See END */ #ifdef APPLY_FOR_EXTERNAL_ERROR_CODES @@ -703,7 +707,7 @@ namespace ErrorCodes APPLY_FOR_ERROR_CODES(M) #undef M - constexpr ErrorCode END = 1015; + constexpr ErrorCode END = 1018; #if !defined(CLICKHOUSE_PARSER_MINIMAL_BUILD) /** One `ErrorPairHolder` per error code, each holding two `Error` structs - the last message, diff --git a/src/Common/FailPoint.cpp b/src/Common/FailPoint.cpp index 1e58899e7cc9..d55047e98b14 100644 --- a/src/Common/FailPoint.cpp +++ b/src/Common/FailPoint.cpp @@ -46,6 +46,8 @@ static struct InitFiu PAUSEABLE_ONCE(remote_query_executor_finish_drain_pause) \ ONCE(connection_stale_on_establish) \ REGULAR(cluster_discovery_faults) \ + REGULAR(cluster_discovery_unregister_fail) \ + ONCE(cluster_discovery_retry_signal_fail) \ REGULAR(stripe_log_sink_write_fallpoint) \ REGULAR(file_checker_update_and_save_fail_reading_sizes) \ REGULAR(file_checker_update_and_save_fail_persisting) \ @@ -239,6 +241,14 @@ static struct InitFiu ONCE(iceberg_writes_cleanup) \ REGULAR(iceberg_slow_manifest_read) \ REGULAR(storage_cluster_read_sleep) \ + ONCE(iceberg_writes_non_retry_cleanup) \ + ONCE(iceberg_writes_post_publish_throw) \ + ONCE(iceberg_export_after_commit_before_zk_completed) \ + REGULAR(export_partition_commit_always_throw) \ + ONCE(export_partition_status_change_throw) \ + REGULAR(export_partition_processed_paths_sync_fail) \ + REGULAR(export_part_non_retryable_throw) \ + REGULAR(export_part_retryable_throw) \ ONCE(backup_add_empty_memory_table) \ ONCE(backup_from_snapshot_fail_after_batch) \ ONCE(backup_from_snapshot_fail_after_lock_file_creation) \ diff --git a/src/Common/ProfileEvents.cpp b/src/Common/ProfileEvents.cpp index 3814d82fc18b..7e7d268a88ad 100644 --- a/src/Common/ProfileEvents.cpp +++ b/src/Common/ProfileEvents.cpp @@ -40,6 +40,10 @@ M(FailedInitialQuery, "Number of failed initial queries.", ValueType::Number) \ M(FailedInitialSelectQuery, "Same as FailedInitialQuery, but only for SELECT queries.", ValueType::Number) \ M(FailedQuery, "Number of total failed queries, both internal and user queries.", ValueType::Number) \ + M(PartsExports, "Number of successful part exports.", ValueType::Number) \ + M(PartsExportFailures, "Number of failed part exports.", ValueType::Number) \ + M(PartsExportDuplicated, "Number of part exports that failed because target already exists.", ValueType::Number) \ + M(PartsExportTotalMilliseconds, "Total time spent on part export operations.", ValueType::Milliseconds) \ M(FailedSelectQuery, "Same as FailedQuery, but only for SELECT queries.", ValueType::Number) \ M(FailedInsertQuery, "Same as FailedQuery, but only for INSERT queries.", ValueType::Number) \ M(FailedAsyncInsertQuery, "Number of failed ASYNC INSERT queries.", ValueType::Number) \ @@ -267,6 +271,8 @@ M(UserThrottlerSleepMicroseconds, "Total time a query was sleeping to conform 'max_network_bandwidth_for_user' throttling.", ValueType::Microseconds) \ M(AllUsersThrottlerBytes, "Bytes passed through 'max_network_bandwidth_for_all_users' throttler.", ValueType::Bytes) \ M(AllUsersThrottlerSleepMicroseconds, "Total time a query was sleeping to conform 'max_network_bandwidth_for_all_users' throttling.", ValueType::Microseconds) \ + M(ExportsThrottlerBytes, "Bytes passed through 'max_exports_bandwidth_for_server' throttler.", ValueType::Bytes) \ + M(ExportsThrottlerSleepMicroseconds, "Total time a query was sleeping to conform 'max_exports_bandwidth_for_server' throttling.", ValueType::Microseconds) \ M(QueryRemoteReadThrottlerBytes, "Bytes passed through 'max_remote_read_network_bandwidth' throttler.", ValueType::Bytes) \ M(QueryRemoteReadThrottlerSleepMicroseconds, "Total time a query was sleeping to conform 'max_remote_read_network_bandwidth' throttling.", ValueType::Microseconds) \ M(ReaderExecutorSourceRequests, "Number of source-side requests opened by ReaderExecutor (excludes live-buffer reuses).", ValueType::Number) \ @@ -398,6 +404,19 @@ M(ZooKeeperBytesSent, "Number of bytes send over network while communicating with ZooKeeper.", ValueType::Bytes) \ M(ZooKeeperBytesReceived, "Number of bytes received over network while communicating with ZooKeeper.", ValueType::Bytes) \ \ + M(ExportPartitionZooKeeperRequests, "Total number of ZooKeeper requests made by the export partition feature.", ValueType::Number) \ + M(ExportPartitionZooKeeperGet, "Number of 'get' requests to ZooKeeper made by the export partition feature.", ValueType::Number) \ + M(ExportPartitionZooKeeperGetChildren, "Number of 'getChildren' requests to ZooKeeper made by the export partition feature.", ValueType::Number) \ + M(ExportPartitionZooKeeperGetChildrenWatch, "Number of 'getChildrenWatch' requests to ZooKeeper made by the export partition feature.", ValueType::Number) \ + M(ExportPartitionZooKeeperGetWatch, "Number of 'getWatch' requests to ZooKeeper made by the export partition feature.", ValueType::Number) \ + M(ExportPartitionZooKeeperCreate, "Number of 'create' requests to ZooKeeper made by the export partition feature.", ValueType::Number) \ + M(ExportPartitionZooKeeperSet, "Number of 'set' requests to ZooKeeper made by the export partition feature.", ValueType::Number) \ + M(ExportPartitionZooKeeperRemove, "Number of 'remove' requests to ZooKeeper made by the export partition feature.", ValueType::Number) \ + M(ExportPartitionZooKeeperRemoveRecursive, "Number of 'removeRecursive' requests to ZooKeeper made by the export partition feature.", ValueType::Number) \ + M(ExportPartitionZooKeeperMulti, "Number of 'multi' requests to ZooKeeper made by the export partition feature.", ValueType::Number) \ + M(ExportPartitionZooKeeperExists, "Number of 'exists' requests to ZooKeeper made by the export partition feature.", ValueType::Number) \ + M(ExportPartsRejectedByMemoryLimit, "Number of background export part tasks rejected due to background memory limit.", ValueType::Number) \ + \ M(DistributedConnectionTries, "Total count of distributed connection attempts.", ValueType::Number) \ M(DistributedConnectionUsable, "Total count of successful distributed connections to a usable server (with required table, but maybe stale).", ValueType::Number) \ M(DistributedConnectionFailTry, "Total count when distributed connection fails with retry.", ValueType::Number) \ @@ -440,6 +459,11 @@ M(IcebergTrivialCountOptimizationApplied, "Trivial count optimization applied while reading from Iceberg", ValueType::Number) \ M(IcebergVersionHintUsed, "Number of times version-hint.text has been used.", ValueType::Number) \ M(IcebergMinMaxIndexPrunedFiles, "Number of skipped files by using MinMax index in Iceberg", ValueType::Number) \ + M(IcebergAvroFileParsing, "Number of times avro metadata files have been parsed.", ValueType::Number) \ + M(IcebergAvroFileParsingMicroseconds, "Time spent for parsing avro metadata files for Iceberg tables.", ValueType::Microseconds) \ + M(IcebergJsonFileParsing, "Number of times json metadata files have been parsed.", ValueType::Number) \ + M(IcebergJsonFileParsingMicroseconds, "Time spent for parsing json metadata files for Iceberg tables.", ValueType::Microseconds) \ + \ M(JoinBuildTableRowCount, "Total number of rows in the build table for a JOIN operation.", ValueType::Number) \ M(JoinProbeTableRowCount, "Total number of rows in the probe table for a JOIN operation.", ValueType::Number) \ M(JoinResultRowCount, "Total number of rows in the result of a JOIN operation.", ValueType::Number) \ @@ -793,8 +817,10 @@ The server successfully detected this situation and will download merged part fr M(S3DeleteObjects, "Number of S3 API DeleteObject(s) calls.", ValueType::Number) \ M(S3CopyObject, "Number of S3 API CopyObject calls.", ValueType::Number) \ M(S3ListObjects, "Number of S3 API ListObjects calls.", ValueType::Number) \ + M(S3ListObjectsMicroseconds, "Time of S3 API ListObjects execution.", ValueType::Microseconds) \ M(S3HeadObject, "Number of S3 API HeadObject calls.", ValueType::Number) \ M(S3GetObjectTagging, "Number of S3 API GetObjectTagging calls.", ValueType::Number) \ + M(S3HeadObjectMicroseconds, "Time of S3 API HeadObject execution.", ValueType::Microseconds) \ M(S3CreateMultipartUpload, "Number of S3 API CreateMultipartUpload calls.", ValueType::Number) \ M(S3UploadPartCopy, "Number of S3 API UploadPartCopy calls.", ValueType::Number) \ M(S3UploadPart, "Number of S3 API UploadPart calls.", ValueType::Number) \ @@ -850,6 +876,7 @@ The server successfully detected this situation and will download merged part fr M(AzureCopyObject, "Number of Azure blob storage API CopyObject calls", ValueType::Number) \ M(AzureDeleteObjects, "Number of Azure blob storage API DeleteObject(s) calls.", ValueType::Number) \ M(AzureListObjects, "Number of Azure blob storage API ListObjects calls.", ValueType::Number) \ + M(AzureListObjectsMicroseconds, "Time of Azure blob storage API ListObjects execution.", ValueType::Microseconds) \ M(AzureGetProperties, "Number of Azure blob storage API GetProperties calls.", ValueType::Number) \ M(AzureCreateContainer, "Number of Azure blob storage API CreateContainer calls.", ValueType::Number) \ \ @@ -1693,6 +1720,11 @@ The server successfully detected this situation and will download merged part fr M(StatelessWorkerServerHeartbeatErrors, "Number of failed heartbeats from stateless workers to the stateless worker discovery service (any error or timeout).", ValueType::Number) \ M(StatelessWorkerServerTenantBindings, "Number of times a stateless worker was assigned to a tenant to serve its queries.", ValueType::Number) \ M(StatelessWorkerServerLeasesLost, "Number of times a stateless worker's lease ended and it began awaiting shutdown.", ValueType::Number) \ + M(ObjectStorageListObjectsCacheHits, "Number of times object storage list objects operation hit the cache.", ValueType::Number) \ + M(ObjectStorageListObjectsCacheMisses, "Number of times object storage list objects operation miss the cache.", ValueType::Number) \ + M(ObjectStorageListObjectsCacheExactMatchHits, "Number of times object storage list objects operation hit the cache with an exact match.", ValueType::Number) \ + M(ObjectStorageListObjectsCachePrefixMatchHits, "Number of times object storage list objects operation miss the cache using prefix matching.", ValueType::Number) \ + \ \ M(StatelessWorkerDiscoveryCreateLeaseRequests, "Number of create_lease requests handled by the stateless worker discovery service.", ValueType::Number) \ M(StatelessWorkerDiscoveryCreateLeaseErrors, "Number of create_lease requests the stateless worker discovery service answered with an error.", ValueType::Number) \ @@ -1729,6 +1761,10 @@ The server successfully detected this situation and will download merged part fr M(StatelessWorkerDiscoveryHeartbeatsRejected, "Number of heartbeats the stateless worker discovery service rejected because the worker had already been evicted.", ValueType::Number) \ M(StatelessWorkerDiscoveryKeeperTransactionRetries, "Number of write transactions the stateless worker discovery service retried because its coordination store (Keeper) state was modified concurrently.", ValueType::Number) \ \ + M(ObjectStorageClusterSentToMatchedReplica, "Number of tasks in ObjectStorageCluster request sent to matched replica.", ValueType::Number) \ + M(ObjectStorageClusterSentToNonMatchedReplica, "Number of tasks in ObjectStorageCluster request sent to non-matched replica.", ValueType::Number) \ + M(ObjectStorageClusterProcessedTasks, "Number of processed tasks in ObjectStorageCluster request.", ValueType::Number) \ + M(ObjectStorageClusterWaitingMicroseconds, "Time of waiting for tasks in ObjectStorageCluster request.", ValueType::Microseconds) \ M(DataLakeRestCatalogLoadConfig, "Number of 'load config' requests to Iceberg REST catalog.", ValueType::Number) \ M(DataLakeRestCatalogLoadConfigMicroseconds, "Total time of 'load config' requests to Iceberg REST catalog.", ValueType::Microseconds) \ M(DataLakeRestCatalogGetNamespaces, "Number of 'get namespaces' requests to Iceberg REST catalog.", ValueType::Number) \ diff --git a/src/Common/TTLCachePolicy.h b/src/Common/TTLCachePolicy.h index d481d0290330..1f894cb35bdb 100644 --- a/src/Common/TTLCachePolicy.h +++ b/src/Common/TTLCachePolicy.h @@ -278,10 +278,10 @@ class TTLCachePolicy : public ICachePolicy; Cache cache; - +private: /// TODO To speed up removal of stale entries, we could also add another container sorted on expiry times which maps keys to iterators /// into the cache. To insert an entry, add it to the cache + add the iterator to the sorted container. To remove stale entries, do a /// binary search on the sorted container and erase all left of the found key. diff --git a/src/Common/ThreadStatus.h b/src/Common/ThreadStatus.h index 7b6b6a4b8556..c8f60bcd18d5 100644 --- a/src/Common/ThreadStatus.h +++ b/src/Common/ThreadStatus.h @@ -131,6 +131,16 @@ class ThreadGroup void attachQueryForLog(const String & query_, UInt64 normalized_hash = 0); void attachInternalProfileEventsQueue(const InternalProfileEventsQueuePtr & profile_queue); + /// Override the cancellation predicate. All threads that subsequently attach to this + /// group via ThreadGroupSwitcher inherit the predicate in their local_data, making + /// isQueryCanceled() reflect task-level cancellation without a process-list entry. + /// Required for part and partition export cancellation during S3 outage. + void setCancelPredicate(QueryIsCanceledPredicate predicate) + { + std::lock_guard lock(mutex); + shared_data.query_is_canceled_predicate = std::move(predicate); + } + /// When new query starts, new thread group is created for it, current thread becomes master thread of the query static ThreadGroupPtr createForQuery(ContextPtr query_context_, FatalErrorCallback fatal_error_callback_ = {}); diff --git a/src/Common/setThreadName.h b/src/Common/setThreadName.h index 9993ba64400b..fbd355262586 100644 --- a/src/Common/setThreadName.h +++ b/src/Common/setThreadName.h @@ -182,6 +182,7 @@ namespace DB M(DISK_OBJECT_STORAGE_COPY, "DiskObjStCopy") \ M(DISTRIBUTED_CACHE, "DistribCache") \ M(DISTRIBUTED_CACHE_DROP, "DropDistCache") \ + M(EXPORT_PART, "ExportPart") \ enum class ThreadName : uint8_t diff --git a/src/Core/Joins.h b/src/Core/Joins.h index 0741cf15639f..3e18e6725a4e 100644 --- a/src/Core/Joins.h +++ b/src/Core/Joins.h @@ -142,6 +142,25 @@ enum class JoinTableSide : uint8_t const char * toString(JoinTableSide join_table_side); +/** Whether ordinary columns from this side of a JOIN can be used as filter inputs + * before the JOIN. Skip the null-producing side of an outer JOIN, the right side + * of an `ASOF JOIN`, and both sides of a `PASTE JOIN` or `FULL JOIN`. + * Attaching an equivalent-key filter to the other child, and dictionary / lookup + * fill, are separate (`JoinStep::allowPushDownToRight`). + */ +constexpr bool canPrefilterJoinSide(JoinKind kind, JoinStrictness strictness, JoinTableSide side) +{ + if (isPaste(kind) || isFull(kind)) + return false; + if (strictness == JoinStrictness::Asof && side == JoinTableSide::Right) + return false; + if (isLeft(kind) && side == JoinTableSide::Right) + return false; + if (isRight(kind) && side == JoinTableSide::Left) + return false; + return true; +} + enum class JoinOrderAlgorithm : uint8_t { GREEDY = 0, diff --git a/src/Core/Protocol.h b/src/Core/Protocol.h index a1be11a54307..39da9f7bb62f 100644 --- a/src/Core/Protocol.h +++ b/src/Core/Protocol.h @@ -97,8 +97,10 @@ namespace Protocol MergeTreeReadTaskRequest = 16, /// Request from a MergeTree replica to a coordinator TimezoneUpdate = 17, /// Receive server's (session-wide) default timezone SSHChallenge = 18, /// Return challenge for SSH signature signing + MAX = SSHChallenge, + ConnectionLost = 255, /// Exception that occurred on the client side. }; /// Returns the packet name, or the numeric value itself when it is out of range diff --git a/src/Core/Range.cpp b/src/Core/Range.cpp index 744c977a1bc1..3217353d4de6 100644 --- a/src/Core/Range.cpp +++ b/src/Core/Range.cpp @@ -3,13 +3,21 @@ #include #include #include +#include #include #include +#include namespace DB { +namespace ErrorCodes +{ + extern const int INCORRECT_DATA; +}; + + FieldRef::FieldRef(ColumnsWithTypeAndName * columns_, size_t row_idx_, size_t column_idx_) : Field((*(*columns_)[column_idx_].column)[row_idx_]), columns(columns_), row_idx(row_idx_), column_idx(column_idx_) { @@ -167,6 +175,13 @@ bool Range::isInfinite() const return left.isNegativeInfinity() && right.isPositiveInfinity(); } +/// [x, x] +bool Range::isPoint() const +{ + return fullBounded() && left_included && right_included && equals(left, right) + && !left.isNegativeInfinity() && !left.isPositiveInfinity(); +} + bool Range::intersectsRange(const Range & r) const { /// r to the left of me. @@ -292,6 +307,32 @@ bool Range::nearByWith(const Range & r) const return false; } +String Range::serialize(bool base64) const +{ + WriteBufferFromOwnString str; + + str << left_included << right_included; + writeFieldBinary(left, str); + writeFieldBinary(right, str); + + if (base64) + return base64Encode(str.str()); + else + return str.str(); +} + +void Range::deserialize(const String & range, bool base64) +{ + if (range.empty()) + throw Exception(ErrorCodes::INCORRECT_DATA, "Empty range dump"); + + ReadBufferFromOwnString str(base64 ? base64Decode(range) : range); + + str >> left_included >> right_included; + left = readFieldBinary(str); + right = readFieldBinary(str); +} + Range intersect(const Range & a, const Range & b) { Range res = Range::createWholeUniverse(); diff --git a/src/Core/Range.h b/src/Core/Range.h index 3ebe94de1569..735fd4448bed 100644 --- a/src/Core/Range.h +++ b/src/Core/Range.h @@ -100,6 +100,8 @@ struct Range bool isBlank() const; + bool isPoint() const; + bool intersectsRange(const Range & r) const; bool containsRange(const Range & r) const; @@ -120,6 +122,9 @@ struct Range bool nearByWith(const Range & r) const; String toString() const; + + String serialize(bool base64 = false) const; + void deserialize(const String & range, bool base64 = false); }; Range intersect(const Range & a, const Range & b); diff --git a/src/Core/ServerSettings.cpp b/src/Core/ServerSettings.cpp index f8778118b392..67f3f7933032 100644 --- a/src/Core/ServerSettings.cpp +++ b/src/Core/ServerSettings.cpp @@ -202,6 +202,7 @@ A value of `0` means unlimited. DECLARE(UInt64, max_unexpected_parts_loading_thread_pool_size, 8, R"(The number of threads to load inactive set of data parts (Unexpected ones) at startup.)", 0) \ DECLARE(UInt64, max_parts_cleaning_thread_pool_size, 128, R"(The number of threads for concurrent removal of inactive data parts.)", 0) \ DECLARE(UInt64, max_mutations_bandwidth_for_server, 0, R"(The maximum read speed of all mutations on server in bytes per second. Zero means unlimited.)", 0) \ + DECLARE(UInt64, max_exports_bandwidth_for_server, 0, R"(The maximum read speed of all exports on server in bytes per second. Zero means unlimited.)", 0) \ DECLARE(UInt64, max_merges_bandwidth_for_server, 0, R"(The maximum read speed of all merges on server in bytes per second. Zero means unlimited.)", 0) \ DECLARE(UInt64, max_replicated_fetches_network_bandwidth_for_server, 0, R"(The maximum speed of data exchange over the network in bytes per second for replicated fetches. Zero means unlimited.)", 0) \ DECLARE(UInt64, max_replicated_sends_network_bandwidth_for_server, 0, R"(The maximum speed of data exchange over the network in bytes per second for replicated sends. Zero means unlimited.)", 0) \ @@ -1849,7 +1850,11 @@ If set to true, server settings will not be checked for correctness. ```xml 1 ``` -)", 0) +)", 0) \ + DECLARE(UInt64, object_storage_list_objects_cache_size, 500000000, "Maximum size of ObjectStorage list objects cache in bytes. Zero means disabled.", 0) \ + DECLARE(UInt64, object_storage_list_objects_cache_max_entries, 1000, "Maximum size of ObjectStorage list objects cache in entries. Zero means disabled.", 0) \ + DECLARE(UInt64, object_storage_list_objects_cache_ttl, 3600, "Time to live of records in ObjectStorage list objects cache in seconds. Zero means unlimited", 0) \ + DECLARE(Bool, allow_experimental_export_merge_tree_partition, false, "Enable export replicated merge tree partition feature. It is experimental and not yet ready for production use.", 0) /// Settings with a path are server settings with at least one layer of nesting that have a fixed structure (no lists, lists, enumerations, repetitions, ...). #define LIST_OF_SERVER_SETTINGS_WITH_PATH(DECLARE, ALIAS) \ diff --git a/src/Core/Settings.cpp b/src/Core/Settings.cpp index bb42fc5d983b..881dd7186dd6 100644 --- a/src/Core/Settings.cpp +++ b/src/Core/Settings.cpp @@ -2285,6 +2285,22 @@ Possible values: - `global` — Replaces the `IN`/`JOIN` query with `GLOBAL IN`/`GLOBAL JOIN.` - `allow` — Allows the use of these types of subqueries. )", IMPORTANT) \ + DECLARE(ObjectStorageClusterJoinMode, object_storage_cluster_join_mode, ObjectStorageClusterJoinMode::ALLOW, R"( +Changes the behaviour of object storage cluster function or table. + +ClickHouse applies this setting when the query contains the product of object storage cluster function or table, i.e. when the query for a object storage cluster function or table contains a non-GLOBAL subquery for the object storage cluster function or table. + +Restrictions: + +- Only applied for JOIN subqueries. +- Only if the FROM section uses a object storage cluster function or table. + +Possible values: + +- `local` — Replaces the database and table in the subquery with local ones for the destination server (shard), leaving the normal `IN`/`JOIN.` +- `global` — Replaces the `IN`/`JOIN` query with `GLOBAL IN`/`GLOBAL JOIN.` Right table executes first and is added to the secondary query as temporay table. +- `allow` — Default value. Allows the use of these types of subqueries. +)", 0) \ \ DECLARE(UInt64, max_concurrent_queries_for_all_users, 0, R"( Throw exception if the value of this setting is less or equal than the current number of simultaneously processed queries. @@ -8712,6 +8728,94 @@ Enable converting the hash table to a flat array for joins when the key is a sin )", 0) \ DECLARE(UInt64, query_plan_min_columns_for_join_lazy_indexing, 3, R"( Control the minimum number of payload columns from the left side required for enabling lazy indexing optimization in JOIN. 0 means the optimization is disabled. +)", 0) \ + DECLARE(Timezone, iceberg_timezone_for_timestamptz, "UTC", R"( +Timezone for Iceberg timestamptz field. + +Possible values: + +- Any valid timezone, e.g. `Europe/Berlin`, `UTC` or `Zulu` +- `` (empty value) - use session timezone + +Default value is `UTC`. +)", 0) \ + DECLARE(Timezone, iceberg_partition_timezone, "", R"( +Time zone by which partitioning of Iceberg tables was performed. +Possible values: + +- Any valid timezone, e.g. `Europe/Berlin`, `UTC` or `Zulu` +- `` (empty value) - use server or session timezone + +Default value is empty. +)", 0) \ + DECLARE(Bool, export_merge_tree_part_overwrite_file_if_exists, false, R"( +Overwrite file if it already exists when exporting a merge tree part +)", 0) \ + DECLARE(Bool, export_merge_tree_partition_force_export, false, R"( +Ignore existing partition export and overwrite the zookeeper entry +)", 0) \ + DECLARE(UInt64, export_merge_tree_partition_retry_initial_backoff_seconds, 5, R"( +Initial delay (in seconds) before retrying a failed part export in an export partition task. +The delay grows exponentially with the per-replica retry count (capped doubling): `delay = min(initial << (attempts - 1), max)`, where `max` is `export_merge_tree_partition_retry_max_backoff_seconds`. +The back-off is per-replica in-memory state: it only spaces this replica's retries out in time and never prevents another replica from attempting the same part. Retryable failures are retried until the task succeeds or `export_merge_tree_partition_task_timeout_seconds` elapses. +To survive a long transient outage (e.g. object storage downtime), raise `export_merge_tree_partition_task_timeout_seconds`. +)", 0) \ + DECLARE(UInt64, export_merge_tree_partition_retry_max_backoff_seconds, 300, R"( +Maximum delay (in seconds) between retries of a failed part export in an export partition task. Caps the exponential growth controlled by `export_merge_tree_partition_retry_initial_backoff_seconds`. +)", 0) \ + DECLARE(UInt64, export_merge_tree_partition_task_timeout_seconds, 86400, R"( +Maximum wall-clock duration (in seconds) an export partition task is allowed to remain in the PENDING state before it is auto-killed by the background cleanup loop. +The timeout is measured from the manifest's create_time. Set to 0 to disable the timeout. +When the timeout is exceeded the task transitions to KILLED (same terminal state as `KILL QUERY ... EXPORT PARTITION`), and `last_exception` is populated with a timeout reason. + +IMPORTANT: In case the storage is managed by a 3rd party application that cleans up old manifest files, it is important that the TTL of such files are greater than the timeout of export partition tasks. +If it is not configured in such a way, it is possible to accidentally duplicate data in the extremely rare case a ClickHouse node is the only node working on a given export task, commits the data to Iceberg, crashes before marking the task as done and only boots up after the manifest cleanup has deleted the commit manifest. +In such scenario, ClickHouse would attempt to commit those files again producing duplicates. + +Notes: +- Enforcement is best-effort: actual kill latency is bounded by one manifest-updater poll cycle (~30s) plus ZooKeeper watch propagation. +)", 0) \ + DECLARE(MergeTreePartExportFileAlreadyExistsPolicy, export_merge_tree_part_file_already_exists_policy, MergeTreePartExportFileAlreadyExistsPolicy::skip, R"( +Possible values: +- skip - Skip the file if it already exists. +- error - Throw an error if the file already exists. +- overwrite - Overwrite the file. +)", 0) \ + DECLARE(UInt64, export_merge_tree_part_max_bytes_per_file, 0, R"( +Maximum number of bytes to write to a single file when exporting a merge tree part. 0 means no limit. +This is not a hard limit, and it highly depends on the output format granularity and input source chunk size. +)", 0) \ + DECLARE(UInt64, export_merge_tree_part_max_rows_per_file, 0, R"( +Maximum number of rows to write to a single file when exporting a merge tree part. 0 means no limit. +This is not a hard limit, and it highly depends on the output format granularity and input source chunk size. +)", 0) \ + DECLARE(Bool, export_merge_tree_part_throw_on_pending_mutations, true, R"( +Throw an error if there are pending mutations when exporting a merge tree part. +)", 0) \ + DECLARE(Bool, export_merge_tree_part_throw_on_pending_patch_parts, true, R"( +Throw an error if there are pending patch parts when exporting a merge tree part. +)", 0) \ + DECLARE(ExportPartitionAllOnError, export_merge_tree_partition_all_on_error, ExportPartitionAllOnError::throw_first, R"( +Failure handling for `ALTER TABLE ... EXPORT PARTITION ALL ...`. +Possible values: +- `throw_first` (default) - stop at the first failed partition; partitions already scheduled remain scheduled. +- `collect` - try every partition and throw a single aggregated exception at the end if any failed; partitions that succeeded remain scheduled. +- `skip_conflicts` - silently skip partitions that are already exported / being exported (errors with code EXPORT_PARTITION_ALREADY_EXPORTED); fail-fast on every other error. +Has no effect on `EXPORT PARTITION ` (single-partition export). +)", 0) \ + DECLARE(String, export_merge_tree_part_filename_pattern, "{part_name}_{checksum}", R"( +Pattern for the filename of the exported merge tree part. The `part_name` and `checksum` are calculated and replaced on the fly. Additional macros are supported. +)", 0) \ + DECLARE(Bool, export_merge_tree_part_allow_lossy_cast, false, R"( +Allow `EXPORT PART`/`EXPORT PARTITION` to apply lossy (non-value-preserving) casts when the source and destination column types differ. When disabled, an export that would require a lossy cast throws instead. + +When exporting to Apache Iceberg, the partition value written to the metadata is derived from the source partition columns by casting them to the destination partition-field types and applying the destination partition transform — the same computation the exported data files use, so the metadata stays consistent with the data. A lossy cast on a partition column remains semantically truncating: both the data files and the metadata contain the truncated value, and such casts require this setting to be enabled. +)", 0) \ + DECLARE(MergeTreePartExportSchemaMismatchMode, export_merge_tree_part_schema_mismatch_mode, MergeTreePartExportSchemaMismatchMode::strict, R"( +Controls whether `EXPORT PART`/`EXPORT PARTITION` allows a column-count mismatch between the source `MergeTree` table and the destination table. Columns are matched positionally, like `INSERT INTO dest SELECT * FROM src`. +Possible values: +- `strict` (default) - the source and destination must have the same number of columns. A mismatch in either direction throws `NUMBER_OF_COLUMNS_DOESNT_MATCH`. +- `ignore_extra_source_columns_by_position` - the source may have more columns than the destination. The extra trailing source columns (by position) are dropped and not exported. The destination having more columns than the source is still rejected in this mode. )", 0) \ \ /* ####################################################### */ \ @@ -8984,6 +9088,15 @@ Source SQL dialect for the polyglot transpiler (e.g. 'sqlite', 'mysql', 'postgre )", EXPERIMENTAL) \ DECLARE(Bool, enable_adaptive_memory_spill_scheduler, false, R"( Trigger processor to spill data into external storage adpatively. grace join is supported at present. +)", EXPERIMENTAL) \ + DECLARE(String, object_storage_cluster, "", R"( +Cluster to make distributed requests to object storages with alternative syntax. +)", EXPERIMENTAL) \ + DECLARE(UInt64, object_storage_max_nodes, 0, R"( +Limit for hosts used for request in object storage cluster table functions - azureBlobStorageCluster, s3Cluster, hdfsCluster, etc. +Possible values: +- Positive integer. +- 0 — All hosts in cluster. )", EXPERIMENTAL) \ DECLARE_WITH_ALIAS(Bool, allow_delta_kernel_rs, true, R"( Allow the `delta-kernel-rs` implementation for reading Delta Lake tables. @@ -9016,6 +9129,9 @@ Write full paths (including s3://) into iceberg metadata files. )", EXPERIMENTAL) \ DECLARE(String, iceberg_metadata_compression_method, "", R"( Method to compress `.metadata.json` file. +)", EXPERIMENTAL) \ + DECLARE(Bool, use_object_storage_list_objects_cache, false, R"( +Cache the list of objects returned by list objects calls in object storage )", EXPERIMENTAL) \ DECLARE(Bool, make_distributed_plan, false, R"( Make distributed query plan. @@ -9046,6 +9162,19 @@ How many stateless workers will be used to execute this query. Zero disables sta )", EXPERIMENTAL) \ DECLARE(UInt64, distributed_plan_workers_provisioning_timeout_ms, 10000, R"( Total wall-clock time, in milliseconds, a query may spend provisioning stateless workers before execution: leasing them from the discovery service and verifying they are reachable. The query blocks up to this budget for the leased workers to become ready; when it elapses the query proceeds with the workers verified so far, or fails if none became available. Zero waits only for the initial lease-and-verify pass (no retries). +)", EXPERIMENTAL) \ + DECLARE(UInt64, lock_object_storage_task_distribution_ms, 500, R"( +In object storage distribution queries do not distribute tasks on non-prefetched nodes until prefetched node is active. +Determines how long the free executor node (one that finished processing all of it assigned tasks) should wait before "stealing" tasks from queue of currently busy executor nodes. + +Possible values: + +- 0 - steal tasks immediately after freeing up. +- >0 - wait for specified period of time before stealing tasks. + +Having this `>0` helps with cache reuse and might improve overall query time. +Because busy node might have warmed-up caches for this specific task, while free node needs to fetch lots of data from S3. +Which might take longer than just waiting for the busy node and generate extra traffic. )", EXPERIMENTAL) \ DECLARE(String, distributed_plan_force_exchange_kind, "", R"( Force specified kind of Exchange operators between distributed query stages. @@ -9114,12 +9243,24 @@ Use hash table size statistics collected from previous executions to size the JO )", 0) \ DECLARE(Bool, rewrite_in_to_join, false, R"( Rewrite expressions like 'x IN subquery' to JOIN. This might be useful for optimizing the whole query with join reordering. +)", EXPERIMENTAL) \ + DECLARE(Bool, object_storage_remote_initiator, false, R"( +Execute request to object storage as remote on one of object_storage_cluster nodes. +)", EXPERIMENTAL) \ + DECLARE(String, object_storage_remote_initiator_cluster, "", R"( +Cluster to choose remote initiator, when `object_storage_remote_initiator` is true. When empty, `object_storage_cluster` is used. +)", EXPERIMENTAL) \ + DECLARE(Bool, allow_experimental_iceberg_read_optimization, true, R"( +Allow Iceberg read optimization based on Iceberg metadata. )", EXPERIMENTAL) \ \ /** Experimental timeSeries* aggregate functions. */ \ DECLARE_WITH_ALIAS(Bool, allow_experimental_time_series_aggregate_functions, false, R"( Experimental timeSeries* aggregate functions for Prometheus-like timeseries resampling, rate, delta calculation. )", EXPERIMENTAL, allow_experimental_ts_to_grid_aggregate_function) \ + DECLARE(Bool, allow_experimental_export_merge_tree_part, true, R"( +Experimental export merge tree part. +)", EXPERIMENTAL) \ \ DECLARE(String, promql_database, "", R"( Specifies the database name used by the 'promql' dialect. Empty string means the current database. @@ -9189,6 +9330,8 @@ Enable experimental table function `eval`. #define OBSOLETE_SETTINGS(M, ALIAS) \ /** Obsolete settings which are kept around for compatibility reasons. They have no effect anymore. */ \ MAKE_OBSOLETE(M, Bool, distributed_cache_use_clients_cache_for_write, false) \ + MAKE_OBSOLETE(M, UInt64, export_merge_tree_partition_manifest_ttl, 86400) \ + MAKE_OBSOLETE(M, UInt64, export_merge_tree_partition_max_retries, 3) \ MAKE_OBSOLETE(M, Bool, allow_experimental_query_deduplication, false) \ MAKE_OBSOLETE(M, Bool, allow_experimental_ai_functions, false) \ MAKE_OBSOLETE(M, Bool, query_condition_cache_store_conditions_as_plaintext, false) \ @@ -9309,7 +9452,8 @@ Enable experimental table function `eval`. MAKE_OBSOLETE(M, Bool, use_text_index_dictionary_cache, false) \ MAKE_OBSOLETE(M, Bool, query_plan_use_logical_join_step, true) \ MAKE_OBSOLETE(M, Bool, query_plan_use_new_logical_join_step, true) \ - MAKE_OBSOLETE(M, UInt64, cloud_mode_database_engine, 1) + MAKE_OBSOLETE(M, UInt64, cloud_mode_database_engine, 1) \ + MAKE_OBSOLETE(M, Bool, allow_retries_in_cluster_requests, false) /** The section above is for obsolete settings. Do not add anything there. */ #endif /// __CLION_IDE__ diff --git a/src/Core/Settings.h b/src/Core/Settings.h index 449589b2431f..d0fdbeb22519 100644 --- a/src/Core/Settings.h +++ b/src/Core/Settings.h @@ -64,6 +64,7 @@ class WriteBuffer; M(CLASS_NAME, DistributedCachePoolBehaviourOnLimit) /* Cloud only */ \ M(CLASS_NAME, DistributedDDLOutputMode) \ M(CLASS_NAME, DistributedProductMode) \ + M(CLASS_NAME, ObjectStorageClusterJoinMode) \ M(CLASS_NAME, Double) \ M(CLASS_NAME, EscapingRule) \ M(CLASS_NAME, ExplainQueryPlanDefault) \ @@ -90,6 +91,8 @@ class WriteBuffer; M(CLASS_NAME, LogsLevel) \ M(CLASS_NAME, Map) \ M(CLASS_NAME, MaxThreads) \ + M(CLASS_NAME, MergeTreePartExportFileAlreadyExistsPolicy) \ + M(CLASS_NAME, MergeTreePartExportSchemaMismatchMode) \ M(CLASS_NAME, Milliseconds) \ M(CLASS_NAME, MsgPackUUIDRepresentation) \ M(CLASS_NAME, MySQLDataTypesSupport) \ @@ -128,7 +131,8 @@ class WriteBuffer; M(CLASS_NAME, DeduplicateInsertMode) \ M(CLASS_NAME, FileLikeEngineDefaultPartitionStrategy) \ M(CLASS_NAME, UniqueKeyProbeImplementation) \ - M(CLASS_NAME, SkipUnavailableShardsMode) + M(CLASS_NAME, SkipUnavailableShardsMode) \ + M(CLASS_NAME, ExportPartitionAllOnError) COMMON_SETTINGS_SUPPORTED_TYPES(Settings, DECLARE_SETTING_TRAIT) diff --git a/src/Core/SettingsChangesHistory.cpp b/src/Core/SettingsChangesHistory.cpp index 68d4a76bf996..441325235ea7 100644 --- a/src/Core/SettingsChangesHistory.cpp +++ b/src/Core/SettingsChangesHistory.cpp @@ -265,6 +265,12 @@ const VersionToSettingsChangesMap & getSettingsChangesHistory() {"allow_experimental_query_deduplication", false, false, "The setting is obsolete, the feature has been removed."}, {"query_plan_min_columns_for_join_lazy_indexing", 0, 3, "Control the minimum number of payload columns from the left side required for enabling lazy indexing optimization in JOIN"}, {"query_plan_max_limit_for_join_lazy_indexing", 1000, 1000, "Added new setting to control maximum limit value that allows to use query plan for lazy join indexing optimization. If zero, there is no limit"}, + {"object_storage_cluster_join_mode", "allow", "allow", "New setting"}, + {"export_merge_tree_partition_task_timeout_seconds", "3600", "86400", "Increase default value to make it more realistic"}, + {"export_merge_tree_part_allow_lossy_cast", false, false, "New setting to gate lossy casts in EXPORT PART/PARTITION behind explicit acknowledgment"}, + {"export_merge_tree_part_schema_mismatch_mode", "strict", "strict", "New setting to allow EXPORT PART/EXPORT PARTITION when the source table has more columns than the destination"}, + {"export_merge_tree_partition_retry_initial_backoff_seconds", 5, 5, "New setting for exponential back-off between failed part export retries in an export partition task"}, + {"export_merge_tree_partition_retry_max_backoff_seconds", 300, 300, "New setting capping the exponential back-off between failed part export retries in an export partition task"}, {"allow_experimental_database_s3_tables", false, false, "New setting to enable experimental database S3 tables (AWS Iceberg REST catalog)."}, {"statistics_max_set_size_for_exact_selectivity_estimation", 10000, 10000, "The bound on the cost of estimating the selectivity of `IN` with a large set is kept under `compatibility` with an earlier version: the previous value is deliberately equal to the new one, so that the uncapped estimation, which could add hundreds of milliseconds to the planning of a single query, is not restored."}, }); @@ -461,13 +467,15 @@ const VersionToSettingsChangesMap & getSettingsChangesHistory() }); addSettingsChanges(settings_changes_history, "26.1.3.20001.altinityantalya", { - // {"iceberg_partition_timezone", "", "", "New setting."}, + {"iceberg_partition_timezone", "", "", "New setting."}, // {"s3_propagate_credentials_to_other_storages", false, false, "New setting"}, - // {"export_merge_tree_part_filename_pattern", "", "{part_name}_{checksum}", "New setting"}, + {"export_merge_tree_part_filename_pattern", "", "{part_name}_{checksum}", "New setting"}, // {"use_parquet_metadata_cache", false, true, "Enables cache of parquet file metadata."}, // {"input_format_parquet_use_metadata_cache", true, false, "Obsolete. No-op"}, // https://github.com/Altinity/ClickHouse/pull/586 - // {"object_storage_remote_initiator_cluster", "", "", "New setting."}, + {"object_storage_remote_initiator_cluster", "", "", "New setting."}, // {"iceberg_metadata_staleness_ms", 0, 0, "New setting allowing using cached metadata version at READ operations to prevent fetching from remote catalog"}, + {"export_merge_tree_partition_task_timeout_seconds", 0, 3600, "New setting to control the timeout for export partition tasks."}, + {"export_merge_tree_partition_manifest_ttl", 180, 86400, "Reasonable default for real usage"}, }); addSettingsChanges(settings_changes_history, "26.1", { @@ -551,7 +559,6 @@ const VersionToSettingsChangesMap & getSettingsChangesHistory() {"insert_select_deduplicate", Field{"auto"}, Field{"auto"}, "New setting"}, {"output_format_pretty_named_tuples_as_json", false, true, "New setting to control whether named tuples in Pretty format are output as JSON objects"}, {"deduplicate_insert_select", "enable_even_for_bad_queries", "enable_even_for_bad_queries", "New setting, replace insert_select_deduplicate"}, - }); addSettingsChanges(settings_changes_history, "25.11", { @@ -656,29 +663,30 @@ const VersionToSettingsChangesMap & getSettingsChangesHistory() // {"allow_database_unity_catalog", false, true, "Turned ON by default for Antalya (alias)."}, // {"allow_database_glue_catalog", false, true, "Turned ON by default for Antalya (alias)."}, // {"input_format_parquet_use_metadata_cache", true, true, "New setting, turned ON by default"}, // https://github.com/Altinity/ClickHouse/pull/586 - // {"iceberg_timezone_for_timestamptz", "UTC", "UTC", "New setting."}, - // {"object_storage_remote_initiator", false, false, "New setting."}, - // {"allow_experimental_iceberg_read_optimization", true, true, "New setting."}, - // {"object_storage_cluster_join_mode", "allow", "allow", "New setting"}, - // {"lock_object_storage_task_distribution_ms", 500, 500, "New setting."}, - // {"allow_retries_in_cluster_requests", false, false, "New setting"}, - // {"allow_experimental_export_merge_tree_part", false, true, "Turned ON by default for Antalya."}, - // {"export_merge_tree_part_overwrite_file_if_exists", false, false, "New setting."}, - // {"export_merge_tree_partition_force_export", false, false, "New setting."}, - // {"export_merge_tree_partition_max_retries", 3, 3, "New setting."}, - // {"export_merge_tree_partition_manifest_ttl", 180, 180, "New setting."}, - // {"export_merge_tree_part_file_already_exists_policy", "skip", "skip", "New setting."}, + {"iceberg_timezone_for_timestamptz", "UTC", "UTC", "New setting."}, + {"object_storage_remote_initiator", false, false, "New setting."}, + {"allow_experimental_iceberg_read_optimization", true, true, "New setting."}, + {"lock_object_storage_task_distribution_ms", 500, 500, "New setting."}, + {"allow_retries_in_cluster_requests", false, false, "New setting"}, + {"allow_experimental_export_merge_tree_part", false, true, "Turned ON by default for Antalya."}, + {"export_merge_tree_part_overwrite_file_if_exists", false, false, "New setting."}, + {"export_merge_tree_partition_force_export", false, false, "New setting."}, + {"export_merge_tree_partition_max_retries", 3, 3, "New setting."}, + {"export_merge_tree_partition_manifest_ttl", 180, 180, "New setting."}, + {"export_merge_tree_part_file_already_exists_policy", "skip", "skip", "New setting."}, // {"hybrid_table_auto_cast_columns", true, true, "New setting to automatically cast Hybrid table columns when segments disagree on types. Default enabled."}, // {"allow_experimental_hybrid_table", false, false, "Added new setting to allow the Hybrid table engine."}, // {"enable_alias_marker", true, true, "New setting."}, - // {"export_merge_tree_part_max_bytes_per_file", 0, 0, "New setting."}, - // {"export_merge_tree_part_max_rows_per_file", 0, 0, "New setting."}, + {"export_merge_tree_part_max_bytes_per_file", 0, 0, "New setting."}, + {"export_merge_tree_part_max_rows_per_file", 0, 0, "New setting."}, // {"export_merge_tree_partition_lock_inside_the_task", false, false, "New setting."}, // {"export_merge_tree_partition_system_table_prefer_remote_information", true, true, "New setting."}, - // {"export_merge_tree_part_throw_on_pending_mutations", true, true, "New setting."}, - // {"export_merge_tree_part_throw_on_pending_patch_parts", true, true, "New setting."}, - // {"object_storage_cluster", "", "", "Antalya: New setting"}, - // {"object_storage_max_nodes", 0, 0, "Antalya: New setting"}, + {"export_merge_tree_part_throw_on_pending_mutations", true, true, "New setting."}, + {"export_merge_tree_part_throw_on_pending_patch_parts", true, true, "New setting."}, + {"export_merge_tree_partition_all_on_error", "throw_first", "throw_first", "New setting."}, + {"object_storage_cluster", "", "", "Antalya: New setting"}, + {"object_storage_max_nodes", 0, 0, "Antalya: New setting"}, + {"use_object_storage_list_objects_cache", false, false, "New setting."}, }); addSettingsChanges(settings_changes_history, "25.8", { diff --git a/src/Core/SettingsEnums.cpp b/src/Core/SettingsEnums.cpp index 2388a20b75c3..354eea7e032e 100644 --- a/src/Core/SettingsEnums.cpp +++ b/src/Core/SettingsEnums.cpp @@ -102,6 +102,11 @@ IMPLEMENT_SETTING_ENUM(DistributedProductMode, ErrorCodes::UNKNOWN_DISTRIBUTED_P {"global", DistributedProductMode::GLOBAL}, {"allow", DistributedProductMode::ALLOW}}) +IMPLEMENT_SETTING_ENUM(ObjectStorageClusterJoinMode, ErrorCodes::BAD_ARGUMENTS, + {{"local", ObjectStorageClusterJoinMode::LOCAL}, + {"global", ObjectStorageClusterJoinMode::GLOBAL}, + {"allow", ObjectStorageClusterJoinMode::ALLOW}}) + IMPLEMENT_SETTING_ENUM(QueryResultCacheNondeterministicFunctionHandling, ErrorCodes::BAD_ARGUMENTS, {{"throw", QueryResultCacheNondeterministicFunctionHandling::Throw}, @@ -566,4 +571,11 @@ IMPLEMENT_SETTING_ENUM( ErrorCodes::BAD_ARGUMENTS, {{"wildcard", FileLikeEngineDefaultPartitionStrategy::WILDCARD}, {"hive", FileLikeEngineDefaultPartitionStrategy::HIVE}}) + +IMPLEMENT_SETTING_AUTO_ENUM(MergeTreePartExportFileAlreadyExistsPolicy, ErrorCodes::BAD_ARGUMENTS); + +IMPLEMENT_SETTING_AUTO_ENUM(MergeTreePartExportSchemaMismatchMode, ErrorCodes::BAD_ARGUMENTS); + +IMPLEMENT_SETTING_AUTO_ENUM(ExportPartitionAllOnError, ErrorCodes::BAD_ARGUMENTS); + } diff --git a/src/Core/SettingsEnums.h b/src/Core/SettingsEnums.h index 4f05e00c38a0..2203a313aa2b 100644 --- a/src/Core/SettingsEnums.h +++ b/src/Core/SettingsEnums.h @@ -168,6 +168,16 @@ enum class DistributedProductMode : uint8_t DECLARE_SETTING_ENUM(DistributedProductMode) +/// The setting for executing object storage cluster function or table JOIN sections. +enum class ObjectStorageClusterJoinMode : uint8_t +{ + LOCAL, /// Convert to local query + GLOBAL, /// Convert to global query + ALLOW /// Enable +}; + +DECLARE_SETTING_ENUM(ObjectStorageClusterJoinMode) + /// How the query result cache handles queries with non-deterministic functions, e.g. now() enum class QueryResultCacheNondeterministicFunctionHandling : uint8_t { @@ -661,4 +671,30 @@ enum class FileLikeEngineDefaultPartitionStrategy : uint8_t }; DECLARE_SETTING_ENUM(FileLikeEngineDefaultPartitionStrategy) +enum class MergeTreePartExportFileAlreadyExistsPolicy : uint8_t +{ + skip, + error, + overwrite, +}; + +DECLARE_SETTING_ENUM(MergeTreePartExportFileAlreadyExistsPolicy) + +enum class MergeTreePartExportSchemaMismatchMode : uint8_t +{ + strict, + ignore_extra_source_columns_by_position, +}; + +DECLARE_SETTING_ENUM(MergeTreePartExportSchemaMismatchMode) + +enum class ExportPartitionAllOnError : uint8_t +{ + throw_first, + collect, + skip_conflicts, +}; + +DECLARE_SETTING_ENUM(ExportPartitionAllOnError) + } diff --git a/src/Databases/DataLake/Common.cpp b/src/Databases/DataLake/Common.cpp index 3046f73b115b..2b313b4b7655 100644 --- a/src/Databases/DataLake/Common.cpp +++ b/src/Databases/DataLake/Common.cpp @@ -61,14 +61,14 @@ std::vector splitTypeArguments(const String & type_str) return args; } -DB::DataTypePtr getType(const String & type_name, bool nullable, const String & prefix) +DB::DataTypePtr getType(const String & type_name, bool nullable, DB::ContextPtr context, const String & prefix) { String name = trim(type_name); if (name.starts_with("array<") && name.ends_with(">")) { String inner = name.substr(6, name.size() - 7); - return std::make_shared(getType(inner, nullable)); + return std::make_shared(getType(inner, nullable, context)); } if (name.starts_with("map<") && name.ends_with(">")) @@ -79,7 +79,7 @@ DB::DataTypePtr getType(const String & type_name, bool nullable, const String & if (args.size() != 2) throw DB::Exception(DB::ErrorCodes::DATALAKE_DATABASE_ERROR, "Invalid data type {}", type_name); - return std::make_shared(getType(args[0], false), getType(args[1], nullable)); + return std::make_shared(getType(args[0], false, context), getType(args[1], nullable, context)); } if (name.starts_with("struct<") && name.ends_with(">")) @@ -101,13 +101,13 @@ DB::DataTypePtr getType(const String & type_name, bool nullable, const String & String full_field_name = prefix.empty() ? field_name : prefix + "." + field_name; field_names.push_back(full_field_name); - field_types.push_back(getType(field_type, nullable, full_field_name)); + field_types.push_back(getType(field_type, nullable, context, full_field_name)); } return std::make_shared(field_types, field_names); } - return nullable ? DB::makeNullable(DB::Iceberg::IcebergSchemaProcessor::getSimpleType(name)) - : DB::Iceberg::IcebergSchemaProcessor::getSimpleType(name); + return nullable ? DB::makeNullable(DB::Iceberg::IcebergSchemaProcessor::getSimpleType(name, context)) + : DB::Iceberg::IcebergSchemaProcessor::getSimpleType(name, context); } std::pair parseTableName(const std::string & name) diff --git a/src/Databases/DataLake/Common.h b/src/Databases/DataLake/Common.h index cd4b6214e343..9b0dd7c626a6 100644 --- a/src/Databases/DataLake/Common.h +++ b/src/Databases/DataLake/Common.h @@ -2,6 +2,7 @@ #include #include +#include namespace DataLake { @@ -10,7 +11,7 @@ String trim(const String & str); std::vector splitTypeArguments(const String & type_str); -DB::DataTypePtr getType(const String & type_name, bool nullable, const String & prefix = ""); +DB::DataTypePtr getType(const String & type_name, bool nullable, DB::ContextPtr context, const String & prefix = ""); /// Parse a string, containing at least one dot, into a two substrings: /// A.B.C.D.E -> A.B.C.D and E, where diff --git a/src/Databases/DataLake/DataLakeConstants.h b/src/Databases/DataLake/DataLakeConstants.h index bc5c96ac093f..0e2decbdb43d 100644 --- a/src/Databases/DataLake/DataLakeConstants.h +++ b/src/Databases/DataLake/DataLakeConstants.h @@ -9,6 +9,7 @@ namespace DataLake { static constexpr auto DATABASE_ENGINE_NAME = "DataLakeCatalog"; +static constexpr auto DATABASE_ALIAS_NAME = "Iceberg"; static constexpr std::string_view FILE_PATH_PREFIX = "file:/"; /// Some catalogs (Unity or Glue) may store not only Iceberg/DeltaLake tables but other kinds of "tables" diff --git a/src/Databases/DataLake/DatabaseDataLake.cpp b/src/Databases/DataLake/DatabaseDataLake.cpp index 2295ffe70173..d0c25a2cb838 100644 --- a/src/Databases/DataLake/DatabaseDataLake.cpp +++ b/src/Databases/DataLake/DatabaseDataLake.cpp @@ -72,6 +72,7 @@ namespace DatabaseDataLakeSetting extern const DatabaseDataLakeSettingsString oauth_server_uri; extern const DatabaseDataLakeSettingsBool oauth_server_use_request_body; extern const DatabaseDataLakeSettingsBool vended_credentials; + extern const DatabaseDataLakeSettingsString object_storage_cluster; extern const DatabaseDataLakeSettingsString aws_access_key_id; extern const DatabaseDataLakeSettingsString aws_secret_access_key; extern const DatabaseDataLakeSettingsString region; @@ -86,6 +87,7 @@ namespace DatabaseDataLakeSetting extern const DatabaseDataLakeSettingsBool onelake_use_blob_endpoint; extern const DatabaseDataLakeSettingsString dlf_access_key_id; extern const DatabaseDataLakeSettingsString dlf_access_key_secret; + extern const DatabaseDataLakeSettingsString namespaces; extern const DatabaseDataLakeSettingsString google_project_id; extern const DatabaseDataLakeSettingsString google_service_account; extern const DatabaseDataLakeSettingsString google_metadata_service; @@ -247,6 +249,7 @@ void DatabaseDataLake::initialize() const .aws_access_key_id = settings[DatabaseDataLakeSetting::aws_access_key_id].value, .aws_secret_access_key = settings[DatabaseDataLakeSetting::aws_secret_access_key].value, .region = settings[DatabaseDataLakeSetting::region].value, + .namespaces = settings[DatabaseDataLakeSetting::namespaces].value, .aws_role_arn = settings[DatabaseDataLakeSetting::aws_role_arn].value, .aws_role_session_name = settings[DatabaseDataLakeSetting::aws_role_session_name].value, .aws_external_id = settings[DatabaseDataLakeSetting::aws_external_id].value, @@ -264,6 +267,7 @@ void DatabaseDataLake::initialize() const settings[DatabaseDataLakeSetting::auth_header], settings[DatabaseDataLakeSetting::oauth_server_uri].value, settings[DatabaseDataLakeSetting::oauth_server_use_request_body].value, + settings[DatabaseDataLakeSetting::namespaces].value, Context::getGlobalContextInstance()); break; } @@ -279,6 +283,7 @@ void DatabaseDataLake::initialize() const settings[DatabaseDataLakeSetting::auth_header], settings[DatabaseDataLakeSetting::oauth_server_uri].value, settings[DatabaseDataLakeSetting::oauth_server_use_request_body].value, + settings[DatabaseDataLakeSetting::namespaces].value, Context::getGlobalContextInstance()); break; } @@ -295,6 +300,7 @@ void DatabaseDataLake::initialize() const settings[DatabaseDataLakeSetting::auth_header], settings[DatabaseDataLakeSetting::oauth_server_uri].value, settings[DatabaseDataLakeSetting::oauth_server_use_request_body].value, + settings[DatabaseDataLakeSetting::namespaces].value, Context::getGlobalContextInstance()); break; } @@ -316,6 +322,7 @@ void DatabaseDataLake::initialize() const onelake_auth_scope, settings[DatabaseDataLakeSetting::oauth_server_uri].value, settings[DatabaseDataLakeSetting::oauth_server_use_request_body].value, + settings[DatabaseDataLakeSetting::namespaces].value, Context::getGlobalContextInstance()); break; } @@ -346,6 +353,7 @@ void DatabaseDataLake::initialize() const google_adc_client_secret, google_adc_refresh_token, google_adc_quota_project_id, + settings[DatabaseDataLakeSetting::namespaces].value, Context::getGlobalContextInstance(), allow_server_credentials_in_user_queries); break; @@ -356,6 +364,7 @@ void DatabaseDataLake::initialize() const settings[DatabaseDataLakeSetting::warehouse].value, url, settings[DatabaseDataLakeSetting::catalog_credential].value, + settings[DatabaseDataLakeSetting::namespaces].value, Context::getGlobalContextInstance()); break; } @@ -496,12 +505,15 @@ void DatabaseDataLake::resetCatalog(String reason) const catalog_unavailable_reason = std::move(reason); } -std::shared_ptr DatabaseDataLake::getConfiguration( +StorageObjectStorageConfigurationPtr DatabaseDataLake::getConfiguration( DatabaseDataLakeStorageType type, DataLakeStorageSettingsPtr storage_settings) const { /// TODO: add tests for azure, local storage types. + const auto settings_version = database_settings.get(); + const DatabaseDataLakeSettings & settings = *settings_version; + auto catalog = getCatalog(); switch (catalog->getCatalogType()) { @@ -533,24 +545,24 @@ std::shared_ptr DatabaseDataLake::getConfigur #if USE_AWS_S3 case DB::DatabaseDataLakeStorageType::S3: { - return std::make_shared(storage_settings); + return std::make_shared(storage_settings, settings[DatabaseDataLakeSetting::namespaces].value); } #endif #if USE_AZURE_BLOB_STORAGE case DB::DatabaseDataLakeStorageType::Azure: { - return std::make_shared(storage_settings); + return std::make_shared(storage_settings, settings[DatabaseDataLakeSetting::namespaces].value); } #endif #if USE_HDFS case DB::DatabaseDataLakeStorageType::HDFS: { - return std::make_shared(storage_settings); + return std::make_shared(storage_settings, settings[DatabaseDataLakeSetting::namespaces].value); } #endif case DB::DatabaseDataLakeStorageType::Local: { - return std::make_shared(storage_settings); + return std::make_shared(storage_settings, settings[DatabaseDataLakeSetting::namespaces].value); } /// Fake storage in case when catalog store not only /// primary-type tables (DeltaLake or Iceberg), but for @@ -562,7 +574,7 @@ std::shared_ptr DatabaseDataLake::getConfigur /// dependencies and the most lightweight case DB::DatabaseDataLakeStorageType::Other: { - return std::make_shared(storage_settings); + return std::make_shared(storage_settings, settings[DatabaseDataLakeSetting::namespaces].value); } #if !USE_AWS_S3 || !USE_AZURE_BLOB_STORAGE || !USE_HDFS default: @@ -579,7 +591,7 @@ std::shared_ptr DatabaseDataLake::getConfigur #if USE_AWS_S3 case DB::DatabaseDataLakeStorageType::S3: { - return std::make_shared(storage_settings); + return std::make_shared(storage_settings, settings[DatabaseDataLakeSetting::namespaces].value); } #endif #if USE_AZURE_BLOB_STORAGE @@ -590,7 +602,7 @@ std::shared_ptr DatabaseDataLake::getConfigur #endif case DB::DatabaseDataLakeStorageType::Local: { - return std::make_shared(storage_settings); + return std::make_shared(storage_settings, settings[DatabaseDataLakeSetting::namespaces].value); } /// Fake storage in case when catalog store not only /// primary-type tables (DeltaLake or Iceberg), but for @@ -602,7 +614,7 @@ std::shared_ptr DatabaseDataLake::getConfigur /// dependencies and the most lightweight case DB::DatabaseDataLakeStorageType::Other: { - return std::make_shared(storage_settings); + return std::make_shared(storage_settings, settings[DatabaseDataLakeSetting::namespaces].value); } default: throw Exception(ErrorCodes::BAD_ARGUMENTS, @@ -617,12 +629,12 @@ std::shared_ptr DatabaseDataLake::getConfigur #if USE_AWS_S3 case DB::DatabaseDataLakeStorageType::S3: { - return std::make_shared(storage_settings); + return std::make_shared(storage_settings, settings[DatabaseDataLakeSetting::namespaces].value); } #endif case DB::DatabaseDataLakeStorageType::Other: { - return std::make_shared(storage_settings); + return std::make_shared(storage_settings, settings[DatabaseDataLakeSetting::namespaces].value); } default: throw Exception(ErrorCodes::BAD_ARGUMENTS, @@ -735,7 +747,7 @@ StoragePtr DatabaseDataLake::tryGetTableImpl(const String & name, ContextPtr con auto [namespace_name, table_name] = DataLake::parseTableName(name); - if (!catalog->tryGetTableMetadata(namespace_name, table_name, table_metadata)) + if (!catalog->tryGetTableMetadata(namespace_name, table_name, context_, table_metadata)) return nullptr; if (ignore_if_not_iceberg && !table_metadata.isDefaultReadableTable()) return nullptr; @@ -903,7 +915,7 @@ StoragePtr DatabaseDataLake::tryGetTableImpl(const String & name, ContextPtr con /// with_table_structure = false: because there will be /// no table structure in table definition AST. - StorageObjectStorageConfiguration::initialize(*configuration, args, context_copy, /* with_table_structure */false); + configuration->initialize(args, context_copy, /* with_table_structure */false); const auto & query_settings = context_->getSettingsRef(); @@ -936,65 +948,37 @@ StoragePtr DatabaseDataLake::tryGetTableImpl(const String & name, ContextPtr con const auto catalog_uuid = table_metadata.getTableUUID(); const UUID table_uuid = catalog_uuid ? parseFromString(*catalog_uuid) : UUIDHelpers::Nil; - if (can_use_parallel_replicas && !is_secondary_query) - { - auto storage_id = StorageID(getDatabaseName(), name, table_uuid); - auto storage_cluster = std::make_shared( - parallel_replicas_cluster_name, - configuration, - configuration->createObjectStorage(context_copy, /* is_readonly */ false, get_credentials_refresh_callback(storage_id)), - storage_id, - columns, - ConstraintsDescription{}, - nullptr, - context_, - /// Use is_table_function = true, - /// because this table is actually stateless like a table function. - /* is_table_function */true, - getFormatSettings(context_copy), - getCatalog()); - - if (context_->hasQueryContext() && context_->getSettingsRef()[Setting::log_queries]) - context_->getQueryContext()->addQueryFactoriesInfo(Context::QueryLogFactories::Storage, storage_cluster->getName()); - - storage_cluster->startup(); - return storage_cluster; - } + std::string cluster_name = configuration->isClusterSupported() ? settings[DatabaseDataLakeSetting::object_storage_cluster].value : ""; - /// Unlike table functions (s3, url, etc.), DataLake tables are queried as - /// `SELECT * FROM catalog.table` — the query sent to shards cannot be rewritten - /// into a Cluster table function variant. So when the initiator created a - /// StorageObjectStorageCluster (the branch above) and the shard is collaborating - /// with it, we need distributed_processing=true to use the task iterator. - const bool distributed_processing = - context_->getClientInfo().collaborate_with_initiator - && can_use_parallel_replicas; + if (cluster_name.empty() && can_use_parallel_replicas && !is_secondary_query) + cluster_name = parallel_replicas_cluster_name; - auto result_storage = std::make_shared( + auto storage_cluster = std::make_shared( + cluster_name, configuration, configuration->createObjectStorage(context_copy, /* is_readonly */ false, get_credentials_refresh_callback(StorageID(getDatabaseName(), name, table_uuid))), - context_copy, StorageID(getDatabaseName(), name, table_uuid), /* columns */columns, /* constraints */ConstraintsDescription{}, - /* comment */"", + /* partition_by */nullptr, + /* order_by */nullptr, + context_copy, + /* comment */ "", getFormatSettings(context_copy), LoadingStrictnessLevel::CREATE, getCatalog(), /* if_not_exists*/true, /* is_datalake_query*/true, - distributed_processing, - /* partition_by */nullptr, - /* order_by */nullptr, /// Use is_table_function = true, /// because this table is actually stateless like a table function. /* is_table_function */true, /* lazy_init */true); if (context_->hasQueryContext() && context_->getSettingsRef()[Setting::log_queries]) - context_->getQueryContext()->addQueryFactoriesInfo(Context::QueryLogFactories::Storage, result_storage->getName()); + context_->getQueryContext()->addQueryFactoriesInfo(Context::QueryLogFactories::Storage, storage_cluster->getName()); - return result_storage; + storage_cluster->startup(); + return storage_cluster; } void DatabaseDataLake::dropTable( /// NOLINT @@ -1355,7 +1339,7 @@ void DatabaseDataLake::applySettingsChanges(const SettingsChanges & settings_cha ASTPtr DatabaseDataLake::getCreateTableQueryImpl( const String & name, - ContextPtr /* context_ */, + ContextPtr context_, bool throw_on_error) const { const auto settings_version = database_settings.get(); @@ -1368,7 +1352,7 @@ ASTPtr DatabaseDataLake::getCreateTableQueryImpl( const auto [namespace_name, table_name] = DataLake::parseTableName(name); - if (!catalog->tryGetTableMetadata(namespace_name, table_name, table_metadata)) + if (!catalog->tryGetTableMetadata(namespace_name, table_name, context_, table_metadata)) { if (throw_on_error) throw Exception(ErrorCodes::CANNOT_GET_CREATE_TABLE_QUERY, "Table `{}` doesn't exist", name); @@ -1799,6 +1783,18 @@ SELECT count() from database_name.table_name; )DOCS_MD", .syntax = "ENGINE = DataLakeCatalog('catalog_url'[, 'user', 'password']) SETTINGS catalog_type = '...'", .related = {}}); + factory.registerDatabase("Iceberg", create_fn, { + .supports_arguments = true, + .supports_settings = true, + .is_external = true, + }, Documentation{ + .description = R"DOCS_MD( +The `Iceberg` database engine is the legacy name of [`DataLakeCatalog`](/engines/database-engines/datalake) +and is kept for compatibility with databases created before the rename. It accepts the same arguments and +the same settings, and supports the same catalogs. Use `DataLakeCatalog` for new databases. +)DOCS_MD", + .syntax = "ENGINE = Iceberg('catalog_url'[, 'user', 'password']) SETTINGS catalog_type = '...'", + .related = {"DataLakeCatalog"}}); } } diff --git a/src/Databases/DataLake/DatabaseDataLake.h b/src/Databases/DataLake/DatabaseDataLake.h index bdf9c057153e..89e1baee906c 100644 --- a/src/Databases/DataLake/DatabaseDataLake.h +++ b/src/Databases/DataLake/DatabaseDataLake.h @@ -135,7 +135,7 @@ class DatabaseDataLake final : public IDatabase, WithContext /// recording `reason` when it is dropped because it could not be built (empty otherwise). void resetCatalog(String reason) const TSA_REQUIRES(catalog_mutex); - std::shared_ptr getConfiguration( + StorageObjectStorageConfigurationPtr getConfiguration( DatabaseDataLakeStorageType type, DataLakeStorageSettingsPtr storage_settings) const; diff --git a/src/Databases/DataLake/DatabaseDataLakeSettings.cpp b/src/Databases/DataLake/DatabaseDataLakeSettings.cpp index 421eae178c53..866d124fbe0f 100644 --- a/src/Databases/DataLake/DatabaseDataLakeSettings.cpp +++ b/src/Databases/DataLake/DatabaseDataLakeSettings.cpp @@ -51,6 +51,7 @@ namespace ErrorCodes DECLARE(String, dlf_access_key_id, "", "Access id of DLF token for Paimon REST Catalog", 0) \ DECLARE(String, dlf_access_key_secret, "", "Access secret of DLF token for Paimon REST Catalog", 0) \ DECLARE(Bool, force_add_bucket, false, "When constructing object-storage URLs from the catalog-provided table location and storage_endpoint, prepend the bucket/container name even if the endpoint already contains it. Useful for catalogs that hand back paths without the bucket and expect it to be added at URL construction (Polaris-style paths).", 0) \ + DECLARE(String, namespaces, "*", "Comma-separated list of allowed namespaces", 0) \ #define LIST_OF_DATABASE_ICEBERG_SETTINGS(M, ALIAS) \ DATABASE_ICEBERG_RELATED_SETTINGS(M, ALIAS) \ diff --git a/src/Databases/DataLake/GlueCatalog.cpp b/src/Databases/DataLake/GlueCatalog.cpp index 859a73c0fc0a..060d8fdd5742 100644 --- a/src/Databases/DataLake/GlueCatalog.cpp +++ b/src/Databases/DataLake/GlueCatalog.cpp @@ -55,11 +55,15 @@ #include #include +#include +#include + namespace DB::ErrorCodes { extern const int BAD_ARGUMENTS; extern const int DATALAKE_DATABASE_ERROR; extern const int FAULT_INJECTED; + extern const int CATALOG_NAMESPACE_DISABLED; } namespace DB::FailPoints @@ -212,9 +216,9 @@ GlueCatalog::GlueCatalog( LOG_TRACE(log, "Creating AWS glue client with credentials empty {}, region '{}', endpoint '{}'", credentials.IsEmpty(), region, endpoint); } + boost::split(allowed_namespaces, settings.namespaces, boost::is_any_of(", "), boost::token_compress_on); credentials_provider = DB::S3::getCredentialsProvider(poco_config, credentials, creds_config); glue_client = std::make_unique(credentials_provider, endpoint_provider, client_configuration); - } GlueCatalog::~GlueCatalog() = default; @@ -247,8 +251,9 @@ DataLake::ICatalog::Namespaces GlueCatalog::getDatabases(const std::string & pre for (const auto & db : dbs) { const auto & db_name = db.GetName(); - if (!db_name.starts_with(prefix)) + if (!isNamespaceAllowed(db_name) || !db_name.starts_with(prefix)) continue; + result.push_back(db_name); if (limit != 0 && result.size() >= limit) break; @@ -357,15 +362,22 @@ CatalogTables GlueCatalog::listTablesInNamespaceDirect(const std::string & names bool GlueCatalog::existsTable(const std::string & database_name, const std::string & table_name) const { + if (!isNamespaceAllowed(database_name)) + throw DB::Exception(DB::ErrorCodes::CATALOG_NAMESPACE_DISABLED, "Namespace {} is filtered by `namespaces` database parameter", database_name); + TableMetadata metadata; - return tryGetTableMetadata(database_name, table_name, metadata); + return tryGetTableMetadata(database_name, table_name, getContext(), metadata); } bool GlueCatalog::tryGetTableMetadata( const std::string & database_name, const std::string & table_name, + DB::ContextPtr /* context_ */, TableMetadata & result) const { + if (!isNamespaceAllowed(database_name)) + throw DB::Exception(DB::ErrorCodes::CATALOG_NAMESPACE_DISABLED, "Namespace {} is filtered by `namespaces` database parameter", database_name); + Aws::Glue::Model::GetTableRequest request; request.SetDatabaseName(database_name); request.SetName(table_name); @@ -461,7 +473,7 @@ bool GlueCatalog::tryGetTableMetadata( column_type = getActualTimestampType(column.GetName(), result, column_type); } - schema.push_back({column.GetName(), getType(column_type, can_be_nullable)}); + schema.push_back({column.GetName(), getType(column_type, can_be_nullable, getContext())}); } result.setSchema(schema); } @@ -483,9 +495,10 @@ bool GlueCatalog::tryGetTableMetadata( void GlueCatalog::getTableMetadata( const std::string & database_name, const std::string & table_name, + DB::ContextPtr context_, TableMetadata & result) const { - if (!tryGetTableMetadata(database_name, table_name, result)) + if (!tryGetTableMetadata(database_name, table_name, context_, result)) { throw DB::Exception( DB::ErrorCodes::DATALAKE_DATABASE_ERROR, @@ -615,8 +628,8 @@ GlueCatalog::ObjectStorageWithPath GlueCatalog::createObjectStorageForEarlyTable auto storage_settings = std::make_shared(); storage_settings->loadFromSettingsChanges(settings.allChanged()); - auto configuration = std::make_shared(storage_settings); - DB::StorageObjectStorageConfiguration::initialize(*configuration, args, getContext(), false); + auto configuration = std::make_shared(storage_settings, settings.namespaces); + configuration->initialize(args, getContext(), false); auto object_storage = configuration->createObjectStorage(getContext(), true, {}); @@ -685,6 +698,11 @@ void GlueCatalog::createNamespaceIfNotExists(const String & namespace_name, cons void GlueCatalog::createTable(const String & namespace_name, const String & table_name, const String & new_metadata_path, Poco::JSON::Object::Ptr /*metadata_content*/) const { + if (!isNamespaceAllowed(namespace_name)) + throw DB::Exception(DB::ErrorCodes::CATALOG_NAMESPACE_DISABLED, + "Failed to create table {}, namespace {} is filtered by `namespaces` database parameter", + table_name, namespace_name); + Aws::Glue::Model::CreateTableRequest request; request.SetDatabaseName(namespace_name); @@ -777,6 +795,11 @@ bool GlueCatalog::updateSchema( void GlueCatalog::dropTable(const String & namespace_name, const String & table_name, bool /*delete_data*/) const { + if (!isNamespaceAllowed(namespace_name)) + throw DB::Exception(DB::ErrorCodes::CATALOG_NAMESPACE_DISABLED, + "Failed to drop table {}, namespace {} is filtered by `namespaces` database parameter", + table_name, namespace_name); + Aws::Glue::Model::DeleteTableRequest request; request.SetDatabaseName(namespace_name); request.SetName(table_name); @@ -796,6 +819,11 @@ void GlueCatalog::dropTable(const String & namespace_name, const String & table_ response.GetError().GetMessage()); } +bool GlueCatalog::isNamespaceAllowed(const std::string & namespace_) const +{ + return allowed_namespaces.contains("*") || allowed_namespaces.contains(namespace_); +} + } #endif diff --git a/src/Databases/DataLake/GlueCatalog.h b/src/Databases/DataLake/GlueCatalog.h index 5007e5e93974..3277e1a5f1cb 100644 --- a/src/Databases/DataLake/GlueCatalog.h +++ b/src/Databases/DataLake/GlueCatalog.h @@ -49,11 +49,13 @@ class GlueCatalog final : public ICatalog, private DB::WithContext void getTableMetadata( const std::string & database_name, const std::string & table_name, + DB::ContextPtr context_, TableMetadata & result) const override; bool tryGetTableMetadata( const std::string & database_name, const std::string & table_name, + DB::ContextPtr context_, TableMetadata & result) const override; std::optional getStorageType() const override @@ -103,6 +105,9 @@ class GlueCatalog final : public ICatalog, private DB::WithContext std::string region; CatalogSettings settings; DB::ASTPtr table_engine_definition; + std::unordered_set allowed_namespaces; + + bool isNamespaceAllowed(const std::string & namespace_) const; DataLake::ICatalog::Namespaces getDatabases(const std::string & prefix, size_t limit = 0) const; CatalogTables getTablesForDatabase(const std::string & db_name, size_t limit = 0) const; diff --git a/src/Databases/DataLake/HiveCatalog.cpp b/src/Databases/DataLake/HiveCatalog.cpp index 086f389e81a1..42aeadc6fdc5 100644 --- a/src/Databases/DataLake/HiveCatalog.cpp +++ b/src/Databases/DataLake/HiveCatalog.cpp @@ -206,13 +206,21 @@ bool HiveCatalog::existsTable(const std::string & namespace_name, const std::str return true; } -void HiveCatalog::getTableMetadata(const std::string & namespace_name, const std::string & table_name, TableMetadata & result) const +void HiveCatalog::getTableMetadata( + const std::string & namespace_name, + const std::string & table_name, + DB::ContextPtr context_, + TableMetadata & result) const { - if (!tryGetTableMetadata(namespace_name, table_name, result)) + if (!tryGetTableMetadata(namespace_name, table_name, context_, result)) throw DB::Exception(DB::ErrorCodes::DATALAKE_DATABASE_ERROR, "No response from iceberg catalog"); } -bool HiveCatalog::tryGetTableMetadata(const std::string & namespace_name, const std::string & table_name, TableMetadata & result) const +bool HiveCatalog::tryGetTableMetadata( + const std::string & namespace_name, + const std::string & table_name, + DB::ContextPtr context_, + TableMetadata & result) const { Apache::Hadoop::Hive::Table table; @@ -237,7 +245,7 @@ bool HiveCatalog::tryGetTableMetadata(const std::string & namespace_name, const auto columns = table.sd.cols; for (const auto & column : columns) { - schema.push_back({column.name, getType(column.type, true)}); + schema.push_back({column.name, getType(column.type, true, context_)}); } result.setSchema(schema); } diff --git a/src/Databases/DataLake/HiveCatalog.h b/src/Databases/DataLake/HiveCatalog.h index 0d99a8c33051..aa8980fc7a14 100644 --- a/src/Databases/DataLake/HiveCatalog.h +++ b/src/Databases/DataLake/HiveCatalog.h @@ -40,9 +40,17 @@ class HiveCatalog final : public ICatalog, private DB::WithContext bool existsTable(const std::string & namespace_name, const std::string & table_name) const override; - void getTableMetadata(const std::string & namespace_name, const std::string & table_name, TableMetadata & result) const override; - - bool tryGetTableMetadata(const std::string & namespace_name, const std::string & table_name, TableMetadata & result) const override; + void getTableMetadata( + const std::string & namespace_name, + const std::string & table_name, + DB::ContextPtr context_, + TableMetadata & result) const override; + + bool tryGetTableMetadata( + const std::string & namespace_name, + const std::string & table_name, + DB::ContextPtr context_, + TableMetadata & result) const override; std::optional getStorageType() const override; diff --git a/src/Databases/DataLake/ICatalog.h b/src/Databases/DataLake/ICatalog.h index e79ebf686f68..6aa932f26a3f 100644 --- a/src/Databases/DataLake/ICatalog.h +++ b/src/Databases/DataLake/ICatalog.h @@ -16,6 +16,9 @@ namespace DB { struct DatabaseDataLakeSettings; + +class Context; +using ContextPtr = std::shared_ptr; } namespace DataLake @@ -186,6 +189,7 @@ struct CatalogSettings String aws_access_key_id; String aws_secret_access_key; String region; + String namespaces; String aws_role_arn; String aws_role_session_name; String aws_external_id; @@ -232,6 +236,7 @@ class ICatalog virtual void getTableMetadata( const std::string & namespace_name, const std::string & table_name, + DB::ContextPtr context, TableMetadata & result) const = 0; /// Get table metadata in the given namespace. @@ -239,6 +244,7 @@ class ICatalog virtual bool tryGetTableMetadata( const std::string & namespace_name, const std::string & table_name, + DB::ContextPtr context, TableMetadata & result) const = 0; /// Get storage type, where Iceberg tables' data is stored. diff --git a/src/Databases/DataLake/PaimonRestCatalog.cpp b/src/Databases/DataLake/PaimonRestCatalog.cpp index ea0207a4ee26..854532e8f0a7 100644 --- a/src/Databases/DataLake/PaimonRestCatalog.cpp +++ b/src/Databases/DataLake/PaimonRestCatalog.cpp @@ -485,7 +485,7 @@ bool PaimonRestCatalog::existsTable(const String & database_name, const String & return true; } -bool PaimonRestCatalog::tryGetTableMetadata(const String & database_name, const String & table_name, TableMetadata & result) const +bool PaimonRestCatalog::tryGetTableMetadata(const String & database_name, const String & table_name, DB::ContextPtr /*context_*/, TableMetadata & result) const { try { @@ -611,9 +611,9 @@ Poco::JSON::Object::Ptr PaimonRestCatalog::requestRest( return json.extract(); } -void PaimonRestCatalog::getTableMetadata(const String & database_name, const String & table_name, TableMetadata & result) const +void PaimonRestCatalog::getTableMetadata(const String & database_name, const String & table_name, DB::ContextPtr context_, TableMetadata & result) const { - if (!tryGetTableMetadata(database_name, table_name, result)) + if (!tryGetTableMetadata(database_name, table_name, context_, result)) { throw DB::Exception(DB::ErrorCodes::DATALAKE_DATABASE_ERROR, "No response from paimon rest catalog"); } diff --git a/src/Databases/DataLake/PaimonRestCatalog.h b/src/Databases/DataLake/PaimonRestCatalog.h index ff5d9dee9720..b77225a7636b 100644 --- a/src/Databases/DataLake/PaimonRestCatalog.h +++ b/src/Databases/DataLake/PaimonRestCatalog.h @@ -90,9 +90,9 @@ class PaimonRestCatalog final : public ICatalog, private DB::WithContext bool existsTable(const String & database_name, const String & table_name) const override; - void getTableMetadata(const String & database_name, const String & table_name, TableMetadata & result) const override; + void getTableMetadata(const String & database_name, const String & table_name, DB::ContextPtr context_, TableMetadata & result) const override; - bool tryGetTableMetadata(const String & database_name, const String & table_name, TableMetadata & result) const override; + bool tryGetTableMetadata(const String & database_name, const String & table_name, DB::ContextPtr /*context_*/, TableMetadata & result) const override; std::optional getStorageType() const override { return storage_type; } diff --git a/src/Databases/DataLake/RestCatalog.cpp b/src/Databases/DataLake/RestCatalog.cpp index 9c15ff5774b6..08a98c21322b 100644 --- a/src/Databases/DataLake/RestCatalog.cpp +++ b/src/Databases/DataLake/RestCatalog.cpp @@ -56,6 +56,9 @@ #include #include +#include +#include + namespace DB::ErrorCodes { @@ -64,6 +67,7 @@ namespace DB::ErrorCodes extern const int BAD_ARGUMENTS; extern const int FAULT_INJECTED; extern const int ACCESS_DENIED; + extern const int CATALOG_NAMESPACE_DISABLED; } namespace DB::Setting @@ -222,6 +226,7 @@ RestCatalog::RestCatalog( const std::string & auth_header_, const std::string & oauth_server_uri_, bool oauth_server_use_request_body_, + const std::string & namespaces_, DB::ContextPtr context_) : ICatalog(warehouse_) , DB::WithContext(context_) @@ -230,6 +235,7 @@ RestCatalog::RestCatalog( , auth_scope(auth_scope_) , oauth_server_uri(oauth_server_uri_) , oauth_server_use_request_body(oauth_server_use_request_body_) + , allowed_namespaces(namespaces_) { CatalogState initial_state; if (!catalog_credential_.empty()) @@ -256,6 +262,7 @@ RestCatalog::RestCatalog( const std::string & auth_scope_, const std::string & oauth_server_uri_, bool oauth_server_use_request_body_, + const std::string & namespaces_, DB::ContextPtr context_) : ICatalog(warehouse_) , DB::WithContext(context_) @@ -264,6 +271,7 @@ RestCatalog::RestCatalog( , auth_scope(auth_scope_) , oauth_server_uri(oauth_server_uri_) , oauth_server_use_request_body(oauth_server_use_request_body_) + , allowed_namespaces(namespaces_) { } @@ -381,8 +389,9 @@ OneLakeCatalog::OneLakeCatalog( const std::string & auth_scope_, const std::string & oauth_server_uri_, bool oauth_server_use_request_body_, + const std::string & namespaces_, DB::ContextPtr context_) - : RestCatalog(warehouse_, base_url_, auth_scope_, oauth_server_uri_, oauth_server_use_request_body_, context_) + : RestCatalog(warehouse_, base_url_, auth_scope_, oauth_server_uri_, oauth_server_use_request_body_, namespaces_, context_) { CatalogState initial_state; initial_state.tenant_id = onelake_tenant_id; @@ -670,8 +679,9 @@ HorizonCatalog::HorizonCatalog( const std::string & auth_header_, const std::string & oauth_server_uri_, bool oauth_server_use_request_body_, + const std::string & namespaces_, DB::ContextPtr context_) - : RestCatalog(warehouse_, base_url_, auth_scope_, oauth_server_uri_, oauth_server_use_request_body_, context_) + : RestCatalog(warehouse_, base_url_, auth_scope_, oauth_server_uri_, oauth_server_use_request_body_, namespaces_, context_) { CatalogState initial_state; if (!catalog_credential_.empty()) @@ -1018,9 +1028,10 @@ BigLakeCatalog::BigLakeCatalog( const std::string & google_adc_client_secret_, const std::string & google_adc_refresh_token_, const std::string & google_adc_quota_project_id_, + const std::string & namespaces_, DB::ContextPtr context_, bool allow_server_credentials_in_user_queries_) - : RestCatalog(warehouse_, base_url_, "", "", false, context_) + : RestCatalog(warehouse_, base_url_, "", "", false, namespaces_, context_) , google_project_id(google_project_id_) , google_service_account(google_service_account_) , google_metadata_service(google_metadata_service_) @@ -1287,6 +1298,10 @@ bool RestCatalog::empty() const { if (found_table) return true; + + if (!allowed_namespaces.isNamespaceAllowed(namespace_name, /*nested*/ false)) + return false; + const auto tables = listTablesInNamespace(namespace_name, /* limit */1); if (!tables.empty()) found_table = true; @@ -1311,6 +1326,8 @@ CatalogTables RestCatalog::getTables() const auto execute_for_each_namespace = [&](const std::string & current_namespace) { + if (!allowed_namespaces.isNamespaceAllowed(current_namespace, /*nested*/ false)) + return; runner.enqueueAndKeepTrack( [=, &tables, &mutex, this] { @@ -1382,9 +1399,21 @@ void RestCatalog::getNamespacesRecursive( break; if (func) - func(current_namespace); + { + if (allowed_namespaces.isNamespaceAllowed(current_namespace, /*nested*/ false)) + func(current_namespace); + else + { + LOG_DEBUG(log, "Tables in namespace {} are filtered", current_namespace); + } + } - getNamespacesRecursive(current_namespace, result, stop_condition, func); + if (allowed_namespaces.isNamespaceAllowed(current_namespace, /*nested*/ true)) + getNamespacesRecursive(current_namespace, result, stop_condition, func); + else + { + LOG_DEBUG(log, "Nested namespaces in namespace {} are filtered", current_namespace); + } } } @@ -1573,6 +1602,10 @@ RestCatalog::Namespaces RestCatalog::parseNamespaces(DB::ReadBuffer & buf, const DB::Names RestCatalog::listTablesInNamespace(const std::string & base_namespace, size_t limit) const { + if (!allowed_namespaces.isNamespaceAllowed(base_namespace, /*nested*/ false)) + throw DB::Exception(DB::ErrorCodes::CATALOG_NAMESPACE_DISABLED, + "Namespace {} is filtered by `namespaces` database parameter", base_namespace); + const auto state_snapshot = state.get(); auto encoded_namespace = encodeNamespaceForURI(base_namespace); @@ -1687,17 +1720,18 @@ DB::Names RestCatalog::parseTables(DB::ReadBuffer & buf, const std::string & bas bool RestCatalog::existsTable(const std::string & namespace_name, const std::string & table_name) const { TableMetadata table_metadata; - return tryGetTableMetadata(namespace_name, table_name, table_metadata); + return tryGetTableMetadata(namespace_name, table_name, getContext(), table_metadata); } bool RestCatalog::tryGetTableMetadata( const std::string & namespace_name, const std::string & table_name, + DB::ContextPtr context_, TableMetadata & result) const { try { - return getTableMetadataImpl(namespace_name, table_name, result); + return getTableMetadataImpl(namespace_name, table_name, context_, result); } catch (const DB::HTTPException & ex) { @@ -1717,19 +1751,25 @@ bool RestCatalog::tryGetTableMetadata( void RestCatalog::getTableMetadata( const std::string & namespace_name, const std::string & table_name, + DB::ContextPtr context_, TableMetadata & result) const { - if (!getTableMetadataImpl(namespace_name, table_name, result)) + if (!getTableMetadataImpl(namespace_name, table_name, context_, result)) throw DB::Exception(DB::ErrorCodes::DATALAKE_DATABASE_ERROR, "No response from iceberg catalog"); } bool RestCatalog::getTableMetadataImpl( const std::string & namespace_name, const std::string & table_name, + DB::ContextPtr context_, TableMetadata & result) const { LOG_DEBUG(log, "Checking table {} in namespace {}", table_name, namespace_name); + if (!allowed_namespaces.isNamespaceAllowed(namespace_name, /*nested*/ false)) + throw DB::Exception(DB::ErrorCodes::CATALOG_NAMESPACE_DISABLED, + "Namespace {} is filtered by `namespaces` database parameter", namespace_name); + DB::HTTPHeaderEntries headers; if (result.requiresCredentials()) { @@ -1794,8 +1834,8 @@ bool RestCatalog::getTableMetadataImpl( { const bool allow_geo_parser = getContext()->getSettingsRef()[DB::Setting::allow_experimental_geo_types_in_iceberg].value; - auto schema_processor = DB::Iceberg::IcebergSchemaProcessor(allow_geo_parser); - auto id = DB::IcebergMetadata::parseTableSchema(metadata_object, schema_processor, log); + auto schema_processor = DB::Iceberg::IcebergSchemaProcessor(context_, allow_geo_parser); + auto id = DB::IcebergMetadata::parseTableSchema(metadata_object, schema_processor, context_, log); auto schema = schema_processor.getClickHouseTableSchemaById(id); result.setSchema(*schema); } @@ -1957,6 +1997,10 @@ void RestCatalog::createNamespaceIfNotExists(const String & namespace_name, cons void RestCatalog::createTable(const String & namespace_name, const String & table_name, const String & /*new_metadata_path*/, Poco::JSON::Object::Ptr metadata_content) const { + if (!allowed_namespaces.isNamespaceAllowed(namespace_name, /*nested*/ false)) + throw DB::Exception(DB::ErrorCodes::CATALOG_NAMESPACE_DISABLED, + "Failed to create table {}, namespace {} is filtered by `namespaces` database parameter", table_name, namespace_name); + const auto state_snapshot = state.get(); const std::string endpoint = (base_url / state_snapshot->config.prefix / NAMESPACES_ENDPOINT / encodeNamespaceForURI(namespace_name) / "tables").generic_string(); @@ -2146,6 +2190,11 @@ bool RestCatalog::updateSchema( void RestCatalog::dropTable(const String & namespace_name, const String & table_name, bool /*delete_data*/) const { + if (!allowed_namespaces.isNamespaceAllowed(namespace_name, /*nested*/ false)) + throw DB::Exception(DB::ErrorCodes::CATALOG_NAMESPACE_DISABLED, + "Failed to drop table {}, namespace {} is filtered by `namespaces` database parameter", + table_name, namespace_name); + const auto state_snapshot = state.get(); const std::string endpoint = fmt::format("{}/namespaces/{}/tables/{}?purgeRequested=False", base_url, namespace_name, table_name); @@ -2292,6 +2341,70 @@ ICatalog::CredentialsRefreshCallback RestCatalog::getCredentialsConfigurationCal }; } +/// "alpha,alpha.a1,bravo,bravo.*,charlie,delta.d1,echo.*" +/// allows tables from +/// - "alpha" namespace +/// - "alpha.a1" namespace +/// - "bravo" namespace +/// - any nested namespaces of "bravo" +/// - "charlie" namespace, but not from nested of "charlie" +/// - "delta.d1" namespace, but not from "delta" +/// - any nested namespaces of "echo", but not "echo" itself +/// "bravo.*.b2" makes no sense for now, asterisk allows all nested +RestCatalog::AllowedNamespaces::AllowedNamespaces(const std::string & namespaces_) +{ + std::vector list_of_namespaces; + boost::split(list_of_namespaces, namespaces_, boost::is_any_of(", "), boost::token_compress_on); + for (const auto & ns : list_of_namespaces) + { + std::vector list_of_nested_namespaces; + boost::split(list_of_nested_namespaces, ns, boost::is_any_of(".")); + + size_t len = list_of_nested_namespaces.size(); + if (!len) + continue; + + AllowedNamespaces * current = &(nested_namespaces[list_of_nested_namespaces[0]]); + for (size_t i = 1; i <= len; ++i) + { + if (i == len) + current->allow_tables = true; + else + { + current = &(current->nested_namespaces[list_of_nested_namespaces[i]]); + if (list_of_nested_namespaces[i] == "*") + { + current->allow_tables = true; + break; + } + } + } + } +} + +bool RestCatalog::AllowedNamespaces::isNamespaceAllowed(const std::string & namespace_, bool nested) const +{ + // Trivial case, check here to avoid split namespace on nested + if (nested_namespaces.contains("*")) + return true; + + std::vector list_of_nested_namespaces; + boost::split(list_of_nested_namespaces, namespace_, boost::is_any_of(".")); + + const AllowedNamespaces * current = this; + for (const auto & nns : list_of_nested_namespaces) + { + if (current->nested_namespaces.contains("*")) + return true; + auto it = current->nested_namespaces.find(nns); + if (it == current->nested_namespaces.end()) + return false; + current = &(it->second); + } + + return nested ? !current->nested_namespaces.empty() : current->allow_tables; +} + } #endif diff --git a/src/Databases/DataLake/RestCatalog.h b/src/Databases/DataLake/RestCatalog.h index 4444c3eed5f6..077f186395ca 100644 --- a/src/Databases/DataLake/RestCatalog.h +++ b/src/Databases/DataLake/RestCatalog.h @@ -44,6 +44,7 @@ class RestCatalog : public ICatalog, public DB::WithContext const std::string & auth_header_, const std::string & oauth_server_uri_, bool oauth_server_use_request_body_, + const std::string & namespaces_, DB::ContextPtr context_); ~RestCatalog() override = default; @@ -59,11 +60,13 @@ class RestCatalog : public ICatalog, public DB::WithContext void getTableMetadata( const std::string & namespace_name, const std::string & table_name, + DB::ContextPtr context_, TableMetadata & result) const override; bool tryGetTableMetadata( const std::string & namespace_name, const std::string & table_name, + DB::ContextPtr context_, TableMetadata & result) const override; std::optional getStorageType() const override; @@ -140,6 +143,7 @@ class RestCatalog : public ICatalog, public DB::WithContext const std::string & auth_scope_, const std::string & oauth_server_uri_, bool oauth_server_use_request_body_, + const std::string & namespaces_, DB::ContextPtr context_); void createNamespaceIfNotExists(const String & namespace_name, const String & location) const override; @@ -156,6 +160,26 @@ class RestCatalog : public ICatalog, public DB::WithContext bool oauth_server_use_request_body; mutable MultiVersion access_token; +public: + class AllowedNamespaces + { + public: + AllowedNamespaces() {} + explicit AllowedNamespaces(const std::string & namespaces_); + + /// Check if nested namespaces (nested=true) or tables (nested=false) are allowed in namespace + bool isNamespaceAllowed(const std::string & namespace_, bool nested) const; + + private: + /// List of allowed nested namespaces + std::unordered_map nested_namespaces; + /// Tables from current level are allowed + bool allow_tables = false; + }; + +protected: + AllowedNamespaces allowed_namespaces; + Poco::Net::HTTPBasicCredentials credentials{}; /// `catalog_state` is the snapshot the caller derived the endpoint from, so that one @@ -203,6 +227,7 @@ class RestCatalog : public ICatalog, public DB::WithContext bool getTableMetadataImpl( const std::string & namespace_name, const std::string & table_name, + DB::ContextPtr context_, TableMetadata & result) const; /// Load catalog config (special http handler) utilizing information from catalog_state and auth_headers. @@ -275,6 +300,7 @@ class OneLakeCatalog : public RestCatalog const std::string & auth_scope_, const std::string & oauth_server_uri_, bool oauth_server_use_request_body_, + const std::string & namespaces_, DB::ContextPtr context_); DB::DatabaseDataLakeCatalogType getCatalogType() const override @@ -324,6 +350,7 @@ class BigLakeCatalog : public RestCatalog const std::string & google_adc_client_secret_, const std::string & google_adc_refresh_token_, const std::string & google_adc_quota_project_id_, + const std::string & namespaces_, DB::ContextPtr context_, bool allow_server_credentials_in_user_queries_); @@ -395,6 +422,7 @@ class HorizonCatalog : public RestCatalog const std::string & auth_header_, const std::string & oauth_server_uri_, bool oauth_server_use_request_body_, + const std::string & namespaces_, DB::ContextPtr context_); DB::DatabaseDataLakeCatalogType getCatalogType() const override diff --git a/src/Databases/DataLake/S3TablesCatalog.cpp b/src/Databases/DataLake/S3TablesCatalog.cpp index bbf1365250b5..7b1d5ff57a3a 100644 --- a/src/Databases/DataLake/S3TablesCatalog.cpp +++ b/src/Databases/DataLake/S3TablesCatalog.cpp @@ -64,7 +64,7 @@ S3TablesCatalog::S3TablesCatalog( const CatalogSettings & catalog_settings_, DB::ContextPtr context_, bool allow_server_credentials_in_user_queries_) - : RestCatalog(warehouse_, base_url_, "", "", false, context_) + : RestCatalog(warehouse_, base_url_, "", "", false, catalog_settings_.namespaces, context_) , region(region_) , storage_endpoint(catalog_settings_.storage_endpoint) , signing_service("s3tables") @@ -170,9 +170,10 @@ CatalogTables S3TablesCatalog::getTables() const bool S3TablesCatalog::tryGetTableMetadata( const std::string & namespace_name, const std::string & table_name, + DB::ContextPtr context_, TableMetadata & result) const { - if (!RestCatalog::tryGetTableMetadata(namespace_name, table_name, result)) + if (!RestCatalog::tryGetTableMetadata(namespace_name, table_name, context_, result)) return false; /// For S3 Tables the catalog and the underlying data live in AWS S3 under the same diff --git a/src/Databases/DataLake/S3TablesCatalog.h b/src/Databases/DataLake/S3TablesCatalog.h index 65ca66500ed1..2bb706df9796 100644 --- a/src/Databases/DataLake/S3TablesCatalog.h +++ b/src/Databases/DataLake/S3TablesCatalog.h @@ -43,6 +43,7 @@ class S3TablesCatalog final : public RestCatalog bool tryGetTableMetadata( const std::string & namespace_name, const std::string & table_name, + DB::ContextPtr context_, TableMetadata & result) const override; void dropTable(const String & namespace_name, const String & table_name, bool delete_data) const override; diff --git a/src/Databases/DataLake/UnityCatalog.cpp b/src/Databases/DataLake/UnityCatalog.cpp index da9ece058891..9d1859343c19 100644 --- a/src/Databases/DataLake/UnityCatalog.cpp +++ b/src/Databases/DataLake/UnityCatalog.cpp @@ -16,6 +16,9 @@ #include #include +#include +#include + namespace ProfileEvents { extern const Event DataLakeUnityCatalogGetTables; @@ -35,6 +38,7 @@ namespace DB::ErrorCodes extern const int DATALAKE_DATABASE_ERROR; extern const int LOGICAL_ERROR; extern const int BAD_ARGUMENTS; + extern const int CATALOG_NAMESPACE_DISABLED; } namespace @@ -141,9 +145,10 @@ CatalogTables UnityCatalog::listTablesInNamespaceDirect(const std::string & name void UnityCatalog::getTableMetadata( const std::string & namespace_name, const std::string & table_name, + DB::ContextPtr context_, TableMetadata & result) const { - if (!tryGetTableMetadata(namespace_name, table_name, result)) + if (!tryGetTableMetadata(namespace_name, table_name, context_, result)) throw DB::Exception(DB::ErrorCodes::DATALAKE_DATABASE_ERROR, "No response from unity catalog"); } @@ -215,8 +220,12 @@ void UnityCatalog::getCredentials(const String & table_id, TableMetadata & metad bool UnityCatalog::tryGetTableMetadata( const std::string & schema_name, const std::string & table_name, + DB::ContextPtr /* context_ */, TableMetadata & result) const { + if (!isNamespaceAllowed(schema_name)) + throw DB::Exception(DB::ErrorCodes::CATALOG_NAMESPACE_DISABLED, "Namespace {} is filtered by `namespaces` database parameter", schema_name); + auto full_table_name = warehouse + "." + schema_name + "." + table_name; Poco::Dynamic::Var json; std::string json_str; @@ -343,6 +352,9 @@ bool UnityCatalog::tryGetTableMetadata( bool UnityCatalog::existsTable(const std::string & schema_name, const std::string & table_name) const { + if (!isNamespaceAllowed(schema_name)) + throw DB::Exception(DB::ErrorCodes::CATALOG_NAMESPACE_DISABLED, "Namespace {} is filtered by `namespaces` database parameter", schema_name); + String json_str; Poco::Dynamic::Var json; try @@ -467,7 +479,7 @@ DataLake::ICatalog::Namespaces UnityCatalog::getSchemas(const std::string & base chassert(schema_info->get("catalog_name").extract() == warehouse); UnityCatalogFullSchemaName schema_name = parseFullSchemaName(schema_info->get("full_name").extract()); - if (schema_name.schema_name.starts_with(base_prefix)) + if (isNamespaceAllowed(schema_name.schema_name) && schema_name.schema_name.starts_with(base_prefix)) schemas.push_back(schema_name.schema_name); if (limit && schemas.size() > limit) @@ -509,6 +521,7 @@ UnityCatalog::UnityCatalog( const std::string & catalog_, const std::string & base_url_, const std::string & catalog_credential_, + const std::string & namespaces_, DB::ContextPtr context_) : ICatalog(catalog_) , DB::WithContext(context_) @@ -516,6 +529,12 @@ UnityCatalog::UnityCatalog( , log(getLogger("UnityCatalog(" + catalog_ + ")")) , bearer_token(catalog_credential_) { + boost::split(allowed_namespaces, namespaces_, boost::is_any_of(", "), boost::token_compress_on); +} + +bool UnityCatalog::isNamespaceAllowed(const std::string & namespace_) const +{ + return allowed_namespaces.contains("*") || allowed_namespaces.contains(namespace_); } /// getCredentialsConfigurationCallback method is supported only for S3 storage diff --git a/src/Databases/DataLake/UnityCatalog.h b/src/Databases/DataLake/UnityCatalog.h index 840822b0e0da..7ad9684d7a5b 100644 --- a/src/Databases/DataLake/UnityCatalog.h +++ b/src/Databases/DataLake/UnityCatalog.h @@ -22,6 +22,7 @@ class UnityCatalog final : public ICatalog, private DB::WithContext const std::string & catalog_, const std::string & base_url_, const std::string & catalog_credential_, + const std::string & namespaces_, DB::ContextPtr context_); ~UnityCatalog() override = default; @@ -37,11 +38,13 @@ class UnityCatalog final : public ICatalog, private DB::WithContext void getTableMetadata( const std::string & namespace_name, const std::string & table_name, + DB::ContextPtr context_, TableMetadata & result) const override; bool tryGetTableMetadata( const std::string & schema_name, const std::string & table_name, + DB::ContextPtr context_, TableMetadata & result) const override; std::optional getStorageType() const override { return std::nullopt; } @@ -60,6 +63,11 @@ class UnityCatalog final : public ICatalog, private DB::WithContext std::pair getJSONRequest(const std::string & route, const Poco::URI::QueryParameters & params = {}) const; std::pair postJSONRequest(const std::string & route, std::function out_stream_callaback) const; + std::unordered_set allowed_namespaces; + + bool isNamespaceAllowed(const std::string & namespace_) const; + + DataLake::ICatalog::Namespaces getSchemas(const std::string & base_prefix, size_t limit = 0) const; CatalogTables getTablesForSchema(const std::string & schema, size_t limit = 0) const; diff --git a/src/Databases/DataLake/tests/gtest_rest_catalog.cpp b/src/Databases/DataLake/tests/gtest_rest_catalog.cpp index 1b37532ea7d4..7cec11446624 100644 --- a/src/Databases/DataLake/tests/gtest_rest_catalog.cpp +++ b/src/Databases/DataLake/tests/gtest_rest_catalog.cpp @@ -327,6 +327,7 @@ bool restCatalogEmpty(CatalogShape shape) /* auth_header */"", /* oauth_server_uri */"", /* oauth_server_use_request_body */false, + /* namespaces */"*", context); return catalog.empty(); @@ -346,6 +347,7 @@ bool deltaSharingCatalogEmpty(CatalogShape shape) /* auth_header */"", /* oauth_server_uri */"", /* oauth_server_use_request_body */false, + /* namespaces */"*", context); return catalog.empty(); @@ -391,19 +393,20 @@ TEST(RestCatalog, TryGetTableMetadataDistinguishesMissingTableFromOtherErrors) /* auth_header */"", /* oauth_server_uri */"", /* oauth_server_use_request_body */false, + /* namespaces */"*", context); auto existing = TableMetadata().withLocation(); - EXPECT_TRUE(catalog.tryGetTableMetadata("namespace", "table_a", existing)); + EXPECT_TRUE(catalog.tryGetTableMetadata("namespace", "table_a", context, existing)); EXPECT_EQ(existing.getLocation(), "s3://bucket/table_a"); EXPECT_TRUE(catalog.existsTable("namespace", "table_a")); TableMetadata missing; - EXPECT_FALSE(catalog.tryGetTableMetadata("namespace", "missing_table", missing)); + EXPECT_FALSE(catalog.tryGetTableMetadata("namespace", "missing_table", context, missing)); EXPECT_FALSE(catalog.existsTable("namespace", "missing_table")); TableMetadata unauthorized; - EXPECT_THROW(catalog.tryGetTableMetadata("namespace", "unauthorized_table", unauthorized), DB::HTTPException); + EXPECT_THROW(catalog.tryGetTableMetadata("namespace", "unauthorized_table", context, unauthorized), DB::HTTPException); EXPECT_THROW(catalog.existsTable("namespace", "unauthorized_table"), DB::HTTPException); } @@ -427,12 +430,13 @@ TEST(RestCatalog, TryGetTableMetadataAuthErrorPropagates) /* auth_scope */"", /* oauth_server_uri */"", /* oauth_server_use_request_body */false, + /* namespaces */"*", context); TableMetadata metadata; try { - catalog.tryGetTableMetadata("namespace", "expired_token_table", metadata); + catalog.tryGetTableMetadata("namespace", "expired_token_table", context, metadata); FAIL() << "expected the HTTP 401 from the catalog to propagate"; } catch (const DB::HTTPException & e) @@ -458,6 +462,7 @@ TEST(RestCatalog, ApplySettingsChangesWithoutAuthenticationRejected) /* auth_header */"", /* oauth_server_uri */"", /* oauth_server_use_request_body */false, + /* namespaces */"*", context); DB::SettingsChanges changes; @@ -479,6 +484,7 @@ TEST(RestCatalog, ApplySettingsChangesCredentialMode) /* auth_header */"", /* oauth_server_uri */"", /* oauth_server_use_request_body */false, + /* namespaces */"*", context); EXPECT_EQ(catalog.getStateSnapshot()->client_id, "client-1"); @@ -520,6 +526,7 @@ TEST(RestCatalog, ApplySettingsChangesAuthHeaderMode) /* auth_header */"Authorization: Bearer token-1", /* oauth_server_uri */"", /* oauth_server_use_request_body */false, + /* namespaces */"*", context); DB::SettingsChanges changes; @@ -552,6 +559,7 @@ TEST(RestCatalog, OneLakeApplySettingsChangesBearerMode) /* auth_scope */"", /* oauth_server_uri */"", /* oauth_server_use_request_body */false, + /* namespaces */"*", context); const auto snapshot_before = catalog.getStateSnapshot(); @@ -612,6 +620,7 @@ TEST(RestCatalog, OneLakeRejectsMalformedBearerToken) /* auth_scope */ "", /* oauth_server_uri */ "", /* oauth_server_use_request_body */ false, + /* namespaces */"*", context); }, DB::ErrorCodes::BAD_ARGUMENTS); @@ -652,6 +661,7 @@ TEST(RestCatalog, OneLakeRefreshTokenTransparentRenewal) /* auth_scope */"https://storage.azure.com/.default", /* oauth_server_uri */server.getUrl() + "/token", /* oauth_server_use_request_body */true, + /* namespaces */"*", context); const auto requests_after_construction = server.tokenRequests(); @@ -693,6 +703,7 @@ TEST(RestCatalog, OneLakeRefreshTokenExpiredThrowsWithAlterHint) /* auth_scope */"https://storage.azure.com/.default", /* oauth_server_uri */server.getUrl() + "/token", /* oauth_server_use_request_body */true, + /* namespaces */"*", context); /// ADD_FAILURE (rather than FAIL) does not return from the test, so the /// profile event check below is reached on every path; FAIL would make @@ -727,6 +738,7 @@ TEST(RestCatalog, OneLakeApplySettingsChangesRefreshMode) /* auth_scope */"https://storage.azure.com/.default", /* oauth_server_uri */server.getUrl() + "/token", /* oauth_server_use_request_body */true, + /* namespaces */"*", context); DB::SettingsChanges changes; @@ -780,6 +792,7 @@ TEST(RestCatalog, HorizonCatalogAuthenticatesWithBarePAT) /* auth_header */"", /* oauth_server_uri */"", /* oauth_server_use_request_body */true, + /* namespaces */"*", context); EXPECT_EQ(catalog.getCatalogType(), DB::DatabaseDataLakeCatalogType::ICEBERG_HORIZON); @@ -789,7 +802,7 @@ TEST(RestCatalog, HorizonCatalogAuthenticatesWithBarePAT) TableMetadata metadata; metadata.withLocation(); - catalog.getTableMetadata("namespace", "table_a", metadata); + catalog.getTableMetadata("namespace", "table_a", context, metadata); EXPECT_TRUE(metadata.hasLocation()); EXPECT_EQ(metadata.getLocation(), "s3://bucket/table_a"); } @@ -811,6 +824,7 @@ TEST(RestCatalog, HorizonCatalogRequiresCredentialOrAuthHeader) /* auth_header */"", /* oauth_server_uri */"", /* oauth_server_use_request_body */true, + /* namespaces */"*", context); }, DB::ErrorCodes::BAD_ARGUMENTS); @@ -830,6 +844,7 @@ TEST(RestCatalog, HorizonApplySettingsChangesBarePAT) /* auth_header */"", /* oauth_server_uri */"", /* oauth_server_use_request_body */true, + /* namespaces */"*", context); DB::SettingsChanges changes; diff --git a/src/Databases/DataLake/tests/gtest_rest_catalog_allowed_namespaces.cpp b/src/Databases/DataLake/tests/gtest_rest_catalog_allowed_namespaces.cpp new file mode 100644 index 000000000000..7a1981511c3f --- /dev/null +++ b/src/Databases/DataLake/tests/gtest_rest_catalog_allowed_namespaces.cpp @@ -0,0 +1,69 @@ +#include +#include + + +TEST(TestRestCatalogAllowedNamespaces, TestAllAllowed) +{ + DataLake::RestCatalog::AllowedNamespaces namespaces("*"); + EXPECT_TRUE(namespaces.isNamespaceAllowed("foo", /*nested*/ true)); + EXPECT_TRUE(namespaces.isNamespaceAllowed("foo", /*nested*/ false)); + EXPECT_TRUE(namespaces.isNamespaceAllowed("foo.bar", /*nested*/ true)); + EXPECT_TRUE(namespaces.isNamespaceAllowed("foo.bar", /*nested*/ false)); +} + +TEST(TestRestCatalogAllowedNamespaces, TestAllBlocked) +{ + DataLake::RestCatalog::AllowedNamespaces namespaces(""); + EXPECT_FALSE(namespaces.isNamespaceAllowed("foo", /*nested*/ true)); + EXPECT_FALSE(namespaces.isNamespaceAllowed("foo", /*nested*/ false)); + EXPECT_FALSE(namespaces.isNamespaceAllowed("foo.bar", /*nested*/ true)); + EXPECT_FALSE(namespaces.isNamespaceAllowed("foo.bar", /*nested*/ false)); +} + +TEST(TestRestCatalogAllowedNamespaces, TestTableInNamespaceAllowed) +{ + DataLake::RestCatalog::AllowedNamespaces namespaces("foo"); + EXPECT_FALSE(namespaces.isNamespaceAllowed("foo", /*nested*/ true)); + EXPECT_TRUE(namespaces.isNamespaceAllowed("foo", /*nested*/ false)); + EXPECT_FALSE(namespaces.isNamespaceAllowed("foo.bar", /*nested*/ true)); + EXPECT_FALSE(namespaces.isNamespaceAllowed("foo.bar", /*nested*/ false)); + EXPECT_FALSE(namespaces.isNamespaceAllowed("biz", /*nested*/ true)); + EXPECT_FALSE(namespaces.isNamespaceAllowed("biz", /*nested*/ false)); +} + +TEST(TestRestCatalogAllowedNamespaces, TestSpecificNestedNamespaceAllowed) +{ + DataLake::RestCatalog::AllowedNamespaces namespaces("foo.bar"); + EXPECT_TRUE(namespaces.isNamespaceAllowed("foo", /*nested*/ true)); + EXPECT_FALSE(namespaces.isNamespaceAllowed("foo", /*nested*/ false)); + EXPECT_FALSE(namespaces.isNamespaceAllowed("foo.bar", /*nested*/ true)); + EXPECT_TRUE(namespaces.isNamespaceAllowed("foo.bar", /*nested*/ false)); + EXPECT_FALSE(namespaces.isNamespaceAllowed("bar", /*nested*/ true)); + EXPECT_FALSE(namespaces.isNamespaceAllowed("bar", /*nested*/ false)); + EXPECT_FALSE(namespaces.isNamespaceAllowed("biz", /*nested*/ true)); + EXPECT_FALSE(namespaces.isNamespaceAllowed("biz", /*nested*/ false)); + EXPECT_FALSE(namespaces.isNamespaceAllowed("foo.biz", /*nested*/ true)); + EXPECT_FALSE(namespaces.isNamespaceAllowed("foo.biz", /*nested*/ false)); +} + +TEST(TestRestCatalogAllowedNamespaces, TestNestedNamespacesAllowed) +{ + DataLake::RestCatalog::AllowedNamespaces namespaces("foo.*"); + EXPECT_TRUE(namespaces.isNamespaceAllowed("foo", /*nested*/ true)); + EXPECT_FALSE(namespaces.isNamespaceAllowed("foo", /*nested*/ false)); + EXPECT_TRUE(namespaces.isNamespaceAllowed("foo.bar", /*nested*/ true)); + EXPECT_TRUE(namespaces.isNamespaceAllowed("foo.bar", /*nested*/ false)); + EXPECT_FALSE(namespaces.isNamespaceAllowed("biz", /*nested*/ true)); + EXPECT_FALSE(namespaces.isNamespaceAllowed("biz", /*nested*/ false)); +} + +TEST(TestRestCatalogAllowedNamespaces, TestTablesAndNestedNamespacesAllowed) +{ + DataLake::RestCatalog::AllowedNamespaces namespaces("foo,foo.*"); + EXPECT_TRUE(namespaces.isNamespaceAllowed("foo", /*nested*/ true)); + EXPECT_TRUE(namespaces.isNamespaceAllowed("foo", /*nested*/ false)); + EXPECT_TRUE(namespaces.isNamespaceAllowed("foo.bar", /*nested*/ true)); + EXPECT_TRUE(namespaces.isNamespaceAllowed("foo.bar", /*nested*/ false)); + EXPECT_FALSE(namespaces.isNamespaceAllowed("biz", /*nested*/ true)); + EXPECT_FALSE(namespaces.isNamespaceAllowed("biz", /*nested*/ false)); +} diff --git a/src/Databases/DatabaseReplicated.cpp b/src/Databases/DatabaseReplicated.cpp index 19906928b8c8..3def0e07a11c 100644 --- a/src/Databases/DatabaseReplicated.cpp +++ b/src/Databases/DatabaseReplicated.cpp @@ -2927,7 +2927,7 @@ bool DatabaseReplicated::shouldReplicateQuery(const ContextPtr & query_context, if (const auto * alter = query_ptr->as()) { if (alter->isAttachAlter() || alter->isFetchAlter() || alter->isDropPartitionAlter() || alter->isFreezeAlter() - || alter->isUnlockSnapshot()) + || alter->isUnlockSnapshot() || alter->isExportPartOrExportPartitionAlter()) return false; // Allowed ALTER operation on KeeperMap still should be replicated diff --git a/src/Disks/DiskObjectStorage/ObjectStorages/AzureBlobStorage/AzureObjectStorage.cpp b/src/Disks/DiskObjectStorage/ObjectStorages/AzureBlobStorage/AzureObjectStorage.cpp index 21641d98d0b8..a36ed00fbfc6 100644 --- a/src/Disks/DiskObjectStorage/ObjectStorages/AzureBlobStorage/AzureObjectStorage.cpp +++ b/src/Disks/DiskObjectStorage/ObjectStorages/AzureBlobStorage/AzureObjectStorage.cpp @@ -27,6 +27,7 @@ #include #include #include +#include namespace CurrentMetrics @@ -39,6 +40,7 @@ namespace CurrentMetrics namespace ProfileEvents { extern const Event AzureListObjects; + extern const Event AzureListObjectsMicroseconds; extern const Event DiskAzureListObjects; extern const Event AzureDeleteObjects; extern const Event DiskAzureDeleteObjects; @@ -90,6 +92,7 @@ class AzureIteratorAsync final : public IObjectStorageIteratorAsync ProfileEvents::increment(ProfileEvents::AzureListObjects); if (client->IsClientForDisk()) ProfileEvents::increment(ProfileEvents::DiskAzureListObjects); + ProfileEventTimeIncrement watch(ProfileEvents::AzureListObjectsMicroseconds); chassert(batch.empty()); auto blob_list_response = client->ListBlobs(options); @@ -201,7 +204,11 @@ void AzureObjectStorage::listObjects(const std::string & path, RelativePathsWith /// MoveToNextPage refetches pages 2..N directly and would leave the raw Azure prefix on their blob names. while (true) { - auto blob_list_response = client_ptr->ListBlobs(options); + AzureBlobStorage::ListBlobsPagedResponse blob_list_response; + { + ProfileEventTimeIncrement watch(ProfileEvents::AzureListObjectsMicroseconds); + blob_list_response = client_ptr->ListBlobs(options); + } ProfileEvents::increment(ProfileEvents::AzureListObjects); if (client_ptr->IsClientForDisk()) diff --git a/src/Disks/DiskObjectStorage/ObjectStorages/AzureBlobStorage/AzureObjectStorage.h b/src/Disks/DiskObjectStorage/ObjectStorages/AzureBlobStorage/AzureObjectStorage.h index 88adfe903284..88420b30472f 100644 --- a/src/Disks/DiskObjectStorage/ObjectStorages/AzureBlobStorage/AzureObjectStorage.h +++ b/src/Disks/DiskObjectStorage/ObjectStorages/AzureBlobStorage/AzureObjectStorage.h @@ -36,6 +36,8 @@ class AzureObjectStorage : public IObjectStorage const String & description_, const String & common_key_prefix_); + bool supportsListObjectsCache() override { return true; } + void listObjects(const std::string & path, RelativePathsWithMetadata & children, size_t max_keys) const override; /// Sanitizer build may crash with max_keys=1; this looks like a false positive. diff --git a/src/Disks/DiskObjectStorage/ObjectStorages/IObjectStorage.cpp b/src/Disks/DiskObjectStorage/ObjectStorages/IObjectStorage.cpp index 4bfc33395682..2d4088d813b0 100644 --- a/src/Disks/DiskObjectStorage/ObjectStorages/IObjectStorage.cpp +++ b/src/Disks/DiskObjectStorage/ObjectStorages/IObjectStorage.cpp @@ -9,6 +9,11 @@ #include #include #include +#include + +#include +#include +#include namespace DB @@ -113,4 +118,55 @@ void IObjectStorage::prepareRead( pipeline.setSource(std::move(storage), objects, read_settings, read_hint); } +RelativePathWithMetadata::RelativePathWithMetadata(const DataFileInfo & info, std::optional metadata_) + : metadata(std::move(metadata_)) +{ + relative_path = info.file_path; + file_meta_info = info.file_meta_info; +} + +RelativePathWithMetadata::CommandInTaskResponse::CommandInTaskResponse(const std::string & task) +{ + Poco::JSON::Parser parser; + try + { + auto json = parser.parse(task).extract(); + if (!json) + return; + + is_valid = true; + + if (json->has("file_path")) + file_path = json->getValue("file_path"); + if (json->has("retry_after_us")) + retry_after_us = json->getValue("retry_after_us"); + if (json->has("meta_info")) + file_meta_info = std::make_shared(json->getObject("meta_info")); + } + catch (const Poco::JSON::JSONException &) + { /// Not a JSON + return; + } + catch (const Poco::SyntaxException &) + { /// Not a JSON + return; + } +} + +std::string RelativePathWithMetadata::CommandInTaskResponse::toString() const +{ + Poco::JSON::Object json; + if (file_path.has_value()) + json.set("file_path", file_path.value()); + if (retry_after_us.has_value()) + json.set("retry_after_us", retry_after_us.value()); + if (file_meta_info.has_value()) + json.set("meta_info", file_meta_info.value()->toJson()); + + std::ostringstream oss; + oss.exceptions(std::ios::failbit); + Poco::JSON::Stringifier::stringify(json, oss); + return oss.str(); +} + } diff --git a/src/Disks/DiskObjectStorage/ObjectStorages/IObjectStorage.h b/src/Disks/DiskObjectStorage/ObjectStorages/IObjectStorage.h index 8c0a5c2858fb..b156519c30b6 100644 --- a/src/Disks/DiskObjectStorage/ObjectStorages/IObjectStorage.h +++ b/src/Disks/DiskObjectStorage/ObjectStorages/IObjectStorage.h @@ -157,10 +157,51 @@ struct ObjectMetadata bool isEtagUsableAsCacheKey() const { return !etag.empty() && etag_is_strong; } }; + +struct DataFileInfo; +class DataFileMetaInfo; +using DataFileMetaInfoPtr = std::shared_ptr; + struct DataLakeObjectMetadata; struct RelativePathWithMetadata { + class CommandInTaskResponse + { + public: + CommandInTaskResponse() = default; + explicit CommandInTaskResponse(const std::string & task); + + bool isValid() const { return is_valid; } + void setFilePath(const std::string & file_path_ ) + { + file_path = file_path_; + is_valid = true; + } + void setRetryAfterUs(Poco::Timestamp::TimeDiff time_us) + { + retry_after_us = time_us; + is_valid = true; + } + void setFileMetaInfo(DataFileMetaInfoPtr file_meta_info_ ) + { + file_meta_info = file_meta_info_; + is_valid = true; + } + + std::string toString() const; + + std::optional getFilePath() const { return file_path; } + std::optional getRetryAfterUs() const { return retry_after_us; } + std::optional getFileMetaInfo() const { return file_meta_info; } + + private: + bool is_valid = false; + std::optional file_path; + std::optional retry_after_us; + std::optional file_meta_info; + }; + String relative_path; std::optional read_source_index; std::optional path_for_glob_matching; @@ -168,13 +209,26 @@ struct RelativePathWithMetadata bool derive_file_name_from_url_path = false; /// Object metadata: size, modification time, etc. std::optional metadata; + /// Information about columns + std::optional file_meta_info; + /// Retry request after short pause + CommandInTaskResponse command; RelativePathWithMetadata() = default; - explicit RelativePathWithMetadata(String relative_path_, std::optional metadata_ = std::nullopt) - : relative_path(std::move(relative_path_)) + explicit RelativePathWithMetadata(String command_or_path, std::optional metadata_ = std::nullopt) + : relative_path(std::move(command_or_path)) , metadata(std::move(metadata_)) - {} + , command(relative_path) + { + if (command.isValid()) + { + relative_path = command.getFilePath().value_or(""); + file_meta_info = command.getFileMetaInfo(); + } + } + + explicit RelativePathWithMetadata(const DataFileInfo & info, std::optional metadata_ = std::nullopt); RelativePathWithMetadata(String relative_path_, std::optional read_source_index_, std::optional metadata_ = std::nullopt) : relative_path(std::move(relative_path_)) @@ -201,6 +255,12 @@ struct RelativePathWithMetadata } std::string getPath() const { return relative_path; } std::string getPathForGlobMatching() const { return path_for_glob_matching.value_or(relative_path); } + + void setFileMetaInfo(std::optional file_meta_info_ ) { file_meta_info = file_meta_info_; } + std::optional getFileMetaInfo() const { return file_meta_info; } + + const CommandInTaskResponse & getCommand() const { return command; } + std::string getFileNameWithoutExtension() const { return std::filesystem::path(relative_path).stem(); } }; struct ObjectKeyWithMetadata @@ -462,6 +522,8 @@ class IObjectStorage /// Returns the inner (unwrapped) object storage for decorator types such as `CachedObjectStorage`. /// Returns nullptr for non-decorator types, meaning this storage is already the base. virtual ObjectStoragePtr getUnderlying() { return nullptr; } + + virtual bool supportsListObjectsCache() { return false; } }; using ObjectStoragePtr = std::shared_ptr; diff --git a/src/Disks/DiskObjectStorage/ObjectStorages/S3/S3ObjectStorage.cpp b/src/Disks/DiskObjectStorage/ObjectStorages/S3/S3ObjectStorage.cpp index 90f374c23aa1..5fdd13cef073 100644 --- a/src/Disks/DiskObjectStorage/ObjectStorages/S3/S3ObjectStorage.cpp +++ b/src/Disks/DiskObjectStorage/ObjectStorages/S3/S3ObjectStorage.cpp @@ -35,6 +35,7 @@ #include #include #include +#include #include #include @@ -42,6 +43,7 @@ namespace ProfileEvents { extern const Event S3ListObjects; + extern const Event S3ListObjectsMicroseconds; extern const Event DiskS3DeleteObjects; extern const Event DiskS3ListObjects; } @@ -172,7 +174,12 @@ class S3IteratorAsync final : public IObjectStorageIteratorAsync ProfileEvents::increment(ProfileEvents::S3ListObjects); ProfileEvents::increment(ProfileEvents::DiskS3ListObjects); - auto outcome = client->ListObjectsV2(*request); + Aws::S3::Model::ListObjectsV2Outcome outcome; + + { + ProfileEventTimeIncrement watch(ProfileEvents::S3ListObjectsMicroseconds); + outcome = client->ListObjectsV2(*request); + } /// Outcome failure will be handled on the caller side. if (outcome.IsSuccess()) @@ -373,7 +380,11 @@ void S3ObjectStorage::listObjects(const std::string & path, RelativePathsWithMet ProfileEvents::increment(ProfileEvents::S3ListObjects); ProfileEvents::increment(ProfileEvents::DiskS3ListObjects); - outcome = client.get()->ListObjectsV2(request); + { + ProfileEventTimeIncrement watch(ProfileEvents::S3ListObjectsMicroseconds); + outcome = client.get()->ListObjectsV2(request); + } + throwIfError(outcome, "while listing objects in bucket '{}' with prefix '{}' on disk '{}'", uri.bucket, path, disk_name); auto result = outcome.GetResult(); diff --git a/src/Disks/DiskObjectStorage/ObjectStorages/S3/S3ObjectStorage.h b/src/Disks/DiskObjectStorage/ObjectStorages/S3/S3ObjectStorage.h index ab26f5e998d0..52ba5691bf91 100644 --- a/src/Disks/DiskObjectStorage/ObjectStorages/S3/S3ObjectStorage.h +++ b/src/Disks/DiskObjectStorage/ObjectStorages/S3/S3ObjectStorage.h @@ -72,6 +72,8 @@ class S3ObjectStorage : public IObjectStorage ObjectStorageType getType() const override { return ObjectStorageType::S3; } + bool supportsListObjectsCache() override { return true; } + bool exists(const StoredObject & object) const override; std::unique_ptr readObject( /// NOLINT diff --git a/src/Disks/DiskType.cpp b/src/Disks/DiskType.cpp index d60ef273e263..186e169ba483 100644 --- a/src/Disks/DiskType.cpp +++ b/src/Disks/DiskType.cpp @@ -10,7 +10,7 @@ namespace ErrorCodes extern const int LOGICAL_ERROR; } -MetadataStorageType metadataTypeFromString(const String & type) +MetadataStorageType metadataTypeFromString(const std::string & type) { auto check_type = Poco::toLower(type); if (check_type == "local") @@ -60,25 +60,7 @@ String DataSourceDescription::name() const case DataSourceType::RAM: return "memory"; case DataSourceType::ObjectStorage: - { - switch (object_storage_type) - { - case ObjectStorageType::S3: - return "s3"; - case ObjectStorageType::HDFS: - return "hdfs"; - case ObjectStorageType::Azure: - return "azure_blob_storage"; - case ObjectStorageType::Local: - return "local_blob_storage"; - case ObjectStorageType::Web: - return "web"; - case ObjectStorageType::None: - return "none"; - case ObjectStorageType::Max: - throw Exception(ErrorCodes::LOGICAL_ERROR, "Unexpected object storage type: Max"); - } - } + return DB::toString(object_storage_type); } } @@ -88,4 +70,45 @@ String DataSourceDescription::toString() const name(), description, is_encrypted, is_cached, zookeeper_name); } +ObjectStorageType objectStorageTypeFromString(const std::string & type) +{ + auto check_type = Poco::toLower(type); + if (check_type == "s3") + return ObjectStorageType::S3; + if (check_type == "hdfs") + return ObjectStorageType::HDFS; + if (check_type == "azure_blob_storage" || check_type == "azure") + return ObjectStorageType::Azure; + if (check_type == "local_blob_storage" || check_type == "local") + return ObjectStorageType::Local; + if (check_type == "web") + return ObjectStorageType::Web; + if (check_type == "none") + return ObjectStorageType::None; + + throw Exception(ErrorCodes::UNKNOWN_ELEMENT_IN_CONFIG, + "Unknown object storage type: {}", type); +} + +std::string toString(ObjectStorageType type) +{ + switch (type) + { + case ObjectStorageType::S3: + return "s3"; + case ObjectStorageType::HDFS: + return "hdfs"; + case ObjectStorageType::Azure: + return "azure_blob_storage"; + case ObjectStorageType::Local: + return "local_blob_storage"; + case ObjectStorageType::Web: + return "web"; + case ObjectStorageType::None: + return "none"; + case ObjectStorageType::Max: + throw Exception(ErrorCodes::LOGICAL_ERROR, "Unexpected object storage type: Max"); + } +} + } diff --git a/src/Disks/DiskType.h b/src/Disks/DiskType.h index 778435581500..f1b9aebef261 100644 --- a/src/Disks/DiskType.h +++ b/src/Disks/DiskType.h @@ -37,7 +37,10 @@ enum class MetadataStorageType : uint8_t Memory, }; -MetadataStorageType metadataTypeFromString(const String & type); +MetadataStorageType metadataTypeFromString(const std::string & type); + +ObjectStorageType objectStorageTypeFromString(const std::string & type); +std::string toString(ObjectStorageType type); struct DataSourceDescription { diff --git a/src/Functions/FunctionJoinGet.cpp b/src/Functions/FunctionJoinGet.cpp index e3d7b87f97d8..6e38c15d524a 100644 --- a/src/Functions/FunctionJoinGet.cpp +++ b/src/Functions/FunctionJoinGet.cpp @@ -82,6 +82,8 @@ class FunctionJoinGet final : public IFunctionBase String getName() const override { return function_name; } + bool isDeterministic() const override { return false; } + bool isSuitableForShortCircuitArgumentsExecution(const DataTypesWithConstInfo & /*arguments*/) const override { return true; } const DataTypes & getArgumentTypes() const override { return argument_types; } diff --git a/src/Functions/generateSnowflakeID.cpp b/src/Functions/generateSnowflakeID.cpp index 7f39c5991dd0..14a6c8d76d81 100644 --- a/src/Functions/generateSnowflakeID.cpp +++ b/src/Functions/generateSnowflakeID.cpp @@ -155,6 +155,11 @@ uint64_t generateSnowflakeID() return fromSnowflakeId(snowflake_id); } +std::string generateSnowflakeIDString() +{ + return std::to_string(generateSnowflakeID()); +} + class FunctionGenerateSnowflakeID final : public IFunction { public: diff --git a/src/Functions/generateSnowflakeID.h b/src/Functions/generateSnowflakeID.h index 38fa684a9b4b..4fc173dcf1be 100644 --- a/src/Functions/generateSnowflakeID.h +++ b/src/Functions/generateSnowflakeID.h @@ -7,4 +7,6 @@ namespace DB uint64_t generateSnowflakeID(); +std::string generateSnowflakeIDString(); + } diff --git a/src/IO/ReadBufferFromS3.cpp b/src/IO/ReadBufferFromS3.cpp index 2a7137cb6cfd..d904e749d224 100644 --- a/src/IO/ReadBufferFromS3.cpp +++ b/src/IO/ReadBufferFromS3.cpp @@ -582,6 +582,12 @@ Aws::S3::Model::GetObjectResult ReadBufferFromS3::sendRequest(size_t attempt, si log, "Read S3 object. Bucket: {}, Key: {}, Version: {}, Offset: {}", bucket, key, version_id.empty() ? "Latest" : version_id, range_begin); } + else + { + LOG_TEST( + log, "Read S3 object. Bucket: {}, Key: {}, Version: {}", + bucket, key, version_id.empty() ? "Latest" : version_id); + } ProfileEvents::increment(ProfileEvents::S3GetObject); if (client_ptr->isClientForDisk()) diff --git a/src/IO/S3/Client.cpp b/src/IO/S3/Client.cpp index 6bf9c7ddd0b6..7f16eb8fb0db 100644 --- a/src/IO/S3/Client.cpp +++ b/src/IO/S3/Client.cpp @@ -475,7 +475,7 @@ Model::HeadObjectOutcome Client::headObjectInternal(HeadObjectRequest & request) auto bucket_uri = getURIForBucket(bucket); if (!bucket_uri) { - if (auto maybe_error = updateURIForBucketForHead(bucket); maybe_error.has_value()) + if (auto maybe_error = updateURIForBucketForHead(bucket, request.GetKey()); maybe_error.has_value()) return *maybe_error; if (auto region = getRegionForBucket(bucket); !region.empty()) @@ -692,7 +692,6 @@ Client::doRequest(RequestType & request, RequestFn request_fn) const if (auto uri = getURIForBucket(bucket); uri.has_value()) request.overrideURI(std::move(*uri)); - bool found_new_endpoint = false; // if we found correct endpoint after 301 responses, update the cache for future requests SCOPE_EXIT( @@ -1071,12 +1070,15 @@ std::optional Client::getURIFromError(const Aws::S3::S3Error & error) c } // Do a list request because head requests don't have body in response -std::optional Client::updateURIForBucketForHead(const std::string & bucket) const +// S3 Tables don't support ListObjects, so made dirty workaroung - changed on GetObject +std::optional Client::updateURIForBucketForHead(const std::string & bucket, const std::string & key) const { - ListObjectsV2Request req; + GetObjectRequest req; req.SetBucket(bucket); - req.SetMaxKeys(1); - auto result = ListObjectsV2(req); + req.SetKey(key); + req.SetRange("bytes=0-1"); + auto result = GetObject(req); + if (result.IsSuccess()) return std::nullopt; return result.GetError(); diff --git a/src/IO/S3/Client.h b/src/IO/S3/Client.h index ae5312812549..4f679588689a 100644 --- a/src/IO/S3/Client.h +++ b/src/IO/S3/Client.h @@ -299,7 +299,7 @@ class Client : private Aws::S3::S3Client void updateURIForBucket(const std::string & bucket, S3::URI new_uri) const; std::optional getURIFromError(const Aws::S3::S3Error & error) const; - std::optional updateURIForBucketForHead(const std::string & bucket) const; + std::optional updateURIForBucketForHead(const std::string & bucket, const std::string & key) const; Model::HeadObjectOutcome headObjectInternal(HeadObjectRequest & request) const; diff --git a/src/IO/S3/URI.cpp b/src/IO/S3/URI.cpp index b671cb447531..a07fc73456e8 100644 --- a/src/IO/S3/URI.cpp +++ b/src/IO/S3/URI.cpp @@ -215,10 +215,72 @@ bool URI::tryInitVirtualHostedStyle(bool is_using_aws_private_link_interface, bo return true; } +bool URI::isAWSRegion(std::string_view region) +{ + /// List from https://docs.aws.amazon.com/general/latest/gr/s3.html + static const std::unordered_set regions = { + "us-east-2", + "us-east-1", + "us-west-1", + "us-west-2", + "af-south-1", + "ap-east-1", + "ap-south-2", + "ap-southeast-3", + "ap-southeast-5", + "ap-southeast-4", + "ap-south-1", + "ap-northeast-3", + "ap-northeast-2", + "ap-southeast-1", + "ap-southeast-2", + "ap-east-2", + "ap-southeast-7", + "ap-northeast-1", + "ca-central-1", + "ca-west-1", + "eu-central-1", + "eu-west-1", + "eu-west-2", + "eu-south-1", + "eu-west-3", + "eu-south-2", + "eu-north-1", + "eu-central-2", + "il-central-1", + "mx-central-1", + "me-south-1", + "me-central-1", + "sa-east-1", + "us-gov-east-1", + "us-gov-west-1" + }; + + /// 's3-us-west-2' is a legacy region format for S3 storage, equals to 'us-west-2' + /// See https://docs.aws.amazon.com/AmazonS3/latest/userguide/VirtualHosting.html#VirtualHostingBackwardsCompatibility + if (region.substr(0, 3) == "s3-") + region = region.substr(3); + + return regions.contains(region); +} + void URI::addRegionToURI(const std::string ®ion) { if (auto pos = endpoint.find(".amazonaws.com"); pos != std::string::npos) + { + if (pos > 0) + { /// Check if region is already in endpoint to avoid add it second time + auto prev_pos = endpoint.find_last_of("/.", pos - 1); + if (prev_pos == std::string::npos) + prev_pos = 0; + else + ++prev_pos; + std::string_view endpoint_region = std::string_view(endpoint).substr(prev_pos, pos - prev_pos); + if (isAWSRegion(endpoint_region)) + return; + } endpoint = endpoint.substr(0, pos) + "." + region + endpoint.substr(pos); + } } void URI::validateBucket(const String & bucket, const Poco::URI & uri) diff --git a/src/IO/S3/URI.h b/src/IO/S3/URI.h index a8122aa94f40..49378cf46f69 100644 --- a/src/IO/S3/URI.h +++ b/src/IO/S3/URI.h @@ -47,6 +47,10 @@ struct URI static void validateBucket(const std::string & bucket, const Poco::URI & uri); static void validateKey(const std::string & key, const Poco::URI & uri); + /// Returns true if 'region' string is an AWS S3 region + /// https://docs.aws.amazon.com/general/latest/gr/s3.html + static bool isAWSRegion(std::string_view region); + private: bool tryInitPathStyle(); bool tryInitVirtualHostedStyle(bool is_using_aws_private_link_interface, bool use_strict_pattern); diff --git a/src/IO/S3/getObjectInfo.cpp b/src/IO/S3/getObjectInfo.cpp index 1cdcd9f94e37..deec76d3fdc6 100644 --- a/src/IO/S3/getObjectInfo.cpp +++ b/src/IO/S3/getObjectInfo.cpp @@ -1,6 +1,7 @@ #include #include #include +#include #if USE_AWS_S3 @@ -8,6 +9,7 @@ namespace ProfileEvents { extern const Event S3GetObjectTagging; extern const Event S3HeadObject; + extern const Event S3HeadObjectMicroseconds; extern const Event DiskS3GetObjectTagging; extern const Event DiskS3HeadObject; } @@ -27,6 +29,7 @@ namespace ProfileEvents::increment(ProfileEvents::S3HeadObject); if (client.isClientForDisk()) ProfileEvents::increment(ProfileEvents::DiskS3HeadObject); + ProfileEventTimeIncrement watch(ProfileEvents::S3HeadObjectMicroseconds); S3::HeadObjectRequest req; req.SetBucket(bucket); diff --git a/src/Interpreters/CancellationCode.h b/src/Interpreters/CancellationCode.h index e37a7f13105f..571f42c6c5bd 100644 --- a/src/Interpreters/CancellationCode.h +++ b/src/Interpreters/CancellationCode.h @@ -1,5 +1,7 @@ #pragma once +#include + namespace DB { diff --git a/src/Interpreters/Cluster.cpp b/src/Interpreters/Cluster.cpp index f24818dbca0c..35829910f8c4 100644 --- a/src/Interpreters/Cluster.cpp +++ b/src/Interpreters/Cluster.cpp @@ -386,36 +386,40 @@ void Clusters::updateClusters(const Poco::Util::AbstractConfiguration & new_conf std::lock_guard lock(mutex); - /// If old config is set, remove deleted clusters from impl, otherwise just clear it. + /// If old config is set, remove deleted clusters; otherwise rebuild ownership from scratch + /// while preserving non-automatic entries (e.g. clusters added via setCluster). if (old_config) { for (const auto & key : deleted_keys) { - if (!automatic_clusters.contains(key)) - impl.erase(key); + automatic_clusters.erase(key); + impl.erase(key); } } else { - if (!automatic_clusters.empty()) - std::erase_if(impl, [this](const auto & e) { return automatic_clusters.contains(e.first); }); - else - impl.clear(); + for (const auto & name : automatic_clusters) + impl.erase(name); + automatic_clusters.clear(); } - for (const auto & key : new_config_keys) { if (new_config.has(config_prefix + "." + key + ".discovery")) { - /// Handled in ClusterDiscovery + /// Handled in ClusterDiscovery — must not leave a prior static Cluster in impl, + /// or Context::getCluster / getClusters would prefer the stale static entry. automatic_clusters.insert(key); + impl.erase(key); continue; } if (key.contains('.')) throw Exception(ErrorCodes::SYNTAX_ERROR, "Cluster names with dots are not supported: '{}'", key); + /// Leaving discovery (or never was discovery): drop automatic ownership for this name. + automatic_clusters.erase(key); + /// If old config is set and cluster config wasn't changed, don't update this cluster. if (!old_config || !isSameConfiguration(new_config, *old_config, config_prefix + "." + key)) impl[key] = std::make_shared(new_config, settings, config_prefix, key); @@ -754,9 +758,9 @@ void Cluster::initMisc() } } -std::unique_ptr Cluster::getClusterWithReplicasAsShards(const Settings & settings, size_t max_replicas_from_shard) const +std::unique_ptr Cluster::getClusterWithReplicasAsShards(const Settings & settings, size_t max_replicas_from_shard, size_t max_hosts) const { - return std::unique_ptr{ new Cluster(ReplicasAsShardsTag{}, *this, settings, max_replicas_from_shard)}; + return std::unique_ptr{ new Cluster(ReplicasAsShardsTag{}, *this, settings, max_replicas_from_shard, max_hosts)}; } std::unique_ptr Cluster::getClusterWithSingleShard(size_t index) const @@ -805,7 +809,7 @@ void shuffleReplicas(std::vector & replicas, const Settings & } -Cluster::Cluster(Cluster::ReplicasAsShardsTag, const Cluster & from, const Settings & settings, size_t max_replicas_from_shard) +Cluster::Cluster(Cluster::ReplicasAsShardsTag, const Cluster & from, const Settings & settings, size_t max_replicas_from_shard, size_t max_hosts) { if (from.addresses_with_failover.empty()) throw Exception(ErrorCodes::LOGICAL_ERROR, "Cluster is empty"); @@ -827,6 +831,7 @@ Cluster::Cluster(Cluster::ReplicasAsShardsTag, const Cluster & from, const Setti if (address.is_local) info.local_addresses.push_back(address); + addresses_with_failover.emplace_back(Addresses({address})); auto pool = ConnectionPoolFactory::instance().get( static_cast(settings[Setting::distributed_connections_pool_size]), @@ -850,9 +855,6 @@ Cluster::Cluster(Cluster::ReplicasAsShardsTag, const Cluster & from, const Setti info.per_replica_pools = {std::move(pool)}; info.default_database = address.default_database; - addresses_with_failover.emplace_back(Addresses{address}); - - slot_to_shard.insert(std::end(slot_to_shard), info.weight, shards_info.size()); shards_info.emplace_back(std::move(info)); } }; @@ -874,10 +876,37 @@ Cluster::Cluster(Cluster::ReplicasAsShardsTag, const Cluster & from, const Setti secret = from.secret; name = from.name; + constrainShardInfoAndAddressesToMaxHosts(max_hosts); + + for (size_t i = 0; i < shards_info.size(); ++i) + slot_to_shard.insert(std::end(slot_to_shard), shards_info[i].weight, i); + initMisc(); } +void Cluster::constrainShardInfoAndAddressesToMaxHosts(size_t max_hosts) +{ + if (max_hosts == 0 || shards_info.size() <= max_hosts) + return; + + pcg64_fast gen{randomSeed()}; + std::shuffle(shards_info.begin(), shards_info.end(), gen); + shards_info.resize(max_hosts); + + AddressesWithFailover addresses_with_failover_; + + UInt32 shard_num = 0; + for (auto & shard_info : shards_info) + { + addresses_with_failover_.push_back(addresses_with_failover[shard_info.shard_num - 1]); + shard_info.shard_num = ++shard_num; + } + + addresses_with_failover.swap(addresses_with_failover_); +} + + Cluster::Cluster(Cluster::SubclusterTag, const Cluster & from, const std::vector & indices) { for (size_t index : indices) diff --git a/src/Interpreters/Cluster.h b/src/Interpreters/Cluster.h index 2b74c2b9e6c8..2d707b51265f 100644 --- a/src/Interpreters/Cluster.h +++ b/src/Interpreters/Cluster.h @@ -285,7 +285,7 @@ class Cluster std::unique_ptr getClusterWithMultipleShards(const std::vector & indices) const; /// Get a new Cluster that contains all servers (all shards with all replicas) from existing cluster as independent shards. - std::unique_ptr getClusterWithReplicasAsShards(const Settings & settings, size_t max_replicas_from_shard = 0) const; + std::unique_ptr getClusterWithReplicasAsShards(const Settings & settings, size_t max_replicas_from_shard = 0, size_t max_hosts = 0) const; /// Returns false if cluster configuration doesn't allow to use it for cross-replication. /// NOTE: true does not mean, that it's actually a cross-replication cluster. @@ -311,7 +311,7 @@ class Cluster /// For getClusterWithReplicasAsShards implementation struct ReplicasAsShardsTag {}; - Cluster(ReplicasAsShardsTag, const Cluster & from, const Settings & settings, size_t max_replicas_from_shard); + Cluster(ReplicasAsShardsTag, const Cluster & from, const Settings & settings, size_t max_replicas_from_shard, size_t max_hosts); void addShard( const Settings & settings, @@ -322,6 +322,9 @@ class Cluster UInt32 weight = 1, bool internal_replication = false); + /// Reduce size of cluster to max_hosts + void constrainShardInfoAndAddressesToMaxHosts(size_t max_hosts); + /// Inter-server secret String secret; diff --git a/src/Interpreters/ClusterDiscovery.cpp b/src/Interpreters/ClusterDiscovery.cpp index 20efaebf2f4b..fdbe423e3d8f 100644 --- a/src/Interpreters/ClusterDiscovery.cpp +++ b/src/Interpreters/ClusterDiscovery.cpp @@ -2,7 +2,9 @@ #include #include #include +#include #include +#include #include #include @@ -54,6 +56,8 @@ namespace ErrorCodes namespace FailPoints { extern const char cluster_discovery_faults[]; + extern const char cluster_discovery_unregister_fail[]; + extern const char cluster_discovery_retry_signal_fail[]; } namespace @@ -78,7 +82,7 @@ ClusterDiscovery::ClusterInfo::ClusterInfo(const String & name_, size_t shard_id, bool observer_mode, bool invisible, - size_t zk_root_index_ + const String & multicluster_full_path_ ) : name(name_) , zk_name(zk_name_) @@ -90,7 +94,7 @@ ClusterDiscovery::ClusterInfo::ClusterInfo(const String & name_, , username(username_) , password(password_) , cluster_secret(cluster_secret_) - , zk_root_index(zk_root_index_) + , multicluster_full_path(multicluster_full_path_) { } @@ -103,12 +107,7 @@ template class ClusterDiscovery::Flags { public: - template - Flags(It begin, It end) - { - for (auto it = begin; it != end; ++it) - flags.emplace(*it, false); - } + Flags() = default; void set(const T & key, bool value = true) { @@ -120,6 +119,21 @@ class ClusterDiscovery::Flags cv.notify_one(); } + /// Set an existing key only. Late Keeper watch callbacks must use this so a removed + /// cluster name cannot be reinserted after Flags::remove. + void setIfPresent(const T & key, bool value = true) + { + std::unique_lock lk(mu); + if (stop_flag) + return; + auto it = flags.find(key); + if (it == flags.end()) + return; + it->second = value; + any_need_update |= value; + cv.notify_one(); + } + /// Just notify the condition variable. void set() { @@ -157,9 +171,22 @@ class ClusterDiscovery::Flags cv.notify_one(); } + bool isStopped() const + { + std::unique_lock lk(mu); + return stop_flag; + } + + void wakeup() + { + std::unique_lock lk(mu); + any_need_update = true; + cv.notify_one(); + } + private: std::condition_variable cv; - std::mutex mu; + mutable std::mutex mu; /// flag indicates that update is required std::unordered_map flags; @@ -167,17 +194,12 @@ class ClusterDiscovery::Flags bool stop_flag = false; }; -ClusterDiscovery::ClusterDiscovery( +ClusterDiscovery::ParsedDiscoveryConfig ClusterDiscovery::parseDiscoveryConfig( const Poco::Util::AbstractConfiguration & config, - ContextPtr context_, - MultiVersion::Version macros_, + ContextPtr context, const String & config_prefix) - : context(Context::createCopy(context_)) - , current_node_name(toString(ServerUUID::get())) - , log(getLogger("ClusterDiscovery")) - , macros(macros_) { - LOG_DEBUG(log, "Cluster discovery is enabled"); + ParsedDiscoveryConfig result; Poco::Util::AbstractConfiguration::Keys config_keys; config.keys(config_prefix, config_keys); @@ -213,16 +235,14 @@ ClusterDiscovery::ClusterDiscovery( String zk_root = zkutil::extractZooKeeperPath(zk_multicluster_name_and_root, true); String zk_name = zkutil::extractZooKeeperName(zk_multicluster_name_and_root); - MulticlusterDiscovery mcd( - /* zk_name */ zk_name, - /* zk_path */ zk_root, - /* is_secure_connection */ config.getBool(cluster_config_prefix + ".secure", false), - /* username */ config.getString(cluster_config_prefix + ".user", context->getUserName()), - /* password */ password, - /* cluster_secret */ cluster_secret - ); - - multicluster_discovery_paths.push_back(std::move(mcd)); + result.multicluster_roots.push_back(ParsedMulticlusterDiscovery{ + .zk_name = zk_name, + .zk_path = zk_root, + .is_secure_connection = config.getBool(cluster_config_prefix + ".secure", false), + .username = config.getString(cluster_config_prefix + ".user", context->getUserName()), + .password = password, + .cluster_secret = cluster_secret, + }); continue; } @@ -232,51 +252,433 @@ ClusterDiscovery::ClusterDiscovery( String zk_root = zkutil::extractZooKeeperPath(zk_name_and_root, true); String zk_name = zkutil::extractZooKeeperName(zk_name_and_root); - clusters_info.emplace( - key, - ClusterInfo( - /* name_= */ key, - /* zk_name_= */ zk_name, - /* zk_root_= */ zk_root, - /* host_name= */ config.getString(cluster_config_prefix + ".my_hostname", getFQDNOrHostName()), - /* username= */ config.getString(cluster_config_prefix + ".user", context->getUserName()), - /* password= */ password, - /* cluster_secret= */ cluster_secret, - /* port= */ context->getTCPPort(), - /* secure= */ config.getBool(cluster_config_prefix + ".secure", false), - /* shard_id= */ config.getUInt(cluster_config_prefix + ".shard", 0), - /* observer_mode= */ is_observer, - /* invisible= */ ConfigHelper::getBool(config, cluster_config_prefix + ".invisible") - ) - ); + result.static_clusters.push_back(ParsedStaticDiscovery{ + .name = key, + .zk_name = zk_name, + .zk_root = zk_root, + .host_name = config.getString(cluster_config_prefix + ".my_hostname", getFQDNOrHostName()), + .username = config.getString(cluster_config_prefix + ".user", context->getUserName()), + .password = password, + .cluster_secret = cluster_secret, + .secure = config.getBool(cluster_config_prefix + ".secure", false), + .shard_id = config.getUInt(cluster_config_prefix + ".shard", 0), + .observer = is_observer, + .invisible = ConfigHelper::getBool(config, cluster_config_prefix + ".invisible"), + }); } + return result; +} + +void ClusterDiscovery::validateConfig( + const Poco::Util::AbstractConfiguration & config, + ContextPtr context, + const String & config_prefix) +{ + /// Discard result; throws on invalid discovery subtrees. + parseDiscoveryConfig(config, context, config_prefix); +} + +ClusterDiscovery::ClusterDiscovery( + const Poco::Util::AbstractConfiguration & config, + ContextPtr context_, + MultiVersion::Version macros_, + const String & config_prefix) + : context(Context::createCopy(context_)) + , current_node_name(toString(ServerUUID::get())) + , clusters_to_update(std::make_shared()) + , log(getLogger("ClusterDiscovery")) + , macros(macros_) +{ + LOG_DEBUG(log, "Cluster discovery is enabled"); + + auto parsed = parseDiscoveryConfig(config, context, config_prefix); + + for (auto & static_cluster : parsed.static_clusters) + addStaticCluster(std::move(static_cluster)); + + for (auto & root : parsed.multicluster_roots) + addMulticlusterRoot(std::move(root)); + std::vector clusters_info_names; clusters_info_names.reserve(clusters_info.size()); for (const auto & e : clusters_info) clusters_info_names.emplace_back(e.first); LOG_TRACE(log, "Clusters in discovery mode: {}", fmt::join(clusters_info_names, ", ")); - clusters_to_update = std::make_shared(clusters_info_names.begin(), clusters_info_names.end()); +} - /// Init get_nodes_callbacks after init clusters_to_update. - for (const auto & e : clusters_info) - get_nodes_callbacks[e.first] = std::make_shared( - [cluster_name = e.first, my_clusters_to_update = clusters_to_update](auto) - { - my_clusters_to_update->set(cluster_name); - }); +void ClusterDiscovery::addStaticCluster(ParsedStaticDiscovery && parsed) +{ + const String name = parsed.name; - for (auto & path : multicluster_discovery_paths) + if (auto existing = clusters_info.find(name); existing != clusters_info.end()) { - path.watch_callback = std::make_shared( - [my_need_update = path.need_update, my_flag = clusters_to_update](auto) - { - my_need_update->store(true); - my_flag->set(); - } - ); + if (existing->second.isDynamic()) + { + /// Static config wins over multicluster discovery (same as findDynamicClusters). + removeDynamicCluster(name); + } + else + { + LOG_DEBUG(log, "Static discovery cluster '{}' already exists, skip add", name); + return; + } + } + + clusters_info.emplace( + name, + ClusterInfo( + /* name_= */ parsed.name, + /* zk_name_= */ parsed.zk_name, + /* zk_root_= */ parsed.zk_root, + /* host_name= */ parsed.host_name, + /* username= */ parsed.username, + /* password= */ parsed.password, + /* cluster_secret= */ parsed.cluster_secret, + /* port= */ context->getTCPPort(), + /* secure= */ parsed.secure, + /* shard_id= */ parsed.shard_id, + /* observer_mode= */ parsed.observer, + /* invisible= */ parsed.invisible)); + + get_nodes_callbacks[name] = std::make_shared( + [cluster_name = name, my_clusters_to_update = clusters_to_update](auto) + { + my_clusters_to_update->setIfPresent(cluster_name); + }); + + clusters_to_update->set(name); +} + +void ClusterDiscovery::removeStaticCluster(const String & name) +{ + auto it = clusters_info.find(name); + if (it == clusters_info.end() || it->second.isDynamic()) + return; + + /// Drop local tracking even if Keeper remove fails: config already removed the cluster. + /// Keep enough identity to retry ephemeral cleanup so peers stop seeing this node. + /// Aliases that share zk_name/zk_root share one ephemeral; do not remove it while another + /// participant alias still needs the registration. + if (pathHasActiveParticipant(it->second.zk_name, it->second.zk_root, &name)) + { + LOG_DEBUG( + log, + "Skip unregister for removed cluster '{}': another participant still owns path {}:{}", + name, + it->second.zk_name, + it->second.zk_root); + } + else if (!unregisterFromZk(it->second.zk_name, it->second.zk_root, name)) + { + pending_zk_unregisters.push_back(PendingZkUnregister{ + .zk_name = it->second.zk_name, + .zk_root = it->second.zk_root, + .cluster_name = name, + }); + LOG_WARNING( + log, + "Failed to unregister current node from cluster '{}' on config remove; will retry", + name); + } + + clusters_to_update->remove(name); + get_nodes_callbacks.erase(name); + clusters_info.erase(it); + + { + std::lock_guard lock(mutex); + cluster_impls.erase(name); + } + + /// A static entry may have been shadowing a same-named cluster under a multicluster root. + /// Roots are otherwise skipped while need_update is false and no children event fires. + if (!multicluster_discovery_paths.empty()) + { + markMulticlusterRootsNeedUpdate(); + clusters_to_update->set(); + } + + LOG_DEBUG(log, "Static discovery cluster '{}' removed due to config change", name); +} + +void ClusterDiscovery::removeDynamicCluster(const String & name) +{ + auto it = clusters_info.find(name); + if (it == clusters_info.end() || !it->second.isDynamic()) + return; + + removeCluster(name, /* is_dynamic */ true); + clusters_info.erase(name); + LOG_DEBUG(log, "Dynamic discovery cluster '{}' removed to make way for static config", name); +} + +bool ClusterDiscovery::updateStaticClusterFields(ClusterInfo & info, const ParsedStaticDiscovery & parsed) +{ + bool identity_changed = info.zk_name != parsed.zk_name || info.zk_root != parsed.zk_root; + if (identity_changed) + return false; + + bool credentials_changed + = info.username != parsed.username + || info.password != parsed.password + || info.cluster_secret != parsed.cluster_secret + || info.is_secure_connection != parsed.secure; + + String expected_address = parsed.host_name + ":" + toString(context->getTCPPort()); + bool registration_changed + = info.current_node_is_observer != parsed.observer + || info.current_node.shard_id != parsed.shard_id + || info.current_node.address != expected_address + || info.current_node.secure != parsed.secure; + + bool invisible_changed = info.current_cluster_is_invisible != parsed.invisible; + + if (!credentials_changed && !registration_changed && !invisible_changed) + return true; + + info.username = parsed.username; + info.password = parsed.password; + info.cluster_secret = parsed.cluster_secret; + info.is_secure_connection = parsed.secure; + info.current_node_is_observer = parsed.observer; + info.current_cluster_is_invisible = parsed.invisible; + info.current_node = NodeInfo(expected_address, parsed.secure, parsed.shard_id); + + if (registration_changed || invisible_changed) + { + /// Force upsertCluster to re-read ZK payloads; membership UUIDs alone do not change + /// when only address/shard/secure are updated. + if (registration_changed) + info.nodes_info.clear(); + clusters_to_update->set(info.name); + } + else + rebuildClusterObject(info); + + return true; +} + +void ClusterDiscovery::addMulticlusterRoot(ParsedMulticlusterDiscovery && parsed) +{ + const String full_path = parsed.getFullPath(); + if (multicluster_discovery_paths.contains(full_path)) + return; + + MulticlusterDiscovery mcd( + /* zk_name */ parsed.zk_name, + /* zk_path */ parsed.zk_path, + /* is_secure_connection */ parsed.is_secure_connection, + /* username */ parsed.username, + /* password */ parsed.password, + /* cluster_secret */ parsed.cluster_secret); + + mcd.watch_callback = std::make_shared( + [my_need_update = mcd.need_update, my_flag = clusters_to_update](auto) + { + my_need_update->store(true); + my_flag->set(); + }); + + multicluster_discovery_paths.emplace(full_path, std::move(mcd)); + clusters_to_update->set(); + + LOG_DEBUG(log, "Added multicluster discovery root '{}'", full_path); +} + +void ClusterDiscovery::removeMulticlusterRoot(const String & full_path) +{ + if (!multicluster_discovery_paths.erase(full_path)) + return; + + std::vector dynamic_clusters; + for (const auto & [name, info] : clusters_info) + { + if (info.multicluster_full_path == full_path) + dynamic_clusters.push_back(name); + } + + for (const auto & name : dynamic_clusters) + removeDynamicCluster(name); + + clusters_to_update->set(); + + LOG_DEBUG(log, "Removed multicluster discovery root '{}'", full_path); +} + +void ClusterDiscovery::markMulticlusterRootsNeedUpdate() +{ + for (auto & [_, path] : multicluster_discovery_paths) + path.need_update->store(true); +} + +bool ClusterDiscovery::updateMulticlusterRootFields(MulticlusterDiscovery & path, const ParsedMulticlusterDiscovery & parsed) +{ + bool credentials_changed + = path.username != parsed.username + || path.password != parsed.password + || path.cluster_secret != parsed.cluster_secret + || path.is_secure_connection != parsed.is_secure_connection; + + if (!credentials_changed) + return true; + + path.username = parsed.username; + path.password = parsed.password; + path.cluster_secret = parsed.cluster_secret; + path.is_secure_connection = parsed.is_secure_connection; + + const String full_path = path.getFullPath(); + for (auto & [_, info] : clusters_info) + { + if (info.multicluster_full_path != full_path) + continue; + info.username = parsed.username; + info.password = parsed.password; + info.cluster_secret = parsed.cluster_secret; + info.is_secure_connection = parsed.is_secure_connection; + info.current_node.secure = parsed.is_secure_connection; + rebuildClusterObject(info); + } + + return true; +} + +void ClusterDiscovery::rebuildClusterObject(const ClusterInfo & info) +{ + if (info.current_cluster_is_invisible || info.nodes_info.empty()) + { + std::lock_guard lock(mutex); + cluster_impls.erase(info.name); + return; + } + + auto cluster = makeCluster(info); + std::lock_guard lock(mutex); + cluster_impls[info.name] = std::move(cluster); +} + +void ClusterDiscovery::applyParsedConfig(ParsedDiscoveryConfig && parsed) +{ + std::unordered_map desired_static; + for (auto & c : parsed.static_clusters) + desired_static.emplace(c.name, std::move(c)); + + std::unordered_map desired_multi; + for (auto & r : parsed.multicluster_roots) + desired_multi.emplace(r.getFullPath(), std::move(r)); + + std::vector static_to_remove; + std::vector static_identity_replace; + std::vector static_to_add; + std::vector multi_to_remove; + std::vector multi_to_add; + + for (auto & [name, info] : clusters_info) + { + if (info.isDynamic()) + continue; + + auto it = desired_static.find(name); + if (it == desired_static.end()) + { + static_to_remove.push_back(name); + continue; + } + + if (!updateStaticClusterFields(info, it->second)) + static_identity_replace.push_back(name); + else + desired_static.erase(it); + } + + for (auto & [full_path, path] : multicluster_discovery_paths) + { + auto it = desired_multi.find(full_path); + if (it == desired_multi.end()) + { + multi_to_remove.push_back(full_path); + continue; + } + + updateMulticlusterRootFields(path, it->second); + desired_multi.erase(it); + } + + for (const auto & name : static_identity_replace) + { + auto it = desired_static.find(name); + if (it != desired_static.end()) + { + static_to_add.push_back(std::move(it->second)); + desired_static.erase(it); + } + static_to_remove.push_back(name); + } + + for (auto & [_, c] : desired_static) + static_to_add.push_back(std::move(c)); + for (auto & [_, r] : desired_multi) + multi_to_add.push_back(std::move(r)); + + for (const auto & name : static_to_remove) + removeStaticCluster(name); + for (auto & c : static_to_add) + addStaticCluster(std::move(c)); + + for (const auto & full_path : multi_to_remove) + removeMulticlusterRoot(full_path); + for (auto & r : multi_to_add) + addMulticlusterRoot(std::move(r)); +} + +void ClusterDiscovery::updateFromConfig( + const Poco::Util::AbstractConfiguration & config, + const String & config_prefix) +{ + LOG_DEBUG(log, "Scheduling cluster discovery config update"); + auto parsed = parseDiscoveryConfig(config, context, config_prefix); + { + std::lock_guard lock(pending_config_mutex); + pending_config_update = std::move(parsed); + } + ensureWorkerStarted(); + clusters_to_update->set(); +} + +bool ClusterDiscovery::consumePendingConfigUpdate() +{ + std::optional pending; + { + std::lock_guard lock(pending_config_mutex); + pending.swap(pending_config_update); + } + if (!pending) + return false; + + LOG_DEBUG(log, "Applying pending cluster discovery config update"); + applyParsedConfig(std::move(*pending)); + return true; +} + +void ClusterDiscovery::ensureWorkerStarted() +{ + std::lock_guard lock(start_mutex); + if (main_thread.joinable()) + return; + + if (clusters_info.empty() && multicluster_discovery_paths.empty()) + { + std::lock_guard pending_lock(pending_config_mutex); + if (!pending_config_update) + return; + /// Pending update may add the first discovery path; apply it before start(). } + + /// If worker is not running yet, apply pending config inline so startImpl() sees new clusters. + consumePendingConfigUpdate(); + startImpl(); } /// List node in zookeper for cluster @@ -285,7 +687,7 @@ Strings ClusterDiscovery::getNodeNames(zkutil::ZooKeeperPtr & zk, const String & cluster_name, int * version, bool set_callback, - size_t zk_root_index) + const String & multicluster_full_path) { Coordination::Stat stat; Strings nodes; @@ -295,10 +697,18 @@ Strings ClusterDiscovery::getNodeNames(zkutil::ZooKeeperPtr & zk, auto callback = get_nodes_callbacks.find(cluster_name); if (callback == get_nodes_callbacks.end()) { + std::shared_ptr need_update; + if (!multicluster_full_path.empty()) + { + auto path_it = multicluster_discovery_paths.find(multicluster_full_path); + if (path_it != multicluster_discovery_paths.end()) + need_update = path_it->second.need_update; + } + auto watch_dynamic_callback = std::make_shared([ cluster_name, my_clusters_to_update = clusters_to_update, - my_discovery_paths_need_update = multicluster_discovery_paths[zk_root_index - 1].need_update + my_discovery_paths_need_update = need_update ](auto) { my_discovery_paths_need_update->store(true); @@ -339,10 +749,8 @@ ClusterDiscovery::NodesInfo ClusterDiscovery::getNodes(zkutil::ZooKeeperPtr & zk return result; } -/// Checks if cluster nodes set is changed. -/// Returns true if update required. -/// It performs only shallow check (set of nodes' uuids). -/// So, if node's hostname are changed, then cluster won't be updated. +/// Checks if cluster membership (set of node UUIDs) changed. +/// Used for logging; payload refresh is decided separately in upsertCluster. bool ClusterDiscovery::needUpdate(const Strings & node_uuids, const NodesInfo & nodes) { bool has_difference = node_uuids.size() != nodes.size() || @@ -430,15 +838,18 @@ bool ClusterDiscovery::upsertCluster(ClusterInfo & cluster_info) LOG_DEBUG(log, "Updating cluster '{}'", cluster_info.name); auto zk = context->getDefaultOrAuxiliaryZooKeeper(cluster_info.zk_name); + registerInZk(zk, cluster_info); int start_version = 0; - Strings node_uuids = getNodeNames(zk, cluster_info.zk_root, cluster_info.name, &start_version, false, cluster_info.zk_root_index); + Strings node_uuids = getNodeNames( + zk, cluster_info.zk_root, cluster_info.name, &start_version, false, cluster_info.multicluster_full_path); auto & nodes_info = cluster_info.nodes_info; auto on_exit = [this, start_version, &zk, &cluster_info, &nodes_info]() { /// in case of successful update we still need to check if configuration of cluster still valid and also set watch callback int current_version = 0; - getNodeNames(zk, cluster_info.zk_root, cluster_info.name, ¤t_version, true, cluster_info.zk_root_index); + getNodeNames( + zk, cluster_info.zk_root, cluster_info.name, ¤t_version, true, cluster_info.multicluster_full_path); if (current_version != start_version) { @@ -449,7 +860,9 @@ bool ClusterDiscovery::upsertCluster(ClusterInfo & cluster_info) return true; }; - if (!cluster_info.current_node_is_observer && !contains(node_uuids, current_node_name)) + if (!cluster_info.current_node_is_observer + && context->isSwarmModeEnabled() + && !contains(node_uuids, current_node_name)) { LOG_ERROR(log, "Can't find current node in cluster '{}', will register again", cluster_info.name); registerInZk(zk, cluster_info); @@ -460,15 +873,17 @@ bool ClusterDiscovery::upsertCluster(ClusterInfo & cluster_info) if (cluster_info.current_cluster_is_invisible) { LOG_DEBUG(log, "Cluster '{}' is invisible.", cluster_info.name); + std::lock_guard lock(mutex); + cluster_impls.erase(cluster_info.name); return true; } if (!needUpdate(node_uuids, nodes_info)) - { - LOG_DEBUG(log, "No update required for cluster '{}'", cluster_info.name); - return on_exit(); - } + LOG_DEBUG(log, "Membership unchanged for cluster '{}', refreshing node payloads", cluster_info.name); + /// Always re-read ephemeral payloads so hostname/shard/secure updates propagate even when + /// the UUID set is unchanged (createOrUpdate does not fire children watches by itself; + /// registerInZk recreates the node when data changes to notify peers). nodes_info = getNodes(zk, cluster_info.zk_root, node_uuids); if (bool ok = on_exit(); !ok) @@ -478,14 +893,15 @@ bool ClusterDiscovery::upsertCluster(ClusterInfo & cluster_info) if (nodes_info.empty()) { - removeCluster(cluster_info.name, /* is_dynamic_cluster */cluster_info.zk_root_index != 0); + String name = cluster_info.name; + if (cluster_info.isDynamic()) + removeDynamicCluster(name); + else + removeCluster(name, /* is_dynamic */ false); return true; } - auto cluster = makeCluster(cluster_info); - std::lock_guard lock(mutex); - cluster_impls[cluster_info.name] = cluster; - + rebuildClusterObject(cluster_info); return true; } @@ -514,16 +930,147 @@ void ClusterDiscovery::registerInZk(zkutil::ZooKeeperPtr & zk, ClusterInfo & inf if (info.current_node_is_observer) { + /// Drop leftover ephemeral registration when transitioning from participant to observer + /// (or if a stale node remained). ZNONODE means already absent. + /// Shared-path aliases: another participant may still own this ephemeral. + if (pathHasActiveParticipant(info.zk_name, info.zk_root)) + { + LOG_DEBUG( + log, + "Current node {} is observer of cluster {} (shared path {}:{} still owned by another participant)", + current_node_name, + info.name, + info.zk_name, + info.zk_root); + return; + } + fiu_do_on(FailPoints::cluster_discovery_retry_signal_fail, + { + throw Exception( + ErrorCodes::KEEPER_EXCEPTION, + "Failpoint cluster_discovery_retry_signal_fail is triggered in observer tryRemove"); + }); + auto code = zk->tryRemove(node_path); + if (code != Coordination::Error::ZOK && code != Coordination::Error::ZNONODE) + throw Exception( + ErrorCodes::KEEPER_EXCEPTION, + "Cannot remove discovery registration for observer node {}: {}", + node_path, + Coordination::errorMessage(code)); LOG_DEBUG(log, "Current node {} is observer of cluster {}", current_node_name, info.name); return; } + if (!context->isSwarmModeEnabled()) + { + LOG_DEBUG(log, "STOP SWARM MODE called, skip self-registering current node {} in cluster {}", current_node_name, info.name); + return; + } + LOG_DEBUG(log, "Registering current node {} in cluster {}", current_node_name, info.name); - zk->createOrUpdate(node_path, info.current_node.serialize(), zkutil::CreateMode::Ephemeral); + const String payload = info.current_node.serialize(); + String existing; + if (zk->tryGet(node_path, existing)) + { + if (existing == payload) + { + LOG_DEBUG(log, "Current node {} already registered in cluster {} with up-to-date data", current_node_name, info.name); + return; + } + /// Recreate ephemeral so children watches fire; setData alone does not notify peers. + zk->tryRemove(node_path); + } + + zk->create(node_path, payload, zkutil::CreateMode::Ephemeral); LOG_DEBUG(log, "Current node {} registered in cluster {}", current_node_name, info.name); } +bool ClusterDiscovery::unregisterFromZk(const String & zk_name, const String & zk_root, const String & cluster_name) +{ + try + { + fiu_do_on(FailPoints::cluster_discovery_unregister_fail, + { + throw Exception( + ErrorCodes::KEEPER_EXCEPTION, + "Failpoint cluster_discovery_unregister_fail is triggered for cluster '{}'", + cluster_name); + }); + + auto zk = context->getDefaultOrAuxiliaryZooKeeper(zk_name); + String node_path = getShardsListPath(zk_root) / current_node_name; + auto code = zk->tryRemove(node_path); + if (code != Coordination::Error::ZOK && code != Coordination::Error::ZNONODE) + { + LOG_WARNING( + log, + "Cannot unregister current node {} from cluster '{}': {}", + current_node_name, + cluster_name, + Coordination::errorMessage(code)); + return false; + } + + LOG_DEBUG(log, "Current node {} unregistered from cluster {}", current_node_name, cluster_name); + return true; + } + catch (...) + { + tryLogCurrentException(log, "Error while unregistering node from cluster '" + cluster_name + "'"); + return false; + } +} + +bool ClusterDiscovery::pathHasActiveParticipant( + const String & zk_name, + const String & zk_root, + const String * exclude_cluster_name) const +{ + for (const auto & [name, info] : clusters_info) + { + if (exclude_cluster_name && name == *exclude_cluster_name) + continue; + if (!info.current_node_is_observer && info.zk_name == zk_name && info.zk_root == zk_root) + return true; + } + return false; +} + +bool ClusterDiscovery::retryPendingUnregisters() +{ + if (pending_zk_unregisters.empty()) + return true; + + std::vector still_pending; + still_pending.reserve(pending_zk_unregisters.size()); + + for (const auto & pending : pending_zk_unregisters) + { + /// Re-add put a participant back on this path; drop stale cleanup instead of deleting the live ephemeral. + if (pathHasActiveParticipant(pending.zk_name, pending.zk_root)) + continue; + + if (!unregisterFromZk(pending.zk_name, pending.zk_root, pending.cluster_name)) + still_pending.push_back(pending); + } + + pending_zk_unregisters = std::move(still_pending); + return pending_zk_unregisters.empty(); +} + +void ClusterDiscovery::unregisterFromZk(zkutil::ZooKeeperPtr & zk, ClusterInfo & info) +{ + if (info.current_node_is_observer) + return; + + String node_path = getShardsListPath(info.zk_root) / current_node_name; + LOG_DEBUG(log, "Removing current node {} from cluster {}", current_node_name, info.name); + + zk->remove(node_path); + LOG_DEBUG(log, "Current node {} removed from cluster {}", current_node_name, info.name); +} + void ClusterDiscovery::initialUpdate() { LOG_DEBUG(log, "Initializing"); @@ -539,7 +1086,7 @@ void ClusterDiscovery::initialUpdate() throw Exception(ErrorCodes::KEEPER_EXCEPTION, "Failpoint cluster_discovery_faults is triggered"); }); - for (const auto & path : multicluster_discovery_paths) + for (const auto & [_, path] : multicluster_discovery_paths) { auto zk = context->getDefaultOrAuxiliaryZooKeeper(path.zk_name); @@ -549,37 +1096,58 @@ void ClusterDiscovery::initialUpdate() findDynamicClusters(clusters_info); - for (auto & [_, info] : clusters_info) + std::vector cluster_names; + cluster_names.reserve(clusters_info.size()); + for (const auto & [name, _] : clusters_info) + cluster_names.push_back(name); + + for (const auto & name : cluster_names) { + auto it = clusters_info.find(name); + if (it == clusters_info.end()) + continue; + + auto & info = it->second; auto zk = context->getDefaultOrAuxiliaryZooKeeper(info.zk_name); registerInZk(zk, info); if (!upsertCluster(info)) { - LOG_WARNING(log, "Error on initial cluster '{}' update, will retry in background", info.name); - clusters_to_update->set(info.name); + LOG_WARNING(log, "Error on initial cluster '{}' update, will retry in background", name); + clusters_to_update->set(name); } - else if (info.zk_root_index) - clusters_to_update->set(info.name, false); + else if (auto after = clusters_info.find(name); after != clusters_info.end() && after->second.isDynamic()) + clusters_to_update->set(name, false); } LOG_DEBUG(log, "Initialized"); is_initialized = true; } +void ClusterDiscovery::registerAll() +{ + register_change_flag = RegisterChangeFlag::RCF_REGISTER_ALL; + clusters_to_update->wakeup(); +} + +void ClusterDiscovery::unregisterAll() +{ + register_change_flag = RegisterChangeFlag::RCF_UNREGISTER_ALL; + clusters_to_update->wakeup(); +} + void ClusterDiscovery::findDynamicClusters( std::unordered_map & info, - std::unordered_set * unchanged_roots) + std::unordered_set * unchanged_roots) { using namespace std::chrono_literals; constexpr auto force_update_interval = 2min; - size_t zk_root_index = 0; - - for (const auto & path : multicluster_discovery_paths) + for (auto & [full_path, path] : multicluster_discovery_paths) { - ++zk_root_index; - + /// Cleared before Keeper I/O so a throw must restore the bit; otherwise the worker + /// may sleep forever with no children watch reinstalled. + bool cleared_need_update = false; if (unchanged_roots) { if (!path.need_update->exchange(false)) @@ -588,63 +1156,92 @@ void ClusterDiscovery::findDynamicClusters( bool force_update = path.watch.elapsedSeconds() > std::chrono::seconds(force_update_interval).count(); if (!force_update) { - unchanged_roots->insert(zk_root_index); + unchanged_roots->insert(full_path); continue; } } + else + cleared_need_update = true; } - auto zk = context->getDefaultOrAuxiliaryZooKeeper(path.zk_name); - - auto clusters = zk->getChildrenWatch( - path.zk_path, - nullptr, - Coordination::WatchCallbackPtrOrEventPtr{path.watch_callback, ProfileEvents::ZooKeeperWatchTriggeredClusterDiscovery}); - - for (const auto & cluster : clusters) + try { - auto p = clusters_info.find(cluster); - if (p != clusters_info.end() && !p->second.zk_root_index) + fiu_do_on(FailPoints::cluster_discovery_retry_signal_fail, { - /// Not a warning - node can register itsefs in one cluster and discover other clusters - LOG_TRACE(log, "Found dynamic duplicate of cluster '{}' in config and Keeper, skipped", cluster); - continue; - } + throw Exception( + ErrorCodes::KEEPER_EXCEPTION, + "Failpoint cluster_discovery_retry_signal_fail is triggered in findDynamicClusters"); + }); - if (info.contains(cluster)) + auto zk = context->getDefaultOrAuxiliaryZooKeeper(path.zk_name); + zk->createAncestors(path.zk_path); + zk->createIfNotExists(path.zk_path, ""); + + auto clusters = zk->getChildrenWatch( + path.zk_path, + nullptr, + Coordination::WatchCallbackPtrOrEventPtr{path.watch_callback, ProfileEvents::ZooKeeperWatchTriggeredClusterDiscovery}); + + for (const auto & cluster : clusters) { - /// Possible with several root paths, it's a configuration error - LOG_WARNING(log, "Found dynamic duplicate of cluster '{}' in Keeper, skipped record by path {}:{}", - cluster, path.zk_name, path.zk_path); - continue; + auto p = clusters_info.find(cluster); + if (p != clusters_info.end() && !p->second.isDynamic()) + { + /// Not a warning - node can register itsefs in one cluster and discover other clusters + LOG_TRACE(log, "Found dynamic duplicate of cluster '{}' in config and Keeper, skipped", cluster); + continue; + } + + if (info.contains(cluster)) + { + /// Possible with several root paths, it's a configuration error + LOG_WARNING(log, "Found dynamic duplicate of cluster '{}' in Keeper, skipped record by path {}:{}", + cluster, path.zk_name, path.zk_path); + continue; + } + + info.emplace( + cluster, + ClusterInfo( + /* name_= */ cluster, + /* zk_name_= */ path.zk_name, + /* zk_root_= */ path.zk_path + "/" + cluster, + /* host_name= */ "", + /* username= */ path.username, + /* password= */ path.password, + /* cluster_secret= */ path.cluster_secret, + /* port= */ context->getTCPPort(), + /* secure= */ path.is_secure_connection, + /* shard_id= */ 0, + /* observer_mode= */ true, + /* invisible= */ false, + /* multicluster_full_path_= */ full_path + ) + ); } - info.emplace( - cluster, - ClusterInfo( - /* name_= */ cluster, - /* zk_name_= */ path.zk_name, - /* zk_root_= */ path.zk_path + "/" + cluster, - /* host_name= */ "", - /* username= */ path.username, - /* password= */ path.password, - /* cluster_secret= */ path.cluster_secret, - /* port= */ context->getTCPPort(), - /* secure= */ path.is_secure_connection, - /* shard_id= */ 0, - /* observer_mode= */ true, - /* invisible= */ false, - /* zk_root_index= */ zk_root_index - ) - ); + path.watch.restart(); + } + catch (...) + { + if (cleared_need_update) + path.need_update->store(true); + throw; } - - path.watch.restart(); } } void ClusterDiscovery::start() { + std::lock_guard lock(start_mutex); + startImpl(); +} + +void ClusterDiscovery::startImpl() +{ + if (main_thread.joinable()) + return; + if (clusters_info.empty() && multicluster_discovery_paths.empty()) { LOG_DEBUG(log, "No defined clusters for discovery"); @@ -654,6 +1251,8 @@ void ClusterDiscovery::start() try { auto component_guard = Coordination::setCurrentComponent("ClusterDiscovery::start"); + /// Apply any queued reload before the first init attempt (same rationale as runMainThread). + consumePendingConfigUpdate(); initialUpdate(); } catch (...) @@ -683,8 +1282,25 @@ void ClusterDiscovery::start() * should not stop discovery forever */ tryLogCurrentException(log, "Caught exception in cluster discovery runMainThread"); + if (clusters_to_update->isStopped()) + break; + /// Flags::wait may have already cleared the only wake bit before the throw. + /// Re-arm so backoff retry does not sleep until an unrelated event. + clusters_to_update->wakeup(); + } + if (finish || clusters_to_update->isStopped()) + break; + + /// Interruptible backoff so shutdown does not wait for a long sleep after errors. + for (auto remaining = backoff_timeout; remaining.count() > 0 && !clusters_to_update->isStopped();) + { + constexpr auto slice = std::chrono::milliseconds(50); + auto step = remaining < slice ? remaining : slice; + std::this_thread::sleep_for(step); + remaining -= step; } - std::this_thread::sleep_for(backoff_timeout); + if (clusters_to_update->isStopped()) + break; backoff_timeout = std::min(backoff_timeout * 2, std::chrono::milliseconds(3min)); } }); @@ -702,6 +1318,12 @@ bool ClusterDiscovery::runMainThread(std::function up_to_date_callback) constexpr auto force_update_interval = 2min; + /// Pending reloads must be applied before retrying init. Otherwise a worker that keeps + /// failing initialUpdate (bad Keeper path, etc.) never reaches the loop body consumer and + /// updateFromConfig stays stuck while ensureWorkerStarted no-ops on the running thread. + consumePendingConfigUpdate(); + retryPendingUnregisters(); + if (!is_initialized) initialUpdate(); @@ -713,89 +1335,172 @@ bool ClusterDiscovery::runMainThread(std::function up_to_date_callback) if (finished) break; - std::unordered_map new_dynamic_clusters_info; - std::unordered_set unchanged_roots; - findDynamicClusters(new_dynamic_clusters_info, &unchanged_roots); - - std::unordered_set clusters_to_insert; - std::unordered_set clusters_to_remove; + /// Snapshot of work acknowledged by wait(). Must be restored if this iteration throws + /// before watches / register commands are reinstalled; otherwise the worker blocks forever. + RegisterChangeFlag consumed_register_flag = RegisterChangeFlag::RCF_NONE; - /// Remove clusters that are not found in new_dynamic_clusters_info - for (const auto & [cluster_name, info] : clusters_info) + try { - if (!info.zk_root_index) - continue; - if (!new_dynamic_clusters_info.erase(cluster_name) - && !unchanged_roots.contains(info.zk_root_index)) - clusters_to_remove.insert(cluster_name); - } - /// new_dynamic_clusters_info now contains only new clusters - for (const auto & [cluster_name, _] : new_dynamic_clusters_info) - clusters_to_insert.insert(cluster_name); - - for (const auto & cluster_name : clusters_to_remove) - removeCluster(cluster_name, /* is_dynamic_cluster */true); + fiu_do_on(FailPoints::cluster_discovery_retry_signal_fail, + { + throw Exception( + ErrorCodes::KEEPER_EXCEPTION, + "Failpoint cluster_discovery_retry_signal_fail is triggered after Flags::wait"); + }); - clusters_info.merge(new_dynamic_clusters_info); + consumePendingConfigUpdate(); - for (const auto & [cluster_name, need_update] : clusters) - { - auto cluster_info_it = clusters_info.find(cluster_name); - if (cluster_info_it == clusters_info.end()) + if (!retryPendingUnregisters()) { - LOG_ERROR(log, "Unknown cluster '{}'", cluster_name); - continue; + /// Keep waking the loop with a short interruptible backoff so ephemeral cleanup + /// retries without failing / rolling back the already-applied config update. + using namespace std::chrono_literals; + for (auto remaining = std::chrono::milliseconds(1000); + remaining.count() > 0 && !clusters_to_update->isStopped();) + { + constexpr auto slice = std::chrono::milliseconds(50); + auto step = remaining < slice ? remaining : slice; + std::this_thread::sleep_for(step); + remaining -= step; + } + if (!clusters_to_update->isStopped()) + clusters_to_update->set(); } - auto & cluster_info = cluster_info_it->second; - if (!need_update) + std::unordered_map new_dynamic_clusters_info; + std::unordered_set unchanged_roots; + findDynamicClusters(new_dynamic_clusters_info, &unchanged_roots); + + std::unordered_set clusters_to_insert; + std::unordered_set clusters_to_remove; + + /// Remove clusters that are not found in new_dynamic_clusters_info + for (const auto & [cluster_name, info] : clusters_info) { - /// force updating periodically - bool force_update = cluster_info.watch.elapsedSeconds() > std::chrono::seconds(force_update_interval).count(); - if (!force_update) + if (!info.isDynamic()) continue; + if (!new_dynamic_clusters_info.erase(cluster_name) + && !unchanged_roots.contains(info.multicluster_full_path)) + clusters_to_remove.insert(cluster_name); } + /// new_dynamic_clusters_info now contains only new clusters + for (const auto & [cluster_name, _] : new_dynamic_clusters_info) + clusters_to_insert.insert(cluster_name); + + for (const auto & cluster_name : clusters_to_remove) + removeDynamicCluster(cluster_name); + + clusters_info.merge(new_dynamic_clusters_info); - if (upsertCluster(cluster_info)) + for (const auto & [cluster_name, need_update] : clusters) { - cluster_info.watch.restart(); - LOG_DEBUG(log, "Cluster '{}' updated successfully", cluster_name); + auto cluster_info_it = clusters_info.find(cluster_name); + if (cluster_info_it == clusters_info.end()) + { + LOG_ERROR(log, "Unknown cluster '{}'", cluster_name); + /// Drop keys resurrected by late Keeper callbacks after removal. + clusters_to_update->remove(cluster_name); + continue; + } + + auto & cluster_info = cluster_info_it->second; + if (!need_update) + { + /// force updating periodically + bool force_update = cluster_info.watch.elapsedSeconds() > std::chrono::seconds(force_update_interval).count(); + if (!force_update) + continue; + } + + String name = cluster_name; + if (upsertCluster(cluster_info)) + { + cluster_info_it = clusters_info.find(name); + if (cluster_info_it != clusters_info.end()) + cluster_info_it->second.watch.restart(); + LOG_DEBUG(log, "Cluster '{}' updated successfully", name); + } + else + { + all_up_to_date = false; + /// no need to trigger convar, will retry after timeout in `wait` + clusters_to_update->set(name); + LOG_WARNING(log, "Cluster '{}' wasn't updated, will retry", name); + } } - else + + for (const auto & cluster_name : clusters_to_insert) { - all_up_to_date = false; - /// no need to trigger convar, will retry after timeout in `wait` - clusters_to_update->set(cluster_name); - LOG_WARNING(log, "Cluster '{}' wasn't updated, will retry", cluster_name); + auto cluster_info_it = clusters_info.find(cluster_name); + if (cluster_info_it == clusters_info.end()) + { + LOG_ERROR(log, "Unknown dynamic cluster '{}'", cluster_name); + clusters_to_update->remove(cluster_name); + continue; + } + auto & cluster_info = cluster_info_it->second; + String name = cluster_name; + if (upsertCluster(cluster_info)) + { + cluster_info_it = clusters_info.find(name); + if (cluster_info_it != clusters_info.end()) + cluster_info_it->second.watch.restart(); + LOG_DEBUG(log, "Dynamic cluster '{}' inserted successfully", name); + } + else + { + all_up_to_date = false; + /// no need to trigger convar, will retry after timeout in `wait` + clusters_to_update->set(name); + LOG_WARNING(log, "Dynamic cluster '{}' wasn't inserted, will retry", name); + } } - } - for (const auto & cluster_name : clusters_to_insert) - { - auto cluster_info_it = clusters_info.find(cluster_name); - if (cluster_info_it == clusters_info.end()) + if (all_up_to_date) { - LOG_ERROR(log, "Unknown dynamic cluster '{}'", cluster_name); - continue; + up_to_date_callback(); } - auto & cluster_info = cluster_info_it->second; - if (upsertCluster(cluster_info)) + + consumed_register_flag = register_change_flag.exchange(RegisterChangeFlag::RCF_NONE); + + if (consumed_register_flag == RegisterChangeFlag::RCF_REGISTER_ALL) { - cluster_info.watch.restart(); - LOG_DEBUG(log, "Dynamic cluster '{}' inserted successfully", cluster_name); + LOG_DEBUG(log, "Register in all dynamic clusters"); + for (auto & [_, info] : clusters_info) + { + auto zk = context->getDefaultOrAuxiliaryZooKeeper(info.zk_name); + registerInZk(zk, info); + } } - else + else if (consumed_register_flag == RegisterChangeFlag::RCF_UNREGISTER_ALL) { - all_up_to_date = false; - /// no need to trigger convar, will retry after timeout in `wait` - clusters_to_update->set(cluster_name); - LOG_WARNING(log, "Dynamic cluster '{}' wasn't inserted, will retry", cluster_name); + LOG_DEBUG(log, "Unregister in all dynamic clusters"); + for (auto & [_, info] : clusters_info) + { + auto zk = context->getDefaultOrAuxiliaryZooKeeper(info.zk_name); + unregisterFromZk(zk, info); + } } - } - if (all_up_to_date) + consumed_register_flag = RegisterChangeFlag::RCF_NONE; + } + catch (...) { - up_to_date_callback(); + for (const auto & [cluster_name, need_update] : clusters) + { + if (need_update) + clusters_to_update->set(cluster_name); + } + + if (consumed_register_flag != RegisterChangeFlag::RCF_NONE) + { + /// Do not overwrite a newer registerAll/unregisterAll posted while we failed. + RegisterChangeFlag expected = RegisterChangeFlag::RCF_NONE; + register_change_flag.compare_exchange_strong(expected, consumed_register_flag); + } + + clusters_to_update->wakeup(); + throw; } } LOG_DEBUG(log, "Worker thread stopped"); @@ -821,12 +1526,65 @@ std::unordered_map ClusterDiscovery::getClusters() const void ClusterDiscovery::shutdown() { LOG_DEBUG(log, "Shutting down"); - clusters_to_update->stop(); + if (clusters_to_update) + clusters_to_update->stop(); + /// Wait for any in-flight startImpl() before joining so we do not race ThreadFromGlobalPool assign. + std::lock_guard lock(start_mutex); if (main_thread.joinable()) main_thread.join(); } +void ClusterDiscovery::disableAndShutdown() +{ + LOG_DEBUG(log, "Disabling cluster discovery"); + + /// Stop the worker before touching clusters_info so upsert cannot re-register. + shutdown(); + + /// Config-reloader thread has no ZooKeeper component; unregister requires one. + auto component_guard = Coordination::setCurrentComponent("ClusterDiscovery::disableAndShutdown"); + + for (const auto & [name, info] : clusters_info) + { + if (info.current_node_is_observer) + continue; + if (!unregisterFromZk(info.zk_name, info.zk_root, name)) + { + LOG_WARNING( + log, + "Failed to unregister current node from cluster '{}' while disabling discovery", + name); + } + } + + for (const auto & pending : pending_zk_unregisters) + { + if (!unregisterFromZk(pending.zk_name, pending.zk_root, pending.cluster_name)) + { + LOG_WARNING( + log, + "Failed to complete pending unregister for cluster '{}' while disabling discovery", + pending.cluster_name); + } + } + pending_zk_unregisters.clear(); + + clusters_info.clear(); + multicluster_discovery_paths.clear(); + get_nodes_callbacks.clear(); + register_change_flag.store(RegisterChangeFlag::RCF_NONE); + + { + std::lock_guard lock(pending_config_mutex); + pending_config_update.reset(); + } + { + std::lock_guard lock(mutex); + cluster_impls.clear(); + } +} + ClusterDiscovery::~ClusterDiscovery() { try diff --git a/src/Interpreters/ClusterDiscovery.h b/src/Interpreters/ClusterDiscovery.h index 2bea12d9f1e0..06bfb815d57c 100644 --- a/src/Interpreters/ClusterDiscovery.h +++ b/src/Interpreters/ClusterDiscovery.h @@ -9,7 +9,10 @@ #include +#include #include +#include +#include namespace DB { @@ -33,11 +36,31 @@ class ClusterDiscovery void start(); + /// Apply changes from reloaded remote_servers config (credentials, add/remove discovery paths). + /// Safe to call from the config-reloader thread; the update is applied on the discovery worker. + void updateFromConfig( + const Poco::Util::AbstractConfiguration & config, + const String & config_prefix = "remote_servers"); + + /// Throws if discovery subtrees under `config_prefix` are invalid. No side effects. + /// Call before committing Clusters / clusters_config so a bad reload cannot partially apply. + static void validateConfig( + const Poco::Util::AbstractConfiguration & config, + ContextPtr context, + const String & config_prefix = "remote_servers"); + ClusterPtr getCluster(const String & cluster_name) const; std::unordered_map getClusters() const; + /// Stop the worker, remove participant ephemerals, and drop published clusters. + /// Used when allow_experimental_cluster_discovery is turned off on reload. + void disableAndShutdown(); + ~ClusterDiscovery(); + void registerAll(); + void unregisterAll(); + private: struct NodeInfo { @@ -89,9 +112,11 @@ class ClusterDiscovery String password; String cluster_secret; - /// For dynamic clusters, index+1 in multicluster_discovery_paths where cluster was found - /// 0 for static clusters - size_t zk_root_index; + /// For dynamic clusters: MulticlusterDiscovery::getFullPath() where cluster was found. + /// Empty for static clusters defined with . + String multicluster_full_path; + + bool isDynamic() const { return !multicluster_full_path.empty(); } ClusterInfo(const String & name_, const String & zk_name_, @@ -105,20 +130,128 @@ class ClusterDiscovery size_t shard_id, bool observer_mode, bool invisible, - size_t zk_root_index_ = 0 + const String & multicluster_full_path_ = {} ); }; + struct ParsedStaticDiscovery + { + String name; + String zk_name; + String zk_root; + String host_name; + String username; + String password; + String cluster_secret; + bool secure = false; + size_t shard_id = 0; + bool observer = false; + bool invisible = false; + }; + + struct ParsedMulticlusterDiscovery + { + String zk_name; + String zk_path; + bool is_secure_connection = false; + String username; + String password; + String cluster_secret; + + String getFullPath() const { return zk_name + ":" + zk_path; } + }; + + struct ParsedDiscoveryConfig + { + std::vector static_clusters; + std::vector multicluster_roots; + }; + + struct MulticlusterDiscovery + { + const String zk_name; + const String zk_path; + bool is_secure_connection; + String username; + String password; + String cluster_secret; + + mutable Stopwatch watch; + mutable std::shared_ptr need_update; + Coordination::WatchCallbackPtr watch_callback; + + MulticlusterDiscovery(const String & zk_name_, + const String & zk_path_, + bool is_secure_connection_, + const String & username_, + const String & password_, + const String & cluster_secret_) + : zk_name(zk_name_) + , zk_path(zk_path_) + , is_secure_connection(is_secure_connection_) + , username(username_) + , password(password_) + , cluster_secret(cluster_secret_) + , need_update(std::make_shared(true)) + {} + + String getFullPath() const { return zk_name + ":" + zk_path; } + }; + + static ParsedDiscoveryConfig parseDiscoveryConfig( + const Poco::Util::AbstractConfiguration & config, + ContextPtr context, + const String & config_prefix); + + void applyParsedConfig(ParsedDiscoveryConfig && parsed); + void addStaticCluster(ParsedStaticDiscovery && parsed); + void removeStaticCluster(const String & name); + /// Remove a multicluster-discovered cluster so a static config entry can take its name. + void removeDynamicCluster(const String & name); + bool updateStaticClusterFields(ClusterInfo & info, const ParsedStaticDiscovery & parsed); + void addMulticlusterRoot(ParsedMulticlusterDiscovery && parsed); + void removeMulticlusterRoot(const String & full_path); + bool updateMulticlusterRootFields(MulticlusterDiscovery & path, const ParsedMulticlusterDiscovery & parsed); + /// Force findDynamicClusters to rescan roots (e.g. after a static name stops shadowing). + void markMulticlusterRootsNeedUpdate(); + + void rebuildClusterObject(const ClusterInfo & info); + void ensureWorkerStarted(); + bool consumePendingConfigUpdate(); + + /// Assumes start_mutex is held. Starts the worker at most once. + void startImpl(); + void initialUpdate(); void registerInZk(zkutil::ZooKeeperPtr & zk, ClusterInfo & info); + void unregisterFromZk(zkutil::ZooKeeperPtr & zk, ClusterInfo & info); + + struct PendingZkUnregister + { + String zk_name; + String zk_root; + String cluster_name; + }; + + /// Returns false if Keeper remove failed; caller should queue a retry. + bool unregisterFromZk(const String & zk_name, const String & zk_root, const String & cluster_name); + /// True if a non-observer entry still owns this Keeper registration path. + /// When removing `exclude_cluster_name`, pass it so the cluster being dropped is ignored. + bool pathHasActiveParticipant( + const String & zk_name, + const String & zk_root, + const String * exclude_cluster_name = nullptr) const; + /// Retries failed unregisters. Returns true when the queue is empty. + /// Drops pending entries whose path already has an active participant again. + bool retryPendingUnregisters(); Strings getNodeNames(zkutil::ZooKeeperPtr & zk, const String & zk_root, const String & cluster_name, int * version, bool set_callback, - size_t zk_root_index); + const String & multicluster_full_path); NodesInfo getNodes(zkutil::ZooKeeperPtr & zk, const String & zk_root, const Strings & node_uuids); @@ -131,9 +264,12 @@ class ClusterDiscovery bool runMainThread(std::function up_to_date_callback); void shutdown(); - void findDynamicClusters(std::unordered_map & info, std::unordered_set * unchanged_roots = nullptr); + void findDynamicClusters( + std::unordered_map & info, + std::unordered_set * unchanged_roots = nullptr); /// cluster name -> cluster info (zk root, set of nodes) + /// Mutated only from constructor (before start) and the discovery worker thread. std::unordered_map clusters_info; ContextMutablePtr context; @@ -156,44 +292,39 @@ class ClusterDiscovery std::unordered_map cluster_impls; bool is_initialized = false; + + /// Serializes start() / ensureWorkerStarted() so concurrent config reload and + /// startClusterDiscovery cannot double-assign main_thread (ThreadFromGlobalPool aborts). + /// Lock order: start_mutex before pending_config_mutex. + mutable std::mutex start_mutex; ThreadFromGlobalPool main_thread; LoggerPtr log; - struct MulticlusterDiscovery - { - const String zk_name; - const String zk_path; - bool is_secure_connection; - String username; - String password; - String cluster_secret; + /// Keyed by MulticlusterDiscovery::getFullPath() + std::unordered_map multicluster_discovery_paths; - mutable Stopwatch watch; - mutable std::shared_ptr need_update; - Coordination::WatchCallbackPtr watch_callback; + /// Config reload posts parsed config here; worker applies it. + /// Never take this lock while a caller without start_mutex may later take start_mutex + /// while holding this one: lock order is start_mutex -> pending_config_mutex. + mutable std::mutex pending_config_mutex; + std::optional pending_config_update; - MulticlusterDiscovery(const String & zk_name_, - const String & zk_path_, - bool is_secure_connection_, - const String & username_, - const String & password_, - const String & cluster_secret_) - : zk_name(zk_name_) - , zk_path(zk_path_) - , is_secure_connection(is_secure_connection_) - , username(username_) - , password(password_) - , cluster_secret(cluster_secret_) - , need_update(std::make_shared(true)) - {} + /// Ephemeral registrations that failed to remove during config apply. + /// Local cluster state is already dropped; retry from the worker thread. + /// Accessed only from the discovery worker (same as clusters_info). + std::vector pending_zk_unregisters; - String getFullPath() const { return zk_name + ":" + zk_path; } - }; + MultiVersion::Version macros; - std::vector multicluster_discovery_paths; + enum RegisterChangeFlag + { + RCF_NONE, + RCF_REGISTER_ALL, + RCF_UNREGISTER_ALL, + }; - MultiVersion::Version macros; + std::atomic register_change_flag = RegisterChangeFlag::RCF_NONE; }; } diff --git a/src/Interpreters/ClusterFunctionReadTask.cpp b/src/Interpreters/ClusterFunctionReadTask.cpp index f8d57daccaf2..b3c6074f00d3 100644 --- a/src/Interpreters/ClusterFunctionReadTask.cpp +++ b/src/Interpreters/ClusterFunctionReadTask.cpp @@ -41,9 +41,18 @@ ClusterFunctionReadTaskResponse::ClusterFunctionReadTaskResponse(ObjectInfoPtr o } #endif - const bool send_over_whole_archive = !context->getSettingsRef()[Setting::cluster_function_process_archive_on_multiple_nodes]; - path = send_over_whole_archive ? object->getPathOrPathToArchiveIfArchive() : object->getPath(); + file_meta_info = object->relative_path_with_metadata.file_meta_info; + + if (object->relative_path_with_metadata.getCommand().isValid()) + path = object->relative_path_with_metadata.getCommand().toString(); + else + { + const bool send_over_whole_archive = !context->getSettingsRef()[Setting::cluster_function_process_archive_on_multiple_nodes]; + path = send_over_whole_archive ? object->getPathOrPathToArchiveIfArchive() : object->getPath(); + } + read_source_index = object->relative_path_with_metadata.read_source_index; + file_bucket_info = object->file_bucket_info; } @@ -76,6 +85,8 @@ ObjectInfoPtr ClusterFunctionReadTaskResponse::getObjectInfo() const object->relative_path_with_metadata.read_source_index = read_source_index; object->data_lake_metadata = data_lake_metadata; object->file_bucket_info = file_bucket_info; + if (file_meta_info.has_value()) + object->relative_path_with_metadata.file_meta_info = file_meta_info; return object; } diff --git a/src/Interpreters/ClusterFunctionReadTask.h b/src/Interpreters/ClusterFunctionReadTask.h index e092d4397004..78649e071483 100644 --- a/src/Interpreters/ClusterFunctionReadTask.h +++ b/src/Interpreters/ClusterFunctionReadTask.h @@ -25,6 +25,8 @@ struct ClusterFunctionReadTaskResponse DataLakeObjectMetadata data_lake_metadata; /// Iceberg object metadata std::optional iceberg_info; + /// File's columns info + std::optional file_meta_info; /// Convert received response into ObjectInfo. ObjectInfoPtr getObjectInfo() const; diff --git a/src/Interpreters/ClusterProxy/executeQuery.cpp b/src/Interpreters/ClusterProxy/executeQuery.cpp index 7fa2d6fadecb..2dc5c9b83767 100644 --- a/src/Interpreters/ClusterProxy/executeQuery.cpp +++ b/src/Interpreters/ClusterProxy/executeQuery.cpp @@ -695,6 +695,13 @@ void executeQuery( std::move(unavailable_shard_tracker)); read_from_remote->setStepDescription("Read from remote replica"); + /// The remote node is allowed to act as an initiator (and distribute the query further, e.g. over a swarm) + /// only when `remote()` wraps a table function, like `remote(host, s3Cluster(...))`. + /// For `remote(host, db, table)` the remote node must stay a secondary query: for intermediate processing + /// stages it returns a header made of internal column identifiers which the initiator matches by name, + /// and query tree optimizations enabled for initial queries change those names (see + /// 04045_merge_function_missing_columns_remote). + read_from_remote->setIsRemoteFunction(is_remote_function && table_func_ptr != nullptr); plan->addStep(std::move(read_from_remote)); plan->addInterpreterContext(new_context); plans.emplace_back(std::move(plan)); diff --git a/src/Interpreters/Context.cpp b/src/Interpreters/Context.cpp index 5ea14c9289cb..8ebdb2463295 100644 --- a/src/Interpreters/Context.cpp +++ b/src/Interpreters/Context.cpp @@ -49,6 +49,7 @@ #include #include #include +#include #include #include #include @@ -187,6 +188,8 @@ namespace ProfileEvents extern const Event BackupThrottlerSleepMicroseconds; extern const Event MergesThrottlerBytes; extern const Event MergesThrottlerSleepMicroseconds; + extern const Event ExportsThrottlerBytes; + extern const Event ExportsThrottlerSleepMicroseconds; extern const Event MutationsThrottlerBytes; extern const Event MutationsThrottlerSleepMicroseconds; extern const Event QueryLocalReadThrottlerBytes; @@ -288,6 +291,7 @@ namespace CurrentMetrics extern const Metric IndexUncompressedCacheCells; extern const Metric ZooKeeperSessionExpired; extern const Metric ZooKeeperConnectionLossStartedTimestampSeconds; + extern const Metric IsSwarmModeEnabled; } @@ -424,6 +428,7 @@ namespace ServerSetting extern const ServerSettingsUInt64 max_local_write_bandwidth_for_server; extern const ServerSettingsUInt64 max_merges_bandwidth_for_server; extern const ServerSettingsUInt64 max_mutations_bandwidth_for_server; + extern const ServerSettingsUInt64 max_exports_bandwidth_for_server; extern const ServerSettingsUInt64 max_remote_read_network_bandwidth_for_server; extern const ServerSettingsUInt64 max_remote_write_network_bandwidth_for_server; extern const ServerSettingsUInt64 max_replicated_fetches_network_bandwidth_for_server; @@ -648,6 +653,7 @@ struct ContextSharedPart : boost::noncopyable GlobalOvercommitTracker global_overcommit_tracker; MergeList merge_list; /// The list of executable merge (for (Replicated)?MergeTree) MovesList moves_list; /// The list of executing moves (for (Replicated)?MergeTree) + ExportsList exports_list; /// The list of executing exports (for (Replicated)?MergeTree) ReplicatedFetchList replicated_fetch_list; RefreshSet refresh_set; /// The list of active refreshes (for MaterializedView) ConfigurationPtr users_config TSA_GUARDED_BY(mutex); /// Config with the users, profiles and quotas sections. @@ -700,6 +706,8 @@ struct ContextSharedPart : boost::noncopyable mutable ThrottlerPtr distributed_cache_read_throttler; /// A server-wide throttler for distributed cache read mutable ThrottlerPtr distributed_cache_write_throttler; /// A server-wide throttler for distributed cache write + mutable ThrottlerPtr exports_throttler; /// A server-wide throttler for exports + MultiVersion macros; /// Substitutions extracted from config. std::unique_ptr ddl_worker TSA_GUARDED_BY(mutex); /// Process ddl commands from zk. LoadTaskPtr ddl_worker_startup_task; /// To postpone `ddl_worker->startup()` after all tables startup @@ -803,6 +811,7 @@ struct ContextSharedPart : boost::noncopyable std::map server_ports TSA_GUARDED_BY(server_ports_mutex); std::atomic shutdown_called = false; + std::atomic swarm_mode_enabled = true; Stopwatch uptime_watch TSA_GUARDED_BY(mutex); @@ -1001,6 +1010,8 @@ struct ContextSharedPart : boost::noncopyable */ void shutdown() TSA_NO_THREAD_SAFETY_ANALYSIS { + swarm_mode_enabled = false; + CurrentMetrics::set(CurrentMetrics::IsSwarmModeEnabled, 0); bool is_shutdown_called = shutdown_called.exchange(true); if (is_shutdown_called) return; @@ -1336,6 +1347,9 @@ struct ContextSharedPart : boost::noncopyable if (auto bandwidth = server_settings[ServerSetting::max_merges_bandwidth_for_server]) merges_throttler = std::make_shared(bandwidth, ProfileEvents::MergesThrottlerBytes, ProfileEvents::MergesThrottlerSleepMicroseconds); + + if (auto bandwidth = server_settings[ServerSetting::max_exports_bandwidth_for_server]) + exports_throttler = std::make_shared(bandwidth, ProfileEvents::ExportsThrottlerBytes, ProfileEvents::ExportsThrottlerSleepMicroseconds); } }; @@ -1512,6 +1526,8 @@ MergeList & Context::getMergeList() { return shared->merge_list; } const MergeList & Context::getMergeList() const { return shared->merge_list; } MovesList & Context::getMovesList() { return shared->moves_list; } const MovesList & Context::getMovesList() const { return shared->moves_list; } +ExportsList & Context::getExportsList() { return shared->exports_list; } +const ExportsList & Context::getExportsList() const { return shared->exports_list; } ReplicatedFetchList & Context::getReplicatedFetchList() { return shared->replicated_fetch_list; } const ReplicatedFetchList & Context::getReplicatedFetchList() const { return shared->replicated_fetch_list; } RefreshSet & Context::getRefreshSet() { return shared->refresh_set; } @@ -3729,8 +3745,11 @@ void Context::setCurrentQueryId(const String & query_id) client_info.current_query_id = query_id_to_set; - if (client_info.query_kind == ClientInfo::QueryKind::INITIAL_QUERY) + if (client_info.query_kind == ClientInfo::QueryKind::INITIAL_QUERY + && (getApplicationType() != ApplicationType::SERVER || client_info.initial_query_id.empty())) + { client_info.initial_query_id = client_info.current_query_id; + } } void Context::killCurrentQuery() const @@ -3912,6 +3931,13 @@ void Context::makeQueryContextForMutate(const MergeTreeSettings & merge_tree_set = merge_tree_settings[MergeTreeSetting::mutation_workload].value.empty() ? getMutationWorkload() : merge_tree_settings[MergeTreeSetting::mutation_workload]; } +void Context::makeQueryContextForExportPart() +{ + makeQueryContext(); + classifier.reset(); // It is assumed that there are no active queries running using this classifier, otherwise this will lead to crashes + // Export part operations don't have a specific workload setting, so we leave the default workload +} + void Context::makeSessionContext() { session_context = shared_from_this(); @@ -5844,6 +5870,11 @@ ThrottlerPtr Context::getDistributedCacheWriteThrottler() const return shared->distributed_cache_write_throttler; } +ThrottlerPtr Context::getExportsThrottler() const +{ + return shared->exports_throttler; +} + void Context::reloadRemoteThrottlerConfig(size_t read_bandwidth, size_t write_bandwidth) const { if (read_bandwidth) @@ -6666,7 +6697,6 @@ std::shared_ptr Context::getCluster(const std::string & cluster_name) c throw Exception(ErrorCodes::CLUSTER_DOESNT_EXIST, "Requested cluster '{}' not found", cluster_name); } - std::shared_ptr Context::tryGetCluster(const std::string & cluster_name) const { std::shared_ptr res = nullptr; @@ -6685,6 +6715,21 @@ std::shared_ptr Context::tryGetCluster(const std::string & cluster_name return res; } +void Context::unregisterInAutodiscoveryClusters() +{ + std::lock_guard lock(shared->clusters_mutex); + if (!shared->cluster_discovery) + return; + shared->cluster_discovery->unregisterAll(); +} + +void Context::registerInAutodiscoveryClusters() +{ + std::lock_guard lock(shared->clusters_mutex); + if (!shared->cluster_discovery) + return; + shared->cluster_discovery->registerAll(); +} void Context::reloadClusterConfig() const { @@ -6761,12 +6806,12 @@ void Context::startClusterDiscovery() /// On repeating calls updates existing clusters and adds new clusters, doesn't delete old clusters void Context::setClustersConfig(const ConfigurationPtr & config, bool enable_discovery, const String & config_name) { + ClusterDiscovery * discovery_to_update = nullptr; + ClusterDiscovery * discovery_just_created_ptr = nullptr; + std::unique_ptr discovery_to_disable; + bool clusters_changed = false; { std::lock_guard lock(shared->clusters_mutex); - if (ConfigHelper::getBool(*config, "allow_experimental_cluster_discovery") && enable_discovery && !shared->cluster_discovery) - { - shared->cluster_discovery = std::make_unique(*config, getGlobalContext(), getMacros()); - } /// Do not update clusters if this part of config wasn't changed. /// Note: clusters_config must be checked for null separately from clusters, because @@ -6774,19 +6819,76 @@ void Context::setClustersConfig(const ConfigurationPtr & config, bool enable_dis /// shared->clusters using the fallback getConfigRef() without setting shared->clusters_config. /// If setClustersConfig() then runs before the config reloader stores its ConfigurationPtr, /// dereferencing shared->clusters_config would throw Poco::NullPointerException. - if (shared->clusters && shared->clusters_config && isSameConfiguration(*config, *shared->clusters_config, config_name)) - return; + /// + /// Still start a discovery object created after server start when only the allow-flag + /// flipped (remote_servers subtree unchanged) — otherwise the worker never runs. + /// The reverse transition (allow 1 -> 0) must tear discovery down even when remote_servers + /// is unchanged; otherwise the worker stays registered until restart. + const bool remote_servers_unchanged + = shared->clusters && shared->clusters_config + && isSameConfiguration(*config, *shared->clusters_config, config_name); + + const bool discovery_enabled + = ConfigHelper::getBool(*config, "allow_experimental_cluster_discovery") && enable_discovery; + + /// Validate discovery before creating the object or committing Clusters so a bad reload + /// cannot leave clusters_config advanced while discovery stays on the previous view. + /// Also validate when allow is turned off: an existing ClusterDiscovery is still updated. + if (!remote_servers_unchanged && (discovery_enabled || shared->cluster_discovery)) + ClusterDiscovery::validateConfig(*config, getGlobalContext(), config_name); + + bool discovery_just_created = false; + if (discovery_enabled) + { + if (!shared->cluster_discovery) + { + shared->cluster_discovery = std::make_unique(*config, getGlobalContext(), getMacros()); + discovery_just_created = true; + } + } + else if (shared->cluster_discovery) + { + discovery_to_disable = std::move(shared->cluster_discovery); + } - auto old_clusters_config = shared->clusters_config; - shared->clusters_config = config; + if (!remote_servers_unchanged) + { + auto old_clusters_config = shared->clusters_config; + shared->clusters_config = config; - if (!shared->clusters) - shared->clusters = std::make_shared(*shared->clusters_config, *settings, getMacros(), config_name); - else - shared->clusters->updateClusters(*shared->clusters_config, *settings, config_name, old_clusters_config); + if (!shared->clusters) + shared->clusters = std::make_shared(*shared->clusters_config, *settings, getMacros(), config_name); + else + shared->clusters->updateClusters(*shared->clusters_config, *settings, config_name, old_clusters_config); + + if (shared->cluster_discovery && !discovery_just_created) + discovery_to_update = shared->cluster_discovery.get(); + + ++shared->clusters_version; + clusters_changed = true; + } - ++shared->clusters_version; + /// Constructor already applied config. Start outside this lock if the server is ready; + /// otherwise programs/server/Server.cpp calls startClusterDiscovery() after listen. + if (discovery_just_created) + discovery_just_created_ptr = shared->cluster_discovery.get(); } + + /// Tear down outside clusters_mutex: joins the worker and may touch ZooKeeper. + if (discovery_to_disable) + discovery_to_disable->disableAndShutdown(); + + /// Apply discovery updates outside clusters_mutex: may start the worker and touch ZooKeeper. + if (discovery_to_update) + discovery_to_update->updateFromConfig(*config, config_name); + + /// Re-check server readiness without clusters_mutex (isServerCompletelyStarted takes shared->mutex). + if (discovery_just_created_ptr && getApplicationType() == ApplicationType::SERVER && isServerCompletelyStarted()) + discovery_just_created_ptr->start(); + + /// Avoid DDL host-id refresh / log noise when remote_servers (and discovery) did not change. + /// Still notify when discovery was just created or disabled (e.g. allow-flag-only reload). + if (clusters_changed || discovery_to_update || discovery_just_created_ptr || discovery_to_disable) { SharedLockGuard lock(shared->mutex); if (shared->ddl_worker) @@ -7730,12 +7832,35 @@ void Context::stopServers(const ServerType & server_type) const shared->stop_servers_callback(server_type); } - void Context::shutdown() TSA_NO_THREAD_SAFETY_ANALYSIS { shared->shutdown(); } +bool Context::stopSwarmMode() +{ + bool expected_is_enabled = true; + bool is_stopped_now = shared->swarm_mode_enabled.compare_exchange_strong(expected_is_enabled, false); + if (is_stopped_now) + CurrentMetrics::set(CurrentMetrics::IsSwarmModeEnabled, 0); + // return true if stop successful + return is_stopped_now; +} + +bool Context::startSwarmMode() +{ + bool expected_is_enabled = false; + bool is_started_now = shared->swarm_mode_enabled.compare_exchange_strong(expected_is_enabled, true); + if (is_started_now) + CurrentMetrics::set(CurrentMetrics::IsSwarmModeEnabled, 1); + // return true if start successful + return is_started_now; +} + +bool Context::isSwarmModeEnabled() const +{ + return shared->swarm_mode_enabled; +} Context::ApplicationType Context::getApplicationType() const { diff --git a/src/Interpreters/Context.h b/src/Interpreters/Context.h index c66f26bd53fc..e0fc54ccb600 100644 --- a/src/Interpreters/Context.h +++ b/src/Interpreters/Context.h @@ -97,6 +97,7 @@ class InterserverIOHandler; class AsynchronousMetrics; class MergeList; class MovesList; +class ExportsList; class ReplicatedFetchList; class RefreshSet; class Cluster; @@ -1422,6 +1423,7 @@ class Context: public ContextData, public std::enable_shared_from_this void makeQueryContext(); void makeQueryContextForMerge(const MergeTreeSettings & merge_tree_settings); void makeQueryContextForMutate(const MergeTreeSettings & merge_tree_settings); + void makeQueryContextForExportPart(); void makeSessionContext(); void makeGlobalContext(); void makeBackgroundContext(const Poco::Util::AbstractConfiguration & config); @@ -1463,6 +1465,9 @@ class Context: public ContextData, public std::enable_shared_from_this MovesList & getMovesList(); const MovesList & getMovesList() const; + ExportsList & getExportsList(); + const ExportsList & getExportsList() const; + ReplicatedFetchList & getReplicatedFetchList(); const ReplicatedFetchList & getReplicatedFetchList() const; @@ -1706,6 +1711,8 @@ class Context: public ContextData, public std::enable_shared_from_this size_t getClustersVersion() const; void startClusterDiscovery(); + void registerInAutodiscoveryClusters(); + void unregisterInAutodiscoveryClusters(); /// Sets custom cluster, but doesn't update configuration void setCluster(const String & cluster_name, const std::shared_ptr & cluster); @@ -1842,6 +1849,15 @@ class Context: public ContextData, public std::enable_shared_from_this void shutdown(); + /// Stop some works to allow graceful shutdown later. + /// Returns true if stop successful. + bool stopSwarmMode(); + /// Resume some works if we change our mind. + /// Returns true if start successful. + bool startSwarmMode(); + /// Return current swarm mode state. + bool isSwarmModeEnabled() const; + bool isInternalQuery() const { return is_internal_query; } void setInternalQuery(bool internal) { is_internal_query = internal; } @@ -2129,6 +2145,7 @@ class Context: public ContextData, public std::enable_shared_from_this ThrottlerPtr getMutationsThrottler() const; ThrottlerPtr getMergesThrottler() const; + ThrottlerPtr getExportsThrottler() const; ThrottlerPtr getDistributedCacheReadThrottler() const; ThrottlerPtr getDistributedCacheWriteThrottler() const; diff --git a/src/Interpreters/DDLWorker.cpp b/src/Interpreters/DDLWorker.cpp index 854fbc82ba7d..5df18840c6b7 100644 --- a/src/Interpreters/DDLWorker.cpp +++ b/src/Interpreters/DDLWorker.cpp @@ -840,7 +840,8 @@ bool DDLWorker::taskShouldBeExecutedOnLeader(const ASTPtr & ast_ddl, const Stora alter->isUnlockSnapshot() || alter->isMovePartitionToDiskOrVolumeAlter() || alter->isCommentAlter() || - alter->isSettingsOrCommentAlter()) + alter->isSettingsOrCommentAlter() || + alter->isExportPartOrExportPartitionAlter()) return false; } diff --git a/src/Interpreters/IcebergMetadataLog.cpp b/src/Interpreters/IcebergMetadataLog.cpp index 637091ae4aac..170dc0f8d7fa 100644 --- a/src/Interpreters/IcebergMetadataLog.cpp +++ b/src/Interpreters/IcebergMetadataLog.cpp @@ -12,6 +12,7 @@ #include #include #include +#include #include #include #include diff --git a/src/Interpreters/InterpreterAlterQuery.cpp b/src/Interpreters/InterpreterAlterQuery.cpp index ed545cf51aca..86c7089c6b63 100644 --- a/src/Interpreters/InterpreterAlterQuery.cpp +++ b/src/Interpreters/InterpreterAlterQuery.cpp @@ -836,6 +836,20 @@ AccessRightsElements InterpreterAlterQuery::getRequiredAccessForCommand( required_access.emplace_back(AccessType::ALTER_DELETE | AccessType::INSERT, database, table); break; } + case ASTAlterCommand::EXPORT_PART: + { + required_access.emplace_back(AccessType::ALTER_EXPORT_PART, database, table); + /// For table functions, access control is handled by the table function itself + if (!command.to_table_function) + required_access.emplace_back(AccessType::INSERT, command.to_database, command.to_table); + break; + } + case ASTAlterCommand::EXPORT_PARTITION: + { + required_access.emplace_back(AccessType::ALTER_EXPORT_PARTITION, database, table); + required_access.emplace_back(AccessType::INSERT, command.to_database, command.to_table); + break; + } case ASTAlterCommand::FETCH_PARTITION: { required_access.emplace_back(AccessType::ALTER_FETCH_PARTITION, database, table); diff --git a/src/Interpreters/InterpreterCreateQuery.cpp b/src/Interpreters/InterpreterCreateQuery.cpp index 73a9fd4bd051..f955487b1dd7 100644 --- a/src/Interpreters/InterpreterCreateQuery.cpp +++ b/src/Interpreters/InterpreterCreateQuery.cpp @@ -2405,8 +2405,7 @@ bool InterpreterCreateQuery::doCreateTable(ASTCreateQuery & create, auto table_function_ast = create.as_table_function->ptr(); auto table_function = TableFunctionFactory::instance().get(table_function_ast, getContext()); - if (!table_function->canBeUsedToCreateTable()) - throw Exception(ErrorCodes::BAD_ARGUMENTS, "Table function '{}' cannot be used to create a table", table_function->getName()); + table_function->validateUseToCreateTable(); /// In case of CREATE AS table_function() query we should use global context /// in storage creation because there will be no query context on server startup diff --git a/src/Interpreters/InterpreterInsertQuery.cpp b/src/Interpreters/InterpreterInsertQuery.cpp index afbd4a86e552..861f2265d3ff 100644 --- a/src/Interpreters/InterpreterInsertQuery.cpp +++ b/src/Interpreters/InterpreterInsertQuery.cpp @@ -1092,6 +1092,9 @@ std::optional InterpreterInsertQuery::distributedWriteIntoReplica if (!src_storage_cluster) return {}; + if (src_storage_cluster->getClusterName(local_context).empty()) + return {}; + if (!isInsertSelectTrivialEnoughForDistributedExecution(query)) return {}; diff --git a/src/Interpreters/InterpreterKillQueryQuery.cpp b/src/Interpreters/InterpreterKillQueryQuery.cpp index 177ca1a051c0..06e06f5e6054 100644 --- a/src/Interpreters/InterpreterKillQueryQuery.cpp +++ b/src/Interpreters/InterpreterKillQueryQuery.cpp @@ -19,6 +19,7 @@ #include #include #include +#include #include #include #include @@ -37,10 +38,16 @@ namespace Setting extern const SettingsUInt64 max_parser_depth; } +namespace ServerSetting +{ + extern const ServerSettingsBool allow_experimental_export_merge_tree_partition; +} + namespace ErrorCodes { extern const int ACCESS_DENIED; extern const int NOT_IMPLEMENTED; + extern const int SUPPORT_IS_DISABLED; } @@ -249,6 +256,83 @@ BlockIO InterpreterKillQueryQuery::execute() break; } + case ASTKillQueryQuery::Type::ExportPartition: + { + if (!getContext()->getServerSettings()[ServerSetting::allow_experimental_export_merge_tree_partition]) + { + throw Exception(ErrorCodes::SUPPORT_IS_DISABLED, + "Exporting merge tree partition is experimental. Set the server setting `allow_experimental_export_merge_tree_partition` to enable it"); + } + + Block exports_block = getSelectResult( + "source_database, source_table, transaction_id, destination_database, destination_table, partition_id", + "system.replicated_partition_exports"); + if (exports_block.empty()) + return res_io; + + const ColumnString & src_db_col = typeid_cast(*exports_block.getByName("source_database").column); + const ColumnString & src_table_col = typeid_cast(*exports_block.getByName("source_table").column); + const ColumnString & dst_db_col = typeid_cast(*exports_block.getByName("destination_database").column); + const ColumnString & dst_table_col = typeid_cast(*exports_block.getByName("destination_table").column); + const ColumnString & tx_col = typeid_cast(*exports_block.getByName("transaction_id").column); + + auto header = exports_block.cloneEmpty(); + header.insert(0, {ColumnString::create(), std::make_shared(), "kill_status"}); + + MutableColumns res_columns = header.cloneEmptyColumns(); + AccessRightsElements required_access_rights; + auto access = getContext()->getAccess(); + bool access_denied = false; + + for (size_t i = 0; i < exports_block.rows(); ++i) + { + const auto src_database = src_db_col.getDataAt(i); + const auto src_table = src_table_col.getDataAt(i); + const auto dst_database = dst_db_col.getDataAt(i); + const auto dst_table = dst_table_col.getDataAt(i); + + const auto table_id = StorageID{std::string{src_database}, std::string{src_table}}; + const auto transaction_id = tx_col.getDataAt(i); + + CancellationCode code = CancellationCode::Unknown; + if (!query.test) + { + auto storage = DatabaseCatalog::instance().tryGetTable(table_id, getContext()); + if (!storage) + code = CancellationCode::NotFound; + else + { + ASTAlterCommand alter_command{}; + alter_command.type = ASTAlterCommand::EXPORT_PARTITION; + alter_command.move_destination_type = DataDestinationType::TABLE; + alter_command.from_database = src_database; + alter_command.from_table = src_table; + alter_command.to_database = dst_database; + alter_command.to_table = dst_table; + + required_access_rights = InterpreterAlterQuery::getRequiredAccessForCommand( + alter_command, table_id.database_name, table_id.table_name, + InterpreterAlterQuery::isRowExistsLightweightDeleteMarker(storage, getContext())); + if (!access->isGranted(required_access_rights)) + { + access_denied = true; + continue; + } + code = storage->killExportPartition(std::string{transaction_id}); + } + } + + insertResultRow(i, code, exports_block, header, res_columns); + } + + if (res_columns[0]->empty() && access_denied) + throw Exception(ErrorCodes::ACCESS_DENIED, "Not allowed to kill export partition. " + "To execute this query, it's necessary to have the grant {}", required_access_rights.toString()); + + res_io.pipeline = QueryPipeline(Pipe(std::make_shared(std::make_shared(header.cloneWithColumns(std::move(res_columns)))))); + + break; + } case ASTKillQueryQuery::Type::Mutation: { Block mutations_block = getSelectResult("database, table, mutation_id, command", "system.mutations"); @@ -485,6 +569,9 @@ AccessRightsElements InterpreterKillQueryQuery::getRequiredAccessForDDLOnCluster | AccessType::ALTER_MATERIALIZE_TTL | AccessType::ALTER_REWRITE_PARTS ); + /// todo arthur think about this + else if (query.type == ASTKillQueryQuery::Type::ExportPartition) + required_access.emplace_back(AccessType::ALTER_EXPORT_PARTITION); return required_access; } diff --git a/src/Interpreters/InterpreterOptimizeQuery.cpp b/src/Interpreters/InterpreterOptimizeQuery.cpp index 756b35a9727e..7d9fe8936269 100644 --- a/src/Interpreters/InterpreterOptimizeQuery.cpp +++ b/src/Interpreters/InterpreterOptimizeQuery.cpp @@ -13,6 +13,7 @@ #include #include #include +#include #if USE_AVRO #include @@ -59,15 +60,26 @@ BlockIO InterpreterOptimizeQuery::execute() throw Exception(ErrorCodes::BAD_ARGUMENTS, "OPTIMIZE MANIFEST is incompatible with FINAL, PARTITION, DEDUPLICATE, CLEANUP, and DRY RUN options"); #if USE_AVRO + /// Object storage engines are created as StorageObjectStorageCluster, which wraps a plain + /// StorageObjectStorage, so both have to be accepted here. auto * object_storage_table = dynamic_cast(table.get()); - if (!object_storage_table) + auto * object_storage_cluster_table = dynamic_cast(table.get()); + if (!object_storage_table && !object_storage_cluster_table) throw Exception(ErrorCodes::NOT_IMPLEMENTED, "OPTIMIZE MANIFEST is only supported for Iceberg tables"); - auto iceberg_metadata = std::dynamic_pointer_cast(object_storage_table->getExternalMetadata(getContext())); + auto external_metadata = object_storage_table + ? object_storage_table->getExternalMetadata(getContext()) + : object_storage_cluster_table->getExternalMetadata(getContext()); + + auto iceberg_metadata = std::dynamic_pointer_cast(external_metadata); if (!iceberg_metadata) throw Exception(ErrorCodes::NOT_IMPLEMENTED, "OPTIMIZE MANIFEST is only supported for Iceberg tables"); - iceberg_metadata->optimizeManifestFiles(metadata_snapshot, getContext(), object_storage_table->getCatalog(), table_id); + auto catalog = object_storage_table + ? object_storage_table->getCatalog() + : object_storage_cluster_table->getCatalog(); + + iceberg_metadata->optimizeManifestFiles(metadata_snapshot, getContext(), catalog, table_id); return {}; #else throw Exception(ErrorCodes::NOT_IMPLEMENTED, "OPTIMIZE MANIFEST is only supported for Iceberg tables"); diff --git a/src/Interpreters/InterpreterSystemQuery.cpp b/src/Interpreters/InterpreterSystemQuery.cpp index 89f8e0409994..e070cd3aafcf 100644 --- a/src/Interpreters/InterpreterSystemQuery.cpp +++ b/src/Interpreters/InterpreterSystemQuery.cpp @@ -59,6 +59,7 @@ #include #include #include +#include #include #include #include @@ -606,7 +607,12 @@ BlockIO InterpreterSystemQuery::execute() #else throw Exception(ErrorCodes::SUPPORT_IS_DISABLED, "The server was compiled without the support for AWS S3"); #endif - + case Type::DROP_OBJECT_STORAGE_LIST_OBJECTS_CACHE: + { + getContext()->checkAccess(AccessType::SYSTEM_DROP_OBJECT_STORAGE_LIST_OBJECTS_CACHE); + ObjectStorageListObjectsCache::instance().clear(); + break; + } case Type::CLEAR_FILESYSTEM_CACHE: { getContext()->checkAccess(AccessType::SYSTEM_DROP_FILESYSTEM_CACHE); @@ -926,6 +932,20 @@ BlockIO InterpreterSystemQuery::execute() case Type::START_MOVES: startStopAction(ActionLocks::PartsMove, true); break; + case Type::STOP_SWARM_MODE: + { + getContext()->checkAccess(AccessType::SYSTEM_SWARM); + if (getContext()->stopSwarmMode()) + getContext()->unregisterInAutodiscoveryClusters(); + break; + } + case Type::START_SWARM_MODE: + { + getContext()->checkAccess(AccessType::SYSTEM_SWARM); + if (getContext()->startSwarmMode()) + getContext()->registerInAutodiscoveryClusters(); + break; + } case Type::STOP_FETCHES: startStopAction(ActionLocks::PartsFetch, false); break; @@ -2874,6 +2894,9 @@ AccessRightsElements InterpreterSystemQuery::getRequiredAccessForDDLOnCluster() case Type::CLEAR_S3_CLIENT_CACHE: required_access.emplace_back(AccessType::SYSTEM_DROP_S3_CLIENT_CACHE); break; + case Type::DROP_OBJECT_STORAGE_LIST_OBJECTS_CACHE: + required_access.emplace_back(AccessType::SYSTEM_DROP_OBJECT_STORAGE_LIST_OBJECTS_CACHE); + break; case Type::CLEAR_DISTRIBUTED_CACHE: { required_access.emplace_back(AccessType::SYSTEM_DROP_DISTRIBUTED_CACHE); @@ -2960,6 +2983,12 @@ AccessRightsElements InterpreterSystemQuery::getRequiredAccessForDDLOnCluster() required_access.emplace_back(AccessType::SYSTEM_MOVES, query.getDatabase(), query.getTable()); break; } + case Type::STOP_SWARM_MODE: + case Type::START_SWARM_MODE: + { + required_access.emplace_back(AccessType::SYSTEM_SWARM); + break; + } case Type::STOP_PULLING_REPLICATION_LOG: case Type::START_PULLING_REPLICATION_LOG: { diff --git a/src/Interpreters/PartLog.cpp b/src/Interpreters/PartLog.cpp index b5481e32335e..a6e6aa5f112b 100644 --- a/src/Interpreters/PartLog.cpp +++ b/src/Interpreters/PartLog.cpp @@ -72,6 +72,7 @@ ColumnsDescription PartLogElement::getColumnsDescription() {"MovePart", static_cast(MOVE_PART)}, {"MergePartsStart", static_cast(MERGE_PARTS_START)}, {"MutatePartStart", static_cast(MUTATE_PART_START)}, + {"ExportPart", static_cast(EXPORT_PART)}, } ); @@ -113,7 +114,8 @@ ColumnsDescription PartLogElement::getColumnsDescription() "RemovePart — Removing or detaching a data part using [DETACH PARTITION](/reference/statements/alter/partition#detach-partitionpart)." "MutatePartStart — Mutating of a data part has started, " "MutatePart — Mutating of a data part has finished, " - "MovePart — Moving the data part from the one disk to another one."}, + "MovePart — Moving the data part from the one disk to another one." + "ExportPart — Exporting the data part from a MergeTree table into a target table that represents external storage (e.g., object storage or a data lake).."}, {"merge_reason", std::move(merge_reason_datatype), "The reason for the event with type MERGE_PARTS. Can have one of the following values: " "NotAMerge — The current event has the type other than MERGE_PARTS, " @@ -137,6 +139,7 @@ ColumnsDescription PartLogElement::getColumnsDescription() {"part_storage_type", std::make_shared(), "The type of `DataPartStorage`. Possible values: `Packed` - most part files are stored in a single archive (projections and a few service files such as `txn_version.txt` are written separately), `Full` - each file is stored separately."}, {"disk_name", std::make_shared(), "The disk name data part lies on."}, {"path_on_disk", std::make_shared(), "Absolute path to the folder with data part files."}, + {"remote_file_paths", std::make_shared(std::make_shared()), "In case of an export operation to remote storages, the file paths a given export generated"}, {"rows", std::make_shared(), "The number of rows in the data part."}, {"size_in_bytes", std::make_shared(), "Size of the data part on disk in bytes."}, @@ -197,6 +200,12 @@ void PartLogElement::appendToBlock(MutableColumns & columns) const columns[i++]->insert(disk_name); columns[i++]->insert(path_on_disk); + Array remote_file_paths_array; + remote_file_paths_array.reserve(remote_file_paths.size()); + for (const auto & remote_file_path : remote_file_paths) + remote_file_paths_array.push_back(remote_file_path); + columns[i++]->insert(remote_file_paths_array); + columns[i++]->insert(rows); columns[i++]->insert(bytes_compressed_on_disk); diff --git a/src/Interpreters/PartLog.h b/src/Interpreters/PartLog.h index 1143ff188143..780e4422410d 100644 --- a/src/Interpreters/PartLog.h +++ b/src/Interpreters/PartLog.h @@ -25,6 +25,7 @@ struct PartLogElement MOVE_PART = 6, MERGE_PARTS_START = 7, MUTATE_PART_START = 8, + EXPORT_PART = 9, }; /// Copy of MergeAlgorithm since values are written to disk. @@ -68,6 +69,7 @@ struct PartLogElement String disk_name; String path_on_disk; Strings deduplication_block_ids; + std::vector remote_file_paths; MergeTreeDataPartFormat part_format; diff --git a/src/Interpreters/PreparedSets.cpp b/src/Interpreters/PreparedSets.cpp index 9316912f0618..1d332973319e 100644 --- a/src/Interpreters/PreparedSets.cpp +++ b/src/Interpreters/PreparedSets.cpp @@ -326,6 +326,12 @@ SetAndKeyPtr FutureSetFromSubquery::detachSetAndKey() } SetPtr FutureSetFromSubquery::get() const +{ + std::lock_guard lock(mutex); + return get_unsafe(); +} + +SetPtr FutureSetFromSubquery::get_unsafe() const { if (set_and_key->set != nullptr && set_and_key->set->isCreated()) return set_and_key->set; @@ -335,6 +341,7 @@ SetPtr FutureSetFromSubquery::get() const void FutureSetFromSubquery::setQueryPlan(std::unique_ptr source_) { + std::lock_guard lock(mutex); source = std::move(source_); set_and_key->set->setHeader(source->getCurrentHeader()->getColumnsWithTypeAndName()); } @@ -386,6 +393,8 @@ void FutureSetFromSubquery::buildExternalTableFromInplaceSet(StoragePtr external void FutureSetFromSubquery::setExternalTable(StoragePtr external_table_) { + std::lock_guard lock(mutex); + if (set_and_key->set->isCreated()) { if (!set_and_key->set->hasExplicitSetElements()) @@ -399,6 +408,7 @@ void FutureSetFromSubquery::setExternalTable(StoragePtr external_table_) DataTypes FutureSetFromSubquery::getTypes() const { + std::lock_guard lock(mutex); return set_and_key->set->getElementsTypes(); } @@ -415,6 +425,12 @@ bool FutureSetFromSubquery::hasExternalTable() const FutureSet::Hash FutureSetFromSubquery::getHash() const { return hash; } std::unique_ptr FutureSetFromSubquery::build(const SizeLimits & network_transfer_limits, const PreparedSetsCachePtr & prepared_sets_cache) +{ + std::lock_guard lock(mutex); + return build_unsafe(network_transfer_limits, prepared_sets_cache); +} + +std::unique_ptr FutureSetFromSubquery::build_unsafe(const SizeLimits & network_transfer_limits, const PreparedSetsCachePtr & prepared_sets_cache) { if (set_and_key->set->isCreated()) return nullptr; @@ -459,6 +475,8 @@ void FutureSetFromSubquery::prepareForDistributedPlan(const ContextPtr & context void FutureSetFromSubquery::buildSetInplace(const ContextPtr & context) { + std::lock_guard lock(mutex); + if (external_table_set) external_table_set->buildSetInplace(context); @@ -477,7 +495,7 @@ void FutureSetFromSubquery::buildSetInplace(const ContextPtr & context) prepared_sets_cache = nullptr; } - auto plan = build(network_transfer_limits, prepared_sets_cache); + auto plan = build_unsafe(network_transfer_limits, prepared_sets_cache); if (!plan) return; @@ -503,7 +521,9 @@ SetPtr FutureSetFromSubquery::buildOrderedSetInplace(const ContextPtr & context) if (!context->getSettingsRef()[Setting::use_index_for_in_with_subqueries]) return nullptr; - if (auto set = get()) + std::lock_guard lock(mutex); + + if (auto set = get_unsafe()) { if (set->hasExplicitSetElements()) return set; @@ -650,7 +670,7 @@ SetPtr FutureSetFromSubquery::buildOrderedSetInplace(const ContextPtr & context) /// `CreatingSetStep` to the canonical `set_and_key` (as this code always did). On a silent failure /// `source` is gone, so the deferred build cannot rebuild — exactly the previous behavior; the set /// is never reused with partial rows, because the deferred build throws "Not-ready Set" instead. - plan = build(network_transfer_limits, prepared_sets_cache); + plan = build_unsafe(network_transfer_limits, prepared_sets_cache); if (!plan) return nullptr; diff --git a/src/Interpreters/PreparedSets.h b/src/Interpreters/PreparedSets.h index f0771403586d..e1df40752fdf 100644 --- a/src/Interpreters/PreparedSets.h +++ b/src/Interpreters/PreparedSets.h @@ -219,6 +219,11 @@ class FutureSetFromSubquery final : public FutureSet bool hasExternalTable() const; private: + SetPtr get_unsafe() const; + std::unique_ptr build_unsafe( + const SizeLimits & network_transfer_limits, + const PreparedSetsCachePtr & prepared_sets_cache); + Hash hash; ASTPtr ast; SetAndKeyPtr set_and_key; @@ -231,6 +236,8 @@ class FutureSetFromSubquery final : public FutureSet /// The set can never be built once that happened, so `build` rethrows this instead of returning a null /// plan, which its callers would silently take for "nothing left to build". std::exception_ptr in_place_build_failure; + + mutable std::mutex mutex; }; using FutureSetFromSubqueryPtr = std::shared_ptr; diff --git a/src/Interpreters/executeDDLQueryOnCluster.cpp b/src/Interpreters/executeDDLQueryOnCluster.cpp index 53c7a2ac824d..98d09ee80ce7 100644 --- a/src/Interpreters/executeDDLQueryOnCluster.cpp +++ b/src/Interpreters/executeDDLQueryOnCluster.cpp @@ -58,6 +58,8 @@ bool isSupportedAlterTypeForOnClusterDDLQuery(int type) ASTAlterCommand::ATTACH_PARTITION, /// Usually followed by ATTACH PARTITION ASTAlterCommand::FETCH_PARTITION, + /// Data operation that should be executed locally on each replica + ASTAlterCommand::EXPORT_PART, /// Logical error ASTAlterCommand::NO_TYPE, }; diff --git a/src/Interpreters/tests/gtest_cluster_discovery_flags.cpp b/src/Interpreters/tests/gtest_cluster_discovery_flags.cpp new file mode 100644 index 000000000000..34343ca2ccfc --- /dev/null +++ b/src/Interpreters/tests/gtest_cluster_discovery_flags.cpp @@ -0,0 +1,77 @@ +#include + +#include +#include +#include +#include + +namespace +{ + +/// Mirrors ClusterDiscovery::Flags set / setIfPresent / remove contract used by watch callbacks. +template +class UpdateFlags +{ +public: + void set(const T & key, bool value = true) + { + std::unique_lock lk(mu); + flags[key] = value; + any_need_update |= value; + } + + void setIfPresent(const T & key, bool value = true) + { + std::unique_lock lk(mu); + auto it = flags.find(key); + if (it == flags.end()) + return; + it->second = value; + any_need_update |= value; + } + + void remove(const T & key) + { + std::unique_lock lk(mu); + flags.erase(key); + } + + bool contains(const T & key) const + { + std::unique_lock lk(mu); + return flags.contains(key); + } + +private: + mutable std::mutex mu; + std::unordered_map flags; + bool any_need_update = true; +}; + +} + +TEST(ClusterDiscoveryFlags, SetIfPresentDoesNotResurrectRemovedKey) +{ + UpdateFlags flags; + flags.set("gone"); + ASSERT_TRUE(flags.contains("gone")); + + flags.remove("gone"); + ASSERT_FALSE(flags.contains("gone")); + + /// Late Keeper callback after removal. + flags.setIfPresent("gone"); + EXPECT_FALSE(flags.contains("gone")); + + /// Intentional re-registration may insert again. + flags.set("gone"); + EXPECT_TRUE(flags.contains("gone")); +} + +TEST(ClusterDiscoveryFlags, SetIfPresentUpdatesExistingKey) +{ + UpdateFlags flags; + flags.set("alive", false); + flags.setIfPresent("alive", true); + EXPECT_TRUE(flags.contains("alive")); +} diff --git a/src/Interpreters/tests/gtest_cluster_discovery_start.cpp b/src/Interpreters/tests/gtest_cluster_discovery_start.cpp new file mode 100644 index 000000000000..eee373d5ad1c --- /dev/null +++ b/src/Interpreters/tests/gtest_cluster_discovery_start.cpp @@ -0,0 +1,80 @@ +#include +#include +#include +#include + +#include +#include + +#include + +#include +#include +#include +#include + +using namespace DB; + +namespace +{ + +Poco::AutoPtr makeDiscoveryConfig() +{ + /// Observer mode avoids ephemeral registration; initialUpdate may still fail without ZooKeeper, + /// which is fine — startImpl() still assigns the worker thread after catching the exception. + std::istringstream config_stream{R"( + + + + + /clickhouse/discovery/test_cluster_concurrent_start + + + + + + )"}; + return new Poco::Util::XMLConfiguration(config_stream); +} + +} + +/// Regression: concurrent start() / updateFromConfig (ensureWorkerStarted) must not +/// double-assign ThreadFromGlobalPool (which aborts if already initialized). +TEST(ClusterDiscovery, ConcurrentStartDoesNotAbort) +{ + ServerUUID::setRandomForUnitTests(); + + auto context = Context::createCopy(getContext().context); + auto config = makeDiscoveryConfig(); + auto discovery = std::make_unique(*config, context, context->getMacros()); + + constexpr size_t num_threads = 8; + constexpr size_t iterations = 40; + std::atomic started_calls{0}; + + std::vector threads; + threads.reserve(num_threads); + for (size_t i = 0; i < num_threads; ++i) + { + threads.emplace_back([&] + { + for (size_t j = 0; j < iterations; ++j) + { + if ((j % 2) == 0) + discovery->start(); + else + discovery->updateFromConfig(*config); + started_calls.fetch_add(1, std::memory_order_relaxed); + } + }); + } + + for (auto & t : threads) + t.join(); + + EXPECT_EQ(started_calls.load(), num_threads * iterations); + + /// Destructor joins the worker; surviving to here means no abort on double-start. + discovery.reset(); +} diff --git a/src/Parsers/ASTAlterQuery.cpp b/src/Parsers/ASTAlterQuery.cpp index eb4ba28cbd81..cb88ef990fed 100644 --- a/src/Parsers/ASTAlterQuery.cpp +++ b/src/Parsers/ASTAlterQuery.cpp @@ -91,6 +91,10 @@ ASTPtr ASTAlterCommand::clone() const res->add_enum_values = res->children.emplace_back(add_enum_values->clone()); if (refresh) res->refresh = res->children.emplace_back(refresh->clone()).get(); + if (to_table_function) + res->to_table_function = res->children.emplace_back(to_table_function->clone()).get(); + if (partition_by_expr) + res->partition_by_expr = res->children.emplace_back(partition_by_expr->clone()).get(); return res; } @@ -925,6 +929,49 @@ void ASTAlterCommand::formatImpl(WriteBuffer & ostr, const FormatSettings & sett ostr << quoteString(move_destination_name); } } + else if (type == ASTAlterCommand::EXPORT_PART) + { + ostr << "EXPORT PART "; + partition->format(ostr, settings, state, frame); + ostr << " TO "; + switch (move_destination_type) + { + case DataDestinationType::TABLE: + ostr << "TABLE "; + if (to_table_function) + { + ostr << "FUNCTION "; + to_table_function->format(ostr, settings, state, frame); + if (partition_by_expr) + { + ostr << " PARTITION BY "; + partition_by_expr->format(ostr, settings, state, frame); + } + } + else + { + if (!to_database.empty()) + ostr << backQuoteIfNeed(to_database) << "."; + + ostr << backQuoteIfNeed(to_table); + } + return; + default: + break; + } + + } + else if (type == ASTAlterCommand::EXPORT_PARTITION) + { + ostr << "EXPORT PARTITION "; + partition->format(ostr, settings, state, frame); + ostr << " TO TABLE "; + if (!to_database.empty()) + { + ostr << backQuoteIfNeed(to_database) << "."; + } + ostr << backQuoteIfNeed(to_table); + } else if (type == ASTAlterCommand::REPLACE_PARTITION) { ostr << (replace ? "REPLACE" : "ATTACH") << " PARTITION " @@ -1158,6 +1205,8 @@ void ASTAlterCommand::forEachPointerToChild(std::function #include #include +#if !defined(CLICKHOUSE_PARSER_MINIMAL_BUILD) +#include +#include +#endif #include #include +#include namespace DB { @@ -32,12 +37,13 @@ void FunctionSecretArgumentsFinder::markSecretArgument(size_t index, bool argume { if (index >= function->arguments->size()) return; + auto real_index = function->arguments->getRealIndex(index); chassert(result.replacement.empty()); /// We shouldn't use replacement with masking other arguments /// Each argument is masked individually: valid S3 syntax can interleave secrets with non-secret /// arguments, which a contiguous span cannot represent without hiding the arguments in between. /// A malformed query can mark the same index as both named and positional; the positional form /// wins, hiding the argument whole (fail closed). - auto [it, inserted] = result.masked_arguments.emplace(index, argument_is_named); + auto [it, inserted] = result.masked_arguments.emplace(real_index, argument_is_named); if (!inserted) it->second &= argument_is_named; } @@ -222,9 +228,17 @@ void FunctionSecretArgumentsFinder::findOrdinaryFunctionSecretArguments() { findMongoDBSecretArguments(); } + else if (function->name() == "iceberg") + { + findIcebergFunctionSecretArguments(/* is_cluster_function= */ false); + } + else if (function ->name() == "icebergCluster") + { + findIcebergFunctionSecretArguments(/* is_cluster_function= */ true); + } else if ((function->name() == "s3") || (function->name() == "cosn") || (function->name() == "oss") || (function->name() == "deltaLake") || (function->name() == "deltaLakeS3") || (function->name() == "hudi") || - (function->name() == "iceberg") || (function->name() == "gcs") || (function->name() == "icebergS3") || + (function->name() == "gcs") || (function->name() == "icebergS3") || (function->name() == "paimon") || (function->name() == "paimonS3")) { /// s3('url', 'aws_access_key_id', 'aws_secret_access_key', ...) @@ -232,7 +246,7 @@ void FunctionSecretArgumentsFinder::findOrdinaryFunctionSecretArguments() } else if ((function->name() == "s3Cluster") || (function ->name() == "hudiCluster") || (function ->name() == "deltaLakeCluster") || (function ->name() == "deltaLakeS3Cluster") || - (function ->name() == "icebergS3Cluster") || (function ->name() == "icebergCluster") || + (function ->name() == "icebergS3Cluster") || (function ->name() == "paimonCluster") || (function ->name() == "paimonS3Cluster")) { /// s3Cluster('cluster_name', 'url', 'aws_access_key_id', 'aws_secret_access_key', ...) @@ -512,6 +526,52 @@ void FunctionSecretArgumentsFinder::findS3FunctionSecretArguments(bool is_cluste maskS3PositionalSecrets(positional, url_slot, with_structure); } +std::string FunctionSecretArgumentsFinder::findIcebergStorageType(bool is_cluster_function) +{ + std::string storage_type = "s3"; + + size_t count = function->arguments->size(); + if (!count) + return storage_type; + + auto storage_type_idx = findNamedArgument(&storage_type, "storage_type"); + if (storage_type_idx != -1) + { + storage_type = Poco::toLower(storage_type); + function->arguments->skipArgument(storage_type_idx); + } + /// The standalone parser links no named collection registry, so there is nothing to look up + /// there and the storage type stays at its default. +#if !defined(CLICKHOUSE_PARSER_MINIMAL_BUILD) + else if (isNamedCollectionName(is_cluster_function ? 1 : 0)) + { + std::string collection_name; + if (function->arguments->at(is_cluster_function ? 1 : 0)->tryGetString(&collection_name, true)) + { + NamedCollectionPtr collection = NamedCollectionFactory::instance().tryGet(collection_name); + if (collection && collection->has("storage_type")) + { + storage_type = Poco::toLower(collection->get("storage_type")); + } + } + } +#endif + + return storage_type; +} + +void FunctionSecretArgumentsFinder::findIcebergFunctionSecretArguments(bool is_cluster_function) +{ + auto storage_type = findIcebergStorageType(is_cluster_function); + + if (storage_type == "s3") + findS3FunctionSecretArguments(is_cluster_function); + else if (storage_type == "azure") + findAzureBlobStorageFunctionSecretArguments(is_cluster_function); + + function->arguments->unskipArguments(); +} + void FunctionSecretArgumentsFinder::findAzureBlobStorageFunctionSecretArguments(bool is_cluster_function) { /// azureBlobStorageCluster('cluster_name', 'conn_string/storage_account_url', ...) has 'conn_string/storage_account_url' as its second argument. @@ -578,7 +638,7 @@ bool FunctionSecretArgumentsFinder::maskAzureConnectionString(ssize_t url_arg_id if (maskConnectionStringKey(url_arg, "AccountKey=")) { chassert(result.count == 0); /// We shouldn't use replacement with masking other arguments - result.start = url_arg_idx; + result.start = function->arguments->getRealIndex(url_arg_idx); result.are_named = argument_is_named; result.count = 1; result.replacement = url_arg; @@ -588,7 +648,7 @@ bool FunctionSecretArgumentsFinder::maskAzureConnectionString(ssize_t url_arg_id if (maskConnectionStringKey(url_arg, "SharedAccessSignature=")) { chassert(result.count == 0); /// We shouldn't use replacement with masking other arguments - result.start = url_arg_idx; + result.start = function->arguments->getRealIndex(url_arg_idx); result.are_named = argument_is_named; result.count = 1; result.replacement = url_arg; @@ -818,9 +878,13 @@ void FunctionSecretArgumentsFinder::findTableEngineSecretArguments() { findMongoDBSecretArguments(); } + else if (engine_name == "Iceberg") + { + findIcebergTableEngineSecretArguments(); + } else if ((engine_name == "S3") || (engine_name == "COSN") || (engine_name == "OSS") || (engine_name == "GCS") || (engine_name == "DeltaLake") || (engine_name == "DeltaLakeS3") || (engine_name == "Hudi") - || (engine_name == "Iceberg") || (engine_name == "IcebergS3") + || (engine_name == "IcebergS3") || (engine_name == "Paimon") || (engine_name == "PaimonS3") || (engine_name == "S3Queue")) { @@ -831,7 +895,7 @@ void FunctionSecretArgumentsFinder::findTableEngineSecretArguments() { findURLSecretArguments(); } - else if (engine_name == "AzureBlobStorage" || engine_name == "AzureQueue") + else if (engine_name == "AzureBlobStorage" || engine_name == "AzureQueue" || engine_name == "IcebergAzure") { findAzureBlobStorageTableEngineSecretArguments(); } @@ -1089,6 +1153,18 @@ void FunctionSecretArgumentsFinder::findBigQuerySecretArguments() } } +void FunctionSecretArgumentsFinder::findIcebergTableEngineSecretArguments() +{ + auto storage_type = findIcebergStorageType(0); + + if (storage_type == "s3") + findS3TableEngineSecretArguments(); + else if (storage_type == "azure") + findAzureBlobStorageTableEngineSecretArguments(); + + function->arguments->unskipArguments(); +} + void FunctionSecretArgumentsFinder::findDatabaseEngineSecretArguments() { const String & engine_name = function->name(); @@ -1115,7 +1191,7 @@ void FunctionSecretArgumentsFinder::findDatabaseEngineSecretArguments() /// S3('url', 'access_key_id', 'secret_access_key') findS3DatabaseSecretArguments(); } - else if (engine_name == "DataLakeCatalog") + else if (engine_name == "DataLakeCatalog" || engine_name == "Iceberg") { findDataLakeCatalogSecretArguments(); } diff --git a/src/Parsers/FunctionSecretArgumentsFinder.h b/src/Parsers/FunctionSecretArgumentsFinder.h index 10342f18cc6b..baacde355038 100644 --- a/src/Parsers/FunctionSecretArgumentsFinder.h +++ b/src/Parsers/FunctionSecretArgumentsFinder.h @@ -3,6 +3,7 @@ #include #include #include +#include #include #include #include @@ -35,6 +36,21 @@ class AbstractFunction virtual ~Arguments() = default; virtual size_t size() const = 0; virtual std::unique_ptr at(size_t n) const = 0; + void skipArgument(size_t n) { skipped_indexes.insert(n); } + void unskipArguments() { skipped_indexes.clear(); } + size_t getRealIndex(size_t n) const + { + for (auto idx : skipped_indexes) + { + if (n < idx) + break; + ++n; + } + return n; + } + size_t skippedSize() const { return skipped_indexes.size(); } + private: + std::set skipped_indexes; }; virtual ~AbstractFunction() = default; @@ -169,6 +185,8 @@ class FunctionSecretArgumentsFinder void maskXDBCSecretNamedArgument(std::string_view key, size_t start); void findS3FunctionSecretArguments(bool is_cluster_function); + std::string findIcebergStorageType(bool is_cluster_function); + void findIcebergFunctionSecretArguments(bool is_cluster_function); void findAzureBlobStorageFunctionSecretArguments(bool is_cluster_function); bool maskAzureConnectionString(ssize_t url_arg_idx, bool argument_is_named = false, size_t start = 0); /// Masks the secrets of every URL form (`url`/`urlCluster` table functions, the `URL` table @@ -195,6 +213,7 @@ class FunctionSecretArgumentsFinder void findTableEngineSecretArguments(); void findExternalDistributedTableEngineSecretArguments(); void findS3TableEngineSecretArguments(); + void findIcebergTableEngineSecretArguments(); void findAzureBlobStorageTableEngineSecretArguments(); void findRedisFunctionSecretArguments(); void findYTsaurusStorageTableEngineSecretArguments(); diff --git a/src/Parsers/FunctionSecretArgumentsFinderAST.h b/src/Parsers/FunctionSecretArgumentsFinderAST.h index 754411535c8d..838844d9e8ff 100644 --- a/src/Parsers/FunctionSecretArgumentsFinderAST.h +++ b/src/Parsers/FunctionSecretArgumentsFinderAST.h @@ -64,10 +64,13 @@ class FunctionAST : public AbstractFunction { public: explicit ArgumentsAST(const ASTs * arguments_) : arguments(arguments_) {} - size_t size() const override { return arguments ? arguments->size() : 0; } + size_t size() const override + { /// size withous skipped indexes + return arguments ? arguments->size() - skippedSize() : 0; + } std::unique_ptr at(size_t n) const override - { - return std::make_unique(arguments->at(n).get()); + { /// n is relative index, some can be skipped + return std::make_unique(arguments->at(getRealIndex(n)).get()); } private: const ASTs * arguments = nullptr; diff --git a/src/Parsers/ParserAlterQuery.cpp b/src/Parsers/ParserAlterQuery.cpp index 1fd41ba286a0..25152694441a 100644 --- a/src/Parsers/ParserAlterQuery.cpp +++ b/src/Parsers/ParserAlterQuery.cpp @@ -87,6 +87,8 @@ bool ParserAlterCommand::parseImpl(Pos & pos, ASTPtr & node, Expected & expected ParserKeyword s_forget_partition(Keyword::FORGET_PARTITION); ParserKeyword s_move_partition(Keyword::MOVE_PARTITION); ParserKeyword s_move_part(Keyword::MOVE_PART); + ParserKeyword s_export_part(Keyword::EXPORT_PART); + ParserKeyword s_export_partition(Keyword::EXPORT_PARTITION); ParserKeyword s_drop_detached_partition(Keyword::DROP_DETACHED_PARTITION); ParserKeyword s_drop_detached_part(Keyword::DROP_DETACHED_PART); ParserKeyword s_fetch_partition(Keyword::FETCH_PARTITION); @@ -96,6 +98,7 @@ bool ParserAlterCommand::parseImpl(Pos & pos, ASTPtr & node, Expected & expected ParserKeyword s_unfreeze(Keyword::UNFREEZE); ParserKeyword s_unlock_snapshot(Keyword::UNLOCK_SNAPSHOT); ParserKeyword s_partition(Keyword::PARTITION); + ParserKeyword s_partition_by(Keyword::PARTITION_BY); ParserKeyword s_first(Keyword::FIRST); ParserKeyword s_after(Keyword::AFTER); @@ -110,6 +113,7 @@ bool ParserAlterCommand::parseImpl(Pos & pos, ASTPtr & node, Expected & expected ParserKeyword s_to_volume(Keyword::TO_VOLUME); ParserKeyword s_to_table(Keyword::TO_TABLE); ParserKeyword s_to_shard(Keyword::TO_SHARD); + ParserKeyword s_function(Keyword::FUNCTION); ParserKeyword s_delete(Keyword::DELETE); ParserKeyword s_update(Keyword::UPDATE); @@ -185,6 +189,8 @@ bool ParserAlterCommand::parseImpl(Pos & pos, ASTPtr & node, Expected & expected ASTPtr command_sql_security; ASTPtr command_snapshot_desc; ASTPtr command_refresh; + ASTPtr export_table_function; + ASTPtr export_table_function_partition_by_expr; if (with_round_bracket) { @@ -559,6 +565,57 @@ bool ParserAlterCommand::parseImpl(Pos & pos, ASTPtr & node, Expected & expected command->move_destination_name = ast_space_name->as().value.safeGet(); } + else if (s_export_part.ignore(pos, expected)) + { + if (!parser_string_and_substituion.parse(pos, command_partition, expected)) + return false; + + command->type = ASTAlterCommand::EXPORT_PART; + command->part = true; + + if (!s_to_table.ignore(pos, expected)) + { + return false; + } + + if (s_function.ignore(pos, expected)) + { + ParserFunction table_function_parser(/*allow_function_parameters=*/true, /*is_table_function=*/true); + + if (!table_function_parser.parse(pos, export_table_function, expected)) + return false; + + if (s_partition_by.ignore(pos, expected)) + if (!parser_exp_elem.parse(pos, export_table_function_partition_by_expr, expected)) + return false; + + command->to_table_function = export_table_function.get(); + command->partition_by_expr = export_table_function_partition_by_expr.get(); + command->move_destination_type = DataDestinationType::TABLE; + } + else + { + if (!parseDatabaseAndTableName(pos, expected, command->to_database, command->to_table)) + return false; + command->move_destination_type = DataDestinationType::TABLE; + } + } + else if (s_export_partition.ignore(pos, expected)) + { + if (!parser_partition.parse(pos, command_partition, expected)) + return false; + + command->type = ASTAlterCommand::EXPORT_PARTITION; + + if (!s_to_table.ignore(pos, expected)) + { + return false; + } + + if (!parseDatabaseAndTableName(pos, expected, command->to_database, command->to_table)) + return false; + command->move_destination_type = DataDestinationType::TABLE; + } else if (s_move_partition.ignore(pos, expected)) { if (!parser_partition.parse(pos, command_partition, expected)) @@ -1169,6 +1226,10 @@ bool ParserAlterCommand::parseImpl(Pos & pos, ASTPtr & node, Expected & expected command->snapshot_desc = command->children.emplace_back(std::move(command_snapshot_desc)).get(); if (command_refresh) command->refresh = command->children.emplace_back(std::move(command_refresh)).get(); + if (export_table_function) + command->to_table_function = command->children.emplace_back(std::move(export_table_function)).get(); + if (export_table_function_partition_by_expr) + command->partition_by_expr = command->children.emplace_back(std::move(export_table_function_partition_by_expr)).get(); return true; } diff --git a/src/Parsers/ParserKillQueryQuery.cpp b/src/Parsers/ParserKillQueryQuery.cpp index 97e58566af67..99f2d6fd2d64 100644 --- a/src/Parsers/ParserKillQueryQuery.cpp +++ b/src/Parsers/ParserKillQueryQuery.cpp @@ -17,6 +17,7 @@ bool ParserKillQueryQuery::parseImpl(Pos & pos, ASTPtr & node, Expected & expect ParserKeyword p_kill{Keyword::KILL}; ParserKeyword p_query{Keyword::QUERY}; ParserKeyword p_mutation{Keyword::MUTATION}; + ParserKeyword p_export_partition{Keyword::EXPORT_PARTITION}; ParserKeyword p_part_move_to_shard{Keyword::PART_MOVE_TO_SHARD}; ParserKeyword p_transaction{Keyword::TRANSACTION}; ParserKeyword p_on{Keyword::ON}; @@ -33,6 +34,8 @@ bool ParserKillQueryQuery::parseImpl(Pos & pos, ASTPtr & node, Expected & expect query->type = ASTKillQueryQuery::Type::Query; else if (p_mutation.ignore(pos, expected)) query->type = ASTKillQueryQuery::Type::Mutation; + else if (p_export_partition.ignore(pos, expected)) + query->type = ASTKillQueryQuery::Type::ExportPartition; else if (p_part_move_to_shard.ignore(pos, expected)) query->type = ASTKillQueryQuery::Type::PartMoveToShard; else if (p_transaction.ignore(pos, expected)) diff --git a/src/Planner/Planner.cpp b/src/Planner/Planner.cpp index 5fba02502364..29ca045e8cda 100644 --- a/src/Planner/Planner.cpp +++ b/src/Planner/Planner.cpp @@ -48,6 +48,7 @@ #include #include #include +#include #include #include @@ -66,6 +67,7 @@ #include #include #include +#include #include @@ -187,6 +189,7 @@ namespace Setting extern const SettingsBool make_distributed_plan; extern const SettingsBool query_plan_enable_optimizations; extern const SettingsUInt64 query_plan_max_limit_for_top_k_optimization; + extern const SettingsBool use_hive_partitioning; } namespace ServerSetting @@ -237,6 +240,11 @@ void checkStoragesSupportTransactions(const PlannerContextPtr & planner_context) } } +} + +namespace +{ + /** Storages can rely that filters that for storage will be available for analysis before * getQueryProcessingStage method will be called. * @@ -416,6 +424,8 @@ FiltersForTableExpressionMap collectFiltersForAnalysis(const QueryTreeNodePtr & return res; } +} + FiltersForTableExpressionMap collectFiltersForAnalysis(const QueryTreeNodePtr & query_tree_node, const SelectQueryOptions & select_query_options, const ActionsDAG * post_filter) { if (select_query_options.only_analyze) @@ -437,6 +447,9 @@ FiltersForTableExpressionMap collectFiltersForAnalysis(const QueryTreeNodePtr & return collectFiltersForAnalysis(query_tree_node, table_expressions_nodes, context, post_filter); } +namespace +{ + /// Extend lifetime of query context, storages, and table locks void extendQueryContextAndStoragesLifetime(QueryPlan & query_plan, const PlannerContextPtr & planner_context) { @@ -646,6 +659,21 @@ ALWAYS_INLINE void addFilterStep( query_plan.addStep(std::move(where_step)); } +template +ALWAYS_INLINE void addObjectFilterStep( + QueryPlan & query_plan, + FilterAnalysisResult & filter_analysis_result, + const char (&step_description)[size]) +{ + auto actions = std::move(filter_analysis_result.filter_actions->dag); + + auto where_step = std::make_unique(query_plan.getCurrentHeader(), + std::move(actions), + filter_analysis_result.filter_column_name); + where_step->setStepDescription(step_description); + query_plan.addStep(std::move(where_step)); +} + Aggregator::Params getAggregatorParams(const PlannerContextPtr & planner_context, const AggregationAnalysisResult & aggregation_analysis_result, const QueryAnalysisResult & query_analysis_result, @@ -2839,6 +2867,16 @@ void Planner::buildPlanForQueryNode() if (query_processing_info.isSecondStage() || query_processing_info.isFromAggregationState()) { + if (settings[Setting::use_hive_partitioning] + && !query_processing_info.isFirstStage() + && expression_analysis_result.hasWhere()) + { + if (typeid_cast(query_plan.getRootNode()->step.get())) + { + addObjectFilterStep(query_plan, expression_analysis_result.getWhere(), "WHERE"); + } + } + if (query_processing_info.isFromAggregationState()) { /// Aggregation was performed on remote shards diff --git a/src/Planner/Planner.h b/src/Planner/Planner.h index 7e1c87d5f41f..7b6d7a35c80b 100644 --- a/src/Planner/Planner.h +++ b/src/Planner/Planner.h @@ -7,6 +7,7 @@ #include #include +#include namespace DB { @@ -89,4 +90,9 @@ class Planner QueryNodeToPlanStepMapping query_node_to_plan_step_mapping; }; +FiltersForTableExpressionMap collectFiltersForAnalysis( + const QueryTreeNodePtr & query_tree_node, + const SelectQueryOptions & select_query_options, + const ActionsDAG * post_filter); + } diff --git a/src/Planner/PlannerJoinTree.cpp b/src/Planner/PlannerJoinTree.cpp index 2c27d8acef42..3ddeaacd9305 100644 --- a/src/Planner/PlannerJoinTree.cpp +++ b/src/Planner/PlannerJoinTree.cpp @@ -79,6 +79,7 @@ #include #include #include +#include #include #include #include @@ -348,6 +349,91 @@ void checkAccessRightsForSubquery(const QueryTreeNodePtr & subquery_node, const } } +/// Same restrictions as JOIN filter pushdown (`canPrefilterJoinSide`). +bool joinTreePreservesRowsForTable(const QueryTreeNodePtr & join_tree, const QueryTreeNodePtr & table) +{ + std::vector stack = {join_tree}; + while (!stack.empty()) + { + auto node = std::move(stack.back()); + stack.pop_back(); + if (!node) + continue; + + if (const auto * join = node->as()) + { + const bool table_on_left = extractTableExpressionsSet(join->getLeftTableExpressionNodeTyped()).contains(table.get()); + const bool table_on_right = extractTableExpressionsSet(join->getRightTableExpressionNodeTyped()).contains(table.get()); + + if (table_on_left && !canPrefilterJoinSide(join->getKind(), join->getStrictness(), JoinTableSide::Left)) + return false; + if (table_on_right && !canPrefilterJoinSide(join->getKind(), join->getStrictness(), JoinTableSide::Right)) + return false; + stack.push_back(join->getLeftTableExpressionNode()); + stack.push_back(join->getRightTableExpressionNode()); + } + else if (const auto * array_join = node->as()) + { + stack.push_back(array_join->getTableExpressionNode()); + } + else if (const auto * cross_join = node->as()) + { + for (const auto & expr : cross_join->getTableExpressions()) + stack.push_back(expr); + } + } + return true; +} + +/// `IStorageCluster` JOINs wrap the left table in a subquery. Attach dummy-analysis +/// filters to `ReadFromCluster` for listing only; do not add a FilterStep, which would +/// drop unused columns from the wrap header. Other wrap sources (`ReadFromMergeTree` +/// for a remote `Distributed` replica, `ReadFromObjectStorageStep` after cluster +/// fallback) keep the optimizer's later `applyFilters`. +void tryAddClusterWrapFilter(QueryPlan & query_plan, const TableExpressionData & table_expression_data) +{ + const auto & filter_actions = table_expression_data.getFilterActions(); + if (!filter_actions || !query_plan.isInitialized()) + return; + + QueryPlan::Node * node = query_plan.getRootNode(); + while (node && !node->children.empty()) + node = node->children.front(); + + auto * source = node ? typeid_cast(node->step.get()) : nullptr; + if (!source) + return; + + auto filter_dag = filter_actions->clone(); + + if (filter_dag.getOutputs().size() != 1) + throw Exception(ErrorCodes::LOGICAL_ERROR, "Filter DAG must have single output"); + + const auto filter_column_name = filter_dag.getOutputs().at(0)->result_name; + const auto & header = source->getOutputHeader(); + ActionsDAG rename_dag(header->getColumnsWithTypeAndName()); + const auto & identifier_to_name = table_expression_data.getColumnIdentifierToColumnName(); + + for (const auto * input : filter_dag.getInputs()) + { + if (header->has(input->result_name)) + continue; + + auto it = identifier_to_name.find(input->result_name); + if (it == identifier_to_name.end() || !header->has(it->second)) + continue; + + const auto & physical = rename_dag.findInOutputs(it->second); + rename_dag.addOrReplaceInOutputs(rename_dag.addAlias(physical, input->result_name)); + } + + filter_dag = ActionsDAG::merge(std::move(rename_dag), std::move(filter_dag)); + source->addFilter(std::move(filter_dag), filter_column_name); + /// Wrap subquery planning already called `applyFilters` with no predicate. + /// Apply now so `ReadFromCluster` can keep the copied `WHERE` for listing. + source->SourceStepWithFilterBase::applyFilters(); +} + bool shouldIgnoreQuotaAndLimits(const TableNode & table_node) { const auto & storage_id = table_node.getStorageID(); @@ -1601,11 +1687,58 @@ JoinTreeQueryPlan buildQueryPlanForTableExpression(TableExpressionNodePtr table_ auto & table_expression_data = planner_context->getTableExpressionDataOrThrow(table_expression); QueryProcessingStage::Enum till_stage = QueryProcessingStage::Enum::FetchColumns; + bool can_prefilter_wrapped_table = false; if (wrap_read_columns_in_subquery) { + auto original_table_expression = table_expression; + + const auto * parent_query = select_query_info.query_tree + ? select_query_info.query_tree->as() + : nullptr; + can_prefilter_wrapped_table = parent_query + && joinTreePreservesRowsForTable(parent_query->getJoinTreeNode(), original_table_expression); + + /// Subqueries inherit the outer GlobalPlannerContext, whose filter map is keyed by + /// outer table nodes. Collect filters for this JOIN query so icebergCluster listing + /// still sees left-only WHERE after the wrap. Skip the same join sides as + /// `joinTreePreservesRowsForTable` so listing cannot change `ASOF` / `PASTE` matches. + if (can_prefilter_wrapped_table && !table_expression_data.getFilterActions()) + { + auto collected = collectFiltersForAnalysis(select_query_info.query_tree, select_query_options, nullptr); + auto it = collected.find(table_expression); + if (it != collected.end() && it->second.filter_actions) + table_expression_data.setFilterActions(it->second.filter_actions->clone()); + } + auto columns = table_expression_data.getColumns(); - table_expression = buildSubqueryToReadColumnsFromTableExpression(columns, table_expression, query_context); + table_expression = buildSubqueryToReadColumnsFromTableExpression(columns, original_table_expression, query_context); + + /// Wrap is planned as `SELECT cols FROM icebergCluster` with no JOIN. Copy left-only + /// WHERE/PREWHERE so initiator file listing sees the same predicate as a single-table + /// `icebergCluster` read. Same helper as `IStorageCluster::updateQueryWithJoinToSendIfNeeded`. + if (can_prefilter_wrapped_table) + { + auto copy_left_only = [&](const QueryTreeNodePtr & predicate) -> QueryTreeNodePtr + { + auto cloned = predicate->clone(); + removeExpressionsThatDoNotDependOnTableIdentifiers(cloned, original_table_expression, query_context); + removeExpressionsThatAreUnsafeToDuplicate(cloned, query_context); + return cloned; + }; + + auto & wrap_query = table_expression->as(); + if (parent_query->hasWhere()) + { + if (auto pred = copy_left_only(parent_query->getWhere())) + wrap_query.getWhere() = std::move(pred); + } + if (parent_query->hasPrewhere()) + { + if (auto pred = copy_left_only(parent_query->getPrewhere())) + wrap_query.getPrewhere() = std::move(pred); + } + } } auto * table_node = table_expression->as(); @@ -1901,10 +2034,11 @@ JoinTreeQueryPlan buildQueryPlanForTableExpression(TableExpressionNodePtr table_ /// The filter is built against this table's schema, but read() hands it to wrapper /// storages' children (Merge, Buffer), which re-derive it against their own types. /// Push it down only if every column it consumes is in the PREWHERE contract. - /// A remote storage cannot carry it at all: read() only ships query text to the - /// remote servers and never lowers the filter into it, so pushing would silently - /// drop an access-control filter. Refuse, and let the stage check fail closed. - bool can_push_down_filter = storage->supportsPrewhere() && !storage->isRemote(); + /// A storage that ships query text instead of lowering the filter into the read + /// cannot carry it at all, so pushing would silently drop an access-control + /// filter. Refuse, and let the stage check fail closed. + bool can_push_down_filter + = storage->supportsPrewhere() && storage->appliesRowLevelFilterInRead(query_context); if (can_push_down_filter) { if (const auto supported_prewhere_columns = storage->supportedPrewhereColumns()) @@ -2750,12 +2884,15 @@ JoinTreeQueryPlan buildQueryPlanForTableExpression(TableExpressionNodePtr table_ else { std::shared_ptr subquery_planner_context; + auto subquery_options = select_query_options.subquery(); if (wrap_read_columns_in_subquery) - subquery_planner_context = std::make_shared(nullptr, nullptr, nullptr, FiltersForTableExpressionMap{}); + { + subquery_planner_context = std::make_shared( + nullptr, nullptr, nullptr, collectFiltersForAnalysis(table_expression, subquery_options, nullptr)); + } else subquery_planner_context = planner_context->getGlobalPlannerContext(); - auto subquery_options = select_query_options.subquery(); Planner subquery_planner(table_expression, subquery_options, subquery_planner_context); /// Propagate storage limits to subquery subquery_planner.addStorageLimits(*select_query_info.storage_limits); @@ -2763,6 +2900,8 @@ JoinTreeQueryPlan buildQueryPlanForTableExpression(TableExpressionNodePtr table_ const auto & mapping = subquery_planner.getQueryNodeToPlanStepMapping(); query_node_to_plan_step_mapping.insert(mapping.begin(), mapping.end()); query_plan = std::move(subquery_planner).extractQueryPlan(); + if (wrap_read_columns_in_subquery && till_stage == QueryProcessingStage::FetchColumns && can_prefilter_wrapped_table) + tryAddClusterWrapFilter(query_plan, table_expression_data); } auto & alias_column_expressions = table_expression_data.getAliasColumnExpressions(); @@ -2814,7 +2953,9 @@ JoinTreeQueryPlan buildQueryPlanForTableExpression(TableExpressionNodePtr table_ /// Overall, IStorage::read -> FetchColumns returns normal column names (except Distributed, which is inconsistent) /// Interpreter::getQueryPlan -> FetchColumns returns identifiers (why?) and this the reason for the bug ^ in Distributed /// Hopefully there is no other case when we read from Distributed up to FetchColumns. - if (table_node && table_node->getStorage()->isRemote() && select_query_options.to_stage == QueryProcessingStage::FetchColumns) + if (table_node && table_node->getStorage()->isRemote()) + updated_actions_dag_outputs.push_back(output_node); + else if (table_function_node && table_function_node->getStorage()->isRemote()) updated_actions_dag_outputs.push_back(output_node); } else diff --git a/src/Processors/Chunk.cpp b/src/Processors/Chunk.cpp index bd5e8027801c..689190b6a0e9 100644 --- a/src/Processors/Chunk.cpp +++ b/src/Processors/Chunk.cpp @@ -109,7 +109,12 @@ void Chunk::addColumn(ColumnPtr column) void Chunk::addColumn(size_t position, ColumnPtr column) { - if (position >= columns.size()) + if (position == columns.size()) + { + addColumn(column); + return; + } + if (position > columns.size()) throw Exception(ErrorCodes::POSITION_OUT_OF_BOUND, "Position {} out of bound in Chunk::addColumn(), max position = {}", position, !columns.empty() ? columns.size() - 1 : 0); diff --git a/src/Processors/QueryPlan/Optimizations/actionsDAGUtils.cpp b/src/Processors/QueryPlan/Optimizations/actionsDAGUtils.cpp index fa04c059cff6..c20c89c58cdb 100644 --- a/src/Processors/QueryPlan/Optimizations/actionsDAGUtils.cpp +++ b/src/Processors/QueryPlan/Optimizations/actionsDAGUtils.cpp @@ -291,16 +291,7 @@ MatchedTrees::Matches matchTrees( } -struct PossiblyMonotonicChain -{ - const ActionsDAG::Node * input_node = nullptr; - std::vector non_const_arg_pos; - bool changes_order = false; - bool is_strict = true; -}; - -/// Build a chain of functions which may be monotonic. -static PossiblyMonotonicChain buildPossiblyMonitinicChain(const ActionsDAG::Node * node) +PossiblyMonotonicChain buildPossiblyMonotonicChain(const ActionsDAG::Node * node) { std::vector chain; @@ -353,8 +344,7 @@ static PossiblyMonotonicChain buildPossiblyMonitinicChain(const ActionsDAG::Node return {node, std::move(chain)}; } -/// Check whether all the function in chain are monotonic -static bool isMonotonicChain(const ActionsDAG::Node * node, PossiblyMonotonicChain & chain) +bool isMonotonicChain(const ActionsDAG::Node * node, PossiblyMonotonicChain & chain) { auto it = chain.non_const_arg_pos.begin(); while (node != chain.input_node) @@ -434,7 +424,7 @@ void applyActionsToSortDescription( if (output == output_to_skip) continue; - auto chain = buildPossiblyMonitinicChain(output); + auto chain = buildPossiblyMonotonicChain(output); if (!chain.input_node) break; diff --git a/src/Processors/QueryPlan/Optimizations/actionsDAGUtils.h b/src/Processors/QueryPlan/Optimizations/actionsDAGUtils.h index c9e14970fb20..9e30c712f1fb 100644 --- a/src/Processors/QueryPlan/Optimizations/actionsDAGUtils.h +++ b/src/Processors/QueryPlan/Optimizations/actionsDAGUtils.h @@ -56,6 +56,22 @@ MatchedTrees::Matches matchTrees( bool check_monotonicity = true, size_t max_size_for_sets_from_tuple_to_compare = 0); +/// A path from a node down to an input, where every function on the path has a single non-constant argument. +/// `non_const_arg_pos` holds the position of that argument for every function on the path, top-down. +struct PossiblyMonotonicChain +{ + const ActionsDAG::Node * input_node = nullptr; + std::vector non_const_arg_pos; + bool changes_order = false; + bool is_strict = true; +}; + +/// Build a chain of functions which may be monotonic. `input_node` is nullptr if the node is not such a chain. +PossiblyMonotonicChain buildPossiblyMonotonicChain(const ActionsDAG::Node * node); + +/// Check whether all the function in chain are monotonic +bool isMonotonicChain(const ActionsDAG::Node * node, PossiblyMonotonicChain & chain); + /// Update SortDescription (inplace) by applying ActionsDAG. /// /// Assuming that sorting properties are fulfilled for inputs, calculate sorting properties for the outputs. diff --git a/src/Processors/QueryPlan/Optimizations/filterPushDown.cpp b/src/Processors/QueryPlan/Optimizations/filterPushDown.cpp index 67653da47fae..475176db0221 100644 --- a/src/Processors/QueryPlan/Optimizations/filterPushDown.cpp +++ b/src/Processors/QueryPlan/Optimizations/filterPushDown.cpp @@ -1,6 +1,7 @@ #include #include #include +#include #include #include @@ -509,15 +510,24 @@ static size_t tryPushDownOverJoinStep(QueryPlan::Node * parent_node, QueryPlan:: const auto & left_stream_input_header = child->getInputHeaders().front(); const auto & right_stream_input_header = child->getInputHeaders().back(); - if (table_join_ptr && table_join_ptr->kind() == JoinKind::Full) - return 0; - if (logical_join && logical_join->getJoinOperator().kind == JoinKind::Full) - return 0; + JoinKind kind = JoinKind::Inner; + JoinStrictness strictness = JoinStrictness::Unspecified; + const bool have_join_kind = table_join_ptr || logical_join; + if (table_join_ptr) + { + kind = table_join_ptr->kind(); + strictness = table_join_ptr->strictness(); + } + else if (logical_join) + { + kind = logical_join->getJoinOperator().kind; + strictness = logical_join->getJoinOperator().strictness; + } - /// PASTE JOIN aligns rows from both sides by position, and pushing filters - /// to either side may change relative alignment - if ((table_join_ptr && table_join_ptr->kind() == JoinKind::Paste) - || (logical_join && logical_join->getJoinOperator().kind == JoinKind::Paste)) + /// `FULL` / `PASTE` cannot prefilter either side (`canPrefilterJoinSide`). + if (have_join_kind + && !canPrefilterJoinSide(kind, strictness, JoinTableSide::Left) + && !canPrefilterJoinSide(kind, strictness, JoinTableSide::Right)) return 0; std::unordered_map equivalent_left_stream_column_to_right_stream_column; @@ -574,6 +584,10 @@ static size_t tryPushDownOverJoinStep(QueryPlan::Node * parent_node, QueryPlan:: equivalent_expressions.append_range(std::move(extra_equivalent_expressions)); } + NameSet filter_input_names; + for (const auto * input_node : filter->getExpression().getInputs()) + filter_input_names.emplace(input_node->result_name); + auto get_available_columns_for_filter = [&](bool push_to_left_stream, bool filter_push_down_input_columns_available, bool require_stable_types = false) { Names available_input_columns_for_filter; @@ -582,11 +596,24 @@ static size_t tryPushDownOverJoinStep(QueryPlan::Node * parent_node, QueryPlan:: return available_input_columns_for_filter; const auto & input_header = push_to_left_stream ? left_stream_input_header : right_stream_input_header; - const auto & input_columns_names = input_header->getNames(); + NameSet already_added; + + auto try_add = [&](const String & name) + { + if (!already_added.insert(name).second) + return; + + available_input_columns_for_filter.push_back(name); + }; - for (const auto & name : input_columns_names) + for (const auto & name : input_header->getNames()) { - if (!join_header->has(name)) + const bool in_join_output = join_header->has(name); + + /// JOIN output may drop a left-only column (unused-column removal after + /// `count()` of `SELECT * … JOIN … WHERE left.col …`) while the Filter DAG + /// still references it. That name is still valid on this stream. + if (!in_join_output && (require_stable_types || !filter_input_names.contains(name))) continue; /// For the legacy JoinStep (not JoinStepLogical), there is no mechanism to adjust @@ -597,11 +624,44 @@ static size_t tryPushDownOverJoinStep(QueryPlan::Node * parent_node, QueryPlan:: /// /// The disjunction (partial predicate) push-down path has no such type-fixup, so it /// passes require_stable_types to also exclude type-changing columns for JoinStepLogical. - if ((!logical_join || require_stable_types) + if (in_join_output + && (!logical_join || require_stable_types) && !input_header->getByName(name).type->equals(*join_header->getByName(name).type)) continue; - available_input_columns_for_filter.push_back(name); + try_add(name); + } + + /// JoinStepLogical may alias a side's input (`bid`) to a JOIN-output / filter name + /// (`__table1.bid`). `splitActionsForJOINFilterPushDown` matches filter inputs, so + /// the output name must be listed; `fix_predicate_for_join_logical_step` remaps it. + if (logical_join) + { + for (const auto & output_action : logical_join->getOutputActions()) + { + if (push_to_left_stream ? !output_action.fromLeft() : !output_action.fromRight()) + continue; + + const auto & output_name = output_action.getColumnName(); + if (!join_header->has(output_name) && !filter_input_names.contains(output_name)) + continue; + + if (require_stable_types) + { + auto resolved = output_action.resolveAliases(); + if (resolved.getNode()->type != ActionsDAG::ActionType::INPUT + || !input_header->has(resolved.getColumnName())) + continue; + + const auto & output_type = join_header->has(output_name) + ? join_header->getByName(output_name).type + : output_action.getType(); + if (!input_header->getByName(resolved.getColumnName()).type->equals(*output_type)) + continue; + } + + try_add(output_name); + } } return available_input_columns_for_filter; @@ -610,15 +670,11 @@ static size_t tryPushDownOverJoinStep(QueryPlan::Node * parent_node, QueryPlan:: bool left_stream_filter_push_down_input_columns_available = true; bool right_stream_filter_push_down_input_columns_available = true; - if (table_join_ptr && table_join_ptr->kind() == JoinKind::Left) - right_stream_filter_push_down_input_columns_available = false; - else if (table_join_ptr && table_join_ptr->kind() == JoinKind::Right) - left_stream_filter_push_down_input_columns_available = false; - - if (logical_join && logical_join->getJoinOperator().kind == JoinKind::Left) - right_stream_filter_push_down_input_columns_available = false; - else if (logical_join && logical_join->getJoinOperator().kind == JoinKind::Right) - left_stream_filter_push_down_input_columns_available = false; + if (have_join_kind) + { + left_stream_filter_push_down_input_columns_available = canPrefilterJoinSide(kind, strictness, JoinTableSide::Left); + right_stream_filter_push_down_input_columns_available = canPrefilterJoinSide(kind, strictness, JoinTableSide::Right); + } /** `ANY INNER` join emits at most one row per key, deduplicating both sides. * Both sides are blocked: filtering the right stream can change which match is taken for the left row. @@ -637,15 +693,20 @@ static size_t tryPushDownOverJoinStep(QueryPlan::Node * parent_node, QueryPlan:: left_stream_filter_push_down_input_columns_available = false; } - /** We disable push down to right table in cases: + /** `canPrefilterJoinSide` only decides whether this side's own columns may be + * used as ordinary filter inputs (false on the null-producing outer-JOIN side). + * Equivalent-key filters can still be attached to that child. That attach is + * gated by `allow_push_down_to_right`: * 1. Right side is already filled. Example: JOIN with Dictionary. - * 2. ASOF Right join is not supported. + * 2. `ASOF` right join is not supported. */ - bool allow_push_down_to_right = join && join->allowPushDownToRight() && table_join_ptr && table_join_ptr->strictness() != JoinStrictness::Asof; + bool allow_push_down_to_right = join && join->allowPushDownToRight() && table_join_ptr + && table_join_ptr->strictness() != JoinStrictness::Asof; if (logical_join) { bool has_logical_lookup = typeid_cast(child_node->children.back()->step.get()); - allow_push_down_to_right = !has_logical_lookup && logical_join->getJoinOperator().strictness != JoinStrictness::Asof; + allow_push_down_to_right = !has_logical_lookup + && logical_join->getJoinOperator().strictness != JoinStrictness::Asof; } if (!allow_push_down_to_right) @@ -1070,22 +1131,41 @@ static size_t tryPushDownOverJoinStep(QueryPlan::Node * parent_node, QueryPlan:: return updated_steps; } - /// Unlike the main push-down above, addFilterOnTop builds the partial FilterStep directly - /// against the join input header without fix_predicate_for_join_logical_step. So a function - /// node whose type was computed for the join output (e.g. equals over a USING key widened to - /// Nullable) would be applied to the non-widened input column and trip the result-type check - /// in updateHeader. Restrict the partial predicate to columns with stable types across the join. + /// Restrict the partial predicate to columns with stable types across the join. + /// `addFilterOnTop` builds the FilterStep against the join input header, so + /// `fix_predicate_for_join_logical_step` remaps JOIN-output aliases + /// (`__table1.bid`) to the child's physical names (`bid`). Type-changing + /// columns still have no partial-path conversion, so they stay excluded. Names left_stream_stable_columns_to_push_down = get_available_columns_for_filter( true /*push_to_left_stream*/, left_stream_filter_push_down_input_columns_available, /*require_stable_types=*/true); Names right_stream_stable_columns_to_push_down = get_available_columns_for_filter( false /*push_to_left_stream*/, right_stream_filter_push_down_input_columns_available, /*require_stable_types=*/true); + auto remap_partial_predicate_for_logical_join = [&](ActionsDAG filter_dag) -> ActionsDAG + { + if (!logical_join) + return filter_dag; + + auto required_actions = get_required_pre_actions(logical_join->getOutputActions(), filter_dag.getInputs()); + if (required_actions.empty()) + return filter_dag; + + filter_dag = fix_predicate_for_join_logical_step( + std::move(filter_dag), JoinExpressionActions::getSubDAG(required_actions)); + /// `fix_predicate_for_join_logical_step` projects unused inputs as extra outputs. + /// `addFilterOnTop` expects a single filter column and rebuilds the header itself. + filter_dag.getOutputs().resize(1); + filter_dag.removeUnusedActions(); + return filter_dag; + }; + { auto left_partial_filter_dag = tryToExtractPartialPredicate(filter->getExpression(), filter->getFilterColumnName(), left_stream_stable_columns_to_push_down); if (left_partial_filter_dag.has_value()) { - const auto partial_predicate_column_name = left_partial_filter_dag->getOutputs().front()->result_name; - addFilterOnTop(*child_node, 0, nodes, std::move(*left_partial_filter_dag)); + auto remapped = remap_partial_predicate_for_logical_join(std::move(*left_partial_filter_dag)); + const auto partial_predicate_column_name = remapped.getOutputs().front()->result_name; + addFilterOnTop(*child_node, 0, nodes, std::move(remapped)); ++updated_steps; LOG_DEBUG(&Poco::Logger::get("QueryPlanOptimizations"), "Pushed down partial filter {} to the {} side of join", @@ -1098,8 +1178,9 @@ static size_t tryPushDownOverJoinStep(QueryPlan::Node * parent_node, QueryPlan:: auto right_partial_filter_dag = tryToExtractPartialPredicate(filter->getExpression(), filter->getFilterColumnName(), right_stream_stable_columns_to_push_down); if (right_partial_filter_dag.has_value()) { - const auto partial_predicate_column_name = right_partial_filter_dag->getOutputs().front()->result_name; - addFilterOnTop(*child_node, 1, nodes, std::move(*right_partial_filter_dag)); + auto remapped = remap_partial_predicate_for_logical_join(std::move(*right_partial_filter_dag)); + const auto partial_predicate_column_name = remapped.getOutputs().front()->result_name; + addFilterOnTop(*child_node, 1, nodes, std::move(remapped)); ++updated_steps; LOG_DEBUG(&Poco::Logger::get("QueryPlanOptimizations"), "Pushed down partial filter {} to the {} side of join", diff --git a/src/Processors/QueryPlan/ReadFromObjectStorageStep.cpp b/src/Processors/QueryPlan/ReadFromObjectStorageStep.cpp index a36a43bbbcc3..88df5f661570 100644 --- a/src/Processors/QueryPlan/ReadFromObjectStorageStep.cpp +++ b/src/Processors/QueryPlan/ReadFromObjectStorageStep.cpp @@ -79,7 +79,7 @@ void ReadFromObjectStorageStep::applyFilters(ActionDAGNodes added_filter_nodes) if (!filter_actions_dag) return; - if (boost::iequals(configuration->format, "Parquet") || boost::iequals(configuration->format, "ORC")) + if (boost::iequals(configuration->getFormat(), "Parquet") || boost::iequals(configuration->getFormat(), "ORC")) prepareEagerKeyConditionSets( filter_actions_dag, storage_snapshot, info.source_header, @@ -155,7 +155,7 @@ void ReadFromObjectStorageStep::initializePipeline(QueryPipelineBuilder & pipeli size_t output_ports = pipe.numOutputPorts(); const bool parallelize_output = context->getSettingsRef()[Setting::parallelize_output_from_storages]; if (parallelize_output - && FormatFactory::instance().checkParallelizeOutputAfterReading(configuration->format, context) + && FormatFactory::instance().checkParallelizeOutputAfterReading(configuration->getFormat(), context) && output_ports > 0 && output_ports < max_num_streams) pipe.resize(max_num_streams); diff --git a/src/Processors/QueryPlan/ReadFromRemote.cpp b/src/Processors/QueryPlan/ReadFromRemote.cpp index aace7e4a7f94..a427437326d1 100644 --- a/src/Processors/QueryPlan/ReadFromRemote.cpp +++ b/src/Processors/QueryPlan/ReadFromRemote.cpp @@ -611,7 +611,8 @@ void ReadFromRemote::addLazyPipe( my_stage = stage, my_storage = storage, add_agg_info, add_totals, add_extremes, async_read, async_query_sending, query_tree = shard.query_tree, planner_context = shard.planner_context, - pushed_down_filters, parallel_marshalling_threads]() mutable + pushed_down_filters, parallel_marshalling_threads, + my_is_remote_function = is_remote_function]() mutable -> QueryPipelineBuilder { auto current_settings = my_context->getSettingsRef(); @@ -723,6 +724,8 @@ void ReadFromRemote::addLazyPipe( /// Attach the shared tracker so exception-based shard skips on the lazy path are also bounded by /// `max_skip_unavailable_shards_num` / `max_skip_unavailable_shards_ratio`, like the non-lazy path. remote_query_executor->setUnavailableShardTracker(my_unavailable_shard_tracker); + remote_query_executor->setRemoteFunction(my_is_remote_function); + remote_query_executor->setShardCount(my_shard_count); auto pipe = createRemoteSourcePipe( remote_query_executor, add_agg_info, add_totals, add_extremes, async_read, async_query_sending, parallel_marshalling_threads); @@ -816,6 +819,8 @@ void ReadFromRemote::addPipe( remote_query_executor->setPoolMode(PoolMode::GET_ONE); remote_query_executor->setDistributedFanout(shards.size() * shard.shard_info.per_replica_pools.size()); remote_query_executor->setUnavailableShardTracker(unavailable_shard_tracker); + remote_query_executor->setRemoteFunction(is_remote_function); + remote_query_executor->setShardCount(shard_count); if (!table_func_ptr) remote_query_executor->setMainTable(shard.main_table ? shard.main_table : main_table); @@ -846,6 +851,8 @@ void ReadFromRemote::addPipe( remote_query_executor->setLogger(log); remote_query_executor->setDistributedFanout(shards.size()); remote_query_executor->setUnavailableShardTracker(unavailable_shard_tracker); + remote_query_executor->setRemoteFunction(is_remote_function); + remote_query_executor->setShardCount(shard_count); // Several connections to a shard are correct only when every replica reads its own part of the data, // which is the case only for the offset based modes (`SAMPLING_KEY`, `CUSTOM_KEY_SAMPLING`, diff --git a/src/Processors/QueryPlan/ReadFromRemote.h b/src/Processors/QueryPlan/ReadFromRemote.h index afa452c10620..1be0509f9ac8 100644 --- a/src/Processors/QueryPlan/ReadFromRemote.h +++ b/src/Processors/QueryPlan/ReadFromRemote.h @@ -51,6 +51,7 @@ class ReadFromRemote final : public SourceStepWithFilterBase void enableMemoryBoundMerging(); void enforceAggregationInOrder(const SortDescription & sort_description); + void setIsRemoteFunction(bool is_remote_function_ = true) { is_remote_function = is_remote_function_; } bool hasSerializedPlan() const; @@ -69,6 +70,7 @@ class ReadFromRemote final : public SourceStepWithFilterBase const String cluster_name; UnavailableShardTrackerPtr unavailable_shard_tracker; std::optional priority_func_factory; + bool is_remote_function = false; Pipes addPipes(const ClusterProxy::SelectStreamFactory::Shards & used_shards, const SharedHeader & out_header); diff --git a/src/Processors/Sources/ConstChunkGenerator.h b/src/Processors/Sources/ConstChunkGenerator.h index 7f9a2b84abfa..1807f792c45b 100644 --- a/src/Processors/Sources/ConstChunkGenerator.h +++ b/src/Processors/Sources/ConstChunkGenerator.h @@ -1,6 +1,7 @@ #pragma once #include +#include namespace DB @@ -13,7 +14,7 @@ class ConstChunkGenerator final : public ISource public: ConstChunkGenerator(SharedHeader header, size_t total_num_rows, size_t max_block_size_) : ISource(std::move(header)) - , remaining_rows(total_num_rows), max_block_size(max_block_size_) + , generated_rows(0), remaining_rows(total_num_rows), max_block_size(max_block_size_) { } @@ -27,10 +28,14 @@ class ConstChunkGenerator final : public ISource size_t num_rows = std::min(max_block_size, remaining_rows); remaining_rows -= num_rows; - return cloneConstWithDefault(Chunk{getPort().getHeader().getColumns(), 0}, num_rows); + auto chunk = cloneConstWithDefault(Chunk{getPort().getHeader().getColumns(), 0}, num_rows); + chunk.getChunkInfos().add(std::make_shared(generated_rows)); + generated_rows += num_rows; + return chunk; } private: + size_t generated_rows; size_t remaining_rows; size_t max_block_size; }; diff --git a/src/QueryPipeline/RemoteQueryExecutor.cpp b/src/QueryPipeline/RemoteQueryExecutor.cpp index 87bce9b87531..b6bf2f73d840 100644 --- a/src/QueryPipeline/RemoteQueryExecutor.cpp +++ b/src/QueryPipeline/RemoteQueryExecutor.cpp @@ -497,7 +497,16 @@ void RemoteQueryExecutor::sendQueryUnlocked(ClientInfo::QueryKind query_kind, As auto timeouts = ConnectionTimeouts::getTCPTimeoutsWithFailover(settings); ClientInfo modified_client_info = context->getClientInfo(); - modified_client_info.query_kind = query_kind; + + /// Doesn't support now "remote('1.1.1.{1,2}')"" + if (is_remote_function && (shard_count == 1)) + { + modified_client_info.setInitialQuery(); + modified_client_info.client_name = "ClickHouse server"; + modified_client_info.interface = ClientInfo::Interface::TCP; + } + else + modified_client_info.query_kind = query_kind; /// A distributed query must carry a known initiator version: the receiving server uses it for /// version-gated compatibility decisions (e.g. whether to enable the analyzer, see `TCPHandler`). @@ -758,6 +767,8 @@ RemoteQueryExecutor::ReadResult RemoteQueryExecutor::processPacket(Packet packet if (!packet.block.empty() && (packet.block.rows() > 0)) { got_data_from_replica = true; + if (extension && extension->replica_info) + replica_has_processed_data.insert(extension->replica_info->number_of_current_replica); return ReadResult(adaptBlockStructure(packet.block, *header)); } break; /// If the block is empty - we will receive other packets before EndOfStream. @@ -838,6 +849,19 @@ RemoteQueryExecutor::ReadResult RemoteQueryExecutor::processPacket(Packet packet case Protocol::Server::TimezoneUpdate: break; + case Protocol::Server::ConnectionLost: + if (extension && extension->task_iterator && extension->task_iterator->supportRerunTask() && extension->replica_info) + { + if (!replica_has_processed_data.contains(extension->replica_info->number_of_current_replica)) + { + finished = true; + extension->task_iterator->rescheduleTasksFromReplica(extension->replica_info->number_of_current_replica); + return ReadResult(Block{}); + } + } + packet.exception->rethrow(); + break; + default: got_unknown_packet_from_replica = true; throw Exception( @@ -1216,6 +1240,11 @@ void RemoteQueryExecutor::setProfileInfoCallback(ProfileInfoCallback callback) profile_info_callback = std::move(callback); } +bool RemoteQueryExecutor::skipUnavailableShards() const +{ + return context->getSettingsRef()[Setting::skip_unavailable_shards]; +} + bool RemoteQueryExecutor::needToSkipUnavailableShard() { if (context->getSettingsRef()[Setting::skip_unavailable_shards] && (0 == connections->size())) diff --git a/src/QueryPipeline/RemoteQueryExecutor.h b/src/QueryPipeline/RemoteQueryExecutor.h index cbeda47e98c9..8cd35f2a18fa 100644 --- a/src/QueryPipeline/RemoteQueryExecutor.h +++ b/src/QueryPipeline/RemoteQueryExecutor.h @@ -37,7 +37,22 @@ class RemoteQueryExecutorReadContext; class ParallelReplicasReadingCoordinator; -using TaskIterator = std::function; +namespace ErrorCodes +{ + extern const int NOT_IMPLEMENTED; +}; + +class TaskIterator +{ +public: + virtual ~TaskIterator() = default; + virtual bool supportRerunTask() const { return false; } + virtual void rescheduleTasksFromReplica(size_t /* number_of_current_replica */) + { + throw Exception(ErrorCodes::NOT_IMPLEMENTED, "Method rescheduleTasksFromReplica is not implemented"); + } + virtual ClusterFunctionReadTaskResponsePtr operator()(size_t number_of_current_replica) const = 0; +}; /// This class allows one to launch queries on remote replicas of one shard and get results class RemoteQueryExecutor @@ -219,11 +234,17 @@ class RemoteQueryExecutor void setDistributedFanout(size_t total_connections) { distributed_fanout = total_connections; } + void setRemoteFunction(bool is_remote_function_ = true) { is_remote_function = is_remote_function_; } + + void setShardCount(UInt32 shard_count_) { shard_count = shard_count_; } + const Block & getHeader() const { return *header; } const SharedHeader & getSharedHeader() const { return header; } IConnections & getConnections() { return *connections; } + bool skipUnavailableShards() const; + bool needToSkipUnavailableShard(); /// Reports a skipped shard to `unavailable_shard_tracker` (if any), enforcing the @@ -339,6 +360,9 @@ class RemoteQueryExecutor bool packet_in_progress = false; #endif + bool is_remote_function = false; + UInt32 shard_count = 0; + PoolMode pool_mode = PoolMode::GET_MANY; StorageID main_table = StorageID::createEmpty(); @@ -354,6 +378,8 @@ class RemoteQueryExecutor const bool read_packet_type_separately = false; + std::unordered_set replica_has_processed_data; + /// Send all scalars to remote servers void sendScalars(); diff --git a/src/QueryPipeline/RemoteQueryExecutorReadContext.cpp b/src/QueryPipeline/RemoteQueryExecutorReadContext.cpp index dc8769562707..ae8bdedaed75 100644 --- a/src/QueryPipeline/RemoteQueryExecutorReadContext.cpp +++ b/src/QueryPipeline/RemoteQueryExecutorReadContext.cpp @@ -21,10 +21,13 @@ namespace ErrorCodes extern const int CANNOT_READ_FROM_SOCKET; extern const int CANNOT_OPEN_FILE; extern const int SOCKET_TIMEOUT; + extern const int ATTEMPT_TO_READ_AFTER_EOF; } RemoteQueryExecutorReadContext::RemoteQueryExecutorReadContext( - RemoteQueryExecutor & executor_, bool suspend_when_query_sent_, bool read_packet_type_separately_) + RemoteQueryExecutor & executor_, + bool suspend_when_query_sent_, + bool read_packet_type_separately_) : AsyncTaskExecutor(std::make_unique(*this), "RemoteQueryExecutorReadContext") , executor(executor_) , suspend_when_query_sent(suspend_when_query_sent_) @@ -71,19 +74,42 @@ void RemoteQueryExecutorReadContext::Task::run(AsyncCallback async_callback, Sus if (read_context.executor.needToSkipUnavailableShard()) return; - while (true) + try { - read_context.has_read_packet_part = PacketPart::None; + while (true) + { + read_context.has_read_packet_part = PacketPart::None; + + if (read_context.read_packet_type_separately) + { + read_context.packet.type = read_context.executor.getConnections().receivePacketTypeUnlocked(async_callback); + read_context.has_read_packet_part = PacketPart::Type; + suspend_callback(); + } + read_context.packet = read_context.executor.getConnections().receivePacketUnlocked(async_callback); + read_context.has_read_packet_part = PacketPart::Body; + if (read_context.packet.type == Protocol::Server::Data && read_context.packet.block.rows() > 0) + read_context.has_data_packets = true; - if (read_context.read_packet_type_separately) + suspend_callback(); + } + } + catch (const Exception & e) + { + /// If cluster node unxepectedly shutted down (kill/segfault/power off/etc.) socket just closes. + /// If initiator did not process any data packets before, this fact can be ignored. + /// Unprocessed tasks will be executed on other nodes. + if (e.code() == ErrorCodes::ATTEMPT_TO_READ_AFTER_EOF + && !read_context.has_data_packets.load() + && read_context.executor.skipUnavailableShards()) { - read_context.packet.type = read_context.executor.getConnections().receivePacketTypeUnlocked(async_callback); - read_context.has_read_packet_part = PacketPart::Type; + read_context.packet.type = Protocol::Server::ConnectionLost; + read_context.packet.exception = std::make_unique(getCurrentExceptionMessageAndPattern(true), getCurrentExceptionCode()); + read_context.has_read_packet_part = PacketPart::Body; suspend_callback(); } - read_context.packet = read_context.executor.getConnections().receivePacketUnlocked(async_callback); - read_context.has_read_packet_part = PacketPart::Body; - suspend_callback(); + else + throw; } } diff --git a/src/QueryPipeline/RemoteQueryExecutorReadContext.h b/src/QueryPipeline/RemoteQueryExecutorReadContext.h index 6baa2d3916dc..ad654457f012 100644 --- a/src/QueryPipeline/RemoteQueryExecutorReadContext.h +++ b/src/QueryPipeline/RemoteQueryExecutorReadContext.h @@ -26,7 +26,9 @@ class RemoteQueryExecutorReadContext : public AsyncTaskExecutor { public: explicit RemoteQueryExecutorReadContext( - RemoteQueryExecutor & executor_, bool suspend_when_query_sent_, bool read_packet_type_separately_); + RemoteQueryExecutor & executor_, + bool suspend_when_query_sent_, + bool read_packet_type_separately_); ~RemoteQueryExecutorReadContext() override; @@ -85,6 +87,7 @@ class RemoteQueryExecutorReadContext : public AsyncTaskExecutor /// None -> Type -> Body -> None /// None -> Body -> None std::atomic has_read_packet_part = PacketPart::None; + std::atomic_bool has_data_packets = false; Packet packet; RemoteQueryExecutor & executor; diff --git a/src/Server/TCPHandler.cpp b/src/Server/TCPHandler.cpp index 18d9711bf0c5..c6884f08a95b 100644 --- a/src/Server/TCPHandler.cpp +++ b/src/Server/TCPHandler.cpp @@ -28,6 +28,7 @@ #include #include #include +#include #include #include #include @@ -38,7 +39,6 @@ #include #include #include -#include #include #include #include diff --git a/src/Storages/Cache/ObjectStorageListObjectsCache.cpp b/src/Storages/Cache/ObjectStorageListObjectsCache.cpp new file mode 100644 index 000000000000..78b73381c843 --- /dev/null +++ b/src/Storages/Cache/ObjectStorageListObjectsCache.cpp @@ -0,0 +1,226 @@ +#include +#include +#include +#include + +namespace ProfileEvents +{ +extern const Event ObjectStorageListObjectsCacheHits; +extern const Event ObjectStorageListObjectsCacheMisses; +extern const Event ObjectStorageListObjectsCacheExactMatchHits; +extern const Event ObjectStorageListObjectsCachePrefixMatchHits; +} + +namespace DB +{ + +template +class ObjectStorageListObjectsCachePolicy : public TTLCachePolicy +{ +public: + using BasePolicy = TTLCachePolicy; + using typename BasePolicy::MappedPtr; + using typename BasePolicy::KeyMapped; + using BasePolicy::cache; + + ObjectStorageListObjectsCachePolicy() + : BasePolicy(CurrentMetrics::end(), CurrentMetrics::end(), std::make_unique()) + { + } + + std::optional getWithKey(const Key & key) override + { + if (const auto it = cache.find(key); it != cache.end()) + { + if (!IsStaleFunction()(it->first)) + { + return std::make_optional({it->first, it->second}); + } + // found a stale entry, remove it but don't return. We still want to perform the prefix matching search + BasePolicy::remove(it->first); + } + + if (const auto it = findBestMatchingPrefixAndRemoveExpiredEntries(key); it != cache.end()) + { + return std::make_optional({it->first, it->second}); + } + + return std::nullopt; + } + +private: + auto findBestMatchingPrefixAndRemoveExpiredEntries(Key key) + { + while (!key.prefix.empty()) + { + if (const auto it = cache.find(key); it != cache.end()) + { + if (IsStaleFunction()(it->first)) + { + BasePolicy::remove(it->first); + } + else + { + return it; + } + } + + key.prefix.pop_back(); + } + + return cache.end(); + } +}; + +ObjectStorageListObjectsCache::Key::Key( + const String & storage_description_, + const String & bucket_, + const String & prefix_, + bool with_tags_, + const std::chrono::steady_clock::time_point & expires_at_, + std::optional user_id_) + : storage_description(storage_description_), bucket(bucket_), prefix(prefix_), with_tags(with_tags_), expires_at(expires_at_), user_id(user_id_) {} + +bool ObjectStorageListObjectsCache::Key::operator==(const Key & other) const +{ + return storage_description == other.storage_description && bucket == other.bucket && prefix == other.prefix && with_tags == other.with_tags; +} + +size_t ObjectStorageListObjectsCache::KeyHasher::operator()(const Key & key) const +{ + std::size_t seed = 0; + + boost::hash_combine(seed, key.storage_description); + boost::hash_combine(seed, key.bucket); + boost::hash_combine(seed, key.prefix); + boost::hash_combine(seed, key.with_tags); + + return seed; +} + +bool ObjectStorageListObjectsCache::IsStale::operator()(const Key & key) const +{ + return key.expires_at < std::chrono::steady_clock::now(); +} + +size_t ObjectStorageListObjectsCache::WeightFunction::operator()(const Value & value) const +{ + std::size_t weight = 0; + + for (const auto & object : value) + { + const auto object_metadata = object->getObjectMetadata(); + + weight += object->getPath().capacity() + sizeof(object_metadata); + + // variable size + if (object_metadata) + { + weight += object_metadata->etag.capacity(); + weight += object_metadata->attributes.size() * (sizeof(std::string) * 2); + + for (const auto & [k, v] : object_metadata->attributes) + { + weight += k.capacity() + v.capacity(); + } + + for (const auto & [k, v] : object_metadata->tags) + { + weight += k.capacity() + v.capacity(); + } + } + } + + return weight; +} + +ObjectStorageListObjectsCache::ObjectStorageListObjectsCache() + : cache(std::make_unique>()) +{ +} + +void ObjectStorageListObjectsCache::set( + const Key & key, + const std::shared_ptr & value) +{ + auto key_with_ttl = key; + + if (ttl_in_seconds == 0) + { + key_with_ttl.expires_at = std::chrono::steady_clock::time_point::max(); + } + else + { + key_with_ttl.expires_at = std::chrono::steady_clock::now() + std::chrono::seconds(ttl_in_seconds); + } + + cache.set(key_with_ttl, value); +} + +void ObjectStorageListObjectsCache::clear() +{ + cache.clear(); +} + +std::optional ObjectStorageListObjectsCache::get(const Key & key, bool filter_by_prefix) +{ + const auto pair = cache.getWithKey(key); + + if (!pair) + { + ProfileEvents::increment(ProfileEvents::ObjectStorageListObjectsCacheMisses); + return {}; + } + + ProfileEvents::increment(ProfileEvents::ObjectStorageListObjectsCacheHits); + + if (pair->key == key) + { + ProfileEvents::increment(ProfileEvents::ObjectStorageListObjectsCacheExactMatchHits); + return *pair->mapped; + } + + ProfileEvents::increment(ProfileEvents::ObjectStorageListObjectsCachePrefixMatchHits); + + if (!filter_by_prefix) + { + return *pair->mapped; + } + + Value filtered_objects; + + filtered_objects.reserve(pair->mapped->size()); + + for (const auto & object : *pair->mapped) + { + if (object->getPath().starts_with(key.prefix)) + { + filtered_objects.push_back(object); + } + } + + return filtered_objects; +} + +void ObjectStorageListObjectsCache::setMaxSizeInBytes(std::size_t size_in_bytes_) +{ + cache.setMaxSizeInBytes(size_in_bytes_); +} + +void ObjectStorageListObjectsCache::setMaxCount(std::size_t count) +{ + cache.setMaxCount(count); +} + +void ObjectStorageListObjectsCache::setTTL(std::size_t ttl_in_seconds_) +{ + ttl_in_seconds = ttl_in_seconds_; +} + +ObjectStorageListObjectsCache & ObjectStorageListObjectsCache::instance() +{ + static ObjectStorageListObjectsCache instance; + return instance; +} + +} diff --git a/src/Storages/Cache/ObjectStorageListObjectsCache.h b/src/Storages/Cache/ObjectStorageListObjectsCache.h new file mode 100644 index 000000000000..ab1eec7efb4e --- /dev/null +++ b/src/Storages/Cache/ObjectStorageListObjectsCache.h @@ -0,0 +1,80 @@ +#pragma once + +#include +#include +#include +#include + +namespace DB +{ + +class ObjectStorageListObjectsCache +{ + friend class ObjectStorageListObjectsCacheTest; +public: + ObjectStorageListObjectsCache(const ObjectStorageListObjectsCache &) = delete; + ObjectStorageListObjectsCache(ObjectStorageListObjectsCache &&) noexcept = delete; + + ObjectStorageListObjectsCache& operator=(const ObjectStorageListObjectsCache &) = delete; + ObjectStorageListObjectsCache& operator=(ObjectStorageListObjectsCache &&) noexcept = delete; + + static ObjectStorageListObjectsCache & instance(); + + struct Key + { + Key( + const String & storage_description_, + const String & bucket_, + const String & prefix_, + bool with_tags_, + const std::chrono::steady_clock::time_point & expires_at_ = std::chrono::steady_clock::now(), + std::optional user_id_ = std::nullopt); + + std::string storage_description; + std::string bucket; + std::string prefix; + bool with_tags; + std::chrono::steady_clock::time_point expires_at; + std::optional user_id; + + bool operator==(const Key & other) const; + }; + + using Value = ObjectInfos; + struct KeyHasher + { + size_t operator()(const Key & key) const; + }; + + struct IsStale + { + bool operator()(const Key & key) const; + }; + + struct WeightFunction + { + size_t operator()(const Value & value) const; + }; + + using Cache = CacheBase; + + void set( + const Key & key, + const std::shared_ptr & value); + + std::optional get(const Key & key, bool filter_by_prefix = true); + + void clear(); + + void setMaxSizeInBytes(std::size_t size_in_bytes_); + void setMaxCount(std::size_t count); + void setTTL(std::size_t ttl_in_seconds_); + +private: + ObjectStorageListObjectsCache(); + + Cache cache; + size_t ttl_in_seconds {0}; +}; + +} diff --git a/src/Storages/Cache/tests/gtest_object_storage_list_objects_cache.cpp b/src/Storages/Cache/tests/gtest_object_storage_list_objects_cache.cpp new file mode 100644 index 000000000000..8eb45d520c96 --- /dev/null +++ b/src/Storages/Cache/tests/gtest_object_storage_list_objects_cache.cpp @@ -0,0 +1,244 @@ +#include +#include +#include +#include + +namespace DB +{ + +class ObjectStorageListObjectsCacheTest : public ::testing::Test +{ +protected: + void SetUp() override + { + cache = std::unique_ptr(new ObjectStorageListObjectsCache()); + cache->setTTL(3); + cache->setMaxCount(100); + cache->setMaxSizeInBytes(1000000); + } + + std::unique_ptr cache; + static ObjectStorageListObjectsCache::Key default_key; + + static std::shared_ptr createTestValue(const std::vector& paths) + { + auto value = std::make_shared(); + for (const auto & path : paths) + { + value->push_back(std::make_shared(path)); + } + return value; + } +}; + +ObjectStorageListObjectsCache::Key ObjectStorageListObjectsCacheTest::default_key {"default", "test-bucket", "test-prefix/", false}; + +TEST_F(ObjectStorageListObjectsCacheTest, BasicSetAndGet) +{ + cache->clear(); + auto value = createTestValue({"test-prefix/file1.txt", "test-prefix/file2.txt"}); + + cache->set(default_key, value); + + auto result = cache->get(default_key).value(); + + ASSERT_EQ(result.size(), 2); + EXPECT_EQ(result[0]->getPath(), "test-prefix/file1.txt"); + EXPECT_EQ(result[1]->getPath(), "test-prefix/file2.txt"); +} + +TEST_F(ObjectStorageListObjectsCacheTest, CacheMiss) +{ + cache->clear(); + + EXPECT_FALSE(cache->get(default_key)); +} + +TEST_F(ObjectStorageListObjectsCacheTest, ClearCache) +{ + cache->clear(); + auto value = createTestValue({"test-prefix/file1.txt", "test-prefix/file2.txt"}); + + cache->set(default_key, value); + cache->clear(); + + EXPECT_FALSE(cache->get(default_key)); +} + +TEST_F(ObjectStorageListObjectsCacheTest, PrefixMatching) +{ + cache->clear(); + + auto short_prefix_key = default_key; + short_prefix_key.prefix = "parent/"; + + auto mid_prefix_key = default_key; + mid_prefix_key.prefix = "parent/child/"; + + auto long_prefix_key = default_key; + long_prefix_key.prefix = "parent/child/grandchild/"; + + auto value = createTestValue( + { + "parent/child/grandchild/file1.txt", + "parent/child/grandchild/file2.txt"}); + + cache->set(mid_prefix_key, value); + + auto result1 = cache->get(mid_prefix_key).value(); + EXPECT_EQ(result1.size(), 2); + + auto result2 = cache->get(long_prefix_key).value(); + EXPECT_EQ(result2.size(), 2); + + EXPECT_FALSE(cache->get(short_prefix_key)); +} + +TEST_F(ObjectStorageListObjectsCacheTest, PrefixFiltering) +{ + cache->clear(); + + auto key_with_short_prefix = default_key; + key_with_short_prefix.prefix = "parent/"; + + auto key_with_mid_prefix = default_key; + key_with_mid_prefix.prefix = "parent/child1/"; + + auto value = createTestValue({ + "parent/file1.txt", + "parent/child1/file2.txt", + "parent/child2/file3.txt" + }); + + cache->set(key_with_short_prefix, value); + + auto result = cache->get(key_with_mid_prefix, true).value(); + EXPECT_EQ(result.size(), 1); + EXPECT_EQ(result[0]->getPath(), "parent/child1/file2.txt"); +} + +TEST_F(ObjectStorageListObjectsCacheTest, TTLExpiration) +{ + cache->clear(); + auto value = createTestValue({"test-prefix/file1.txt"}); + + cache->set(default_key, value); + + // Verify we can get it immediately + auto result1 = cache->get(default_key).value(); + EXPECT_EQ(result1.size(), 1); + + std::this_thread::sleep_for(std::chrono::seconds(4)); + + EXPECT_FALSE(cache->get(default_key)); +} + +TEST_F(ObjectStorageListObjectsCacheTest, TTLUnlimited) +{ + cache->clear(); + cache->setTTL(0); // 0 means unlimited + auto value = createTestValue({"test-prefix/file1.txt"}); + + cache->set(default_key, value); + + // Verify we can get it immediately + auto result1 = cache->get(default_key).value(); + EXPECT_EQ(result1.size(), 1); + + // Sleep for a reasonable amount (longer than the default 3 second TTL from SetUp) + std::this_thread::sleep_for(std::chrono::seconds(5)); + + // Should still be available since TTL is unlimited + auto result2 = cache->get(default_key).value(); + EXPECT_EQ(result2.size(), 1); + EXPECT_EQ(result2[0]->getPath(), "test-prefix/file1.txt"); +} + +TEST_F(ObjectStorageListObjectsCacheTest, TTLSwitchFromUnlimitedToFinite) +{ + cache->clear(); + cache->setTTL(0); // Start with unlimited + auto value1 = createTestValue({"test-prefix/file1.txt"}); + auto key1 = default_key; + key1.prefix = "unlimited/"; + + cache->set(key1, value1); + + // Switch to finite TTL and add another entry + cache->setTTL(1); + auto value2 = createTestValue({"test-prefix/file2.txt"}); + auto key2 = default_key; + key2.prefix = "finite/"; + + cache->set(key2, value2); + + // Verify both are available immediately + EXPECT_TRUE(cache->get(key1).has_value()); + EXPECT_TRUE(cache->get(key2).has_value()); + + // Wait for finite TTL entry to expire + std::this_thread::sleep_for(std::chrono::seconds(2)); + + // Unlimited entry should still be there, finite should be gone + EXPECT_TRUE(cache->get(key1).has_value()); + EXPECT_FALSE(cache->get(key2).has_value()); +} + +TEST_F(ObjectStorageListObjectsCacheTest, BestPrefixMatch) +{ + cache->clear(); + + auto short_prefix_key = default_key; + short_prefix_key.prefix = "a/b/"; + + auto mid_prefix_key = default_key; + mid_prefix_key.prefix = "a/b/c/"; + + auto long_prefix_key = default_key; + long_prefix_key.prefix = "a/b/c/d/"; + + auto short_prefix = createTestValue({"a/b/c/d/file1.txt", "a/b/c/file1.txt", "a/b/file2.txt"}); + auto mid_prefix = createTestValue({"a/b/c/d/file1.txt", "a/b/c/file1.txt"}); + + cache->set(short_prefix_key, short_prefix); + cache->set(mid_prefix_key, mid_prefix); + + // should pick mid_prefix, which has size 2. filter_by_prefix=false so we can assert by size + auto result = cache->get(long_prefix_key, false).value(); + EXPECT_EQ(result.size(), 2u); +} + +TEST_F(ObjectStorageListObjectsCacheTest, WithTags) +{ + cache->clear(); + + auto key_with_tags = default_key; + key_with_tags.with_tags = true; + + auto value_with_tags = createTestValue({"test.txt"}); + + cache->set(key_with_tags, value_with_tags); + + /// we have set with tags, we should be able to retrieve it + auto result_with_tags = cache->get(key_with_tags).value(); + EXPECT_EQ(result_with_tags.size(), 1u); + EXPECT_EQ(result_with_tags[0]->getPath(), "test.txt"); + + /// querying by a key without tags should return nothing + auto result_without_tags = cache->get(default_key); + EXPECT_FALSE(result_without_tags.has_value()); + + cache->clear(); + + cache->set(default_key, value_with_tags); + + /// querying by a key with tags should return nothing + EXPECT_FALSE(cache->get(key_with_tags).has_value()); + + /// querying by a key without tags should return the value + auto result_without_tags_2 = cache->get(default_key).value(); + EXPECT_EQ(result_without_tags_2.size(), 1u); + EXPECT_EQ(result_without_tags_2[0]->getPath(), "test.txt"); +} + +} diff --git a/src/Storages/ColumnsDescription.cpp b/src/Storages/ColumnsDescription.cpp index c04428ca5ca9..9fab15c72315 100644 --- a/src/Storages/ColumnsDescription.cpp +++ b/src/Storages/ColumnsDescription.cpp @@ -613,6 +613,15 @@ NamesAndTypesList ColumnsDescription::getInsertable() const return ret; } +NamesAndTypesList ColumnsDescription::getReadable() const +{ + NamesAndTypesList ret; + for (const auto & col : columns) + if (col.default_desc.kind != ColumnDefaultKind::Ephemeral) + ret.emplace_back(col.name, col.type); + return ret; +} + NamesAndTypesList ColumnsDescription::getMaterialized() const { NamesAndTypesList ret; @@ -1013,7 +1022,6 @@ std::optional ColumnsDescription::getDefault(const String & colum return {}; } - bool ColumnsDescription::hasCompressionCodec(const String & column_name) const { const auto it = columns.get<1>().find(column_name); diff --git a/src/Storages/ColumnsDescription.h b/src/Storages/ColumnsDescription.h index ed8e967004b2..757ff1c26eb8 100644 --- a/src/Storages/ColumnsDescription.h +++ b/src/Storages/ColumnsDescription.h @@ -175,6 +175,7 @@ class ColumnsDescription : public IHints<> NamesAndTypesList getOrdinary() const; NamesAndTypesList getMaterialized() const; NamesAndTypesList getInsertable() const; /// ordinary + ephemeral + NamesAndTypesList getReadable() const; /// ordinary + materialized + aliases (no ephemeral) NamesAndTypesList getAliases() const; NamesAndTypesList getEphemeral() const; NamesAndTypesList getAllPhysical() const; /// ordinary + materialized. diff --git a/src/Storages/ExportReplicatedMergeTreePartitionManifest.h b/src/Storages/ExportReplicatedMergeTreePartitionManifest.h new file mode 100644 index 000000000000..7857302b1261 --- /dev/null +++ b/src/Storages/ExportReplicatedMergeTreePartitionManifest.h @@ -0,0 +1,410 @@ +#pragma once + +#include +#include +#include +#include +#include +#include +#include +#include + +namespace DB +{ + +struct ExportReplicatedMergeTreePartitionProcessingPartEntry +{ + + enum class Status + { + PENDING, + COMPLETED, + FAILED + }; + + String part_name; + Status status; + String finished_by; + + std::string toJsonString() const + { + Poco::JSON::Object json; + + json.set("part_name", part_name); + json.set("status", String(magic_enum::enum_name(status))); + json.set("finished_by", finished_by); + std::ostringstream oss; // STYLE_CHECK_ALLOW_STD_STRING_STREAM + oss.exceptions(std::ios::failbit); + Poco::JSON::Stringifier::stringify(json, oss); + + return oss.str(); + } + + static ExportReplicatedMergeTreePartitionProcessingPartEntry fromJsonString(const std::string & json_string) + { + Poco::JSON::Parser parser; + auto json = parser.parse(json_string).extract(); + chassert(json); + + ExportReplicatedMergeTreePartitionProcessingPartEntry entry; + + entry.part_name = json->getValue("part_name"); + entry.status = magic_enum::enum_cast(json->getValue("status")).value(); + if (json->has("finished_by")) + { + entry.finished_by = json->getValue("finished_by"); + } + return entry; + } +}; + +/// Per-task "last exception" record persisted at /last_exception. +/// +/// Single znode per export task. Updated atomically with the surrounding state +/// transition (status flip / lock release / retry counter bump) via a single +/// `tryMulti` Set op. The `count` field is best-effort and non-atomic: writers +/// `tryGet` the current value and write `count + 1` back without a version +/// check, so concurrent writers may under-count. This matches the semantics +/// used by `commit_attempts` and is documented in the system table. +struct LastExceptionEntry +{ + String message; + String part; /// empty for task-level exceptions (commit failure, timeout) + String replica; + time_t time = 0; + size_t count = 0; + + std::string toJsonString() const + { + Poco::JSON::Object json; + json.set("message", message); + json.set("part", part); + json.set("replica", replica); + json.set("time", time); + json.set("count", count); + std::ostringstream oss; // STYLE_CHECK_ALLOW_STD_STRING_STREAM + oss.exceptions(std::ios::failbit); + Poco::JSON::Stringifier::stringify(json, oss); + return oss.str(); + } + + static LastExceptionEntry fromJsonString(const std::string & json_string) + { + LastExceptionEntry entry; + if (json_string.empty()) + return entry; + + Poco::JSON::Parser parser; + auto json = parser.parse(json_string).extract(); + chassert(json); + + if (json->has("message")) + entry.message = json->getValue("message"); + if (json->has("part")) + entry.part = json->getValue("part"); + if (json->has("replica")) + entry.replica = json->getValue("replica"); + if (json->has("time")) + entry.time = json->getValue("time"); + if (json->has("count")) + entry.count = json->getValue("count"); + return entry; + } +}; + +struct ExportReplicatedMergeTreePartitionProcessedPartEntry +{ + String part_name; + std::vector paths_in_destination; + String finished_by; + + std::string toJsonString() const + { + Poco::JSON::Object json; + json.set("part_name", part_name); + json.set("paths_in_destination", paths_in_destination); + json.set("finished_by", finished_by); + std::ostringstream oss; // STYLE_CHECK_ALLOW_STD_STRING_STREAM + oss.exceptions(std::ios::failbit); + Poco::JSON::Stringifier::stringify(json, oss); + return oss.str(); + } + + static ExportReplicatedMergeTreePartitionProcessedPartEntry fromJsonString(const std::string & json_string) + { + Poco::JSON::Parser parser; + auto json = parser.parse(json_string).extract(); + chassert(json); + + ExportReplicatedMergeTreePartitionProcessedPartEntry entry; + + entry.part_name = json->getValue("part_name"); + + const auto paths_in_destination_array = json->getArray("paths_in_destination"); + for (size_t i = 0; i < paths_in_destination_array->size(); ++i) + entry.paths_in_destination.emplace_back(paths_in_destination_array->getElement(static_cast(i))); + + entry.finished_by = json->getValue("finished_by"); + + return entry; + } +}; + +/// Per-task "commit info" record persisted at /commit_info. +/// +/// Written exactly once, atomically with the status -> COMPLETED transition +/// (see ExportPartitionUtils::commit). Captures the metadata-layer file paths +/// produced by the destination storage during commit so they can be surfaced in +/// system.replicated_partition_exports for debugging. +/// +/// All Iceberg fields are empty for non-Iceberg destinations. They may also be +/// empty for an Iceberg destination if the committing replica crashed between +/// writing the object-storage files and writing this znode; in that case the +/// task still transitions to COMPLETED via the recovery path but commit_info +/// remains absent. This is best-effort observability and acceptable. +struct ExportReplicatedMergeTreePartitionCommitInfoEntry +{ + /// Iceberg: path (in destination object storage) of the new vN.metadata.json + /// written by the commit. + String iceberg_metadata_file; + + /// Iceberg: path of the snap---.avro manifest list + /// referenced by the new snapshot. + String iceberg_manifest_list; + + /// Iceberg: path of the manifest entry file (*.avro) referenced by the + /// manifest list. + String iceberg_manifest_file; + + /// Plain object storage: path of the commit marker file written by + /// StorageObjectStorage::commitExportPartitionTransaction. Empty for Iceberg. + String commit_marker_file; + + std::string toJsonString() const + { + Poco::JSON::Object json; + json.set("iceberg_metadata_file", iceberg_metadata_file); + json.set("iceberg_manifest_list", iceberg_manifest_list); + json.set("iceberg_manifest_file", iceberg_manifest_file); + json.set("commit_marker_file", commit_marker_file); + + std::ostringstream oss; // STYLE_CHECK_ALLOW_STD_STRING_STREAM + oss.exceptions(std::ios::failbit); + Poco::JSON::Stringifier::stringify(json, oss); + return oss.str(); + } + + static ExportReplicatedMergeTreePartitionCommitInfoEntry fromJsonString(const std::string & json_string) + { + ExportReplicatedMergeTreePartitionCommitInfoEntry entry; + if (json_string.empty()) + return entry; + + Poco::JSON::Parser parser; + auto json = parser.parse(json_string).extract(); + + if (json->has("iceberg_metadata_file")) + entry.iceberg_metadata_file = json->getValue("iceberg_metadata_file"); + if (json->has("iceberg_manifest_list")) + entry.iceberg_manifest_list = json->getValue("iceberg_manifest_list"); + + if (json->has("iceberg_manifest_file")) + entry.iceberg_manifest_file = json->getValue("iceberg_manifest_file"); + + if (json->has("commit_marker_file")) + entry.commit_marker_file = json->getValue("commit_marker_file"); + + return entry; + } +}; + +struct ExportReplicatedMergeTreePartitionManifest +{ + String transaction_id; + String query_id; + String partition_id; + String destination_database; + String destination_table; + String source_replica; + size_t number_of_parts; + std::vector parts; + time_t create_time; + size_t retry_initial_backoff_seconds = 5; + size_t retry_max_backoff_seconds = 300; + size_t task_timeout_seconds; + size_t max_threads; + bool parallel_formatting; + bool parquet_parallel_encoding; + size_t max_bytes_per_file; + size_t max_rows_per_file; + MergeTreePartExportManifest::FileAlreadyExistsPolicy file_already_exists_policy; + String filename_pattern; + bool write_full_path_in_iceberg_metadata = false; + bool allow_lossy_cast = false; + String iceberg_metadata_json; + + /// Optional because of backwards compatibility + std::optional parquet_compression_method; + std::optional output_format_compression_level; + std::optional parquet_row_group_size; + std::optional parquet_row_group_size_bytes; + std::optional schema_mismatch_mode; + + /// this is a controversial setting. As far as I can infer from the iceberg docs, the transforms are always UTC. + /// this setting allows to specify different timezones. Since it is already implemented, we must respect it. + /// At the same time, we don't allow transforms with timezones, so this is very weird. + std::optional iceberg_partition_timezone; + + std::string toJsonString() const + { + Poco::JSON::Object json; + json.set("transaction_id", transaction_id); + json.set("query_id", query_id); + json.set("partition_id", partition_id); + json.set("destination_database", destination_database); + json.set("destination_table", destination_table); + json.set("source_replica", source_replica); + json.set("number_of_parts", number_of_parts); + + if (!iceberg_metadata_json.empty()) + { + json.set("iceberg_metadata_json", iceberg_metadata_json); + } + + Poco::JSON::Array::Ptr parts_array = new Poco::JSON::Array(); + for (const auto & part : parts) + parts_array->add(part); + json.set("parts", parts_array); + json.set("parallel_formatting", parallel_formatting); + json.set("max_threads", max_threads); + json.set("parquet_parallel_encoding", parquet_parallel_encoding); + json.set("max_bytes_per_file", max_bytes_per_file); + json.set("max_rows_per_file", max_rows_per_file); + json.set("file_already_exists_policy", String(magic_enum::enum_name(file_already_exists_policy))); + json.set("filename_pattern", filename_pattern); + json.set("create_time", create_time); + json.set("retry_initial_backoff_seconds", retry_initial_backoff_seconds); + json.set("retry_max_backoff_seconds", retry_max_backoff_seconds); + json.set("task_timeout_seconds", task_timeout_seconds); + json.set("write_full_path_in_iceberg_metadata", write_full_path_in_iceberg_metadata); + json.set("allow_lossy_cast", allow_lossy_cast); + if (parquet_compression_method) + json.set("parquet_compression_method", *parquet_compression_method); + if (output_format_compression_level) + json.set("output_format_compression_level", *output_format_compression_level); + if (parquet_row_group_size) + json.set("parquet_row_group_size", *parquet_row_group_size); + if (parquet_row_group_size_bytes) + json.set("parquet_row_group_size_bytes", *parquet_row_group_size_bytes); + if (iceberg_partition_timezone) + json.set("iceberg_partition_timezone", *iceberg_partition_timezone); + if (schema_mismatch_mode) + json.set("schema_mismatch_mode", String(magic_enum::enum_name(*schema_mismatch_mode))); + std::ostringstream oss; // STYLE_CHECK_ALLOW_STD_STRING_STREAM + oss.exceptions(std::ios::failbit); + Poco::JSON::Stringifier::stringify(json, oss); + return oss.str(); + } + + static ExportReplicatedMergeTreePartitionManifest fromJsonString(const std::string & json_string) + { + Poco::JSON::Parser parser; + auto json = parser.parse(json_string).extract(); + chassert(json); + + ExportReplicatedMergeTreePartitionManifest manifest; + manifest.transaction_id = json->getValue("transaction_id"); + manifest.query_id = json->getValue("query_id"); + manifest.partition_id = json->getValue("partition_id"); + manifest.destination_database = json->getValue("destination_database"); + manifest.destination_table = json->getValue("destination_table"); + manifest.source_replica = json->getValue("source_replica"); + manifest.number_of_parts = json->getValue("number_of_parts"); + + if (json->has("retry_initial_backoff_seconds")) + { + manifest.retry_initial_backoff_seconds = json->getValue("retry_initial_backoff_seconds"); + } + + if (json->has("retry_max_backoff_seconds")) + { + manifest.retry_max_backoff_seconds = json->getValue("retry_max_backoff_seconds"); + } + + if (json->has("iceberg_metadata_json")) + { + manifest.iceberg_metadata_json = json->getValue("iceberg_metadata_json"); + } + + auto parts_array = json->getArray("parts"); + for (size_t i = 0; i < parts_array->size(); ++i) + manifest.parts.push_back(parts_array->getElement(static_cast(i))); + + manifest.create_time = json->getValue("create_time"); + manifest.task_timeout_seconds = json->getValue("task_timeout_seconds"); + manifest.max_threads = json->getValue("max_threads"); + manifest.parallel_formatting = json->getValue("parallel_formatting"); + manifest.parquet_parallel_encoding = json->getValue("parquet_parallel_encoding"); + manifest.max_bytes_per_file = json->getValue("max_bytes_per_file"); + manifest.max_rows_per_file = json->getValue("max_rows_per_file"); + manifest.filename_pattern = json->getValue("filename_pattern"); + + if (json->has("file_already_exists_policy")) + { + const auto file_already_exists_policy = magic_enum::enum_cast(json->getValue("file_already_exists_policy")); + if (file_already_exists_policy) + { + manifest.file_already_exists_policy = file_already_exists_policy.value(); + } + + /// what to do if it's not a valid value? + } + + manifest.write_full_path_in_iceberg_metadata = json->getValue("write_full_path_in_iceberg_metadata"); + + /// Default to true for tasks created before this field existed, so an in-flight + /// export scheduled with the old permissive worker behavior is not wrongly rejected + /// on upgrade. New tasks always persist the initiator's actual choice. + manifest.allow_lossy_cast = json->has("allow_lossy_cast") ? json->getValue("allow_lossy_cast") : true; + + /// Left unset (nullopt) for tasks created before this field existed - such tasks were + /// always scheduled under the old, strict column-count check (a mismatch could never + /// reach scheduling in the first place), so callers should treat an absent value as + /// `strict`. + if (json->has("schema_mismatch_mode")) + { + const auto schema_mismatch_mode = magic_enum::enum_cast(json->getValue("schema_mismatch_mode")); + if (schema_mismatch_mode) + manifest.schema_mismatch_mode = schema_mismatch_mode; + } + + if (json->has("parquet_compression_method")) + { + manifest.parquet_compression_method = json->getValue("parquet_compression_method"); + } + + if (json->has("output_format_compression_level")) + { + manifest.output_format_compression_level = json->getValue("output_format_compression_level"); + } + + if (json->has("parquet_row_group_size")) + { + manifest.parquet_row_group_size = json->getValue("parquet_row_group_size"); + } + + if (json->has("parquet_row_group_size_bytes")) + { + manifest.parquet_row_group_size_bytes = json->getValue("parquet_row_group_size_bytes"); + } + + if (json->has("iceberg_partition_timezone")) + { + manifest.iceberg_partition_timezone = json->getValue("iceberg_partition_timezone"); + } + + return manifest; + } +}; + +} diff --git a/src/Storages/ExportReplicatedMergeTreePartitionTaskEntry.h b/src/Storages/ExportReplicatedMergeTreePartitionTaskEntry.h new file mode 100644 index 000000000000..36c0ef303fbf --- /dev/null +++ b/src/Storages/ExportReplicatedMergeTreePartitionTaskEntry.h @@ -0,0 +1,104 @@ +#pragma once + +#include +#include +#include +#include +#include "Core/QualifiedTableName.h" +#include +#include +#include +#include + +namespace DB +{ +struct ExportReplicatedMergeTreePartitionTaskEntry +{ + using DataPartPtr = std::shared_ptr; + ExportReplicatedMergeTreePartitionManifest manifest; + + enum class Status + { + PENDING, + COMPLETED, + FAILED, + KILLED + }; + + /// Allows us to skip completed / failed entries during scheduling + mutable Status status; + + /// References to the parts that should be exported + /// This is used to prevent the parts from being deleted before finishing the export operation + /// It does not mean this replica will export all the parts + /// There is also a chance this replica does not contain a given part and it is totally ok. + mutable std::vector part_references; + + /// In-memory mirror of /last_exception/ leaves in ZK, + /// keyed by replica name (verbatim, not escaped). Refreshed on every poll() cycle + /// and on every status-change handler invocation; served verbatim to + /// system.replicated_partition_exports without any extra ZK read. + /// An empty map means no replica has recorded an exception yet for this task. + mutable std::map last_exception_per_replica; + + /// In-memory mirror of /processed/ leaves in ZK, keyed by + /// part name. Each value is the list of destination file paths produced by the + /// per-part export (typically Parquet object-storage keys). Refreshed on every + /// poll() cycle and on status-change handler invocations; served verbatim to + /// system.replicated_partition_exports without any extra ZK read at query time. + /// An empty map means no part has finished exporting yet for this task. + /// Incomplete Keeper refreshes (or unreadable processed leaves) publish + /// "" as a whole-map key, or as the sole path value + /// for the affected part leaf. + mutable std::map> destination_file_paths_per_part; + + /// In-memory mirror of the /commit_info znode (written atomically + /// with the COMPLETED status transition; see ExportPartitionUtils::commit). + /// nullopt until commit_info is observed in ZK. Empty fields inside the struct + /// for non-Iceberg destinations. + mutable std::optional commit_info; + + std::string getCompositeKey() const + { + const auto qualified_table_name = QualifiedTableName {manifest.destination_database, manifest.destination_table}; + return manifest.partition_id + "_" + qualified_table_name.getFullName(); + } + + std::string getTransactionId() const + { + return manifest.transaction_id; + } + + /// Get create_time for sorted iteration + time_t getCreateTime() const + { + return manifest.create_time; + } +}; + +struct ExportPartitionTaskEntryTagByCompositeKey {}; +struct ExportPartitionTaskEntryTagByCreateTime {}; +struct ExportPartitionTaskEntryTagByTransactionId {}; + +// Multi-index container for export partition task entries +// - Index 0 (TagByCompositeKey): hashed_unique on composite key for O(1) lookup +// - Index 1 (TagByCreateTime): ordered_non_unique on create_time for sorted iteration +using ExportPartitionTaskEntriesContainer = boost::multi_index_container< + ExportReplicatedMergeTreePartitionTaskEntry, + boost::multi_index::indexed_by< + boost::multi_index::hashed_unique< + boost::multi_index::tag, + boost::multi_index::const_mem_fun + >, + boost::multi_index::ordered_non_unique< + boost::multi_index::tag, + boost::multi_index::const_mem_fun + >, + boost::multi_index::hashed_unique< + boost::multi_index::tag, + boost::multi_index::const_mem_fun + > + > +>; + +} diff --git a/src/Storages/HivePartitioningUtils.cpp b/src/Storages/HivePartitioningUtils.cpp index be72281a9ed6..f635ce13b14a 100644 --- a/src/Storages/HivePartitioningUtils.cpp +++ b/src/Storages/HivePartitioningUtils.cpp @@ -226,9 +226,9 @@ HivePartitionColumnsWithFileColumnsPair setupHivePartitioningForObjectStorage( * Otherwise, in case `use_hive_partitioning=1`, we can keep the old behavior of extracting it from the sample path. * And if the schema was inferred (not specified in the table definition), we need to enrich it with the path partition columns */ - if (configuration->partition_strategy && configuration->partition_strategy_type == PartitionStrategyFactory::StrategyType::HIVE) + if (configuration->getPartitionStrategy() && configuration->getPartitionStrategyType() == PartitionStrategyFactory::StrategyType::HIVE) { - hive_partition_columns_to_read_from_file_path = configuration->partition_strategy->getPartitionColumns(); + hive_partition_columns_to_read_from_file_path = configuration->getPartitionStrategy()->getPartitionColumns(); sanityCheckSchemaAndHivePartitionColumns(hive_partition_columns_to_read_from_file_path, columns, /* check_contained_in_schema */true); } else if (context->getSettingsRef()[Setting::use_hive_partitioning]) @@ -242,7 +242,7 @@ HivePartitionColumnsWithFileColumnsPair setupHivePartitioningForObjectStorage( sanityCheckSchemaAndHivePartitionColumns(hive_partition_columns_to_read_from_file_path, columns, /* check_contained_in_schema */false); } - if (configuration->partition_columns_in_data_file) + if (configuration->getPartitionColumnsInDataFile()) { file_columns = columns.getAllPhysical(); } diff --git a/src/Storages/IPartitionStrategy.cpp b/src/Storages/IPartitionStrategy.cpp index 079e5f07ecea..01c85f481427 100644 --- a/src/Storages/IPartitionStrategy.cpp +++ b/src/Storages/IPartitionStrategy.cpp @@ -312,17 +312,15 @@ ColumnPtr WildcardPartitionStrategy::computePartitionKey(const Chunk & chunk) co return block_with_partition_by_expr.getByName(actions_with_column.column_name).column; } -std::string WildcardPartitionStrategy::getPathForRead( - const std::string & prefix) +ColumnPtr WildcardPartitionStrategy::computePartitionKey(Block & block) const { - return prefix; -} + auto actions_with_column = getCachedOrBuildActions( + cached_result, + *this, + [&] { return buildToStringPartitionAST(partition_key_description.definition_ast); }); -std::string WildcardPartitionStrategy::getPathForWrite( - const std::string & prefix, - const std::string & partition_key) -{ - return PartitionedSink::replaceWildcards(prefix, partition_key); + actions_with_column.actions->execute(block); + return block.getByName(actions_with_column.column_name).column; } HiveStylePartitionStrategy::HiveStylePartitionStrategy( @@ -350,41 +348,6 @@ HiveStylePartitionStrategy::HiveStylePartitionStrategy( cacheDeterministicActions(cached_result, actions_with_column); } -std::string HiveStylePartitionStrategy::getPathForRead(const std::string & prefix) -{ - return prefix + "**." + Poco::toLower(file_format); -} - -std::string HiveStylePartitionStrategy::getPathForWrite( - const std::string & prefix, - const std::string & partition_key) -{ - std::string path; - - if (!prefix.empty()) - { - path += prefix; - if (path.back() != '/') - { - path += '/'; - } - } - - /// Not adding '/' because buildExpressionHive() always adds a trailing '/' - path += partition_key; - - /* - * File extension is toLower(format) - * This isn't ideal, but I guess multiple formats can be specified and introduced. - * So I think it is simpler to keep it this way. - * - * Or perhaps implement something like `IInputFormat::getFileExtension()` - */ - path += std::to_string(generateSnowflakeID()) + "." + Poco::toLower(file_format); - - return path; -} - ColumnPtr HiveStylePartitionStrategy::computePartitionKey(const Chunk & chunk) const { auto actions_with_column = getCachedOrBuildActions( @@ -399,6 +362,17 @@ ColumnPtr HiveStylePartitionStrategy::computePartitionKey(const Chunk & chunk) c return block_with_partition_by_expr.getByName(actions_with_column.column_name).column; } +ColumnPtr HiveStylePartitionStrategy::computePartitionKey(Block & block) const +{ + auto actions_with_column = getCachedOrBuildActions( + cached_result, + *this, + [&] { return buildHivePartitionAST(partition_key_description.definition_ast, getPartitionColumns()); }); + + actions_with_column.actions->execute(block); + return block.getByName(actions_with_column.column_name).column; +} + ColumnRawPtrs HiveStylePartitionStrategy::getFormatChunkColumns(const Chunk & chunk) { ColumnRawPtrs result; diff --git a/src/Storages/IPartitionStrategy.h b/src/Storages/IPartitionStrategy.h index b2899b0e4d0a..1378762c6911 100644 --- a/src/Storages/IPartitionStrategy.h +++ b/src/Storages/IPartitionStrategy.h @@ -29,8 +29,7 @@ struct IPartitionStrategy virtual ColumnPtr computePartitionKey(const Chunk & chunk) const = 0; - virtual std::string getPathForRead(const std::string & prefix) = 0; - virtual std::string getPathForWrite(const std::string & prefix, const std::string & partition_key) = 0; + virtual ColumnPtr computePartitionKey(Block & block) const = 0; virtual ColumnRawPtrs getFormatChunkColumns(const Chunk & chunk) { @@ -93,8 +92,8 @@ struct WildcardPartitionStrategy : IPartitionStrategy WildcardPartitionStrategy(KeyDescription partition_key_description_, const Block & sample_block_, ContextPtr context_); ColumnPtr computePartitionKey(const Chunk & chunk) const override; - std::string getPathForRead(const std::string & prefix) override; - std::string getPathForWrite(const std::string & prefix, const std::string & partition_key) override; + + ColumnPtr computePartitionKey(Block & block) const override; }; /* @@ -112,8 +111,8 @@ struct HiveStylePartitionStrategy : IPartitionStrategy bool partition_columns_in_data_file_); ColumnPtr computePartitionKey(const Chunk & chunk) const override; - std::string getPathForRead(const std::string & prefix) override; - std::string getPathForWrite(const std::string & prefix, const std::string & partition_key) override; + + ColumnPtr computePartitionKey(Block & block) const override; ColumnRawPtrs getFormatChunkColumns(const Chunk & chunk) override; Block getFormatHeader() override; diff --git a/src/Storages/IStorage.cpp b/src/Storages/IStorage.cpp index 09d70bdd2804..a3b512ad19d4 100644 --- a/src/Storages/IStorage.cpp +++ b/src/Storages/IStorage.cpp @@ -312,6 +312,11 @@ CancellationCode IStorage::killPartMoveToShard(const UUID & /*task_uuid*/) throw Exception(ErrorCodes::NOT_IMPLEMENTED, "Part moves between shards are not supported by storage {}", getName()); } +CancellationCode IStorage::killExportPartition(const String & /*transaction_id*/) +{ + throw Exception(ErrorCodes::NOT_IMPLEMENTED, "Export partition is not supported by storage {}", getName()); +} + StorageID IStorage::getStorageID() const { std::lock_guard lock(id_mutex); diff --git a/src/Storages/IStorage.h b/src/Storages/IStorage.h index 4307ed1df1c0..fe7da5864f47 100644 --- a/src/Storages/IStorage.h +++ b/src/Storages/IStorage.h @@ -1,5 +1,6 @@ #pragma once +#include #include #include #include @@ -19,6 +20,7 @@ #include #include #include +#include #include #include @@ -60,6 +62,9 @@ struct StreamLocalLimits; class EnabledQuota; struct SelectQueryInfo; +/// Declared opaquely (definition in Core/SettingsEnums.h) to keep this widely included header light. +enum class MergeTreePartExportFileAlreadyExistsPolicy : uint8_t; + using NameDependencies = std::unordered_map>; using DatabaseAndTableName = std::pair; @@ -71,6 +76,9 @@ using ConditionSelectivityEstimatorPtr = std::shared_ptr; + class ActionsDAG; /** Storage. Describes the table. Responsible for @@ -171,6 +179,13 @@ class IStorage : public std::enable_shared_from_this, public TypePromo /// Returns true if the storage supports optimization of moving conditions to PREWHERE section. virtual bool canMoveConditionsToPrewhere() const { return supportsPrewhere(); } + /// Returns true if read() lowers `query_info.row_level_filter` into the reading step. A storage + /// that instead ships query text to other servers must return false: the filter is a plan-level + /// structure that does not travel with the text, so pushing it down would silently drop an + /// access-control filter. Wrappers that only delegate to a remote read for some queries decide + /// per query, hence the context. + virtual bool appliesRowLevelFilterInRead(ContextPtr) const { return !isRemote(); } + /// Returns true if the storage replicates SELECT, INSERT and ALTER commands among replicas. virtual bool supportsReplication() const { return false; } @@ -444,6 +459,7 @@ class IStorage : public std::enable_shared_from_this, public TypePromo size_t /*max_block_size*/, size_t /*num_streams*/); +public: /// Should we process blocks of data returned by the storage in parallel /// even when the storage returned only one stream of data for reading? /// It is beneficial, for example, when you read from a file quickly, @@ -454,7 +470,6 @@ class IStorage : public std::enable_shared_from_this, public TypePromo /// useless). virtual bool parallelizeOutputAfterReading(ContextPtr) const { return !isSystemStorage(); } -public: /// Other version of read which adds reading step to query plan. /// Default implementation creates ReadFromStorageStep and uses usual read. /// Can be called after `shutdown`, but not after `drop`. @@ -497,6 +512,64 @@ class IStorage : public std::enable_shared_from_this, public TypePromo */ virtual void checkInsertIsAllowed(ContextPtr /*context*/) const {} + virtual bool supportsImport(ContextPtr) const + { + return false; + } + + /* +It is currently only implemented in StorageObjectStorage. + It is meant to be used to import merge tree data parts into object storage. It is similar to the write API, + but it won't re-partition the data and should allow the filename to be set by the caller. + */ + virtual SinkToStoragePtr import( + const std::string & /* file_name */, + Block & /* block_with_partition_values */, + const std::function & /* new_file_path_callback */, + MergeTreePartExportFileAlreadyExistsPolicy /* file_already_exists_policy */, + std::size_t /* max_bytes_per_file */, + std::size_t /* max_rows_per_file */, + const std::optional & /* iceberg_metadata_json_string */, + const std::optional & /* format_settings */, + ContextPtr /* context */) + { + throw Exception(ErrorCodes::NOT_IMPLEMENTED, "Import is not implemented for storage {}", getName()); + } + + struct IcebergCommitExportPartitionArguments + { + std::string metadata_json_string; + /// Representative source partition-key columns from one exported part (the part's + /// minmax block). The destination derives the Iceberg partition tuple from a row of + /// this block by casting to the destination column types and applying the partition + /// transform, so the metadata partition value matches the exported data files. + Block partition_source_block; + }; + + /// Paths produced by the destination storage during commit. Surfaced via + /// system.replicated_partition_exports for debugging + struct ExportPartitionCommitInfo + { + /// Iceberg destinations only. + String iceberg_metadata_file; + String iceberg_manifest_list; + String iceberg_manifest_file; + + /// Plain object storage destinations only: path of the commit marker file + /// written/observed by StorageObjectStorage::commitExportPartitionTransaction. + String commit_marker_file; + }; + + virtual ExportPartitionCommitInfo commitExportPartitionTransaction( + const String & /* transaction_id */, + const String & /* partition_id */, + const Strings & /* exported_paths */, + const IcebergCommitExportPartitionArguments & /* iceberg_commit_export_partition_arguments */, + ContextPtr /* local_context */) + { + throw Exception(ErrorCodes::NOT_IMPLEMENTED, "commitExportPartitionTransaction is not implemented for storage type {}", getName()); + } + /** Writes the data to a table in distributed manner. * It is supposed that implementation looks into SELECT part of the query and executes distributed * INSERT SELECT if it is possible with current storage as a receiver and query SELECT part as a producer. @@ -609,6 +682,9 @@ class IStorage : public std::enable_shared_from_this, public TypePromo virtual void setMutationCSN(const String & /*mutation_id*/, UInt64 /*csn*/); + /// Cancel a replicated partition export by transaction id. + virtual CancellationCode killExportPartition(const String & /*transaction_id*/); + /// Cancel a part move to shard. virtual CancellationCode killPartMoveToShard(const UUID & /*task_uuid*/); diff --git a/src/Storages/IStorageCluster.cpp b/src/Storages/IStorageCluster.cpp index 87c61dcae9cc..2b8d2eb7353a 100644 --- a/src/Storages/IStorageCluster.cpp +++ b/src/Storages/IStorageCluster.cpp @@ -1,9 +1,16 @@ #include +#include +#include + +#include +#include + #include #include #include #include +#include #include #include #include @@ -13,13 +20,32 @@ #include #include #include +#include +#include +#include #include +#include #include #include #include #include #include #include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include #include @@ -41,13 +67,18 @@ namespace Setting extern const SettingsBool async_query_sending_for_remote; extern const SettingsBool async_socket_for_remote; extern const SettingsBool skip_unavailable_shards; - extern const SettingsBool parallel_replicas_local_plan; - extern const SettingsString cluster_for_parallel_replicas; extern const SettingsNonZeroUInt64 max_parallel_replicas; + extern const SettingsUInt64 object_storage_max_nodes; + extern const SettingsBool object_storage_remote_initiator; + extern const SettingsString object_storage_remote_initiator_cluster; + extern const SettingsObjectStorageClusterJoinMode object_storage_cluster_join_mode; } namespace ErrorCodes { + extern const int LOGICAL_ERROR; + extern const int NOT_IMPLEMENTED; + extern const int BAD_ARGUMENTS; extern const int ALL_CONNECTION_TRIES_FAILED; } @@ -61,31 +92,335 @@ IStorageCluster::IStorageCluster( { } +void ReadFromCluster::describeActions(FormatSettings & format_settings) const +{ + SourceStepWithFilter::describeActions(format_settings); + if (query_to_send) + format_settings.out << format_settings.detail_prefix << "Query: " + << format({.ctx = getContext(), .query = *query_to_send}) << '\n'; +} + +namespace +{ + +/// Independent filter DAGs (wrap `WHERE`, then a later pushed FilterStep) cannot +/// be `merge`d: that wires the second DAG through the first's boolean output. +/// `mergeNodes` keeps both predicates, then `and` is the listing condition. +ActionsDAG andListingFilterDAGs(ActionsDAG first, ActionsDAG second) +{ + if (first.getOutputs().empty()) + return second; + if (second.getOutputs().empty()) + return first; + + const auto * first_filter = first.getOutputs().front(); + ActionsDAG::NodeRawConstPtrs second_outputs; + first.mergeNodes(std::move(second), &second_outputs); + if (second_outputs.empty()) + return first; + + const auto * second_filter = second_outputs.front(); + if (first_filter == second_filter) + return first; + + FunctionOverloadResolverPtr func_and + = std::make_unique(std::make_shared()); + const auto & and_node = first.addFunction(func_and, {first_filter, second_filter}, {}); + first.getOutputs() = {&and_node}; + first.removeUnusedActions(); + return first; +} + +} + void ReadFromCluster::applyFilters(ActionDAGNodes added_filter_nodes) { SourceStepWithFilter::applyFilters(std::move(added_filter_nodes)); + /// Empty later `applyFilters` (optimizer walk stops at JOIN) wipes + /// `filter_actions_dag` and must not drop wrap `WHERE`. + if (!filter_actions_dag) + return; - const ActionsDAG::Node * predicate = nullptr; - const ActionsDAG * filter = filter_actions_dag ? filter_actions_dag.get() : query_info.filter_actions_dag.get(); - if (filter) - predicate = filter->getOutputs().at(0); + if (!listing_filter_dag) + { + listing_filter_dag = filter_actions_dag; + } + else if (listing_filter_dag->getHash() != filter_actions_dag->getHash()) + { + listing_filter_dag = std::make_shared( + andListingFilterDAGs(listing_filter_dag->clone(), filter_actions_dag->clone())); + } - createExtension(predicate); + VirtualColumnUtils::buildSetsForDAGExcludingGlobalIn(*listing_filter_dag, getContext()); } -void ReadFromCluster::createExtension(const ActionsDAG::Node * predicate) +void ReadFromCluster::createExtension() { if (extension) return; + const ActionsDAG * filter = listing_filter_dag + ? listing_filter_dag.get() + : (filter_actions_dag ? filter_actions_dag.get() : query_info.filter_actions_dag.get()); + const ActionsDAG::Node * predicate = filter ? filter->getOutputs().at(0) : nullptr; extension = storage->getTaskIteratorExtension( predicate, - filter_actions_dag ? filter_actions_dag.get() : query_info.filter_actions_dag.get(), + filter, context, cluster, getStorageSnapshot()->metadata); } +namespace +{ + +/* +Helping class to find in query tree first node of required type +*/ +class SearcherVisitor : public InDepthQueryTreeVisitorWithContext +{ +public: + using Base = InDepthQueryTreeVisitorWithContext; + using Base::Base; + + explicit SearcherVisitor(std::unordered_set types_, size_t entry_, ContextPtr context) + : Base(context) + , types(types_) + , entry(entry_) {} + + bool needChildVisit(QueryTreeNodePtr & /*parent*/, QueryTreeNodePtr & /*child*/) + { + return getSubqueryDepth() <= 2 && !passed_node && !current_entry; + } + + void enterImpl(QueryTreeNodePtr & node) + { + if (passed_node) + return; + + auto node_type = node->getNodeType(); + + if (types.contains(node_type)) + { + ++current_entry; + if (current_entry == entry) + passed_node = node; + } + } + + QueryTreeNodePtr getNode() const { return passed_node; } + +private: + std::unordered_set types; + size_t entry; + size_t current_entry = 0; + QueryTreeNodePtr passed_node; +}; + +/* +Helping class to find all used columns with specific source +*/ +class CollectUsedColumnsForSourceVisitor : public InDepthQueryTreeVisitorWithContext +{ +public: + using Base = InDepthQueryTreeVisitorWithContext; + using Base::Base; + + explicit CollectUsedColumnsForSourceVisitor( + QueryTreeNodePtr source_, + ContextPtr context, + bool collect_columns_from_other_sources_ = false) + : Base(context) + , source(source_) + , collect_columns_from_other_sources(collect_columns_from_other_sources_) + {} + + void enterImpl(QueryTreeNodePtr & node) + { + auto node_type = node->getNodeType(); + + if (node_type != QueryTreeNodeType::COLUMN) + return; + + auto & column_node = node->as(); + auto column_source = column_node.getColumnSourceOrNull(); + if (!column_source) + return; + + if ((column_source == source) != collect_columns_from_other_sources) + { + const auto & name = column_node.getColumnName(); + if (!names.count(name)) + { + columns.emplace_back(column_node.getColumn()); + names.insert(name); + } + } + } + + const NamesAndTypes & getColumns() const { return columns; } + +private: + std::unordered_set names; + QueryTreeNodePtr source; + NamesAndTypes columns; + bool collect_columns_from_other_sources; +}; + +}; + +/* +Try to make subquery to send on nodes +Converts + + SELECT s3.c1, s3.c2, t.c3 + FROM + s3Cluster(...) AS s3 + JOIN + localtable as t + ON s3.key == t.key + +to (object_storage_cluster_join_mode='local') + + SELECT s3.c1, s3.c2, s3.key + FROM + s3Cluster(...) AS s3 + +or (object_storage_cluster_join_mode='global') + + SELECT s3.c1, s3.c2, t.c3 + FROM + s3Cluster(...) as s3 + JOIN + values('key UInt32, data String', (1, 'one'), (2, 'two'), ...) as t + ON s3.key == t.key +*/ +void IStorageCluster::updateQueryWithJoinToSendIfNeeded( + ASTPtr & query_to_send, + SelectQueryInfo query_info, + const ContextPtr & context) +{ + auto object_storage_cluster_join_mode = context->getSettingsRef()[Setting::object_storage_cluster_join_mode]; + switch (object_storage_cluster_join_mode) + { + case ObjectStorageClusterJoinMode::LOCAL: + { + if (!context->getSettingsRef()[Setting::allow_experimental_analyzer]) + throw Exception(ErrorCodes::NOT_IMPLEMENTED, + "object_storage_cluster_join_mode!='allow' is not supported without allow_experimental_analyzer=true"); + + auto info = getQueryTreeInfo(query_info.query_tree, context); + + if (info.has_join || info.has_cross_join || info.has_local_columns_in_where) + { + auto modified_query_tree = query_info.query_tree->clone(); + + SearcherVisitor left_table_expression_searcher({QueryTreeNodeType::TABLE, QueryTreeNodeType::TABLE_FUNCTION}, 1, context); + left_table_expression_searcher.visit(modified_query_tree); + auto table_function_node = left_table_expression_searcher.getNode(); + if (!table_function_node) + throw Exception(ErrorCodes::LOGICAL_ERROR, "Can't find left table function node"); + + QueryTreeNodePtr query_tree_distributed; + + auto & query_node = modified_query_tree->as(); + + if (info.has_join) + { + const auto & join_node = query_node.getJoinTreeNode(); + query_tree_distributed = join_node->as()->getLeftTableExpressionNode()->clone(); + } + else if (info.has_cross_join) + { + SearcherVisitor join_searcher({QueryTreeNodeType::CROSS_JOIN}, 1, context); + join_searcher.visit(modified_query_tree); + auto cross_join_node = join_searcher.getNode(); + if (!cross_join_node) + throw Exception(ErrorCodes::LOGICAL_ERROR, "Can't find CROSS JOIN node"); + // CrossJoinNode contains vector of nodes. 0 is left expression, always exists. + query_tree_distributed = cross_join_node->as()->getTableExpressions()[0]->clone(); + } + + // Find add used columns from table function to make proper projection list + // Need to do before changing WHERE condition + CollectUsedColumnsForSourceVisitor collector(table_function_node, context); + collector.visit(modified_query_tree); + const auto & columns = collector.getColumns(); + + if (columns.empty()) + { + auto column_nodes_to_select = std::make_shared(); + column_nodes_to_select->getNodes().reserve(1); + column_nodes_to_select->getNodes().emplace_back(std::make_shared(1)); + query_node.getProjectionNode() = column_nodes_to_select; + } + else + { + query_node.resolveProjectionColumns(columns); + auto column_nodes_to_select = std::make_shared(); + column_nodes_to_select->getNodes().reserve(columns.size()); + for (auto & column : columns) + column_nodes_to_select->getNodes().emplace_back( + std::make_shared(column, std::static_pointer_cast(table_function_node))); + query_node.getProjectionNode() = column_nodes_to_select; + } + + if (info.has_local_columns_in_where) + { + if (query_node.getPrewhere()) + removeExpressionsThatDoNotDependOnTableIdentifiers(query_node.getPrewhere(), table_function_node, context); + if (query_node.getWhere()) + removeExpressionsThatDoNotDependOnTableIdentifiers(query_node.getWhere(), table_function_node, context); + } + + if (query_node.getPrewhere()) + removeExpressionsThatAreUnsafeToDuplicate(query_node.getPrewhere(), context); + if (query_node.getWhere()) + removeExpressionsThatAreUnsafeToDuplicate(query_node.getWhere(), context); + + query_node.getOrderByNode() = std::make_shared(); + query_node.getGroupByNode() = std::make_shared(); + + if (query_tree_distributed) + { + // Left only table function to send on cluster nodes + modified_query_tree = modified_query_tree->cloneAndReplace( + query_node.getJoinTreeNodeTyped(), std::static_pointer_cast(query_tree_distributed)); + } + + query_to_send = queryNodeToDistributedSelectQuery(modified_query_tree); + } + + return; + } + case ObjectStorageClusterJoinMode::GLOBAL: + { + auto info = getQueryTreeInfo(query_info.query_tree, context); + + if (info.has_join || info.has_cross_join || info.has_local_columns_in_where) + { + auto modified_query_tree = query_info.query_tree->clone(); + + rewriteJoinToGlobalJoin(modified_query_tree, context, /*force_prefer_global_join*/ true); + + if (info.has_local_columns_in_where) + rewriteInToGlobalIn(modified_query_tree, context, /*rewrite_for_distributed*/ true); + + modified_query_tree = buildQueryTreeForShard( + query_info.planner_context, + modified_query_tree, + /*allow_global_join_for_right_table*/ false, + /*find_cross_join*/ true); + query_to_send = queryNodeToDistributedSelectQuery(modified_query_tree); + } + + return; + } + case ObjectStorageClusterJoinMode::ALLOW: // Do nothing special + return; + } +} + /// The code executes on initiator void IStorageCluster::read( QueryPlan & query_plan, @@ -94,36 +429,100 @@ void IStorageCluster::read( SelectQueryInfo & query_info, ContextPtr context, QueryProcessingStage::Enum processed_stage, - size_t /*max_block_size*/, - size_t /*num_streams*/) + size_t max_block_size, + size_t num_streams) { + updateBeforeRead(context); + + if (!isClusterSupported()) + { + readFallBackToPure(query_plan, column_names, storage_snapshot, query_info, context, processed_stage, max_block_size, num_streams); + return; + } + + auto cluster_name_from_settings = getClusterName(context); + const auto & settings = context->getSettingsRef(); + ASTPtr query_to_send = query_info.query; + + if (cluster_name_from_settings.empty()) + { + if (settings[Setting::object_storage_remote_initiator]) + { + auto remote_initiator_cluster_name = settings[Setting::object_storage_remote_initiator_cluster].value; + if (remote_initiator_cluster_name.empty()) + throw Exception(ErrorCodes::BAD_ARGUMENTS, + "Setting 'object_storage_remote_initiator' can be used only with 'object_storage_remote_initiator_cluster', 'object_storage_cluster', or cluster name in arguments"); + + /// rewrite query to execute `remote('remote_host', s3(...))` + /// remote_host can execute query itself or make on-cluster query depends on own `object_storage_cluster` setting + updateConfigurationIfNeeded(context); + updateQueryWithJoinToSendIfNeeded(query_to_send, query_info, context); + updateQueryToSendIfNeeded(query_to_send, storage_snapshot, context, /*make_cluster_function*/ false); + + auto remote_initiator_cluster = getClusterImpl(context, remote_initiator_cluster_name); + auto storage_and_context = convertToRemote(remote_initiator_cluster, context, remote_initiator_cluster_name, query_to_send); + auto src_distributed = std::dynamic_pointer_cast(storage_and_context.storage); + auto modified_query_info = query_info; + modified_query_info.cluster = src_distributed->getCluster(); + auto new_storage_snapshot = storage_and_context.storage->getStorageSnapshot(storage_snapshot->metadata, storage_and_context.context); + storage_and_context.storage->read(query_plan, column_names, new_storage_snapshot, modified_query_info, storage_and_context.context, processed_stage, max_block_size, num_streams); + return; + } + + readFallBackToPure(query_plan, column_names, storage_snapshot, query_info, context, processed_stage, max_block_size, num_streams); + return; + } + updateConfigurationIfNeeded(context); storage_snapshot->check(column_names); - updateBeforeRead(context); - auto cluster = getCluster(context); - /// Calculate the header. This is significant, because some columns could be thrown away in some cases like query with count(*) SharedHeader sample_block; - ASTPtr query_to_send = query_info.query; - if (context->getSettingsRef()[Setting::allow_experimental_analyzer]) + updateQueryWithJoinToSendIfNeeded(query_to_send, query_info, context); + + if (settings[Setting::allow_experimental_analyzer]) { - sample_block = InterpreterSelectQueryAnalyzer::getSampleBlock(query_info.query, context, SelectQueryOptions(processed_stage)); + sample_block = InterpreterSelectQueryAnalyzer::getSampleBlock(query_to_send, context, SelectQueryOptions(processed_stage)); } else { - auto interpreter = InterpreterSelectQuery(query_info.query, context, SelectQueryOptions(processed_stage).analyze()); + auto interpreter = InterpreterSelectQuery(query_to_send, context, SelectQueryOptions(processed_stage).analyze()); sample_block = interpreter.getSampleBlock(); query_to_send = interpreter.getQueryInfo().query->clone(); } - updateQueryToSendIfNeeded(query_to_send, storage_snapshot, context); + updateQueryToSendIfNeeded(query_to_send, storage_snapshot, context, /*make_cluster_function*/ true); + + /// In case the current node is not supposed to initiate the clustered query + /// Sends this query to a remote initiator using the `remote` table function + if (settings[Setting::object_storage_remote_initiator]) + { + /// Re-writes queries in the form of: + /// Input: SELECT * FROM iceberg(...) SETTINGS object_storage_cluster='swarm', object_storage_remote_initiator=1 + /// Output: SELECT * FROM remote('remote_host', icebergCluster('swarm', ...) + /// Where `remote_host` is a random host from the cluster which will execute the query + /// This means the initiator node belongs to the same cluster that will execute the query + /// In case remote_initiator_cluster_name is set, the initiator might be set to a different cluster + auto remote_initiator_cluster_name = settings[Setting::object_storage_remote_initiator_cluster].value; + if (remote_initiator_cluster_name.empty()) + remote_initiator_cluster_name = cluster_name_from_settings; + auto remote_initiator_cluster = getClusterImpl(context, remote_initiator_cluster_name); + auto storage_and_context = convertToRemote(remote_initiator_cluster, context, remote_initiator_cluster_name, query_to_send); + auto src_distributed = std::dynamic_pointer_cast(storage_and_context.storage); + auto modified_query_info = query_info; + modified_query_info.cluster = src_distributed->getCluster(); + auto new_storage_snapshot = storage_and_context.storage->getStorageSnapshot(storage_snapshot->metadata, storage_and_context.context); + storage_and_context.storage->read(query_plan, column_names, new_storage_snapshot, modified_query_info, storage_and_context.context, processed_stage, max_block_size, num_streams); + return; + } + + auto cluster = getClusterImpl(context, cluster_name_from_settings, isObjectStorage() ? settings[Setting::object_storage_max_nodes] : 0); RestoreQualifiedNamesVisitor::Data data; - data.distributed_table = DatabaseAndTableWithAlias(*getTableExpression(query_info.query->as(), 0)); + data.distributed_table = DatabaseAndTableWithAlias(*getTableExpression(query_to_send->as(), 0)); data.remote_table.database = context->getCurrentDatabase(); data.remote_table.table = getName(); RestoreQualifiedNamesVisitor(data).visit(query_to_send); @@ -140,6 +539,10 @@ void IStorageCluster::read( auto this_ptr = std::static_pointer_cast(shared_from_this()); + std::optional external_tables = std::nullopt; + if (query_info.planner_context && query_info.planner_context->getMutableQueryContext()) + external_tables = query_info.planner_context->getMutableQueryContext()->getExternalTables(); + auto reading = std::make_unique( column_names, query_info, @@ -150,11 +553,111 @@ void IStorageCluster::read( std::move(query_to_send), processed_stage, cluster, - log); + log, + external_tables); query_plan.addStep(std::move(reading)); } +IStorageCluster::RemoteCallVariables IStorageCluster::convertToRemote( + ClusterPtr cluster, + ContextPtr context, + const std::string & cluster_name_from_settings, + ASTPtr query_to_send) +{ + /// TODO: Allow to use secret for remote queries + if (!cluster->getSecret().empty()) + throw Exception(ErrorCodes::NOT_IMPLEMENTED, "Can't convert query to remote when cluster uses secret"); + + auto host_addresses = cluster->getShardsAddresses(); + if (host_addresses.empty()) + throw Exception(ErrorCodes::BAD_ARGUMENTS, "Empty cluster {}", cluster_name_from_settings); + + pcg64 rng(randomSeed()); + size_t shard_num = rng() % host_addresses.size(); + auto shard_addresses = host_addresses[shard_num]; + /// After getClusterImpl each shard must have exactly 1 replica + if (shard_addresses.size() != 1) + throw Exception(ErrorCodes::BAD_ARGUMENTS, "Size of shard {} in cluster {} is not equal 1", shard_num, cluster_name_from_settings); + std::string host_name; + Poco::URI::decode(shard_addresses[0].toString(), host_name); + + LOG_INFO(log, "Choose remote initiator '{}'", host_name); + + bool secure = shard_addresses[0].secure == Protocol::Secure::Enable; + std::string remote_function_name = secure ? "remoteSecure" : "remote"; + + /// Clean object_storage_remote_initiator setting to avoid infinite remote call + auto new_context = Context::createCopy(context); + std::vector settings_to_remove = {"object_storage_remote_initiator", "object_storage_remote_initiator_cluster"}; + new_context->resetSettingsToDefaultValue(settings_to_remove); + + auto * select_query = query_to_send->as(); + if (!select_query) + throw Exception(ErrorCodes::LOGICAL_ERROR, "Expected SELECT query"); + + auto query_settings = select_query->settings(); + if (query_settings) + { + auto & settings_ast = query_settings->as(); + bool settings_changed = false; + for (const auto & setting_to_remove : settings_to_remove) + settings_changed |= settings_ast.changes.removeSetting(setting_to_remove); + if (settings_changed && settings_ast.changes.empty()) + select_query->setExpression(ASTSelectQuery::Expression::SETTINGS, {}); + } + + ASTTableExpression * table_expression = extractTableExpressionASTPtrFromSelectQuery(query_to_send); + if (!table_expression) + throw Exception(ErrorCodes::LOGICAL_ERROR, "Can't find table expression"); + if (!table_expression->table_function) + throw Exception(ErrorCodes::LOGICAL_ERROR, "Can't find table function in table expression"); + + boost::intrusive_ptr remote_query; + + if (shard_addresses[0].user_specified) + { // with user/password for clsuter access remote query is executed from this user, add it in query parameters + remote_query = makeASTFunction(remote_function_name, + make_intrusive(host_name), + table_expression->table_function, + make_intrusive(shard_addresses[0].user), + make_intrusive(shard_addresses[0].password)); + } + else + { // without specified user/password remote query is executed from default user + remote_query = makeASTFunction(remote_function_name, make_intrusive(host_name), table_expression->table_function); + } + + table_expression->table_function = remote_query; + + auto remote_function = TableFunctionFactory::instance().get(remote_query, new_context); + + std::shared_ptr remote_table_function = std::dynamic_pointer_cast(remote_function); + if (remote_table_function) + { + auto metadata_snapshot = getInMemoryMetadataPtr(context, false); + remote_table_function->setActualTableStructure(metadata_snapshot->columns); + } + + auto storage = remote_function->execute(query_to_send, new_context, remote_function_name); + + return RemoteCallVariables{storage, new_context}; +} + +SinkToStoragePtr IStorageCluster::write( + const ASTPtr & query, + const StorageMetadataPtr & metadata_snapshot, + ContextPtr context, + bool async_insert) +{ + auto cluster_name_from_settings = getClusterName(context); + + if (cluster_name_from_settings.empty()) + return writeFallBackToPure(query, metadata_snapshot, context, async_insert); + + throw Exception(ErrorCodes::NOT_IMPLEMENTED, "Method write is not supported by storage {}", getName()); +} + void ReadFromCluster::initializePipeline(QueryPipelineBuilder & pipeline, const BuildQueryPipelineSettings &) { const Scalars & scalars = context->hasQueryContext() ? context->getQueryContext()->getScalars() : Scalars{}; @@ -170,7 +673,7 @@ void ReadFromCluster::initializePipeline(QueryPipelineBuilder & pipeline, const if (current_settings[Setting::max_parallel_replicas] > 1) max_replicas_to_use = std::min(max_replicas_to_use, current_settings[Setting::max_parallel_replicas].value); - createExtension(nullptr); + createExtension(); ProfileEvents::increment(ProfileEvents::Shards, max_replicas_to_use); @@ -200,7 +703,7 @@ void ReadFromCluster::initializePipeline(QueryPipelineBuilder & pipeline, const new_context, /*throttler=*/nullptr, scalars, - Tables(), + external_tables.has_value() ? *external_tables : Tables(), processed_stage, nullptr, RemoteQueryExecutor::Extension{.task_iterator = extension->task_iterator, .replica_info = std::move(replica_info)}, @@ -226,9 +729,62 @@ void ReadFromCluster::initializePipeline(QueryPipelineBuilder & pipeline, const pipeline.init(std::move(pipe)); } +IStorageCluster::QueryTreeInfo IStorageCluster::getQueryTreeInfo(QueryTreeNodePtr query_tree, ContextPtr context) +{ + QueryTreeInfo info; + + auto & query_node = query_tree->as(); + if (const auto & join_node = query_node.getJoinTreeNode()) + { + if (join_node->getNodeType() == QueryTreeNodeType::JOIN) + info.has_join = true; + else if (join_node->getNodeType() == QueryTreeNodeType::CROSS_JOIN) + info.has_cross_join = true; + } + + SearcherVisitor left_table_expression_searcher({QueryTreeNodeType::TABLE, QueryTreeNodeType::TABLE_FUNCTION}, 1, context); + left_table_expression_searcher.visit(query_tree); + auto table_function_node = left_table_expression_searcher.getNode(); + if (!table_function_node) + throw Exception(ErrorCodes::LOGICAL_ERROR, "Can't find table or table function node"); + + if (query_node.hasWhere() || query_node.hasPrewhere()) + { + CollectUsedColumnsForSourceVisitor collector_where(table_function_node, context, true); + if (query_node.hasPrewhere()) + collector_where.visit(query_node.getPrewhere()); + if (query_node.hasWhere()) + collector_where.visit(query_node.getWhere()); + + // SELECT x FROM datalake.table WHERE x IN local.table. + // Need to modify 'WHERE' on remote node if it contains columns from other sources + // because remote node might not have those sources. + if (!collector_where.getColumns().empty()) + info.has_local_columns_in_where = true; + } + + return info; +} + QueryProcessingStage::Enum IStorageCluster::getQueryProcessingStage( - ContextPtr context, QueryProcessingStage::Enum to_stage, const StorageSnapshotPtr &, SelectQueryInfo &) const + ContextPtr context, QueryProcessingStage::Enum to_stage, const StorageSnapshotPtr &, SelectQueryInfo & query_info) const { + auto object_storage_cluster_join_mode = context->getSettingsRef()[Setting::object_storage_cluster_join_mode]; + + if (object_storage_cluster_join_mode != ObjectStorageClusterJoinMode::ALLOW) + { + if (!context->getSettingsRef()[Setting::allow_experimental_analyzer]) + throw Exception(ErrorCodes::NOT_IMPLEMENTED, + "object_storage_cluster_join_mode!='allow' is not supported without allow_experimental_analyzer=true"); + + if (object_storage_cluster_join_mode == ObjectStorageClusterJoinMode::LOCAL) + { + auto info = getQueryTreeInfo(query_info.query_tree, context); + if (info.has_join || info.has_cross_join || info.has_local_columns_in_where) + return QueryProcessingStage::Enum::FetchColumns; + } + } + /// Only a follower reached by another node's cluster function (SECONDARY_QUERY) just fetches /// raw data. Everything else is the initiator of the distributed read, including internal /// contexts that never set the kind (NO_QUERY), e.g. a Replicated database DDL worker. @@ -240,6 +796,20 @@ QueryProcessingStage::Enum IStorageCluster::getQueryProcessingStage( return QueryProcessingStage::Enum::FetchColumns; } +NamesAndTypesList IStorageCluster::getHivePartitionColumnsWithoutVirtuals(const StorageMetadataPtr & metadata_snapshot) const +{ + // Virtual columns can contain hive columns, so we remove these hive coulmns to avoid duplicates. + // In non-cluster case these columns are filtered in DB::prepareReadingFromFormat function. + auto virtual_columns = metadata_snapshot->virtuals.getSampleBlock(VirtualsKind::All, VirtualsMaterializationPlace::Reader).getNamesAndTypesList(); + NamesAndTypesList hive_partition_filtered; + for (const auto & hive_name_and_type : hive_partition_columns_to_read_from_file_path) + { + if (!virtual_columns.contains(hive_name_and_type.name)) + hive_partition_filtered.emplace_back(hive_name_and_type); + } + return hive_partition_filtered; +} + ContextPtr ReadFromCluster::updateSettings(const Settings & settings) { Settings new_settings{settings}; @@ -258,9 +828,9 @@ ContextPtr ReadFromCluster::updateSettings(const Settings & settings) return new_context; } -ClusterPtr IStorageCluster::getCluster(ContextPtr context) const +ClusterPtr IStorageCluster::getClusterImpl(ContextPtr context, const String & cluster_name_, size_t max_hosts) { - return context->getCluster(cluster_name)->getClusterWithReplicasAsShards(context->getSettingsRef()); + return context->getCluster(cluster_name_)->getClusterWithReplicasAsShards(context->getSettingsRef(), /* max_replicas_from_shard */ 0, max_hosts); } } diff --git a/src/Storages/IStorageCluster.h b/src/Storages/IStorageCluster.h index 43b2d690955d..123c05adab1b 100644 --- a/src/Storages/IStorageCluster.h +++ b/src/Storages/IStorageCluster.h @@ -31,10 +31,16 @@ class IStorageCluster : public IStorage SelectQueryInfo & query_info, ContextPtr context, QueryProcessingStage::Enum processed_stage, - size_t /*max_block_size*/, - size_t /*num_streams*/) override; + size_t max_block_size, + size_t num_streams) override; - ClusterPtr getCluster(ContextPtr context) const; + SinkToStoragePtr write( + const ASTPtr & query, + const StorageMetadataPtr & metadata_snapshot, + ContextPtr context, + bool async_insert) override; + + ClusterPtr getCluster(ContextPtr context) const { return getClusterImpl(context, cluster_name); } /// Query is needed for pruning by virtual columns (_file, _path) virtual RemoteQueryExecutor::Extension getTaskIteratorExtension( @@ -54,15 +60,74 @@ class IStorageCluster : public IStorage const String & getClusterName() const { return cluster_name; } + const String & getOriginalClusterName() const { return cluster_name; } + virtual String getClusterName(ContextPtr /* context */) const { return getOriginalClusterName(); } + protected: virtual void updateBeforeRead(const ContextPtr &) {} - virtual void updateQueryToSendIfNeeded(ASTPtr & /*query*/, const StorageSnapshotPtr & /*storage_snapshot*/, const ContextPtr & /*context*/) {} + virtual void updateQueryToSendIfNeeded( + ASTPtr & /*query*/, + const StorageSnapshotPtr & /*storage_snapshot*/, + const ContextPtr & /*context*/, + bool /*make_cluster_function*/) {} + void updateQueryWithJoinToSendIfNeeded(ASTPtr & query_to_send, SelectQueryInfo query_info, const ContextPtr & context); virtual void updateConfigurationIfNeeded(ContextPtr /* context */) {} + struct RemoteCallVariables + { + StoragePtr storage; + ContextPtr context; + }; + + RemoteCallVariables convertToRemote( + ClusterPtr cluster, + ContextPtr context, + const std::string & cluster_name_from_settings, + ASTPtr query_to_send); + + virtual void readFallBackToPure( + QueryPlan & /* query_plan */, + const Names & /* column_names */, + const StorageSnapshotPtr & /* storage_snapshot */, + SelectQueryInfo & /* query_info */, + ContextPtr /* context */, + QueryProcessingStage::Enum /* processed_stage */, + size_t /* max_block_size */, + size_t /* num_streams */) + { + throw Exception(ErrorCodes::NOT_IMPLEMENTED, "Method readFallBackToPure is not supported by storage {}", getName()); + } + + virtual SinkToStoragePtr writeFallBackToPure( + const ASTPtr & /*query*/, + const StorageMetadataPtr & /*metadata_snapshot*/, + ContextPtr /*context*/, + bool /*async_insert*/) + { + throw Exception(ErrorCodes::NOT_IMPLEMENTED, "Method writeFallBackToPure is not supported by storage {}", getName()); + } + + NamesAndTypesList getHivePartitionColumnsWithoutVirtuals(const StorageMetadataPtr & metadata_snapshot) const; + + NamesAndTypesList hive_partition_columns_to_read_from_file_path; + private: + static ClusterPtr getClusterImpl(ContextPtr context, const String & cluster_name_, size_t max_hosts = 0); + + virtual bool isClusterSupported() const { return true; } + LoggerPtr log; String cluster_name; + + struct QueryTreeInfo + { + bool has_join = false; + bool has_cross_join = false; + bool has_local_columns_in_where = false; + }; + + static QueryTreeInfo getQueryTreeInfo(QueryTreeNodePtr query_tree, ContextPtr context); }; @@ -72,6 +137,7 @@ class ReadFromCluster : public SourceStepWithFilter std::string getName() const override { return "ReadFromCluster"; } void initializePipeline(QueryPipelineBuilder & pipeline, const BuildQueryPipelineSettings &) override; void applyFilters(ActionDAGNodes added_filter_nodes) override; + void describeActions(FormatSettings & format_settings) const override; ReadFromCluster( const Names & column_names_, @@ -83,7 +149,8 @@ class ReadFromCluster : public SourceStepWithFilter ASTPtr query_to_send_, QueryProcessingStage::Enum processed_stage_, ClusterPtr cluster_, - LoggerPtr log_) + LoggerPtr log_, + std::optional external_tables_) : SourceStepWithFilter( std::move(sample_block), column_names_, @@ -95,6 +162,7 @@ class ReadFromCluster : public SourceStepWithFilter , processed_stage(processed_stage_) , cluster(std::move(cluster_)) , log(log_) + , external_tables(external_tables_) { } @@ -106,8 +174,10 @@ class ReadFromCluster : public SourceStepWithFilter LoggerPtr log; std::optional extension; + std::shared_ptr listing_filter_dag; + std::optional external_tables; - void createExtension(const ActionsDAG::Node * predicate); + void createExtension(); ContextPtr updateSettings(const Settings & settings); }; diff --git a/src/Storages/MergeTree/BackgroundJobsAssignee.cpp b/src/Storages/MergeTree/BackgroundJobsAssignee.cpp index 4c6a226fc616..810da3446297 100644 --- a/src/Storages/MergeTree/BackgroundJobsAssignee.cpp +++ b/src/Storages/MergeTree/BackgroundJobsAssignee.cpp @@ -99,6 +99,10 @@ bool BackgroundJobsAssignee::scheduleCommonTask(ExecutableTaskPtr common_task, b return schedule_res; } +std::size_t BackgroundJobsAssignee::getAvailableMoveExecutors() const +{ + return getContext()->getMovesExecutor()->getAvailableSlots(); +} String BackgroundJobsAssignee::toString(Type type) { diff --git a/src/Storages/MergeTree/BackgroundJobsAssignee.h b/src/Storages/MergeTree/BackgroundJobsAssignee.h index 25d3806ba4d2..3a50ad85cfd7 100644 --- a/src/Storages/MergeTree/BackgroundJobsAssignee.h +++ b/src/Storages/MergeTree/BackgroundJobsAssignee.h @@ -74,6 +74,8 @@ class BackgroundJobsAssignee : public WithContext bool scheduleMoveTask(ExecutableTaskPtr move_task); bool scheduleCommonTask(ExecutableTaskPtr common_task, bool need_trigger); + std::size_t getAvailableMoveExecutors() const; + /// Just call finish ~BackgroundJobsAssignee(); diff --git a/src/Storages/MergeTree/ExportList.cpp b/src/Storages/MergeTree/ExportList.cpp new file mode 100644 index 000000000000..018c1f091ef9 --- /dev/null +++ b/src/Storages/MergeTree/ExportList.cpp @@ -0,0 +1,74 @@ +#include + +namespace DB +{ + +ExportsListElement::ExportsListElement( + const StorageID & source_table_id_, + const StorageID & destination_table_id_, + UInt64 part_size_, + const String & part_name_, + const std::vector & destination_file_paths_, + UInt64 total_rows_to_read_, + UInt64 total_size_bytes_compressed_, + UInt64 total_size_bytes_uncompressed_, + time_t create_time_, + const String & query_id_, + const ContextPtr & context) +: source_table_id(source_table_id_) +, destination_table_id(destination_table_id_) +, part_size(part_size_) +, part_name(part_name_) +, destination_file_paths(destination_file_paths_) +, total_rows_to_read(total_rows_to_read_) +, total_size_bytes_compressed(total_size_bytes_compressed_) +, total_size_bytes_uncompressed(total_size_bytes_uncompressed_) +, create_time(create_time_) +, query_id(query_id_) +{ + thread_group = ThreadGroup::createForMergeMutate(context); +} + +ExportsListElement::~ExportsListElement() +{ + background_memory_tracker.adjustOnBackgroundTaskEnd(&thread_group->memory_tracker); +} + +ExportInfo ExportsListElement::getInfo() const +{ + ExportInfo res; + res.source_database = source_table_id.database_name; + res.source_table = source_table_id.table_name; + res.destination_database = destination_table_id.database_name; + res.destination_table = destination_table_id.table_name; + res.part_name = part_name; + + { + std::shared_lock lock(destination_file_paths_mutex); + res.destination_file_paths = destination_file_paths; + } + + res.rows_read = rows_read.load(std::memory_order_relaxed); + res.total_rows_to_read = total_rows_to_read; + res.total_size_bytes_compressed = total_size_bytes_compressed; + res.total_size_bytes_uncompressed = total_size_bytes_uncompressed; + res.bytes_read_uncompressed = bytes_read_uncompressed.load(std::memory_order_relaxed); + res.memory_usage = getMemoryUsage(); + res.peak_memory_usage = getPeakMemoryUsage(); + res.create_time = create_time; + res.elapsed = watch.elapsedSeconds(); + res.query_id = query_id; + return res; +} + +UInt64 ExportsListElement::getMemoryUsage() const +{ + return thread_group->memory_tracker.get(); +} + +UInt64 ExportsListElement::getPeakMemoryUsage() const +{ + return thread_group->memory_tracker.getPeak(); +} + +} diff --git a/src/Storages/MergeTree/ExportList.h b/src/Storages/MergeTree/ExportList.h new file mode 100644 index 000000000000..4a02826dfe44 --- /dev/null +++ b/src/Storages/MergeTree/ExportList.h @@ -0,0 +1,96 @@ +#pragma once + +#include +#include +#include +#include +#include +#include +#include +#include + +namespace CurrentMetrics +{ + extern const Metric Export; +} + +namespace DB +{ + +struct ExportInfo +{ + String source_database; + String source_table; + String destination_database; + String destination_table; + String part_name; + std::vector destination_file_paths; + UInt64 rows_read; + UInt64 total_rows_to_read; + UInt64 total_size_bytes_compressed; + UInt64 total_size_bytes_uncompressed; + UInt64 bytes_read_uncompressed; + UInt64 memory_usage; + UInt64 peak_memory_usage; + time_t create_time = 0; + Float64 elapsed; + String query_id; +}; + +struct ExportsListElement : private boost::noncopyable +{ + const StorageID source_table_id; + const StorageID destination_table_id; + const UInt64 part_size; + const String part_name; + + /// see destination_file_paths_mutex + std::vector destination_file_paths; + std::atomic rows_read {0}; + UInt64 total_rows_to_read {0}; + UInt64 total_size_bytes_compressed {0}; + UInt64 total_size_bytes_uncompressed {0}; + std::atomic bytes_read_uncompressed {0}; + time_t create_time {0}; + String query_id; + + Stopwatch watch; + ThreadGroupPtr thread_group; + mutable std::shared_mutex destination_file_paths_mutex; + + ExportsListElement( + const StorageID & source_table_id_, + const StorageID & destination_table_id_, + UInt64 part_size_, + const String & part_name_, + const std::vector & destination_file_paths_, + UInt64 total_rows_to_read_, + UInt64 total_size_bytes_compressed_, + UInt64 total_size_bytes_uncompressed_, + time_t create_time_, + const String & query_id_, + const ContextPtr & context); + + ~ExportsListElement(); + + ExportInfo getInfo() const; + + UInt64 getMemoryUsage() const; + UInt64 getPeakMemoryUsage() const; +}; + + +class ExportsList final : public BackgroundProcessList +{ +private: + using Parent = BackgroundProcessList; + +public: + ExportsList() + : Parent(CurrentMetrics::Export) + {} +}; + +using ExportsListEntry = BackgroundProcessListEntry; + +} diff --git a/src/Storages/MergeTree/ExportPartTask.cpp b/src/Storages/MergeTree/ExportPartTask.cpp new file mode 100644 index 000000000000..1d250401e84f --- /dev/null +++ b/src/Storages/MergeTree/ExportPartTask.cpp @@ -0,0 +1,536 @@ +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include "Common/setThreadName.h" +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +namespace ProfileEvents +{ + extern const Event PartsExportDuplicated; + extern const Event PartsExportFailures; + extern const Event PartsExports; + extern const Event PartsExportTotalMilliseconds; +} + +namespace DB +{ + +namespace ErrorCodes +{ + extern const int UNKNOWN_TABLE; + extern const int FILE_ALREADY_EXISTS; + extern const int LOGICAL_ERROR; + extern const int QUERY_WAS_CANCELLED; + extern const int BAD_ARGUMENTS; + extern const int FAULT_INJECTED; +} + +namespace FailPoints +{ + /// Throw a non-retryable (denylisted) error from the part-export worker, so the whole + /// export task transitions to FAILED immediately regardless of any timeout. + extern const char export_part_non_retryable_throw[]; + /// Throw a retryable error from the part-export worker, so the part is retried with the + /// per-replica back-off until the task succeeds or the absolute timeout fires. + extern const char export_part_retryable_throw[]; +} + +namespace Setting +{ + extern const SettingsUInt64 min_bytes_to_use_direct_io; + extern const SettingsUInt64 export_merge_tree_part_max_bytes_per_file; + extern const SettingsUInt64 export_merge_tree_part_max_rows_per_file; + extern const SettingsBool allow_experimental_analyzer; + extern const SettingsString export_merge_tree_part_filename_pattern; + extern const SettingsMergeTreePartExportSchemaMismatchMode export_merge_tree_part_schema_mismatch_mode; +} + +namespace +{ + void materializeSpecialColumns( + const SharedHeader & header, + const StorageMetadataPtr & storage_metadata, + const ContextPtr & local_context, + QueryPlan & plan_for_part + ) + { + const auto readable_columns = storage_metadata->getColumns().getReadable(); + + // Enable all experimental settings for default expressions + // (same pattern as in IMergeTreeReader::evaluateMissingDefaults) + auto context_for_defaults = Context::createCopy(local_context); + enableAllExperimentalSettings(context_for_defaults); + + /// Copy the behavior of `IMergeTreeReader`, see https://github.com/ClickHouse/ClickHouse/blob/c45224e3f0a6dd9a9217e5d75723f378ffe0a86a/src/Storages/MergeTree/IMergeTreeReader.cpp#L215 + context_for_defaults->setSetting("enable_analyzer", local_context->getSettingsRef()[Setting::allow_experimental_analyzer].value); + + auto defaults_dag = evaluateMissingDefaults( + *header, + readable_columns, + storage_metadata->getColumns(), + context_for_defaults); + + if (defaults_dag) + { + ActionsDAG base_dag(header->getColumnsWithTypeAndName()); + + /// `evaluateMissingDefaults` has a new analyzer path since https://github.com/ClickHouse/ClickHouse/pull/87585 + /// which returns a DAG that does not contain all columns. We need to merge it with the base DAG to get all columns. + auto merged = ActionsDAG::merge(std::move(base_dag), std::move(*defaults_dag)); + + /// Ensure columns are in the correct order matching readable_columns + merged.removeUnusedActions(readable_columns.getNames(), false); + merged.addMaterializingOutputActions(/*materialize_sparse=*/ false); + + auto expression_step = std::make_unique( + header, + std::move(merged)); + expression_step->setStepDescription("Compute alias and default expressions for export"); + plan_for_part.addStep(std::move(expression_step)); + } + } + + /// Mirrors `InterpreterInsertQuery::addInsertToSelectPipeline`: positional match, + /// destination header = `getSampleBlockNonMaterialized()`, all type bridging is done + /// by the CAST inside `makeConvertingActions`. No pre-validation, no per-column + /// lossy/non-lossy classification — restrictions are exactly what INSERT SELECT enforces. + /// + /// Exception: when `export_merge_tree_part_schema_mismatch_mode = 'ignore_extra_source_columns_by_position'` + /// and the source has more columns than the destination, the extra trailing source + /// columns (by position) are dropped by a preliminary projection step before the + /// positional convert, so `makeConvertingActions` always sees equal-sized inputs. + void addExportConvertingActions( + QueryPlan & plan_for_part, + const IStorage & destination_storage, + const ContextPtr & local_context) + { + const auto destination_metadata = destination_storage.getInMemoryMetadataPtr(local_context, false); + const auto destination_header = destination_metadata->getSampleBlockNonMaterialized(); + const auto & destination_columns = destination_header.getColumnsWithTypeAndName(); + + const bool ignore_extra_source_columns_by_position = + local_context->getSettingsRef()[Setting::export_merge_tree_part_schema_mismatch_mode] + == MergeTreePartExportSchemaMismatchMode::ignore_extra_source_columns_by_position; + + auto source_columns = plan_for_part.getCurrentHeader()->getColumnsWithTypeAndName(); + + if (ignore_extra_source_columns_by_position && source_columns.size() > destination_columns.size()) + { + LOG_DEBUG(getLogger("ExportPartTask"), + "Source has {} columns while destination has {} columns, " + "the {} extra trailing source column(s) will be ignored", + source_columns.size(), destination_columns.size(), + source_columns.size() - destination_columns.size()); + + Names kept_names; + kept_names.reserve(destination_columns.size()); + for (size_t i = 0; i < destination_columns.size(); ++i) + kept_names.push_back(source_columns[i].name); + + /// `allow_remove_inputs = false` keeps the dropped columns registered as DAG + /// inputs (just not as outputs), so `ExpressionActions::execute` still + /// recognizes and consumes them from the block instead of passing them through + /// unchanged. See the `defaults_dag` merge above for the same pattern. + ActionsDAG trim_dag(source_columns); + trim_dag.removeUnusedActions(kept_names, false); + + auto trim_step = std::make_unique( + plan_for_part.getCurrentHeader(), + std::move(trim_dag)); + trim_step->setStepDescription("Drop source columns beyond destination schema for export"); + plan_for_part.addStep(std::move(trim_step)); + + source_columns = plan_for_part.getCurrentHeader()->getColumnsWithTypeAndName(); + } + + auto dag = ActionsDAG::makeConvertingActions( + source_columns, + destination_columns, + ActionsDAG::MatchColumnsMode::Position, + local_context); + + auto expression_step = std::make_unique( + plan_for_part.getCurrentHeader(), + std::move(dag)); + expression_step->setStepDescription("Convert source columns to destination types for export"); + plan_for_part.addStep(std::move(expression_step)); + } + + String buildDestinationFilename( + const MergeTreePartExportManifest & manifest, + const StorageID & storage_id, + const ContextPtr & local_context) + { + auto filename = manifest.settings[Setting::export_merge_tree_part_filename_pattern].value; + + boost::replace_all(filename, "{part_name}", manifest.data_part->name); + boost::replace_all(filename, "{checksum}", manifest.data_part->checksums.getTotalChecksumHex()); + + Macros::MacroExpansionInfo macro_info; + macro_info.table_id = storage_id; + + if (auto database = DatabaseCatalog::instance().tryGetDatabase(storage_id.database_name)) + { + if (const auto replicated = dynamic_cast(database.get())) + { + macro_info.shard = replicated->getShardName(); + macro_info.replica = replicated->getReplicaName(); + } + } + + filename = local_context->getMacros()->expand(filename, macro_info); + + return filename; + } +} + +ExportPartTask::ExportPartTask(MergeTreeData & storage_, const MergeTreePartExportManifest & manifest_) + : storage(storage_), + manifest(manifest_) +{ +} + +const MergeTreePartExportManifest & ExportPartTask::getManifest() const +{ + return manifest; +} + +bool ExportPartTask::executeStep() +{ + auto local_context = Context::createCopy(storage.getContext()); + local_context->makeQueryContextForExportPart(); + local_context->setCurrentQueryId(manifest.query_id); + local_context->setSettings(manifest.settings); + + const auto & metadata_snapshot = manifest.metadata_snapshot; + + /// Read only physical columns from the part + const auto columns_to_read = metadata_snapshot->getColumns().getNamesOfPhysical(); + + MergeTreeSequentialSourceType read_type = MergeTreeSequentialSourceType::Export; + + Block block_with_partition_values; + if (metadata_snapshot->hasPartitionKey()) + { + /// todo arthur do I need to init minmax_idx? + block_with_partition_values = manifest.data_part->getMinMaxIndex()->getBlock(storage); + } + + const auto & destination_storage = manifest.destination_storage_ptr; + const auto destination_storage_id = destination_storage->getStorageID(); + + auto exports_list_entry = storage.getContext()->getExportsList().insert( + getStorageID(), + destination_storage_id, + manifest.data_part->getBytesOnDisk(), + manifest.data_part->name, + std::vector{}, + manifest.data_part->rows_count, + manifest.data_part->getBytesOnDisk(), + manifest.data_part->getBytesUncompressedOnDisk(), + manifest.create_time, + manifest.query_id, + local_context); + + /* + This is a hack to fix the issue where S3 is out, ClickHouse keeps retrying S3 requests deep + in the AWS SDK and never check for the `isCancelled()` flag. That prevents the task from being killed / cancelled. It also prevents the table from being dropped. + + Merges and mutations don't suffer from this problem because they don't make requests to S3 :). Select statements + do make requests to S3, but the cancel predicate is properly setup for regular queries. + + I think this is the first time we have a background operation that makes requests to S3, so we need to connect the dots. + + The simples way is this one, and given the release timeline, I am opting for it. + */ + (*exports_list_entry)->thread_group->setCancelPredicate( + [weak_this = weak_from_this()]() -> bool + { + if (auto shared_this = weak_this.lock()) + { + return shared_this->isCancelled(); + } + + return true; + }); + + SinkToStoragePtr sink; + + const auto new_file_path_callback = [&exports_list_entry](const std::string & file_path) + { + std::unique_lock lock((*exports_list_entry)->destination_file_paths_mutex); + (*exports_list_entry)->destination_file_paths.push_back(file_path); + }; + + try + { + ThreadGroupSwitcher switcher((*exports_list_entry)->thread_group, ThreadName::EXPORT_PART); + + fiu_do_on(FailPoints::export_part_non_retryable_throw, + { + throw Exception(ErrorCodes::BAD_ARGUMENTS, + "Failpoint: export_part_non_retryable_throw"); + }); + + fiu_do_on(FailPoints::export_part_retryable_throw, + { + throw Exception(ErrorCodes::FAULT_INJECTED, + "Failpoint: export_part_retryable_throw"); + }); + + const auto filename = buildDestinationFilename(manifest, storage.getStorageID(), local_context); + + sink = destination_storage->import( + filename, + block_with_partition_values, + new_file_path_callback, + manifest.file_already_exists_policy, + manifest.settings[Setting::export_merge_tree_part_max_bytes_per_file], + manifest.settings[Setting::export_merge_tree_part_max_rows_per_file], + manifest.iceberg_metadata_json, + getFormatSettings(local_context), + local_context); + + bool apply_deleted_mask = true; + bool read_with_direct_io = local_context->getSettingsRef()[Setting::min_bytes_to_use_direct_io] > manifest.data_part->getBytesOnDisk(); + bool prefetch = false; + + MergeTreeData::IMutationsSnapshot::Params mutations_snapshot_params + { + .metadata_version = metadata_snapshot->getMetadataVersion(), + .min_part_metadata_version = manifest.data_part->getMetadataVersion() + }; + + auto mutations_snapshot = storage.getMutationsSnapshot(mutations_snapshot_params); + auto alter_conversions = MergeTreeData::getAlterConversionsForPart( + manifest.data_part, + mutations_snapshot, + local_context); + + QueryPlan plan_for_part; + + createReadFromPartStep( + read_type, + plan_for_part, + storage, + storage.getStorageSnapshot(metadata_snapshot, local_context), + RangesInDataPart(manifest.data_part), + alter_conversions, + nullptr, + columns_to_read, + nullptr, + apply_deleted_mask, + std::nullopt, + read_with_direct_io, + prefetch, + local_context, + getLogger("ExportPartition")); + + /// We need to support exporting materialized and alias columns to object storage. For some reason, object storage engines don't support them. + /// This is a hack that materializes the columns before the export so they can be exported to tables that have matching columns + materializeSpecialColumns(plan_for_part.getCurrentHeader(), metadata_snapshot, local_context, plan_for_part); + + /// Align the pipeline header with the destination's non-materialized sample block, + /// using the same `makeConvertingActions(Position)` call INSERT SELECT performs. + addExportConvertingActions(plan_for_part, *destination_storage, local_context); + + QueryPlanOptimizationSettings optimization_settings(local_context); + auto pipeline_settings = BuildQueryPipelineSettings(local_context); + auto builder = plan_for_part.buildQueryPipeline(optimization_settings, pipeline_settings); + + builder->setProgressCallback([&exports_list_entry](const Progress & progress) + { + (*exports_list_entry)->bytes_read_uncompressed += progress.read_bytes; + (*exports_list_entry)->rows_read += progress.read_rows; + }); + + pipeline = QueryPipelineBuilder::getPipeline(std::move(*builder)); + + pipeline.complete(sink); + + CompletedPipelineExecutor exec(pipeline); + + auto is_cancelled_callback = [this]() + { + return isCancelled(); + }; + + exec.setCancelCallback(is_cancelled_callback, 100); + + if (isCancelled()) + { + throw Exception(ErrorCodes::QUERY_WAS_CANCELLED, "Export part was cancelled"); + } + + exec.execute(); + + if (isCancelled()) + { + throw Exception(ErrorCodes::QUERY_WAS_CANCELLED, "Export part was cancelled"); + } + + /// For the direct EXPORT PART → Iceberg path there is no deferred-commit callback + /// (the partition-export path provides one that writes to ZooKeeper). + /// Commit the Iceberg metadata inline here so the rows become visible immediately. + if (destination_storage->isDataLake() && !manifest.completion_callback) + { + IStorage::IcebergCommitExportPartitionArguments iceberg_args; + iceberg_args.metadata_json_string = manifest.iceberg_metadata_json; + iceberg_args.partition_source_block = block_with_partition_values; + + destination_storage->commitExportPartitionTransaction( + manifest.transaction_id, + manifest.data_part->info.getPartitionId(), + (*exports_list_entry)->destination_file_paths, + iceberg_args, + local_context); + } + + std::lock_guard inner_lock(storage.export_manifests_mutex); + storage.writePartLog( + PartLogElement::Type::EXPORT_PART, + {}, + (*exports_list_entry)->watch.elapsed(), + manifest.data_part->name, + manifest.data_part, + {manifest.data_part}, + nullptr, + nullptr, + {}, + {}, + exports_list_entry.get()); + + storage.export_manifests.erase(manifest); + + ProfileEvents::increment(ProfileEvents::PartsExports); + ProfileEvents::increment(ProfileEvents::PartsExportTotalMilliseconds, (*exports_list_entry)->watch.elapsedMilliseconds()); + + if (manifest.completion_callback) + manifest.completion_callback(MergeTreePartExportManifest::CompletionCallbackResult::createSuccess((*exports_list_entry)->destination_file_paths)); + } + catch (const Exception & e) + { + /// If an exception is thrown before the pipeline is started, the sink will not be canceled and might leave buffers open. + /// Cancel it manually to ensure the buffers are closed. + if (sink) + { + sink->cancel(); + } + + if (e.code() == ErrorCodes::FILE_ALREADY_EXISTS) + { + ProfileEvents::increment(ProfileEvents::PartsExportDuplicated); + + /// File already exists and the policy is NO_OP, treat it as success. + if (manifest.file_already_exists_policy == MergeTreePartExportManifest::FileAlreadyExistsPolicy::skip) + { + storage.writePartLog( + PartLogElement::Type::EXPORT_PART, + {}, + (*exports_list_entry)->watch.elapsed(), + manifest.data_part->name, + manifest.data_part, + {manifest.data_part}, + nullptr, + nullptr, + {}, + {}, + exports_list_entry.get()); + + std::lock_guard inner_lock(storage.export_manifests_mutex); + storage.export_manifests.erase(manifest); + + ProfileEvents::increment(ProfileEvents::PartsExports); + ProfileEvents::increment(ProfileEvents::PartsExportTotalMilliseconds, (*exports_list_entry)->watch.elapsedMilliseconds()); + + if (manifest.completion_callback) + { + manifest.completion_callback(MergeTreePartExportManifest::CompletionCallbackResult::createSuccess((*exports_list_entry)->destination_file_paths)); + } + + return false; + } + } + + ProfileEvents::increment(ProfileEvents::PartsExportFailures); + + storage.writePartLog( + PartLogElement::Type::EXPORT_PART, + ExecutionStatus::fromCurrentException("", true), + (*exports_list_entry)->watch.elapsed(), + manifest.data_part->name, + manifest.data_part, + {manifest.data_part}, + nullptr, + nullptr, + {}, + {}, + exports_list_entry.get()); + + std::lock_guard inner_lock(storage.export_manifests_mutex); + storage.export_manifests.erase(manifest); + + if (manifest.completion_callback) + manifest.completion_callback(MergeTreePartExportManifest::CompletionCallbackResult::createFailure(e)); + return false; + } + + return false; +} + +void ExportPartTask::cancel() noexcept +{ + LOG_INFO(getLogger("ExportPartTask"), "Export part {} task cancel() method called", manifest.data_part->name); + cancel_requested.store(true); + pipeline.cancel(); +} + +bool ExportPartTask::isCancelled() const +{ + return cancel_requested.load() || storage.parts_mover.moves_blocker.isCancelled(); +} + +void ExportPartTask::onCompleted() +{ +} + +StorageID ExportPartTask::getStorageID() const +{ + return storage.getStorageID(); +} + +Priority ExportPartTask::getPriority() const +{ + return Priority{}; +} + +String ExportPartTask::getQueryId() const +{ + return manifest.query_id; +} + +} diff --git a/src/Storages/MergeTree/ExportPartTask.h b/src/Storages/MergeTree/ExportPartTask.h new file mode 100644 index 000000000000..1596f2bf23c9 --- /dev/null +++ b/src/Storages/MergeTree/ExportPartTask.h @@ -0,0 +1,34 @@ +#pragma once + +#include +#include +#include + +namespace DB +{ + +class ExportPartTask : public IExecutableTask, public std::enable_shared_from_this +{ +public: + explicit ExportPartTask( + MergeTreeData & storage_, + const MergeTreePartExportManifest & manifest_); + bool executeStep() override; + void onCompleted() override; + StorageID getStorageID() const override; + Priority getPriority() const override; + String getQueryId() const override; + const MergeTreePartExportManifest & getManifest() const; + + void cancel() noexcept override; + +private: + MergeTreeData & storage; + MergeTreePartExportManifest manifest; + QueryPipeline pipeline; + std::atomic cancel_requested = false; + + bool isCancelled() const; +}; + +} diff --git a/src/Storages/MergeTree/ExportPartitionManifestUpdatingTask.cpp b/src/Storages/MergeTree/ExportPartitionManifestUpdatingTask.cpp new file mode 100644 index 000000000000..077f06349c5d --- /dev/null +++ b/src/Storages/MergeTree/ExportPartitionManifestUpdatingTask.cpp @@ -0,0 +1,949 @@ +#include +#include +#include +#include "Storages/MergeTree/ExportPartitionUtils.h" +#include "Common/logger_useful.h" +#include +#include +#include +#include +#include +#include +#include +#include +#include + +namespace ProfileEvents +{ + extern const Event ExportPartitionZooKeeperRequests; + extern const Event ExportPartitionZooKeeperGet; + extern const Event ExportPartitionZooKeeperGetChildren; + extern const Event ExportPartitionZooKeeperGetChildrenWatch; + extern const Event ExportPartitionZooKeeperGetWatch; + extern const Event ExportPartitionZooKeeperRemoveRecursive; + extern const Event ExportPartitionZooKeeperMulti; +} + +namespace DB +{ + +namespace ErrorCodes +{ + extern const int FAULT_INJECTED; +} + +namespace FailPoints +{ + extern const char export_partition_status_change_throw[]; + extern const char export_partition_processed_paths_sync_fail[]; +} + +namespace +{ + /// Value published into destination_file_paths when a processed/ Keeper refresh + /// is incomplete (or a leaf is unreadable), so system.replicated_partition_exports + /// can show that the in-memory mirror failed to sync instead of silently under-counting. + constexpr std::string_view zk_sync_failed_marker = ""; + + /// Describes pending commits + struct CommitRecoveryWork + { + ExportReplicatedMergeTreePartitionManifest metadata; + std::string entry_path; + StoragePtr destination_storage; + ContextPtr context; + }; + + /// Fetch all per-replica last_exception leaves under /last_exception and build + /// a fresh map keyed by replica name. + std::optional> readLastExceptionPerReplica( + const zkutil::ZooKeeperPtr & zk, + const std::filesystem::path & entry_path, + const std::string & log_key, + const LoggerPtr & log) + { + std::map out; + + const auto container_path = entry_path / "last_exception"; + + Strings children; + ProfileEvents::increment(ProfileEvents::ExportPartitionZooKeeperRequests); + ProfileEvents::increment(ProfileEvents::ExportPartitionZooKeeperGetChildren); + if (Coordination::Error::ZOK != zk->tryGetChildren(container_path, children)) + { + LOG_WARNING(log, "ExportPartition Manifest Updating Task: failed to list last_exception leaves for {}, leaving in-memory copy untouched", log_key); + return std::nullopt; + } + + if (children.empty()) + return out; + + std::vector paths; + paths.reserve(children.size()); + for (const auto & child : children) + paths.emplace_back(container_path / child); + + ProfileEvents::increment(ProfileEvents::ExportPartitionZooKeeperRequests); + ProfileEvents::increment(ProfileEvents::ExportPartitionZooKeeperGet, paths.size()); + auto responses = zk->tryGet(paths); + responses.waitForResponses(); + + for (size_t i = 0; i < paths.size(); ++i) + { + Coordination::GetResponse response; + try + { + /// MultiTryGetResponse::operator[] swallows ZNONODE but rethrows on + /// other errors; treat any unexpected Keeper error as "skip this + /// leaf, retry on the next poll". Matches the lenient semantics of + /// the previous per-leaf tryGet implementation. + response = responses[i]; + } + catch (...) + { + LOG_WARNING(log, "ExportPartition Manifest Updating Task: ZK error fetching last_exception leaf {} for {}, skipping", children[i], log_key); + continue; + } + + if (response.error != Coordination::Error::ZOK) + continue; /// ZNONODE: child concurrently removed (recursive cleanup race). + + try + { + auto entry = LastExceptionEntry::fromJsonString(response.data); + String replica = entry.replica.empty() ? unescapeForFileName(children[i]) : entry.replica; + out.emplace(std::move(replica), std::move(entry)); + } + catch (...) + { + LOG_WARNING(log, "ExportPartition Manifest Updating Task: malformed last_exception JSON for {} (leaf {}), ignoring", log_key, children[i]); + } + } + + return out; + } + + std::map> readDestinationFilePathsPerPart( + const zkutil::ZooKeeperPtr & zk, + const std::filesystem::path & entry_path, + const std::string & log_key, + const LoggerPtr & log) + { + std::map> out; + + const auto container_path = entry_path / "processed"; + + Strings children; + ProfileEvents::increment(ProfileEvents::ExportPartitionZooKeeperRequests); + ProfileEvents::increment(ProfileEvents::ExportPartitionZooKeeperGetChildren); + if (Coordination::Error::ZOK != zk->tryGetChildren(container_path, children)) + { + LOG_INFO(log, "ExportPartition Manifest Updating Task: failed to list processed leaves for {}, publishing sync-failed marker", log_key); + out.emplace(String(zk_sync_failed_marker), std::vector{String(zk_sync_failed_marker)}); + return out; + } + + if (children.empty()) + return out; + + std::vector paths; + paths.reserve(children.size()); + for (const auto & child : children) + paths.emplace_back(container_path / child); + + ProfileEvents::increment(ProfileEvents::ExportPartitionZooKeeperRequests); + ProfileEvents::increment(ProfileEvents::ExportPartitionZooKeeperGet, paths.size()); + auto responses = zk->tryGet(paths); + responses.waitForResponses(); + + for (size_t i = 0; i < paths.size(); ++i) + { + Coordination::GetResponse response; + try + { + /// Simulate a non-ZNONODE multi-get failure so the catch path below + /// publishes the sync-failed marker (same shape as operator[] rethrow). + fiu_do_on(FailPoints::export_partition_processed_paths_sync_fail, + { + throw zkutil::KeeperException(Coordination::Error::ZCONNECTIONLOSS); + }); + response = responses[i]; + } + catch (...) + { + LOG_WARNING(log, "ExportPartition Manifest Updating Task: ZK error fetching processed leaf {} for {}, publishing sync-failed marker", children[i], log_key); + out.emplace(children[i], std::vector{String(zk_sync_failed_marker)}); + continue; + } + + if (response.error != Coordination::Error::ZOK) + { + LOG_WARNING(log, "ExportPartition Manifest Updating Task: could not read processed leaf {} for {} (error {}), publishing sync-failed marker", children[i], log_key, response.error); + out.emplace(children[i], std::vector{String(zk_sync_failed_marker)}); + continue; + } + + try + { + auto entry = ExportReplicatedMergeTreePartitionProcessedPartEntry::fromJsonString(response.data); + out.emplace(std::move(entry.part_name), std::move(entry.paths_in_destination)); + } + catch (...) + { + LOG_WARNING(log, "ExportPartition Manifest Updating Task: malformed processed JSON for {} (leaf {}), publishing sync-failed marker", log_key, children[i]); + out.emplace(children[i], std::vector{String(zk_sync_failed_marker)}); + } + } + + return out; + } + + /// True when the cached `/processed` mirror carries a `zk_sync_failed_marker` sentinel, + /// published whenever a listing or a leaf read/parse failed. Such a mirror is incomplete + /// and must be refreshed again on the next poll. + bool destinationFilePathsMirrorHasSyncFailure(const std::map> & cached_paths) + { + for (const auto & [part_name, destination_paths] : cached_paths) + { + if (part_name == zk_sync_failed_marker) + return true; + for (const auto & destination_path : destination_paths) + if (destination_path == zk_sync_failed_marker) + return true; + } + return false; + } + + bool skipReadingDestinationFilePaths( + ExportReplicatedMergeTreePartitionTaskEntry::Status status, + const std::map> & cached_paths, + size_t number_of_parts) + { + if (status == ExportReplicatedMergeTreePartitionTaskEntry::Status::PENDING) + return false; + if (destinationFilePathsMirrorHasSyncFailure(cached_paths)) + return false; + if (status == ExportReplicatedMergeTreePartitionTaskEntry::Status::COMPLETED) + return cached_paths.size() == number_of_parts; + return true; + } + + /// Read the optional /commit_info znode and return the parsed entry. + /// Returns nullopt when the znode is absent (task has not committed yet, peer + /// crashed before writing it, or transient ZK error). Callers should treat + /// nullopt as "leave the in-memory copy untouched". + std::optional readCommitInfo( + const zkutil::ZooKeeperPtr & zk, + const std::filesystem::path & entry_path, + const std::string & log_key, + const LoggerPtr & log) + { + const auto commit_info_path = entry_path / "commit_info"; + + std::string data; + ProfileEvents::increment(ProfileEvents::ExportPartitionZooKeeperRequests); + ProfileEvents::increment(ProfileEvents::ExportPartitionZooKeeperGet); + if (!zk->tryGet(commit_info_path, data)) + return std::nullopt; + + try + { + return ExportReplicatedMergeTreePartitionCommitInfoEntry::fromJsonString(data); + } + catch (...) + { + LOG_WARNING(log, "ExportPartition Manifest Updating Task: malformed commit_info JSON for {}, ignoring", log_key); + return std::nullopt; + } + } + + /// collects pending commits and kills tasks that have timed out + void tryCleanup( + const zkutil::ZooKeeperPtr & zk, + const std::string & entry_path, + const LoggerPtr & log, + const ContextPtr & storage_context, + StorageReplicatedMergeTree & storage, + const ExportReplicatedMergeTreePartitionManifest & metadata, + const time_t now, + const bool is_pending, + std::vector & deferred_commits + ) + { + bool task_timed_out = is_pending + && metadata.task_timeout_seconds > 0 + && metadata.create_time + static_cast(metadata.task_timeout_seconds) < now; + + if (task_timed_out) + { + /// Serialize against commit(): don't kill a task whose commit is in progress. + auto commit_lock = zkutil::EphemeralNodeHolder::tryCreate( + fs::path(entry_path) / "commit_lock", *zk, storage.getReplicaName()); + if (!commit_lock) + { + LOG_DEBUG(log, "ExportPartition Manifest Updating Task: commit in progress for {}, skipping timeout kill", entry_path); + return; + } + + const std::string status_path = fs::path(entry_path) / "status"; + + Coordination::Stat status_stat; + std::string status_string; + + ProfileEvents::increment(ProfileEvents::ExportPartitionZooKeeperRequests); + ProfileEvents::increment(ProfileEvents::ExportPartitionZooKeeperGet); + if (!zk->tryGet(status_path, status_string, &status_stat)) + { + LOG_WARNING(log, "ExportPartition Manifest Updating Task: Failed to read status for {} while enforcing task timeout, skipping", entry_path); + return; + } + + const auto current_status = magic_enum::enum_cast(status_string); + if (!current_status || *current_status != ExportReplicatedMergeTreePartitionTaskEntry::Status::PENDING) + { + LOG_DEBUG(log, "ExportPartition Manifest Updating Task: Task {} is not PENDING, can't set to KILLED, skipping", entry_path); + return; + } + + const auto timeout_message = fmt::format( + "Export partition task timed out: exceeded export_merge_tree_partition_task_timeout_seconds={} (created at {}, now {})", + metadata.task_timeout_seconds, metadata.create_time, now); + + const auto killed_name = String(magic_enum::enum_name(ExportReplicatedMergeTreePartitionTaskEntry::Status::KILLED)); + + Coordination::Requests ops; + ExportPartitionUtils::appendExceptionOps( + ops, zk, fs::path(entry_path), storage.getReplicaName(), + /*part_name=*/"", timeout_message, log); + + ops.emplace_back(zkutil::makeSetRequest(status_path, killed_name, status_stat.version)); + + ProfileEvents::increment(ProfileEvents::ExportPartitionZooKeeperRequests); + ProfileEvents::increment(ProfileEvents::ExportPartitionZooKeeperMulti); + + Coordination::Responses responses; + const auto rc = zk->tryMulti(ops, responses); + + if (rc == Coordination::Error::ZOK) + { + LOG_WARNING(log, + "ExportPartition Manifest Updating Task: task {} exceeded task_timeout_seconds={}s, " + "transitioned PENDING -> KILLED (atomic with exception record)", + entry_path, metadata.task_timeout_seconds); + } + else + { + /// ZBADVERSION (status changed), ZNODEEXISTS (lazy-create race with the scheduler), + /// counter race, or ZNONODE (entry concurrently removed). In all cases the batch + /// was rolled back atomically and the task will be re-evaluated on the next poll. + LOG_DEBUG(log, + "ExportPartition Manifest Updating Task: atomic kill for {} failed (rc={}); " + "status was concurrently updated or a ZK op conflicted, will retry on next poll", + entry_path, rc); + } + + /// The entry remains in entries_by_key; the status watch will drive + /// handleStatusChanges -> killExportPart on every replica, mirroring user-initiated KILL. + return; + } + else if (is_pending) + { + auto context = ExportPartitionUtils::getContextCopyWithTaskSettings(storage_context, metadata); + + ProfileEvents::increment(ProfileEvents::ExportPartitionZooKeeperRequests); + ProfileEvents::increment(ProfileEvents::ExportPartitionZooKeeperGetChildren); + std::vector parts_in_processing_or_pending; + if (Coordination::Error::ZOK != zk->tryGetChildren(fs::path(entry_path) / "processing", parts_in_processing_or_pending)) + { + + LOG_WARNING(log, "ExportPartition Manifest Updating Task: Failed to get parts in processing or pending, skipping"); + return; + } + + if (parts_in_processing_or_pending.empty()) + { + LOG_DEBUG(log, "ExportPartition Manifest Updating Task: Cleanup found PENDING for {} with all parts exported, deferring commit recovery to post-lock phase", entry_path); + + const auto destination_storage_id = StorageID(QualifiedTableName {metadata.destination_database, metadata.destination_table}); + const auto destination_storage = DatabaseCatalog::instance().tryGetTable(destination_storage_id, context); + if (!destination_storage) + { + LOG_WARNING(log, "ExportPartition Manifest Updating Task: Failed to reconstruct destination storage: {}, skipping", destination_storage_id.getNameForLogs()); + return; + } + + /// A replica exported the last part but the commit never landed + deferred_commits.push_back(CommitRecoveryWork{ + .metadata = metadata, + .entry_path = entry_path, + .destination_storage = destination_storage, + .context = context, + }); + } + } + } +} + +ExportPartitionManifestUpdatingTask::ExportPartitionManifestUpdatingTask(StorageReplicatedMergeTree & storage_) + : storage(storage_) +{ +} + +std::vector ExportPartitionManifestUpdatingTask::getPartitionExportsInfo() const +{ + const auto model = storage.export_partition_manifests.get(); + + if (!model) + return {}; + + const auto backoff = storage.export_merge_tree_partition_task_scheduler->getLocalBackoffSnapshot(); + + std::vector infos; + infos.reserve(model->size()); + + for (const auto & entry : model->get()) + { + const auto & manifest = entry.manifest; + + ReplicatedPartitionExportInfo info; + + info.destination_database = manifest.destination_database; + info.destination_table = manifest.destination_table; + info.partition_id = manifest.partition_id; + info.transaction_id = manifest.transaction_id; + info.query_id = manifest.query_id; + info.create_time = manifest.create_time; + info.source_replica = manifest.source_replica; + info.parts_count = manifest.number_of_parts; + info.parts_to_do = manifest.parts.size(); + info.parts = manifest.parts; + info.status = magic_enum::enum_name(entry.status); + + info.last_exception_per_replica.reserve(entry.last_exception_per_replica.size()); + size_t total_exception_count = 0; + for (const auto & [_, ex] : entry.last_exception_per_replica) + { + total_exception_count += ex.count; + info.last_exception_per_replica.push_back(ex); + } + info.exception_count = total_exception_count; + + info.destination_file_paths_per_part = entry.destination_file_paths_per_part; + + if (entry.commit_info) + { + info.committed_metadata_file = entry.commit_info->iceberg_metadata_file; + info.committed_manifest_list = entry.commit_info->iceberg_manifest_list; + info.committed_manifest_file = entry.commit_info->iceberg_manifest_file; + info.committed_marker_file = entry.commit_info->commit_marker_file; + } + + if (const auto it = backoff.find(entry.getTransactionId()); it != backoff.end()) + { + info.backoff_per_part.reserve(it->second.size()); + for (const auto & [part_name, state] : it->second) + info.backoff_per_part.push_back({part_name, state.attempts, state.next_retry_time}); + } + + infos.emplace_back(std::move(info)); + } + + return infos; +} + +void ExportPartitionManifestUpdatingTask::poll() +{ + /// Commit-recovery work collected while the storage-wide mutex is held. + /// Executed AFTER the mutex is released - committing to Iceberg/REST-catalog can take + /// many seconds (up to MAX_TRANSACTION_RETRIES=100 catalog round-trips) and blocking + /// `system.replicated_partition_exports` for that long is what we are fixing here. + std::vector deferred_commits; + + auto zk = storage.getZooKeeper(); + const auto log = storage.log.load(); + + const std::string exports_path = fs::path(storage.zookeeper_path) / "exports"; + const std::string cleanup_lock_path = fs::path(storage.zookeeper_path) / "exports_cleanup_lock"; + + /// The `exports_cleanup_lock` is an ephemeral ZK node that serializes cleanup work + /// across replicas: only the replica holding it walks `tryCleanup` (task-timeout + /// enforcement + commit recovery). It MUST outlive the deferred-commit loop below; otherwise a peer + /// replica's next poll() could acquire it and race us on the same commit-recovery work, + /// duplicating REST-catalog round-trips and snapshot writes. + auto cleanup_lock = zkutil::EphemeralNodeHolder::tryCreate(cleanup_lock_path, *zk, storage.replica_name); + if (cleanup_lock) + { + LOG_DEBUG(log, "ExportPartition Manifest Updating Task: Cleanup lock acquired, will remove stale entries"); + } + + { + /// M_task: serializes poll() vs handleStatusChanges(). We copy the current read-model into a + /// private mutable container, mutate that copy across the ZooKeeper reads below, and publish + /// it atomically via export_read_model.set() at the end. Readers never see partial updates. + std::lock_guard task_guard(background_task_serialization_mutex); + + const auto current_model = storage.export_partition_manifests.get(); + + auto working_model = current_model + ? std::make_unique(*current_model) + : std::make_unique(); + + auto & entries_by_key = working_model->get(); + + LOG_DEBUG(log, "ExportPartition Manifest Updating Task: Polling for new entries for table {}. Current number of entries: {}", storage.getStorageID().getNameForLogs(), entries_by_key.size()); + + ProfileEvents::increment(ProfileEvents::ExportPartitionZooKeeperRequests); + ProfileEvents::increment(ProfileEvents::ExportPartitionZooKeeperGetChildrenWatch); + + Coordination::Stat stat; + const auto children = zk->getChildrenWatch(exports_path, &stat, storage.export_merge_tree_partition_watch_callback); + const std::unordered_set zk_children(children.begin(), children.end()); + + const auto now = time(nullptr); + + /// Load new entries + /// If we have the cleanup lock, also remove stale entries from zk and local + /// Upload dangling commit files if any + for (const auto & key : zk_children) + { + const std::string entry_path = fs::path(exports_path) / key; + + ProfileEvents::increment(ProfileEvents::ExportPartitionZooKeeperRequests); + ProfileEvents::increment(ProfileEvents::ExportPartitionZooKeeperGet); + std::string metadata_json; + if (!zk->tryGet(fs::path(entry_path) / "metadata.json", metadata_json)) + { + LOG_WARNING(log, "ExportPartition Manifest Updating Task: Skipping {}: missing metadata.json", key); + continue; + } + + ExportReplicatedMergeTreePartitionManifest metadata; + try + { + metadata = ExportReplicatedMergeTreePartitionManifest::fromJsonString(metadata_json); + } + catch (...) + { + /// A single unparseable metadata.json (e.g. genuinely corrupt, or written by a + /// future incompatible format) must not abort the whole poll and stall discovery, + /// cleanup and status convergence for every other task. Skip just this entry. + tryLogCurrentException(log, __PRETTY_FUNCTION__); + LOG_WARNING(log, "ExportPartition Manifest Updating Task: Skipping {}: could not parse metadata.json", key); + continue; + } + + auto last_exception_per_replica = readLastExceptionPerReplica( + zk, fs::path(entry_path), key, log); + + /// If the zk entry has been replaced with export_merge_tree_partition_force_export, checking only for the export key is not enough + /// we need to make sure it is the same transaction id. If it is not, it needs to be replaced. + const auto local_entry = entries_by_key.find(key); + const bool has_local_entry = local_entry != entries_by_key.end() + && local_entry->manifest.transaction_id == metadata.transaction_id; + + std::string status_string; + + /// In theory, we should be notified when the status changes by the status watch + /// but in practice, the watch is not always reliable (e.g. if the ZooKeeper session is lost) + /// so we need to read the status from the ZK node directly. + if (has_local_entry) + { + ProfileEvents::increment(ProfileEvents::ExportPartitionZooKeeperRequests); + ProfileEvents::increment(ProfileEvents::ExportPartitionZooKeeperGet); + + zk->tryGet(fs::path(entry_path) / "status", status_string); + } + else + { + /// If we don't have a local entry, we need to arm a status watch to be notified when the status changes + std::weak_ptr weak_manifest_updater = storage.export_merge_tree_partition_manifest_updater; + auto status_watch_callback = std::make_shared([weak_manifest_updater, key](const Coordination::WatchResponse &) + { + /// If the table is dropped but the watch is not removed, we need to prevent use after free + /// below code assumes that if manifest updater is still alive, the status handling task is also alive + if (auto manifest_updater = weak_manifest_updater.lock()) + { + manifest_updater->addStatusChange(key); + manifest_updater->storage.export_merge_tree_partition_status_handling_task->schedule(); + } + }); + + ProfileEvents::increment(ProfileEvents::ExportPartitionZooKeeperRequests); + ProfileEvents::increment(ProfileEvents::ExportPartitionZooKeeperGetWatch); + + zk->tryGetWatch(fs::path(entry_path) / "status", status_string, nullptr, status_watch_callback); + } + + if (status_string.empty()) + { + LOG_WARNING(log, "ExportPartition Manifest Updating Task: Skipping {}: missing status", key); + continue; + } + + const auto status = magic_enum::enum_cast(status_string); + if (!status) + { + LOG_WARNING(log, "ExportPartition Manifest Updating Task: Invalid status {} for task {}, skipping", status_string, key); + continue; + } + + const bool skip_processed_refresh = + has_local_entry + && skipReadingDestinationFilePaths(*status, local_entry->destination_file_paths_per_part, metadata.number_of_parts); + + std::optional>> destination_file_paths_per_part; + if (!skip_processed_refresh) + destination_file_paths_per_part = readDestinationFilePathsPerPart( + zk, fs::path(entry_path), key, log); + + /// If we hold the cleanup lock, enforce the task timeout and recover uncommitted exports. + /// Entries are never removed here, so we always fall through to refresh / addTask below. + if (cleanup_lock) + { + tryCleanup( + zk, + entry_path, + log, + storage.getContext(), + storage, + metadata, + now, + *status == ExportReplicatedMergeTreePartitionTaskEntry::Status::PENDING, + deferred_commits); + } + + if (!has_local_entry) + { + addTask( + metadata, + *status, + last_exception_per_replica ? std::move(*last_exception_per_replica) : std::map{}, + destination_file_paths_per_part ? std::move(*destination_file_paths_per_part) : std::map>{}, + readCommitInfo(zk, fs::path(entry_path), key, log), + key, + entries_by_key); + LOG_INFO(log, "ExportPartition Manifest Updating Task: Added new entry for task {}", key); + continue; + } + + if (!local_entry->commit_info && *status == ExportReplicatedMergeTreePartitionTaskEntry::Status::COMPLETED) + { + local_entry->commit_info = readCommitInfo(zk, fs::path(entry_path), key, log); + } + + /// If we already have the local entry, we need to update it + if (last_exception_per_replica) + local_entry->last_exception_per_replica = std::move(*last_exception_per_replica); + if (destination_file_paths_per_part) + local_entry->destination_file_paths_per_part = std::move(*destination_file_paths_per_part); + + const bool status_changed = local_entry->status != *status; + if (status_changed) + { + local_entry->status = *status; + if (local_entry->status != ExportReplicatedMergeTreePartitionTaskEntry::Status::PENDING) + { + /// terminal now - we no longer need to keep the data parts alive + local_entry->part_references.clear(); + + /// looks like we missed a status change event, we should kill local operations. + if (local_entry->status == ExportReplicatedMergeTreePartitionTaskEntry::Status::KILLED) + { + storage.killExportPart(local_entry->manifest.transaction_id); + } + } + } + + LOG_DEBUG(log, "ExportPartition Manifest Updating Task: Skipping {}: already exists", key); + + } + + removeStaleEntries(zk_children, entries_by_key); + + const auto entries_count = entries_by_key.size(); + + /// Publish the updated copy atomically. `working_model` is moved out here, so + /// `entries_by_key` (a reference into it) must not be used afterwards. + storage.export_partition_manifests.set(std::move(working_model)); + + LOG_DEBUG(log, "ExportPartition Manifest Updating task: finished polling for new entries. Number of entries: {}", entries_count); + } + + /// Execute pending commits + for (const auto & work : deferred_commits) + { + /// A replica exported the last part but the commit never landed. Try to fix it. + try + { + ExportPartitionUtils::commit(work.metadata, work.destination_storage, zk, log, work.entry_path, work.context, storage, storage.getReplicaName()); + } + catch (const Exception & e) + { + LOG_WARNING(log, + "ExportPartition Manifest Updating Task: " + "Caught exception while committing export for {}: {}", + work.entry_path, e.message()); + + const bool became_failed = ExportPartitionUtils::handleCommitFailure( + zk, + work.entry_path, + e.code(), + storage.getReplicaName(), + e.message(), + log); + + if (became_failed) + { + LOG_WARNING(log, + "ExportPartition Manifest Updating Task: " + "Commit for {} transitioned to FAILED due to non-retryable error (code {})", + work.entry_path, e.code()); + } + } + } + + storage.export_merge_tree_partition_select_task->schedule(); +} + +void ExportPartitionManifestUpdatingTask::addTask( + const ExportReplicatedMergeTreePartitionManifest & metadata, + ExportReplicatedMergeTreePartitionTaskEntry::Status status, + std::map last_exception_per_replica, + std::map> destination_file_paths_per_part, + std::optional commit_info, + const std::string & key, + auto & entries_by_key +) +{ + std::vector part_references; + + /// If the status is PENDING, we grab references to the data parts to prevent them from being deleted from the disk + /// Otherwise, the operation has already been completed and there is no need to keep the data parts alive + /// You might also ask: why bother adding tasks that have already been completed (i.e, status != PENDING)? + /// The reason is the `replicated_partition_exports` table might miss entries if they are not added here. + if (status == ExportReplicatedMergeTreePartitionTaskEntry::Status::PENDING) + { + for (const auto & part_name : metadata.parts) + { + if (const auto part = storage.getPartIfExists(part_name, {MergeTreeDataPartState::Active, MergeTreeDataPartState::Outdated})) + { + part_references.push_back(part); + } + } + } + + /// Called from poll() under M_task (sole mutator), so no extra locking is required. + ExportReplicatedMergeTreePartitionTaskEntry entry { + metadata, + status, + std::move(part_references), + std::move(last_exception_per_replica), + std::move(destination_file_paths_per_part), + std::move(commit_info)}; + + auto it = entries_by_key.find(key); + if (it != entries_by_key.end()) + { + if (!entries_by_key.replace(it, entry)) + LOG_ERROR(storage.log, + "ExportPartition Manifest Updating Task: failed to replace in-memory entry for {} (transaction_id {}). " + "This most likely means another export already holds the same transaction_id (id collision); " + "this export will be missing from system.replicated_partition_exports.", + key, entry.getTransactionId()); + } + else if (!entries_by_key.insert(entry).second) + { + LOG_ERROR(storage.log, + "ExportPartition Manifest Updating Task: failed to insert in-memory entry for {} (transaction_id {}). " + "Another entry already holds this transaction_id (id collision); " + "this export will be invisible in system.replicated_partition_exports.", + key, entry.getTransactionId()); + } +} + +void ExportPartitionManifestUpdatingTask::removeStaleEntries( + const std::unordered_set & zk_children, + auto & entries_by_key +) +{ + for (auto it = entries_by_key.begin(); it != entries_by_key.end();) + { + const auto key = it->getCompositeKey(); + if (zk_children.contains(key)) + { + ++it; + continue; + } + + LOG_INFO(storage.log, "ExportPartition Manifest Updating Task: Export task {} was deleted, calling killExportPartition for transaction {}", key, it->manifest.transaction_id); + + try + { + storage.killExportPart(it->manifest.transaction_id); + } + catch (...) + { + tryLogCurrentException(storage.log, __PRETTY_FUNCTION__); + } + + it = entries_by_key.erase(it); + } +} + +void ExportPartitionManifestUpdatingTask::addStatusChange(const std::string & key) +{ + std::lock_guard lock(status_changes_mutex); + status_changes.emplace(key); +} + +void ExportPartitionManifestUpdatingTask::handleStatusChanges() +{ + /// copy the events to a local queue to avoid holding status_changes_mutex under M_task + std::queue local_status_changes; + { + std::lock_guard lock(status_changes_mutex); + std::swap(status_changes, local_status_changes); + } + + /// Take a snapshot of all status changes. If an exception is thrown, we will requeue the whole batch. + const std::queue batch = local_status_changes; + const auto log = storage.log.load(); + + try + { + /// M_task: serializes this against poll(). We copy the current read-model into a private + /// mutable container, apply this batch's status transitions to that copy across the ZooKeeper + /// reads below, and publish it atomically via export_read_model.set() at the end. Readers + /// never see partial updates. + std::lock_guard task_guard(background_task_serialization_mutex); + auto zk = storage.getZooKeeper(); + + const bool had_changes = !local_status_changes.empty(); + + LOG_DEBUG(log, "ExportPartition Manifest Updating task: handling status changes. Number of status changes: {}", local_status_changes.size()); + + const auto current_model = storage.export_partition_manifests.get(); + auto working_model = current_model + ? std::make_unique(*current_model) + : std::make_unique(); + auto & entries_by_key = working_model->get(); + + while (!local_status_changes.empty()) + { + const auto & key = local_status_changes.front(); + LOG_INFO(log, "ExportPartition Manifest Updating task: handling status change for task {}", key); + + fiu_do_on(FailPoints::export_partition_status_change_throw, + { + throw Exception(ErrorCodes::FAULT_INJECTED, + "Failpoint: simulating exception during status change handling for key {}", key); + }); + + const auto it = entries_by_key.find(key); + if (it == entries_by_key.end()) + { + local_status_changes.pop(); + continue; + } + + const auto export_path = fs::path(storage.zookeeper_path) / "exports" / key; + + ProfileEvents::increment(ProfileEvents::ExportPartitionZooKeeperRequests); + ProfileEvents::increment(ProfileEvents::ExportPartitionZooKeeperGet); + /// get new status from zk + std::string new_status_string; + if (!zk->tryGet(export_path / "status", new_status_string)) + { + LOG_WARNING(log, "ExportPartition Manifest Updating Task: Failed to get new status for task {}, skipping", key); + local_status_changes.pop(); + continue; + } + + const auto new_status = magic_enum::enum_cast(new_status_string); + if (!new_status) + { + LOG_WARNING(log, "ExportPartition Manifest Updating Task: Invalid status {} for task {}, skipping", new_status_string, key); + local_status_changes.pop(); + continue; + } + + LOG_INFO(log, "ExportPartition Manifest Updating task: status changed for task {}. New status: {}", key, magic_enum::enum_name(*new_status).data()); + + auto fetched = readLastExceptionPerReplica( + zk, export_path, key, log); + + if (!skipReadingDestinationFilePaths(*new_status, it->destination_file_paths_per_part, it->manifest.number_of_parts)) + { + auto destination_file_paths_per_part = readDestinationFilePathsPerPart( + zk, export_path, key, log); + it->destination_file_paths_per_part = std::move(destination_file_paths_per_part); + } + + if (*new_status == ExportReplicatedMergeTreePartitionTaskEntry::Status::COMPLETED) + { + if (auto fetched_commit_info = readCommitInfo(zk, export_path, key, log)) + it->commit_info = std::move(fetched_commit_info); + } + + /// If status changed to KILLED, cancel local export operations + if (*new_status == ExportReplicatedMergeTreePartitionTaskEntry::Status::KILLED) + { + try + { + LOG_INFO(log, "ExportPartition Manifest Updating task: killing export partition for task {}", key); + storage.killExportPart(it->manifest.transaction_id); + } + catch (...) + { + tryLogCurrentException(log, __PRETTY_FUNCTION__); + } + } + + /// Apply the in-memory updates directly (poll() cannot run concurrently under M_task). + if (fetched) + it->last_exception_per_replica = std::move(*fetched); + + it->status = *new_status; + + if (it->status != ExportReplicatedMergeTreePartitionTaskEntry::Status::PENDING) + { + /// we no longer need to keep the data parts alive + it->part_references.clear(); + } + + local_status_changes.pop(); + } + + /// Publish this batch's status transitions to readers. `working_model` is moved out here, + /// so `entries_by_key` (a reference into it) must not be used afterwards. + if (had_changes) + storage.export_partition_manifests.set(std::move(working_model)); + } + catch (...) + { + tryLogCurrentException(log, __PRETTY_FUNCTION__); + + LOG_WARNING(log, "ExportPartition Manifest Updating task: exception thrown while handling status changes; nothing was published, requeuing the whole batch. Batch size: {}", batch.size()); + + std::lock_guard lock(status_changes_mutex); + + /// upon exception, requeue the whole batch + if (!batch.empty()) + { + std::queue requeued = batch; + while (!status_changes.empty()) + { + requeued.push(std::move(status_changes.front())); + status_changes.pop(); + } + + std::swap(status_changes, requeued); + } + + LOG_DEBUG(log, "ExportPartition Manifest Updating task: pending status changes after requeue: {}", status_changes.size()); + + throw; + } +} + +} diff --git a/src/Storages/MergeTree/ExportPartitionManifestUpdatingTask.h b/src/Storages/MergeTree/ExportPartitionManifestUpdatingTask.h new file mode 100644 index 000000000000..129629f6ed9b --- /dev/null +++ b/src/Storages/MergeTree/ExportPartitionManifestUpdatingTask.h @@ -0,0 +1,57 @@ +#pragma once + +#include +#include +#include +#include +#include +#include +namespace DB +{ + +class StorageReplicatedMergeTree; +struct ExportReplicatedMergeTreePartitionManifest; + +class ExportPartitionManifestUpdatingTask +{ +public: + ExportPartitionManifestUpdatingTask(StorageReplicatedMergeTree & storage); + + void poll(); + + void handleStatusChanges(); + + void addStatusChange(const std::string & key); + + /// Returns a snapshot of every replicated partition export task tracked by this + /// replica's in-memory mirror. No ZooKeeper traffic; safe to call from query threads. + std::vector getPartitionExportsInfo() const; + +private: + StorageReplicatedMergeTree & storage; + + void addTask( + const ExportReplicatedMergeTreePartitionManifest & metadata, + ExportReplicatedMergeTreePartitionTaskEntry::Status status, + std::map last_exception_per_replica, + std::map> destination_file_paths_per_part, + std::optional commit_info, + const std::string & key, + auto & entries_by_key + ); + + void removeStaleEntries( + const std::unordered_set & zk_children, + auto & entries_by_key + ); + + std::mutex status_changes_mutex; + std::queue status_changes; + + /// M_task: serializes poll() and handleStatusChanges(). Each builds a private mutable copy of + /// the current read-model, mutates it, and atomically publishes it via export_read_model.set(). + /// Held across ZooKeeper I/O; no reader takes it (readers use export_read_model.get()). + std::mutex background_task_serialization_mutex; +}; + +} diff --git a/src/Storages/MergeTree/ExportPartitionTaskScheduler.cpp b/src/Storages/MergeTree/ExportPartitionTaskScheduler.cpp new file mode 100644 index 000000000000..de271bc01ec7 --- /dev/null +++ b/src/Storages/MergeTree/ExportPartitionTaskScheduler.cpp @@ -0,0 +1,672 @@ +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include "Storages/MergeTree/ExportPartitionUtils.h" +#include "Storages/MergeTree/MergeTreePartExportManifest.h" +#include "Formats/FormatFactory.h" +#include +#include + +namespace ProfileEvents +{ + extern const Event ExportPartitionZooKeeperRequests; + extern const Event ExportPartitionZooKeeperGet; + extern const Event ExportPartitionZooKeeperGetChildren; + extern const Event ExportPartitionZooKeeperCreate; + extern const Event ExportPartitionZooKeeperSet; + extern const Event ExportPartitionZooKeeperRemove; + extern const Event ExportPartitionZooKeeperMulti; + extern const Event ExportPartsRejectedByMemoryLimit; +} + + +namespace DB +{ + +namespace Setting +{ + extern const SettingsMergeTreePartExportFileAlreadyExistsPolicy export_merge_tree_part_file_already_exists_policy; +} + +namespace ErrorCodes +{ + extern const int QUERY_WAS_CANCELLED; + extern const int LOGICAL_ERROR; +} + +namespace +{ + /// Capped exponential back-off, matching the standard ClickHouse convention + /// (see ZooKeeperRetriesControl): delay = min(initial << (retry_count - 1), max). + /// `retry_count` is the number of failures so far (>= 1 when a retry is pending). + /// The shift is guarded against overflow by saturating to `max_backoff_seconds`. + size_t computeRetryBackoffSeconds(size_t retry_count, size_t initial_backoff_seconds, size_t max_backoff_seconds) + { + const size_t initial = std::min(initial_backoff_seconds, max_backoff_seconds); + + if (retry_count <= 1 || initial == 0) + return initial; + + const size_t shift = retry_count - 1; + + /// If shifting would overflow size_t, the result is certainly clamped to the cap. + static constexpr size_t bits = sizeof(size_t) * 8; + if (shift >= bits) + return max_backoff_seconds; + + const size_t headroom = std::numeric_limits::max() >> shift; + if (initial > headroom) + return max_backoff_seconds; + + return std::min(initial << shift, max_backoff_seconds); + } +} + +ExportPartitionTaskScheduler::ExportPartitionTaskScheduler(StorageReplicatedMergeTree & storage_) + : storage(storage_) +{ +} + +std::optional ExportPartitionTaskScheduler::run() +{ + std::optional earliest_backoff_retry; + + const auto available_move_executors = storage.background_moves_assignee.getAvailableMoveExecutors(); + + /// this is subject to TOCTOU - but for now we choose to live with it. + if (available_move_executors == 0) + { + LOG_DEBUG(storage.log, "ExportPartition scheduler task: No available move executors, skipping"); + return earliest_backoff_retry; + } + + /// Respect the background memory soft-limit: refuse to schedule new export-part tasks when + /// background tasks are already pressing the limit. The task is rescheduled by the parent + /// background pool a few seconds later, so this just defers work without losing it. + if (!canEnqueueBackgroundTask()) + { + ProfileEvents::increment(ProfileEvents::ExportPartsRejectedByMemoryLimit); + LOG_TRACE(storage.log, + "ExportPartition scheduler task: Reached memory limit for the background tasks ({}), " + "so won't select new parts to export. Current background tasks memory usage: {}.", + formatReadableSizeWithBinarySuffix(background_memory_tracker.getSoftLimit()), + formatReadableSizeWithBinarySuffix(background_memory_tracker.get())); + return earliest_backoff_retry; + } + + LOG_DEBUG(storage.log, "ExportPartition scheduler task: Available move executors: {}", available_move_executors); + + std::size_t scheduled_exports_count = 0; + + const uint32_t seed = uint32_t(std::hash{}(storage.replica_name)) ^ uint32_t(scheduled_exports_count); + pcg64_fast rng(seed); + + /// Hold the published snapshot for the whole pass and iterate it directly (sorted by + /// create_time). It is immutable and the shared_ptr copy never blocks the writer. The scheduler + /// is a pure reader; status converges via the status watch -> handleStatusChanges and poll(). + const auto model = storage.export_partition_manifests.get(); + if (!model) + return earliest_backoff_retry; + + auto zk = storage.getZooKeeper(); + + pruneLocalBackoff(model->get()); + + // Iterate sorted by create_time + for (const auto & entry : model->get()) + { + if (scheduled_exports_count >= available_move_executors) + { + LOG_DEBUG(storage.log, "ExportPartition scheduler task: Scheduled exports count is greater than available move executors, skipping"); + break; + } + + /// No need to query zk for status if the local one is not PENDING + if (entry.status != ExportReplicatedMergeTreePartitionTaskEntry::Status::PENDING) + { + LOG_DEBUG(storage.log, "ExportPartition scheduler task: Skipping... Local status is {}", magic_enum::enum_name(entry.status).data()); + continue; + } + + const auto & manifest = entry.manifest; + const auto key = entry.getCompositeKey(); + const auto database = storage.getContext()->resolveDatabase(manifest.destination_database); + const auto & table = manifest.destination_table; + + const auto destination_storage_id = StorageID(QualifiedTableName {database, table}); + + const auto destination_storage = DatabaseCatalog::instance().tryGetTable(destination_storage_id, storage.getContext()); + + if (!destination_storage) + { + LOG_WARNING(storage.log, "ExportPartition scheduler task: Failed to reconstruct destination storage: {}, skipping", destination_storage_id.getNameForLogs()); + continue; + } + + ProfileEvents::increment(ProfileEvents::ExportPartitionZooKeeperRequests); + ProfileEvents::increment(ProfileEvents::ExportPartitionZooKeeperGet); + std::string status_in_zk_string; + if (!zk->tryGet(fs::path(storage.zookeeper_path) / "exports" / key / "status", status_in_zk_string)) + { + LOG_WARNING(storage.log, "ExportPartition scheduler task: Failed to get status, skipping"); + continue; + } + + const auto status_in_zk = magic_enum::enum_cast(status_in_zk_string); + + if (!status_in_zk) + { + LOG_WARNING(storage.log, "ExportPartition scheduler task: Failed to get status from zk, skipping"); + continue; + } + + if (status_in_zk.value() != ExportReplicatedMergeTreePartitionTaskEntry::Status::PENDING) + { + LOG_DEBUG(storage.log, "ExportPartition scheduler task: Skipping {}... Status from zk is {}", key, magic_enum::enum_name(status_in_zk.value()).data()); + continue; + } + + ProfileEvents::increment(ProfileEvents::ExportPartitionZooKeeperRequests); + ProfileEvents::increment(ProfileEvents::ExportPartitionZooKeeperGetChildren); + std::vector parts_in_processing_or_pending; + + if (Coordination::Error::ZOK != zk->tryGetChildren(fs::path(storage.zookeeper_path) / "exports" / key / "processing", parts_in_processing_or_pending)) + { + LOG_WARNING(storage.log, "ExportPartition scheduler task: Failed to get parts in processing or pending, skipping"); + continue; + } + + + if (parts_in_processing_or_pending.empty()) + { + LOG_DEBUG(storage.log, "ExportPartition scheduler task: No parts in processing or pending, skipping"); + continue; + } + + /// shuffle the parts to reduce the risk of lock collisions + std::shuffle(parts_in_processing_or_pending.begin(), parts_in_processing_or_pending.end(), rng); + + ProfileEvents::increment(ProfileEvents::ExportPartitionZooKeeperRequests); + ProfileEvents::increment(ProfileEvents::ExportPartitionZooKeeperGetChildren); + std::vector locked_parts; + + if (Coordination::Error::ZOK != zk->tryGetChildren(fs::path(storage.zookeeper_path) / "exports" / key / "locks", locked_parts)) + { + LOG_WARNING(storage.log, "ExportPartition scheduler task: Failed to get locked parts, skipping"); + continue; + } + + std::unordered_set locked_parts_set(locked_parts.begin(), locked_parts.end()); + + const auto now = time(nullptr); + + for (const auto & zk_part_name : parts_in_processing_or_pending) + { + if (scheduled_exports_count >= available_move_executors) + { + LOG_DEBUG(storage.log, "ExportPartition scheduler task: Scheduled exports count is greater than available move executors, skipping"); + break; + } + + if (locked_parts_set.contains(zk_part_name)) + { + LOG_DEBUG(storage.log, "ExportPartition scheduler task: Part {} is locked, skipping", zk_part_name); + continue; + } + + if (shouldBackOff(entry.getTransactionId(), zk_part_name, now, earliest_backoff_retry)) + { + continue; + } + + const auto part = storage.getPartIfExists(zk_part_name, {MergeTreeDataPartState::Active, MergeTreeDataPartState::Outdated}); + if (!part) + { + LOG_DEBUG(storage.log, "ExportPartition scheduler task: Part {} not found locally, skipping", zk_part_name); + continue; + } + + LOG_INFO(storage.log, "ExportPartition scheduler task: Scheduling part export: {}", zk_part_name); + + auto context = ExportPartitionUtils::getContextCopyWithTaskSettings(storage.getContext(), manifest); + + try + { + LOG_DEBUG(storage.log, "ExportPartition scheduler task: Exporting part to table"); + + LOG_INFO(storage.log, "ExportPartition scheduler task: Attempting to lock part: {}", zk_part_name); + + ProfileEvents::increment(ProfileEvents::ExportPartitionZooKeeperRequests); + ProfileEvents::increment(ProfileEvents::ExportPartitionZooKeeperCreate); + if (Coordination::Error::ZOK != zk->tryCreate(fs::path(storage.zookeeper_path) / "exports" / key / "locks" / zk_part_name, storage.replica_name, zkutil::CreateMode::Ephemeral)) + { + LOG_INFO(storage.log, "ExportPartition scheduler task: Failed to lock part {}, skipping", zk_part_name); + continue; + } + + LOG_INFO(storage.log, "ExportPartition scheduler task: Locked part: {}", zk_part_name); + + storage.exportPartToTable( + part->name, + destination_storage_id, + manifest.transaction_id, + context, + manifest.iceberg_metadata_json, + /*allow_outdated_parts*/ true, + [this, key, zk_part_name, manifest, destination_storage] + (MergeTreePartExportManifest::CompletionCallbackResult result) + { + handlePartExportCompletion(key, zk_part_name, manifest, destination_storage, result); + }); + + scheduled_exports_count++; + } + catch (const Exception &) + { + tryLogCurrentException(__PRETTY_FUNCTION__); + ProfileEvents::increment(ProfileEvents::ExportPartitionZooKeeperRequests); + ProfileEvents::increment(ProfileEvents::ExportPartitionZooKeeperRemove); + zk->tryRemove(fs::path(storage.zookeeper_path) / "exports" / key / "locks" / zk_part_name); + /// Dispatch-time failure (e.g. Keeper node full). We do not arm the local + /// back-off here: the export never started, so the part stays immediately + /// eligible for this or another replica on the next tick. + } + } + } + + return earliest_backoff_retry; +} + +bool ExportPartitionTaskScheduler::shouldBackOff( + const std::string & transaction_id, + const std::string & part_name, + time_t now, + std::optional & earliest_backoff_retry) const +{ + std::lock_guard lock(local_backoff_mutex); + const auto task_it = local_backoff.find(transaction_id); + if (task_it == local_backoff.end()) + return false; + + const auto part_it = task_it->second.find(part_name); + if (part_it == task_it->second.end() || now >= part_it->second.next_retry_time) + return false; + + const auto next_retry_time = part_it->second.next_retry_time; + LOG_TRACE(storage.log, "ExportPartition scheduler task: Part {} is backing off locally, next retry at {} (now {}), skipping", part_name, next_retry_time, now); + earliest_backoff_retry = earliest_backoff_retry + ? std::min(*earliest_backoff_retry, next_retry_time) : next_retry_time; + return true; +} + +time_t ExportPartitionTaskScheduler::registerLocalBackoff( + const std::string & transaction_id, + const std::string & part_name, + const ExportReplicatedMergeTreePartitionManifest & manifest) +{ + std::lock_guard lock(local_backoff_mutex); + + /// First retryable failure for (transaction_id, part_name): create the map entries. + auto & parts = local_backoff.try_emplace(transaction_id).first->second; + auto & backoff = parts.try_emplace(part_name).first->second; + + ++backoff.attempts; + const auto backoff_seconds = computeRetryBackoffSeconds( + backoff.attempts, manifest.retry_initial_backoff_seconds, manifest.retry_max_backoff_seconds); + const auto now = time(nullptr); + /// Clamp so a huge configured back-off cannot overflow time_t (now is a normal wall-clock value). + const size_t headroom = static_cast(std::numeric_limits::max() - now); + backoff.next_retry_time = now + static_cast(std::min(backoff_seconds, headroom)); + return backoff.next_retry_time; +} + +void ExportPartitionTaskScheduler::clearLocalBackoff(const std::string & transaction_id, const std::string & part_name) +{ + std::lock_guard lock(local_backoff_mutex); + if (const auto task_it = local_backoff.find(transaction_id); task_it != local_backoff.end()) + { + task_it->second.erase(part_name); + if (task_it->second.empty()) + local_backoff.erase(task_it); + } +} + +void ExportPartitionTaskScheduler::pruneLocalBackoff(const ExportPartitionTaskEntriesContainer::index::type & model) +{ + std::lock_guard lock(local_backoff_mutex); + for (auto it = local_backoff.begin(); it != local_backoff.end();) + { + const auto found = model.find(it->first); + if (found != model.end() && found->status == ExportReplicatedMergeTreePartitionTaskEntry::Status::PENDING) + { + ++it; + continue; + } + + it = local_backoff.erase(it); + } +} + +ExportPartitionTaskScheduler::LocalBackoffMap ExportPartitionTaskScheduler::getLocalBackoffSnapshot() const +{ + LocalBackoffMap snapshot; + + std::lock_guard lock(local_backoff_mutex); + snapshot.reserve(local_backoff.size()); + for (const auto & [transaction_id, parts] : local_backoff) + { + auto & out_parts = snapshot[transaction_id]; + out_parts.reserve(parts.size()); + for (const auto & [part_name, backoff] : parts) + out_parts.emplace(part_name, LocalBackoff{backoff.attempts, backoff.next_retry_time}); + } + + return snapshot; +} + +void ExportPartitionTaskScheduler::handlePartExportCompletion( + const std::string & export_key, + const std::string & part_name, + const ExportReplicatedMergeTreePartitionManifest & manifest, + const StoragePtr & destination_storage, + const MergeTreePartExportManifest::CompletionCallbackResult & result) +{ + /// Invoked from MergeTreeBackgroundExecutor threads, so the component is not inherited from selectPartsToExport. + auto component_guard = Coordination::setCurrentComponent("ExportPartitionTaskScheduler::handlePartExportCompletion"); + + const auto export_path = fs::path(storage.zookeeper_path) / "exports" / export_key; + const auto processing_parts_path = export_path / "processing"; + const auto processed_part_path = export_path / "processed" / part_name; + const auto zk = storage.getZooKeeper(); + + if (result.success) + { + handlePartExportSuccess(manifest, destination_storage, processing_parts_path, processed_part_path, part_name, export_path, zk, result.relative_paths_in_destination_storage); + } + else + { + handlePartExportFailure(part_name, export_path, zk, result.exception, manifest); + } +} + +void ExportPartitionTaskScheduler::handlePartExportSuccess( + const ExportReplicatedMergeTreePartitionManifest & manifest, + const StoragePtr & destination_storage, + const std::filesystem::path & processing_parts_path, + const std::filesystem::path & processed_part_path, + const std::string & part_name, + const std::filesystem::path & export_path, + const zkutil::ZooKeeperPtr & zk, + const std::vector & relative_paths_in_destination_storage +) +{ + LOG_INFO(storage.log, "ExportPartition scheduler task: Part {} exported successfully, paths size: {}", part_name, relative_paths_in_destination_storage.size()); + + for (const auto & relative_path_in_destination_storage : relative_paths_in_destination_storage) + { + LOG_DEBUG(storage.log, "ExportPartition scheduler task: {}", relative_path_in_destination_storage); + } + + if (!tryToMovePartToProcessed(export_path, processing_parts_path, processed_part_path, part_name, relative_paths_in_destination_storage, zk)) + { + LOG_WARNING(storage.log, "ExportPartition scheduler task: Failed to move part to processed, will not commit export partition"); + return; + } + + /// Part is done on this replica; drop any local back-off state we held for it. + clearLocalBackoff(manifest.transaction_id, part_name); + + LOG_INFO(storage.log, "ExportPartition scheduler task: Marked part export {} as completed", part_name); + + if (!areAllPartsProcessed(export_path, zk)) + { + return; + } + + LOG_INFO(storage.log, "ExportPartition scheduler task: All parts are processed, will try to commit export partition"); + + try + { + auto context = ExportPartitionUtils::getContextCopyWithTaskSettings(storage.getContext(), manifest); + ExportPartitionUtils::commit(manifest, destination_storage, zk, storage.log.load(), export_path, context, storage, storage.replica_name); + } + catch (const Exception & e) + { + LOG_INFO(storage.log, "ExportPartition scheduler task: Caught exception while committing export partition, {}", e.message()); + + /// Classify the commit failure: a non-retryable error (e.g. schema/spec mismatch) + /// transitions the task to FAILED immediately; a retryable one (transient catalog or + /// destination outage) only records the exception and leaves the task PENDING so the + /// commit is retried until the absolute task timeout. + /// The exception is recorded in /last_exception via appendExceptionOps + /// inside the same multi as the (possible) FAILED set. + const bool became_failed = ExportPartitionUtils::handleCommitFailure( + zk, + export_path, + e.code(), + storage.replica_name, + e.message(), + storage.log.load()); + + if (became_failed) + { + LOG_WARNING(storage.log, + "ExportPartition scheduler task: Commit for {} transitioned to FAILED due to non-retryable error (code {})", + export_path.string(), e.code()); + } + } +} + +void ExportPartitionTaskScheduler::handlePartExportFailure( + const std::string & part_name, + const std::filesystem::path & export_path, + const zkutil::ZooKeeperPtr & zk, + const std::optional & exception, + const ExportReplicatedMergeTreePartitionManifest & manifest +) +{ + LOG_INFO(storage.log, "ExportPartition scheduler task: Part {} export failed", part_name); + + if (!exception) + { + throw Exception(ErrorCodes::LOGICAL_ERROR, "ExportPartition scheduler task: No exception provided for error handling. Sounds like a bug"); + } + + Coordination::Stat locked_by_stat; + std::string locked_by; + + ProfileEvents::increment(ProfileEvents::ExportPartitionZooKeeperRequests); + ProfileEvents::increment(ProfileEvents::ExportPartitionZooKeeperGet); + if (!zk->tryGet(export_path / "locks" / part_name, locked_by, &locked_by_stat)) + { + LOG_DEBUG(storage.log, "ExportPartition scheduler task: Part {} is not locked by any replica, will not increment error counts", part_name); + return; + } + + if (locked_by != storage.replica_name) + { + LOG_DEBUG(storage.log, "ExportPartition scheduler task: Part {} is locked by another replica, will not increment error counts", part_name); + return; + } + + /// Early exit if the query was cancelled - no need to increment error counts + if (exception->code() == ErrorCodes::QUERY_WAS_CANCELLED) + { + /// Releasing the lock is important because a query can be cancelled due to SYSTEM STOP MOVES. If this is the case, + /// other replicas should still be able to export this individual part. That's why there is a retry loop here. + /// It is very unlikely this will be a problem in practice. The lock is ephemeral, which means it is automatically released + /// if ClickHouse loses connection to ZooKeeper + std::size_t retry_count = 0; + static constexpr std::size_t max_lock_release_retries = 3; + while (retry_count < max_lock_release_retries) + { + ProfileEvents::increment(ProfileEvents::ExportPartitionZooKeeperRequests); + ProfileEvents::increment(ProfileEvents::ExportPartitionZooKeeperRemove); + + const auto removal_code = zk->tryRemove(export_path / "locks" / part_name, locked_by_stat.version); + + if (Coordination::Error::ZOK == removal_code) + { + break; + } + + if (Coordination::Error::ZBADVERSION == removal_code) + { + LOG_DEBUG(storage.log, "ExportPartition scheduler task: Part {} lock version mismatch, will not increment error counts", part_name); + break; + } + + retry_count++; + } + + LOG_INFO(storage.log, "ExportPartition scheduler task: Part {} export was cancelled, skipping error handling", part_name); + return; + } + + const std::string status_path = export_path / "status"; + Coordination::Stat status_stat; + std::string current_status; + + ProfileEvents::increment(ProfileEvents::ExportPartitionZooKeeperRequests); + ProfileEvents::increment(ProfileEvents::ExportPartitionZooKeeperGet); + if (!zk->tryGet(status_path, current_status, &status_stat)) + { + LOG_DEBUG(storage.log, "ExportPartition scheduler task: /status missing for {}, skipping failure bookkeeping", export_path.string()); + return; + } + + const auto status = magic_enum::enum_cast(current_status); + if (!status || *status != ExportReplicatedMergeTreePartitionTaskEntry::Status::PENDING) + { + LOG_DEBUG(storage.log, "ExportPartition scheduler task: /status for {} is {} (not PENDING), skipping failure bookkeeping", export_path.string(), current_status); + return; + } + + const bool non_retryable = ExportPartitionUtils::isNonRetryableExportError(exception->code()); + + Coordination::Requests ops; + + ops.emplace_back(zkutil::makeRemoveRequest(export_path / "locks" / part_name, locked_by_stat.version)); + + if (non_retryable) + { + /// Deterministic failure (e.g. schema/type incompatibility): retrying cannot help, + /// so fail the whole task immediately instead of waiting for the absolute timeout. + ops.emplace_back(zkutil::makeSetRequest( + status_path, + String(magic_enum::enum_name(ExportReplicatedMergeTreePartitionTaskEntry::Status::FAILED)).data(), + status_stat.version)); + LOG_WARNING(storage.log, "ExportPartition scheduler task: Part {} failed with non-retryable error (code {}), failing the entire task", part_name, exception->code()); + } + else + { + LOG_DEBUG(storage.log, "ExportPartition scheduler task: Part {} failed with retryable error (code {}), will back off and retry until the task timeout", part_name, exception->code()); + } + + ExportPartitionUtils::appendExceptionOps( + ops, zk, export_path, storage.replica_name, part_name, + exception->message(), storage.log.load()); + + ProfileEvents::increment(ProfileEvents::ExportPartitionZooKeeperRequests); + ProfileEvents::increment(ProfileEvents::ExportPartitionZooKeeperMulti); + Coordination::Responses responses; + if (Coordination::Error::ZOK != zk->tryMulti(ops, responses)) + { + LOG_WARNING(storage.log, "ExportPartition scheduler task: All failure mechanism failed, will not try to update it"); + return; + } + + /// Only after the lock release + exception record committed do we arm the local back-off, + /// so a Keeper failure above does not leave this replica skipping the part for no reason. + if (!non_retryable) + { + const auto next_retry_time = registerLocalBackoff(manifest.transaction_id, part_name, manifest); + LOG_INFO(storage.log, "ExportPartition scheduler task: Part {} backing off locally, next retry at {}", part_name, next_retry_time); + } + + LOG_INFO(storage.log, "ExportPartition scheduler task: Successfully recorded failure for part {}", part_name); +} + +bool ExportPartitionTaskScheduler::tryToMovePartToProcessed( + const std::filesystem::path & export_path, + const std::filesystem::path & processing_parts_path, + const std::filesystem::path & processed_part_path, + const std::string & part_name, + const std::vector & relative_paths_in_destination_storage, + const zkutil::ZooKeeperPtr & zk +) +{ + Coordination::Stat locked_by_stat; + std::string locked_by; + + ProfileEvents::increment(ProfileEvents::ExportPartitionZooKeeperRequests); + ProfileEvents::increment(ProfileEvents::ExportPartitionZooKeeperGet); + if (!zk->tryGet(export_path / "locks" / part_name, locked_by, &locked_by_stat)) + { + LOG_DEBUG(storage.log, "ExportPartition scheduler task: Part {} is not locked by any replica, will not commit or set it as completed", part_name); + return false; + } + + /// Is this a good idea? what if the file we just pushed to s3 ends up triggering an exception in the replica that actually locks the part and it does not commit? + /// I guess we should not throw if file already exists for export partition, hard coded. + if (locked_by != storage.replica_name) + { + LOG_DEBUG(storage.log, "ExportPartition scheduler task: Part {} is locked by another replica, will not commit or set it as completed", part_name); + return false; + } + + Coordination::Requests requests; + + ExportReplicatedMergeTreePartitionProcessedPartEntry processed_part_entry; + processed_part_entry.part_name = part_name; + processed_part_entry.paths_in_destination = relative_paths_in_destination_storage; + processed_part_entry.finished_by = storage.replica_name; + + requests.emplace_back(zkutil::makeRemoveRequest(processing_parts_path / part_name, -1)); + requests.emplace_back(zkutil::makeCreateRequest(processed_part_path, processed_part_entry.toJsonString(), zkutil::CreateMode::Persistent)); + requests.emplace_back(zkutil::makeRemoveRequest(export_path / "locks" / part_name, locked_by_stat.version)); + + ProfileEvents::increment(ProfileEvents::ExportPartitionZooKeeperRequests); + ProfileEvents::increment(ProfileEvents::ExportPartitionZooKeeperMulti); + Coordination::Responses responses; + if (Coordination::Error::ZOK != zk->tryMulti(requests, responses)) + { + + /// todo arthur remember what to do here + LOG_WARNING(storage.log, "ExportPartition scheduler task: Failed to update export path, skipping"); + return false; + } + + return true; +} + +bool ExportPartitionTaskScheduler::areAllPartsProcessed( + const std::filesystem::path & export_path, + const zkutil::ZooKeeperPtr & zk) +{ + ProfileEvents::increment(ProfileEvents::ExportPartitionZooKeeperRequests); + ProfileEvents::increment(ProfileEvents::ExportPartitionZooKeeperGetChildren); + Strings parts_in_processing_or_pending; + if (Coordination::Error::ZOK != zk->tryGetChildren(export_path / "processing", parts_in_processing_or_pending)) + { + LOG_WARNING(storage.log, "ExportPartition scheduler task: Failed to get parts in processing or pending, will not try to commit export partition"); + return false; + } + + if (!parts_in_processing_or_pending.empty()) + { + LOG_DEBUG(storage.log, "ExportPartition scheduler task: There are still parts in processing or pending, will not try to commit export partition"); + return false; + } + + return true; +} + +} diff --git a/src/Storages/MergeTree/ExportPartitionTaskScheduler.h b/src/Storages/MergeTree/ExportPartitionTaskScheduler.h new file mode 100644 index 000000000000..038febefd831 --- /dev/null +++ b/src/Storages/MergeTree/ExportPartitionTaskScheduler.h @@ -0,0 +1,118 @@ +#pragma once + +#include +#include +#include +#include +#include +#include +#include +#include +#include + +namespace DB +{ + +class Exception; +class StorageReplicatedMergeTree; + +struct ExportReplicatedMergeTreePartitionManifest; + +/// todo arthur remember to add check(lock, version) when updating stuff because maybe if we believe we have the lock, we might not actually have it +class ExportPartitionTaskScheduler +{ +public: + ExportPartitionTaskScheduler(StorageReplicatedMergeTree & storage); + + /// Returns the earliest future back-off deadline (unix seconds) among parts that were skipped + /// this tick purely because they are still backing off, or nullopt if none. The caller can use + /// it to wake the select task sooner than the default tick interval. + std::optional run(); +private: + StorageReplicatedMergeTree & storage; + + /// todo arthur maybe it is invalid to grab the manifst here + void handlePartExportCompletion( + const std::string & export_key, + const std::string & part_name, + const ExportReplicatedMergeTreePartitionManifest & manifest, + const StoragePtr & destination_storage, + const MergeTreePartExportManifest::CompletionCallbackResult & result); + + void handlePartExportSuccess( + const ExportReplicatedMergeTreePartitionManifest & manifest, + const StoragePtr & destination_storage, + const std::filesystem::path & processing_parts_path, + const std::filesystem::path & processed_part_path, + const std::string & part_name, + const std::filesystem::path & export_path, + const zkutil::ZooKeeperPtr & zk, + const std::vector & relative_paths_in_destination_storage + ); + + void handlePartExportFailure( + const std::string & part_name, + const std::filesystem::path & export_path, + const zkutil::ZooKeeperPtr & zk, + const std::optional & exception, + const ExportReplicatedMergeTreePartitionManifest & manifest); + + bool tryToMovePartToProcessed( + const std::filesystem::path & export_path, + const std::filesystem::path & processing_parts_path, + const std::filesystem::path & processed_part_path, + const std::string & part_name, + const std::vector & relative_paths_in_destination_storage, + const zkutil::ZooKeeperPtr & zk + ); + + bool areAllPartsProcessed( + const std::filesystem::path & export_path, + const zkutil::ZooKeeperPtr & zk + ); + + struct LocalBackoff + { + size_t attempts = 0; + time_t next_retry_time = 0; + }; + + /// transaction_id -> part name -> back-off state. Keyed by transaction_id (not composite + /// key) so a reused composite key does not inherit a prior instance's back-off. Guarded by + /// local_backoff_mutex because run() (schedule-pool thread) reads it while part-export + /// completion callbacks (background-executor threads) write it. + using PartNameToBackOffMap = std::unordered_map; + using TransactionID = std::string; + using LocalBackoffMap = std::unordered_map; + + mutable std::mutex local_backoff_mutex; + LocalBackoffMap local_backoff TSA_GUARDED_BY(local_backoff_mutex); + + bool shouldBackOff( + const std::string & transaction_id, + const std::string & part_name, + time_t now, + std::optional & earliest_backoff_retry) const; + + /// Record a retryable failure for (transaction_id, part_name): grow the attempt counter and + /// compute the next eligible time. Returns the new absolute deadline. + time_t registerLocalBackoff( + const std::string & transaction_id, + const std::string & part_name, + const ExportReplicatedMergeTreePartitionManifest & manifest); + + /// Drop any back-off state for parts of (transaction_id) once they succeed or the task ends. + void clearLocalBackoff(const std::string & transaction_id, const std::string & part_name); + + /// Remove back-off state for tasks whose transaction_id is no longer PENDING in the published + /// model, bounding the map to the parts of currently-active tasks. + void pruneLocalBackoff(const ExportPartitionTaskEntriesContainer::index::type & model); + +public: + /// Snapshot of the local back-off map for system.replicated_partition_exports: + /// transaction_id -> part -> (attempts, next_retry_time). Briefly locks local_backoff_mutex; + /// never held across ZooKeeper I/O. + std::unordered_map getLocalBackoffSnapshot() const; +}; + +} diff --git a/src/Storages/MergeTree/ExportPartitionUtils.cpp b/src/Storages/MergeTree/ExportPartitionUtils.cpp new file mode 100644 index 000000000000..b97b9af4d665 --- /dev/null +++ b/src/Storages/MergeTree/ExportPartitionUtils.cpp @@ -0,0 +1,1037 @@ +#include +#include +#include +#include +#include +#include +#include "Storages/ExportReplicatedMergeTreePartitionManifest.h" +#include "Storages/ExportReplicatedMergeTreePartitionTaskEntry.h" +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#if USE_AVRO +#include +#include +#endif + +namespace ProfileEvents +{ + extern const Event ExportPartitionZooKeeperRequests; + extern const Event ExportPartitionZooKeeperGet; + extern const Event ExportPartitionZooKeeperGetChildren; + extern const Event ExportPartitionZooKeeperSet; + extern const Event ExportPartitionZooKeeperCreate; + extern const Event ExportPartitionZooKeeperMulti; +} + +namespace DB +{ + +namespace ErrorCodes +{ + extern const int FAULT_INJECTED; + extern const int BAD_ARGUMENTS; + extern const int NO_SUCH_DATA_PART; + extern const int CORRUPTED_DATA; + extern const int NETWORK_ERROR; + extern const int LOGICAL_ERROR; + extern const int NOT_IMPLEMENTED; + extern const int SUPPORT_IS_DISABLED; + extern const int TYPE_MISMATCH; + extern const int CANNOT_CONVERT_TYPE; + extern const int ILLEGAL_TYPE_OF_ARGUMENT; + extern const int ILLEGAL_COLUMN; + extern const int NUMBER_OF_COLUMNS_DOESNT_MATCH; + extern const int INCOMPATIBLE_COLUMNS; + extern const int NO_SUCH_COLUMN_IN_TABLE; + extern const int FILE_ALREADY_EXISTS; + extern const int METADATA_MISMATCH; + extern const int CANNOT_PARSE_TEXT; + extern const int CANNOT_PARSE_NUMBER; + extern const int CANNOT_PARSE_DATE; + extern const int CANNOT_PARSE_DATETIME; + extern const int CANNOT_PARSE_BOOL; + extern const int CANNOT_PARSE_UUID; + extern const int CANNOT_PARSE_IPV4; + extern const int CANNOT_PARSE_IPV6; + extern const int CANNOT_PARSE_QUOTED_STRING; + extern const int CANNOT_PARSE_ESCAPE_SEQUENCE; + extern const int CANNOT_PARSE_INPUT_ASSERTION_FAILED; + extern const int CANNOT_PARSE_DOMAIN_VALUE_FROM_STRING; + extern const int VALUE_IS_OUT_OF_RANGE_OF_DATA_TYPE; + extern const int ATTEMPT_TO_READ_AFTER_EOF; + extern const int CANNOT_READ_ARRAY_FROM_TEXT; + extern const int DECIMAL_OVERFLOW; +} + +namespace Setting +{ + extern const SettingsBool export_merge_tree_part_allow_lossy_cast; +#if USE_AVRO + extern const SettingsTimezone iceberg_partition_timezone; +#endif + extern const SettingsMergeTreePartExportSchemaMismatchMode export_merge_tree_part_schema_mismatch_mode; +} + +namespace FailPoints +{ + extern const char iceberg_export_after_commit_before_zk_completed[]; + extern const char export_partition_commit_always_throw[]; +} + +namespace fs = std::filesystem; + +namespace ExportPartitionUtils +{ + bool isNonRetryableExportError(int code) + { + /// Deterministic failures where retrying cannot possibly succeed (schema/type + /// incompatibilities, unsupported features, programming errors). Everything else + /// (memory limits, network/object-storage/Keeper transient errors, ...) is retryable. + /// `QUERY_WAS_CANCELLED` is handled separately by the caller and never reaches here. + /// + /// ErrorCodes values are runtime `extern const int`, not constant expressions, so they + /// cannot be used as `switch` labels; compare against a static set instead. + static const std::unordered_set non_retryable_codes = { + ErrorCodes::BAD_ARGUMENTS, + ErrorCodes::TYPE_MISMATCH, + ErrorCodes::CANNOT_CONVERT_TYPE, + ErrorCodes::ILLEGAL_TYPE_OF_ARGUMENT, + ErrorCodes::ILLEGAL_COLUMN, + ErrorCodes::NUMBER_OF_COLUMNS_DOESNT_MATCH, + ErrorCodes::INCOMPATIBLE_COLUMNS, + ErrorCodes::NO_SUCH_COLUMN_IN_TABLE, + ErrorCodes::NOT_IMPLEMENTED, + ErrorCodes::SUPPORT_IS_DISABLED, + ErrorCodes::LOGICAL_ERROR, + ErrorCodes::FILE_ALREADY_EXISTS, + ErrorCodes::METADATA_MISMATCH, + ErrorCodes::CANNOT_PARSE_TEXT, + ErrorCodes::CANNOT_PARSE_NUMBER, + ErrorCodes::CANNOT_PARSE_DATE, + ErrorCodes::CANNOT_PARSE_DATETIME, + ErrorCodes::CANNOT_PARSE_BOOL, + ErrorCodes::CANNOT_PARSE_UUID, + ErrorCodes::CANNOT_PARSE_IPV4, + ErrorCodes::CANNOT_PARSE_IPV6, + ErrorCodes::CANNOT_PARSE_QUOTED_STRING, + ErrorCodes::CANNOT_PARSE_ESCAPE_SEQUENCE, + ErrorCodes::CANNOT_PARSE_INPUT_ASSERTION_FAILED, + ErrorCodes::CANNOT_PARSE_DOMAIN_VALUE_FROM_STRING, + ErrorCodes::VALUE_IS_OUT_OF_RANGE_OF_DATA_TYPE, + ErrorCodes::ATTEMPT_TO_READ_AFTER_EOF, + ErrorCodes::CANNOT_READ_ARRAY_FROM_TEXT, + ErrorCodes::DECIMAL_OVERFLOW, + }; + return non_retryable_codes.contains(code); + } + + Block getPartitionSourceBlockForIcebergCommit( + MergeTreeData & storage, const String & partition_id, const std::vector & exported_part_names) + { + auto lock = storage.readLockParts(); + const auto parts = storage.getDataPartsVectorInPartitionForInternalUsage( + {MergeTreeDataPartState::Active, MergeTreeDataPartState::Outdated}, partition_id, lock); + + /// Only look at the parts being exported. These parts are guaranteed to map to a single partition. + /// Parts that were later inserted shall be ignored + const std::unordered_set exported(exported_part_names.begin(), exported_part_names.end()); + IMergeTreeDataPart::MinMaxIndex minmax; + for (const auto & part : parts) + if (exported.contains(part->name)) + minmax.merge(*part->getMinMaxIndex()); + + if (!minmax.initialized) + throw Exception(ErrorCodes::NO_SUCH_DATA_PART, + "Cannot find any of the exported parts for partition_id '{}' to derive Iceberg partition " + "values. They may have been merged and cleaned up before this commit, or are not present " + "on this replica. The commit will be retried.", + partition_id); + + const auto metadata_snapshot = storage.getInMemoryMetadataPtr(storage.getContext(), false); + const auto & partition_key = metadata_snapshot->getPartitionKey(); + const auto minmax_columns = MergeTreeData::getMinMaxColumns( + partition_key, storage.getSettings(), MergeTreePartMinMaxIndexColumns::PARTITION_KEY_ONLY); + + if (minmax.hyperrectangle.size() < minmax_columns.size()) + throw Exception(ErrorCodes::LOGICAL_ERROR, + "Cannot derive Iceberg partition values: the exported parts of partition '{}' hold min/max " + "statistics for {} columns, but the partition key has {}.", + partition_id, minmax.hyperrectangle.size(), minmax_columns.size()); + + /// When the query was scheduled, we validated that dst_expression(min) == dst_expression(max). + /// Therefore, we can use only the min value, no need for the max. + Block block; + size_t i = 0; + for (const auto & [column_name, column_type] : minmax_columns) + { + auto column = column_type->createColumn(); + column->insert(minmax.hyperrectangle[i].left); + block.insert(ColumnWithTypeAndName(column->getPtr(), column_type, column_name)); + ++i; + } + + return block; + } + + ContextPtr getContextCopyWithTaskSettings(const ContextPtr & context, const ExportReplicatedMergeTreePartitionManifest & manifest) + { + auto context_copy = Context::createCopy(context); + context_copy->makeQueryContextForExportPart(); + context_copy->setCurrentQueryId(manifest.query_id); + context_copy->setSetting("output_format_parallel_formatting", manifest.parallel_formatting); + context_copy->setSetting("output_format_parquet_parallel_encoding", manifest.parquet_parallel_encoding); + + /// Backwards compatibility + if (manifest.parquet_compression_method) + context_copy->setSetting("output_format_parquet_compression_method", *manifest.parquet_compression_method); + if (manifest.output_format_compression_level) + context_copy->setSetting("output_format_compression_level", *manifest.output_format_compression_level); + if (manifest.parquet_row_group_size) + context_copy->setSetting("output_format_parquet_row_group_size", *manifest.parquet_row_group_size); + if (manifest.parquet_row_group_size_bytes) + context_copy->setSetting("output_format_parquet_row_group_size_bytes", *manifest.parquet_row_group_size_bytes); + /// Manifests written before this setting existed have no value here; such tasks were always + /// scheduled under the old, strict column-count check, so an absent value must resolve to + /// `strict` regardless of the ambient context's setting (which may have since been changed). + context_copy->setSetting( + "export_merge_tree_part_schema_mismatch_mode", + String(magic_enum::enum_name(manifest.schema_mismatch_mode.value_or(MergeTreePartExportSchemaMismatchMode::strict)))); + + context_copy->setSetting("max_threads", manifest.max_threads); + context_copy->setSetting("export_merge_tree_part_file_already_exists_policy", String(magic_enum::enum_name(manifest.file_already_exists_policy))); + context_copy->setSetting("export_merge_tree_part_max_bytes_per_file", manifest.max_bytes_per_file); + context_copy->setSetting("export_merge_tree_part_max_rows_per_file", manifest.max_rows_per_file); + context_copy->setSetting("iceberg_insert_max_bytes_in_data_file", manifest.max_bytes_per_file); + context_copy->setSetting("iceberg_insert_max_rows_in_data_file", manifest.max_rows_per_file); + + /// always skip pending mutations and patch parts because we already validated the parts during query processing + context_copy->setSetting("export_merge_tree_part_throw_on_pending_mutations", false); + context_copy->setSetting("export_merge_tree_part_throw_on_pending_patch_parts", false); + + context_copy->setSetting("export_merge_tree_part_filename_pattern", manifest.filename_pattern); + context_copy->setSetting("write_full_path_in_iceberg_metadata", manifest.write_full_path_in_iceberg_metadata); + + /// The request-time call to exportPartitionToTable has already validated allow_insert_into_iceberg + /// against the initiator's settings. Once the manifest is in ZooKeeper, every replica must be + /// able to execute the task regardless of its own profile - otherwise an export silently + /// stalls when the setting is only set at the query level. + context_copy->setSetting("allow_insert_into_iceberg", true); + + /// Reapply the initiator's lossy-cast decision (persisted in the manifest) so the + /// worker's schema revalidation honors the user's choice. Without this, a task + /// scheduled without the opt-in could still apply a lossy cast if the destination + /// schema drifts to a lossy target between scheduling and execution. + context_copy->setSetting("export_merge_tree_part_allow_lossy_cast", manifest.allow_lossy_cast); + + if (manifest.iceberg_partition_timezone) + { + context_copy->setSetting("iceberg_partition_timezone", *manifest.iceberg_partition_timezone); + } + + return context_copy; + } + + /// Collect all the exported paths from the processed parts + /// If multiRead is supported by the keeper implementation, it is done in a single request + /// Otherwise, multiple async requests are sent + std::vector getExportedPaths(const LoggerPtr & log, const zkutil::ZooKeeperPtr & zk, const std::string & export_path) + { + std::vector exported_paths; + + LOG_DEBUG(log, "ExportPartition: Getting exported paths for {}", export_path); + + const auto processed_parts_path = fs::path(export_path) / "processed"; + + ProfileEvents::increment(ProfileEvents::ExportPartitionZooKeeperRequests); + ProfileEvents::increment(ProfileEvents::ExportPartitionZooKeeperGetChildren); + std::vector processed_parts; + if (Coordination::Error::ZOK != zk->tryGetChildren(processed_parts_path, processed_parts)) + { + /// todo arthur do something here + LOG_WARNING(log, "ExportPartition: Failed to get parts children, exiting"); + return {}; + } + + std::vector get_paths; + + for (const auto & processed_part : processed_parts) + { + get_paths.emplace_back(processed_parts_path / processed_part); + } + + auto responses = zk->tryGet(get_paths); + ProfileEvents::increment(ProfileEvents::ExportPartitionZooKeeperRequests); + ProfileEvents::increment(ProfileEvents::ExportPartitionZooKeeperGet, get_paths.size()); + + responses.waitForResponses(); + + for (size_t i = 0; i < responses.size(); ++i) + { + if (responses[i].error != Coordination::Error::ZOK) + { + /// todo arthur what to do in this case? + /// It could be that zk is corrupt, in that case we should fail the task + /// but it can also be some temporary network issue? not sure + LOG_WARNING(log, "ExportPartition: Failed to get exported path, exiting"); + return {}; + } + + const auto processed_part_entry = ExportReplicatedMergeTreePartitionProcessedPartEntry::fromJsonString(responses[i].data); + + for (const auto & path_in_destination : processed_part_entry.paths_in_destination) + { + exported_paths.emplace_back(path_in_destination); + } + } + + return exported_paths; + } + + void commit( + const ExportReplicatedMergeTreePartitionManifest & manifest, + const StoragePtr & destination_storage, + const zkutil::ZooKeeperPtr & zk, + const LoggerPtr & log, + const std::string & entry_path, + const ContextPtr & context_in, + MergeTreeData & source_storage, + const String & replica_name) + { + auto context = Context::createCopy(context_in); + context->setSetting("write_full_path_in_iceberg_metadata", manifest.write_full_path_in_iceberg_metadata); + + if (manifest.iceberg_partition_timezone) + context->setSetting("iceberg_partition_timezone", *manifest.iceberg_partition_timezone); + + /// Failpoint used by integration tests to force persistent commit failure and exercise + /// the commit-attempts budget / FAILED state transition. + fiu_do_on(FailPoints::export_partition_commit_always_throw, + { + throw Exception(ErrorCodes::FAULT_INJECTED, + "Failpoint: export_partition_commit_always_throw"); + }); + + /// Per-task ephemeral lock that serializes the commit phase across replicas. + /// Without it, `handlePartExportSuccess` (post-last-part path) and `tryCleanup` + /// (poll/recovery path) can drive `commitExportPartitionTransaction` concurrently + /// for the same task. + const auto commit_lock_path = fs::path(entry_path) / "commit_lock"; + auto commit_lock = zkutil::EphemeralNodeHolder::tryCreate(commit_lock_path, *zk, replica_name); + if (!commit_lock) + { + LOG_DEBUG(log, "ExportPartition: commit_lock for {} is held by another replica, skipping commit on this replica", entry_path); + return; + } + LOG_INFO(log, "ExportPartition: commit_lock for {} acquired by replica {}", entry_path, replica_name); + + /// Honor a concurrent KILL: commit_lock serializes us against killExportPartition, + /// so a non-PENDING status here means cancel won the race. + std::string status_str; + if (!zk->tryGet(fs::path(entry_path) / "status", status_str)) + return; + const auto status = magic_enum::enum_cast(status_str); + if (!status || *status != ExportReplicatedMergeTreePartitionTaskEntry::Status::PENDING) + { + LOG_DEBUG(log, "ExportPartition: {} not PENDING, skipping commit", entry_path); + return; + } + + const auto exported_paths = ExportPartitionUtils::getExportedPaths(log, zk, entry_path); + + if (exported_paths.empty()) + { + throw Exception(ErrorCodes::CORRUPTED_DATA, "ExportPartition: No exported paths found, will not commit export. This might be a bug"); + } + + //// not checking for an exact match because a single part might generate multiple files + if (exported_paths.size() < manifest.parts.size()) + { + throw Exception(ErrorCodes::CORRUPTED_DATA, "ExportPartition: Reached the commit phase, but exported paths size is less than the number of parts, will not commit export. This might be a bug"); + } + + IStorage::IcebergCommitExportPartitionArguments iceberg_args; + + if (!manifest.iceberg_metadata_json.empty()) + { + iceberg_args.metadata_json_string = manifest.iceberg_metadata_json; + const auto source_metadata = source_storage.getInMemoryMetadataPtr(context, false); + if (source_metadata->hasPartitionKey()) + iceberg_args.partition_source_block = + getPartitionSourceBlockForIcebergCommit(source_storage, manifest.partition_id, manifest.parts); + } + + const auto destination_commit_info = destination_storage->commitExportPartitionTransaction( + manifest.transaction_id, manifest.partition_id, exported_paths, iceberg_args, context); + + /// Failpoint to simulate a crash after the Iceberg commit succeeds but before + /// ZooKeeper is updated to COMPLETED. Used by idempotency integration tests. + fiu_do_on(FailPoints::iceberg_export_after_commit_before_zk_completed, + { + LOG_INFO(log, "Failpoint: simulating crash after Iceberg commit, before ZK COMPLETED"); + std::this_thread::sleep_for(std::chrono::seconds(10)); + throw Exception(ErrorCodes::FAULT_INJECTED, + "Failpoint: simulating crash after Iceberg commit, before ZK COMPLETED"); + }); + + LOG_INFO(log, "ExportPartition: Committed export, mark as completed"); + + const std::string status_path = fs::path(entry_path) / "status"; + const std::string completed_name = String(magic_enum::enum_name(ExportReplicatedMergeTreePartitionTaskEntry::Status::COMPLETED)).data(); + + Coordination::Requests ops; + ops.emplace_back(zkutil::makeSetRequest(status_path, completed_name, -1)); + + ExportReplicatedMergeTreePartitionCommitInfoEntry commit_info_entry { + destination_commit_info.iceberg_metadata_file, + destination_commit_info.iceberg_manifest_list, + destination_commit_info.iceberg_manifest_file, + destination_commit_info.commit_marker_file}; + + const std::string commit_info_path = fs::path(entry_path) / "commit_info"; + ops.emplace_back(zkutil::makeCreateRequest(commit_info_path, commit_info_entry.toJsonString(), zkutil::CreateMode::Persistent)); + + ProfileEvents::increment(ProfileEvents::ExportPartitionZooKeeperRequests); + ProfileEvents::increment(ProfileEvents::ExportPartitionZooKeeperMulti); + + Coordination::Responses responses; + const auto rc = zk->tryMulti(ops, responses); + + if (rc == Coordination::Error::ZOK) + { + LOG_INFO(log, "ExportPartition: Marked export as completed and persisted commit_info"); + return; + } + + if (rc == Coordination::Error::ZNODEEXISTS) + { + LOG_INFO(log, "ExportPartition: commit_info already present (peer wrote it first); task already COMPLETED"); + return; + } + + throw Exception(ErrorCodes::NETWORK_ERROR, "ExportPartition: Failed to mark export as completed (rc={}), will not try to fix it", rc); + } + + bool handleCommitFailure( + const zkutil::ZooKeeperPtr & zk, + const std::string & entry_path, + int exception_code, + const std::string & replica_name, + const std::string & exception_message, + const LoggerPtr & log) + { + const std::string status_path = fs::path(entry_path) / "status"; + + /// Read /status together with its stat so we can (a) bail early if another + /// replica has already moved the task out of PENDING and (b) use a + /// version-checked Set later to avoid clobbering a concurrent write + /// (e.g. a racing successful commit that marked the task COMPLETED between + /// our read and our tryMulti). + Coordination::Stat status_stat; + std::string current_status; + + ProfileEvents::increment(ProfileEvents::ExportPartitionZooKeeperRequests); + ProfileEvents::increment(ProfileEvents::ExportPartitionZooKeeperGet); + if (!zk->tryGet(status_path, current_status, &status_stat)) + { + /// Task was removed (TTL cleanup or force-overwrite). Nothing to do. + LOG_DEBUG(log, "ExportPartition: /status missing for {}, skipping commit-failure bookkeeping", entry_path); + return false; + } + + const auto status = magic_enum::enum_cast(current_status); + if (!status) + { + LOG_WARNING(log, "ExportPartition: Invalid status {} for task {}, skipping commit-failure bookkeeping", current_status, entry_path); + return false; + } + + if (status != ExportReplicatedMergeTreePartitionTaskEntry::Status::PENDING) + { + /// Another replica already reached a terminal state (COMPLETED or FAILED). + /// Do NOT overwrite — a successful commit by a peer must win. + LOG_DEBUG(log, + "ExportPartition: /status for {} is {} (not PENDING), skipping commit-failure bookkeeping", + entry_path, current_status); + return false; + } + + Coordination::Requests ops; + + /// Record the exception in the same multi as the (possible) FAILED transition, so the + /// user-visible last_exception znode is updated atomically with the state change that + /// exposes it. + appendExceptionOps(ops, zk, fs::path(entry_path), replica_name, /*part_name=*/"", exception_message, log); + + /// A non-retryable error (schema/spec mismatch, ...) can never succeed, + /// so fail the task immediately + const bool non_retryable = isNonRetryableExportError(exception_code); + if (non_retryable) + { + /// Version-checked Set: if /status has changed since we read it (e.g. a peer's + /// commit() succeeded and wrote COMPLETED), the whole multi aborts with + /// ZBADVERSION and we safely do nothing — the winning terminal state stands. + ops.emplace_back(zkutil::makeSetRequest( + status_path, + String(magic_enum::enum_name(ExportReplicatedMergeTreePartitionTaskEntry::Status::FAILED)).data(), + status_stat.version)); + } + + ProfileEvents::increment(ProfileEvents::ExportPartitionZooKeeperRequests); + ProfileEvents::increment(ProfileEvents::ExportPartitionZooKeeperMulti); + Coordination::Responses responses; + const auto rc = zk->tryMulti(ops, responses); + if (rc != Coordination::Error::ZOK) + { + LOG_WARNING(log, "ExportPartition: Failed to persist commit failure bookkeeping for {}: {}", entry_path, rc); + return false; + } + + LOG_INFO(log, + "ExportPartition: Commit failure recorded for {} (code {}){}", + entry_path, exception_code, + non_retryable ? ", task transitioned to FAILED (non-retryable)" : ", will retry until task timeout"); + + return non_retryable; + } + + void appendExceptionOps( + Coordination::Requests & ops, + const zkutil::ZooKeeperPtr & zk, + const std::filesystem::path & entry_path, + const std::string & replica_name, + const std::string & part_name, + const std::string & exception_message, + const LoggerPtr & log) + { + /// Per-replica leaf under the `last_exception/` container created at task setup. + /// Each replica only ever writes its own leaf, so cross-replica updates never + /// race on the count. Concurrent writers within the same replica still race + /// on read+1+write (best-effort), matching the documented column semantics. + const auto last_exception_path + = entry_path / "last_exception" / escapeForFileName(replica_name); + + LastExceptionEntry entry; + std::string current_data; + + ProfileEvents::increment(ProfileEvents::ExportPartitionZooKeeperRequests); + ProfileEvents::increment(ProfileEvents::ExportPartitionZooKeeperGet); + const bool leaf_exists = zk->tryGet(last_exception_path, current_data); + if (leaf_exists) + { + try + { + entry = LastExceptionEntry::fromJsonString(current_data); + } + catch (...) + { + LOG_WARNING(log, "ExportPartition: last_exception JSON at {} is malformed, resetting", last_exception_path.string()); + entry = LastExceptionEntry{}; + } + } + + entry.message = exception_message; + entry.part = part_name; + entry.replica = replica_name; + entry.time = ::time(nullptr); + entry.count += 1; + + if (!leaf_exists) + { + /// Materialize the leaf out-of-band (idempotently) so the op we hand back to the + /// caller's atomic multi is always a conflict-free Set. Two failing parts on the + /// same replica whose first failures race would both pick Create here; one of the + /// enclosing multis would then abort with ZNODEEXISTS and roll back its own part + /// lock removal, stranding that part behind its ephemeral lock until session loss + /// or task timeout. A peer thread winning this create (ZNODEEXISTS) is benign. + ProfileEvents::increment(ProfileEvents::ExportPartitionZooKeeperRequests); + ProfileEvents::increment(ProfileEvents::ExportPartitionZooKeeperCreate); + const auto create_code = zk->tryCreate(last_exception_path, entry.toJsonString(), zkutil::CreateMode::Persistent); + if (create_code != Coordination::Error::ZOK && create_code != Coordination::Error::ZNODEEXISTS) + LOG_INFO(log, "ExportPartition: could not pre-create last_exception leaf {}: {}", last_exception_path.string(), create_code); + } + + /// Always a version -1 Set: it can neither conflict with a peer's create nor abort the + /// enclosing multi, so the lock-release / FAILED-set ops it accompanies always commit. + ops.emplace_back(zkutil::makeSetRequest(last_exception_path, entry.toJsonString(), -1)); + } + +namespace +{ + /// Two types are interchangeable for partitioning only if their canonical names match. IDataType::equals + /// is too weak here: it deliberately treats DateTime and DateTime64 with different time zones as equal, + /// since they are interchangeable for INSERT, but a time zone changes what a temporal transform returns, + /// so the same expression over the two types can produce different partitions. + bool isSameTypeForPartitioning(const DataTypePtr & lhs, const DataTypePtr & rhs) + { + return lhs->getName() == rhs->getName(); + } + + /// The structural match is kind of permissive and is matching terms by name, not by type. + /// We also need to ensure types are the same if they are wrapped by functions. + bool castCannotBreakStructuralMatch( + const ActionsDAG::Node * destination_output, + const Names & minmax_column_names, + const DataTypes & minmax_column_types) + { + if (destination_output->type == ActionsDAG::ActionType::INPUT) + return true; + + for (const auto & required : ActionsDAG::cloneSubDAG({destination_output}, /*remove_aliases=*/ true).getRequiredColumns()) + { + const auto it = std::find(minmax_column_names.begin(), minmax_column_names.end(), required.name); + if (it == minmax_column_names.end()) + return false; + + if (!isSameTypeForPartitioning(minmax_column_types[static_cast(it - minmax_column_names.begin())], required.type)) + return false; + } + + return true; + } + + /// Dynamically verifies the destination expression maps to a single partition by checking its monotonicity over the source range. + void verifyOutputMapsToSinglePartition( + const ActionsDAG::Node * destination_output, + const Names & minmax_column_names, + const DataTypes & minmax_column_types, + const IMergeTreeDataPart::MinMaxIndex & minmax, + const String & partition_id, + const ContextPtr & context) + { + auto chain = buildPossiblyMonotonicChain(destination_output); + if (!chain.input_node) + throw Exception(ErrorCodes::BAD_ARGUMENTS, + "Cannot export partition: the destination partition expression '{}' is not a chain of functions " + "with known monotonicity over a single column, so it cannot be proven that the source partition " + "maps to a single destination partition.", destination_output->result_name); + + const auto & column = chain.input_node->result_name; + const auto slot_it = std::find(minmax_column_names.begin(), minmax_column_names.end(), column); + if (slot_it == minmax_column_names.end()) + throw Exception(ErrorCodes::BAD_ARGUMENTS, + "Cannot export partition: the destination partition expression uses column '{}', which is " + "not part of the source MergeTree partition key.", column); + const size_t slot = static_cast(slot_it - minmax_column_names.begin()); + const auto & source_type = minmax_column_types[slot]; + + /// A NULL value forms its own destination partition, so a Nullable column may split the source + /// partition; min/max cannot rule that out. Require a structural match for such columns. + if (isNullableOrLowCardinalityNullable(source_type)) + throw Exception(ErrorCodes::BAD_ARGUMENTS, + "Cannot export partition: column '{}' is Nullable, so a NULL forms a separate destination " + "partition; partition the source by the matching destination partition expression.", column); + + if (!minmax.initialized || slot >= minmax.hyperrectangle.size()) + throw Exception(ErrorCodes::BAD_ARGUMENTS, + "Cannot export partition: no min/max statistics available for column '{}' in partition " + "'{}'; cannot validate partitioning.", column, partition_id); + const auto & min_value = minmax.hyperrectangle[slot].left; + const auto & max_value = minmax.hyperrectangle[slot].right; + + const auto & destination_type = chain.input_node->result_type; + + /// If the types are not the same, we need to check if the cast is monotonic + if (!isSameTypeForPartitioning(source_type, destination_type)) + { + const auto cast_function + = createInternalCast({source_type, column}, destination_type, CastType::nonAccurate, {}, context); + if (!cast_function->hasInformationAboutMonotonicity() + || !cast_function->getMonotonicityForRange(*source_type, min_value, max_value).is_monotonic) + throw Exception(ErrorCodes::BAD_ARGUMENTS, + "Cannot export partition '{}': values of column '{}' cross a non-monotonic cast boundary to " + "the destination type {}, so it spans multiple destination partitions.", + partition_id, column, destination_type->getName()); + } + + if (!isMonotonicChain(destination_output, chain)) + throw Exception(ErrorCodes::BAD_ARGUMENTS, + "Cannot export partition '{}': the destination partition expression '{}' is not monotonic in " + "column '{}' (a hash such as icebergBucket never is), so its values at the endpoints of the " + "partition do not bound the rows in between.", + partition_id, destination_output->result_name, column); + + auto endpoints = source_type->createColumn(); + endpoints->insert(min_value); + endpoints->insert(max_value); + + Block block{{castColumn({std::move(endpoints), source_type, column}, destination_type), destination_type, column}}; + ExpressionActions(ActionsDAG::cloneSubDAG({destination_output}, /*remove_aliases=*/ true)).execute(block); + + const auto & result = *block.getByName(destination_output->result_name).column; + Field at_min; + Field at_max; + result.get(0, at_min); + result.get(1, at_max); + + if (at_min != at_max) + throw Exception(ErrorCodes::BAD_ARGUMENTS, + "Cannot export partition '{}': the source partition might span multiple destination partitions " + "for expression '{}'. A source MergeTree partition must map to a single destination partition.", + partition_id, destination_output->result_name); + } + + /// A source partition is not split in the destination when every destination partition expression is + /// single-valued over it. That holds structurally when the expression is a deterministic function of the + /// source partition key, because rows agreeing on the source key then agree on it as well; the remaining + /// expressions have to be proven from the partition's min/max values. + void verifyPartitionKeyCompatibility( + const KeyDescription & source_key, + const KeyDescription & destination_key, + const MergeTreeData::DataPartsVector & parts, + const String & partition_id, + const ContextPtr & context) + { + /// An unpartitioned destination holds everything in a single partition. + if (destination_key.column_names.empty()) + return; + + const auto & destination_dag = destination_key.expression->getActionsDAG(); + const auto source_dag = ActionsDAG::cloneSubDAG( + source_key.expression->getActionsDAG().findInOutputs(source_key.column_names), /*remove_aliases=*/ true); + + /// ARRAY JOIN turns one row into many, which neither the tree matcher nor min/max models. + if (source_dag.hasArrayJoin() || destination_dag.hasArrayJoin()) + throw Exception(ErrorCodes::BAD_ARGUMENTS, + "Cannot export partition: a partition key containing ARRAY JOIN is not supported."); + + /// Injective functions do not group rows, so the values they are applied to are what a destination + /// expression has to be a function of. + const auto irreducible_source_nodes = removeInjectiveFunctionsFromResultsRecursively(source_dag); + const auto matches = matchTrees(source_dag.getOutputs(), destination_dag); + + const auto minmax_columns = MergeTreeData::getMinMaxColumns( + source_key, parts.front()->storage.getSettings(), MergeTreePartMinMaxIndexColumns::PARTITION_KEY_ONLY); + const auto minmax_column_names = minmax_columns.getNames(); + const auto minmax_column_types = minmax_columns.getTypes(); + + /// Compute the global min/max index of the parts + IMergeTreeDataPart::MinMaxIndex minmax; + for (const auto & part : parts) + minmax.merge(*part->getMinMaxIndex()); + + /* + 1. If there is a structural match between the source and destination key, we accept it + 2. If there is not a structural match, we check if the destination expression maps to a single partition by checking its monotonicity over the source range. + */ + NodeMap visited; + for (const auto * destination_output : destination_dag.findInOutputs(destination_key.column_names)) + { + if (allOutputsDependsOnlyOnAllowedNodes(irreducible_source_nodes, matches, destination_output, visited) + && castCannotBreakStructuralMatch(destination_output, minmax_column_names, minmax_column_types)) + continue; + + verifyOutputMapsToSinglePartition( + destination_output, minmax_column_names, minmax_column_types, minmax, partition_id, context); + } + } +} + +#if USE_AVRO + void verifyIcebergPartitionCompatibility( + const Poco::JSON::Object::Ptr & metadata_object, + const StorageMetadataPtr & source_metadata, + const StorageMetadataPtr & destination_metadata, + const MergeTreeData::DataPartsVector & parts, + const String & partition_id, + const ContextPtr & context) + { + const auto original_schema_id = metadata_object->getValue(Iceberg::f_current_schema_id); + const auto partition_spec_id = metadata_object->getValue(Iceberg::f_default_spec_id); + + Poco::JSON::Object::Ptr current_schema_json; + { + const auto schemas = metadata_object->getArray(Iceberg::f_schemas); + for (size_t i = 0; i < schemas->size(); ++i) + { + auto s = schemas->getObject(static_cast(i)); + if (s->getValue(Iceberg::f_schema_id) == static_cast(original_schema_id)) + { + current_schema_json = s; + break; + } + } + } + + Poco::JSON::Object::Ptr partition_spec_json; + { + const auto specs = metadata_object->getArray(Iceberg::f_partition_specs); + for (size_t i = 0; i < specs->size(); ++i) + { + auto s = specs->getObject(static_cast(i)); + if (s->getValue(Iceberg::f_spec_id) == partition_spec_id) + { + partition_spec_json = s; + break; + } + } + } + + if (!current_schema_json || !partition_spec_json) + throw Exception(ErrorCodes::BAD_ARGUMENTS, + "Cannot export partition to Iceberg table: destination metadata is malformed, " + "current-schema-id '{}' or default-spec-id '{}' does not resolve to a schema/spec.", + original_schema_id, partition_spec_id); + + std::unordered_map source_id_to_column_name; + { + const auto schema_fields = current_schema_json->getArray(Iceberg::f_fields); + for (size_t i = 0; i < schema_fields->size(); ++i) + { + auto f = schema_fields->getObject(static_cast(i)); + source_id_to_column_name[f->getValue(Iceberg::f_id)] = f->getValue(Iceberg::f_name); + } + } + + const auto spec_fields = partition_spec_json->getArray(Iceberg::f_fields); + const UInt32 spec_size = spec_fields ? static_cast(spec_fields->size()) : 0; + if (spec_size == 0) + return; + + /// Rebuild the destination spec as a ClickHouse partition key, the way the Iceberg read path does in + /// ManifestFileIterator, so the same compatibility rule applies as for a plain object storage + /// destination and the transform arguments keep the order the writer will use. + const String partition_timezone = context->getSettingsRef()[Setting::iceberg_partition_timezone]; + auto partition_key_ast = make_intrusive(); + partition_key_ast->name = "tuple"; + partition_key_ast->arguments = make_intrusive(); + partition_key_ast->children.push_back(partition_key_ast->arguments); + + for (UInt32 i = 0; i < spec_size; ++i) + { + const auto field = spec_fields->getObject(i); + const auto transform = field->getValue(Iceberg::f_transform); + const auto source_id = field->getValue(Iceberg::f_source_id); + + const auto column_it = source_id_to_column_name.find(source_id); + if (column_it == source_id_to_column_name.end()) + throw Exception(ErrorCodes::BAD_ARGUMENTS, + "Cannot export partition to Iceberg table: destination partition spec refers to source_id " + "{}, which is not part of the current schema.", source_id); + + auto transform_ast = Iceberg::getASTFromTransform(transform, column_it->second, partition_timezone); + if (!transform_ast) + throw Exception(ErrorCodes::BAD_ARGUMENTS, + "Cannot export partition to Iceberg table: destination field on column '{}' uses transform " + "'{}', which has no ClickHouse equivalent.", column_it->second, transform); + + partition_key_ast->arguments->children.emplace_back(std::move(transform_ast)); + } + + const auto destination_columns = ColumnsDescription::fromNamesAndTypes( + destination_metadata->getSampleBlockNonMaterialized().getNamesAndTypes()); + + verifyPartitionKeyCompatibility( + source_metadata->getPartitionKey(), + KeyDescription::getKeyFromAST(partition_key_ast, destination_columns, /*virtuals=*/ {}, context), + parts, partition_id, context); + } +#endif + + void verifyPlainPartitionCompatibility( + const StorageMetadataPtr & source_metadata, + const StorageMetadataPtr & destination_metadata, + const MergeTreeData::DataPartsVector & parts, + const String & partition_id, + const ContextPtr & context) + { + verifyPartitionKeyCompatibility( + source_metadata->getPartitionKey(), destination_metadata->getPartitionKey(), parts, partition_id, context); + } + + namespace + { + bool haveSameTupleElementLayout(const DataTypePtr & source_type, const DataTypePtr & destination_type) + { + const auto source_type_unwrapped = removeNullable(removeLowCardinality(source_type)); + const auto destination_type_unwrapped = removeNullable(removeLowCardinality(destination_type)); + + const auto * source_tuple = checkAndGetDataType(source_type_unwrapped.get()); + const auto * destination_tuple = checkAndGetDataType(destination_type_unwrapped.get()); + if (source_tuple || destination_tuple) + { + if (!source_tuple || !destination_tuple) + return false; + + if (source_tuple->hasExplicitNames() && destination_tuple->hasExplicitNames()) + { + if (source_tuple->getElementNames() != destination_tuple->getElementNames()) + return false; + } + else if (source_tuple->getElements().size() != destination_tuple->getElements().size()) + return false; + + const auto & source_elements = source_tuple->getElements(); + const auto & destination_elements = destination_tuple->getElements(); + for (size_t i = 0; i < source_elements.size(); ++i) + if (!haveSameTupleElementLayout(source_elements[i], destination_elements[i])) + return false; + + return true; + } + + const auto * source_array = checkAndGetDataType(source_type_unwrapped.get()); + const auto * destination_array = checkAndGetDataType(destination_type_unwrapped.get()); + if (source_array || destination_array) + { + if (!source_array || !destination_array) + return false; + + return haveSameTupleElementLayout(source_array->getNestedType(), destination_array->getNestedType()); + } + + const auto * source_map = checkAndGetDataType(source_type_unwrapped.get()); + const auto * destination_map = checkAndGetDataType(destination_type_unwrapped.get()); + if (source_map || destination_map) + { + if (!source_map || !destination_map) + return false; + + return haveSameTupleElementLayout(source_map->getKeyType(), destination_map->getKeyType()) + && haveSameTupleElementLayout(source_map->getValueType(), destination_map->getValueType()); + } + + return true; + } + + void verifyPartitionKeyColumn( + const ColumnWithTypeAndName & source_column, + const ColumnWithTypeAndName & destination_column, + size_t position, + const StorageID & destination_storage_id) + { + if (source_column.name != destination_column.name) + throw Exception( + ErrorCodes::BAD_ARGUMENTS, + "Cannot export to {}: partition key column '{}' is at position {} in the source " + "table, but the destination's column at that position is named '{}'. EXPORT " + "PART/PARTITION matches columns by position, so partition key columns must be " + "declared at the same position in both tables.", + destination_storage_id.getFullTableName(), + source_column.name, + position, + destination_column.name); + + if (!haveSameTupleElementLayout(source_column.type, destination_column.type)) + throw Exception( + ErrorCodes::BAD_ARGUMENTS, + "Cannot export to {}: partition key column '{}' has a different Tuple element " + "layout in the source ({}) and destination ({}). Tuple element names must be " + "declared in the same order in both tables.", + destination_storage_id.getFullTableName(), + source_column.name, + source_column.type->getName(), + destination_column.type->getName()); + } + } + + void verifyExportSchemaCastable( + const StorageMetadataPtr & source_metadata, + const StorageMetadataPtr & destination_metadata, + const StorageID & destination_storage_id, + const ContextPtr & context) + { + /// Build (and discard) the same converting DAG the export worker will build + /// later, to surface structural mismatches (column count, untyped casts) early. + Block source_sample_block; + for (const auto & column : source_metadata->getColumns().getReadable()) + source_sample_block.insert({column.type->createColumn(), column.type, column.name}); + + const auto destination_sample_block = destination_metadata->getSampleBlockNonMaterialized(); + + auto source_columns = source_sample_block.getColumnsWithTypeAndName(); + const auto & destination_columns = destination_sample_block.getColumnsWithTypeAndName(); + + /// In `ignore_extra_source_columns_by_position` mode a source with more columns than the destination + /// is allowed: the extra trailing source columns (by position) are dropped, mirroring + /// the trimming `ExportPartTask::addExportConvertingActions` applies to the real data. + /// The reverse (destination has more columns than source) is always rejected below by + /// `makeConvertingActions`, in both modes. + const bool ignore_extra_source_columns_by_position = + context->getSettingsRef()[Setting::export_merge_tree_part_schema_mismatch_mode] + == MergeTreePartExportSchemaMismatchMode::ignore_extra_source_columns_by_position; + + if (ignore_extra_source_columns_by_position && source_columns.size() > destination_columns.size()) + { + LOG_DEBUG(getLogger("ExportPartitionUtils"), + "Source has {} columns while destination has {} columns, " + "the {} extra trailing source column(s) will be ignored", + source_columns.size(), destination_columns.size(), + source_columns.size() - destination_columns.size()); + + source_columns.resize(destination_columns.size()); + } + + (void) ActionsDAG::makeConvertingActions( + source_columns, + destination_columns, + ActionsDAG::MatchColumnsMode::Position, + context); + + const auto & source_columns_description = source_metadata->getColumns(); + /// Collect the top-level columns that own columns or subcolumns required by `PARTITION BY`. + /// For example, both `PARTITION BY t.a` and `PARTITION BY (t.a, t.b)` add `t`. + std::unordered_set partition_key_owner_columns; + for (const auto & column_or_subcolumn_name : source_metadata->getColumnsRequiredForPartitionKey()) + { + auto resolved = source_columns_description.tryGetColumnOrSubcolumn( + GetColumnsOptions::All, column_or_subcolumn_name); + const auto & column_name = resolved ? resolved->getNameInStorage() : column_or_subcolumn_name; + partition_key_owner_columns.insert(column_name); + } + + const bool allow_lossy_cast = context->getSettingsRef()[Setting::export_merge_tree_part_allow_lossy_cast]; + + const size_t num_columns = std::min(source_columns.size(), destination_columns.size()); + for (size_t i = 0; i < num_columns; ++i) + { + const auto & source_column = source_columns[i]; + const auto & destination_column = destination_columns[i]; + + if (partition_key_owner_columns.contains(source_column.name)) + verifyPartitionKeyColumn(source_column, destination_column, i, destination_storage_id); + + /// Lossy casts may silently change values, so reject them unless the user opts in. + if (allow_lossy_cast) + continue; + + if (!canBeSafelyCast(source_column.type, destination_column.type)) + throw Exception(ErrorCodes::INCOMPATIBLE_COLUMNS, + "Cannot export to {}: column '{}' requires a lossy cast from {} to {}, " + "which may change values. Set `export_merge_tree_part_allow_lossy_cast = 1` " + "to allow lossy casts during export.", + destination_storage_id.getFullTableName(), + destination_column.name, + source_column.type->getName(), + destination_column.type->getName()); + } + } +} + +} diff --git a/src/Storages/MergeTree/ExportPartitionUtils.h b/src/Storages/MergeTree/ExportPartitionUtils.h new file mode 100644 index 000000000000..7605bd43ac4a --- /dev/null +++ b/src/Storages/MergeTree/ExportPartitionUtils.h @@ -0,0 +1,132 @@ +#pragma once + +#include +#include +#include +#include +#include +#include +#include "Storages/IStorage.h" +#include +#include +#include + +#if USE_AVRO +#include +#include +#endif + +namespace DB +{ + +class MergeTreeData; +struct ExportReplicatedMergeTreePartitionManifest; + +namespace ExportPartitionUtils +{ + bool isNonRetryableExportError(int code); + + std::vector getExportedPaths(const LoggerPtr & log, const zkutil::ZooKeeperPtr & zk, const std::string & export_path); + + ContextPtr getContextCopyWithTaskSettings(const ContextPtr & context, const ExportReplicatedMergeTreePartitionManifest & manifest); + + /// Get the min/max values from the partition expression columns + Block getPartitionSourceBlockForIcebergCommit( + MergeTreeData & storage, const String & partition_id, const std::vector & exported_part_names); + + void commit( + const ExportReplicatedMergeTreePartitionManifest & manifest, + const StoragePtr & destination_storage, + const zkutil::ZooKeeperPtr & zk, + const LoggerPtr & log, + const std::string & entry_path, + const ContextPtr & context, + MergeTreeData & source_storage, + const String & replica_name + ); + + /// Handles a commit-phase failure for a replicated partition export: + /// - records the exception via appendExceptionOps in the same multi + /// - if `exception_code` is non-retryable (see isNonRetryableExportError), sets + /// /status to FAILED (version-checked against the PENDING read) + /// - otherwise leaves the task PENDING so the commit is retried (by the next + /// last-part success or deferred-commit recovery) until the absolute task timeout + /// + /// There is no per-task commit-attempt budget: retryable commit failures retry until + /// success or timeout, matching the per-part retry semantics. + /// + /// Returns true if this call transitioned the task to FAILED. + bool handleCommitFailure( + const zkutil::ZooKeeperPtr & zk, + const std::string & entry_path, + int exception_code, + const std::string & replica_name, + const std::string & exception_message, + const LoggerPtr & log); + + /// Appends a single ZK op to `ops` that writes the per-replica leaf + /// /last_exception/ + /// with a JSON-encoded LastExceptionEntry containing the message, part, + /// replica, time, and an incremented count. If the leaf does not yet exist + /// the op is a Create; otherwise it is a Set with version -1. + /// + /// Cross-replica updates do not race: each replica only writes its own + /// leaf. Within a single replica the count increment is best-effort and + /// non-atomic (synchronous tryGet + Set with version -1); concurrent + /// failing writers may under-count by one, which is accepted. + void appendExceptionOps( + Coordination::Requests & ops, + const zkutil::ZooKeeperPtr & zk, + const std::filesystem::path & entry_path, + const std::string & replica_name, + const std::string & part_name, + const std::string & exception_message, + const LoggerPtr & log); + + void assertPartitionKeyASTAreEqual( + const StorageMetadataPtr & source_metadata, + const StorageMetadataPtr & destination_metadata); + + /// Validates that source columns can be exported into the destination with the + /// same positional CAST matching as `INSERT INTO dest SELECT * FROM src`. Lossy + /// casts are rejected unless `export_merge_tree_part_allow_lossy_cast` is set. + /// + /// By default the source and destination must have the same number of columns. + /// If `export_merge_tree_part_schema_mismatch_mode = 'ignore_extra_source_columns_by_position'`, a + /// source with more columns than the destination is allowed: the extra trailing + /// source columns (by position) are excluded from the comparison here, matching + /// what `ExportPartTask::addExportConvertingActions` drops from the actual data. + /// + /// Throws BAD_ARGUMENTS on any violation. + void verifyExportSchemaCastable( + const StorageMetadataPtr & source_metadata, + const StorageMetadataPtr & destination_metadata, + const StorageID & destination_storage_id, + const ContextPtr & context); + + void verifyPlainPartitionCompatibility( + const StorageMetadataPtr & source_metadata, + const StorageMetadataPtr & destination_metadata, + const MergeTreeData::DataPartsVector & parts, + const String & partition_id, + const ContextPtr & context); + +#if USE_AVRO + /// Verifies the source MergeTree partition key is compatible with the destination Iceberg + /// partition spec: every destination partition field must be single-valued across the exported + /// source partition (which the commit path requires - it writes one partition tuple per export). + /// A field is proven either structurally (the source key already applies the matching transform + /// on that column) or dynamically, by checking the destination transform is constant over the + /// partition's actual [min, max] folded across `parts`. `bucket` is non-monotonic and can only be + /// matched structurally. Throws BAD_ARGUMENTS when a field cannot be proven. + void verifyIcebergPartitionCompatibility( + const Poco::JSON::Object::Ptr & metadata_object, + const StorageMetadataPtr & source_metadata, + const StorageMetadataPtr & destination_metadata, + const MergeTreeData::DataPartsVector & parts, + const String & partition_id, + const ContextPtr & context); +#endif +} + +} diff --git a/src/Storages/MergeTree/IMergeTreeDataPart.cpp b/src/Storages/MergeTree/IMergeTreeDataPart.cpp index 81ff0ca34e5b..dd0357ed1503 100644 --- a/src/Storages/MergeTree/IMergeTreeDataPart.cpp +++ b/src/Storages/MergeTree/IMergeTreeDataPart.cpp @@ -510,6 +510,42 @@ void IMergeTreeDataPart::setMinMaxIndex(MinMaxIndexPtr minmax_index) const minmax_idx = std::move(minmax_index); } +Block IMergeTreeDataPart::MinMaxIndex::getBlock(const MergeTreeData & data) const +{ + if (!initialized) + throw Exception(ErrorCodes::LOGICAL_ERROR, "Attempt to get block from uninitialized MinMax index."); + + Block block; + + const auto metadata_snapshot = data.getInMemoryMetadataPtr(data.getContext(), false); + const auto & partition_key = metadata_snapshot->getPartitionKey(); + + /// The minmax index may also contain block number/offset columns at the end, + /// they are not part of the partition key, so take only the partition key columns. + const auto minmax_columns = MergeTreeData::getMinMaxColumns( + partition_key, data.getSettings(), MergeTreePartMinMaxIndexColumns::PARTITION_KEY_ONLY); + + size_t i = 0; + for (const auto & [column_name, data_type] : minmax_columns) + { + const auto column = data_type->createColumn(); + + auto range = hyperrectangle.at(i); + range.shrinkToIncludedIfPossible(); + + const auto & min_val = range.left; + const auto & max_val = range.right; + + column->insert(min_val); + column->insert(max_val); + + block.insert(ColumnWithTypeAndName(column->getPtr(), data_type, column_name)); + ++i; + } + + return block; +} + void IMergeTreeDataPart::incrementStateMetric(MergeTreeDataPartState state_) const { switch (state_) diff --git a/src/Storages/MergeTree/IMergeTreeDataPart.h b/src/Storages/MergeTree/IMergeTreeDataPart.h index 86d129a5f38f..ab029ac839ae 100644 --- a/src/Storages/MergeTree/IMergeTreeDataPart.h +++ b/src/Storages/MergeTree/IMergeTreeDataPart.h @@ -447,6 +447,8 @@ class IMergeTreeDataPart : public std::enable_shared_from_this; diff --git a/src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp b/src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp index f04a6a910a1b..7c3bbfcd41bb 100644 --- a/src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp +++ b/src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp @@ -157,6 +157,12 @@ size_t MergeTreeBackgroundExecutor::getMaxTasksCount() const return max_tasks_count.load(std::memory_order_relaxed); } +template +size_t MergeTreeBackgroundExecutor::getAvailableSlots() const +{ + return getMaxTasksCount() - CurrentMetrics::values[metric].load(std::memory_order_relaxed); +} + template bool MergeTreeBackgroundExecutor::trySchedule(ExecutableTaskPtr task) { diff --git a/src/Storages/MergeTree/MergeTreeBackgroundExecutor.h b/src/Storages/MergeTree/MergeTreeBackgroundExecutor.h index e7db0523ea27..8b93a5d4799e 100644 --- a/src/Storages/MergeTree/MergeTreeBackgroundExecutor.h +++ b/src/Storages/MergeTree/MergeTreeBackgroundExecutor.h @@ -330,6 +330,8 @@ class MergeTreeBackgroundExecutor final : boost::noncopyable /// can lead only to some postponing, not logical error. size_t getMaxTasksCount() const; + size_t getAvailableSlots() const; + bool trySchedule(ExecutableTaskPtr task); void removeTasksCorrespondingToStorage(StorageID id); void wait(); diff --git a/src/Storages/MergeTree/MergeTreeData.cpp b/src/Storages/MergeTree/MergeTreeData.cpp index 54a70a444220..df436b80aa27 100644 --- a/src/Storages/MergeTree/MergeTreeData.cpp +++ b/src/Storages/MergeTree/MergeTreeData.cpp @@ -10,12 +10,14 @@ #include #include +#include #include #if CLICKHOUSE_CLOUD #include #include #include #endif +#include #include #include #include @@ -27,6 +29,13 @@ #include #include #include +#include +#include +#include +#include +#include +#include +#include #include #include #include @@ -45,6 +54,7 @@ #include #include #include +#include #include #include #include @@ -94,6 +104,9 @@ #include #include #include +#include +#include +#include #include #include #include @@ -131,6 +144,7 @@ #include #include #include +#include #include #include #include @@ -146,6 +160,7 @@ #include #include #include +#include #include @@ -165,6 +180,7 @@ #include #include #include +#include #include #include @@ -211,6 +227,11 @@ namespace ProfileEvents extern const Event RestorePartsSkippedFiles; extern const Event RestorePartsSkippedBytes; extern const Event LoadedStatisticsMicroseconds; + extern const Event PartsExports; + extern const Event PartsExportTotalMilliseconds; + extern const Event PartsExportFailures; + extern const Event PartsExportDuplicated; + extern const Event ExportPartsRejectedByMemoryLimit; } namespace CurrentMetrics @@ -267,6 +288,14 @@ namespace Setting extern const SettingsBool use_partition_pruning; extern const SettingsBool use_constant_folding_in_index_analysis; extern const SettingsBool use_skip_indexes; + extern const SettingsBool allow_experimental_export_merge_tree_part; + extern const SettingsUInt64 min_bytes_to_use_direct_io; + extern const SettingsMergeTreePartExportFileAlreadyExistsPolicy export_merge_tree_part_file_already_exists_policy; + extern const SettingsBool output_format_parallel_formatting; + extern const SettingsBool output_format_parquet_parallel_encoding; + extern const SettingsBool export_merge_tree_part_throw_on_pending_mutations; + extern const SettingsBool export_merge_tree_part_throw_on_pending_patch_parts; + extern const SettingsBool allow_insert_into_iceberg; } namespace MergeTreeSetting @@ -422,6 +451,9 @@ namespace ErrorCodes extern const int TOO_LARGE_LIGHTWEIGHT_UPDATES; extern const int FAULT_INJECTED; extern const int TABLE_IS_PERMANENTLY_READ_ONLY; + extern const int UNKNOWN_TABLE; + extern const int FILE_ALREADY_EXISTS; + extern const int PENDING_MUTATIONS_NOT_ALLOWED; } namespace FailPoints @@ -6296,8 +6328,6 @@ void MergeTreeData::changeSettings( { if (new_settings) { - bool has_storage_policy_changed = false; - auto new_changes = new_settings->as().changes; MergeTreeSettings::resolveDiskSetting(new_changes, getContext(), /*is_loading_from_existing_metadata=*/true); @@ -6338,8 +6368,6 @@ void MergeTreeData::changeSettings( disk->createDirectories(fs::path(relative_data_path) / DETACHED_DIR_NAME); } /// FIXME how would that be done while reloading configuration??? - - has_storage_policy_changed = true; } } } @@ -6381,9 +6409,6 @@ void MergeTreeData::changeSettings( setInMemoryMetadata(new_metadata); - if (has_storage_policy_changed) - startBackgroundMovesIfNeeded(); - if (has_refresh_statistics_interval_changed) { startStatisticsCache(); @@ -8123,8 +8148,11 @@ void MergeTreeData::checkAlterPartitionIsPossible( const auto * partition_ast = command.partition->as(); if (partition_ast && partition_ast->all) { - if (command.type != PartitionCommand::DROP_PARTITION && command.type != PartitionCommand::ATTACH_PARTITION && !(command.type == PartitionCommand::REPLACE_PARTITION && !command.replace)) - throw DB::Exception(ErrorCodes::SUPPORT_IS_DISABLED, "Only support DROP/DETACH/ATTACH PARTITION ALL currently"); + if (command.type != PartitionCommand::DROP_PARTITION + && command.type != PartitionCommand::ATTACH_PARTITION + && command.type != PartitionCommand::EXPORT_PARTITION + && !(command.type == PartitionCommand::REPLACE_PARTITION && !command.replace)) + throw DB::Exception(ErrorCodes::SUPPORT_IS_DISABLED, "Only support DROP/DETACH/ATTACH/EXPORT PARTITION ALL currently"); } else { @@ -8367,6 +8395,265 @@ void MergeTreeData::movePartitionToTable(const PartitionCommand & command, Conte movePartitionToTable(dest_storage, command.partition, query_context); } +void MergeTreeData::exportPartToTable(const PartitionCommand & command, ContextPtr query_context) +{ + if (!query_context->getSettingsRef()[Setting::allow_experimental_export_merge_tree_part]) + { + throw Exception(ErrorCodes::SUPPORT_IS_DISABLED, + "Exporting merge tree part is experimental. Set `allow_experimental_export_merge_tree_part` to enable it"); + } + + const auto part_name = command.partition->as().value.safeGet(); + + if (!command.to_table_function) + { + const auto database_name = query_context->resolveDatabase(command.to_database); + exportPartToTable(part_name, StorageID{database_name, command.to_table}, generateSnowflakeIDString(), query_context); + + return; + } + + auto table_function_ast = command.to_table_function; + auto table_function_ptr = TableFunctionFactory::instance().get(command.to_table_function, query_context); + + if (table_function_ptr->needStructureHint()) + { + const auto source_metadata_ptr = getInMemoryMetadataPtr(query_context, false); + + /// Grab only the readable columns from the source metadata to skip ephemeral columns + const auto readable_columns = ColumnsDescription(source_metadata_ptr->getColumns().getReadable()); + table_function_ptr->setStructureHint(readable_columns); + } + + if (command.partition_by_expr) + { + table_function_ptr->setPartitionBy(command.partition_by_expr); + } + + auto dest_storage = table_function_ptr->execute( + table_function_ast, + query_context, + table_function_ptr->getName(), + /* cached_columns */ {}, + /* use_global_context */ false, + /* is_insert_query */ true); + + if (!dest_storage) + { + throw Exception(ErrorCodes::BAD_ARGUMENTS, "Failed to reconstruct destination storage"); + } + + exportPartToTable(part_name, dest_storage, generateSnowflakeIDString(), query_context); +} + +void MergeTreeData::exportPartToTable( + const std::string & part_name, + const StorageID & destination_storage_id, + const String & transaction_id, + ContextPtr query_context, + const std::optional & iceberg_metadata_json, + bool allow_outdated_parts, + std::function completion_callback) +{ + auto dest_storage = DatabaseCatalog::instance().getTable(destination_storage_id, query_context); + + if (destination_storage_id == this->getStorageID()) + { + throw Exception(ErrorCodes::BAD_ARGUMENTS, "Exporting to the same table is not allowed"); + } + + exportPartToTable(part_name, dest_storage, transaction_id, query_context, iceberg_metadata_json, allow_outdated_parts, completion_callback); +} + +void MergeTreeData::exportPartToTable( + const std::string & part_name, + const StoragePtr & dest_storage, + const String & transaction_id, + ContextPtr query_context, + const std::optional & iceberg_metadata_json_, + bool allow_outdated_parts, + std::function completion_callback) +{ + if (!dest_storage->supportsImport(query_context)) + throw Exception(ErrorCodes::NOT_IMPLEMENTED, "Destination storage {} does not support MergeTree parts or uses unsupported partitioning", dest_storage->getName()); + + auto source_metadata_ptr = getInMemoryMetadataPtr(query_context, false); + auto destination_metadata_ptr = dest_storage->getInMemoryMetadataPtr(query_context, false); + + if (dest_storage->isDataLake() && !query_context->getSettingsRef()[Setting::allow_insert_into_iceberg]) + { + throw Exception(ErrorCodes::SUPPORT_IS_DISABLED, + "Iceberg writes are experimental. " + "To allow its usage, enable the setting `allow_insert_into_iceberg`."); + } + + ExportPartitionUtils::verifyExportSchemaCastable( + source_metadata_ptr, destination_metadata_ptr, dest_storage->getStorageID(), query_context); + + auto part = getPartIfExists(part_name, {MergeTreeDataPartState::Active, MergeTreeDataPartState::Outdated}); + + if (!part) + throw Exception(ErrorCodes::NO_SUCH_DATA_PART, "No such data part '{}' to export in table '{}'", + part_name, getStorageID().getFullTableName()); + + std::string iceberg_metadata_json; + + if (dest_storage->isDataLake()) + { +#if USE_AVRO + if (iceberg_metadata_json_) + { + iceberg_metadata_json = *iceberg_metadata_json_; + } + else + { + auto * object_storage = dynamic_cast(dest_storage.get()); + auto * object_storage_cluster = dynamic_cast(dest_storage.get()); + + /// in theory this should never happen, but just in case + if (!object_storage && !object_storage_cluster) + { + throw Exception(ErrorCodes::BAD_ARGUMENTS, "Destination storage {} is not a StorageObjectStorage", dest_storage->getName()); + } + + std::shared_ptr iceberg_metadata; + if (object_storage) + iceberg_metadata = std::dynamic_pointer_cast(object_storage->getExternalMetadata(query_context)); + else if (object_storage_cluster) + iceberg_metadata = std::dynamic_pointer_cast(object_storage_cluster->getExternalMetadata(query_context)); + if (!iceberg_metadata) + { + throw Exception(ErrorCodes::BAD_ARGUMENTS, "Destination storage {} is a data lake but not an iceberg table", dest_storage->getName()); + } + + const auto metadata_object = iceberg_metadata->getMetadataJSON(query_context); + + std::ostringstream oss; + metadata_object->stringify(oss); + iceberg_metadata_json = oss.str(); + + ExportPartitionUtils::verifyIcebergPartitionCompatibility( + metadata_object, + source_metadata_ptr, + destination_metadata_ptr, + {part}, + part->info.getPartitionId(), + query_context); + } +#else + (void)iceberg_metadata_json_; + throw Exception(ErrorCodes::NOT_IMPLEMENTED, "Data lake export requires Avro support"); +#endif + } + else + { + /// Plain (hive) object storage writes every row of the part to the one directory computed from + /// the destination PARTITION BY on the part's min row, so the source partition must map to a + /// single destination partition. Equivalent or finer source keys are accepted. + ExportPartitionUtils::verifyPlainPartitionCompatibility( + source_metadata_ptr, + destination_metadata_ptr, + {part}, + part->info.getPartitionId(), + query_context); + } + + if (part->getState() == MergeTreeDataPartState::Outdated && !allow_outdated_parts) + throw Exception( + ErrorCodes::BAD_ARGUMENTS, + "Part {} is in the outdated state and cannot be exported", + part_name); + + const bool throw_on_pending_mutations = query_context->getSettingsRef()[Setting::export_merge_tree_part_throw_on_pending_mutations]; + const bool throw_on_pending_patch_parts = query_context->getSettingsRef()[Setting::export_merge_tree_part_throw_on_pending_patch_parts]; + + MergeTreeData::IMutationsSnapshot::Params mutations_snapshot_params + { + .metadata_version = source_metadata_ptr->getMetadataVersion(), + .min_part_metadata_version = part->getMetadataVersion(), + .need_data_mutations = throw_on_pending_mutations, + .need_alter_mutations = throw_on_pending_mutations || throw_on_pending_patch_parts, + .need_patch_parts = throw_on_pending_patch_parts, + }; + + const auto mutations_snapshot = getMutationsSnapshot(mutations_snapshot_params); + + const auto alter_conversions = getAlterConversionsForPart(part, mutations_snapshot, query_context); + + /// re-check `throw_on_pending_mutations` because `pending_mutations` might have been filled due to `throw_on_pending_patch_parts` + if (throw_on_pending_mutations && alter_conversions->hasMutations()) + { + throw Exception(ErrorCodes::PENDING_MUTATIONS_NOT_ALLOWED, + "Part {} can not be exported because there are pending mutations. Either wait for the mutations to be applied or set `export_merge_tree_part_throw_on_pending_mutations` to false", + part_name); + } + + if (alter_conversions->hasPatches()) + { + throw Exception(ErrorCodes::PENDING_MUTATIONS_NOT_ALLOWED, + "Part {} can not be exported because there are pending patch parts. Either wait for the patch parts to be applied or set `export_merge_tree_part_throw_on_pending_patch_parts` to false", + part_name); + } + + { + if (!canEnqueueBackgroundTask()) + { + ProfileEvents::increment(ProfileEvents::ExportPartsRejectedByMemoryLimit); + throw Exception(ErrorCodes::ABORTED, + "Failed to schedule export part task for data part '{}'. " + "Reached memory limit for the background tasks ({}). Current background tasks memory usage: {}.", + part_name, + formatReadableSizeWithBinarySuffix(background_memory_tracker.getSoftLimit()), + formatReadableSizeWithBinarySuffix(background_memory_tracker.get())); + } + + MergeTreePartExportManifest manifest( + dest_storage, + part, + transaction_id, + query_context->getCurrentQueryId(), + query_context->getSettingsRef()[Setting::export_merge_tree_part_file_already_exists_policy].value, + query_context->getSettingsCopy(), + source_metadata_ptr, + iceberg_metadata_json, + completion_callback); + + std::lock_guard lock(export_manifests_mutex); + + manifest.task = std::make_shared(*this, manifest); + + if (!export_manifests.emplace(manifest).second) + { + throw Exception(ErrorCodes::ABORTED, "Data part '{}' is already being exported", + part_name); + } + + if (!background_moves_assignee.scheduleMoveTask(manifest.task)) + { + export_manifests.erase(manifest); + throw Exception(ErrorCodes::ABORTED, "Failed to schedule export part task for data part '{}'. Background executor is busy", + part_name); + } + } +} + +void MergeTreeData::killExportPart(const String & transaction_id) +{ + std::lock_guard lock(export_manifests_mutex); + + std::erase_if(export_manifests, [&](const auto & manifest) + { + if (manifest.transaction_id == transaction_id) + { + if (manifest.task) + manifest.task->cancel(); + + return true; + } + return false; + }); +} + void MergeTreeData::movePartitionToShard(const ASTPtr & /*partition*/, bool /*move_part*/, const String & /*to*/, ContextPtr /*query_context*/) { throw Exception(ErrorCodes::NOT_IMPLEMENTED, "MOVE PARTITION TO SHARD is not supported by storage {}", getName()); @@ -8445,6 +8732,17 @@ Pipe MergeTreeData::alterPartition( } } break; + case PartitionCommand::EXPORT_PART: + { + exportPartToTable(command, query_context); + break; + } + + case PartitionCommand::EXPORT_PARTITION: + { + exportPartitionToTable(command, query_context); + break; + } case PartitionCommand::DROP_DETACHED_PARTITION: dropDetached(command.partition, command.part, query_context); @@ -11156,6 +11454,33 @@ std::pair MergeTreeData::cloneAn return std::make_pair(dst_data_part, std::move(temporary_directory_lock)); } +std::vector MergeTreeData::getExportsStatus() const +{ + std::lock_guard lock(export_manifests_mutex); + std::vector result; + + auto source_database = getStorageID().database_name; + auto source_table = getStorageID().table_name; + + for (const auto & manifest : export_manifests) + { + MergeTreeExportStatus status; + + status.source_database = source_database; + status.source_table = source_table; + const auto destination_storage_id = manifest.destination_storage_ptr->getStorageID(); + status.destination_database = destination_storage_id.database_name; + status.destination_table = destination_storage_id.table_name; + status.create_time = manifest.create_time; + status.part_name = manifest.data_part->name; + + result.emplace_back(std::move(status)); + } + + return result; +} + + bool MergeTreeData::canUseAdaptiveGranularity() const { const auto settings = getSettings(); @@ -11477,7 +11802,8 @@ void MergeTreeData::writePartLog( const MergeListEntry * merge_entry, std::shared_ptr profile_counters, const Strings & mutation_ids, - const std::map & projections_duration_ms) + const std::map & projections_duration_ms, + const ExportsListEntry * exports_entry) try { auto table_id = getStorageID(); @@ -11545,6 +11871,16 @@ try element.rows = (*merge_entry)->rows_written; element.peak_memory_usage = (*merge_entry)->getMemoryTracker().getPeak(); } + else if (exports_entry) + { + element.rows_read = (*exports_entry)->rows_read; + element.bytes_read_uncompressed = (*exports_entry)->bytes_read_uncompressed; + element.peak_memory_usage = (*exports_entry)->getPeakMemoryUsage(); + element.query_id = (*exports_entry)->query_id; + + /// no need to lock because at this point no one is writing to the destination file paths + element.remote_file_paths = (*exports_entry)->destination_file_paths; + } if (profile_counters) { @@ -11838,6 +12174,10 @@ bool MergeTreeData::canUsePolymorphicParts() const return canUsePolymorphicParts(*getSettings(), unused); } +void MergeTreeData::startBackgroundMoves() +{ + background_moves_assignee.start(); +} void MergeTreeData::checkDropOrRenameCommandDoesntAffectInProgressMutations( const AlterCommand & command, const std::map & unfinished_mutations, ContextPtr local_context) const diff --git a/src/Storages/MergeTree/MergeTreeData.h b/src/Storages/MergeTree/MergeTreeData.h index 8858452c5d79..3caa4b0f4c19 100644 --- a/src/Storages/MergeTree/MergeTreeData.h +++ b/src/Storages/MergeTree/MergeTreeData.h @@ -20,6 +20,7 @@ #include #include #include +#include #include #include #include @@ -41,6 +42,8 @@ #include #include #include +#include +#include #include #include @@ -1131,6 +1134,33 @@ class MergeTreeData : public WithMutableContext, public IStorage, public IBackgr /// Moves partition to specified Table void movePartitionToTable(const PartitionCommand & command, ContextPtr query_context); + void exportPartToTable(const PartitionCommand & command, ContextPtr query_context); + + void exportPartToTable( + const std::string & part_name, + const StoragePtr & destination_storage, + const String & transaction_id, + ContextPtr query_context, + const std::optional & iceberg_metadata_json = std::nullopt, + bool allow_outdated_parts = false, + std::function completion_callback = {}); + + void exportPartToTable( + const std::string & part_name, + const StorageID & destination_storage_id, + const String & transaction_id, + ContextPtr query_context, + const std::optional & iceberg_metadata_json = std::nullopt, + bool allow_outdated_parts = false, + std::function completion_callback = {}); + + void killExportPart(const String & transaction_id); + + virtual void exportPartitionToTable(const PartitionCommand &, ContextPtr) + { + throw Exception(ErrorCodes::NOT_IMPLEMENTED, "EXPORT PARTITION is not implemented for engine {}", getName()); + } + /// Checks that Partition could be dropped right now /// Otherwise - throws an exception with detailed information. /// We do not use mutex because it is not very important that the size could change during the operation. @@ -1224,6 +1254,7 @@ class MergeTreeData : public WithMutableContext, public IStorage, public IBackgr const WriteSettings & write_settings); virtual std::vector getMutationsStatus() const = 0; + std::vector getExportsStatus() const; /// Returns true if table can create new parts with adaptive granularity /// Has additional constraint in replicated version @@ -1446,6 +1477,10 @@ class MergeTreeData : public WithMutableContext, public IStorage, public IBackgr /// Used for streaming queries registration. mutable StreamSubscriptionManager subscription_manager; + mutable std::mutex export_manifests_mutex; + + std::set export_manifests; + PinnedPartUUIDsPtr getPinnedPartUUIDs() const; /// Last-resort guard for the post-vtable-demotion window of STID 3631-4165; @@ -1565,6 +1600,7 @@ class MergeTreeData : public WithMutableContext, public IStorage, public IBackgr friend class VersionMetadataOnKeeper; // for access to log friend class MutationsState; // for access to log friend class UniqueKeyDenseIndexOps; // for access to log + data_parts_by_info + friend class ExportPartTask; bool require_part_metadata; @@ -1634,6 +1670,8 @@ class MergeTreeData : public WithMutableContext, public IStorage, public IBackgr size_t getColumnsDescriptionsCacheSize() const; protected: + void startBackgroundMoves(); + /// Engine-specific methods BrokenPartCallback broken_part_callback; @@ -1925,7 +1963,8 @@ class MergeTreeData : public WithMutableContext, public IStorage, public IBackgr const MergeListEntry * merge_entry, std::shared_ptr profile_counters, const Strings & mutation_ids, - const std::map & projections_duration_ms); + const std::map & projections_duration_ms, + const ExportsListEntry * exports_entry = nullptr); /// If part is assigned to merge or mutation (possibly replicated) /// Should be overridden by children, because they can have different @@ -2156,8 +2195,6 @@ class MergeTreeData : public WithMutableContext, public IStorage, public IBackgr bool canUsePolymorphicParts(const MergeTreeSettings & settings, String & out_reason) const; - virtual void startBackgroundMovesIfNeeded() = 0; - bool allow_nullable_key = false; void addPartContributionToDataVolume(const DataPartPtr & part); diff --git a/src/Storages/MergeTree/MergeTreeExportManifest.h b/src/Storages/MergeTree/MergeTreeExportManifest.h new file mode 100644 index 000000000000..05506ecb004a --- /dev/null +++ b/src/Storages/MergeTree/MergeTreeExportManifest.h @@ -0,0 +1,50 @@ +#include +#include + +namespace DB +{ + +struct MergeTreeExportManifest +{ + using DataPartPtr = std::shared_ptr; + + + MergeTreeExportManifest( + const StorageID & destination_storage_id_, + const DataPartPtr & data_part_, + bool overwrite_file_if_exists_, + const FormatSettings & format_settings_) + : destination_storage_id(destination_storage_id_), + data_part(data_part_), + overwrite_file_if_exists(overwrite_file_if_exists_), + format_settings(format_settings_), + create_time(time(nullptr)) {} + + StorageID destination_storage_id; + DataPartPtr data_part; + bool overwrite_file_if_exists; + FormatSettings format_settings; + + time_t create_time; + mutable bool in_progress = false; + + bool operator<(const MergeTreeExportManifest & rhs) const + { + // Lexicographic comparison: first compare destination storage, then part name + auto lhs_storage = destination_storage_id.getQualifiedName(); + auto rhs_storage = rhs.destination_storage_id.getQualifiedName(); + + if (lhs_storage != rhs_storage) + return lhs_storage < rhs_storage; + + return data_part->name < rhs.data_part->name; + } + + bool operator==(const MergeTreeExportManifest & rhs) const + { + return destination_storage_id.getQualifiedName() == rhs.destination_storage_id.getQualifiedName() + && data_part->name == rhs.data_part->name; + } +}; + +} diff --git a/src/Storages/MergeTree/MergeTreePartExportManifest.h b/src/Storages/MergeTree/MergeTreePartExportManifest.h new file mode 100644 index 000000000000..08d73febf968 --- /dev/null +++ b/src/Storages/MergeTree/MergeTreePartExportManifest.h @@ -0,0 +1,98 @@ +#pragma once + +#include +#include +#include +#include +#include +#include +#include + +namespace DB +{ + +class Exception; + +class IExecutableTask; + +struct MergeTreePartExportManifest +{ + using FileAlreadyExistsPolicy = MergeTreePartExportFileAlreadyExistsPolicy; + + using DataPartPtr = std::shared_ptr; + + struct CompletionCallbackResult + { + private: + CompletionCallbackResult(bool success_, const std::vector & relative_paths_in_destination_storage_, std::optional exception_) + : success(success_), relative_paths_in_destination_storage(relative_paths_in_destination_storage_), exception(std::move(exception_)) {} + public: + + static CompletionCallbackResult createSuccess(const std::vector & relative_paths_in_destination_storage_) + { + return CompletionCallbackResult(true, relative_paths_in_destination_storage_, std::nullopt); + } + + static CompletionCallbackResult createFailure(Exception exception_) + { + return CompletionCallbackResult(false, {}, std::move(exception_)); + } + + bool success = false; + std::vector relative_paths_in_destination_storage; + std::optional exception; + }; + + MergeTreePartExportManifest( + const StoragePtr destination_storage_ptr_, + const DataPartPtr & data_part_, + const String & transaction_id_, + const String & query_id_, + FileAlreadyExistsPolicy file_already_exists_policy_, + const Settings & settings_, + const StorageMetadataPtr & metadata_snapshot_, + const String & iceberg_metadata_json_, + std::function completion_callback_ = {}) + : destination_storage_ptr(destination_storage_ptr_), + data_part(data_part_), + transaction_id(transaction_id_), + query_id(query_id_), + file_already_exists_policy(file_already_exists_policy_), + settings(settings_), + metadata_snapshot(metadata_snapshot_), + iceberg_metadata_json(iceberg_metadata_json_), + completion_callback(completion_callback_), + create_time(time(nullptr)) {} + + StoragePtr destination_storage_ptr; + DataPartPtr data_part; + /// Used for killing the export. + String transaction_id; + String query_id; + FileAlreadyExistsPolicy file_already_exists_policy; + Settings settings; + + /// Metadata snapshot captured at the time of query validation to prevent race conditions with mutations + /// Otherwise the export could fail if the schema changes between validation and execution + StorageMetadataPtr metadata_snapshot; + + String iceberg_metadata_json; + + std::function completion_callback; + + time_t create_time; + /// Required to cancel export tasks + mutable std::shared_ptr task = nullptr; + + bool operator<(const MergeTreePartExportManifest & rhs) const + { + return data_part->name < rhs.data_part->name; + } + + bool operator==(const MergeTreePartExportManifest & rhs) const + { + return data_part->name == rhs.data_part->name; + } +}; + +} diff --git a/src/Storages/MergeTree/MergeTreePartExportStatus.h b/src/Storages/MergeTree/MergeTreePartExportStatus.h new file mode 100644 index 000000000000..e71a2f15e6ed --- /dev/null +++ b/src/Storages/MergeTree/MergeTreePartExportStatus.h @@ -0,0 +1,20 @@ +#pragma once + +#include +#include + + +namespace DB +{ + +struct MergeTreeExportStatus +{ + String source_database; + String source_table; + String destination_database; + String destination_table; + time_t create_time = 0; + std::string part_name; +}; + +} diff --git a/src/Storages/MergeTree/MergeTreePartition.cpp b/src/Storages/MergeTree/MergeTreePartition.cpp index 12dfce7728a8..2cd8b4d73a2a 100644 --- a/src/Storages/MergeTree/MergeTreePartition.cpp +++ b/src/Storages/MergeTree/MergeTreePartition.cpp @@ -10,6 +10,7 @@ #include #include #include +#include #include #include #include @@ -507,6 +508,22 @@ void MergeTreePartition::create(const StorageMetadataPtr & metadata_snapshot, Bl } } +Block MergeTreePartition::getBlockWithPartitionValues(const NamesAndTypesList & partition_columns) const +{ + chassert(partition_columns.size() == value.size()); + + Block result; + + std::size_t i = 0; + for (const auto & partition_column : partition_columns) + { + ColumnPtr column = partition_column.type->createColumnConst(1, value[i++]); + result.insert({column, partition_column.type, partition_column.name}); + } + + return result; +} + NamesAndTypesList MergeTreePartition::executePartitionByExpression(const StorageMetadataPtr & metadata_snapshot, Block & block, ContextPtr context) { auto adjusted_partition_key = adjustPartitionKey(metadata_snapshot, context); diff --git a/src/Storages/MergeTree/MergeTreePartition.h b/src/Storages/MergeTree/MergeTreePartition.h index 17936fc78e31..964a7bcdc0ae 100644 --- a/src/Storages/MergeTree/MergeTreePartition.h +++ b/src/Storages/MergeTree/MergeTreePartition.h @@ -60,6 +60,8 @@ struct MergeTreePartition void create(const StorageMetadataPtr & metadata_snapshot, Block block, size_t row, ContextPtr context); + Block getBlockWithPartitionValues(const NamesAndTypesList & partition_columns) const; + /// Adjust partition key and execute its expression on block. Return sample block according to used expression. static NamesAndTypesList executePartitionByExpression(const StorageMetadataPtr & metadata_snapshot, Block & block, ContextPtr context); diff --git a/src/Storages/MergeTree/MergeTreeSequentialSource.cpp b/src/Storages/MergeTree/MergeTreeSequentialSource.cpp index 91ee5dfa67d6..b28af032ca54 100644 --- a/src/Storages/MergeTree/MergeTreeSequentialSource.cpp +++ b/src/Storages/MergeTree/MergeTreeSequentialSource.cpp @@ -176,6 +176,10 @@ MergeTreeSequentialSource::MergeTreeSequentialSource( addThrottler(read_settings.remote_throttler, context->getMergesThrottler()); addThrottler(read_settings.local_throttler, context->getMergesThrottler()); break; + case Export: + addThrottler(read_settings.local_throttler, context->getExportsThrottler()); + addThrottler(read_settings.remote_throttler, context->getExportsThrottler()); + break; } MergeTreeReadTask::Extras extras = diff --git a/src/Storages/MergeTree/MergeTreeSequentialSource.h b/src/Storages/MergeTree/MergeTreeSequentialSource.h index abba230d9e79..a858adf33bb5 100644 --- a/src/Storages/MergeTree/MergeTreeSequentialSource.h +++ b/src/Storages/MergeTree/MergeTreeSequentialSource.h @@ -15,6 +15,7 @@ enum MergeTreeSequentialSourceType { Mutation, Merge, + Export, }; /// Create stream for reading single part from MergeTree. diff --git a/src/Storages/MergeTree/ReplicatedMergeTreeRestartingThread.cpp b/src/Storages/MergeTree/ReplicatedMergeTreeRestartingThread.cpp index 1f1975170f54..aeb8a9cb4a0e 100644 --- a/src/Storages/MergeTree/ReplicatedMergeTreeRestartingThread.cpp +++ b/src/Storages/MergeTree/ReplicatedMergeTreeRestartingThread.cpp @@ -15,6 +15,7 @@ #include #include #include +#include namespace CurrentMetrics @@ -31,6 +32,11 @@ namespace MergeTreeSetting extern const MergeTreeSettingsSeconds zookeeper_session_expiration_check_period; } +namespace ServerSetting +{ + extern const ServerSettingsBool allow_experimental_export_merge_tree_partition; +} + namespace ErrorCodes { extern const int REPLICA_IS_ALREADY_ACTIVE; @@ -177,11 +183,20 @@ bool ReplicatedMergeTreeRestartingThread::runImpl() storage.mutations_updating_task->activateAndSchedule(); storage.mutations_finalizing_task->activateAndSchedule(); storage.merge_selecting_task->activateAndSchedule(); + + if (storage.getContext()->getServerSettings()[ServerSetting::allow_experimental_export_merge_tree_partition]) + { + storage.export_merge_tree_partition_updating_task->activateAndSchedule(); + storage.export_merge_tree_partition_select_task->activateAndSchedule(); + storage.export_merge_tree_partition_status_handling_task->activateAndSchedule(); + } + storage.cleanup_thread.start(); storage.part_check_thread.start(); storage.deduplication_hashes_cache.start(); + LOG_DEBUG(log, "Table started successfully"); return true; } diff --git a/src/Storages/MergeTree/tests/gtest_export_partition_ordering.cpp b/src/Storages/MergeTree/tests/gtest_export_partition_ordering.cpp new file mode 100644 index 000000000000..df20755ba590 --- /dev/null +++ b/src/Storages/MergeTree/tests/gtest_export_partition_ordering.cpp @@ -0,0 +1,174 @@ +#include +#include +#include +#include +#include +#include +#include +#include +#include + +namespace DB +{ + +namespace Setting +{ + extern const SettingsMergeTreePartExportSchemaMismatchMode export_merge_tree_part_schema_mismatch_mode; +} + +namespace +{ + ExportReplicatedMergeTreePartitionManifest makeValidManifest() + { + ExportReplicatedMergeTreePartitionManifest manifest; + manifest.transaction_id = "tx1"; + manifest.query_id = "query1"; + manifest.partition_id = "2020"; + manifest.destination_database = "db1"; + manifest.destination_table = "table1"; + manifest.source_replica = "r1"; + manifest.number_of_parts = 1; + manifest.create_time = 1000; + manifest.task_timeout_seconds = 60; + manifest.max_threads = 1; + manifest.parallel_formatting = true; + manifest.parquet_parallel_encoding = true; + manifest.max_bytes_per_file = 1000000; + manifest.max_rows_per_file = 1000; + manifest.file_already_exists_policy = MergeTreePartExportManifest::FileAlreadyExistsPolicy::error; + manifest.filename_pattern = "{part_name}"; + return manifest; + } +} + +class ExportPartitionOrderingTest : public ::testing::Test +{ +protected: + ExportPartitionTaskEntriesContainer container; + ExportPartitionTaskEntriesContainer::index::type & by_key; + ExportPartitionTaskEntriesContainer::index::type & by_create_time; + + ExportPartitionOrderingTest() + : by_key(container.get()) + , by_create_time(container.get()) + { + } +}; + +class ExportPartitionManifestBackCompatTest : public ::testing::Test +{ +}; + +TEST_F(ExportPartitionOrderingTest, IterationOrderMatchesCreateTime) +{ + time_t base_time = 1000; + + ExportReplicatedMergeTreePartitionManifest manifest1; + manifest1.partition_id = "2020"; + manifest1.destination_database = "db1"; + manifest1.destination_table = "table1"; + manifest1.transaction_id = "tx1"; + manifest1.create_time = base_time + 300; // Latest + + ExportReplicatedMergeTreePartitionManifest manifest2; + manifest2.partition_id = "2021"; + manifest2.destination_database = "db1"; + manifest2.destination_table = "table1"; + manifest2.transaction_id = "tx2"; + manifest2.create_time = base_time + 100; // Middle + + ExportReplicatedMergeTreePartitionManifest manifest3; + manifest3.partition_id = "2022"; + manifest3.destination_database = "db1"; + manifest3.destination_table = "table1"; + manifest3.transaction_id = "tx3"; + manifest3.create_time = base_time; // Oldest + + ExportReplicatedMergeTreePartitionTaskEntry entry1{manifest1, ExportReplicatedMergeTreePartitionTaskEntry::Status::PENDING, {}, {}, {}, {}}; + ExportReplicatedMergeTreePartitionTaskEntry entry2{manifest2, ExportReplicatedMergeTreePartitionTaskEntry::Status::PENDING, {}, {}, {}, {}}; + ExportReplicatedMergeTreePartitionTaskEntry entry3{manifest3, ExportReplicatedMergeTreePartitionTaskEntry::Status::PENDING, {}, {}, {}, {}}; + + // Insert in reverse order + by_key.insert(entry1); + by_key.insert(entry2); + by_key.insert(entry3); + + // Verify iteration order matches create_time (ascending) + auto it = by_create_time.begin(); + ASSERT_NE(it, by_create_time.end()); + EXPECT_EQ(it->manifest.partition_id, "2022"); // Oldest first + EXPECT_EQ(it->manifest.create_time, base_time); + + ++it; + ASSERT_NE(it, by_create_time.end()); + EXPECT_EQ(it->manifest.partition_id, "2021"); + EXPECT_EQ(it->manifest.create_time, base_time + 100); + + ++it; + ASSERT_NE(it, by_create_time.end()); + EXPECT_EQ(it->manifest.partition_id, "2020"); + EXPECT_EQ(it->manifest.create_time, base_time + 300); + + ++it; + EXPECT_EQ(it, by_create_time.end()); +} + + +TEST_F(ExportPartitionManifestBackCompatTest, MissingSchemaMismatchModeParsesAsNullopt) +{ + auto manifest = makeValidManifest(); + manifest.schema_mismatch_mode = MergeTreePartExportSchemaMismatchMode::ignore_extra_source_columns_by_position; + + Poco::JSON::Parser parser; + auto json = parser.parse(manifest.toJsonString()).extract(); + json->remove("schema_mismatch_mode"); + std::ostringstream oss; + oss.exceptions(std::ios::failbit); + Poco::JSON::Stringifier::stringify(json, oss); + + auto parsed = ExportReplicatedMergeTreePartitionManifest::fromJsonString(oss.str()); + EXPECT_FALSE(parsed.schema_mismatch_mode.has_value()); +} + +TEST_F(ExportPartitionManifestBackCompatTest, SchemaMismatchModeRoundTripsForEveryValue) +{ + for (const auto value : magic_enum::enum_values()) + { + auto manifest = makeValidManifest(); + manifest.schema_mismatch_mode = value; + + auto parsed = ExportReplicatedMergeTreePartitionManifest::fromJsonString(manifest.toJsonString()); + + ASSERT_TRUE(parsed.schema_mismatch_mode.has_value()) << "value=" << magic_enum::enum_name(value); + EXPECT_EQ(*parsed.schema_mismatch_mode, value) << "value=" << magic_enum::enum_name(value); + } +} + +TEST_F(ExportPartitionManifestBackCompatTest, MissingSchemaMismatchModeFallsBackToStrictInWorkerContext) +{ + auto manifest = makeValidManifest(); + ASSERT_FALSE(manifest.schema_mismatch_mode.has_value()); + + auto worker_context = ExportPartitionUtils::getContextCopyWithTaskSettings(getContext().context, manifest); + + EXPECT_EQ( + worker_context->getSettingsRef()[Setting::export_merge_tree_part_schema_mismatch_mode].value, + MergeTreePartExportSchemaMismatchMode::strict); +} + +TEST_F(ExportPartitionManifestBackCompatTest, SchemaMismatchModeAppliedToWorkerContextForEveryValue) +{ + for (const auto value : magic_enum::enum_values()) + { + auto manifest = makeValidManifest(); + manifest.schema_mismatch_mode = value; + + auto worker_context = ExportPartitionUtils::getContextCopyWithTaskSettings(getContext().context, manifest); + + EXPECT_EQ( + worker_context->getSettingsRef()[Setting::export_merge_tree_part_schema_mismatch_mode].value, + value) << "value=" << magic_enum::enum_name(value); + } +} + +} diff --git a/src/Storages/ObjectStorage/Azure/Configuration.cpp b/src/Storages/ObjectStorage/Azure/Configuration.cpp index 4ca0c16780bd..abb2f3d13ece 100644 --- a/src/Storages/ObjectStorage/Azure/Configuration.cpp +++ b/src/Storages/ObjectStorage/Azure/Configuration.cpp @@ -65,6 +65,7 @@ const std::unordered_set optional_configuration_keys = { "partition_columns_in_data_file", "client_id", "tenant_id", + "storage_type", }; void StorageAzureConfiguration::check(ContextPtr context) @@ -212,10 +213,6 @@ void AzureStorageParsedArguments::fromNamedCollection(const NamedCollection & co String connection_url; String container_name; - std::optional account_name; - std::optional account_key; - std::optional client_id; - std::optional tenant_id; if (collection.has("connection_string")) connection_url = collection.get("connection_string"); @@ -408,16 +405,10 @@ void AzureStorageParsedArguments::fromAST(ASTs & engine_args, ContextPtr context std::unordered_map engine_args_to_idx; - String connection_url = checkAndGetLiteralArgument(engine_args[0], "connection_string/storage_account_url"); String container_name = checkAndGetLiteralArgument(engine_args[1], "container"); blob_path = checkAndGetLiteralArgument(engine_args[2], "blobpath"); - std::optional account_name; - std::optional account_key; - std::optional client_id; - std::optional tenant_id; - collectCredentials(extra_credentials, client_id, tenant_id, context); auto is_format_arg = [] (const std::string & s) -> bool @@ -851,6 +842,26 @@ void StorageAzureConfiguration::initializeFromParsedArguments(const AzureStorage StorageObjectStorageConfiguration::initializeFromParsedArguments(parsed_arguments); blob_path = parsed_arguments.blob_path; connection_params = parsed_arguments.connection_params; + account_name = parsed_arguments.account_name; + account_key = parsed_arguments.account_key; + client_id = parsed_arguments.client_id; + tenant_id = parsed_arguments.tenant_id; +} + +ASTPtr StorageAzureConfiguration::createArgsWithAccessData() const +{ + auto arguments = make_intrusive(); + + arguments->children.push_back(make_intrusive(connection_params.endpoint.storage_account_url)); + arguments->children.push_back(make_intrusive(connection_params.endpoint.container_name)); + arguments->children.push_back(make_intrusive(blob_path.path)); + if (account_name && account_key) + { + arguments->children.push_back(make_intrusive(*account_name)); + arguments->children.push_back(make_intrusive(*account_key)); + } + + return arguments; } void StorageAzureConfiguration::addStructureAndFormatToArgsIfNeeded( @@ -858,13 +869,13 @@ void StorageAzureConfiguration::addStructureAndFormatToArgsIfNeeded( { if (disk) { - if (format == "auto") + if (getFormat() == "auto") { ASTs format_equal_func_args = {make_intrusive("format"), make_intrusive(format_)}; auto format_equal_func = makeASTFunction("equals", std::move(format_equal_func_args)); args.push_back(format_equal_func); } - if (structure == "auto") + if (getStructure() == "auto") { ASTs structure_equal_func_args = {make_intrusive("structure"), make_intrusive(structure_)}; auto structure_equal_func = makeASTFunction("equals", std::move(structure_equal_func_args)); diff --git a/src/Storages/ObjectStorage/Azure/Configuration.h b/src/Storages/ObjectStorage/Azure/Configuration.h index c9d45d0bdb8a..0a97cf78f8f8 100644 --- a/src/Storages/ObjectStorage/Azure/Configuration.h +++ b/src/Storages/ObjectStorage/Azure/Configuration.h @@ -77,6 +77,11 @@ struct AzureStorageParsedArguments : private StorageParsedArguments Path blob_path; AzureBlobStorage::ConnectionParams connection_params; + + std::optional account_name; + std::optional account_key; + std::optional client_id; + std::optional tenant_id; }; class StorageAzureConfiguration : public StorageObjectStorageConfiguration @@ -141,6 +146,7 @@ class StorageAzureConfiguration : public StorageObjectStorageConfiguration onelake_use_blob_endpoint = use_blob_endpoint_; is_onelake = true; } + ASTPtr createArgsWithAccessData() const override; protected: void fromDisk(const String & disk_name, ASTs & args, ContextPtr context, bool with_structure) override; @@ -152,7 +158,11 @@ class StorageAzureConfiguration : public StorageObjectStorageConfiguration Path blob_path; Paths blobs_paths; AzureBlobStorage::ConnectionParams connection_params; - DiskPtr disk; + + std::optional account_name; + std::optional account_key; + std::optional client_id; + std::optional tenant_id; String onelake_client_id; String onelake_client_secret; @@ -164,8 +174,11 @@ class StorageAzureConfiguration : public StorageObjectStorageConfiguration bool onelake_use_blob_endpoint = true; bool is_onelake = false; + DiskPtr disk; + void initializeFromParsedArguments(const AzureStorageParsedArguments & parsed_arguments); }; + } #endif diff --git a/src/Storages/ObjectStorage/DataLakes/Common/AvroForIcebergDeserializer.cpp b/src/Storages/ObjectStorage/DataLakes/Common/AvroForIcebergDeserializer.cpp index fe983f88a825..a9534abdd383 100644 --- a/src/Storages/ObjectStorage/DataLakes/Common/AvroForIcebergDeserializer.cpp +++ b/src/Storages/ObjectStorage/DataLakes/Common/AvroForIcebergDeserializer.cpp @@ -17,6 +17,7 @@ #include #include #include +#include #include namespace DB::ErrorCodes @@ -25,6 +26,12 @@ namespace DB::ErrorCodes extern const int INCORRECT_DATA; } +namespace ProfileEvents +{ + extern const Event IcebergAvroFileParsing; + extern const Event IcebergAvroFileParsingMicroseconds; +} + namespace DB::Iceberg { @@ -37,6 +44,9 @@ AvroForIcebergDeserializer::AvroForIcebergDeserializer( try : manifest_file_path(manifest_file_path_) { + ProfileEvents::increment(ProfileEvents::IcebergAvroFileParsing); + ProfileEventTimeIncrement watch(ProfileEvents::IcebergAvroFileParsingMicroseconds); + auto buffer = std::move(buffer_); auto manifest_file_reader = std::make_unique(std::make_unique(*buffer), MAX_AVRO_SCHEMA_DEPTH); diff --git a/src/Storages/ObjectStorage/DataLakes/DataLakeConfiguration.h b/src/Storages/ObjectStorage/DataLakes/DataLakeConfiguration.h index b122defb4d7f..03a34c420412 100644 --- a/src/Storages/ObjectStorage/DataLakes/DataLakeConfiguration.h +++ b/src/Storages/ObjectStorage/DataLakes/DataLakeConfiguration.h @@ -7,6 +7,7 @@ #include #include +#include #include #include #include @@ -17,11 +18,15 @@ #include #include #include -#include +#include #include #include #include #include +#include +#include +#include +#include #include #include #include @@ -55,21 +60,21 @@ namespace ErrorCodes namespace DataLakeStorageSetting { - extern DataLakeStorageSettingsDatabaseDataLakeCatalogType storage_catalog_type; - extern DataLakeStorageSettingsString object_storage_endpoint; - extern DataLakeStorageSettingsString storage_aws_access_key_id; - extern DataLakeStorageSettingsString storage_aws_secret_access_key; - extern DataLakeStorageSettingsString storage_region; - extern DataLakeStorageSettingsString storage_aws_role_arn; - extern DataLakeStorageSettingsString storage_aws_role_session_name; - extern DataLakeStorageSettingsString storage_catalog_url; - extern DataLakeStorageSettingsString storage_warehouse; - extern DataLakeStorageSettingsString storage_catalog_credential; - - extern DataLakeStorageSettingsString storage_auth_scope; - extern DataLakeStorageSettingsString storage_auth_header; - extern DataLakeStorageSettingsString storage_oauth_server_uri; - extern DataLakeStorageSettingsBool storage_oauth_server_use_request_body; + extern const DataLakeStorageSettingsDatabaseDataLakeCatalogType storage_catalog_type; + extern const DataLakeStorageSettingsString object_storage_endpoint; + extern const DataLakeStorageSettingsString storage_aws_access_key_id; + extern const DataLakeStorageSettingsString storage_aws_secret_access_key; + extern const DataLakeStorageSettingsString storage_region; + extern const DataLakeStorageSettingsString storage_aws_role_arn; + extern const DataLakeStorageSettingsString storage_aws_role_session_name; + extern const DataLakeStorageSettingsString storage_catalog_url; + extern const DataLakeStorageSettingsString storage_warehouse; + extern const DataLakeStorageSettingsString storage_catalog_credential; + extern const DataLakeStorageSettingsString storage_auth_scope; + extern const DataLakeStorageSettingsString storage_auth_header; + extern const DataLakeStorageSettingsString storage_oauth_server_uri; + extern const DataLakeStorageSettingsBool storage_oauth_server_use_request_body; + extern const DataLakeStorageSettingsString iceberg_metadata_file_path; } struct FormatParserSharedResources; @@ -78,11 +83,17 @@ using FormatParserSharedResourcesPtr = std::shared_ptr concept StorageConfiguration = std::derived_from; -template +template class DataLakeConfiguration : public BaseStorageConfiguration, public std::enable_shared_from_this { public: - explicit DataLakeConfiguration(DataLakeStorageSettingsPtr settings_) : settings(settings_) {} + DataLakeConfiguration() {} + + explicit DataLakeConfiguration( + DataLakeStorageSettingsPtr settings_, + std::optional catalog_namespaces_ = std::nullopt) + : settings(settings_) + , catalog_namespaces(catalog_namespaces_.value_or("*")) {} bool isDataLakeConfiguration() const override { return true; } @@ -107,6 +118,7 @@ class DataLakeConfiguration : public BaseStorageConfiguration, public std::enabl return StorageObjectStorageConfiguration::Path(result.ends_with('/') ? result : result + "/"); } + void setRawPath(const StorageObjectStorageConfiguration::Path & path) override { BaseStorageConfiguration::setRawPath(path); } void update(ObjectStoragePtr object_storage, ContextPtr local_context) override { @@ -434,9 +446,48 @@ class DataLakeConfiguration : public BaseStorageConfiguration, public std::enabl return true; } + bool isClusterSupported() const override { return is_cluster_supported; } + + ASTPtr createArgsWithAccessData() const override + { + auto res = BaseStorageConfiguration::createArgsWithAccessData(); + + auto iceberg_metadata_file_path = (*settings)[DataLakeStorageSetting::iceberg_metadata_file_path]; + + if (iceberg_metadata_file_path.changed) + { + auto * arguments = res->template as(); + if (!arguments) + throw Exception(ErrorCodes::LOGICAL_ERROR, "Arguments are not an expression list"); + + bool has_settings = false; + + for (auto & arg : arguments->children) + { + if (auto * settings_ast = arg->template as()) + { + has_settings = true; + settings_ast->changes.setSetting("iceberg_metadata_file_path", iceberg_metadata_file_path.value); + break; + } + } + + if (!has_settings) + { + boost::intrusive_ptr settings_ast = make_intrusive(); + settings_ast->is_standalone = false; + settings_ast->changes.setSetting("iceberg_metadata_file_path", iceberg_metadata_file_path.value); + arguments->children.push_back(settings_ast); + } + } + + return res; + } + private: const DataLakeStorageSettingsPtr settings; ObjectStoragePtr ready_object_storage; + std::string catalog_namespaces; mutable std::mutex metadata_mutex; /// Readers take a copy of this pointer under the lock and use that copy, so a concurrent /// republish in update() cannot destroy the object they are still calling into. @@ -463,6 +514,7 @@ class DataLakeConfiguration : public BaseStorageConfiguration, public std::enabl std::shared_ptr getMetadata() const { + BaseStorageConfiguration::assertInitialized(); auto metadata = tryGetMetadata(); if (!metadata) throw Exception(ErrorCodes::LOGICAL_ERROR, "Metadata is not initialized"); @@ -505,18 +557,399 @@ using StorageS3IcebergConfiguration = DataLakeConfiguration; #endif -#if USE_AZURE_BLOB_STORAGE +# if USE_AZURE_BLOB_STORAGE using StorageAzureIcebergConfiguration = DataLakeConfiguration; using StorageAzurePaimonConfiguration = DataLakeConfiguration; #endif -#if USE_HDFS +# if USE_HDFS using StorageHDFSIcebergConfiguration = DataLakeConfiguration; using StorageHDFSPaimonConfiguration = DataLakeConfiguration; #endif using StorageLocalIcebergConfiguration = DataLakeConfiguration; -using StorageLocalPaimonConfiguration = DataLakeConfiguration; +using StorageLocalPaimonConfiguration = DataLakeConfiguration; + +/// Class detects storage type by `storage_type` parameter if exists +/// and uses appropriate implementation - S3, Azure, HDFS or Local +class StorageIcebergConfiguration : public StorageObjectStorageConfiguration, public std::enable_shared_from_this +{ + friend class StorageObjectStorageConfiguration; + +public: + StorageIcebergConfiguration() {} + + explicit StorageIcebergConfiguration(DataLakeStorageSettingsPtr settings_) : settings(settings_) {} + + void initialize( + ASTs & engine_args, + ContextPtr local_context, + bool with_table_structure, + const StorageID * table_id = nullptr) override + { + createDynamicConfiguration(engine_args, local_context); + getImpl().initialize(engine_args, local_context, with_table_structure, table_id); + } + + ObjectStorageType getType() const override { return getImpl().getType(); } + + std::string getTypeName() const override { return getImpl().getTypeName(); } + std::string getEngineName() const override { return getImpl().getEngineName(); } + std::string getNamespaceType() const override { return getImpl().getNamespaceType(); } + + Path getRawPath() const override { return getImpl().getRawPath(); } + void setRawPath(const Path & path) override { getImpl().setRawPath(path); } + const String & getRawURI() const override { return getImpl().getRawURI(); } + const Path & getPathForRead() const override { return getImpl().getPathForRead(); } + Path getPathForWrite(const std::string & partition_id) const override { return getImpl().getPathForWrite(partition_id); } + + void setPathForRead(const Path & path) override { getImpl().setPathForRead(path); } + + const Paths & getPaths() const override { return getImpl().getPaths(); } + void setPaths(const Paths & paths) override { getImpl().setPaths(paths); } + + String getDataSourceDescription() const override { return getImpl().getDataSourceDescription(); } + String getNamespace() const override { return getImpl().getNamespace(); } + + StorageObjectStorageQuerySettings getQuerySettings(const ContextPtr & context) const override + { return getImpl().getQuerySettings(context); } + + void addStructureAndFormatToArgsIfNeeded( + ASTs & args, const String & structure_, const String & format_, ContextPtr context, bool with_structure) override + { getImpl().addStructureAndFormatToArgsIfNeeded(args, structure_, format_, context, with_structure); } + + bool isNamespaceWithGlobs() const override { return getImpl().isNamespaceWithGlobs(); } + + bool isArchive() const override { return getImpl().isArchive(); } + bool isPathInArchiveWithGlobs() const override { return getImpl().isPathInArchiveWithGlobs(); } + std::string getPathInArchive() const override { return getImpl().getPathInArchive(); } + + void check(ContextPtr context) override { getImpl().check(context); } + void validateNamespace(const String & name) const override { getImpl().validateNamespace(name); } + + ObjectStoragePtr createObjectStorage(ContextPtr context, bool is_readonly, CredentialsConfigurationCallback refresh_credentials_callback) override + { return getImpl().createObjectStorage(context, is_readonly, refresh_credentials_callback); } + bool isStaticConfiguration() const override { return getImpl().isStaticConfiguration(); } + + bool isDataLakeConfiguration() const override { return getImpl().isDataLakeConfiguration(); } + + bool supportsTotalRows(ContextPtr context, ObjectStorageType storage_type) const override { return getImpl().supportsTotalRows(context, storage_type); } + std::optional totalRows(ContextPtr context) override { return getImpl().totalRows(context); } + bool supportsTotalBytes(ContextPtr context, ObjectStorageType storage_type) const override { return getImpl().supportsTotalBytes(context, storage_type); } + std::optional totalBytes(ContextPtr context) override { return getImpl().totalBytes(context); } + bool isDataSortedBySortingKey(StorageMetadataPtr storage_metadata, ContextPtr context) const override + { return getImpl().isDataSortedBySortingKey(storage_metadata, context); } + + std::shared_ptr getExternalMetadata() override { return getImpl().getExternalMetadata(); } + + std::shared_ptr getInitialSchemaByPath(ContextPtr context, ObjectInfoPtr object_info) const override + { return getImpl().getInitialSchemaByPath(context, object_info); } + + std::shared_ptr getSchemaTransformer(ContextPtr context, ObjectInfoPtr object_info) const override + { return getImpl().getSchemaTransformer(context, object_info); } + + void modifyFormatSettings(FormatSettings & settings_, const Context & context) const override + { getImpl().modifyFormatSettings(settings_, context); } + + void addDeleteTransformers( + ObjectInfoPtr object_info, + QueryPipelineBuilder & builder, + const std::optional & format_settings, + FormatParserSharedResourcesPtr parser_shared_resources, + ContextPtr local_context) const override + { getImpl().addDeleteTransformers(object_info, builder, format_settings, parser_shared_resources, local_context); } + + ReadFromFormatInfo prepareReadingFromFormat( + ObjectStoragePtr object_storage, + const Strings & requested_columns, + const StorageSnapshotPtr & storage_snapshot, + bool supports_subset_of_columns, + bool supports_tuple_elements, + ContextPtr local_context, + const PrepareReadingFromFormatHiveParams & hive_parameters) override + { + return getImpl().prepareReadingFromFormat( + object_storage, + requested_columns, + storage_snapshot, + supports_subset_of_columns, + supports_tuple_elements, + local_context, + hive_parameters); + } + + void setSchemaHash(const String & hash) override { getImpl().setSchemaHash(hash); } + + void initPartitionStrategy(ASTPtr partition_by, const ColumnsDescription & columns, ContextPtr context) override + { getImpl().initPartitionStrategy(partition_by, columns, context); } + + std::optional getTableStateSnapshot(ContextPtr local_context) const override { return getImpl().getTableStateSnapshot(local_context); } + std::unique_ptr buildStorageMetadataFromState(const DataLakeTableStateSnapshot & state, ContextPtr local_context) const override + { return getImpl().buildStorageMetadataFromState(state, local_context); } + bool shouldReloadSchemaForConsistency(ContextPtr local_context) const override { return getImpl().shouldReloadSchemaForConsistency(local_context); } + std::optional tryGetTableStructureFromMetadata(ContextPtr local_context) const override + { return getImpl().tryGetTableStructureFromMetadata(local_context); } + + bool supportsFileIterator() const override { return getImpl().supportsFileIterator(); } + bool supportsParallelInsert() const override { return getImpl().supportsParallelInsert(); } + bool supportsWrites() const override { return getImpl().supportsWrites(); } + + bool supportsPartialPathPrefix() const override { return getImpl().supportsPartialPathPrefix(); } + + ObjectIterator iterate( + const ActionsDAG * filter_dag, + IDataLakeMetadata::FileProgressCallback callback, + size_t list_batch_size, + StorageMetadataPtr storage_metadata, + ContextPtr context) override + { + return getImpl().iterate(filter_dag, callback, list_batch_size, storage_metadata, context); + } + + void update( + ObjectStoragePtr object_storage_ptr, + ContextPtr context) override + { + getImpl().update(object_storage_ptr, context); + } + void lazyInitializeIfNeeded(ObjectStoragePtr object_storage, ContextPtr local_context) override + { return getImpl().lazyInitializeIfNeeded(object_storage, local_context); } + + void create( + ObjectStoragePtr object_storage, + ContextPtr local_context, + const std::optional & columns, + ASTPtr partition_by, + ASTPtr order_by, + bool if_not_exists, + std::shared_ptr catalog, + const StorageID & table_id_) override + { + getImpl().create(object_storage, local_context, columns, partition_by, order_by, if_not_exists, catalog, table_id_); + } + + SinkToStoragePtr write( + SharedHeader sample_block, + const StorageID & table_id, + ObjectStoragePtr object_storage, + const std::optional & format_settings, + ContextPtr context, + std::shared_ptr catalog) override + { + return getImpl().write(sample_block, table_id, object_storage, format_settings, context, catalog); + } + + bool supportsDelete() const override { return getImpl().supportsDelete(); } + void mutate(const MutationCommands & commands, + ContextPtr context, + StoragePtr storage_ptr, + const StorageID & storage_id, + StorageMetadataPtr metadata_snapshot, + std::shared_ptr catalog, + const std::optional & format_settings) override + { + getImpl().mutate(commands, context, storage_ptr, storage_id, metadata_snapshot, catalog, format_settings); + } + void checkMutationIsPossible(ObjectStoragePtr object_storage, ContextPtr context, const MutationCommands & commands) override + { getImpl().checkMutationIsPossible(object_storage, context, commands); } + + void checkAlterIsPossible(ObjectStoragePtr object_storage, ContextPtr context, const AlterCommands & commands) override + { getImpl().checkAlterIsPossible(object_storage, context, commands); } + + void alter( + ObjectStoragePtr object_storage, + const AlterCommands & params, + ContextPtr context, + const StorageID & storage_id, + std::shared_ptr catalog) override + { + getImpl().alter(object_storage, params, context, storage_id, catalog); + } + + const DataLakeStorageSettings & getDataLakeSettings() const override { return getImpl().getDataLakeSettings(); } + + ASTPtr createArgsWithAccessData() const override + { + return getImpl().createArgsWithAccessData(); + } + + void fromNamedCollection(const NamedCollection & collection, ContextPtr context) override + { getImpl().fromNamedCollection(collection, context); } + void fromAST(ASTs & args, ContextPtr context, bool with_structure) override + { getImpl().fromAST(args, context, with_structure); } + void fromDisk(const String & disk_name, ASTs & args, ContextPtr context, bool with_structure) override + { getImpl().fromDisk(disk_name, args, context, with_structure); } + + /// Find storage_type argument and remove it from args if exists. + /// Return storage type. + ObjectStorageType extractDynamicStorageType(ASTs & args, ContextPtr context, ASTPtr * type_arg, bool cluster_name_first) const override + { + static const auto * const storage_type_name = "storage_type"; + + { + auto args_copy = args; + if (cluster_name_first) + { + // Remove cluster name from args to avoid confusing cluster name and named collection name + args_copy.erase(args_copy.begin()); + } + + if (auto named_collection = tryGetNamedCollectionWithOverrides(args_copy, context)) + { + if (named_collection->has(storage_type_name)) + { + return objectStorageTypeFromString(named_collection->get(storage_type_name)); + } + } + } + + auto type_it = args.end(); + + /// S3 by default for backward compatibility + /// Iceberg without storage_type == IcebergS3 + ObjectStorageType type = ObjectStorageType::S3; + + for (auto arg_it = args.begin(); arg_it != args.end(); ++arg_it) + { + const auto * type_ast_function = (*arg_it)->as(); + + if (type_ast_function && type_ast_function->name == "equals" + && type_ast_function->arguments && type_ast_function->arguments->children.size() == 2) + { + auto * name = type_ast_function->arguments->children[0]->as(); + + if (name && name->name() == storage_type_name) + { + if (type_it != args.end()) + { + throw Exception( + ErrorCodes::BAD_ARGUMENTS, + "DataLake can have only one key-value argument: storage_type='type'."); + } + + auto * value = type_ast_function->arguments->children[1]->as(); + + if (!value) + { + throw Exception( + ErrorCodes::BAD_ARGUMENTS, + "DataLake parameter 'storage_type' has wrong type, string literal expected."); + } + + if (value->value.getType() != Field::Types::String) + { + throw Exception( + ErrorCodes::BAD_ARGUMENTS, + "DataLake parameter 'storage_type' has wrong value type, string expected."); + } + + type = objectStorageTypeFromString(value->value.safeGet()); + + type_it = arg_it; + } + } + } + + if (type_it != args.end()) + { + if (type_arg) + *type_arg = *type_it; + args.erase(type_it); + } + + return type; + } + + const String & getFormat() const override { return getImpl().getFormat(); } + const String & getCompressionMethod() const override { return getImpl().getCompressionMethod(); } + const String & getStructure() const override { return getImpl().getStructure(); } + + PartitionStrategyFactory::StrategyType getPartitionStrategyType() const override { return getImpl().getPartitionStrategyType(); } + bool getPartitionColumnsInDataFile() const override { return getImpl().getPartitionColumnsInDataFile(); } + std::shared_ptr getPartitionStrategy() const override { return getImpl().getPartitionStrategy(); } + + void setFormat(const String & format_) override { getImpl().setFormat(format_); } + void setCompressionMethod(const String & compression_method_) override { getImpl().setCompressionMethod(compression_method_); } + void setStructure(const String & structure_) override { getImpl().setStructure(structure_); } + + void setPartitionStrategyType(PartitionStrategyFactory::StrategyType partition_strategy_type_) override + { getImpl().setPartitionStrategyType(partition_strategy_type_); } + void setPartitionColumnsInDataFile(bool partition_columns_in_data_file_) override + { getImpl().setPartitionColumnsInDataFile(partition_columns_in_data_file_); } + void setPartitionStrategy(const std::shared_ptr & partition_strategy_) override + { getImpl().setPartitionStrategy(partition_strategy_); } + + void assertInitialized() const override { getImpl().assertInitialized(); } + + ColumnMapperPtr getColumnMapperForObject(ObjectInfoPtr obj) const override { return getImpl().getColumnMapperForObject(obj); } + + ColumnMapperPtr getColumnMapperForCurrentSchema(StorageMetadataPtr storage_metadata_snapshot, ContextPtr context) const override + { return getImpl().getColumnMapperForCurrentSchema(storage_metadata_snapshot, context); } + + std::shared_ptr getCatalog(ContextPtr context, const StorageID & table_id) const override + { return getImpl().getCatalog(context, table_id); } + + bool optimize(ObjectStoragePtr object_storage, const StorageMetadataPtr & metadata_snapshot, ContextPtr context, const std::optional & format_settings) override + { return getImpl().optimize(object_storage, metadata_snapshot, context, format_settings); } + + bool supportsPrewhere() const override { return getImpl().supportsPrewhere(); } + + void drop(ContextPtr context) override { getImpl().drop(context); } + +protected: + void createDynamicConfiguration(ASTs & args, ContextPtr context) + { + ObjectStorageType type = extractDynamicStorageType(args, context, nullptr, false); + createDynamicStorage(type); + } + +private: + inline StorageObjectStorageConfiguration & getImpl() const + { + if (!impl) + throw Exception(ErrorCodes::LOGICAL_ERROR, "Dynamic DataLake storage not initialized"); + + return *impl; + } + + void createDynamicStorage(ObjectStorageType type) + { + if (impl) + { + if (impl->getType() == type) + return; + + throw Exception(ErrorCodes::LOGICAL_ERROR, "Can't change datalake engine storage"); + } + + switch (type) + { +# if USE_AWS_S3 + case ObjectStorageType::S3: + impl = std::make_unique(settings); + break; +# endif +# if USE_AZURE_BLOB_STORAGE + case ObjectStorageType::Azure: + impl = std::make_unique(settings); + break; +# endif +# if USE_HDFS + case ObjectStorageType::HDFS: + impl = std::make_unique(settings); + break; +# endif + case ObjectStorageType::Local: + impl = std::make_unique(settings); + break; + default: + throw Exception(ErrorCodes::LOGICAL_ERROR, "Unsuported DataLake storage {}", type); + } + } + + StorageObjectStorageConfigurationPtr impl; + DataLakeStorageSettingsPtr settings; +}; #endif #if USE_PARQUET @@ -528,7 +961,7 @@ using StorageS3DeltaLakeConfiguration = DataLakeConfiguration; #endif -using StorageLocalDeltaLakeConfiguration = DataLakeConfiguration; +using StorageLocalDeltaLakeConfiguration = DataLakeConfiguration; #endif diff --git a/src/Storages/ObjectStorage/DataLakes/DataLakeStorageSettings.h b/src/Storages/ObjectStorage/DataLakes/DataLakeStorageSettings.h index 6738c3252919..31e1d412450e 100644 --- a/src/Storages/ObjectStorage/DataLakes/DataLakeStorageSettings.h +++ b/src/Storages/ObjectStorage/DataLakes/DataLakeStorageSettings.h @@ -66,6 +66,9 @@ The period in milliseconds to asynchronously prefetch the latest metadata snapsh )", 0) \ DECLARE(Bool, iceberg_use_version_hint, false, R"( Get latest metadata path from version-hint.text file. +)", 0) \ + DECLARE(String, object_storage_cluster, "", R"( +Cluster for distributed requests )", 0) \ DECLARE(NonZeroUInt64, iceberg_format_version, 2, R"( Metadata format version. diff --git a/src/Storages/ObjectStorage/DataLakes/DeltaLakeMetadataDeltaKernel.cpp b/src/Storages/ObjectStorage/DataLakes/DeltaLakeMetadataDeltaKernel.cpp index 47bbc604ad4b..e52e23b667d5 100644 --- a/src/Storages/ObjectStorage/DataLakes/DeltaLakeMetadataDeltaKernel.cpp +++ b/src/Storages/ObjectStorage/DataLakes/DeltaLakeMetadataDeltaKernel.cpp @@ -118,7 +118,7 @@ DeltaLakeMetadataDeltaKernel::DeltaLakeMetadataDeltaKernel( : log(getLogger("DeltaLakeMetadata")) , kernel_helper(DB::getKernelHelper(configuration_.lock(), object_storage_)) , object_storage(object_storage_) - , format_name(configuration_.lock()->format) + , format_name(configuration_.lock()->getFormat()) /// TODO: Supports size limit, not just elements limit. /// TODO: Support weight function (by default weight = 1 for all elements). /// TODO: Add a setting for cache size. @@ -654,8 +654,8 @@ SinkToStoragePtr DeltaLakeMetadataDeltaKernel::write( context, sample_block, format_settings, - configuration->format, - configuration->compression_method); + configuration->getFormat(), + configuration->getCompressionMethod()); } return std::make_shared( @@ -665,8 +665,8 @@ SinkToStoragePtr DeltaLakeMetadataDeltaKernel::write( context, sample_block, format_settings, - configuration->format, - configuration->compression_method); + configuration->getFormat(), + configuration->getCompressionMethod()); } void DeltaLakeMetadataDeltaKernel::logMetadataFiles(ContextPtr context) const diff --git a/src/Storages/ObjectStorage/DataLakes/HudiMetadata.cpp b/src/Storages/ObjectStorage/DataLakes/HudiMetadata.cpp index aeb4f9989dd2..c0f527d63d87 100644 --- a/src/Storages/ObjectStorage/DataLakes/HudiMetadata.cpp +++ b/src/Storages/ObjectStorage/DataLakes/HudiMetadata.cpp @@ -92,11 +92,11 @@ HudiMetadata::HudiMetadata(ObjectStoragePtr object_storage_, StorageObjectStorag : WithContext(context_) , object_storage(object_storage_) , table_path(configuration_->getPathForRead().path) - , format(configuration_->format) + , format(configuration_->getFormat()) { } -Strings HudiMetadata::getDataFiles(const ActionsDAG *) const +Strings HudiMetadata::getDataFiles() const { if (data_files.empty()) data_files = getDataFilesImpl(); @@ -104,13 +104,13 @@ Strings HudiMetadata::getDataFiles(const ActionsDAG *) const } ObjectIterator HudiMetadata::iterate( - const ActionsDAG * filter_dag, + const ActionsDAG * /* filter_dag */, FileProgressCallback callback, size_t /* list_batch_size */, StorageMetadataPtr /* storage_metadata_snapshot*/, ContextPtr /* context */) const { - return createKeysIterator(getDataFiles(filter_dag), object_storage, callback); + return createKeysIterator(getDataFiles(), object_storage, callback); } } diff --git a/src/Storages/ObjectStorage/DataLakes/HudiMetadata.h b/src/Storages/ObjectStorage/DataLakes/HudiMetadata.h index d2700f405fc8..b941a84a3747 100644 --- a/src/Storages/ObjectStorage/DataLakes/HudiMetadata.h +++ b/src/Storages/ObjectStorage/DataLakes/HudiMetadata.h @@ -65,7 +65,7 @@ class HudiMetadata final : public IDataLakeMetadata, private WithContext mutable Strings data_files; Strings getDataFilesImpl() const; - Strings getDataFiles(const ActionsDAG * filter_dag) const; + Strings getDataFiles() const; }; } diff --git a/src/Storages/ObjectStorage/DataLakes/IDataLakeMetadata.cpp b/src/Storages/ObjectStorage/DataLakes/IDataLakeMetadata.cpp index 6eed86d81e7d..e553907b4a04 100644 --- a/src/Storages/ObjectStorage/DataLakes/IDataLakeMetadata.cpp +++ b/src/Storages/ObjectStorage/DataLakes/IDataLakeMetadata.cpp @@ -1,9 +1,21 @@ #include #include +#include +#include +#include +#include +#include +#include +#include namespace DB { +namespace ErrorCodes +{ + extern const int INCORRECT_DATA; +}; + namespace { @@ -87,4 +99,229 @@ ReadFromFormatInfo IDataLakeMetadata::prepareReadingFromFormat( return DB::prepareReadingFromFormat(requested_columns, storage_snapshot, context, supports_subset_of_columns, supports_tuple_elements); } +DataFileMetaInfo::DataFileMetaInfo( + const Iceberg::IcebergSchemaProcessor & schema_processor, + Int32 table_schema_id, + Int32 file_schema_id, + const std::unordered_map & columns_info_, + const std::unordered_map> & value_bounds_) +{ +#if USE_AVRO + std::vector column_ids; + for (const auto & column : columns_info_) + column_ids.push_back(column.first); + + /// Names are resolved via the table schema so that the resulting `columns_info` + /// map is keyed by the current column names that callers know about. + auto table_name_and_types = schema_processor.tryGetFieldsCharacteristics(table_schema_id, column_ids); + std::unordered_map name_by_index; + for (const auto & name_and_type : table_name_and_types) + { + const auto name = name_and_type.getNameInStorage(); + auto index = schema_processor.tryGetColumnIDByName(table_schema_id, name); + if (index.has_value()) + name_by_index[index.value()] = name; + } + + /// Types come from the file's schema because `value_bounds_` are encoded with + /// that schema's column types — see Iceberg single-value serialization spec. + std::unordered_map type_by_index; + auto file_name_and_types = schema_processor.tryGetFieldsCharacteristics(file_schema_id, column_ids); + for (const auto & name_and_type : file_name_and_types) + { + auto index = schema_processor.tryGetColumnIDByName(file_schema_id, name_and_type.getNameInStorage()); + if (index.has_value()) + type_by_index[index.value()] = name_and_type.type; + } + + for (const auto & column : columns_info_) + { + auto i_name = name_by_index.find(column.first); + if (i_name == name_by_index.end()) + continue; + + std::optional hyperrectangle; + + auto i_bounds = value_bounds_.find(column.first); + auto i_type = type_by_index.find(column.first); + if (i_bounds != value_bounds_.end() && i_type != type_by_index.end()) + { + const auto & type = i_type->second; + if (const auto type_id = type->getTypeId(); + type_id != TypeIndex::Tuple && type_id != TypeIndex::Map && type_id != TypeIndex::Array) + { + String left_str; + String right_str; + if (i_bounds->second.first.tryGet(left_str) && i_bounds->second.second.tryGet(right_str)) + { + auto left = Iceberg::deserializeFieldFromBinaryRepr(left_str, type, true); + auto right = Iceberg::deserializeFieldFromBinaryRepr(right_str, type, false); + if (left && right) + hyperrectangle = DB::Range(*left, true, *right, true); + } + } + } + + columns_info[i_name->second] = {column.second.rows_count, column.second.nulls_count, hyperrectangle}; + } +#else + (void)schema_processor; + (void)table_schema_id; + (void)file_schema_id; + (void)columns_info_; + (void)value_bounds_; +#endif +} + +DataFileMetaInfo::DataFileMetaInfo(Poco::JSON::Object::Ptr file_info) +{ + if (!file_info) + return; + + auto log = getLogger("DataFileMetaInfo"); + + if (file_info->has("columns")) + { + auto columns = file_info->getArray("columns"); + for (size_t i = 0; i < columns->size(); ++i) + { + auto column = columns->getObject(static_cast(i)); + + std::string name; + if (column->has("name")) + name = column->get("name").toString(); + else + { + LOG_WARNING(log, "Can't read column name, ignored"); + continue; + } + + DB::DataFileMetaInfo::ColumnInfo column_info; + if (column->has("rows")) + column_info.rows_count = column->get("rows"); + if (column->has("nulls")) + column_info.nulls_count = column->get("nulls"); + if (column->has("range")) + { + Range range(""); + std::string r = column->get("range"); + try + { + range.deserialize(r, /*base64*/ true); + column_info.hyperrectangle = std::move(range); + } + catch (const Exception & e) + { + LOG_WARNING(log, "Can't read range for column {}, range '{}' ignored, error: {}", name, r, e.what()); + } + } + + columns_info[name] = column_info; + } + } +} + +Poco::JSON::Object::Ptr DataFileMetaInfo::toJson() const +{ + Poco::JSON::Object::Ptr file_info = new Poco::JSON::Object(); + + if (!columns_info.empty()) + { + Poco::JSON::Array::Ptr columns = new Poco::JSON::Array(); + + for (const auto & column : columns_info) + { + Poco::JSON::Object::Ptr column_info = new Poco::JSON::Object(); + column_info->set("name", column.first); + if (column.second.rows_count.has_value()) + column_info->set("rows", column.second.rows_count.value()); + if (column.second.nulls_count.has_value()) + column_info->set("nulls", column.second.nulls_count.value()); + if (column.second.hyperrectangle.has_value()) + column_info->set("range", column.second.hyperrectangle.value().serialize(/*base64*/ true)); + + columns->add(column_info); + } + + file_info->set("columns", columns); + } + + return file_info; +} + +constexpr size_t FIELD_MASK_ROWS = 0x1; +constexpr size_t FIELD_MASK_NULLS = 0x2; +constexpr size_t FIELD_MASK_RECT = 0x4; +constexpr size_t FIELD_MASK_ALL = 0x7; + +void DataFileMetaInfo::serialize(WriteBuffer & out) const +{ + auto size = columns_info.size(); + writeIntBinary(size, out); + for (const auto & column : columns_info) + { + writeStringBinary(column.first, out); + size_t field_mask = 0; + if (column.second.rows_count.has_value()) + field_mask |= FIELD_MASK_ROWS; + if (column.second.nulls_count.has_value()) + field_mask |= FIELD_MASK_NULLS; + if (column.second.hyperrectangle.has_value()) + field_mask |= FIELD_MASK_RECT; + writeIntBinary(field_mask, out); + + if (column.second.rows_count.has_value()) + writeIntBinary(column.second.rows_count.value(), out); + if (column.second.nulls_count.has_value()) + writeIntBinary(column.second.nulls_count.value(), out); + if (column.second.hyperrectangle.has_value()) + { + writeFieldBinary(column.second.hyperrectangle.value().left, out); + writeFieldBinary(column.second.hyperrectangle.value().right, out); + } + } +} + +DataFileMetaInfo DataFileMetaInfo::deserialize(ReadBuffer & in) +{ + DataFileMetaInfo result; + + size_t size; + readIntBinary(size, in); + + for (size_t i = 0; i < size; ++i) + { + std::string name; + readStringBinary(name, in); + size_t field_mask; + readIntBinary(field_mask, in); + if ((field_mask & FIELD_MASK_ALL) != field_mask) + throw Exception(ErrorCodes::INCORRECT_DATA, "Unexpected field mask: {}", field_mask); + + ColumnInfo & column = result.columns_info[name]; + + if (field_mask & FIELD_MASK_ROWS) + { + Int64 value; + readIntBinary(value, in); + column.rows_count = value; + } + if (field_mask & FIELD_MASK_NULLS) + { + Int64 value; + readIntBinary(value, in); + column.nulls_count = value; + } + if (field_mask & FIELD_MASK_RECT) + { + FieldRef left = readFieldBinary(in); + FieldRef right = readFieldBinary(in); + column.hyperrectangle = Range(left, true, right, true); + } + } + + return result; +} + + } diff --git a/src/Storages/ObjectStorage/DataLakes/IDataLakeMetadata.h b/src/Storages/ObjectStorage/DataLakes/IDataLakeMetadata.h index 8347164b9a46..73c6d64e2ccb 100644 --- a/src/Storages/ObjectStorage/DataLakes/IDataLakeMetadata.h +++ b/src/Storages/ObjectStorage/DataLakes/IDataLakeMetadata.h @@ -3,20 +3,26 @@ #include #include +#include #include #include +#include +#include #include #include #include #include #include #include +#include #include #include #include #include #include #include +#include + namespace DataLake { @@ -29,7 +35,73 @@ namespace DB namespace ErrorCodes { extern const int UNSUPPORTED_METHOD; -} +}; + +namespace Iceberg +{ +struct ColumnInfo; +}; + +class DataFileMetaInfo +{ +public: + DataFileMetaInfo() = default; + + // Deserialize from json in distributed requests + explicit DataFileMetaInfo(const Poco::JSON::Object::Ptr file_info); + + // Serialize to json in distributed requests + Poco::JSON::Object::Ptr toJson() const; + + // subset of Iceberg::ColumnInfo now + struct ColumnInfo + { + std::optional rows_count; + std::optional nulls_count; + std::optional hyperrectangle; + }; + + // Extract metadata from Iceberg structure. + // table_schema_id is the current table schema, used to resolve column names that + // appear as keys in the resulting `columns_info` map. + // file_schema_id is the schema the data file (and its `value_bounds_`) was written + // with — bounds bytes are encoded with that schema's column types, so they must be + // deserialized using those types. After schema evolution (e.g. `int` -> `long`) + // the two ids differ, and using the table schema's type would misinterpret the + // bytes and produce a garbage hyperrectangle. + explicit DataFileMetaInfo( + const Iceberg::IcebergSchemaProcessor & schema_processor, + Int32 table_schema_id, + Int32 file_schema_id, + const std::unordered_map & columns_info_, + const std::unordered_map> & value_bounds_); + + void serialize(WriteBuffer & out) const; + static DataFileMetaInfo deserialize(ReadBuffer & in); + + bool empty() const { return columns_info.empty(); } + + std::unordered_map columns_info; +}; + +using DataFileMetaInfoPtr = std::shared_ptr; + +struct DataFileInfo +{ + std::string file_path; + std::optional file_meta_info; + + explicit DataFileInfo(const std::string & file_path_) + : file_path(file_path_) {} + + explicit DataFileInfo(std::string && file_path_) + : file_path(std::move(file_path_)) {} + + bool operator==(const DataFileInfo & rhs) const + { + return file_path == rhs.file_path; + } +}; class BackgroundJobsAssignee; class SinkToStorage; @@ -38,7 +110,6 @@ class StorageObjectStorageConfiguration; using StorageObjectStorageConfigurationPtr = std::shared_ptr; struct StorageID; struct IObjectIterator; -struct RelativePathWithMetadata; class IObjectStorage; struct ObjectInfo; using ObjectInfoPtr = std::shared_ptr; @@ -140,6 +211,37 @@ class IDataLakeMetadata : boost::noncopyable throwNotImplemented("write"); } + virtual bool supportsImport(ContextPtr) const + { + return false; + } + + virtual SinkToStoragePtr import( + std::shared_ptr /* catalog */, + const std::function & /* new_file_path_callback */, + SharedHeader /* sample_block */, + const std::string & /* iceberg_metadata_json_string */, + const std::optional & /* format_settings_ */, + ContextPtr /* context */) + { + throwNotImplemented("import"); + } + + virtual IStorage::ExportPartitionCommitInfo commitExportPartitionTransaction( + std::shared_ptr /* catalog */, + const StorageID & /* table_id */, + const String & /* transaction_id */, + Int64 /* original_schema_id */, + Int64 /* partition_spec_id */, + const Block & /* partition_source_block */, + SharedHeader /* sample_block */, + const std::vector & /* data_file_paths */, + StorageObjectStorageConfigurationPtr /* configuration */, + ContextPtr /* context */) + { + throwNotImplemented("commitExportPartitionTransaction"); + } + virtual bool optimize( const StorageMetadataPtr & /*metadata_snapshot*/, ContextPtr /*context*/, const std::optional & /*format_settings*/) { @@ -190,6 +292,9 @@ class IDataLakeMetadata : boost::noncopyable virtual Int32 getBiasBackoffSeconds() const { return 0; } virtual bool isBackgroundExecutable() const { return false; } + virtual std::optional partitionKey(ContextPtr) const { return {}; } + virtual std::optional sortingKey(ContextPtr) const { return {}; } + protected: virtual ObjectIterator createKeysIterator(Strings && data_files_, ObjectStoragePtr object_storage_, IDataLakeMetadata::FileProgressCallback callback_) const; diff --git a/src/Storages/ObjectStorage/DataLakes/Iceberg/AvroSchema.h b/src/Storages/ObjectStorage/DataLakes/Iceberg/AvroSchema.h index 4e70988735b3..97c832760a11 100644 --- a/src/Storages/ObjectStorage/DataLakes/Iceberg/AvroSchema.h +++ b/src/Storages/ObjectStorage/DataLakes/Iceberg/AvroSchema.h @@ -581,4 +581,42 @@ static constexpr const char * manifest_entry_v2_schema = R"( } )"; +/// Schema for the per-data-file sidecar Avro files written alongside every data file +/// during import/export. The sidecar carries the row count and byte size that cannot +/// be cheaply inferred from the data file itself without a full scan. +static constexpr const char * data_file_sidecar_schema = R"( +{ + "type": "record", + "name": "data_file_metadata", + "fields": [ + {"name": "record_count", "type": "long"}, + {"name": "file_size_in_bytes", "type": "long"}, + { + "name": "column_sizes", + "type": {"type": "array", "items": {"type": "record", "name": "cs_entry", + "fields": [{"name": "key", "type": "int"}, {"name": "value", "type": "long"}]}}, + "default": [] + }, + { + "name": "null_value_counts", + "type": {"type": "array", "items": {"type": "record", "name": "nvc_entry", + "fields": [{"name": "key", "type": "int"}, {"name": "value", "type": "long"}]}}, + "default": [] + }, + { + "name": "lower_bounds", + "type": {"type": "array", "items": {"type": "record", "name": "lb_entry", + "fields": [{"name": "key", "type": "int"}, {"name": "value", "type": "bytes"}]}}, + "default": [] + }, + { + "name": "upper_bounds", + "type": {"type": "array", "items": {"type": "record", "name": "ub_entry", + "fields": [{"name": "key", "type": "int"}, {"name": "value", "type": "bytes"}]}}, + "default": [] + } + ] +} +)"; + } diff --git a/src/Storages/ObjectStorage/DataLakes/Iceberg/ChunkPartitioner.cpp b/src/Storages/ObjectStorage/DataLakes/Iceberg/ChunkPartitioner.cpp index 6e5579c79740..5a169f07d563 100644 --- a/src/Storages/ObjectStorage/DataLakes/Iceberg/ChunkPartitioner.cpp +++ b/src/Storages/ObjectStorage/DataLakes/Iceberg/ChunkPartitioner.cpp @@ -6,6 +6,7 @@ #include #include #include +#include #include #include #include @@ -18,6 +19,7 @@ namespace DB namespace Setting { extern const SettingsUInt64 iceberg_insert_max_partitions; + extern const SettingsTimezone iceberg_partition_timezone; } namespace ErrorCodes @@ -52,7 +54,7 @@ ChunkPartitioner::ChunkPartitioner( auto & factory = FunctionFactory::instance(); - auto transform_and_argument = Iceberg::parseTransformAndArgument(transform_name); + auto transform_and_argument = Iceberg::parseTransformAndArgument(transform_name, context->getSettingsRef()[Setting::iceberg_partition_timezone]); if (!transform_and_argument) throw Exception(ErrorCodes::BAD_ARGUMENTS, "Unknown transform {}", transform_name); @@ -66,6 +68,7 @@ ChunkPartitioner::ChunkPartitioner( result_data_types.push_back(function->getReturnType(columns_for_function)); functions.push_back(function); function_params.push_back(transform_and_argument->argument); + function_time_zones.push_back(transform_and_argument->time_zone); columns_to_apply.push_back(column_name); } } @@ -109,6 +112,14 @@ ChunkPartitioner::partitionChunk(const Chunk & chunk) arguments.push_back(ColumnWithTypeAndName(const_column->clone(), type, "#")); } arguments.push_back(name_to_column[columns_to_apply[transform_ind]]); + if (function_time_zones[transform_ind].has_value()) + { + auto type = std::make_shared(); + auto column_value = ColumnString::create(); + column_value->insert(*function_time_zones[transform_ind]); + auto const_column = ColumnConst::create(std::move(column_value), chunk.getNumRows()); + arguments.push_back(ColumnWithTypeAndName(const_column->clone(), type, "PartitioningTimezone")); + } auto result = functions[transform_ind]->build(arguments)->execute(arguments, result_data_types[transform_ind], chunk.getNumRows(), false); functions_columns.push_back(result); diff --git a/src/Storages/ObjectStorage/DataLakes/Iceberg/ChunkPartitioner.h b/src/Storages/ObjectStorage/DataLakes/Iceberg/ChunkPartitioner.h index ce6a057938e1..9f6582337436 100644 --- a/src/Storages/ObjectStorage/DataLakes/Iceberg/ChunkPartitioner.h +++ b/src/Storages/ObjectStorage/DataLakes/Iceberg/ChunkPartitioner.h @@ -44,6 +44,7 @@ class ChunkPartitioner std::vector functions; std::vector> function_params; + std::vector> function_time_zones; std::vector columns_to_apply; DataTypes result_data_types; diff --git a/src/Storages/ObjectStorage/DataLakes/Iceberg/Compaction.cpp b/src/Storages/ObjectStorage/DataLakes/Iceberg/Compaction.cpp index beb714c61a23..bcd7b88f49ad 100644 --- a/src/Storages/ObjectStorage/DataLakes/Iceberg/Compaction.cpp +++ b/src/Storages/ObjectStorage/DataLakes/Iceberg/Compaction.cpp @@ -534,7 +534,7 @@ static bool writeConsolidatedManifestFile( /// Derive partition value types from a schema that defines every source column the spec references, preferring the current schema then any historical one; register all schemas first so they can be queried by id. for (UInt32 i = 0; i < schemas->size(); ++i) - persistent_table_components.schema_processor->addIcebergTableSchema(schemas->getObject(i)); + persistent_table_components.schema_processor->addIcebergTableSchema(schemas->getObject(i), context); auto build_sample_block = [&](Int32 schema_id) -> std::optional { diff --git a/src/Storages/ObjectStorage/DataLakes/Iceberg/Constant.h b/src/Storages/ObjectStorage/DataLakes/Iceberg/Constant.h index f823774b1cff..73dbb5699cbe 100644 --- a/src/Storages/ObjectStorage/DataLakes/Iceberg/Constant.h +++ b/src/Storages/ObjectStorage/DataLakes/Iceberg/Constant.h @@ -172,6 +172,7 @@ DEFINE_ICEBERG_FIELD_ALIAS(max_ref_age_ms, history.expire.max-ref-age-ms); DEFINE_ICEBERG_FIELD_ALIAS(ref_min_snapshots_to_keep, min-snapshots-to-keep); DEFINE_ICEBERG_FIELD_ALIAS(ref_max_snapshot_age_ms, max-snapshot-age-ms); DEFINE_ICEBERG_FIELD_ALIAS(ref_max_ref_age_ms, max-ref-age-ms); +DEFINE_ICEBERG_FIELD_ALIAS(clickhouse_export_partition_transaction_id, clickhouse.export-partition-transaction-id); /// These are compound fields like `data_file.file_path`, we use prefix 'c_' to distinguish them. DEFINE_ICEBERG_FIELD_COMPOUND(data_file, file_path); DEFINE_ICEBERG_FIELD_COMPOUND(data_file, file_format); diff --git a/src/Storages/ObjectStorage/DataLakes/Iceberg/IcebergDataFileEntry.h b/src/Storages/ObjectStorage/DataLakes/Iceberg/IcebergDataFileEntry.h new file mode 100644 index 000000000000..61fa7be9005b --- /dev/null +++ b/src/Storages/ObjectStorage/DataLakes/Iceberg/IcebergDataFileEntry.h @@ -0,0 +1,50 @@ +#pragma once + +#include "config.h" + +#if USE_AVRO + +#include +#include +#include +#include + +namespace DB +{ + +/// Column-level statistics for a single Iceberg data file stored in Iceberg wire format. +/// Bounds are pre-serialized to bytes so the struct can be persisted to sidecar Avro files +/// and used directly at manifest-commit time without requiring the original ClickHouse +/// DataFileStatistics or a live Block schema. +struct IcebergSerializedFileStats +{ + Int64 record_count = 0; + Int64 file_size_in_bytes = 0; + + /// field_id → compressed byte size of column in the file + std::vector> column_sizes; + /// field_id → number of null values in the file + std::vector> null_value_counts; + /// field_id → Iceberg-serialized lower bound (same binary format as manifest) + std::vector>> lower_bounds; + /// field_id → Iceberg-serialized upper bound (same binary format as manifest) + std::vector>> upper_bounds; +}; + +/// One entry describing a data file that will be registered in an Iceberg manifest. +/// Carries per-file statistics so that each manifest entry gets accurate metadata +/// (column sizes, null counts, min/max bounds, record count, file size). +struct IcebergDataFileEntry +{ + String path; + Int64 record_count = 0; + Int64 file_size_in_bytes = 0; + + /// Per-file column statistics (null counts, min/max bounds, column sizes). + /// Pass std::nullopt when statistics are not available or not yet computed. + std::optional statistics; +}; + +} + +#endif diff --git a/src/Storages/ObjectStorage/DataLakes/Iceberg/IcebergFieldParseHelpers.cpp b/src/Storages/ObjectStorage/DataLakes/Iceberg/IcebergFieldParseHelpers.cpp index 0874b663303a..54383d7ac834 100644 --- a/src/Storages/ObjectStorage/DataLakes/Iceberg/IcebergFieldParseHelpers.cpp +++ b/src/Storages/ObjectStorage/DataLakes/Iceberg/IcebergFieldParseHelpers.cpp @@ -4,7 +4,11 @@ #include #include +#include +#include #include +#include +#include #include #include @@ -98,6 +102,97 @@ std::vector fieldToInt64Array(const Field & value, std::string_view conte return result; } +namespace +{ + /// Iceberg store decimal values as unscaled value with two's-complement big-endian binary + /// using the minimum number of bytes for the value + /// Our decimal binary representation is little endian + /// so we cannot reuse our default code for parsing it. + /// + /// NOTE: It's very weird, but Decimal values for lower bound and upper bound + /// are stored rounded, without fractional part. What is more strange + /// the integer part is rounded mathematically correctly according to fractional part. + /// Example: 17.22 -> 17, 8888.999 -> 8889, 1423.77 -> 1424. + /// I've checked two implementations: Spark and Amazon Athena and both of them + /// do this. + /// + /// The problem is -- we cannot use rounded values for lower bounds and upper bounds. + /// Example: upper_bound(x) = 17.22, but it's rounded 17.00, now condition WHERE x >= 17.21 will + /// check rounded value and say: "Oh largest value is 17, so values bigger than 17.21 cannot be in this file, + /// let's skip it". But it will produce incorrect result since actual value (17.22 >= 17.21) is stored in this file. + /// + /// To handle this issue we subtract 1 from the integral part for lower_bound and add 1 to integral + /// part of upper_bound. This produces: 17.22 -> [16.0, 18.0]. So this is more rough boundary, + /// but at least it doesn't lead to incorrect results. + /// `compensate_rounding` widens the bound as described above; pass false to read the value exactly + /// as the manifest declares it. + template + std::optional deserializeDecimalBound(const std::string & str, UInt32 scale, bool lower_bound, bool compensate_rounding = true) + { + using NativeType = typename DecimalType::NativeType; + using UnsignedType = make_unsigned_t; + + if (str.size() > sizeof(NativeType)) + return std::nullopt; + + /// Accumulate into the unsigned counterpart, pre-filled with the sign bits, + /// so that the sign extension comes out of the shifts themselves. + UnsignedType unscaled = (str[0] & 0x80) ? ~UnsignedType(0) : UnsignedType(0); + for (const auto byte : str) + unscaled = (unscaled << 8) | static_cast(byte); + + NativeType unscaled_value = static_cast(unscaled); + + if (compensate_rounding && scale) + { + NativeType scaler = lower_bound ? -10 : 10; + for (UInt32 i = 1; i < scale; ++i) + scaler *= 10; + + /// The bound is stored as raw bytes and is never checked against the declared precision, so + /// widening it can leave the type. A value that has no widened form is not a usable bound. + if (common::addOverflow(unscaled_value, scaler, unscaled_value)) + return std::nullopt; + } + + return DecimalField(unscaled_value, scale); + } + +} + +std::optional +deserializeFieldFromBinaryRepr(const std::string & str, DataTypePtr expected_type, bool lower_bound, bool compensate_rounding) +{ + auto non_nullable_type = removeNullable(expected_type); + auto column = non_nullable_type->createColumn(); + if (WhichDataType(non_nullable_type).isDecimal()) + { + if (str.empty()) + return std::nullopt; + + const UInt32 scale = getDecimalScale(*non_nullable_type); + if (checkDecimal(*non_nullable_type)) + return deserializeDecimalBound(str, scale, lower_bound, compensate_rounding); + if (checkDecimal(*non_nullable_type)) + return deserializeDecimalBound(str, scale, lower_bound, compensate_rounding); + if (checkDecimal(*non_nullable_type)) + return deserializeDecimalBound(str, scale, lower_bound, compensate_rounding); + if (checkDecimal(*non_nullable_type)) + return deserializeDecimalBound(str, scale, lower_bound, compensate_rounding); + return std::nullopt; + } + + if (non_nullable_type->getTypeId() == TypeIndex::Variant) + return std::nullopt; + + /// For all other types except decimal binary representation + /// matches our internal representation + column->insertData(str.data(), str.length()); + Field result; + column->get(0, result); + return result; +} + } } diff --git a/src/Storages/ObjectStorage/DataLakes/Iceberg/IcebergFieldParseHelpers.h b/src/Storages/ObjectStorage/DataLakes/Iceberg/IcebergFieldParseHelpers.h index 285062cb3d4f..26f37f0aa9a3 100644 --- a/src/Storages/ObjectStorage/DataLakes/Iceberg/IcebergFieldParseHelpers.h +++ b/src/Storages/ObjectStorage/DataLakes/Iceberg/IcebergFieldParseHelpers.h @@ -4,11 +4,14 @@ #if USE_AVRO +#include +#include #include #include #include #include +#include namespace DB::Iceberg { @@ -25,6 +28,13 @@ Int64 fieldToPeriodMs(const Field & value, std::string_view context, std::string /// Convert a Field containing an Array to vector, validating each element. std::vector fieldToInt64Array(const Field & value, std::string_view context, std::string_view arg_name); +/// Deserialize a single lower/upper bound value from Iceberg's binary representation. +/// See https://iceberg.apache.org/spec/#appendix-d-single-value-serialization +/// `compensate_rounding` widens a decimal bound by one integral unit; pass false to read the value +/// exactly as the manifest declares it. +std::optional +deserializeFieldFromBinaryRepr(const std::string & str, DataTypePtr expected_type, bool lower_bound, bool compensate_rounding = true); + } #endif diff --git a/src/Storages/ObjectStorage/DataLakes/Iceberg/IcebergIterator.cpp b/src/Storages/ObjectStorage/DataLakes/Iceberg/IcebergIterator.cpp index baa575fdc03a..3ccc4248d9c8 100644 --- a/src/Storages/ObjectStorage/DataLakes/Iceberg/IcebergIterator.cpp +++ b/src/Storages/ObjectStorage/DataLakes/Iceberg/IcebergIterator.cpp @@ -315,6 +315,7 @@ IcebergIterator::IcebergIterator( persistent_components_) , blocking_queue(100) , callback(std::move(callback_)) + , table_schema_id(table_snapshot_->schema_id) { /// Decoding any manifest reads settings from the context, so a missing one is fatal either way. if (!local_context) @@ -609,6 +610,12 @@ ObjectInfoPtr IcebergIterator::next(size_t) object_info->info.data_object_file_path_key); } + object_info->relative_path_with_metadata.setFileMetaInfo(std::make_shared( + *persistent_components.schema_processor, + table_schema_id, /// current schema id to use current column names + manifest_file_entry->resolved_schema_id, /// file's schema id to interpret value_bounds bytes + manifest_file_entry->parsed_entry->columns_infos, + manifest_file_entry->parsed_entry->value_bounds)); ProfileEvents::increment(ProfileEvents::IcebergMetadataReturnedObjectInfos); if (callback) diff --git a/src/Storages/ObjectStorage/DataLakes/Iceberg/IcebergIterator.h b/src/Storages/ObjectStorage/DataLakes/Iceberg/IcebergIterator.h index 78f6d222c37d..6b27c8a37c81 100644 --- a/src/Storages/ObjectStorage/DataLakes/Iceberg/IcebergIterator.h +++ b/src/Storages/ObjectStorage/DataLakes/Iceberg/IcebergIterator.h @@ -88,7 +88,7 @@ class IcebergIterator : public IObjectIterator IDataLakeMetadata::FileProgressCallback callback_, Iceberg::TableStateSnapshotPtr table_snapshot_, Iceberg::IcebergDataSnapshotPtr data_snapshot_, - Iceberg::PersistentTableComponents persistent_components); + Iceberg::PersistentTableComponents persistent_components_); ObjectInfoPtr next(size_t) override; @@ -120,6 +120,7 @@ class IcebergIterator : public IObjectIterator std::exception_ptr deletes_exception TSA_GUARDED_BY(deletes_mutex); std::exception_ptr exception; std::mutex exception_mutex; + Int32 table_schema_id; }; } diff --git a/src/Storages/ObjectStorage/DataLakes/Iceberg/IcebergMetadata.cpp b/src/Storages/ObjectStorage/DataLakes/Iceberg/IcebergMetadata.cpp index 35a45bde43fd..3fa4471b3f17 100644 --- a/src/Storages/ObjectStorage/DataLakes/Iceberg/IcebergMetadata.cpp +++ b/src/Storages/ObjectStorage/DataLakes/Iceberg/IcebergMetadata.cpp @@ -35,6 +35,7 @@ #include #include +#include #include #include @@ -52,10 +53,12 @@ #include #include #include +#include #include #include #include +#include #include #include #include @@ -80,6 +83,7 @@ #include #include #include +#include #include #include @@ -122,6 +126,9 @@ extern const int S3_ERROR; extern const int TABLE_ALREADY_EXISTS; extern const int SUPPORT_IS_DISABLED; extern const int FILE_ALREADY_EXISTS; +extern const int METADATA_MISMATCH; +extern const int UNFINISHED; +extern const int INCORRECT_DATA; } namespace Setting @@ -154,12 +161,59 @@ extern const SettingsUInt64 iceberg_data_file_size_lower_threshold_compaction; extern const SettingsUInt64 iceberg_data_file_size_upper_threshold_compaction; } +static constexpr size_t MAX_TRANSACTION_RETRIES = 100; + namespace { String dumpMetadataObjectToString(const Poco::JSON::Object::Ptr & metadata_object) { return stringifyJSON(metadata_object); } + +/// Check if a previous attempt already committed this transaction the snapshot +/// (with our transaction_id embedded in its summary) is still present in the snapshots array +/// unless an external engine ran expireSnapshots in the meantime. If found, skip re-committing. +bool isExportPartitionTransactionAlreadyCommitted(const Poco::JSON::Object::Ptr & metadata, const String & transaction_id) +{ + const auto throw_error = [&](const std::string & missing_field_name) + { + throw Exception( + ErrorCodes::ICEBERG_SPECIFICATION_VIOLATION, + "No {} found in metadata for iceberg file while trying to commit export partition transaction", + missing_field_name); + }; + + const auto snapshots = metadata->getArray(Iceberg::f_snapshots); + + if (!snapshots) + { + throw_error(Iceberg::f_snapshots); + } + + for (size_t i = 0; i < snapshots->size(); ++i) + { + const auto snap = snapshots->getObject(static_cast(i)); + const auto summary = snap->getObject(Iceberg::f_summary); + + if (!summary) + { + throw_error(Iceberg::f_summary); + } + + if (summary->has(Iceberg::f_clickhouse_export_partition_transaction_id)) + { + const auto tid = summary->getValue(Iceberg::f_clickhouse_export_partition_transaction_id); + + if (tid == transaction_id) + { + return true; + } + } + } + + return false; +} + } @@ -210,7 +264,8 @@ Iceberg::PersistentTableComponents IcebergMetadata::initializePersistentTableCom auto table_path = configuration->getPathForRead().path; auto root_derivation = IcebergPathResolver::deriveTableRoot(table_location, table_path, metadata_file_path); return PersistentTableComponents{ - .schema_processor = std::make_shared(context_->getSettingsRef()[Setting::allow_experimental_geo_types_in_iceberg]), + .schema_processor = std::make_shared( + context_, context_->getSettingsRef()[Setting::allow_experimental_geo_types_in_iceberg]), .metadata_cache = cache_ptr, .format_version = format_version, .table_location = table_location, @@ -220,6 +275,7 @@ Iceberg::PersistentTableComponents IcebergMetadata::initializePersistentTableCom .path_resolver = IcebergPathResolver( table_location, root_derivation.table_root, configuration->getTypeName(), configuration->getNamespace()), .table_root_was_derived = root_derivation.relation == IcebergPathResolver::RootRelation::AdoptedDescendant, + .common_namespace = configuration->getNamespace(), }; } @@ -247,7 +303,7 @@ IcebergMetadata::IcebergMetadata( , object_storage(std::move(object_storage_)) , persistent_components(std::move(persistent_components_)) , data_lake_settings(configuration_->getDataLakeSettings()) - , write_format(configuration_->format) + , write_format(configuration_->getFormat()) { /// TODO: for now it's okay to start/stop the task via constructor/destructor. Once refactored, we'd need to plumb startup/shutdown and schedule the task from there if (persistent_components.metadata_cache && data_lake_settings[DataLakeStorageSetting::iceberg_metadata_async_prefetch_period_ms] != 0) @@ -324,6 +380,7 @@ void IcebergMetadata::backgroundMetadataPrefetcherThread() Int32 IcebergMetadata::parseTableSchema( const Poco::JSON::Object::Ptr & metadata_object, IcebergSchemaProcessor & schema_processor, + ContextPtr context_, LoggerPtr metadata_logger) { const auto format_version = metadata_object->getValue(f_format_version); @@ -331,7 +388,7 @@ Int32 IcebergMetadata::parseTableSchema( if (format_version == 2) { auto [schema, current_schema_id] = parseTableSchemaV2Method(metadata_object); - schema_processor.addIcebergTableSchema(schema); + schema_processor.addIcebergTableSchema(schema, context_); return current_schema_id; } else @@ -339,7 +396,7 @@ Int32 IcebergMetadata::parseTableSchema( try { auto [schema, current_schema_id] = parseTableSchemaV1Method(metadata_object); - schema_processor.addIcebergTableSchema(schema); + schema_processor.addIcebergTableSchema(schema, context_); return current_schema_id; } catch (const Exception & first_error) @@ -349,7 +406,7 @@ Int32 IcebergMetadata::parseTableSchema( try { auto [schema, current_schema_id] = parseTableSchemaV2Method(metadata_object); - schema_processor.addIcebergTableSchema(schema); + schema_processor.addIcebergTableSchema(schema, context_); LOG_WARNING( metadata_logger, "Iceberg table schema was parsed using v2 specification, but it was impossible to parse it using v1 " @@ -373,7 +430,10 @@ Int32 IcebergMetadata::parseTableSchema( } static Poco::JSON::Object::Ptr traverseMetadataAndFindNecessarySnapshotObject( - Poco::JSON::Object::Ptr metadata_object, Int64 snapshot_id, IcebergSchemaProcessorPtr schema_processor) + Poco::JSON::Object::Ptr metadata_object, + Int64 snapshot_id, + IcebergSchemaProcessorPtr schema_processor, + ContextPtr local_context) { if (!metadata_object->has(f_snapshots)) throw Exception(ErrorCodes::ICEBERG_SPECIFICATION_VIOLATION, "No snapshot set found in metadata for iceberg file"); @@ -381,7 +441,7 @@ static Poco::JSON::Object::Ptr traverseMetadataAndFindNecessarySnapshotObject( for (UInt32 j = 0; j < schemas->size(); ++j) { auto schema = schemas->getObject(j); - schema_processor->addIcebergTableSchema(schema); + schema_processor->addIcebergTableSchema(schema, local_context); } Poco::JSON::Object::Ptr current_snapshot = nullptr; auto snapshots = metadata_object->get(f_snapshots).extract(); @@ -449,7 +509,11 @@ IcebergDataSnapshotPtr IcebergMetadata::createIcebergDataSnapshotFromSnapshotJSO IcebergDataSnapshotPtr IcebergMetadata::getIcebergDataSnapshot(Poco::JSON::Object::Ptr metadata_object, Int64 snapshot_id, ContextPtr local_context) const { - auto object = traverseMetadataAndFindNecessarySnapshotObject(metadata_object, snapshot_id, persistent_components.schema_processor); + auto object = traverseMetadataAndFindNecessarySnapshotObject( + metadata_object, + snapshot_id, + persistent_components.schema_processor, + local_context); if (!object) throw Exception(ErrorCodes::ICEBERG_SPECIFICATION_VIOLATION, "No snapshot found for id `{}`", snapshot_id); @@ -590,7 +654,7 @@ IcebergMetadata::getStateImpl(const ContextPtr & local_context, Poco::JSON::Obje } else { - auto schema_id = parseTableSchema(metadata_object, *persistent_components.schema_processor, log); + auto schema_id = parseTableSchema(metadata_object, *persistent_components.schema_processor, local_context, log); if (!metadata_object->has(f_current_snapshot_id)) { return {nullptr, schema_id}; @@ -615,9 +679,10 @@ IcebergMetadata::getState(const ContextPtr & local_context, const String & metad auto metadata_object = getMetadataJSONObject( metadata_path, object_storage, persistent_components.metadata_cache, local_context, log, persistent_components.metadata_compression_method, persistent_components.table_uuid); + auto dump_metadata = [&]()->String { return dumpMetadataObjectToString(metadata_object); }; insertRowToLogTable( local_context, - [&] { return dumpMetadataObjectToString(metadata_object); }, + dump_metadata, DB::IcebergMetadataLogLevel::Metadata, persistent_components.path_resolver.getTableRoot(), Iceberg::IcebergPathFromMetadata::deserialize(metadata_path), @@ -643,14 +708,16 @@ std::shared_ptr IcebergMetadata::getInitialSchemaByPath(Conte : nullptr; } -std::shared_ptr IcebergMetadata::getSchemaTransformer(ContextPtr, ObjectInfoPtr object_info) const +std::shared_ptr IcebergMetadata::getSchemaTransformer(ContextPtr context_, ObjectInfoPtr object_info) const { IcebergDataObjectInfo * iceberg_object_info = dynamic_cast(object_info.get()); if (!iceberg_object_info) return nullptr; return (iceberg_object_info->info.underlying_format_read_schema_id != iceberg_object_info->info.schema_id_relevant_to_iterator) ? persistent_components.schema_processor->getSchemaTransformationDagByIds( - iceberg_object_info->info.underlying_format_read_schema_id, iceberg_object_info->info.schema_id_relevant_to_iterator) + context_, + iceberg_object_info->info.underlying_format_read_schema_id, + iceberg_object_info->info.schema_id_relevant_to_iterator) : nullptr; } @@ -835,7 +902,7 @@ void IcebergMetadata::createInitial( if (catalog_manages_location) { DataLake::TableMetadata existing_table; - if (catalog->tryGetTableMetadata(namespace_name, table_name, existing_table)) + if (catalog->tryGetTableMetadata(namespace_name, table_name, local_context, existing_table)) { if (if_not_exists) return; @@ -938,7 +1005,7 @@ Iceberg::IcebergDataSnapshotPtr IcebergMetadata::getRelevantDataSnapshotFromTabl if (!table_state_snapshot.snapshot_id.has_value()) return nullptr; Poco::JSON::Object::Ptr snapshot_object = traverseMetadataAndFindNecessarySnapshotObject( - metadata_object, *table_state_snapshot.snapshot_id, persistent_components.schema_processor); + metadata_object, *table_state_snapshot.snapshot_id, persistent_components.schema_processor, local_context); return createIcebergDataSnapshotFromSnapshotJSON(snapshot_object, *table_state_snapshot.snapshot_id, local_context); } @@ -1332,6 +1399,35 @@ std::optional IcebergMetadata::totalBytes(ContextPtr local_context) cons return static_cast(result); } +std::optional IcebergMetadata::partitionKey(ContextPtr context) const +{ + auto [actual_data_snapshot, actual_table_state_snapshot] = getRelevantState(context); + return getPartitionKey(context, actual_table_state_snapshot); +} + +std::optional IcebergMetadata::sortingKey(ContextPtr context) const +{ + auto [actual_data_snapshot, actual_table_state_snapshot] = getRelevantState(context); + auto metadata_object = getMetadataJSONObject( + actual_table_state_snapshot.metadata_file_path, + object_storage, + persistent_components.metadata_cache, + context, + log, + persistent_components.metadata_compression_method, + persistent_components.table_uuid); + auto [schema, current_schema_id] = parseTableSchemaV2Method(metadata_object); + const auto & ch_schema = *persistent_components.schema_processor->getClickHouseTableSchemaById(current_schema_id); + auto display = getSortingKeyDisplayStringFromMetadata(metadata_object, ch_schema); + if (display) + return display; + auto key = getSortingKey(context, actual_table_state_snapshot); + if (!key.expression_list_ast) + return std::nullopt; + return format({context, *key.expression_list_ast}); +} + + ObjectIterator IcebergMetadata::iterate( const ActionsDAG * filter_dag, FileProgressCallback callback, @@ -1416,12 +1512,14 @@ void IcebergMetadata::addDeleteTransformers( if (!iceberg_object_info->info.position_deletes_objects.empty()) { + LOG_DEBUG(log, "Constructing filter transform for position delete, there are {} delete objects", iceberg_object_info->info.position_deletes_objects.size()); builder.addSimpleTransform( [&](const SharedHeader & header) { return iceberg_object_info->getPositionDeleteTransformer(object_storage, header, format_settings, parser_shared_resources, local_context); }); } const auto & delete_files = iceberg_object_info->info.equality_deletes_objects; - LOG_DEBUG(log, "Constructing filter transform for equality delete, there are {} delete files", delete_files.size()); + if (!delete_files.empty()) + LOG_DEBUG(log, "Constructing filter transform for equality delete, there are {} delete files", delete_files.size()); for (const EqualityDeleteObject & delete_file : delete_files) { auto simple_transform_adder = [&](const SharedHeader & header) @@ -1605,6 +1703,23 @@ ColumnMapperPtr IcebergMetadata::getColumnMapperForCurrentSchema(StorageMetadata return persistent_components.schema_processor->getColumnMapperById(iceberg_table_state->schema_id); } +std::optional IcebergMetadata::getPartitionKey(ContextPtr local_context, TableStateSnapshot actual_table_state_snapshot) const +{ + auto metadata_object = getMetadataJSONObject( + actual_table_state_snapshot.metadata_file_path, + object_storage, + persistent_components.metadata_cache, + local_context, + log, + persistent_components.metadata_compression_method, + persistent_components.table_uuid); + auto [schema, current_schema_id] = parseTableSchemaV2Method(metadata_object); + return getPartitionKeyStringFromMetadata( + metadata_object, + *persistent_components.schema_processor->getClickHouseTableSchemaById(current_schema_id), + local_context); +} + KeyDescription IcebergMetadata::getSortingKey(ContextPtr local_context, TableStateSnapshot actual_table_state_snapshot) const { auto metadata_object = getMetadataJSONObject( @@ -1663,11 +1778,597 @@ DataLakeMetadataPtr IcebergMetadata::createWithDeserialization( configuration_ptr->getTypeName(), configuration_ptr->getNamespace()), /// Consistent with the resolver above, which is rooted at `table_path` itself. - .table_root_was_derived = false}; + .table_root_was_derived = false, + .common_namespace = configuration_ptr->getNamespace()}; auto metadata = std::make_unique(object_storage, configuration.lock(), std::move(deserialized_persistent_components), local_context); return metadata; } +SinkToStoragePtr IcebergMetadata::import( + std::shared_ptr catalog, + const std::function & new_file_path_callback, + SharedHeader sample_block, + const std::string & iceberg_metadata_json_string, + const std::optional & format_settings, + ContextPtr context) +{ + Poco::JSON::Parser parser; /// For some reason base/base/JSON.h can not parse this json file + Poco::Dynamic::Var json = parser.parse(iceberg_metadata_json_string); + Poco::JSON::Object::Ptr metadata_json = json.extract(); + + return std::make_shared( + catalog, persistent_components, metadata_json, object_storage, + context, format_settings, write_format, sample_block, data_lake_settings, new_file_path_callback); +} + +namespace FailPoints +{ + extern const char iceberg_writes_cleanup[]; + extern const char iceberg_writes_non_retry_cleanup[]; + extern const char iceberg_writes_post_publish_throw[]; +} + +namespace +{ + +/// Find the partition spec object with the given spec-id inside a metadata JSON document. +/// Throws METADATA_MISMATCH if the spec is not found (indicates metadata/spec-id mismatch). +Poco::JSON::Object::Ptr lookupPartitionSpec(const Poco::JSON::Object::Ptr & meta, Int64 spec_id) +{ + auto specs = meta->getArray(Iceberg::f_partition_specs); + for (size_t i = 0; i < specs->size(); ++i) + { + auto spec = specs->getObject(static_cast(i)); + if (spec->getValue(Iceberg::f_spec_id) == spec_id) + return spec; + } + throw Exception(ErrorCodes::METADATA_MISMATCH, + "Partition spec with id {} not found in table metadata", spec_id); +} + +Poco::JSON::Object::Ptr lookupSchema(const Poco::JSON::Object::Ptr & meta, Int64 schema_id) +{ + auto schemas = meta->getArray(Iceberg::f_schemas); + for (size_t i = 0; i < schemas->size(); ++i) + { + auto schema = schemas->getObject(static_cast(i)); + if (schema->getValue(Iceberg::f_schema_id) == schema_id) + return schema; + } + + throw Exception(ErrorCodes::METADATA_MISMATCH, + "Schema with id {} not found in table metadata", schema_id); +} + +/// Derive the Iceberg partition tuple for an exported part from a representative source row. +/// The MergeTree `partition.value` is the source partition-key expression result; it is neither +/// cast to the destination column types nor expressed through the Iceberg transform, so it must +/// not be written to metadata directly. Within a MergeTree partition the transform result is +/// constant, so a single representative value per partition-source column (taken from the part's +/// minmax block) suffices: cast it to the destination column type and run the same transform the +/// data uses. The result is transform-correct and consistent with the exported data files. +std::vector recomputeExportPartitionValues( + ChunkPartitioner & partitioner, + const SharedHeader & sample_block, + const Block & partition_source_block) +{ + const auto & partition_columns = partitioner.getColumns(); + if (partition_columns.empty()) + return {}; + + for (const auto & column_name : partition_columns) + if (!partition_source_block.has(column_name)) + throw Exception(ErrorCodes::LOGICAL_ERROR, + "Partition source column '{}' required by the Iceberg partition transform is missing " + "from the representative source block while committing an export.", column_name); + + Columns columns; + columns.reserve(sample_block->columns()); + for (size_t i = 0; i < sample_block->columns(); ++i) + { + const auto & dest_column = sample_block->getByPosition(i); + if (partition_source_block.has(dest_column.name)) + { + const auto & source = partition_source_block.getByName(dest_column.name); + ColumnWithTypeAndName representative{source.column->cut(0, 1), source.type, source.name}; + columns.push_back(castColumn(representative, dest_column.type)); + } + else + { + auto column = dest_column.type->createColumn(); + column->insertDefault(); + columns.push_back(std::move(column)); + } + } + + auto partitioned = partitioner.partitionChunk(Chunk(std::move(columns), 1)); + if (partitioned.size() != 1) + throw Exception(ErrorCodes::LOGICAL_ERROR, + "Recomputing Iceberg partition values produced {} partitions for a single representative row; " + "a MergeTree partition must map to exactly one Iceberg partition.", partitioned.size()); + + const auto & key = partitioned.front().first; + return std::vector(key.begin(), key.end()); +} + +} + +std::optional IcebergMetadata::commitImportPartitionTransactionImpl( + FileNamesGenerator & filename_generator, + Poco::JSON::Object::Ptr & metadata, + Poco::JSON::Object::Ptr & partition_spec, + const String & transaction_id, + Int64 original_schema_id, + Int64 partition_spec_id, + const std::vector & partition_values, + const std::vector & partition_columns, + const DataTypes & partition_types, + SharedHeader sample_block, + const std::vector & data_file_paths, + const std::vector & per_file_stats, + Int64 total_data_files, + Int64 total_rows, + Int64 total_chunks_size, + std::shared_ptr catalog, + const StorageID & table_id, + const String & blob_storage_type_name, + const String & blob_storage_namespace_name, + ContextPtr context) +{ + /// this check also exists here because the metadata might have been updated upon retry attempts. + if (isExportPartitionTransactionAlreadyCommitted(metadata, transaction_id)) + { + LOG_INFO(log, + "Export transaction {} already committed, skipping re-commit", + transaction_id); + /// Surface a sentinel so the caller treats this as a successful attempt (non-empty + /// commit info), persists a commit_info znode, and makes the situation visible in + /// system.replicated_partition_exports.committed_metadata_file. We do not know the + /// original committer's paths from here. + IStorage::ExportPartitionCommitInfo already_committed_info; + already_committed_info.iceberg_metadata_file = ""; + return already_committed_info; + } + + const auto & resolver = persistent_components.path_resolver; + + auto metadata_info = filename_generator.generateMetadataPathWithInfo(); + const auto storage_metadata_name = resolver.resolve(metadata_info.path); + + Int64 parent_snapshot = -1; + if (metadata->has(Iceberg::f_current_snapshot_id)) + parent_snapshot = metadata->getValue(Iceberg::f_current_snapshot_id); + + auto [new_snapshot, manifest_list_path] = MetadataGenerator(metadata).generateNextMetadata( + filename_generator, metadata_info.path, parent_snapshot, total_data_files, total_rows, total_chunks_size, total_data_files, /* added_delete_files */0, /* num_deleted_rows */0); + const auto storage_manifest_list_name = resolver.resolve(manifest_list_path); + + /// Embed the stable transaction identifier in the snapshot summary so that a retry after crash + /// can detect the commit already happened by scanning the live snapshots array, without extra S3 + /// files. The field is a ClickHouse extension; Spark/Flink readers ignore unknown summary keys. + new_snapshot->getObject(Iceberg::f_summary)->set( + Iceberg::f_clickhouse_export_partition_transaction_id, transaction_id); + + Iceberg::IcebergPathFromMetadata manifest_entry_path; + String storage_manifest_entry_name; + Int64 manifest_lengths = 0; + + /// Tracks whether the snapshot has become visible to readers. + /// For the file-based layout that happens as soon as writeMetadataFileAndVersionHint + /// succeeds; for a catalog layout it happens when catalog->updateMetadata succeeds. + /// Once published, the manifest entry / manifest list are referenced by the live + /// snapshot and must NOT be deleted by the outer failure cleanup, otherwise the + /// already-published snapshot becomes unreadable. + bool published = false; + + auto cleanup = [&](bool retry_because_of_metadata_conflict) + { + /// We can't cleanup the data files upon retry even if retry_because_of_metadata_conflict == false + /// because this replica or some other replica might attempt to commit the same transaction later + /// todo arthur: in the future, we should consider failing the entire task if retry_because_of_metadata_conflict = true + + object_storage->removeObjectIfExists(StoredObject(storage_manifest_entry_name)); + object_storage->removeObjectIfExists(StoredObject(storage_manifest_list_name)); + + if (retry_because_of_metadata_conflict) + { + MetadataFileWithInfo latest_metadata_file_info; + if (catalog && catalog->isTransactional()) + { + const auto & [namespace_name, table_name] = DataLake::parseTableName(table_id.getTableName()); + DataLake::TableMetadata table_metadata = DataLake::TableMetadata().withLocation().withDataLakeSpecificProperties(); + catalog->getTableMetadata(namespace_name, table_name, context, table_metadata); + + auto table_specific_properties = table_metadata.getDataLakeSpecificProperties(); + if (!table_specific_properties.has_value() || table_specific_properties->iceberg_metadata_file_location.empty()) + throw Exception(ErrorCodes::LOGICAL_ERROR, "Catalog didn't return iceberg metadata location for table {}.{}", namespace_name, table_name); + + String metadata_path = table_metadata.getMetadataLocation(table_specific_properties->iceberg_metadata_file_location); + if (!metadata_path.starts_with(persistent_components.table_path)) + metadata_path = std::filesystem::path(persistent_components.table_path) / metadata_path; + latest_metadata_file_info = Iceberg::getMetadataFileAndVersion(metadata_path); + } + else + { + latest_metadata_file_info = getLatestOrExplicitMetadataFileAndVersion( + object_storage, + persistent_components.table_path, + data_lake_settings, + persistent_components.metadata_cache, + context, + getLogger("IcebergWrites").get(), + persistent_components.table_uuid, + persistent_components.metadata_compression_method, + true); + } + + auto [last_version, metadata_path, compression_method] = latest_metadata_file_info; + + LOG_DEBUG(log, "Rereading metadata file {} with version {}", metadata_path, last_version); + + filename_generator.setVersion(last_version + 1); + filename_generator.setCompressionMethod(compression_method); + + metadata = getMetadataJSONObject( + metadata_path, + object_storage, + persistent_components.metadata_cache, + context, + getLogger("IcebergMetadata"), + compression_method, + persistent_components.table_uuid); + + /// For the export path the schema and partition spec are fixed at the start of the + /// operation (saved in ZooKeeper). If either changed we must fail immediately — + /// the caller has to restart the export from scratch. + const auto new_schema_id = metadata->getValue(Iceberg::f_current_schema_id); + if (new_schema_id != original_schema_id) + throw Exception(ErrorCodes::METADATA_MISMATCH, + "Table schema changed during export (expected schema {}, got {}). Restart the export operation.", + original_schema_id, new_schema_id); + + const Int64 new_partition_spec_id = metadata->getValue(Iceberg::f_default_spec_id); + if (new_partition_spec_id != partition_spec_id) + throw Exception(ErrorCodes::METADATA_MISMATCH, + "Partition spec changed during export (expected spec {}, got {}). Restart the export operation.", + partition_spec_id, new_partition_spec_id); + + partition_spec = lookupPartitionSpec(metadata, partition_spec_id); + + /// partition_values, partition_columns, partition_types, and + /// data_file_paths are all fixed from the saved state — no update needed. + } + }; + + try + { + manifest_entry_path = filename_generator.generateManifestEntryName(); + storage_manifest_entry_name = resolver.resolve(manifest_entry_path); + + auto buffer_manifest_entry = object_storage->writeObject( + StoredObject(storage_manifest_entry_name), WriteMode::Rewrite, std::nullopt, DBMS_DEFAULT_BUFFER_SIZE, context->getWriteSettings()); + + try + { + fiu_do_on(FailPoints::iceberg_writes_non_retry_cleanup, + { + throw Exception(ErrorCodes::BAD_ARGUMENTS, "Failpoint for cleanup enabled"); + }); + + std::vector data_file_metadata_paths; + data_file_metadata_paths.reserve(data_file_paths.size()); + for (const auto & data_file_path : data_file_paths) + data_file_metadata_paths.push_back(Iceberg::IcebergPathFromMetadata::deserialize(data_file_path)); + + generateManifestFile( + metadata, + partition_columns, + partition_values, + partition_types, + data_file_metadata_paths, + /* data_file_row_counts */ {}, + /* data_file_byte_counts */ {}, + std::nullopt, /// per_file_stats is filled, no need for the generic aggregate + sample_block, + new_snapshot, + write_format, + partition_spec, + partition_spec_id, + *buffer_manifest_entry, + Iceberg::FileContentType::DATA, + /* user_defined_sequence_number */ std::nullopt, + /* user_defined_snapshot_id */ std::nullopt, + /* data_file_formats */ {}, + /* per_file_statistics */ {}, + /* data_file_sort_order_ids */ {}, + /* per_file_entry_lineage */ {}, + /* schema_to_serialize */ nullptr, + /* per_file_fresh_statistics */ nullptr, + per_file_stats); + buffer_manifest_entry->finalize(); + manifest_lengths += buffer_manifest_entry->count(); + } + catch (...) + { + cleanup(false); + throw; + } + + { + auto buffer_manifest_list = object_storage->writeObject( + StoredObject(storage_manifest_list_name), WriteMode::Rewrite, std::nullopt, DBMS_DEFAULT_BUFFER_SIZE, context->getWriteSettings()); + + try + { + generateManifestList( + resolver, metadata, object_storage, context, {manifest_entry_path}, new_snapshot, {manifest_lengths}, *buffer_manifest_list, Iceberg::FileContentType::DATA, true); + buffer_manifest_list->finalize(); + } + catch (...) + { + cleanup(false); + throw; + } + } + + { + std::ostringstream oss; // STYLE_CHECK_ALLOW_STD_STRING_STREAM + Poco::JSON::Stringifier::stringify(metadata, oss, 4); + std::string json_representation = removeEscapedSlashes(oss.str()); + + LOG_DEBUG(log, "Writing new metadata file {}", storage_metadata_name); + auto hint_path = filename_generator.generateVersionHint(); + if (!writeMetadataFileAndVersionHint( + resolver, + metadata_info, + json_representation, + hint_path, + object_storage, + context, + data_lake_settings[DataLakeStorageSetting::iceberg_use_version_hint])) + { + LOG_DEBUG(log, "Failed to write metadata {}, retrying", storage_metadata_name); + cleanup(true); + return {}; + } + + LOG_DEBUG(log, "Metadata file {} written", storage_metadata_name); + + if (catalog) + { + String catalog_filename = metadata_info.path.serialize(); + if (!catalog_filename.starts_with(blob_storage_type_name)) + catalog_filename = blob_storage_type_name + "://" + blob_storage_namespace_name + "/" + catalog_filename; + + const auto & [namespace_name, table_name] = DataLake::parseTableName(table_id.getTableName()); + if (!catalog->updateMetadata(namespace_name, table_name, catalog_filename, new_snapshot)) + { + cleanup(true); + return {}; + } + + /// Catalog has accepted the commit - the new snapshot is now live and references + /// storage_manifest_entry_name / storage_manifest_list_name. From here on, any + /// failure must NOT delete those files. + published = true; + } + else + { + /// File-based layout: the snapshot becomes visible via the metadata file and + /// version hint that were just written above. From here on, any failure must + /// NOT delete manifest entry / manifest list. + published = true; + } + } + + /// Fault-injection hook that simulates an exception in the trailing post-publish + /// region (e.g. failure in metadata-cache invalidation). Must be placed AFTER + /// `published = true` to exercise the exception-safety guard in the outer catch. + fiu_do_on(FailPoints::iceberg_writes_post_publish_throw, + { + throw Exception(ErrorCodes::BAD_ARGUMENTS, "Failpoint iceberg_writes_post_publish_throw enabled"); + }); + + if (persistent_components.metadata_cache) + { + /// If there's an active metadata cache + /// We can't just cache 'our' written version as latest, because it could've been overwritten by a concurrent catalog update + /// This is why, we are safely invalidating the cache, and the very next reader will get the most up-to-date latest version + persistent_components.metadata_cache->remove(persistent_components.table_path); + if (persistent_components.table_uuid) + persistent_components.metadata_cache->remove(*persistent_components.table_uuid); + } + } + catch (...) + { + if (published) + { + /// Commit has already become visible to readers. The failure is in trailing + /// post-publish work (e.g. metadata-cache invalidation). Running cleanup() + /// here would delete manifest files referenced by the published snapshot + /// and corrupt it. Log and swallow - any transient state (stale cache) + /// is self-healing on subsequent reads. Surface the published paths anyway + /// so the partition export task can persist them in ZooKeeper. + tryLogCurrentException(log, + "Post-publish work failed after Iceberg snapshot was committed; " + "skipping manifest cleanup to preserve published snapshot"); + IStorage::ExportPartitionCommitInfo published_info; + published_info.iceberg_metadata_file = resolver.resolve(metadata_info.path); + published_info.iceberg_manifest_list = storage_manifest_list_name; + published_info.iceberg_manifest_file = storage_manifest_entry_name; + return published_info; + } + + LOG_ERROR(log, "Failed to commit import partition transaction: {}", getCurrentExceptionMessage(false)); + cleanup(false); + throw; + } + + /// Record the storage paths of the files we just published so the partition + /// export task can persist them in ZooKeeper for observability. Only set here + /// (not on the retry / "already committed" paths) so the struct reflects + /// exactly what this attempt produced. + IStorage::ExportPartitionCommitInfo published_info; + published_info.iceberg_metadata_file = resolver.resolve(metadata_info.path); + published_info.iceberg_manifest_list = storage_manifest_list_name; + published_info.iceberg_manifest_file = storage_manifest_entry_name; + return published_info; +} + +IStorage::ExportPartitionCommitInfo IcebergMetadata::commitExportPartitionTransaction( + std::shared_ptr catalog, + const StorageID & table_id, + const String & transaction_id, + Int64 original_schema_id, + Int64 partition_spec_id, + const Block & partition_source_block, + SharedHeader sample_block, + const std::vector & data_file_paths, + StorageObjectStorageConfigurationPtr configuration, + ContextPtr context) +{ + + MetadataFileWithInfo updated_metadata_file_info = getLatestOrExplicitMetadataFileAndVersion( + object_storage, + persistent_components.table_path, + data_lake_settings, + persistent_components.metadata_cache, + context, + getLogger("IcebergMetadata").get(), + persistent_components.table_uuid, + persistent_components.metadata_compression_method, + true); + + /// Latest metadata is ALWAYS necessary to commit - but we abort in case schema or partition spec changed + Poco::JSON::Object::Ptr metadata = getMetadataJSONObject( + updated_metadata_file_info.path, + object_storage, + persistent_components.metadata_cache, + context, + getLogger("IcebergMetadata"), + updated_metadata_file_info.compression_method, + persistent_components.table_uuid); + + if (isExportPartitionTransactionAlreadyCommitted(metadata, transaction_id)) + { + LOG_INFO(log, + "Export transaction {} already committed, skipping re-commit", + transaction_id); + IStorage::ExportPartitionCommitInfo already_committed_info; + already_committed_info.iceberg_metadata_file = ""; + return already_committed_info; + } + + /// Fail fast if the table schema or partition spec changed between export-start and commit. + /// The exported data files and partition values were produced against the original spec; + const auto latest_schema_id = metadata->getValue(Iceberg::f_current_schema_id); + if (latest_schema_id != original_schema_id) + throw Exception(ErrorCodes::METADATA_MISMATCH, + "Table schema changed before export could commit (expected schema {}, got {}). " + "Restart the export operation.", + original_schema_id, latest_schema_id); + + const auto latest_spec_id = metadata->getValue(Iceberg::f_default_spec_id); + if (latest_spec_id != partition_spec_id) + throw Exception(ErrorCodes::METADATA_MISMATCH, + "Partition spec changed before export could commit (expected spec {}, got {}). " + "Restart the export operation.", + partition_spec_id, latest_spec_id); + + /// Derive partition_columns and partition_types from the schema and partition spec. + /// The IDs are validated equal above so derivation from the latest metadata yields + /// the same result as from the original ZK-pinned snapshot. + + const auto schema = lookupSchema(metadata, original_schema_id); + + auto partition_spec = lookupPartitionSpec(metadata, partition_spec_id); + + ChunkPartitioner partitioner(partition_spec->getArray(Iceberg::f_fields), schema->getArray(Iceberg::f_fields), context, sample_block); + + const auto partition_columns = partitioner.getColumns(); + const auto partition_types = partitioner.getResultTypes(); + + /// Recompute the partition tuple via the destination transform so the metadata partition + /// value matches the exported data (rather than the raw source MergeTree partition value). + const auto partition_values = recomputeExportPartitionValues(partitioner, sample_block, partition_source_block); + + const auto metadata_compression_method = persistent_components.metadata_compression_method; + + /// Generated paths are always expressed relative to the table location, the conversion + /// to a real storage path is done by `persistent_components.path_resolver`. + FileNamesGenerator filename_generator( + persistent_components.path_resolver.getTableLocation(), + (catalog != nullptr && catalog->isTransactional()), + metadata_compression_method, + write_format); + + filename_generator.setVersion(updated_metadata_file_info.version + 1); + + /// Load per-file sidecar stats, necessary to populate the manifest file stats. + std::vector per_file_stats; + const Int64 total_data_files = static_cast(data_file_paths.size()); + Int64 total_rows = 0; + Int64 total_chunks_size = 0; + per_file_stats.reserve(data_file_paths.size()); + for (const auto & path : data_file_paths) + { + const auto sidecar_path = getIcebergExportPartSidecarStoragePath(path); + auto stats = readDataFileSidecar(sidecar_path, object_storage, context); + total_rows += stats.record_count; + total_chunks_size += stats.file_size_in_bytes; + + per_file_stats.push_back(std::move(stats)); + } + + size_t attempt = 0; + while (attempt < MAX_TRANSACTION_RETRIES) + { + auto commit_info = commitImportPartitionTransactionImpl( + filename_generator, + metadata, + partition_spec, + transaction_id, + original_schema_id, + partition_spec_id, + partition_values, + partition_columns, + partition_types, + sample_block, + data_file_paths, + per_file_stats, + total_data_files, + total_rows, + total_chunks_size, + catalog, + table_id, + configuration->getTypeName(), + configuration->getNamespace(), + context); + + if (commit_info) + return *commit_info; + + ++attempt; + } + + throw Exception(ErrorCodes::UNFINISHED, + "Failed to commit export partition transaction after {} attempts due to repeated metadata conflicts.", + attempt); +} + +Poco::JSON::Object::Ptr IcebergMetadata::getMetadataJSON(ContextPtr local_context) const +{ + auto [actual_data_snapshot, actual_table_state_snapshot] = getRelevantState(local_context); + return getMetadataJSONObject( + actual_table_state_snapshot.metadata_file_path, + object_storage, + persistent_components.metadata_cache, + local_context, + log, + persistent_components.metadata_compression_method, + persistent_components.table_uuid); +} + } #endif diff --git a/src/Storages/ObjectStorage/DataLakes/Iceberg/IcebergMetadata.h b/src/Storages/ObjectStorage/DataLakes/Iceberg/IcebergMetadata.h index efd9c0d60029..022f9a73af4e 100644 --- a/src/Storages/ObjectStorage/DataLakes/Iceberg/IcebergMetadata.h +++ b/src/Storages/ObjectStorage/DataLakes/Iceberg/IcebergMetadata.h @@ -1,4 +1,6 @@ #pragma once +#include "Storages/ObjectStorage/DataLakes/Iceberg/ChunkPartitioner.h" +#include "Storages/ObjectStorage/DataLakes/Iceberg/FileNamesGenerator.h" #include "config.h" #if USE_AVRO @@ -23,6 +25,7 @@ #include +#include #include #include #include @@ -101,6 +104,7 @@ class IcebergMetadata : public IDataLakeMetadata static Int32 parseTableSchema( const Poco::JSON::Object::Ptr & metadata_object, Iceberg::IcebergSchemaProcessor & schema_processor, + ContextPtr context_, LoggerPtr metadata_logger); bool supportsUpdate() const override { return true; } @@ -138,6 +142,38 @@ class IcebergMetadata : public IDataLakeMetadata ContextPtr context, std::shared_ptr catalog) override; + bool supportsImport(ContextPtr) const override { return true; } + + SinkToStoragePtr import( + std::shared_ptr catalog, + const std::function & new_file_path_callback, + SharedHeader sample_block, + const std::string & iceberg_metadata_json_string, + const std::optional & format_settings, + ContextPtr context) override; + + /// Commit an export-partition transaction. All parameters that are saved in ZooKeeper at the + /// start of the export operation (schema_id, partition_spec_id, partition_values, + /// partition_columns, partition_types) must be provided by the caller. + /// The partition spec object is derived from the metadata using partition_spec_id. + /// If the live metadata has diverged (schema or partition spec changed) the call throws + /// immediately — the caller must restart from scratch. + /// + /// data_file_paths contains the metadata-path for each exported data file (as recorded in + /// ZooKeeper). For every path a co-located sidecar Avro file (same path, ".avro" extension) + /// must exist in the object storage; it supplies record_count and file_size_in_bytes. + IStorage::ExportPartitionCommitInfo commitExportPartitionTransaction( + std::shared_ptr catalog, + const StorageID & table_id, + const String & transaction_id, + Int64 original_schema_id, + Int64 partition_spec_id, + const Block & partition_source_block, + SharedHeader sample_block, + const std::vector & data_file_paths, + StorageObjectStorageConfigurationPtr configuration, + ContextPtr context) override; + CompressionMethod getCompressionMethod() const { return persistent_components.metadata_compression_method; } bool optimize(const StorageMetadataPtr & metadata_snapshot, ContextPtr context, const std::optional & format_settings) override; @@ -198,6 +234,11 @@ class IcebergMetadata : public IDataLakeMetadata return persistent_components; } + Poco::JSON::Object::Ptr getMetadataJSON(ContextPtr local_context) const; + + std::optional partitionKey(ContextPtr) const override; + std::optional sortingKey(ContextPtr) const override; + private: static Iceberg::PersistentTableComponents initializePersistentTableComponents( ObjectStoragePtr object_storage, @@ -222,6 +263,36 @@ class IcebergMetadata : public IDataLakeMetadata /// scoped to the queried path reaches beyond this table there. void checkTableRootIsQueriedPath(std::string_view operation) const; + /// Non-empty return value means the attempt succeeded (covers both the normal + /// publish path and the `isExportPartitionTransactionAlreadyCommitted` short-circuit). + /// An empty `ExportPartitionCommitInfo` means the caller must retry. The + /// short-circuit branch fills `iceberg_metadata_file` with a sentinel note since + /// the original committer's paths are not trivially recoverable from inside this call. + std::optional commitImportPartitionTransactionImpl( + FileNamesGenerator & filename_generator, + Poco::JSON::Object::Ptr & metadata, + Poco::JSON::Object::Ptr & partition_spec, + const String & transaction_id, + Int64 original_schema_id, + Int64 partition_spec_id, + const std::vector & partition_values, + const std::vector & partition_columns, + const DataTypes & partition_types, + SharedHeader sample_block, + const std::vector & data_file_paths, + const std::vector & per_file_stats, + Int64 total_data_files, + Int64 total_rows, + Int64 total_chunks_size, + std::shared_ptr catalog, + const StorageID & table_id, + const String & blob_storage_type_name, + const String & blob_storage_namespace_name, + ContextPtr context); + + std::optional getPartitionKey(ContextPtr local_context, Iceberg::TableStateSnapshot actual_table_state_snapshot) const; + KeyDescription getSortingKey(ContextPtr local_context, Iceberg::TableStateSnapshot actual_table_state_snapshot) const; + LoggerPtr log; const ObjectStoragePtr object_storage; const DB::Iceberg::PersistentTableComponents persistent_components; @@ -230,8 +301,6 @@ class IcebergMetadata : public IDataLakeMetadata BackgroundSchedulePoolTaskHolder background_metadata_prefetch_task; ObjectIterator prepared_iterator; - KeyDescription getSortingKey(ContextPtr local_context, Iceberg::TableStateSnapshot actual_table_state_snapshot) const; - void backgroundMetadataPrefetcherThread(); }; } diff --git a/src/Storages/ObjectStorage/DataLakes/Iceberg/IcebergWrites.cpp b/src/Storages/ObjectStorage/DataLakes/Iceberg/IcebergWrites.cpp index a83d8b5f3278..1ce3182af3b5 100644 --- a/src/Storages/ObjectStorage/DataLakes/Iceberg/IcebergWrites.cpp +++ b/src/Storages/ObjectStorage/DataLakes/Iceberg/IcebergWrites.cpp @@ -315,6 +315,16 @@ bool canWriteStatistics( } +String getIcebergExportPartSidecarStoragePath(const String & data_file_storage_path) +{ + static constexpr auto postfix = "_clickhouse_export_part_sidecar.avro"; + auto dot_pos = data_file_storage_path.rfind('.'); + auto slash_pos = data_file_storage_path.rfind('/'); + if (dot_pos != String::npos && (slash_pos == String::npos || dot_pos > slash_pos)) + return data_file_storage_path.substr(0, dot_pos) + postfix; + return data_file_storage_path + postfix; +} + String removeEscapedSlashes(const String & json_str) { size_t pos = json_str.find("\\/"); @@ -345,6 +355,160 @@ String stringifyJSON(const Poco::Dynamic::Var & json, unsigned indent) return removeEscapedSlashes(oss.str()); } +IcebergSerializedFileStats readDataFileSidecar( + const String & sidecar_storage_path, + const ObjectStoragePtr & object_storage, + const ContextPtr & context) +{ + auto buf = object_storage->readObject(StoredObject(sidecar_storage_path), context->getReadSettings()); + auto input_stream = std::make_unique(*buf); + avro::DataFileReader reader(std::move(input_stream)); + + avro::GenericDatum datum(reader.readerSchema()); + if (!reader.read(datum)) + throw Exception( + ErrorCodes::BAD_ARGUMENTS, + "Data file sidecar '{}' contains no records", + sidecar_storage_path); + + const auto & record = datum.value(); + IcebergSerializedFileStats result; + result.record_count = record.field("record_count").value(); + result.file_size_in_bytes = record.field("file_size_in_bytes").value(); + + auto read_long_map = [&](const std::string & name, std::vector> & out) + { + const auto & arr = record.field(name).value().value(); + for (const auto & item : arr) + { + const auto & r = item.value(); + out.emplace_back(r.field("key").value(), r.field("value").value()); + } + }; + + auto read_bytes_map = [&](const std::string & name, std::vector>> & out) + { + const auto & arr = record.field(name).value().value(); + for (const auto & item : arr) + { + const auto & r = item.value(); + out.emplace_back(r.field("key").value(), r.field("value").value>()); + } + }; + + read_long_map("column_sizes", result.column_sizes); + read_long_map("null_value_counts", result.null_value_counts); + read_bytes_map("lower_bounds", result.lower_bounds); + read_bytes_map("upper_bounds", result.upper_bounds); + + return result; +} + +void writeDataFileSidecar( + const String & data_file_storage_path, + const IcebergSerializedFileStats & stats, + const ObjectStoragePtr & object_storage, + const ContextPtr & context) +{ + const String sidecar_path = getIcebergExportPartSidecarStoragePath(data_file_storage_path); + auto buf = object_storage->writeObject( + StoredObject(sidecar_path), WriteMode::Rewrite, std::nullopt, DBMS_DEFAULT_BUFFER_SIZE, context->getWriteSettings()); + + { + auto schema = avro::compileJsonSchemaFromString(data_file_sidecar_schema); + auto adapter = std::make_unique(*buf); + avro::DataFileWriter writer(std::move(adapter), schema); + + avro::GenericDatum datum(schema.root()); + avro::GenericRecord & rec = datum.value(); + rec.field("record_count") = avro::GenericDatum(stats.record_count); + rec.field("file_size_in_bytes") = avro::GenericDatum(stats.file_size_in_bytes); + + auto write_long_map = [&](const std::string & name, const std::vector> & entries) + { + auto & field = rec.field(name); + auto & arr = field.value(); + auto schema_element = arr.schema()->leafAt(0); + for (const auto & [k, v] : entries) + { + avro::GenericDatum item(schema_element); + auto & item_rec = item.value(); + item_rec.field("key") = avro::GenericDatum(k); + item_rec.field("value") = avro::GenericDatum(v); + arr.value().push_back(item); + } + }; + + auto write_bytes_map = [&](const std::string & name, const std::vector>> & entries) + { + auto & field = rec.field(name); + auto & arr = field.value(); + auto schema_element = arr.schema()->leafAt(0); + for (const auto & [k, v] : entries) + { + avro::GenericDatum item(schema_element); + auto & item_rec = item.value(); + item_rec.field("key") = avro::GenericDatum(k); + item_rec.field("value") = avro::GenericDatum(v); + arr.value().push_back(item); + } + }; + + write_long_map("column_sizes", stats.column_sizes); + write_long_map("null_value_counts", stats.null_value_counts); + write_bytes_map("lower_bounds", stats.lower_bounds); + write_bytes_map("upper_bounds", stats.upper_bounds); + + writer.write(datum); + writer.flush(); + // writer destructor writes the Avro end-of-file sync marker + } + + buf->finalize(); +} + +/// vibe coded - needs extra attention +IcebergSerializedFileStats serializeDataFileStats( + const DataFileStatistics & stats, + SharedHeader sample_block, + Int64 record_count, + Int64 file_size_in_bytes) +{ + IcebergSerializedFileStats result; + result.record_count = record_count; + result.file_size_in_bytes = file_size_in_bytes; + + for (const auto & [field_id, sz] : stats.getColumnSizes()) + result.column_sizes.emplace_back(static_cast(field_id), static_cast(sz)); + + for (const auto & [field_id, cnt] : stats.getNullCounts()) + result.null_value_counts.emplace_back(static_cast(field_id), static_cast(cnt)); + + std::unordered_map field_id_to_col_idx; + { + auto field_ids = stats.getFieldIds(); + for (size_t i = 0; i < field_ids.size(); ++i) + field_id_to_col_idx[field_ids[i]] = i; + } + + auto serialize_bounds = [&](const std::vector> & bounds, + std::vector>> & out) + { + if (!canWriteStatistics(bounds, field_id_to_col_idx, sample_block)) + return; + for (const auto & [field_id, value] : bounds) + { + auto bytes = dumpFieldToBytes(value, sample_block->getDataTypes()[field_id_to_col_idx.at(field_id)]); + out.emplace_back(static_cast(field_id), std::move(bytes)); + } + }; + + serialize_bounds(stats.getLowerBounds(), result.lower_bounds); + serialize_bounds(stats.getUpperBounds(), result.upper_bounds); + + return result; +} + static void extendSchemaForPartitions( String & schema, const std::vector & partition_columns, @@ -451,7 +615,8 @@ void generateManifestFile( const std::vector> & data_file_sort_order_ids, const std::vector & per_file_entry_lineage, Poco::JSON::Object::Ptr schema_to_serialize, - const std::vector * per_file_fresh_statistics) + const std::vector * per_file_fresh_statistics, + const std::vector & per_file_stats) { /// A throw, not a `chassert`: mis-pairing statistics with data files publishes metadata that is /// wrong in the unsafe direction for external readers, and `chassert` compiles out of release builds. @@ -581,48 +746,102 @@ void generateManifestFile( } }; - if (!per_file_statistics.empty()) + /// Export path: per-file serialized stats override everything (record count, file size, + /// and all column statistics). Existing insert/mutation paths use the branches below. + if (!per_file_stats.empty() && file_idx < per_file_stats.size()) { - /// Manifest-only rewrite: carry over the source file's column stats verbatim. - const auto & stats = per_file_statistics[file_idx]; - /// Bounds are raw bytes; convert to std::vector to produce an Avro `bytes` datum. - auto to_bytes = [](Int32, const String & value) - { return std::vector(value.begin(), value.end()); }; - set_fields(stats.column_sizes, Iceberg::f_column_sizes, [](Int32, Int64 value) { return value; }); - set_fields(stats.value_counts, Iceberg::f_value_counts, [](Int32, Int64 value) { return value; }); - set_fields(stats.null_value_counts, Iceberg::f_null_value_counts, [](Int32, Int64 value) { return value; }); - set_fields(stats.lower_bounds, Iceberg::f_lower_bounds, to_bytes); - set_fields(stats.upper_bounds, Iceberg::f_upper_bounds, to_bytes); - } - else if (effective_statistics) - { - auto statistics = effective_statistics->getColumnSizes(); - set_fields(statistics, Iceberg::f_column_sizes, [](size_t, size_t value) { return static_cast(value); }); + const auto & pf = per_file_stats[file_idx]; - statistics = effective_statistics->getNullCounts(); - set_fields(statistics, Iceberg::f_null_value_counts, [](size_t, size_t value) { return static_cast(value); }); + auto write_long_map = [&](const std::vector> & entries, const String & field_name) + { + if (entries.empty()) + return; + auto & field = data_file.field(field_name); + field.selectBranch(1); + auto & arr = field.value(); + auto schema_element = arr.schema()->leafAt(0); + for (const auto & [k, v] : entries) + { + avro::GenericDatum item(schema_element); + auto & item_rec = item.value(); + item_rec.field(Iceberg::f_key) = avro::GenericDatum(k); + item_rec.field(Iceberg::f_value) = avro::GenericDatum(v); + arr.value().push_back(item); + } + }; - std::unordered_map field_id_to_column_index; - auto field_ids = effective_statistics->getFieldIds(); - for (size_t i = 0; i < field_ids.size(); ++i) - field_id_to_column_index[field_ids[i]] = i; + auto write_bytes_map = [&](const std::vector>> & entries, const String & field_name) + { + if (entries.empty()) + return; + auto & field = data_file.field(field_name); + field.selectBranch(1); + auto & arr = field.value(); + auto schema_element = arr.schema()->leafAt(0); + for (const auto & [k, v] : entries) + { + avro::GenericDatum item(schema_element); + auto & item_rec = item.value(); + item_rec.field(Iceberg::f_key) = avro::GenericDatum(k); + item_rec.field(Iceberg::f_value) = avro::GenericDatum(v); + arr.value().push_back(item); + } + }; - auto dump_fields = [&](size_t field_id, Field value) - { return dumpFieldToBytes(value, sample_block->getDataTypes()[field_id_to_column_index.at(field_id)]); }; + write_long_map(pf.column_sizes, Iceberg::f_column_sizes); + write_long_map(pf.null_value_counts, Iceberg::f_null_value_counts); + write_bytes_map(pf.lower_bounds, Iceberg::f_lower_bounds); + write_bytes_map(pf.upper_bounds, Iceberg::f_upper_bounds); - auto lower_statistics = effective_statistics->getLowerBounds(); - if (canWriteStatistics(lower_statistics, field_id_to_column_index, sample_block)) + data_file.field(Iceberg::f_record_count) = avro::GenericDatum(pf.record_count); + data_file.field(Iceberg::f_file_size_in_bytes) = avro::GenericDatum(pf.file_size_in_bytes); + } + else + { + if (!per_file_statistics.empty()) { - set_fields(lower_statistics, Iceberg::f_lower_bounds, dump_fields); + /// Manifest-only rewrite: carry over the source file's column stats verbatim. + const auto & stats = per_file_statistics[file_idx]; + /// Bounds are raw bytes; convert to std::vector to produce an Avro `bytes` datum. + auto to_bytes = [](Int32, const String & value) + { return std::vector(value.begin(), value.end()); }; + set_fields(stats.column_sizes, Iceberg::f_column_sizes, [](Int32, Int64 value) { return value; }); + set_fields(stats.value_counts, Iceberg::f_value_counts, [](Int32, Int64 value) { return value; }); + set_fields(stats.null_value_counts, Iceberg::f_null_value_counts, [](Int32, Int64 value) { return value; }); + set_fields(stats.lower_bounds, Iceberg::f_lower_bounds, to_bytes); + set_fields(stats.upper_bounds, Iceberg::f_upper_bounds, to_bytes); } - auto upper_statistics = effective_statistics->getUpperBounds(); - if (canWriteStatistics(upper_statistics, field_id_to_column_index, sample_block)) + else if (effective_statistics) { - set_fields(upper_statistics, Iceberg::f_upper_bounds, dump_fields); + auto statistics = effective_statistics->getColumnSizes(); + set_fields(statistics, Iceberg::f_column_sizes, [](size_t, size_t value) { return static_cast(value); }); + + statistics = effective_statistics->getNullCounts(); + set_fields(statistics, Iceberg::f_null_value_counts, [](size_t, size_t value) { return static_cast(value); }); + + std::unordered_map field_id_to_column_index; + auto field_ids = effective_statistics->getFieldIds(); + for (size_t i = 0; i < field_ids.size(); ++i) + field_id_to_column_index[field_ids[i]] = i; + + auto dump_fields = [&](size_t field_id, Field value) + { return dumpFieldToBytes(value, sample_block->getDataTypes()[field_id_to_column_index.at(field_id)]); }; + + auto lower_statistics = effective_statistics->getLowerBounds(); + if (canWriteStatistics(lower_statistics, field_id_to_column_index, sample_block)) + { + set_fields(lower_statistics, Iceberg::f_lower_bounds, dump_fields); + } + auto upper_statistics = effective_statistics->getUpperBounds(); + if (canWriteStatistics(upper_statistics, field_id_to_column_index, sample_block)) + { + set_fields(upper_statistics, Iceberg::f_upper_bounds, dump_fields); + } } + + data_file.field(Iceberg::f_record_count) = avro::GenericDatum(static_cast(data_file_row_counts[file_idx])); + data_file.field(Iceberg::f_file_size_in_bytes) = avro::GenericDatum(static_cast(data_file_byte_counts[file_idx])); } - data_file.field(Iceberg::f_record_count) = avro::GenericDatum(static_cast(data_file_row_counts[file_idx])); - data_file.field(Iceberg::f_file_size_in_bytes) = avro::GenericDatum(static_cast(data_file_byte_counts[file_idx])); /// Preserve the source file's sort_order_id. if (!data_file_sort_order_ids.empty() && data_file_sort_order_ids[file_idx].has_value()) @@ -1011,7 +1230,7 @@ IcebergStorageSink::IcebergStorageSink( , table_id(table_id_) , persistent_table_components(persistent_table_components_) , data_lake_settings(configuration_->getDataLakeSettings()) - , write_format(configuration_->format) + , write_format(configuration_->getFormat()) { auto [last_version, metadata_path, compression_method] = getLatestMetadataFileAndVersionWithCatalog( object_storage, @@ -1523,6 +1742,142 @@ bool IcebergStorageSink::initializeMetadata() return true; } +IcebergImportSink::IcebergImportSink( + std::shared_ptr catalog_, + const Iceberg::PersistentTableComponents & persistent_table_components_, + Poco::JSON::Object::Ptr metadata_json_, + ObjectStoragePtr object_storage_, + ContextPtr context_, + std::optional format_settings_, + const String & write_format_, + SharedHeader sample_block_, + const DataLakeStorageSettings & data_lake_settings_, + std::function new_file_path_callback_) + : SinkToStorage(sample_block_) + , catalog(catalog_) + , persistent_table_components(persistent_table_components_) + , metadata_json(metadata_json_) + , object_storage(object_storage_) + , context(context_) + , format_settings(format_settings_) + , write_format(write_format_) + , sample_block(sample_block_) + , data_lake_settings(data_lake_settings_) + , new_file_path_callback(std::move(new_file_path_callback_)) +{ + const auto current_schema_id = metadata_json->getValue(Iceberg::f_current_schema_id); + const auto schemas = metadata_json->getArray(Iceberg::f_schemas); + + for (size_t i = 0; i < schemas->size(); ++i) + { + if (schemas->getObject(static_cast(i))->getValue(Iceberg::f_schema_id) == current_schema_id) + { + current_schema = schemas->getObject(static_cast(i)); + break; + } + } + + const auto metadata_compression_method = persistent_table_components.metadata_compression_method; + + /// Paths written into Iceberg metadata are always built from the table location, + /// the conversion to the actual storage path is done by the path resolver. + filename_generator = FileNamesGenerator( + persistent_table_components.path_resolver.getTableLocation(), + (catalog != nullptr && catalog->isTransactional()), + metadata_compression_method, + write_format); + + const auto [last_version, unused_meta_path, unused_compression] = getLatestOrExplicitMetadataFileAndVersion( + object_storage, + persistent_table_components.table_path, + data_lake_settings, + persistent_table_components.metadata_cache, + context_, + getLogger("IcebergWrites").get(), + persistent_table_components.table_uuid, + metadata_compression_method, + true); + (void)unused_meta_path; + (void)unused_compression; + + filename_generator.setVersion(last_version + 1); + + writer = std::make_unique( + context->getSettingsRef()[Setting::iceberg_insert_max_rows_in_data_file], + context->getSettingsRef()[Setting::iceberg_insert_max_bytes_in_data_file], + current_schema->getArray(Iceberg::f_fields), + filename_generator, + persistent_table_components.path_resolver, + object_storage, + context, + format_settings, + write_format, + sample_block, + new_file_path_callback); +} + +IcebergImportSink::~IcebergImportSink() +{ + cancelBuffers(); +} + +void IcebergImportSink::consume(Chunk & chunk) +{ + if (isCancelled()) + return; + + writer->consume(chunk); +} + +void IcebergImportSink::onFinish() +{ + if (isCancelled()) + { + cancelBuffers(); + return; + } + + finalizeBuffers(); + + for (const auto & entry : writer->getDataFileEntries()) + { + IcebergSerializedFileStats serialized_stats; + if (entry.statistics) + { + serialized_stats = serializeDataFileStats(*entry.statistics, sample_block, entry.record_count, entry.file_size_in_bytes); + } + else + { + serialized_stats.record_count = entry.record_count; + serialized_stats.file_size_in_bytes = entry.file_size_in_bytes; + } + + writeDataFileSidecar(entry.path, serialized_stats, object_storage, context); + } + + releaseBuffers(); +} + +void IcebergImportSink::onException(std::exception_ptr /* exception */) +{ + cancelBuffers(); +} + +void IcebergImportSink::finalizeBuffers() +{ + writer->finalize(); +} + +void IcebergImportSink::releaseBuffers() +{ + writer->release(); +} + +void IcebergImportSink::cancelBuffers() +{ + writer->cancel(); +} + } // NOLINTEND(clang-analyzer-core.uninitialized.UndefReturn) diff --git a/src/Storages/ObjectStorage/DataLakes/Iceberg/IcebergWrites.h b/src/Storages/ObjectStorage/DataLakes/Iceberg/IcebergWrites.h index efad297beefb..1d3e8c421646 100644 --- a/src/Storages/ObjectStorage/DataLakes/Iceberg/IcebergWrites.h +++ b/src/Storages/ObjectStorage/DataLakes/Iceberg/IcebergWrites.h @@ -25,6 +25,7 @@ #include #include #include +#include #include #include @@ -65,6 +66,37 @@ struct DataFileEntryLineage std::optional file_sequence_number; }; +/// Read a data-file sidecar and return its contents in Iceberg wire format. +/// The returned struct carries the row count, byte size, and per-column statistics. +IcebergSerializedFileStats readDataFileSidecar( + const String & sidecar_storage_path, + const ObjectStoragePtr & object_storage, + const ContextPtr & context); + +/// Write a sidecar Avro file alongside a data file. +/// All six fields are written; empty stat vectors are valid when statistics are unavailable. +void writeDataFileSidecar( + const String & data_file_storage_path, + const IcebergSerializedFileStats & stats, + const ObjectStoragePtr & object_storage, + const ContextPtr & context); + +/// Convert in-memory DataFileStatistics (ClickHouse-internal) to the Iceberg wire format. +/// Bounds are serialized to bytes using the same encoding used in the manifest file, +/// so the result can be stored in sidecar Avro files and used at commit time on any node. +IcebergSerializedFileStats serializeDataFileStats( + const DataFileStatistics & stats, + SharedHeader sample_block, + Int64 record_count, + Int64 file_size_in_bytes); + +/// Generate an Iceberg manifest file for a set of data files. +/// +/// \param data_file_statistics Aggregate column statistics applied to every file (regular +/// INSERT and mutation paths). Ignored when \p per_file_stats is non-empty. +/// \param per_file_stats Per-file pre-serialized statistics (export-commit path). +/// When non-empty each entry overrides both the record count / file size AND the column +/// statistics for the corresponding file. Leave empty to preserve the existing behaviour. void generateManifestFile( Poco::JSON::Object::Ptr metadata, const std::vector & partition_columns, @@ -99,7 +131,10 @@ void generateManifestFile( Poco::JSON::Object::Ptr schema_to_serialize = nullptr, /// Optional freshly-computed per-file statistics parallel to `data_file_names`; when set each entry's stats /// describe only its own data file, else the shared `data_file_statistics` is used for every entry. - const std::vector * per_file_fresh_statistics = nullptr); + const std::vector * per_file_fresh_statistics = nullptr, + /// Optional per-file pre-serialized statistics (export-commit path). When non-empty each entry + /// overrides both the record count / file size AND the column statistics for the corresponding file. + const std::vector & per_file_stats = {}); /// Per manifest-list entry file/row counts and lineage for rewritten manifests. struct ManifestListEntryCounts @@ -137,6 +172,8 @@ void generateManifestList( const std::vector & entry_partition_spec_ids = {}, const std::vector>> & entry_partition_summaries = {}); +std::string getIcebergExportPartSidecarStoragePath(const String & data_file_storage_path); + class IcebergStorageSink final : public SinkToStorage { public: @@ -195,6 +232,50 @@ class IcebergStorageSink final : public SinkToStorage }; +class IcebergImportSink : public SinkToStorage +{ +public: + IcebergImportSink( + std::shared_ptr catalog_, + const Iceberg::PersistentTableComponents & persistent_table_components_, + Poco::JSON::Object::Ptr metadata_json_, + ObjectStoragePtr object_storage_, + ContextPtr context_, + std::optional format_settings_, + const String & write_format_, + SharedHeader sample_block_, + const DataLakeStorageSettings & data_lake_settings_, + std::function new_file_path_callback_ = {}); + + ~IcebergImportSink() override; + + String getName() const override { return "IcebergImportSink"; } + + void consume(Chunk & chunk) override; + + void onFinish() override; + void onException(std::exception_ptr exception) override; + +private: + void finalizeBuffers(); + void releaseBuffers(); + void cancelBuffers(); + + std::shared_ptr catalog; + const Iceberg::PersistentTableComponents & persistent_table_components; + Poco::JSON::Object::Ptr metadata_json; + Poco::JSON::Object::Ptr current_schema; + FileNamesGenerator filename_generator; + ObjectStoragePtr object_storage; + ContextPtr context; + std::optional format_settings; + const String& write_format; + SharedHeader sample_block; + std::unique_ptr writer; + const DataLakeStorageSettings & data_lake_settings; + std::function new_file_path_callback; +}; + } #endif diff --git a/src/Storages/ObjectStorage/DataLakes/Iceberg/ManifestFile.h b/src/Storages/ObjectStorage/DataLakes/Iceberg/ManifestFile.h index 2d9f2389bf90..7b49fdd76d09 100644 --- a/src/Storages/ObjectStorage/DataLakes/Iceberg/ManifestFile.h +++ b/src/Storages/ObjectStorage/DataLakes/Iceberg/ManifestFile.h @@ -1,6 +1,21 @@ #pragma once #include "config.h" +#include + +#include + +namespace DB::Iceberg +{ + +struct ColumnInfo +{ + std::optional rows_count; + std::optional bytes_size; + std::optional nulls_count; +}; + +} #if USE_AVRO @@ -44,13 +59,6 @@ enum class ManifestFileContentType String FileContentTypeToString(FileContentType type); -struct ColumnInfo -{ - std::optional rows_count; - std::optional bytes_size; - std::optional nulls_count; -}; - struct PartitionSpecsEntry { Int32 source_id; diff --git a/src/Storages/ObjectStorage/DataLakes/Iceberg/ManifestFileIterator.cpp b/src/Storages/ObjectStorage/DataLakes/Iceberg/ManifestFileIterator.cpp index b45dfa4f832c..cecfc51c3c4f 100644 --- a/src/Storages/ObjectStorage/DataLakes/Iceberg/ManifestFileIterator.cpp +++ b/src/Storages/ObjectStorage/DataLakes/Iceberg/ManifestFileIterator.cpp @@ -7,16 +7,17 @@ #include #include -#include - +#include #include #include +#include #include #include #include #include +#include #include #include #include @@ -39,6 +40,11 @@ namespace DB::ErrorCodes extern const int BAD_ARGUMENTS; } +namespace DB::Setting +{ + extern const SettingsTimezone iceberg_partition_timezone; +} + namespace ProfileEvents { extern const Event IcebergPartitionPrunedFiles; @@ -50,103 +56,6 @@ namespace DB::Iceberg using namespace DB; -namespace -{ - /// Iceberg store decimal values as unscaled value with two's-complement big-endian binary - /// using the minimum number of bytes for the value - /// Our decimal binary representation is little endian - /// so we cannot reuse our default code for parsing it. - /// - /// NOTE: It's very weird, but Decimal values for lower bound and upper bound - /// are stored rounded, without fractional part. What is more strange - /// the integer part is rounded mathematically correctly according to fractional part. - /// Example: 17.22 -> 17, 8888.999 -> 8889, 1423.77 -> 1424. - /// I've checked two implementations: Spark and Amazon Athena and both of them - /// do this. - /// - /// The problem is -- we cannot use rounded values for lower bounds and upper bounds. - /// Example: upper_bound(x) = 17.22, but it's rounded 17.00, now condition WHERE x >= 17.21 will - /// check rounded value and say: "Oh largest value is 17, so values bigger than 17.21 cannot be in this file, - /// let's skip it". But it will produce incorrect result since actual value (17.22 >= 17.21) is stored in this file. - /// - /// To handle this issue we subtract 1 from the integral part for lower_bound and add 1 to integral - /// part of upper_bound. This produces: 17.22 -> [16.0, 18.0]. So this is more rough boundary, - /// but at least it doesn't lead to incorrect results. - /// `compensate_rounding` widens the bound as described above; pass false to read the value exactly - /// as the manifest declares it. - template - std::optional - deserializeDecimalBound(const std::string & str, UInt32 scale, bool lower_bound, bool compensate_rounding = true) - { - using NativeType = typename DecimalType::NativeType; - using UnsignedType = make_unsigned_t; - - if (str.size() > sizeof(NativeType)) - return std::nullopt; - - /// Accumulate into the unsigned counterpart, pre-filled with the sign bits, - /// so that the sign extension comes out of the shifts themselves. - UnsignedType unscaled = (str[0] & 0x80) ? ~UnsignedType(0) : UnsignedType(0); - for (const auto byte : str) - unscaled = (unscaled << 8) | static_cast(byte); - - NativeType unscaled_value = static_cast(unscaled); - - if (compensate_rounding && scale) - { - NativeType scaler = lower_bound ? -10 : 10; - for (UInt32 i = 1; i < scale; ++i) - scaler *= 10; - - /// The bound is stored as raw bytes and is never checked against the declared precision, so - /// widening it can leave the type. A value that has no widened form is not a usable bound. - if (common::addOverflow(unscaled_value, scaler, unscaled_value)) - return std::nullopt; - } - - return DB::DecimalField(unscaled_value, scale); - } - - /// Iceberg stores lower_bounds and upper_bounds serialized with some custom deserialization as bytes array - /// https://iceberg.apache.org/spec/#appendix-d-single-value-serialization - std::optional deserializeFieldFromBinaryRepr( - std::string str, DB::DataTypePtr expected_type, bool lower_bound, bool compensate_rounding = true) - { - auto non_nullable_type = DB::removeNullable(expected_type); - auto column = non_nullable_type->createColumn(); - if (DB::WhichDataType(non_nullable_type).isDecimal()) - { - if (str.empty()) - return std::nullopt; - - const UInt32 scale = DB::getDecimalScale(*non_nullable_type); - if (DB::checkDecimal(*non_nullable_type)) - return deserializeDecimalBound(str, scale, lower_bound, compensate_rounding); - if (DB::checkDecimal(*non_nullable_type)) - return deserializeDecimalBound(str, scale, lower_bound, compensate_rounding); - if (DB::checkDecimal(*non_nullable_type)) - return deserializeDecimalBound(str, scale, lower_bound, compensate_rounding); - if (DB::checkDecimal(*non_nullable_type)) - return deserializeDecimalBound(str, scale, lower_bound, compensate_rounding); - return std::nullopt; - } - else if (non_nullable_type->getTypeId() == DB::TypeIndex::Variant) - { - return std::nullopt; - } - else - { - /// For all other types except decimal binary representation - /// matches our internal representation - column->insertData(str.data(), str.length()); - DB::Field result; - column->get(0, result); - return result; - } - } - -} - const std::vector & ManifestFileIterator::ManifestFileEntriesHandle::getFilesWithoutDeleted(FileContentType content_type) const { @@ -244,9 +153,10 @@ std::shared_ptr ManifestFileIterator::create( std::shared_ptr filter_dag_, Int32 table_snapshot_schema_id_) { + auto dump_metadata = [&]()->String { return manifest_file_deserializer_->getMetadataContent(); }; insertRowToLogTable( context_, - [&] { return manifest_file_deserializer_->getMetadataContent(); }, + dump_metadata, DB::IcebergMetadataLogLevel::ManifestFileMetadata, path_resolver_.getTableRoot(), path_to_manifest_file_, @@ -293,7 +203,7 @@ std::shared_ptr ManifestFileIterator::create( const Poco::JSON::Object::Ptr & schema_object = json.extract(); Int32 manifest_schema_id = schema_object->getValue(f_schema_id); - schema_processor.addIcebergTableSchema(schema_object); + schema_processor.addIcebergTableSchema(schema_object, context_); PartitionSpecification partition_spec_vec; for (size_t i = 0; i != partition_specification->size(); ++i) @@ -311,7 +221,7 @@ std::shared_ptr ManifestFileIterator::create( auto transform_name = partition_specification_field->getValue(f_partition_transform); auto partition_name = partition_specification_field->getValue(f_partition_name); partition_spec_vec.emplace_back(source_id, transform_name, partition_name, static_cast(i)); - auto partition_ast = getASTFromTransform(transform_name, numeric_column_name); + auto partition_ast = getASTFromTransform(transform_name, numeric_column_name, context_->getSettingsRef()[Setting::iceberg_partition_timezone]); /// Unsupported partition key expression if (partition_ast == nullptr) continue; @@ -396,9 +306,10 @@ ProcessedManifestFileEntryPtr ManifestFileIterator::processRow(size_t row_index) if (parsed_entry->status == ManifestEntryStatus::DELETED) { + auto dump_metadata = [&]()->String { return manifest_file_deserializer->getContent(row_index); }; insertRowToLogTable( context, - [&] { return manifest_file_deserializer->getContent(row_index); }, + dump_metadata, DB::IcebergMetadataLogLevel::ManifestFileEntry, path_resolver.getTableRoot(), path_to_manifest_file, @@ -549,9 +460,10 @@ ProcessedManifestFileEntryPtr ManifestFileIterator::processRow(size_t row_index) const ManifestFilesPruner * current_pruner = getOrCreatePruner(entry->resolved_schema_id); pruning_status = current_pruner->canBePruned(entry, hyperrectangles); } + auto dump_metadata = [&]()->String { return manifest_file_deserializer->getContent(row_index); }; insertRowToLogTable( context, - [&] { return manifest_file_deserializer->getContent(row_index); }, + dump_metadata, DB::IcebergMetadataLogLevel::ManifestFileEntry, path_resolver.getTableRoot(), path_to_manifest_file, diff --git a/src/Storages/ObjectStorage/DataLakes/Iceberg/ManifestFilesPruning.cpp b/src/Storages/ObjectStorage/DataLakes/Iceberg/ManifestFilesPruning.cpp index 011cc96f16a0..aafd19302b7f 100644 --- a/src/Storages/ObjectStorage/DataLakes/Iceberg/ManifestFilesPruning.cpp +++ b/src/Storages/ObjectStorage/DataLakes/Iceberg/ManifestFilesPruning.cpp @@ -34,9 +34,9 @@ namespace DB::ErrorCodes namespace DB::Iceberg { -DB::ASTPtr getASTFromTransform(const String & transform_name_src, const String & column_name) +DB::ASTPtr getASTFromTransform(const String & transform_name_src, const String & column_name, const String & time_zone) { - auto transform_and_argument = parseTransformAndArgument(transform_name_src); + auto transform_and_argument = parseTransformAndArgument(transform_name_src, time_zone); if (!transform_and_argument) { LOG_WARNING(&Poco::Logger::get("Iceberg Partition Pruning"), "Cannot parse iceberg transform name: {}.", transform_name_src); @@ -55,6 +55,13 @@ DB::ASTPtr getASTFromTransform(const String & transform_name_src, const String & return makeASTFunction( transform_and_argument->transform_name, make_intrusive(*transform_and_argument->argument), make_intrusive(column_name)); } + if (transform_and_argument->time_zone) + { + return makeASTFunction( + transform_and_argument->transform_name, + make_intrusive(column_name), + make_intrusive(*transform_and_argument->time_zone)); + } return makeASTFunction(transform_and_argument->transform_name, make_intrusive(column_name)); } diff --git a/src/Storages/ObjectStorage/DataLakes/Iceberg/ManifestFilesPruning.h b/src/Storages/ObjectStorage/DataLakes/Iceberg/ManifestFilesPruning.h index 78c136167a88..d04ced3796ee 100644 --- a/src/Storages/ObjectStorage/DataLakes/Iceberg/ManifestFilesPruning.h +++ b/src/Storages/ObjectStorage/DataLakes/Iceberg/ManifestFilesPruning.h @@ -30,7 +30,7 @@ namespace DB::Iceberg struct ProcessedManifestFileEntry; class ManifestFileIterator; -DB::ASTPtr getASTFromTransform(const String & transform_name_src, const String & column_name); +DB::ASTPtr getASTFromTransform(const String & transform_name_src, const String & column_name, const String & time_zone); /// Prune specific data files based on manifest content class ManifestFilesPruner diff --git a/src/Storages/ObjectStorage/DataLakes/Iceberg/MultipleFileWriter.cpp b/src/Storages/ObjectStorage/DataLakes/Iceberg/MultipleFileWriter.cpp index 06f1ca6fcec6..6433207cc38d 100644 --- a/src/Storages/ObjectStorage/DataLakes/Iceberg/MultipleFileWriter.cpp +++ b/src/Storages/ObjectStorage/DataLakes/Iceberg/MultipleFileWriter.cpp @@ -23,11 +23,12 @@ MultipleFileWriter::MultipleFileWriter( ContextPtr context_, const std::optional & format_settings_, const String & write_format_, - SharedHeader sample_block_) + SharedHeader sample_block_, + std::function new_file_path_callback_) : max_data_file_num_rows(max_data_file_num_rows_) , max_data_file_num_bytes(max_data_file_num_bytes_) , schema(schema_) - , stats(schema_) + , aggregate_stats(schema_) , column_mapper(Iceberg::createColumnMapperFromFields(schema_)) , filename_generator(filename_generator_) , path_resolver(path_resolver_) @@ -36,13 +37,16 @@ MultipleFileWriter::MultipleFileWriter( , format_settings(format_settings_) , write_format(std::move(write_format_)) , sample_block(sample_block_) + , new_file_path_callback(std::move(new_file_path_callback_)) { } void MultipleFileWriter::startNewFile() { if (buffer) + { finalize(); + } current_file_stats = std::make_shared(schema); current_file_num_rows = 0; @@ -51,6 +55,9 @@ void MultipleFileWriter::startNewFile() auto storage_path = path_resolver.resolve(metadata_path); data_file_names.push_back(metadata_path); + if (new_file_path_callback) + new_file_path_callback(storage_path); + buffer = object_storage->writeObject( StoredObject(storage_path), WriteMode::Rewrite, std::nullopt, DBMS_DEFAULT_BUFFER_SIZE, context->getWriteSettings()); @@ -77,7 +84,7 @@ void MultipleFileWriter::consume(const Chunk & chunk) output_format->flush(); *current_file_num_rows += chunk.getNumRows(); *current_file_num_bytes += chunk.bytes(); - stats.update(chunk); + aggregate_stats.update(chunk); current_file_stats->update(chunk); } @@ -95,6 +102,31 @@ void MultipleFileWriter::finalize() data_file_row_counts.push_back(current_file_num_rows.value_or(0)); } +std::vector MultipleFileWriter::getDataFileEntries() const +{ + chassert(data_file_names.size() == data_file_row_counts.size()); + chassert(data_file_names.size() == data_file_byte_counts.size()); + chassert(data_file_names.size() == completed_file_stats.size()); + + std::vector entries; + entries.reserve(data_file_names.size()); + + for (size_t i = 0; i < data_file_names.size(); ++i) + { + std::optional statistics; + if (completed_file_stats[i]) + statistics = *completed_file_stats[i]; + + entries.emplace_back( + path_resolver.resolve(data_file_names[i]), + static_cast(data_file_row_counts[i]), + static_cast(data_file_byte_counts[i]), + std::move(statistics)); + } + + return entries; +} + void MultipleFileWriter::release() { output_format.reset(); diff --git a/src/Storages/ObjectStorage/DataLakes/Iceberg/MultipleFileWriter.h b/src/Storages/ObjectStorage/DataLakes/Iceberg/MultipleFileWriter.h index 17a413f0a919..ef3fa54f0d37 100644 --- a/src/Storages/ObjectStorage/DataLakes/Iceberg/MultipleFileWriter.h +++ b/src/Storages/ObjectStorage/DataLakes/Iceberg/MultipleFileWriter.h @@ -5,6 +5,7 @@ #include #include #include +#include namespace DB { @@ -27,7 +28,8 @@ class MultipleFileWriter ContextPtr context_, const std::optional & format_settings_, const String & write_format_, - SharedHeader sample_block_); + SharedHeader sample_block_, + std::function new_file_path_callback_ = {}); void consume(const Chunk & chunk); void startNewFile(); @@ -55,7 +57,7 @@ class MultipleFileWriter const DataFileStatistics & getResultStatistics() const { - return stats; + return aggregate_stats; } const std::vector & getPerFileStatistics() const @@ -63,12 +65,17 @@ class MultipleFileWriter return completed_file_stats; } + /// Returns one entry per written data file, with the accurate row count, byte size, + /// and per-file column statistics collected during finalization. + /// Must be called only after finalize(). + std::vector getDataFileEntries() const; + private: UInt64 max_data_file_num_rows; UInt64 max_data_file_num_bytes; Poco::JSON::Array::Ptr schema; - DataFileStatistics stats; - DataFileStatisticsPtr current_file_stats; + DataFileStatistics aggregate_stats; /// accumulates across all files + DataFileStatisticsPtr current_file_stats; /// accumulates for the current file only std::vector completed_file_stats; /// Pre-built ColumnMapper for `startNewFile`. Traversing the Iceberg schema is invariant /// for the lifetime of the writer, so we compute the mapping once and reuse it across @@ -89,6 +96,7 @@ class MultipleFileWriter const String& write_format; SharedHeader sample_block; UInt64 total_bytes = 0; + std::function new_file_path_callback; }; #endif diff --git a/src/Storages/ObjectStorage/DataLakes/Iceberg/PersistentTableComponents.h b/src/Storages/ObjectStorage/DataLakes/Iceberg/PersistentTableComponents.h index 9cedaef4207b..dbb41013eb0b 100644 --- a/src/Storages/ObjectStorage/DataLakes/Iceberg/PersistentTableComponents.h +++ b/src/Storages/ObjectStorage/DataLakes/Iceberg/PersistentTableComponents.h @@ -28,6 +28,7 @@ struct PersistentTableComponents /// True when the resolver works against a table root deeper than `table_path`. Operations scoped /// to `table_path` then reach outside this table, so they must refuse to run. const bool table_root_was_derived; + const String common_namespace; /// Invalidate cached metadata for this table under both keys we may have used to cache it /// (`table_path` and `table_uuid`). diff --git a/src/Storages/ObjectStorage/DataLakes/Iceberg/SchemaProcessor.cpp b/src/Storages/ObjectStorage/DataLakes/Iceberg/SchemaProcessor.cpp index c154a025ab3a..f4963807e8e8 100644 --- a/src/Storages/ObjectStorage/DataLakes/Iceberg/SchemaProcessor.cpp +++ b/src/Storages/ObjectStorage/DataLakes/Iceberg/SchemaProcessor.cpp @@ -21,6 +21,7 @@ #include #include #include +#include #include #include @@ -36,6 +37,8 @@ #include #include #include +#include +#include #include @@ -51,6 +54,10 @@ extern const int BAD_ARGUMENTS; extern const int ICEBERG_SPECIFICATION_VIOLATION; } +namespace Setting +{ +extern const SettingsTimezone iceberg_timezone_for_timestamptz; +} namespace { @@ -393,7 +400,7 @@ namespace Iceberg std::string IcebergSchemaProcessor::default_link{}; -void IcebergSchemaProcessor::addIcebergTableSchema(Poco::JSON::Object::Ptr schema_ptr) +void IcebergSchemaProcessor::addIcebergTableSchema(Poco::JSON::Object::Ptr schema_ptr, ContextPtr context_) { std::lock_guard lock(mutex); @@ -444,7 +451,7 @@ void IcebergSchemaProcessor::addIcebergTableSchema(Poco::JSON::Object::Ptr schem auto name = field->getValue(f_name); bool required = field->getValue(f_required); current_full_name = name; - auto type = getFieldType(field, f_type, required, current_full_name, true); + auto type = getFieldType(field, f_type, context_, required, current_full_name, true); clickhouse_schema->push_back(NameAndTypePair{name, type}); clickhouse_types_by_source_ids[{schema_id, field->getValue(f_id)}] = NameAndTypePair{current_full_name, type}; clickhouse_ids_by_source_names[{schema_id, current_full_name}] = field->getValue(f_id); @@ -499,7 +506,7 @@ NamesAndTypesList IcebergSchemaProcessor::tryGetFieldsCharacteristics(Int32 sche return fields; } -DataTypePtr IcebergSchemaProcessor::getSimpleType(const String & type_name_arg, bool allow_geo_parser) +DataTypePtr IcebergSchemaProcessor::getSimpleType(const String & type_name_arg, ContextPtr context_, bool allow_geo_parser) { /// Parameterized primitive type strings (decimal(P, S), fixed[N], geography(...)) can be /// serialized with different inner whitespace across metadata files. Canonicalize by removing @@ -523,7 +530,10 @@ DataTypePtr IcebergSchemaProcessor::getSimpleType(const String & type_name_arg, if (type_name == f_timestamp) return std::make_shared(6); if (type_name == f_timestamptz) - return std::make_shared(6, "UTC"); + { + std::string timezone = context_->getSettingsRef()[Setting::iceberg_timezone_for_timestamptz]; + return std::make_shared(6, timezone); + } if (type_name == f_timestamp_ns) return std::make_shared(9); if (type_name == f_timestamptz_ns) @@ -564,7 +574,11 @@ DataTypePtr IcebergSchemaProcessor::getSimpleType(const String & type_name_arg, } DataTypePtr -IcebergSchemaProcessor::getComplexTypeFromObject(const Poco::JSON::Object::Ptr & type, String & current_full_name, bool is_subfield_of_root) +IcebergSchemaProcessor::getComplexTypeFromObject( + const Poco::JSON::Object::Ptr & type, + String & current_full_name, + ContextPtr context_, + bool is_subfield_of_root) { /// The schema comes from the table metadata and can be nested arbitrarily deeply. checkStackSize(); @@ -573,15 +587,15 @@ IcebergSchemaProcessor::getComplexTypeFromObject(const Poco::JSON::Object::Ptr & if (type_name == f_list) { bool element_required = type->getValue("element-required"); - auto element_type = getFieldType(type, f_element, element_required); + auto element_type = getFieldType(type, f_element, context_, element_required); return std::make_shared(element_type); } if (type_name == f_map) { - auto key_type = getFieldType(type, f_key, true); + auto key_type = getFieldType(type, f_key, context_, true); auto value_required = type->getValue("value-required"); - auto value_type = getFieldType(type, f_value, value_required); + auto value_type = getFieldType(type, f_value, context_, value_required); return std::make_shared(key_type, value_type); } @@ -605,7 +619,7 @@ IcebergSchemaProcessor::getComplexTypeFromObject(const Poco::JSON::Object::Ptr & (current_full_name += ".").append(element_names.back()); scope_guard guard([&] { current_full_name.resize(current_full_name.size() - element_names.back().size() - 1); }); - element_types.push_back(getFieldType(field, f_type, required, current_full_name, true)); + element_types.push_back(getFieldType(field, f_type, context_, required, current_full_name, true)); TSA_SUPPRESS_WARNING_FOR_WRITE(clickhouse_types_by_source_ids) [{schema_id, field->getValue(f_id)}] = NameAndTypePair{current_full_name, element_types.back()}; @@ -614,7 +628,7 @@ IcebergSchemaProcessor::getComplexTypeFromObject(const Poco::JSON::Object::Ptr & } else { - element_types.push_back(getFieldType(field, f_type, required)); + element_types.push_back(getFieldType(field, f_type, context_, required)); } } @@ -625,16 +639,21 @@ IcebergSchemaProcessor::getComplexTypeFromObject(const Poco::JSON::Object::Ptr & } DataTypePtr IcebergSchemaProcessor::getFieldType( - const Poco::JSON::Object::Ptr & field, const String & type_key, bool required, String & current_full_name, bool is_subfield_of_root) + const Poco::JSON::Object::Ptr & field, + const String & type_key, + ContextPtr context_, + bool required, + String & current_full_name, + bool is_subfield_of_root) { if (field->isObject(type_key)) - return getComplexTypeFromObject(field->getObject(type_key), current_full_name, is_subfield_of_root); + return getComplexTypeFromObject(field->getObject(type_key), current_full_name, context_, is_subfield_of_root); auto type = field->get(type_key); if (type.isString()) { const String & type_name = type.extract(); - auto data_type = getSimpleType(type_name, allow_geo_parser); + auto data_type = getSimpleType(type_name, context_, allow_geo_parser); return required || !data_type->canBeInsideNullable() ? data_type : makeNullable(data_type); } @@ -669,7 +688,11 @@ bool IcebergSchemaProcessor::allowPrimitiveTypeConversion(const String & old_typ // Ids are passed only for error logging purposes std::shared_ptr IcebergSchemaProcessor::getSchemaTransformationDag( - const Poco::JSON::Object::Ptr & old_schema, const Poco::JSON::Object::Ptr & new_schema, Int32 old_id, Int32 new_id) + const Poco::JSON::Object::Ptr & old_schema, + const Poco::JSON::Object::Ptr & new_schema, + ContextPtr context_, + Int32 old_id, + Int32 new_id) { std::unordered_map> old_schema_entries; auto old_schema_fields = old_schema->get(f_fields).extract(); @@ -681,7 +704,7 @@ std::shared_ptr IcebergSchemaProcessor::getSchemaTransformationDag( size_t id = field->getValue(f_id); auto name = field->getValue(f_name); bool required = field->getValue(f_required); - old_schema_entries[id] = {field, &dag->addInput(name, getFieldType(field, f_type, required))}; + old_schema_entries[id] = {field, &dag->addInput(name, getFieldType(field, f_type, context_, required))}; } auto new_schema_fields = new_schema->get(f_fields).extract(); for (size_t i = 0; i != new_schema_fields->size(); ++i) @@ -690,7 +713,7 @@ std::shared_ptr IcebergSchemaProcessor::getSchemaTransformationDag( size_t id = field->getValue(f_id); auto name = field->getValue(f_name); bool required = field->getValue(f_required); - auto type = getFieldType(field, f_type, required); + auto type = getFieldType(field, f_type, context_, required); auto old_node_it = old_schema_entries.find(id); if (old_node_it != old_schema_entries.end()) { @@ -709,7 +732,7 @@ std::shared_ptr IcebergSchemaProcessor::getSchemaTransformationDag( old_id, new_id); } - auto old_type = getFieldType(old_json, "type", required); + auto old_type = getFieldType(old_json, "type", context_, required); auto transform = std::make_shared(DataTypes{type}, DataTypes{old_type}, old_json, field); old_node = &dag->addFunction(transform, std::vector{old_node}, name); @@ -742,7 +765,7 @@ std::shared_ptr IcebergSchemaProcessor::getSchemaTransformationDag( } else if (allowPrimitiveTypeConversion(old_type, new_type)) { - node = &dag->addCast(*old_node, getFieldType(field, f_type, required), name, nullptr); + node = &dag->addCast(*old_node, getFieldType(field, f_type, context_, required), name, nullptr); } outputs.push_back(node); } @@ -770,7 +793,10 @@ std::shared_ptr IcebergSchemaProcessor::getSchemaTransformationDag( return dag; } -std::shared_ptr IcebergSchemaProcessor::getSchemaTransformationDagByIds(Int32 old_id, Int32 new_id) +std::shared_ptr IcebergSchemaProcessor::getSchemaTransformationDagByIds( + ContextPtr context_, + Int32 old_id, + Int32 new_id) { if (old_id == new_id) return nullptr; @@ -789,7 +815,7 @@ std::shared_ptr IcebergSchemaProcessor::getSchemaTransformatio throw Exception(ErrorCodes::BAD_ARGUMENTS, "Schema with schema-id {} is unknown", new_id); return transform_dags_by_ids[{old_id, new_id}] - = getSchemaTransformationDag(old_schema_it->second, new_schema_it->second, old_id, new_id); + = getSchemaTransformationDag(old_schema_it->second, new_schema_it->second, context_, old_id, new_id); } Poco::JSON::Object::Ptr IcebergSchemaProcessor::getIcebergTableSchemaById(Int32 id) const diff --git a/src/Storages/ObjectStorage/DataLakes/Iceberg/SchemaProcessor.h b/src/Storages/ObjectStorage/DataLakes/Iceberg/SchemaProcessor.h index a072f5ebc127..5e1d637dba96 100644 --- a/src/Storages/ObjectStorage/DataLakes/Iceberg/SchemaProcessor.h +++ b/src/Storages/ObjectStorage/DataLakes/Iceberg/SchemaProcessor.h @@ -81,18 +81,19 @@ ColumnMapperPtr createColumnMapper(Poco::JSON::Object::Ptr schema_object); * } * } */ -class IcebergSchemaProcessor +class IcebergSchemaProcessor : private WithContext { static std::string default_link; using Node = ActionsDAG::Node; public: - explicit IcebergSchemaProcessor(bool allow_geo_parser_ = false) : allow_geo_parser(allow_geo_parser_) {} + explicit IcebergSchemaProcessor(ContextPtr context_, bool allow_geo_parser_ = false) + : WithContext(context_), allow_geo_parser(allow_geo_parser_) {} - void addIcebergTableSchema(Poco::JSON::Object::Ptr schema_ptr); + void addIcebergTableSchema(Poco::JSON::Object::Ptr schema_ptr, ContextPtr context_); std::shared_ptr getClickHouseTableSchemaById(Int32 id); - std::shared_ptr getSchemaTransformationDagByIds(Int32 old_id, Int32 new_id); + std::shared_ptr getSchemaTransformationDagByIds(ContextPtr context_, Int32 old_id, Int32 new_id); NameAndTypePair getFieldCharacteristics(Int32 schema_version, Int32 source_id) const; std::optional tryGetFieldCharacteristics(Int32 schema_version, Int32 source_id) const; NamesAndTypesList tryGetFieldsCharacteristics(Int32 schema_id, const std::vector & source_ids) const; @@ -100,7 +101,7 @@ class IcebergSchemaProcessor Poco::JSON::Object::Ptr getIcebergTableSchemaById(Int32 id) const; bool hasClickHouseTableSchemaById(Int32 id) const; - static DataTypePtr getSimpleType(const String & type_name, bool allow_geo_parser = true); + static DataTypePtr getSimpleType(const String & type_name, ContextPtr context_, bool allow_geo_parser = true); static std::unordered_map traverseSchema(Poco::JSON::Array::Ptr schema); @@ -127,10 +128,15 @@ class IcebergSchemaProcessor std::unordered_map schema_id_by_snapshot TSA_GUARDED_BY(mutex); NamesAndTypesList getSchemaType(const Poco::JSON::Object::Ptr & schema); - DataTypePtr getComplexTypeFromObject(const Poco::JSON::Object::Ptr & type, String & current_full_name, bool is_subfield_of_root); + DataTypePtr getComplexTypeFromObject( + const Poco::JSON::Object::Ptr & type, + String & current_full_name, + ContextPtr context_, + bool is_subfield_of_root); DataTypePtr getFieldType( const Poco::JSON::Object::Ptr & field, const String & type_key, + ContextPtr context_, bool required, String & current_full_name = default_link, bool is_subfield_of_root = false); @@ -139,7 +145,11 @@ class IcebergSchemaProcessor const Node * getDefaultNodeForField(const Poco::JSON::Object::Ptr & field); std::shared_ptr getSchemaTransformationDag( - const Poco::JSON::Object::Ptr & old_schema, const Poco::JSON::Object::Ptr & new_schema, Int32 old_id, Int32 new_id); + const Poco::JSON::Object::Ptr & old_schema, + const Poco::JSON::Object::Ptr & new_schema, + ContextPtr context_, + Int32 old_id, + Int32 new_id); mutable SharedMutex mutex; bool allow_geo_parser = true; diff --git a/src/Storages/ObjectStorage/DataLakes/Iceberg/Snapshot.h b/src/Storages/ObjectStorage/DataLakes/Iceberg/Snapshot.h index 113bd4982f01..8ed1dc9b798a 100644 --- a/src/Storages/ObjectStorage/DataLakes/Iceberg/Snapshot.h +++ b/src/Storages/ObjectStorage/DataLakes/Iceberg/Snapshot.h @@ -26,6 +26,8 @@ struct IcebergDataSnapshot /// Rows in equality-delete files (snapshot summary). Not a count of deleted data rows; /// used only to fail closed early when present and > 0. std::optional total_equality_delete_rows; + std::optional partition_key; + std::optional sorting_key; std::optional getTotalRows() const { diff --git a/src/Storages/ObjectStorage/DataLakes/Iceberg/StatelessMetadataFileGetter.cpp b/src/Storages/ObjectStorage/DataLakes/Iceberg/StatelessMetadataFileGetter.cpp index 9b3a648938b9..5d9055f020d2 100644 --- a/src/Storages/ObjectStorage/DataLakes/Iceberg/StatelessMetadataFileGetter.cpp +++ b/src/Storages/ObjectStorage/DataLakes/Iceberg/StatelessMetadataFileGetter.cpp @@ -180,9 +180,10 @@ ManifestFileCacheKeys getManifestList( ManifestFileCacheKeys manifest_file_cache_keys; + auto dump_metadata = [&]()->String { return manifest_list_deserializer.getMetadataContent(); }; insertRowToLogTable( local_context, - [&] { return manifest_list_deserializer.getMetadataContent(); }, + dump_metadata, DB::IcebergMetadataLogLevel::ManifestListMetadata, persistent_table_components.path_resolver.getTableRoot(), filename, @@ -245,9 +246,10 @@ ManifestFileCacheKeys getManifestList( manifest_file_name, manifest_length, added_sequence_number, added_snapshot_id.safeGet(), content_type, partition_spec_id); + auto dump_row_metadata = [&]()->String { return manifest_list_deserializer.getContent(i); }; insertRowToLogTable( local_context, - [&] { return manifest_list_deserializer.getContent(i); }, + dump_row_metadata, DB::IcebergMetadataLogLevel::ManifestListEntry, persistent_table_components.path_resolver.getTableRoot(), filename, diff --git a/src/Storages/ObjectStorage/DataLakes/Iceberg/Utils.cpp b/src/Storages/ObjectStorage/DataLakes/Iceberg/Utils.cpp index 497f559d7c95..e499718b383d 100644 --- a/src/Storages/ObjectStorage/DataLakes/Iceberg/Utils.cpp +++ b/src/Storages/ObjectStorage/DataLakes/Iceberg/Utils.cpp @@ -65,6 +65,7 @@ #include #include +#include using namespace DB; @@ -97,12 +98,15 @@ extern const SettingsString iceberg_metadata_compression_method; namespace ProfileEvents { extern const Event IcebergVersionHintUsed; + extern const Event IcebergJsonFileParsing; + extern const Event IcebergJsonFileParsingMicroseconds; } namespace DB::Setting { extern const SettingsUInt64 iceberg_metadata_staleness_ms; extern const SettingsUInt64 output_format_compression_level; + extern const SettingsTimezone iceberg_partition_timezone; } /// Hard to imagine a hint file larger than 10 MB @@ -188,7 +192,7 @@ static Int32 parseMetadataVersion(const String & version_str, const String & fil return version; } -static MetadataFileWithInfo getMetadataFileAndVersion(const std::string & path) +Iceberg::MetadataFileWithInfo getMetadataFileAndVersion(const std::string & path) { String file_name = std::filesystem::path(path).filename(); if (isTemporaryMetadataFile(file_name)) @@ -432,27 +436,31 @@ bool writeMetadataFileAndVersionHint( } -std::optional parseTransformAndArgument(const String & transform_name_src) +std::optional parseTransformAndArgument(const String & transform_name_src, const String & time_zone) { std::string transform_name = Poco::toLower(transform_name_src); + std::optional time_zone_opt; + if (!time_zone.empty()) + time_zone_opt = time_zone; + if (transform_name == "year" || transform_name == "years") - return TransformAndArgument{"toYearNumSinceEpoch", std::nullopt}; + return TransformAndArgument{"toYearNumSinceEpoch", std::nullopt, time_zone_opt}; if (transform_name == "month" || transform_name == "months") - return TransformAndArgument{"toMonthNumSinceEpoch", std::nullopt}; + return TransformAndArgument{"toMonthNumSinceEpoch", std::nullopt, time_zone_opt}; if (transform_name == "day" || transform_name == "date" || transform_name == "days" || transform_name == "dates") - return TransformAndArgument{"toRelativeDayNum", std::nullopt}; + return TransformAndArgument{"toRelativeDayNum", std::nullopt, time_zone_opt}; if (transform_name == "hour" || transform_name == "hours") - return TransformAndArgument{"toRelativeHourNum", std::nullopt}; + return TransformAndArgument{"toRelativeHourNum", std::nullopt, time_zone_opt}; if (transform_name == "identity") - return TransformAndArgument{"identity", std::nullopt}; + return TransformAndArgument{"identity", std::nullopt, std::nullopt}; if (transform_name == "void") - return TransformAndArgument{"tuple", std::nullopt}; + return TransformAndArgument{"tuple", std::nullopt, std::nullopt}; if (transform_name.starts_with("truncate") || transform_name.starts_with("bucket")) { @@ -476,11 +484,11 @@ std::optional parseTransformAndArgument(const String & tra if (transform_name.starts_with("truncate")) { - return TransformAndArgument{"icebergTruncate", argument}; + return TransformAndArgument{"icebergTruncate", argument, std::nullopt}; } else if (transform_name.starts_with("bucket")) { - return TransformAndArgument{"icebergBucket", argument}; + return TransformAndArgument{"icebergBucket", argument, std::nullopt}; } } return std::nullopt; @@ -547,6 +555,9 @@ Poco::JSON::Object::Ptr getMetadataJSONObject( return json_str; }; + ProfileEvents::increment(ProfileEvents::IcebergJsonFileParsing); + ProfileEventTimeIncrement watch(ProfileEvents::IcebergJsonFileParsingMicroseconds); + String metadata_json_str; if (metadata_cache && table_uuid.has_value()) metadata_json_str = metadata_cache->getOrSetTableMetadata( @@ -580,6 +591,8 @@ std::pair getIcebergType(DataTypePtr type, Int32 & ite { switch (type->getTypeId()) { + case TypeIndex::UInt16: + case TypeIndex::Int16: case TypeIndex::UInt32: case TypeIndex::Int32: return {"int", true}; @@ -849,7 +862,7 @@ static Poco::JSON::Object::Ptr getPartitionField( throw Exception(ErrorCodes::BAD_ARGUMENTS, "Unsupported function for iceberg partitioning {}", partition_function->name); } -static std::pair getPartitionSpec( +std::pair getPartitionSpec( ASTPtr partition_by, const std::unordered_map & column_name_to_source_id) { @@ -1412,7 +1425,7 @@ MetadataFileWithInfo getLatestMetadataFileAndVersionWithCatalog( DataLake::TableMetadata table_metadata; table_metadata.withDataLakeSpecificProperties().withLocation(); const auto & [namespace_name, table_name] = DataLake::parseTableName(table_identifier); - catalog->getTableMetadata(namespace_name, table_name, table_metadata); + catalog->getTableMetadata(namespace_name, table_name, local_context, table_metadata); auto specific_properties = table_metadata.getDataLakeSpecificProperties(); if (!specific_properties.has_value() || specific_properties->iceberg_metadata_file_location.empty()) @@ -1489,6 +1502,11 @@ std::pair parseTableSchemaV1Method(const Poco::J KeyDescription getSortingKeyDescriptionFromMetadata(Poco::JSON::Object::Ptr metadata_object, const NamesAndTypesList & ch_schema, ContextPtr local_context) { + // sort-orders / default-sort-order-id are optional in Iceberg V1 metadata + // (required only from V2); an unsorted table uses the no-op order-id 0. + // Treat their absence as "no sort order" rather than dereferencing a missing field. + if (!metadata_object->has(f_default_sort_order_id) || !metadata_object->has(f_sort_orders)) + return KeyDescription{}; auto sort_order_id = metadata_object->getValue(f_default_sort_order_id); Poco::JSON::Array::Ptr sort_orders = metadata_object->getArray(f_sort_orders); std::unordered_map source_id_to_column_name; @@ -1520,7 +1538,8 @@ KeyDescription getSortingKeyDescriptionFromMetadata(Poco::JSON::Object::Ptr meta auto column_name = source_id_to_column_name[source_id]; int direction = field->getValue(f_direction) == "asc" ? 1 : -1; auto iceberg_transform_name = field->getValue(f_transform); - auto clickhouse_transform_name = parseTransformAndArgument(iceberg_transform_name); + auto clickhouse_transform_name = parseTransformAndArgument(iceberg_transform_name, + local_context->getSettingsRef()[Setting::iceberg_partition_timezone]); /// Quote the column name so identifiers with special characters (e.g. `@timestamp`) /// produce a parseable ORDER BY clause. auto quoted_column_name = backQuoteIfNeed(column_name); @@ -1532,7 +1551,10 @@ KeyDescription getSortingKeyDescriptionFromMetadata(Poco::JSON::Object::Ptr meta { full_argument += std::to_string(*clickhouse_transform_name->argument) + ", "; } - full_argument += quoted_column_name + ")"; + full_argument += quoted_column_name; + if (clickhouse_transform_name->time_zone) + full_argument += ", '" + *clickhouse_transform_name->time_zone + "'"; + full_argument += ")"; } else { @@ -1551,6 +1573,124 @@ KeyDescription getSortingKeyDescriptionFromMetadata(Poco::JSON::Object::Ptr meta return KeyDescription::parse(order_by_str, column_description, {}, local_context, true); } +/// Format one partition field for display in Iceberg/Spark style, e.g. "day(ts)" or "bucket(16, id)". +static String formatPartitionFieldDisplay(const String & iceberg_transform_name, const String & column_name) +{ + std::string name = Poco::toLower(iceberg_transform_name); + if (name == "identity") + return column_name; + if (name == "year" || name == "years") + return "year(" + column_name + ")"; + if (name == "month" || name == "months") + return "month(" + column_name + ")"; + if (name == "day" || name == "date" || name == "days" || name == "dates") + return "day(" + column_name + ")"; + if (name == "hour" || name == "hours") + return "hour(" + column_name + ")"; + if (name.starts_with("truncate") && name.back() == ']') + { + auto p = name.find('['); + if (p != std::string::npos) + return "truncate(" + name.substr(p + 1, name.size() - p - 2) + ", " + column_name + ")"; + } + if (name.starts_with("bucket") && name.back() == ']') + { + auto p = name.find('['); + if (p != std::string::npos) + return "bucket(" + name.substr(p + 1, name.size() - p - 2) + ", " + column_name + ")"; + } + return column_name; +} + +std::optional getPartitionKeyStringFromMetadata(Poco::JSON::Object::Ptr metadata_object, const NamesAndTypesList & /* ch_schema */, ContextPtr /* local_context */) +{ + if (!metadata_object->has(f_partition_specs) || !metadata_object->has(f_default_spec_id)) + return std::nullopt; + auto partition_spec_id = metadata_object->getValue(f_default_spec_id); + Poco::JSON::Array::Ptr partition_specs = metadata_object->getArray(f_partition_specs); + std::unordered_map source_id_to_column_name; + auto [schema, current_schema_id] = parseTableSchemaV2Method(metadata_object); + auto mapper = createColumnMapper(schema)->getStorageColumnEncoding(); + for (const auto & [col_name, source_id] : mapper) + source_id_to_column_name[source_id] = col_name; + + Poco::JSON::Object::Ptr partition_spec; + for (size_t i = 0; i < partition_specs->size(); ++i) + { + auto spec = partition_specs->getObject(static_cast(i)); + if (spec->getValue(f_spec_id) == partition_spec_id) + { + partition_spec = spec; + break; + } + } + if (!partition_spec || !partition_spec->has(f_fields)) + return std::nullopt; + auto fields = partition_spec->getArray(f_fields); + if (fields->size() == 0) + return std::nullopt; + + std::vector part_exprs; + for (UInt32 i = 0; i < fields->size(); ++i) + { + auto field = fields->getObject(i); + auto source_id = field->getValue(f_source_id); + auto it = source_id_to_column_name.find(source_id); + if (it == source_id_to_column_name.end()) + return std::nullopt; + String column_name = it->second; + auto iceberg_transform_name = field->getValue(f_transform); + part_exprs.push_back(formatPartitionFieldDisplay(iceberg_transform_name, column_name)); + } + String result; + for (size_t i = 0; i < part_exprs.size(); ++i) + { + if (i != 0) + result += ", "; + result += part_exprs[i]; + } + return result; +} + +std::optional getSortingKeyDisplayStringFromMetadata(Poco::JSON::Object::Ptr metadata_object, const NamesAndTypesList & /* ch_schema */) +{ + if (!metadata_object->has(f_sort_orders) || !metadata_object->has(f_default_sort_order_id)) + return std::nullopt; + auto sort_order_id = metadata_object->getValue(f_default_sort_order_id); + Poco::JSON::Array::Ptr sort_orders = metadata_object->getArray(f_sort_orders); + std::unordered_map source_id_to_column_name; + auto [schema, current_schema_id] = parseTableSchemaV2Method(metadata_object); + auto mapper = createColumnMapper(schema)->getStorageColumnEncoding(); + for (const auto & [col_name, source_id] : mapper) + source_id_to_column_name[source_id] = col_name; + + for (UInt32 i = 0; i < sort_orders->size(); ++i) + { + auto sort_order = sort_orders->getObject(i); + if (sort_order->getValue(f_order_id) != sort_order_id) + continue; + auto sort_fields = sort_order->getArray(f_fields); + String result; + for (UInt32 j = 0; j < sort_fields->size(); ++j) + { + auto field = sort_fields->getObject(j); + auto source_id = field->getValue(f_source_id); + auto it = source_id_to_column_name.find(source_id); + if (it == source_id_to_column_name.end()) + return std::nullopt; + String column_name = it->second; + String direction = field->getValue(f_direction) == "asc" ? " asc" : " desc"; + auto iceberg_transform_name = field->getValue(f_transform); + String expr = formatPartitionFieldDisplay(iceberg_transform_name, column_name); + if (!result.empty()) + result += ", "; + result += expr + direction; + } + return result.empty() ? std::nullopt : std::optional(result); + } + return std::nullopt; +} + DataTypePtr getFunctionResultType(const String & iceberg_transform_name, DataTypePtr source_type) { if (iceberg_transform_name.starts_with("identity") || iceberg_transform_name.starts_with("truncate")) diff --git a/src/Storages/ObjectStorage/DataLakes/Iceberg/Utils.h b/src/Storages/ObjectStorage/DataLakes/Iceberg/Utils.h index 267c6e1f1e02..276791b045d1 100644 --- a/src/Storages/ObjectStorage/DataLakes/Iceberg/Utils.h +++ b/src/Storages/ObjectStorage/DataLakes/Iceberg/Utils.h @@ -20,6 +20,7 @@ #include #include #include +#include #include #include #include @@ -42,6 +43,8 @@ class ICatalog; namespace DB::Iceberg { +Iceberg::MetadataFileWithInfo getMetadataFileAndVersion(const std::string & path); + void writeMessageToFile( const String & data, const String & filename, @@ -67,9 +70,14 @@ struct TransformAndArgument { String transform_name; std::optional argument; + /// When Iceberg table is partitioned by time, splitting by partitions can be made using different timezone + /// (UTC in most cases). This timezone can be set with setting `iceberg_partition_timezone`, value is in this member. + /// When Iceberg partition condition converted to ClickHouse function in `parseTransformAndArgument` method + /// `time_zone` added as second argument to functions like `toRelativeDayNum`, `toYearNumSinceEpoch`, etc. + std::optional time_zone; }; -std::optional parseTransformAndArgument(const String & transform_name_src); +std::optional parseTransformAndArgument(const String & transform_name_src, const String & time_zone); CompressionMethod getCompressionMethodFromMetadataFile(const String & path); @@ -86,6 +94,14 @@ Poco::JSON::Object::Ptr getMetadataJSONObject( std::pair getIcebergType(DataTypePtr type, Int32 & iter); Poco::Dynamic::Var getAvroType(DataTypePtr type, Int32 field_id); +/// Converts a ClickHouse PARTITION BY AST into the corresponding Iceberg partition-spec JSON object. +/// column_name_to_source_id maps each column name to the Iceberg field-id from the table schema. +/// The returned Int32 is the last partition-field-id allocated (useful for tracking the id counter). +/// Throws if the AST contains expressions that cannot be represented as Iceberg transforms. +std::pair getPartitionSpec( + ASTPtr partition_by, + const std::unordered_map & column_name_to_source_id); + /// Spec: https://iceberg.apache.org/spec/?h=metadata.json#table-metadata-fields std::pair createEmptyMetadataFile( String path_location, @@ -141,6 +157,11 @@ FileCategory inspectFileCategory(const String & relative_path); KeyDescription getSortingKeyDescriptionFromMetadata( Poco::JSON::Object::Ptr metadata_object, const NamesAndTypesList & ch_schema, ContextPtr local_context); +/// Returns Iceberg/Spark-style display string for sort order, e.g. "id desc, hour(ts) asc". +std::optional getSortingKeyDisplayStringFromMetadata( + Poco::JSON::Object::Ptr metadata_object, const NamesAndTypesList & ch_schema); +std::optional getPartitionKeyStringFromMetadata( + Poco::JSON::Object::Ptr metadata_object, const NamesAndTypesList & ch_schema, ContextPtr local_context); void sortBlockByKeyDescription(Block & block, const KeyDescription & sort_description, ContextPtr context); void forEachAvroEntry( diff --git a/src/Storages/ObjectStorage/DataLakes/Iceberg/tests/gtest_iceberg_schema_processor.cpp b/src/Storages/ObjectStorage/DataLakes/Iceberg/tests/gtest_iceberg_schema_processor.cpp index 504f0f8fc7e9..1e4fbefb2d91 100644 --- a/src/Storages/ObjectStorage/DataLakes/Iceberg/tests/gtest_iceberg_schema_processor.cpp +++ b/src/Storages/ObjectStorage/DataLakes/Iceberg/tests/gtest_iceberg_schema_processor.cpp @@ -1,5 +1,6 @@ #include +#include #include #include #include @@ -20,109 +21,109 @@ Poco::JSON::Object::Ptr parseSchema(const std::string & json) TEST(IcebergSchemaProcessor, GetSimpleTypeBoolean) { - auto type = IcebergSchemaProcessor::getSimpleType("boolean"); + auto type = IcebergSchemaProcessor::getSimpleType("boolean", getContext().context); EXPECT_EQ(type->getName(), "Bool"); } TEST(IcebergSchemaProcessor, GetSimpleTypeInt) { - auto type = IcebergSchemaProcessor::getSimpleType("int"); + auto type = IcebergSchemaProcessor::getSimpleType("int", getContext().context); EXPECT_EQ(type->getName(), "Int32"); } TEST(IcebergSchemaProcessor, GetSimpleTypeLong) { - auto type = IcebergSchemaProcessor::getSimpleType("long"); + auto type = IcebergSchemaProcessor::getSimpleType("long", getContext().context); EXPECT_EQ(type->getName(), "Int64"); } TEST(IcebergSchemaProcessor, GetSimpleTypeBigint) { - auto type = IcebergSchemaProcessor::getSimpleType("bigint"); + auto type = IcebergSchemaProcessor::getSimpleType("bigint", getContext().context); EXPECT_EQ(type->getName(), "Int64"); } TEST(IcebergSchemaProcessor, GetSimpleTypeFloat) { - auto type = IcebergSchemaProcessor::getSimpleType("float"); + auto type = IcebergSchemaProcessor::getSimpleType("float", getContext().context); EXPECT_EQ(type->getName(), "Float32"); } TEST(IcebergSchemaProcessor, GetSimpleTypeDouble) { - auto type = IcebergSchemaProcessor::getSimpleType("double"); + auto type = IcebergSchemaProcessor::getSimpleType("double", getContext().context); EXPECT_EQ(type->getName(), "Float64"); } TEST(IcebergSchemaProcessor, GetSimpleTypeDate) { - auto type = IcebergSchemaProcessor::getSimpleType("date"); + auto type = IcebergSchemaProcessor::getSimpleType("date", getContext().context); EXPECT_EQ(type->getName(), "Date32"); } TEST(IcebergSchemaProcessor, GetSimpleTypeTime) { - auto type = IcebergSchemaProcessor::getSimpleType("time"); + auto type = IcebergSchemaProcessor::getSimpleType("time", getContext().context); EXPECT_EQ(type->getName(), "Int64"); } TEST(IcebergSchemaProcessor, GetSimpleTypeTimestamp) { - auto type = IcebergSchemaProcessor::getSimpleType("timestamp"); + auto type = IcebergSchemaProcessor::getSimpleType("timestamp", getContext().context); EXPECT_EQ(type->getName(), "DateTime64(6)"); } TEST(IcebergSchemaProcessor, GetSimpleTypeTimestamptz) { - auto type = IcebergSchemaProcessor::getSimpleType("timestamptz"); + auto type = IcebergSchemaProcessor::getSimpleType("timestamptz", getContext().context); EXPECT_EQ(type->getName(), "DateTime64(6, 'UTC')"); } TEST(IcebergSchemaProcessor, GetSimpleTypeTimestampNs) { - auto type = IcebergSchemaProcessor::getSimpleType("timestamp_ns"); + auto type = IcebergSchemaProcessor::getSimpleType("timestamp_ns", getContext().context); EXPECT_EQ(type->getName(), "DateTime64(9)"); } TEST(IcebergSchemaProcessor, GetSimpleTypeTimestamptzNs) { - auto type = IcebergSchemaProcessor::getSimpleType("timestamptz_ns"); + auto type = IcebergSchemaProcessor::getSimpleType("timestamptz_ns", getContext().context); EXPECT_EQ(type->getName(), "DateTime64(9, 'UTC')"); } TEST(IcebergSchemaProcessor, GetSimpleTypeString) { - auto type = IcebergSchemaProcessor::getSimpleType("string"); + auto type = IcebergSchemaProcessor::getSimpleType("string", getContext().context); EXPECT_EQ(type->getName(), "String"); } TEST(IcebergSchemaProcessor, GetSimpleTypeBinary) { - auto type = IcebergSchemaProcessor::getSimpleType("binary"); + auto type = IcebergSchemaProcessor::getSimpleType("binary", getContext().context); EXPECT_EQ(type->getName(), "String"); } TEST(IcebergSchemaProcessor, GetSimpleTypeUuid) { - auto type = IcebergSchemaProcessor::getSimpleType("uuid"); + auto type = IcebergSchemaProcessor::getSimpleType("uuid", getContext().context); EXPECT_EQ(type->getName(), "UUID"); } TEST(IcebergSchemaProcessor, GetSimpleTypeFixed) { - auto type = IcebergSchemaProcessor::getSimpleType("fixed[16]"); + auto type = IcebergSchemaProcessor::getSimpleType("fixed[16]", getContext().context); EXPECT_EQ(type->getName(), "FixedString(16)"); } TEST(IcebergSchemaProcessor, GetSimpleTypeDecimal) { - auto type = IcebergSchemaProcessor::getSimpleType("decimal(10, 2)"); + auto type = IcebergSchemaProcessor::getSimpleType("decimal(10, 2)", getContext().context); EXPECT_EQ(type->getName(), "Decimal(10, 2)"); } TEST(IcebergSchemaProcessor, GetSimpleTypeUnknownThrows) { - EXPECT_THROW(IcebergSchemaProcessor::getSimpleType("unknown_type"), DB::Exception); + EXPECT_THROW(IcebergSchemaProcessor::getSimpleType("unknown_type", getContext().context), DB::Exception); } /// The Iceberg primitive type grammar is a closed set: scalars, decimal(P, S) and fixed[N] whose @@ -132,7 +133,7 @@ TEST(IcebergSchemaProcessor, GetSimpleTypeUnknownThrows) /// any comparison runs, so canonicalizeTypeSpacing never sees whitespace inside a quoted literal. TEST(IcebergSchemaProcessor, GetSimpleTypeWithStringLiteralArgumentThrows) { - EXPECT_THROW(IcebergSchemaProcessor::getSimpleType("MyType('Hello ( world )')"), DB::Exception); + EXPECT_THROW(IcebergSchemaProcessor::getSimpleType("MyType('Hello ( world )')", getContext().context), DB::Exception); } /// The same string-literal-bearing spelling must be rejected as an initial schema type, i.e. the @@ -140,8 +141,8 @@ TEST(IcebergSchemaProcessor, GetSimpleTypeWithStringLiteralArgumentThrows) TEST(IcebergSchemaProcessor, InitialSchemaTypeWithStringLiteralArgumentThrows) { auto schema = parseSchema(R"json({"schema-id":0,"fields":[{"id":1,"name":"c0","required":false,"type":"MyType('Hello ( world )')"}]})json"); - IcebergSchemaProcessor processor; - EXPECT_THROW(processor.addIcebergTableSchema(schema), DB::Exception); + IcebergSchemaProcessor processor(getContext().context); + EXPECT_THROW(processor.addIcebergTableSchema(schema, getContext().context), DB::Exception); } /// The primitive parser must accept the same inner-whitespace spellings that the @@ -150,13 +151,13 @@ TEST(IcebergSchemaProcessor, InitialSchemaTypeWithStringLiteralArgumentThrows) /// "fixed[ 16 ]" fail to parse even though they denote decimal(20, 0) / fixed[16]. TEST(IcebergSchemaProcessor, GetSimpleTypeDecimalInnerWhitespace) { - auto type = IcebergSchemaProcessor::getSimpleType("decimal( 20, 0 )"); + auto type = IcebergSchemaProcessor::getSimpleType("decimal( 20, 0 )", getContext().context); EXPECT_EQ(type->getName(), "Decimal(20, 0)"); } TEST(IcebergSchemaProcessor, GetSimpleTypeFixedInnerWhitespace) { - auto type = IcebergSchemaProcessor::getSimpleType("fixed[ 16 ]"); + auto type = IcebergSchemaProcessor::getSimpleType("fixed[ 16 ]", getContext().context); EXPECT_EQ(type->getName(), "FixedString(16)"); } @@ -170,9 +171,9 @@ TEST(IcebergSchemaProcessor, DecimalTypeWhitespaceIsInsensitive) { auto first = parseSchema(R"json({"schema-id":0,"fields":[{"id":1,"name":"c0","required":false,"type":"decimal(20,0)"}]})json"); auto second = parseSchema(R"json({"schema-id":0,"fields":[{"id":1,"name":"c0","required":false,"type":"decimal(20, 0)"}]})json"); - IcebergSchemaProcessor processor; - processor.addIcebergTableSchema(first); - EXPECT_NO_THROW(processor.addIcebergTableSchema(second)); + IcebergSchemaProcessor processor(getContext().context); + processor.addIcebergTableSchema(first, getContext().context); + EXPECT_NO_THROW(processor.addIcebergTableSchema(second, getContext().context)); } /// A genuinely different type bound to the same schema-id must still be rejected. @@ -180,9 +181,9 @@ TEST(IcebergSchemaProcessor, RebindingSchemaIdToDifferentTypeStillRejected) { auto first = parseSchema(R"json({"schema-id":0,"fields":[{"id":1,"name":"c0","required":false,"type":"decimal(20,0)"}]})json"); auto second = parseSchema(R"json({"schema-id":0,"fields":[{"id":1,"name":"c0","required":false,"type":"decimal(20,2)"}]})json"); - IcebergSchemaProcessor processor; - processor.addIcebergTableSchema(first); - EXPECT_THROW(processor.addIcebergTableSchema(second), DB::Exception); + IcebergSchemaProcessor processor(getContext().context); + processor.addIcebergTableSchema(first, getContext().context); + EXPECT_THROW(processor.addIcebergTableSchema(second, getContext().context), DB::Exception); } /// A renamed field bound to the same schema-id must still be rejected (issue #107316). @@ -190,9 +191,9 @@ TEST(IcebergSchemaProcessor, RebindingSchemaIdToRenamedFieldStillRejected) { auto first = parseSchema(R"json({"schema-id":0,"fields":[{"id":1,"name":"c0","required":false,"type":"long"}]})json"); auto second = parseSchema(R"json({"schema-id":0,"fields":[{"id":1,"name":"c9","required":false,"type":"long"}]})json"); - IcebergSchemaProcessor processor; - processor.addIcebergTableSchema(first); - EXPECT_THROW(processor.addIcebergTableSchema(second), DB::Exception); + IcebergSchemaProcessor processor(getContext().context); + processor.addIcebergTableSchema(first, getContext().context); + EXPECT_THROW(processor.addIcebergTableSchema(second, getContext().context), DB::Exception); } /// The whitespace-insensitive comparison must reach into list/map wrappers: the nested @@ -204,9 +205,9 @@ TEST(IcebergSchemaProcessor, ListElementDecimalWhitespaceIsInsensitive) R"json({"schema-id":0,"fields":[{"id":1,"name":"c0","required":false,"type":{"type":"list","element-id":2,"element-required":false,"element":"decimal(20,0)"}}]})json"); auto second = parseSchema( R"json({"schema-id":0,"fields":[{"id":1,"name":"c0","required":false,"type":{"type":"list","element-id":2,"element-required":false,"element":"decimal(20, 0)"}}]})json"); - IcebergSchemaProcessor processor; - processor.addIcebergTableSchema(first); - EXPECT_NO_THROW(processor.addIcebergTableSchema(second)); + IcebergSchemaProcessor processor(getContext().context); + processor.addIcebergTableSchema(first, getContext().context); + EXPECT_NO_THROW(processor.addIcebergTableSchema(second, getContext().context)); } /// Same for map key/value primitive types (here map). @@ -216,9 +217,9 @@ TEST(IcebergSchemaProcessor, MapKeyValueDecimalWhitespaceIsInsensitive) R"json({"schema-id":0,"fields":[{"id":1,"name":"c0","required":false,"type":{"type":"map","key-id":2,"key":"decimal(20,0)","value-id":3,"value-required":false,"value":"decimal(10,2)"}}]})json"); auto second = parseSchema( R"json({"schema-id":0,"fields":[{"id":1,"name":"c0","required":false,"type":{"type":"map","key-id":2,"key":"decimal(20, 0)","value-id":3,"value-required":false,"value":"decimal(10, 2)"}}]})json"); - IcebergSchemaProcessor processor; - processor.addIcebergTableSchema(first); - EXPECT_NO_THROW(processor.addIcebergTableSchema(second)); + IcebergSchemaProcessor processor(getContext().context); + processor.addIcebergTableSchema(first, getContext().context); + EXPECT_NO_THROW(processor.addIcebergTableSchema(second, getContext().context)); } /// The Iceberg geography/geometry primitives carry parameters too, e.g. @@ -229,9 +230,9 @@ TEST(IcebergSchemaProcessor, GeographyTypeWhitespaceIsInsensitive) { auto first = parseSchema(R"json({"schema-id":0,"fields":[{"id":1,"name":"c0","required":false,"type":"geography(C,A)"}]})json"); auto second = parseSchema(R"json({"schema-id":0,"fields":[{"id":1,"name":"c0","required":false,"type":"geography(C, A)"}]})json"); - IcebergSchemaProcessor processor(/*allow_geo_parser_=*/true); - processor.addIcebergTableSchema(first); - EXPECT_NO_THROW(processor.addIcebergTableSchema(second)); + IcebergSchemaProcessor processor(getContext().context, /*allow_geo_parser_=*/true); + processor.addIcebergTableSchema(first, getContext().context); + EXPECT_NO_THROW(processor.addIcebergTableSchema(second, getContext().context)); } /// A geo type string carrying leading/trailing whitespace must map to its alias just like the @@ -242,9 +243,9 @@ TEST(IcebergSchemaProcessor, GeographyTypeEdgeWhitespaceIsInsensitive) { auto first = parseSchema(R"json({"schema-id":0,"fields":[{"id":1,"name":"c0","required":false,"type":" geography(C,A) "}]})json"); auto second = parseSchema(R"json({"schema-id":0,"fields":[{"id":1,"name":"c0","required":false,"type":"geography(C, A)"}]})json"); - IcebergSchemaProcessor processor(/*allow_geo_parser_=*/true); - processor.addIcebergTableSchema(first); - EXPECT_NO_THROW(processor.addIcebergTableSchema(second)); + IcebergSchemaProcessor processor(getContext().context, /*allow_geo_parser_=*/true); + processor.addIcebergTableSchema(first, getContext().context); + EXPECT_NO_THROW(processor.addIcebergTableSchema(second, getContext().context)); } /// Schema-evolution path: renaming a geo field across two schema-ids while only changing the @@ -255,11 +256,11 @@ TEST(IcebergSchemaProcessor, RenameGeoFieldAcrossSchemaIdsWithWhitespaceIsRename { auto old_schema = parseSchema(R"json({"schema-id":0,"fields":[{"id":1,"name":"a","required":false,"type":"geography(C,A)"}]})json"); auto new_schema = parseSchema(R"json({"schema-id":1,"fields":[{"id":1,"name":"b","required":false,"type":"geography(C, A)"}]})json"); - IcebergSchemaProcessor processor(/*allow_geo_parser_=*/true); - processor.addIcebergTableSchema(old_schema); - processor.addIcebergTableSchema(new_schema); + IcebergSchemaProcessor processor(getContext().context, /*allow_geo_parser_=*/true); + processor.addIcebergTableSchema(old_schema, getContext().context); + processor.addIcebergTableSchema(new_schema, getContext().context); - auto dag = processor.getSchemaTransformationDagByIds(0, 1); + auto dag = processor.getSchemaTransformationDagByIds(getContext().context, 0, 1); ASSERT_TRUE(dag); const auto & outputs = dag->getOutputs(); ASSERT_EQ(outputs.size(), 1u); @@ -272,8 +273,8 @@ TEST(IcebergSchemaProcessor, RenameGeoFieldAcrossSchemaIdsWithWhitespaceIsRename TEST(IcebergSchemaProcessor, InitialSchemaDecimalInnerWhitespaceAccepted) { auto schema = parseSchema(R"json({"schema-id":0,"fields":[{"id":1,"name":"c0","required":false,"type":"decimal( 20, 0 )"}]})json"); - IcebergSchemaProcessor processor; - EXPECT_NO_THROW(processor.addIcebergTableSchema(schema)); + IcebergSchemaProcessor processor(getContext().context); + EXPECT_NO_THROW(processor.addIcebergTableSchema(schema, getContext().context)); } /// Schema-evolution across two schema-ids where a decimal widens (allowed conversion) while its @@ -283,11 +284,11 @@ TEST(IcebergSchemaProcessor, WidenDecimalAcrossSchemaIdsWithInnerWhitespace) { auto old_schema = parseSchema(R"json({"schema-id":0,"fields":[{"id":1,"name":"c0","required":false,"type":"decimal(10,2)"}]})json"); auto new_schema = parseSchema(R"json({"schema-id":1,"fields":[{"id":1,"name":"c0","required":false,"type":"decimal( 20, 2 )"}]})json"); - IcebergSchemaProcessor processor; - processor.addIcebergTableSchema(old_schema); - processor.addIcebergTableSchema(new_schema); + IcebergSchemaProcessor processor(getContext().context); + processor.addIcebergTableSchema(old_schema, getContext().context); + processor.addIcebergTableSchema(new_schema, getContext().context); - auto dag = processor.getSchemaTransformationDagByIds(0, 1); + auto dag = processor.getSchemaTransformationDagByIds(getContext().context, 0, 1); ASSERT_TRUE(dag); const auto & outputs = dag->getOutputs(); ASSERT_EQ(outputs.size(), 1u); @@ -301,9 +302,9 @@ TEST(IcebergSchemaProcessor, RebindingListElementToDifferentTypeStillRejected) R"json({"schema-id":0,"fields":[{"id":1,"name":"c0","required":false,"type":{"type":"list","element-id":2,"element-required":false,"element":"decimal(20,0)"}}]})json"); auto second = parseSchema( R"json({"schema-id":0,"fields":[{"id":1,"name":"c0","required":false,"type":{"type":"list","element-id":2,"element-required":false,"element":"decimal(20,2)"}}]})json"); - IcebergSchemaProcessor processor; - processor.addIcebergTableSchema(first); - EXPECT_THROW(processor.addIcebergTableSchema(second), DB::Exception); + IcebergSchemaProcessor processor(getContext().context); + processor.addIcebergTableSchema(first, getContext().context); + EXPECT_THROW(processor.addIcebergTableSchema(second, getContext().context), DB::Exception); } /// Spacing normalization only removes whitespace adjacent to the delimiters '(', ')', '[', ']', ','. @@ -311,12 +312,12 @@ TEST(IcebergSchemaProcessor, RebindingListElementToDifferentTypeStillRejected) /// "decimal(2 0,0)" or "fixed[1 6]" must NOT canonicalize to a valid type and must still be rejected. TEST(IcebergSchemaProcessor, GetSimpleTypeDecimalMalformedInnerTokenWhitespaceThrows) { - EXPECT_THROW(IcebergSchemaProcessor::getSimpleType("decimal(2 0,0)"), DB::Exception); + EXPECT_THROW(IcebergSchemaProcessor::getSimpleType("decimal(2 0,0)", getContext().context), DB::Exception); } TEST(IcebergSchemaProcessor, GetSimpleTypeFixedMalformedInnerTokenWhitespaceThrows) { - EXPECT_THROW(IcebergSchemaProcessor::getSimpleType("fixed[1 6]"), DB::Exception); + EXPECT_THROW(IcebergSchemaProcessor::getSimpleType("fixed[1 6]", getContext().context), DB::Exception); } /// The same malformed spelling must be rejected when it appears as an initial schema type, i.e. the @@ -324,8 +325,8 @@ TEST(IcebergSchemaProcessor, GetSimpleTypeFixedMalformedInnerTokenWhitespaceThro TEST(IcebergSchemaProcessor, InitialSchemaDecimalMalformedInnerTokenWhitespaceThrows) { auto schema = parseSchema(R"json({"schema-id":0,"fields":[{"id":1,"name":"c0","required":false,"type":"decimal(2 0,0)"}]})json"); - IcebergSchemaProcessor processor; - EXPECT_THROW(processor.addIcebergTableSchema(schema), DB::Exception); + IcebergSchemaProcessor processor(getContext().context); + EXPECT_THROW(processor.addIcebergTableSchema(schema, getContext().context), DB::Exception); } /// Trailing garbage after the scale token must be rejected. Canonicalizing spacing does not remove @@ -333,15 +334,15 @@ TEST(IcebergSchemaProcessor, InitialSchemaDecimalMalformedInnerTokenWhitespaceTh /// stop after reading the scale and silently ignore the rest. This mirrors the fixed[N] handling. TEST(IcebergSchemaProcessor, GetSimpleTypeDecimalTrailingGarbageInScaleThrows) { - EXPECT_THROW(IcebergSchemaProcessor::getSimpleType("decimal(20,0 0)"), DB::Exception); + EXPECT_THROW(IcebergSchemaProcessor::getSimpleType("decimal(20,0 0)", getContext().context), DB::Exception); } /// The same malformed scale spelling must be rejected as an initial schema type. TEST(IcebergSchemaProcessor, InitialSchemaDecimalTrailingGarbageInScaleThrows) { auto schema = parseSchema(R"json({"schema-id":0,"fields":[{"id":1,"name":"c0","required":false,"type":"decimal(20,0 0)"}]})json"); - IcebergSchemaProcessor processor; - EXPECT_THROW(processor.addIcebergTableSchema(schema), DB::Exception); + IcebergSchemaProcessor processor(getContext().context); + EXPECT_THROW(processor.addIcebergTableSchema(schema, getContext().context), DB::Exception); } /// A new schema-id introduced during evolution is parsed at add time (getSimpleType runs on every @@ -351,9 +352,9 @@ TEST(IcebergSchemaProcessor, SchemaEvolutionDecimalTrailingGarbageInScaleThrows) { auto old_schema = parseSchema(R"json({"schema-id":0,"fields":[{"id":1,"name":"c0","required":false,"type":"decimal(10,2)"}]})json"); auto new_schema = parseSchema(R"json({"schema-id":1,"fields":[{"id":1,"name":"c0","required":false,"type":"decimal(20,2 2)"}]})json"); - IcebergSchemaProcessor processor; - processor.addIcebergTableSchema(old_schema); - EXPECT_THROW(processor.addIcebergTableSchema(new_schema), DB::Exception); + IcebergSchemaProcessor processor(getContext().context); + processor.addIcebergTableSchema(old_schema, getContext().context); + EXPECT_THROW(processor.addIcebergTableSchema(new_schema, getContext().context), DB::Exception); } /// A missing scale ("decimal(20,)") or a sign-only scale ("decimal(20,+)") is malformed metadata and @@ -361,10 +362,10 @@ TEST(IcebergSchemaProcessor, SchemaEvolutionDecimalTrailingGarbageInScaleThrows) /// at end of buffer or on a non-digit, matching how the precision is parsed. TEST(IcebergSchemaProcessor, GetSimpleTypeDecimalEmptyScaleThrows) { - EXPECT_THROW(IcebergSchemaProcessor::getSimpleType("decimal(20,)"), DB::Exception); + EXPECT_THROW(IcebergSchemaProcessor::getSimpleType("decimal(20,)", getContext().context), DB::Exception); } TEST(IcebergSchemaProcessor, GetSimpleTypeDecimalSignOnlyScaleThrows) { - EXPECT_THROW(IcebergSchemaProcessor::getSimpleType("decimal(20,+)"), DB::Exception); + EXPECT_THROW(IcebergSchemaProcessor::getSimpleType("decimal(20,+)", getContext().context), DB::Exception); } diff --git a/src/Storages/ObjectStorage/HDFS/Configuration.cpp b/src/Storages/ObjectStorage/HDFS/Configuration.cpp index 8527b4a1be5e..81e89095eb30 100644 --- a/src/Storages/ObjectStorage/HDFS/Configuration.cpp +++ b/src/Storages/ObjectStorage/HDFS/Configuration.cpp @@ -235,6 +235,14 @@ void StorageHDFSConfiguration::addStructureAndFormatToArgsIfNeeded( { addStructureAndFormatToArgsIfNeededHDFS(args, structure_, format_, context, with_structure); } + +ASTPtr StorageHDFSConfiguration::createArgsWithAccessData() const +{ + auto arguments = make_intrusive(); + arguments->children.push_back(make_intrusive(url + path.path)); + return arguments; +} + } #endif diff --git a/src/Storages/ObjectStorage/HDFS/Configuration.h b/src/Storages/ObjectStorage/HDFS/Configuration.h index c52567f4cc72..79c23bbb81b9 100644 --- a/src/Storages/ObjectStorage/HDFS/Configuration.h +++ b/src/Storages/ObjectStorage/HDFS/Configuration.h @@ -81,6 +81,8 @@ class StorageHDFSConfiguration : public StorageObjectStorageConfiguration void addStructureAndFormatToArgsIfNeeded( ASTs & args, const String & structure_, const String & format_, ContextPtr context, bool with_structure) override; + ASTPtr createArgsWithAccessData() const override; + private: void initializeFromParsedArguments(const HDFSStorageParsedArguments & parsed_arguments); void setURL(const std::string & url_); diff --git a/src/Storages/ObjectStorage/Local/Configuration.cpp b/src/Storages/ObjectStorage/Local/Configuration.cpp index 5f28aa06b306..5ab2a4a330d7 100644 --- a/src/Storages/ObjectStorage/Local/Configuration.cpp +++ b/src/Storages/ObjectStorage/Local/Configuration.cpp @@ -147,4 +147,21 @@ void StorageLocalConfiguration::fromNamedCollection(const NamedCollection & coll initializeFromParsedArguments(parsed_arguments); paths = {path}; } + +ASTPtr StorageLocalConfiguration::createArgsWithAccessData() const +{ + auto arguments = make_intrusive(); + + arguments->children.push_back(make_intrusive(path.path)); + if (getFormat() != "auto") + arguments->children.push_back(make_intrusive(getFormat())); + if (getStructure() != "auto") + arguments->children.push_back(make_intrusive(getStructure())); + if (getCompressionMethod() != "auto") + arguments->children.push_back(make_intrusive(getCompressionMethod())); + + return arguments; +} + + } diff --git a/src/Storages/ObjectStorage/Local/Configuration.h b/src/Storages/ObjectStorage/Local/Configuration.h index d72f178aaf6c..d499b3daeab0 100644 --- a/src/Storages/ObjectStorage/Local/Configuration.h +++ b/src/Storages/ObjectStorage/Local/Configuration.h @@ -79,6 +79,8 @@ class StorageLocalConfiguration : public StorageObjectStorageConfiguration void addStructureAndFormatToArgsIfNeeded(ASTs &, const String &, const String &, ContextPtr, bool) override { } + ASTPtr createArgsWithAccessData() const override; + protected: void fromAST(ASTs & args, ContextPtr context, bool with_structure) override; void fromDisk(const String & disk_name_, ASTs & args, ContextPtr context, bool with_structure) override; diff --git a/src/Storages/ObjectStorage/MultiFileStorageObjectStorageSink.cpp b/src/Storages/ObjectStorage/MultiFileStorageObjectStorageSink.cpp new file mode 100644 index 000000000000..8ffd77691213 --- /dev/null +++ b/src/Storages/ObjectStorage/MultiFileStorageObjectStorageSink.cpp @@ -0,0 +1,209 @@ +#include +#include +#include +#include +#include +#include +#include + +namespace DB +{ + +namespace ErrorCodes +{ + extern const int FILE_ALREADY_EXISTS; + extern const int CORRUPTED_DATA; +} + +namespace +{ + /// The commit file lives in the same directory as the data files. + std::string commitFilePath(const std::string & base_path, const String & transaction_id) + { + return (std::filesystem::path(base_path).parent_path() / ("commit_" + transaction_id)).string(); + } +} + +MultiFileStorageObjectStorageSink::MultiFileStorageObjectStorageSink( + const std::string & base_path_, + const String & transaction_id_, + ObjectStoragePtr object_storage_, + StorageObjectStorageConfigurationPtr configuration_, + std::size_t max_bytes_per_file_, + std::size_t max_rows_per_file_, + FileAlreadyExistsPolicy file_already_exists_policy_, + const std::function & new_file_path_callback_, + const std::optional & format_settings_, + SharedHeader sample_block_, + ContextPtr context_) + : SinkToStorage(sample_block_), + base_path(base_path_), + transaction_id(transaction_id_), + commit_file_path(commitFilePath(base_path_, transaction_id_)), + object_storage(object_storage_), + configuration(configuration_), + max_bytes_per_file(max_bytes_per_file_), + max_rows_per_file(max_rows_per_file_), + file_already_exists_policy(file_already_exists_policy_), + new_file_path_callback(new_file_path_callback_), + format_settings(format_settings_), + sample_block(sample_block_), + context(context_) +{ + if (file_already_exists_policy != FileAlreadyExistsPolicy::overwrite) + { + if (auto committed_paths = tryReadCommittedPaths()) + { + if (committed_paths->empty()) + throw Exception(ErrorCodes::CORRUPTED_DATA, + "Commit file {} lists no data files", commit_file_path); + + /// Report the whole committed set before throwing: a caller applying `skip` takes these + /// paths as the part's export result, so it needs every file and not just the first. + for (const auto & committed_path : *committed_paths) + new_file_path_callback(committed_path); + + throw Exception(ErrorCodes::FILE_ALREADY_EXISTS, + "Part was already exported as {} file(s), see commit file {}", + committed_paths->size(), commit_file_path); + } + } + + /// No commit file: either a fresh export, or an attempt that died before finalizing every + /// file. `error` still reports the leftovers as a conflict, but `skip` has to rewrite them -- + /// the files that attempt never reached carry rows no later attempt produces. + overwrite_data_files = file_already_exists_policy != FileAlreadyExistsPolicy::error; + + current_sink = createNewSink(); +} + +MultiFileStorageObjectStorageSink::~MultiFileStorageObjectStorageSink() +{ + if (isCancelled()) + current_sink->cancel(); +} + +/// Adds a counter that represents file index to the file path. +/// Example: +/// Input is `table_root/year=2025/month=12/day=12/file.parquet` +/// Output is `table_root/year=2025/month=12/day=12/file.1.parquet` +std::string MultiFileStorageObjectStorageSink::generateNewFilePath() +{ + const auto file_format = Poco::toLower(configuration->getFormat()); + const auto index_string = std::to_string(file_paths.size() + 1); + std::size_t pos = base_path.rfind(file_format); + + /// normal case - path ends with the file format + if (pos != std::string::npos) + { + const auto path_without_extension = base_path.substr(0, pos); + const auto file_format_extension = "." + file_format; + + return path_without_extension + index_string + file_format_extension; + } + + /// if no extension is found, just append the index - I am not even sure this is possible + return base_path + "." + index_string; +} + +std::shared_ptr MultiFileStorageObjectStorageSink::createNewSink() +{ + auto new_path = generateNewFilePath(); + + /// The callback runs before the conflict check on purpose: under `error` the caller discards + /// the reported path along with the failure, and under the other policies this check is off. + new_file_path_callback(new_path); + + file_paths.emplace_back(std::move(new_path)); + + if (!overwrite_data_files && object_storage->exists(StoredObject(file_paths.back()))) + { + throw Exception(ErrorCodes::FILE_ALREADY_EXISTS, "File {} already exists", file_paths.back()); + } + + return std::make_shared( + file_paths.back(), + object_storage, + format_settings, + sample_block, + context, + configuration->getFormat(), + configuration->getCompressionMethod()); +} + +void MultiFileStorageObjectStorageSink::consume(Chunk & chunk) +{ + if (isCancelled()) + { + current_sink->cancel(); + return; + } + + const auto written_bytes = current_sink->getWrittenBytes(); + + const bool exceeded_bytes_limit = max_bytes_per_file && written_bytes >= max_bytes_per_file; + const bool exceeded_rows_limit = max_rows_per_file && current_sink_written_rows >= max_rows_per_file; + + if (exceeded_bytes_limit || exceeded_rows_limit) + { + current_sink->onFinish(); + current_sink = createNewSink(); + current_sink_written_rows = 0; + } + + current_sink->consume(chunk); + current_sink_written_rows += chunk.getNumRows(); +} + +void MultiFileStorageObjectStorageSink::onFinish() +{ + current_sink->onFinish(); + commit(); +} + +std::optional> MultiFileStorageObjectStorageSink::tryReadCommittedPaths() const +{ + if (!object_storage->exists(StoredObject(commit_file_path))) + return {}; + + auto in = object_storage->readObject(StoredObject(commit_file_path), context->getReadSettings()); + + std::vector committed_paths; + while (!in->eof()) + { + String committed_path; + readStringUntilNewlineInto(committed_path, *in); + in->tryIgnore(1); + if (!committed_path.empty()) + committed_paths.emplace_back(std::move(committed_path)); + } + + return committed_paths; +} + +void MultiFileStorageObjectStorageSink::commit() +{ + /// The constructor already ruled out a pre-existing commit file for every policy but + /// `overwrite`, so seeing one here means another exporter committed this part while we wrote. + if (file_already_exists_policy != FileAlreadyExistsPolicy::overwrite + && object_storage->exists(StoredObject(commit_file_path))) + { + throw Exception(ErrorCodes::FILE_ALREADY_EXISTS, "Commit file {} already exists, aborting {} export", commit_file_path, transaction_id); + } + + auto out = object_storage->writeObject( + StoredObject(commit_file_path), + WriteMode::Rewrite, /* attributes= */ + {}, DBMS_DEFAULT_BUFFER_SIZE, + context->getWriteSettings()); + + for (const auto & p : file_paths) + { + out->write(p.data(), p.size()); + out->write("\n", 1); + } + + out->finalize(); +} + +} diff --git a/src/Storages/ObjectStorage/MultiFileStorageObjectStorageSink.h b/src/Storages/ObjectStorage/MultiFileStorageObjectStorageSink.h new file mode 100644 index 000000000000..5febf0e65da3 --- /dev/null +++ b/src/Storages/ObjectStorage/MultiFileStorageObjectStorageSink.h @@ -0,0 +1,68 @@ +#pragma once + +#include +#include + +namespace DB +{ + +/// This is useful when the data is too large to fit into a single file. +/// It will create a new file when the current file exceeds the max bytes or max rows limit. +/// Ships a commit file including the list of data files to make it transactional +class MultiFileStorageObjectStorageSink : public SinkToStorage +{ +public: + using FileAlreadyExistsPolicy = MergeTreePartExportFileAlreadyExistsPolicy; + + MultiFileStorageObjectStorageSink( + const std::string & base_path_, + const String & transaction_id_, + ObjectStoragePtr object_storage_, + StorageObjectStorageConfigurationPtr configuration_, + std::size_t max_bytes_per_file_, + std::size_t max_rows_per_file_, + FileAlreadyExistsPolicy file_already_exists_policy_, + const std::function & new_file_path_callback_, + const std::optional & format_settings_, + SharedHeader sample_block_, + ContextPtr context_); + + ~MultiFileStorageObjectStorageSink() override; + + void consume(Chunk & chunk) override; + + void onFinish() override; + + String getName() const override { return "MultiFileStorageObjectStorageSink"; } + +private: + const std::string base_path; + const String transaction_id; + /// Written by `commit` only after every data file has been finalized, so its presence -- + /// unlike that of any individual data file -- proves a previous export of this part + /// produced the whole set. + const std::string commit_file_path; + ObjectStoragePtr object_storage; + StorageObjectStorageConfigurationPtr configuration; + std::size_t max_bytes_per_file; + std::size_t max_rows_per_file; + FileAlreadyExistsPolicy file_already_exists_policy; + /// Data files left behind by an attempt that never reached `commit` have to be rewritten. + bool overwrite_data_files = false; + std::function new_file_path_callback; + const std::optional format_settings; + SharedHeader sample_block; + ContextPtr context; + + std::vector file_paths; + std::shared_ptr current_sink; + std::size_t current_sink_written_rows = 0; + + std::string generateNewFilePath(); + std::shared_ptr createNewSink(); + /// The data files a previous export of this part committed, or nothing when it never committed. + std::optional> tryReadCommittedPaths() const; + void commit(); +}; + +} diff --git a/src/Storages/ObjectStorage/ObjectStorageFilePathGenerator.h b/src/Storages/ObjectStorage/ObjectStorageFilePathGenerator.h new file mode 100644 index 000000000000..a1f21dc502d5 --- /dev/null +++ b/src/Storages/ObjectStorage/ObjectStorageFilePathGenerator.h @@ -0,0 +1,83 @@ +#pragma once + +#include +#include +#include +#include +#include + +namespace DB +{ + struct ObjectStorageFilePathGenerator + { + virtual ~ObjectStorageFilePathGenerator() = default; + std::string getPathForWrite(const std::string & partition_id) const { + return getPathForWrite(partition_id, ""); + } + virtual std::string getPathForWrite(const std::string & partition_id, const std::string & /* file_name_override */) const = 0; + virtual std::string getPathForRead() const = 0; + }; + + struct ObjectStorageWildcardFilePathGenerator : ObjectStorageFilePathGenerator + { + static constexpr const char * FILE_WILDCARD = "{_file}"; + explicit ObjectStorageWildcardFilePathGenerator(const std::string & raw_path_) : raw_path(raw_path_) {} + + using ObjectStorageFilePathGenerator::getPathForWrite; // Bring base class overloads into scope + std::string getPathForWrite(const std::string & partition_id, const std::string & file_name_override) const override + { + const auto partition_replaced_path = PartitionedSink::replaceWildcards(raw_path, partition_id); + const auto final_path = boost::replace_all_copy(partition_replaced_path, FILE_WILDCARD, file_name_override); + return final_path; + } + + std::string getPathForRead() const override + { + return raw_path; + } + + private: + std::string raw_path; + + }; + + struct ObjectStorageAppendFilePathGenerator : ObjectStorageFilePathGenerator + { + explicit ObjectStorageAppendFilePathGenerator( + const std::string & raw_path_, + const std::string & file_format_) + : raw_path(raw_path_), file_format(Poco::toLower(file_format_)){} + + using ObjectStorageFilePathGenerator::getPathForWrite; // Bring base class overloads into scope + std::string getPathForWrite(const std::string & partition_id, const std::string & file_name_override) const override + { + std::string result; + + result += raw_path; + + if (!result.empty() && result.back() != '/') + { + result += "/"; + } + + /// Not adding '/' because buildExpressionHive() always adds a trailing '/' + result += partition_id; + + const auto file_name = file_name_override.empty() ? std::to_string(generateSnowflakeID()) : file_name_override; + + result += file_name + "." + file_format; + + return result; + } + + std::string getPathForRead() const override + { + return raw_path + "**." + file_format; + } + + private: + std::string raw_path; + std::string file_format; + }; + +} diff --git a/src/Storages/ObjectStorage/ReadBufferIterator.cpp b/src/Storages/ObjectStorage/ReadBufferIterator.cpp index 40802570831e..264a6fe304c5 100644 --- a/src/Storages/ObjectStorage/ReadBufferIterator.cpp +++ b/src/Storages/ObjectStorage/ReadBufferIterator.cpp @@ -40,8 +40,8 @@ ReadBufferIterator::ReadBufferIterator( , read_keys(read_keys_) , prev_read_keys_size(read_keys_.size()) { - if (configuration->format != "auto") - format = configuration->format; + if (configuration->getFormat() != "auto") + format = configuration->getFormat(); } SchemaCache::Key ReadBufferIterator::getKeyForSchemaCache(const ObjectInfo & object_info, const String & format_name) const @@ -274,13 +274,13 @@ ReadBufferIterator::Data ReadBufferIterator::next() using ObjectInfoInArchive = StorageObjectStorageSource::ArchiveIterator::ObjectInfoInArchive; if (const auto * object_info_in_archive = dynamic_cast(current_object_info.get())) { - compression_method = chooseCompressionMethod(filename, configuration->compression_method); + compression_method = chooseCompressionMethod(filename, configuration->getCompressionMethod()); const auto & archive_reader = object_info_in_archive->archive_reader; read_buf = archive_reader->readFile(object_info_in_archive->path_in_archive, /*throw_on_not_found=*/true); } else { - compression_method = chooseCompressionMethod(filename, configuration->compression_method); + compression_method = chooseCompressionMethod(filename, configuration->getCompressionMethod()); read_buf = createReadBuffer( current_object_info->relative_path_with_metadata, object_storage, getContext(), getLogger("ReadBufferIterator")); } diff --git a/src/Storages/ObjectStorage/S3/Configuration.cpp b/src/Storages/ObjectStorage/S3/Configuration.cpp index 735827eb8222..43757dfbab5f 100644 --- a/src/Storages/ObjectStorage/S3/Configuration.cpp +++ b/src/Storages/ObjectStorage/S3/Configuration.cpp @@ -110,6 +110,7 @@ static const std::unordered_set optional_configuration_keys = "partition_columns_in_data_file", "storage_class_name", "storage_class", /// Interchangeable alias for `storage_class_name`, see issue #68551 + "storage_type", /// Private configuration options "role_arn", /// for extra_credentials "role_session_name", /// for extra_credentials @@ -791,6 +792,7 @@ void S3StorageParsedArguments::fromAST(ASTs & args, ContextPtr context, bool wit compression_method = compression_method_value.value(); } + if (auto partition_strategy_value = getFromPositionOrKeyValue("partition_strategy", args, engine_args_to_idx, key_value_args); partition_strategy_value.has_value()) { @@ -1219,6 +1221,31 @@ void StorageS3Configuration::addStructureAndFormatToArgsIfNeeded( addStructureAndFormatToArgsIfNeededS3( args, structure_, format_, context, with_structure, S3StorageParsedArguments::getMaxNumberOfArguments(with_structure)); } + +ASTPtr StorageS3Configuration::createArgsWithAccessData() const +{ + auto arguments = make_intrusive(); + + arguments->children.push_back(make_intrusive(url.uri_str)); + if (s3_settings->auth_settings[S3AuthSetting::no_sign_request]) + { + arguments->children.push_back(make_intrusive("NOSIGN")); + } + else + { + arguments->children.push_back(make_intrusive(s3_settings->auth_settings[S3AuthSetting::access_key_id].value)); + arguments->children.push_back(make_intrusive(s3_settings->auth_settings[S3AuthSetting::secret_access_key].value)); + if (!s3_settings->auth_settings[S3AuthSetting::session_token].value.empty()) + arguments->children.push_back(make_intrusive(s3_settings->auth_settings[S3AuthSetting::session_token].value)); + if (getFormat() != "auto") + arguments->children.push_back(make_intrusive(getFormat())); + if (!getCompressionMethod().empty()) + arguments->children.push_back(make_intrusive(getCompressionMethod())); + } + + return arguments; +} + } #endif diff --git a/src/Storages/ObjectStorage/S3/Configuration.h b/src/Storages/ObjectStorage/S3/Configuration.h index 13b581c07585..5beda2db5f50 100644 --- a/src/Storages/ObjectStorage/S3/Configuration.h +++ b/src/Storages/ObjectStorage/S3/Configuration.h @@ -145,6 +145,8 @@ class StorageS3Configuration : public StorageObjectStorageConfiguration ContextPtr context, bool with_structure) override; + ASTPtr createArgsWithAccessData() const override; + static bool collectCredentials(ASTPtr maybe_credentials, S3::S3AuthSettings & auth_settings_, ContextPtr local_context); S3::URI url; diff --git a/src/Storages/ObjectStorage/StorageObjectStorage.cpp b/src/Storages/ObjectStorage/StorageObjectStorage.cpp index 3ff032164a9a..de079dc70e98 100644 --- a/src/Storages/ObjectStorage/StorageObjectStorage.cpp +++ b/src/Storages/ObjectStorage/StorageObjectStorage.cpp @@ -1,4 +1,5 @@ #include +#include #include #include @@ -14,6 +15,7 @@ #include #include #include +#include #include #include @@ -46,6 +48,10 @@ #include #include #include +#include +#include +#include +#include namespace DB @@ -68,6 +74,7 @@ namespace ErrorCodes extern const int INCORRECT_DATA; extern const int BAD_ARGUMENTS; extern const int ACCESS_DENIED; + extern const int FILE_ALREADY_EXISTS; } namespace FailPoints @@ -141,13 +148,15 @@ StorageObjectStorage::StorageObjectStorage( std::optional format_settings_, LoadingStrictnessLevel mode, std::shared_ptr catalog_, - bool if_not_exists_, + bool /*if_not_exists_*/, bool is_datalake_query, bool distributed_processing_, ASTPtr partition_by_, - ASTPtr order_by_, + ASTPtr /*order_by_*/, bool is_table_function_, - bool lazy_init) + bool lazy_init, + bool updated_configuration, + std::optional sample_path_) : IStorage(table_id_) , configuration(configuration_) , object_storage(object_storage_) @@ -161,9 +170,9 @@ StorageObjectStorage::StorageObjectStorage( { configuration->initPartitionStrategy(partition_by_, columns_in_table_or_function_definition, context); configuration->check(context); - const bool need_resolve_columns_or_format = columns_in_table_or_function_definition.empty() || (configuration->format == "auto"); + const bool need_resolve_columns_or_format = columns_in_table_or_function_definition.empty() || (configuration->getFormat() == "auto"); const bool need_resolve_sample_path = context->getSettingsRef()[Setting::use_hive_partitioning] - && !configuration->partition_strategy + && !configuration->getPartitionStrategy() && !configuration->isDataLakeConfiguration(); const bool catalog_manages_created_location = catalog_ && catalog_->managesTableLocation() && mode == LoadingStrictnessLevel::CREATE; @@ -183,17 +192,9 @@ StorageObjectStorage::StorageObjectStorage( throw Exception(ErrorCodes::BAD_ARGUMENTS, "Delta lake CDF is allowed only for deltaLake table function"); } - if (!is_table_function && !columns_in_table_or_function_definition.empty() && !is_datalake_query && mode == LoadingStrictnessLevel::CREATE) - { - LOG_DEBUG(log, "Creating new storage with specified columns"); - configuration->create( - object_storage, context, columns_in_table_or_function_definition, partition_by_, order_by_, if_not_exists_, catalog, storage_id); - } - - bool updated_configuration = false; try { - if (!do_lazy_init) + if (!do_lazy_init && !updated_configuration) { if (is_table_function) configuration->lazyInitializeIfNeeded(object_storage, context); @@ -220,7 +221,7 @@ StorageObjectStorage::StorageObjectStorage( tryLogCurrentException(log, /*start of message = */ "", LogsLevel::warning); } - std::string sample_path; + std::string sample_path = sample_path_.value_or(""); ColumnsDescription columns{columns_in_table_or_function_definition}; @@ -229,7 +230,7 @@ StorageObjectStorage::StorageObjectStorage( if (configuration->isDataLakeConfiguration()) throw Exception(ErrorCodes::BAD_ARGUMENTS, "The _schema_hash placeholder is not supported for DataLake engines"); - if (configuration->partition_strategy_type == PartitionStrategyFactory::StrategyType::HIVE) + if (configuration->getPartitionStrategyType() == PartitionStrategyFactory::StrategyType::HIVE) throw Exception(ErrorCodes::BAD_ARGUMENTS, "The _schema_hash placeholder is not supported with hive partition strategy"); if (columns.empty()) @@ -239,7 +240,7 @@ StorageObjectStorage::StorageObjectStorage( } if (need_resolve_columns_or_format) - resolveSchemaAndFormat(columns, configuration->format, object_storage, configuration, format_settings, sample_path, context); + resolveSchemaAndFormat(columns, object_storage, configuration, format_settings, sample_path, context); else validateSupportedColumns(columns, *configuration); @@ -280,7 +281,7 @@ StorageObjectStorage::StorageObjectStorage( sample_path); } - bool format_supports_prewhere = FormatFactory::instance().checkIfFormatSupportsPrewhere(configuration->format, context, format_settings); + bool format_supports_prewhere = FormatFactory::instance().checkIfFormatSupportsPrewhere(configuration->getFormat(), context, format_settings); /// TODO: Known problems with datalake prewhere: /// * If the iceberg table went through schema evolution, columns read from file may need to @@ -336,8 +337,10 @@ StorageObjectStorage::StorageObjectStorage( metadata.setConstraints(constraints_); metadata.setComment(comment); - if (configuration->partition_strategy) - metadata.partition_key = configuration->partition_strategy->getPartitionKeyDescription(); + if (configuration->getPartitionStrategy()) + { + metadata.partition_key = configuration->getPartitionStrategy()->getPartitionKeyDescription(); + } metadata.setVirtuals(createVirtualColumns(metadata.columns, sample_path, context)); @@ -375,17 +378,17 @@ String StorageObjectStorage::getName() const bool StorageObjectStorage::prefersLargeBlocks() const { - return FormatFactory::instance().checkIfOutputFormatPrefersLargeBlocks(configuration->format); + return FormatFactory::instance().checkIfOutputFormatPrefersLargeBlocks(configuration->getFormat()); } bool StorageObjectStorage::parallelizeOutputAfterReading(ContextPtr context) const { - return FormatFactory::instance().checkParallelizeOutputAfterReading(configuration->format, context); + return FormatFactory::instance().checkParallelizeOutputAfterReading(configuration->getFormat(), context); } bool StorageObjectStorage::supportsSubsetOfColumns(const ContextPtr & context) const { - return FormatFactory::instance().checkIfFormatSupportsSubsetOfColumns(configuration->format, context, format_settings); + return FormatFactory::instance().checkIfFormatSupportsSubsetOfColumns(configuration->getFormat(), context, format_settings); } bool StorageObjectStorage::supportsPrewhere() const @@ -547,7 +550,7 @@ void StorageObjectStorage::updateExternalDynamicMetadataIfExists(ContextPtr quer new_metadata.columns, query_context, format_settings, - configuration->partition_strategy_type))); + configuration->getPartitionStrategyType()))); } @@ -655,8 +658,7 @@ void StorageObjectStorage::read( configuration->update(object_storage, local_context); } - - if (configuration->partition_strategy && configuration->partition_strategy_type != PartitionStrategyFactory::StrategyType::HIVE) + if (configuration->getPartitionStrategy() && configuration->getPartitionStrategyType() != PartitionStrategyFactory::StrategyType::HIVE) { throw Exception(ErrorCodes::NOT_IMPLEMENTED, "Reading from a partitioned {} storage is not implemented yet", @@ -800,9 +802,10 @@ SinkToStoragePtr StorageObjectStorage::createSink( /// Not a data lake, just raw object storage - if (configuration->partition_strategy) + if (configuration->getPartitionStrategy()) { - return std::make_shared(object_storage, configuration, format_settings, sample_block, local_context); + auto sink_creator = std::make_shared(object_storage, configuration, format_settings, sample_block, local_context); + return std::make_shared(configuration->getPartitionStrategy(), sink_creator, local_context, sample_block); } auto paths = configuration->getPaths(); @@ -818,8 +821,8 @@ SinkToStoragePtr StorageObjectStorage::createSink( format_settings, sample_block, local_context, - configuration->format, - configuration->compression_method); + configuration->getFormat(), + configuration->getCompressionMethod()); } bool StorageObjectStorage::optimize( @@ -835,6 +838,154 @@ bool StorageObjectStorage::optimize( return configuration->optimize(object_storage, metadata_snapshot, context, format_settings); } +bool StorageObjectStorage::supportsImport(ContextPtr local_context) const +{ + if (isDataLake()) + { + configuration->lazyInitializeIfNeeded(object_storage, local_context); + return configuration->getExternalMetadata()->supportsImport(local_context); + } + + if (!configuration->getPartitionStrategy()) + return false; + + if (configuration->getPartitionStrategyType() == PartitionStrategyFactory::StrategyType::WILDCARD) + return configuration->getRawPath().hasExportFilenameWildcard(); + + return configuration->getPartitionStrategyType() == PartitionStrategyFactory::StrategyType::HIVE; +} + +SinkToStoragePtr StorageObjectStorage::import( + const std::string & file_name, + Block & block_with_partition_values, + const std::function & new_file_path_callback, + MergeTreePartExportFileAlreadyExistsPolicy file_already_exists_policy, + std::size_t max_bytes_per_file, + std::size_t max_rows_per_file, + const std::optional & iceberg_metadata_json_string, + const std::optional & format_settings_, + ContextPtr local_context) +{ + if (isDataLake()) + { + configuration->lazyInitializeIfNeeded(object_storage, local_context); + auto metadata_snapshot = getInMemoryMetadataPtr(local_context, false); + return configuration->getExternalMetadata()->import( + catalog, + new_file_path_callback, + std::make_shared(metadata_snapshot->getSampleBlock()), + *iceberg_metadata_json_string, + format_settings_ ? format_settings_ : format_settings, + local_context); + } + + std::string partition_key; + + auto metadata_snapshot = getInMemoryMetadataPtr(local_context, false); + + if (configuration->getPartitionStrategy()) + { + /// The values still carry the source table's types, but the partition key is rendered as text into the + /// object path and read back in this table's types, so it must be expressed in them first. A DateTime + /// in another time zone is the sharpest case: the epoch is the same, yet its text names another instant. + Block block_in_destination_types = block_with_partition_values; + const auto destination_sample = metadata_snapshot->getSampleBlock(); + for (auto & column : block_in_destination_types) + { + if (!destination_sample.has(column.name)) + continue; + + const auto & destination_type = destination_sample.getByName(column.name).type; + column.column = castColumn(column, destination_type); + /// castColumn is a no-op between types IDataType::equals considers equal, which includes DateTime + /// with different time zones, so relabel the column: serialization follows the type, not the values. + column.type = destination_type; + } + + const auto column_with_partition_key = configuration->getPartitionStrategy()->computePartitionKey(block_in_destination_types); + + if (!column_with_partition_key->empty()) + { + partition_key = column_with_partition_key->getDataAt(0); + } + } + + const auto base_path = configuration->getPathForWrite(partition_key, file_name).path; + + return std::make_shared( + base_path, + /* transaction_id= */ file_name, /// not pretty, but the sink needs some sort of id to generate the commit file name. Using the source part name should be enough + object_storage, + configuration, + max_bytes_per_file, + max_rows_per_file, + file_already_exists_policy, + new_file_path_callback, + format_settings_ ? format_settings_ : format_settings, + std::make_shared(metadata_snapshot->getSampleBlock()), + local_context); +} + +IStorage::ExportPartitionCommitInfo StorageObjectStorage::commitExportPartitionTransaction( + const String & transaction_id, + const String & partition_id, + const Strings & exported_paths, + const IcebergCommitExportPartitionArguments & iceberg_commit_export_partition_arguments, + ContextPtr local_context) +{ + if (isDataLake()) + { + /// Parse the Iceberg metadata snapshot (stored in ZooKeeper at export-start time) only to + /// extract the schema-id and partition-spec-id that were current when the export began. + /// partition_columns and partition_types are derived inside commitExportPartitionTransaction + /// from the same JSON; the representative source partition columns are carried here so the + /// partition tuple can be recomputed through the destination transform. + Poco::JSON::Parser iceberg_parser; + Poco::JSON::Object::Ptr iceberg_metadata = + iceberg_parser.parse(iceberg_commit_export_partition_arguments.metadata_json_string).extract(); + + const auto original_schema_id = iceberg_metadata->getValue(Iceberg::f_current_schema_id); + const auto partition_spec_id = iceberg_metadata->getValue(Iceberg::f_default_spec_id); + + configuration->lazyInitializeIfNeeded(object_storage, local_context); + auto metadata_snapshot = getInMemoryMetadataPtr(local_context, false); + return configuration->getExternalMetadata()->commitExportPartitionTransaction( + catalog, + storage_id, + transaction_id, + original_schema_id, + partition_spec_id, + iceberg_commit_export_partition_arguments.partition_source_block, + std::make_shared(metadata_snapshot->getSampleBlock()), + exported_paths, + configuration, + local_context); + } + + const String commit_object = configuration->getRawPath().path + "/commit_" + partition_id + "_" + transaction_id; + + ExportPartitionCommitInfo result; + result.commit_marker_file = commit_object; + + /// if file already exists, nothing to be done + if (object_storage->exists(StoredObject(commit_object))) + { + LOG_DEBUG(getLogger("StorageObjectStorage"), "Commit file already exists, nothing to be done: {}", commit_object); + /// Still surface the path: observability does not require we wrote it, + /// only that it is the committed marker for this transaction. + return result; + } + + auto out = object_storage->writeObject(StoredObject(commit_object), WriteMode::Rewrite, /* attributes= */ {}, DBMS_DEFAULT_BUFFER_SIZE, local_context->getWriteSettings()); + for (const auto & p : exported_paths) + { + out->write(p.data(), p.size()); + out->write("\n", 1); + } + out->finalize(); + return result; +} + void StorageObjectStorage::truncate( const ASTPtr & /* query */, const StorageMetadataPtr & /* metadata_snapshot */, @@ -926,7 +1077,7 @@ ColumnsDescription StorageObjectStorage::resolveSchemaFromData( { ObjectInfos read_keys; auto iterator = createReadBufferIterator(object_storage, configuration, format_settings, read_keys, context); - auto schema = readSchemaFromFormat(configuration->format, format_settings, *iterator, context); + auto schema = readSchemaFromFormat(configuration->getFormat(), format_settings, *iterator, context); sample_path = iterator->getLastFilePath(); return schema; } @@ -947,7 +1098,7 @@ std::string StorageObjectStorage::resolveFormatFromData( std::pair StorageObjectStorage::resolveSchemaAndFormatFromData( const ObjectStoragePtr & object_storage, - const StorageObjectStorageConfigurationPtr & configuration, + StorageObjectStorageConfigurationPtr & configuration, const std::optional & format_settings, std::string & sample_path, const ContextPtr & context) @@ -956,7 +1107,7 @@ std::pair StorageObjectStorage::resolveSchemaAn auto iterator = createReadBufferIterator(object_storage, configuration, format_settings, read_keys, context); auto [columns, format] = detectFormatAndReadSchema(format_settings, *iterator, context); sample_path = iterator->getLastFilePath(); - configuration->format = format; + configuration->setFormat(format); return std::pair(columns, format); } diff --git a/src/Storages/ObjectStorage/StorageObjectStorage.h b/src/Storages/ObjectStorage/StorageObjectStorage.h index a76d01d4318f..437df90a67fc 100644 --- a/src/Storages/ObjectStorage/StorageObjectStorage.h +++ b/src/Storages/ObjectStorage/StorageObjectStorage.h @@ -6,6 +6,7 @@ #include #include #include +#include "Storages/ObjectStorage/ObjectStorageFilePathGenerator.h" #include #include #include @@ -59,7 +60,9 @@ class StorageObjectStorage : public IStorage, public IBackgroundOperation ASTPtr partition_by_ = nullptr, ASTPtr order_by_ = nullptr, bool is_table_function_ = false, - bool lazy_init = false); + bool lazy_init = false, + bool updated_configuration = false, // avoid double update configuration from cluster and local versions + std::optional sample_path_ = std::nullopt); String getName() const override; @@ -92,6 +95,26 @@ class StorageObjectStorage : public IStorage, public IBackgroundOperation const StorageMetadataPtr & metadata_snapshot, const ContextPtr & context); + bool supportsImport(ContextPtr) const override; + + SinkToStoragePtr import( + const std::string & /* file_name */, + Block & /* block_with_partition_values */, + const std::function & new_file_path_callback, + MergeTreePartExportFileAlreadyExistsPolicy /* file_already_exists_policy */, + std::size_t /* max_bytes_per_file */, + std::size_t /* max_rows_per_file */, + const std::optional & /* iceberg_metadata_json_string */, + const std::optional & /* format_settings_ */, + ContextPtr /* context */) override; + + ExportPartitionCommitInfo commitExportPartitionTransaction( + const String & transaction_id, + const String & partition_id, + const Strings & exported_paths, + const IcebergCommitExportPartitionArguments & iceberg_commit_export_partition_arguments, + ContextPtr local_context) override; + void truncate( const ASTPtr & query, const StorageMetadataPtr & metadata_snapshot, @@ -155,7 +178,7 @@ class StorageObjectStorage : public IStorage, public IBackgroundOperation static std::pair resolveSchemaAndFormatFromData( const ObjectStoragePtr & object_storage, - const StorageObjectStorageConfigurationPtr & configuration, + StorageObjectStorageConfigurationPtr & configuration, const std::optional & format_settings, std::string & sample_path, const ContextPtr & context); @@ -164,6 +187,9 @@ class StorageObjectStorage : public IStorage, public IBackgroundOperation void updateExternalDynamicMetadataIfExists(ContextPtr query_context) override; + /// Valid once a deferred hive partitioning sample path has been resolved. + const NamesAndTypesList & getHivePartitionColumns() const { return hive_partition_columns_to_read_from_file_path; } + std::shared_ptr getExternalMetadata(ContextPtr query_context); std::shared_ptr getCatalog() const { return catalog; } diff --git a/src/Storages/ObjectStorage/StorageObjectStorageCluster.cpp b/src/Storages/ObjectStorage/StorageObjectStorageCluster.cpp index 8e04a3f576f2..f5aa79d5340e 100644 --- a/src/Storages/ObjectStorage/StorageObjectStorageCluster.cpp +++ b/src/Storages/ObjectStorage/StorageObjectStorageCluster.cpp @@ -1,3 +1,4 @@ +#include #include #include @@ -16,9 +17,16 @@ #include #include +#include +#include +#include +#include +#include #include #include #include +#include +#include #include #include @@ -37,11 +45,21 @@ namespace Setting extern const SettingsBool use_hive_partitioning; extern const SettingsBool cluster_function_process_archive_on_multiple_nodes; extern const SettingsObjectStorageGranularityLevel cluster_table_function_split_granularity; + extern const SettingsBool parallel_replicas_for_cluster_engines; + extern const SettingsString object_storage_cluster; + extern const SettingsBool object_storage_remote_initiator; + extern const SettingsString object_storage_remote_initiator_cluster; + extern const SettingsInt64 delta_lake_snapshot_start_version; + extern const SettingsInt64 delta_lake_snapshot_end_version; + extern const SettingsUInt64 lock_object_storage_task_distribution_ms; + extern const SettingsBool allow_experimental_iceberg_read_optimization; } namespace ErrorCodes { extern const int LOGICAL_ERROR; + extern const int BAD_ARGUMENTS; + extern const int INVALID_SETTING_VALUE; extern const int NOT_IMPLEMENTED; } @@ -55,6 +73,14 @@ String StorageObjectStorageCluster::getPathSample(ContextPtr context) auto query_settings = configuration->getQuerySettings(context); /// We don't want to throw an exception if there are no files with specified path. query_settings.throw_on_zero_files_match = false; + + if (!configuration->isArchive()) + { + const auto & path = configuration->getPathForRead(); + if (!path.hasGlobs()) + return path.path; + } + auto file_iterator = StorageObjectStorageSource::createFileIterator( configuration, query_settings, @@ -67,11 +93,14 @@ String StorageObjectStorageCluster::getPathSample(ContextPtr context) {}, // virtual_columns {}, // hive_columns nullptr, // read_keys - {} // file_progress_callback + {}, // file_progress_callback + false, // ignore_archive_globs + true // skip_object_metadata ); if (auto file = file_iterator->next(0)) return file->getPath(); + return ""; } @@ -83,33 +112,115 @@ StorageObjectStorageCluster::StorageObjectStorageCluster( const ColumnsDescription & columns_in_table_or_function_definition, const ConstraintsDescription & constraints_, const ASTPtr & partition_by, + const ASTPtr & order_by, ContextPtr context_, - bool is_table_function, + const String & comment_, std::optional format_settings_, - std::shared_ptr catalog_) + LoadingStrictnessLevel mode_, + std::shared_ptr catalog, + bool if_not_exists, + bool is_datalake_query, + bool is_table_function, + bool lazy_init) : IStorageCluster( cluster_name_, table_id_, getLogger(fmt::format("{}({})", configuration_->getEngineName(), table_id_.table_name))) , configuration{configuration_} , object_storage(object_storage_) - , format_settings(std::move(format_settings_)) - , catalog(std::move(catalog_)) + , cluster_name_in_settings(false) { configuration->initPartitionStrategy(partition_by, columns_in_table_or_function_definition, context_); configuration->check(context_); - /// We allow exceptions to be thrown on update(), - /// because Cluster engine can only be used as table function, - /// so no lazy initialization is allowed. - configuration->update(object_storage, context_); + + const bool need_resolve_columns_or_format = columns_in_table_or_function_definition.empty() || (configuration->getFormat() == "auto"); + const bool do_lazy_init = lazy_init && !need_resolve_columns_or_format && catalog; + + auto log = getLogger("StorageObjectStorageCluster"); + + bool is_delta_lake_cdf = context_->getSettingsRef()[Setting::delta_lake_snapshot_start_version] != -1 + || context_->getSettingsRef()[Setting::delta_lake_snapshot_end_version] != -1; + + if (!is_table_function && is_delta_lake_cdf) + { + throw Exception(ErrorCodes::BAD_ARGUMENTS, "Delta lake CDF is allowed only for deltaLake table function"); + } + + if (!is_table_function && !columns_in_table_or_function_definition.empty() && !is_datalake_query && mode_ == LoadingStrictnessLevel::CREATE) + { + LOG_DEBUG(log, "Creating new storage with specified columns"); + configuration->create( + object_storage, context_, columns_in_table_or_function_definition, partition_by, order_by, if_not_exists, catalog, table_id_); + } + + bool updated_configuration = false; + try + { + if (!do_lazy_init) + { + if (is_table_function) + configuration->lazyInitializeIfNeeded(object_storage, context_); + else + configuration->update(object_storage, context_); + updated_configuration = true; + } + } + catch (...) + { + // If we don't have format or schema yet, we can't ignore failed configuration update, + // because relevant configuration is crucial for format and schema inference + if (mode_ <= LoadingStrictnessLevel::CREATE || need_resolve_columns_or_format) + { + throw; + } + tryLogCurrentException(log); + } ColumnsDescription columns{columns_in_table_or_function_definition}; + + if (configuration->getRawPath().hasSchemaHashWildcard()) + { + if (configuration->isDataLakeConfiguration()) + throw Exception(ErrorCodes::BAD_ARGUMENTS, "The _schema_hash placeholder is not supported for DataLake engines"); + + if (configuration->getPartitionStrategyType() == PartitionStrategyFactory::StrategyType::HIVE) + throw Exception(ErrorCodes::BAD_ARGUMENTS, "The _schema_hash placeholder is not supported with hive partition strategy"); + + if (columns.empty()) + throw Exception(ErrorCodes::BAD_ARGUMENTS, "Cannot use _schema_hash placeholder without explicitly specifying columns"); + + configuration->setSchemaHash(StorageObjectStorageConfiguration::computeSchemaHash(columns)); + } + std::string sample_path; - resolveSchemaAndFormat(columns, configuration->format, object_storage, configuration, {}, sample_path, context_); + if (need_resolve_columns_or_format) + resolveSchemaAndFormat(columns, object_storage, configuration, {}, sample_path, context_); + else + validateSupportedColumns(columns, *configuration); - if (sample_path.empty() - && context_->getSettingsRef()[Setting::use_hive_partitioning] - && !configuration->isDataLakeConfiguration() - && !configuration->partition_strategy) - sample_path = getPathSample(context_); + const bool need_resolve_sample_path = context_->getSettingsRef()[Setting::use_hive_partitioning] + && !configuration->getPartitionStrategy() + && !configuration->isDataLakeConfiguration(); + + /// Resolving the sample path requires listing the object storage. Defer it to the first use of + /// the table, so that CREATE, ATTACH and server startup do not depend on the endpoint. + /// `pure_storage` carries the same deferral and performs the resolution, which updates the + /// metadata this storage serves through `getInMemoryMetadataPtr`. + hive_partitioning_sample_path_deferred = !is_table_function && need_resolve_sample_path && !need_resolve_columns_or_format; + + if (updated_configuration && sample_path.empty() && need_resolve_sample_path && !hive_partitioning_sample_path_deferred) + { + try + { + sample_path = getPathSample(context_); + } + catch (...) + { + LOG_WARNING( + log, + "Failed to list object storage, cannot use hive partitioning. " + "Error: {}", + getCurrentExceptionMessage(true)); + } + } /// Not grabbing the file_columns because it is not necessary to do it here. std::tie(hive_partition_columns_to_read_from_file_path, std::ignore) = HivePartitioningUtils::setupHivePartitioningForObjectStorage( @@ -122,7 +233,8 @@ StorageObjectStorageCluster::StorageObjectStorageCluster( StorageInMemoryMetadata metadata; metadata.setColumns(columns); - if (is_table_function && configuration->isDataLakeConfiguration()) + + if (!do_lazy_init && is_table_function && configuration->isDataLakeConfiguration()) { /// For datalake table functions, always pin the current snapshot version so that /// query execution uses the same snapshot as query analysis (logical-race fix). @@ -139,14 +251,53 @@ StorageObjectStorageCluster::StorageObjectStorageCluster( } metadata.setConstraints(constraints_); + + if (configuration->getPartitionStrategy()) + { + metadata.partition_key = configuration->getPartitionStrategy()->getPartitionKeyDescription(); + } + metadata.setVirtuals(VirtualColumnUtils::getVirtualsForFileLikeStorage( metadata.columns, context_, /* format_settings */std::nullopt, - configuration->partition_strategy_type, + configuration->getPartitionStrategyType(), sample_path)); setInMemoryMetadata(metadata); + + const auto can_use_parallel_replicas = !cluster_name_.empty() + && context_->getSettingsRef()[Setting::parallel_replicas_for_cluster_engines] + && context_->canUseTaskBasedParallelReplicas() + && !context_->isDistributed(); + + bool can_use_distributed_iterator = + context_->getClientInfo().collaborate_with_initiator && + can_use_parallel_replicas; + + pure_storage = std::make_shared( + configuration, + object_storage, + context_, + getStorageID(), + metadata.getColumns(), + metadata.getConstraints(), + comment_, + format_settings_, + mode_, + catalog, + if_not_exists, + is_datalake_query, + /* distributed_processing */can_use_distributed_iterator, + partition_by, + order_by, + /* is_table_function */is_table_function, + /* lazy_init */lazy_init, + updated_configuration, + sample_path); + + /// Virtual columns are a part of StorageInMemoryMetadata, so they are propagated together with it. + pure_storage->setInMemoryMetadata(metadata); } std::string StorageObjectStorageCluster::getName() const @@ -154,129 +305,164 @@ std::string StorageObjectStorageCluster::getName() const return configuration->getEngineName(); } -SinkToStoragePtr StorageObjectStorageCluster::write( - const ASTPtr &, - const StorageMetadataPtr & metadata_snapshot, - ContextPtr local_context, - bool /* async_insert */) +std::optional StorageObjectStorageCluster::totalRows(ContextPtr query_context) const { - if (!configuration->isDataLakeConfiguration()) - configuration->update(object_storage, local_context); - - return StorageObjectStorage::createSink( - configuration, object_storage, getStorageID(), format_settings, catalog, metadata_snapshot, local_context); + if (pure_storage) + return pure_storage->totalRows(query_context); + configuration->lazyInitializeIfNeeded( + object_storage, + query_context); + return configuration->totalRows(query_context); } -bool StorageObjectStorageCluster::supportsParallelInsert() const +std::optional StorageObjectStorageCluster::totalBytes(ContextPtr query_context) const { - if (configuration->isDataLakeConfiguration()) - configuration->lazyInitializeIfNeeded(object_storage, CurrentThread::tryGetQueryContext()); - return configuration->supportsParallelInsert(); + if (pure_storage) + return pure_storage->totalBytes(query_context); + configuration->lazyInitializeIfNeeded( + object_storage, + query_context); + return configuration->totalBytes(query_context); } -bool StorageObjectStorageCluster::supportsDelete() const +bool StorageObjectStorageCluster::updateQueryForDistributedEngineIfNeeded(ASTPtr & query, ContextPtr context, bool make_cluster_function) { - if (configuration->isDataLakeConfiguration()) - configuration->lazyInitializeIfNeeded(object_storage, CurrentThread::tryGetQueryContext()); - return configuration->supportsDelete(); -} + // Change table engine on table function for distributed request + // CREATE TABLE t (...) ENGINE=IcebergS3(...) + // SELECT * FROM t + // change on + // SELECT * FROM icebergS3(...) + // to execute on cluster nodes -bool StorageObjectStorageCluster::optimize( - const ASTPtr & /*query*/, - const StorageMetadataPtr & metadata_snapshot, - const ASTPtr & /*partition*/, - bool /*final*/, - bool /*deduplicate*/, - const Names & /*deduplicate_by_columns*/, - bool /*cleanup*/, - ContextPtr context) -{ - return configuration->optimize(object_storage, metadata_snapshot, context, format_settings); -} + auto * select_query = query->as(); + if (!select_query || !select_query->tables()) + return false; -void StorageObjectStorageCluster::mutate(const MutationCommands & commands, ContextPtr context) -{ - updateExternalDynamicMetadataIfExists(context); - auto metadata_snapshot = getInMemoryMetadataPtr(context, false); - configuration->mutate(commands, context, shared_from_this(), getStorageID(), metadata_snapshot, catalog, format_settings); -} + auto * tables = select_query->tables()->as(); -void StorageObjectStorageCluster::checkMutationIsPossible(const MutationCommands & commands, const Settings & /*settings*/) const -{ - configuration->checkMutationIsPossible(object_storage, CurrentThread::tryGetQueryContext(), commands); -} + if (tables->children.empty()) + throw Exception( + ErrorCodes::LOGICAL_ERROR, + "Expected SELECT query from table with engine {}, got '{}'", + configuration->getEngineName(), query->formatForLogging()); -void StorageObjectStorageCluster::alter(const AlterCommands & params, ContextPtr context, AlterLockHolder & /*alter_lock_holder*/) -{ - auto metadata_snapshot = getInMemoryMetadataPtr(context, false); - StorageInMemoryMetadata new_metadata = *metadata_snapshot; - params.apply(new_metadata, context); + auto * table_expression = tables->children[0]->as()->table_expression->as(); - checkMetadataDoesNotExceedMaxQuerySize(getStorageID(), new_metadata, context); + if (!table_expression) + return false; - configuration->alter(object_storage, params, context, getStorageID(), catalog); + if (!table_expression->database_and_table_name) + return false; - if (catalog) - return; + auto & table_identifier_typed = table_expression->database_and_table_name->as(); - const auto storage_id = getStorageID(); - DatabaseCatalog::instance() - .getDatabase(storage_id.database_name) - ->alterTable(context, storage_id, new_metadata, /*validate_new_create_query=*/true); - setInMemoryMetadata(new_metadata); -} + auto table_alias = table_identifier_typed.tryGetAlias(); -void StorageObjectStorageCluster::checkAlterIsPossible(const AlterCommands & commands, ContextPtr context) const -{ - configuration->checkAlterIsPossible(object_storage, context, commands); -} + auto storage_engine_name = configuration->getEngineName(); + if (storage_engine_name == "Iceberg") + { + switch (configuration->getType()) + { + case ObjectStorageType::S3: + storage_engine_name = "IcebergS3"; + break; + case ObjectStorageType::Azure: + storage_engine_name = "IcebergAzure"; + break; + case ObjectStorageType::HDFS: + storage_engine_name = "IcebergHDFS"; + break; + default: + throw Exception( + ErrorCodes::LOGICAL_ERROR, + "Can't find table function for engine {}", + storage_engine_name + ); + } + } -Pipe StorageObjectStorageCluster::executeCommand(const String & command_name, const ASTPtr & args, ContextPtr context) -{ - if (!configuration->isDataLakeConfiguration()) - throw Exception(ErrorCodes::NOT_IMPLEMENTED, "EXECUTE command '{}' is not supported by this storage", command_name); + static std::unordered_map engine_to_function = { + {"S3", "s3"}, + {"Azure", "azureBlobStorage"}, + {"HDFS", "hdfs"}, + {"Iceberg", "iceberg"}, + {"IcebergS3", "icebergS3"}, + {"IcebergAzure", "icebergAzure"}, + {"IcebergHDFS", "icebergHDFS"}, + {"IcebergLocal", "icebergLocal"}, + {"DeltaLake", "deltaLake"}, + {"DeltaLakeS3", "deltaLakeS3"}, + {"DeltaLakeAzure", "deltaLakeAzure"}, + {"DeltaLakeLocal", "deltaLakeLocal"}, + {"Hudi", "hudi"}, + {"COSN", "cosn"}, + {"GCS", "gcs"}, + {"OSS", "oss"}, + }; - configuration->update(object_storage, context); - auto metadata = configuration->getExternalMetadata(); - if (!metadata) - throw Exception(ErrorCodes::NOT_IMPLEMENTED, "EXECUTE command '{}' is not supported by this storage", command_name); + auto p = engine_to_function.find(storage_engine_name); + if (p == engine_to_function.end()) + { + throw Exception( + ErrorCodes::LOGICAL_ERROR, + "Can't find table function for engine {}", + storage_engine_name + ); + } - return metadata->executeCommand(command_name, args, object_storage, configuration, catalog, context, getStorageID()); -} + std::string table_function_name = p->second; -void StorageObjectStorageCluster::drop() -{ - /// We cannot use query context here, because drop is executed in the background. - auto drop_context = Context::getGlobalContextInstance(); - if (catalog) + auto function_ast = make_intrusive(); + function_ast->name = table_function_name; + + function_ast->arguments = configuration->createArgsWithAccessData(); + function_ast->children.push_back(function_ast->arguments); + function_ast->setAlias(table_alias); + + ASTPtr function_ast_ptr(function_ast); + + table_expression->database_and_table_name = nullptr; + table_expression->table_function = function_ast_ptr; + table_expression->children[0] = function_ast_ptr; + + if (!make_cluster_function) + return false; + + auto cluster_name = getClusterName(context); + + if (cluster_name.empty()) { - const auto [namespace_name, table_name] = DataLake::parseTableName(getStorageID().getTableName()); - catalog->dropTable(namespace_name, table_name, drop_context->getSettingsRef()[Setting::iceberg_delete_data_on_drop]); + throw Exception( + ErrorCodes::LOGICAL_ERROR, + "Can't be here without cluster name, no cluster name in query {}", + query->formatForLogging()); } - configuration->drop(drop_context); -} -std::optional StorageObjectStorageCluster::totalRows(ContextPtr query_context) const -{ - configuration->lazyInitializeIfNeeded( - object_storage, - query_context); - return configuration->totalRows(query_context); -} + auto settings = select_query->settings(); + if (settings) + { + auto & settings_ast = settings->as(); + settings_ast.changes.insertSetting("object_storage_cluster", cluster_name); + } + else + { + auto settings_ast_ptr = make_intrusive(); + settings_ast_ptr->is_standalone = false; + settings_ast_ptr->changes.setSetting("object_storage_cluster", cluster_name); + select_query->setExpression(ASTSelectQuery::Expression::SETTINGS, std::move(settings_ast_ptr)); + } -std::optional StorageObjectStorageCluster::totalBytes(ContextPtr query_context) const -{ - configuration->lazyInitializeIfNeeded( - object_storage, - query_context); - return configuration->totalBytes(query_context); + return true; } void StorageObjectStorageCluster::updateQueryToSendIfNeeded( ASTPtr & query, const DB::StorageSnapshotPtr & storage_snapshot, - const ContextPtr & context) + const ContextPtr & context, + bool make_cluster_function) { + bool cluster_name_added_to_settings = updateQueryForDistributedEngineIfNeeded(query, context, make_cluster_function); + auto * table_function = extractTableFunctionFromSelectQuery(query); if (!table_function) return; @@ -299,6 +485,9 @@ void StorageObjectStorageCluster::updateQueryToSendIfNeeded( configuration->getEngineName()); } + ASTPtr object_storage_type_arg; + configuration->extractDynamicStorageType(args, context, &object_storage_type_arg, !cluster_name_in_settings && !cluster_name_added_to_settings); + ASTPtr settings_temporary_storage = nullptr; for (auto it = args.begin(); it != args.end(); ++it) { @@ -311,42 +500,96 @@ void StorageObjectStorageCluster::updateQueryToSendIfNeeded( } } - if (!endsWith(table_function->name, "Cluster")) + if (cluster_name_in_settings || cluster_name_added_to_settings || !endsWith(table_function->name, "Cluster")) { - configuration->addStructureAndFormatToArgsIfNeeded(args, structure, configuration->format, context, /*with_structure=*/true); + configuration->addStructureAndFormatToArgsIfNeeded(args, structure, configuration->getFormat(), context, /*with_structure=*/true); - /// When a non-cluster table function (e.g. `s3`) was auto-converted to cluster mode - /// by the `parallel_replicas_for_cluster_engines` setting, rename it to the Cluster variant - /// (e.g. `s3Cluster`) and prepend the cluster name argument. This ensures that on the shard, - /// `TableFunctionObjectStorageCluster::executeImpl` is called, which correctly handles - /// `distributed_processing` for task-based file distribution from the initiator. - /// - /// Some table functions (e.g. `paimonLocal`, `deltaLakeLocal`) do not have a Cluster variant, - /// so we only rename when the target function actually exists. - const String cluster_function_name = table_function->name + "Cluster"; - if (TableFunctionFactory::instance().isTableFunctionName(cluster_function_name)) + if (make_cluster_function) { - args.insert(args.begin(), make_intrusive(getClusterName())); - table_function->name = cluster_function_name; + /// Convert to old-stype *Cluster table function. + /// This allows to use old clickhouse versions in cluster. + static std::unordered_map function_to_cluster_function = { + {"s3", "s3Cluster"}, + {"azureBlobStorage", "azureBlobStorageCluster"}, + {"hdfs", "hdfsCluster"}, + {"iceberg", "icebergCluster"}, + {"icebergS3", "icebergS3Cluster"}, + {"icebergAzure", "icebergAzureCluster"}, + {"icebergHDFS", "icebergHDFSCluster"}, + {"icebergLocal", "icebergLocalCluster"}, + {"deltaLake", "deltaLakeCluster"}, + {"deltaLakeS3", "deltaLakeS3Cluster"}, + {"deltaLakeAzure", "deltaLakeAzureCluster"}, + {"hudi", "hudiCluster"}, + {"paimonS3", "paimonS3Cluster"}, + {"paimonAzure", "paimonAzureCluster"}, + }; + + auto p = function_to_cluster_function.find(table_function->name); + if (p == function_to_cluster_function.end()) + { + throw Exception( + ErrorCodes::LOGICAL_ERROR, + "Can't find cluster variant for table function {}", + table_function->name); + } + + table_function->name = p->second; + + auto cluster_name = getClusterName(context); + auto cluster_name_arg = make_intrusive(cluster_name); + args.insert(args.begin(), cluster_name_arg); + + auto * select_query = query->as(); + if (!select_query) + throw Exception( + ErrorCodes::LOGICAL_ERROR, + "Expected SELECT query from table function {}", + configuration->getEngineName()); + + auto settings = select_query->settings(); + if (settings) + { + auto & settings_ast = settings->as(); + if (settings_ast.changes.removeSetting("object_storage_cluster") && settings_ast.changes.empty()) + { + select_query->setExpression(ASTSelectQuery::Expression::SETTINGS, {}); + } + /// No throw if not found - `object_storage_cluster` can be global setting. + } } } else - { + { /// *Cluster function has cluster name as first argument. Temporary remove it before add structure and format ASTPtr cluster_name_arg = args.front(); args.erase(args.begin()); - configuration->addStructureAndFormatToArgsIfNeeded(args, structure, configuration->format, context, /*with_structure=*/true); + configuration->addStructureAndFormatToArgsIfNeeded(args, structure, configuration->getFormat(), context, /*with_structure=*/true); args.insert(args.begin(), cluster_name_arg); } if (settings_temporary_storage) { args.insert(args.end(), std::move(settings_temporary_storage)); } + if (object_storage_type_arg) + args.insert(args.end(), object_storage_type_arg); } void StorageObjectStorageCluster::updateExternalDynamicMetadataIfExists(ContextPtr query_context) { if (!configuration->isDataLakeConfiguration()) + { + /// Resolves a deferred hive partitioning sample path. Called before query analysis, so the + /// hive virtual columns are visible to the triggering query. + if (pure_storage) + { + pure_storage->updateExternalDynamicMetadataIfExists(query_context); + /// A clustered read builds its file iterator from this list, which stayed empty in the + /// constructor because the sample path it comes from was resolved only now. + if (hive_partitioning_sample_path_deferred) + hive_partition_columns_to_read_from_file_path = pure_storage->getHivePartitionColumns(); + } return; + } /// Always force an update to pick up the latest snapshot version. /// Using if_not_updated_before=true would leave latest_snapshot_version @@ -369,13 +612,59 @@ void StorageObjectStorageCluster::updateExternalDynamicMetadataIfExists(ContextP new_metadata = *metadata_snapshot; } - setInMemoryMetadata(new_metadata.withVirtuals(VirtualColumnUtils::getVirtualsForFileLikeStorage( + auto updated_metadata = new_metadata.withVirtuals(VirtualColumnUtils::getVirtualsForFileLikeStorage( new_metadata.columns, query_context, /* format_settings */ std::nullopt, - configuration->partition_strategy_type))); + configuration->getPartitionStrategyType())); + + setInMemoryMetadata(updated_metadata); + + if (pure_storage) + pure_storage->setInMemoryMetadata(updated_metadata); } +class TaskDistributor : public TaskIterator +{ +public: + TaskDistributor(std::shared_ptr iterator, + std::vector && ids_of_hosts, + bool send_over_whole_archive, + uint64_t lock_object_storage_task_distribution_ms, + ContextPtr context_, + bool iceberg_read_optimization_enabled) + : task_distributor( + iterator, + std::move(ids_of_hosts), + send_over_whole_archive, + lock_object_storage_task_distribution_ms, + iceberg_read_optimization_enabled) + , context(context_) {} + ~TaskDistributor() override = default; + bool supportRerunTask() const override { return true; } + void rescheduleTasksFromReplica(size_t number_of_current_replica) override + { + task_distributor.rescheduleTasksFromReplica(number_of_current_replica); + } + + ClusterFunctionReadTaskResponsePtr operator()(size_t number_of_current_replica) const override + { + fiu_do_on(FailPoints::storage_cluster_read_sleep, + { + sleepForSeconds(10); + }); + + auto task = task_distributor.getNextTask(number_of_current_replica); + if (task) + return std::make_shared(std::move(task), context); + return std::make_shared(); + } + +private: + mutable StorageObjectStorageStableTaskDistributor task_distributor; + ContextPtr context; +}; + RemoteQueryExecutor::Extension StorageObjectStorageCluster::getTaskIteratorExtension( const ActionsDAG::Node * predicate, const ActionsDAG * filter, @@ -393,7 +682,7 @@ RemoteQueryExecutor::Extension StorageObjectStorageCluster::getTaskIteratorExten predicate, filter, storage_metadata_snapshot->virtuals.getSampleBlock(VirtualsKind::All, VirtualsMaterializationPlace::Reader).getNamesAndTypesList(), - hive_partition_columns_to_read_from_file_path, + getHivePartitionColumnsWithoutVirtuals(storage_metadata_snapshot), nullptr, local_context->getFileProgressCallback(), /*ignore_archive_globs=*/false, @@ -403,7 +692,7 @@ RemoteQueryExecutor::Extension StorageObjectStorageCluster::getTaskIteratorExten { iterator = std::make_shared( std::move(iterator), - configuration->format, + configuration->getFormat(), object_storage, local_context ); @@ -422,27 +711,525 @@ RemoteQueryExecutor::Extension StorageObjectStorageCluster::getTaskIteratorExten } } - auto task_distributor = std::make_shared( - iterator, - std::move(ids_of_hosts), - /* send_over_whole_archive */!local_context->getSettingsRef()[Setting::cluster_function_process_archive_on_multiple_nodes]); + uint64_t lock_object_storage_task_distribution_ms = local_context->getSettingsRef()[Setting::lock_object_storage_task_distribution_ms]; - auto callback = std::make_shared( - [task_distributor, local_context](size_t number_of_current_replica) mutable -> ClusterFunctionReadTaskResponsePtr - { - fiu_do_on(FailPoints::storage_cluster_read_sleep, - { - sleepForSeconds(10); - }); + /// Check value to avoid negative result after conversion in microseconds. + /// Poco::Timestamp::TimeDiff is signed int 64. + static const uint64_t lock_object_storage_task_distribution_ms_max = 0x0020000000000000ULL; + if (lock_object_storage_task_distribution_ms > lock_object_storage_task_distribution_ms_max) + throw Exception(ErrorCodes::INVALID_SETTING_VALUE, + "Value lock_object_storage_task_distribution_ms is too big: {}, allowed maximum is {}", + lock_object_storage_task_distribution_ms, + lock_object_storage_task_distribution_ms_max + ); - auto task = task_distributor->getNextTask(number_of_current_replica); - if (task) - return std::make_shared(std::move(task), local_context); - return std::make_shared(); - }); + auto callback = std::make_shared(iterator, + std::move(ids_of_hosts), + /* send_over_whole_archive */!local_context->getSettingsRef()[Setting::cluster_function_process_archive_on_multiple_nodes], + lock_object_storage_task_distribution_ms, + local_context, + /* iceberg_read_optimization_enabled */local_context->getSettingsRef()[Setting::allow_experimental_iceberg_read_optimization]); return RemoteQueryExecutor::Extension{ .task_iterator = std::move(callback) }; } +void StorageObjectStorageCluster::readFallBackToPure( + QueryPlan & query_plan, + const Names & column_names, + const StorageSnapshotPtr & storage_snapshot, + SelectQueryInfo & query_info, + ContextPtr context, + QueryProcessingStage::Enum processed_stage, + size_t max_block_size, + size_t num_streams) +{ + pure_storage->read(query_plan, column_names, storage_snapshot, query_info, context, processed_stage, max_block_size, num_streams); +} + +bool StorageObjectStorageCluster::isClusterSupported() const +{ + return configuration->isClusterSupported(); } +SinkToStoragePtr StorageObjectStorageCluster::writeFallBackToPure( + const ASTPtr & query, + const StorageMetadataPtr & metadata_snapshot, + ContextPtr context, + bool async_insert) +{ + return pure_storage->write(query, metadata_snapshot, context, async_insert); +} + +String StorageObjectStorageCluster::getClusterName(ContextPtr context) const +{ + /// StorageObjectStorageCluster is always created for cluster or non-cluster variants. + /// User can specify cluster name in table definition or in setting `object_storage_cluster` + /// only for several queries. When it specified in both places, priority is given to the query setting. + /// When it is empty, non-cluster realization is used. + + if (!isClusterSupported()) + return ""; + + auto cluster_name_from_settings = context->getSettingsRef()[Setting::object_storage_cluster].value; + if (cluster_name_from_settings.empty()) + cluster_name_from_settings = getOriginalClusterName(); + return cluster_name_from_settings; +} + +bool StorageObjectStorageCluster::readsFromPureStorage(ContextPtr context) const +{ + if (!isClusterSupported()) + return true; + + return getClusterName(context).empty() // Not cluster request + && context->getSettingsRef()[Setting::object_storage_remote_initiator_cluster].value.empty(); // Not request with remote initiator +} + +QueryProcessingStage::Enum StorageObjectStorageCluster::getQueryProcessingStage( + ContextPtr context, QueryProcessingStage::Enum to_stage, const StorageSnapshotPtr & storage_snapshot, SelectQueryInfo & query_info) const +{ + /// Full query if fall back to pure storage. + if (readsFromPureStorage(context)) + { + if (isClusterSupported() && context->getSettingsRef()[Setting::object_storage_remote_initiator]) + throw Exception(ErrorCodes::BAD_ARGUMENTS, + "Setting 'object_storage_remote_initiator' can be used only with 'object_storage_remote_initiator_cluster', 'object_storage_cluster', or cluster name in arguments"); + + return QueryProcessingStage::Enum::FetchColumns; + } + + /// Distributed storage. + return IStorageCluster::getQueryProcessingStage(context, to_stage, storage_snapshot, query_info); +} + +bool StorageObjectStorageCluster::appliesRowLevelFilterInRead(ContextPtr context) const +{ + /// A clustered read only ships query text to the swarm nodes, so the filter would be lost. + /// A fallback read is served by `pure_storage`, which lowers it into the reading step. + return pure_storage && readsFromPureStorage(context); +} + +SinkToStoragePtr StorageObjectStorageCluster::write( + const ASTPtr & query, + const StorageMetadataPtr & metadata_snapshot, + ContextPtr context, + bool async_insert) +{ + return pure_storage->write(query, metadata_snapshot, context, async_insert); +} + +std::optional StorageObjectStorageCluster::distributedWrite( + const ASTInsertQuery & query, + ContextPtr context) +{ + if (getClusterName(context).empty()) + return pure_storage->distributedWrite(query, context); + return IStorageCluster::distributedWrite(query, context); +} + +void StorageObjectStorageCluster::drop() +{ + if (pure_storage) + { + pure_storage->drop(); + return; + } + IStorageCluster::drop(); +} + +void StorageObjectStorageCluster::dropInnerTableIfAny(bool sync, ContextPtr context) +{ + if (getClusterName(context).empty()) + { + pure_storage->dropInnerTableIfAny(sync, context); + return; + } + IStorageCluster::dropInnerTableIfAny(sync, context); +} + +void StorageObjectStorageCluster::truncate( + const ASTPtr & query, + const StorageMetadataPtr & metadata_snapshot, + ContextPtr local_context, + TableExclusiveLockHolder & lock_holder) +{ + /// Full query if fall back to pure storage. + if (getClusterName(local_context).empty()) + { + pure_storage->truncate(query, metadata_snapshot, local_context, lock_holder); + return; + } + + throw Exception(ErrorCodes::NOT_IMPLEMENTED, "Truncate is not supported by storage {}", getName()); +} + +void StorageObjectStorageCluster::checkTableCanBeRenamed(const StorageID & new_name) const +{ + if (pure_storage) + pure_storage->checkTableCanBeRenamed(new_name); + IStorageCluster::checkTableCanBeRenamed(new_name); +} + +void StorageObjectStorageCluster::rename(const String & new_path_to_table_data, const StorageID & new_table_id) +{ + if (pure_storage) + pure_storage->rename(new_path_to_table_data, new_table_id); + IStorageCluster::rename(new_path_to_table_data, new_table_id); +} + +void StorageObjectStorageCluster::renameInMemory(const StorageID & new_table_id) +{ + if (pure_storage) + pure_storage->renameInMemory(new_table_id); + IStorageCluster::renameInMemory(new_table_id); +} + +void StorageObjectStorageCluster::alter(const AlterCommands & params, ContextPtr context, AlterLockHolder & alter_lock_holder) +{ + if (getClusterName(context).empty()) + { + pure_storage->alter(params, context, alter_lock_holder); + auto pure_metadata = pure_storage->getInMemoryMetadataPtr(context, false); + setInMemoryMetadata(*pure_metadata); + return; + } + IStorageCluster::alter(params, context, alter_lock_holder); + auto cluster_metadata = IStorageCluster::getInMemoryMetadataPtr(context, false); + pure_storage->setInMemoryMetadata(*cluster_metadata); +} + +void StorageObjectStorageCluster::addInferredEngineArgsToCreateQuery(ASTs & args, const ContextPtr & context) const +{ + /// `pure_storage` shares this configuration and persists an implicit `partition_strategy = 'none'`, + /// which the path shape alone cannot recover on reload. + pure_storage->addInferredEngineArgsToCreateQuery(args, context); +} + +StorageMetadataHandle StorageObjectStorageCluster::getInMemoryMetadataPtr(ContextPtr context, bool bypass_metadata_cache) const +{ + if (pure_storage) + return pure_storage->getInMemoryMetadataPtr(context, bypass_metadata_cache); + return IStorageCluster::getInMemoryMetadataPtr(context, bypass_metadata_cache); +} + +std::shared_ptr StorageObjectStorageCluster::getExternalMetadata(ContextPtr query_context) +{ + if (getClusterName(query_context).empty()) + return pure_storage->getExternalMetadata(query_context); + + configuration->update( + object_storage, + query_context); + + return configuration->getExternalMetadata(); +} + +void StorageObjectStorageCluster::checkAlterIsPossible(const AlterCommands & commands, ContextPtr context) const +{ + if (getClusterName(context).empty()) + { + pure_storage->checkAlterIsPossible(commands, context); + return; + } + IStorageCluster::checkAlterIsPossible(commands, context); +} + +void StorageObjectStorageCluster::checkMutationIsPossible(const MutationCommands & commands, const Settings & settings) const +{ + if (pure_storage) + { + pure_storage->checkMutationIsPossible(commands, settings); + return; + } + IStorageCluster::checkMutationIsPossible(commands, settings); +} + +Pipe StorageObjectStorageCluster::alterPartition( + const StorageMetadataPtr & metadata_snapshot, + const PartitionCommands & commands, + ContextPtr context) +{ + if (getClusterName(context).empty()) + return pure_storage->alterPartition(metadata_snapshot, commands, context); + return IStorageCluster::alterPartition(metadata_snapshot, commands, context); +} + +void StorageObjectStorageCluster::checkAlterPartitionIsPossible( + const PartitionCommands & commands, + const StorageMetadataPtr & metadata_snapshot, + const Settings & settings, + ContextPtr context) const +{ + if (getClusterName(context).empty()) + { + pure_storage->checkAlterPartitionIsPossible(commands, metadata_snapshot, settings, context); + return; + } + IStorageCluster::checkAlterPartitionIsPossible(commands, metadata_snapshot, settings, context); +} + +bool StorageObjectStorageCluster::optimize( + const ASTPtr & query, + const StorageMetadataPtr & metadata_snapshot, + const ASTPtr & partition, + bool final, + bool deduplicate, + const Names & deduplicate_by_columns, + bool cleanup, + ContextPtr context) +{ + if (getClusterName(context).empty()) + return pure_storage->optimize(query, metadata_snapshot, partition, final, deduplicate, deduplicate_by_columns, cleanup, context); + return IStorageCluster::optimize(query, metadata_snapshot, partition, final, deduplicate, deduplicate_by_columns, cleanup, context); +} + +QueryPipeline StorageObjectStorageCluster::updateLightweight(const MutationCommands & commands, ContextPtr context) +{ + if (getClusterName(context).empty()) + return pure_storage->updateLightweight(commands, context); + return IStorageCluster::updateLightweight(commands, context); +} + +void StorageObjectStorageCluster::mutate(const MutationCommands & commands, ContextPtr context) +{ + if (getClusterName(context).empty()) + { + pure_storage->mutate(commands, context); + return; + } + IStorageCluster::mutate(commands, context); +} + +CancellationCode StorageObjectStorageCluster::killMutation(const String & mutation_id) +{ + if (pure_storage) + return pure_storage->killMutation(mutation_id); + return IStorageCluster::killMutation(mutation_id); +} + +void StorageObjectStorageCluster::waitForMutation(const String & mutation_id, bool wait_for_another_mutation) +{ + if (pure_storage) + { + pure_storage->waitForMutation(mutation_id, wait_for_another_mutation); + return; + } + IStorageCluster::waitForMutation(mutation_id, wait_for_another_mutation); +} + +void StorageObjectStorageCluster::setMutationCSN(const String & mutation_id, UInt64 csn) +{ + if (pure_storage) + { + pure_storage->setMutationCSN(mutation_id, csn); + return; + } + IStorageCluster::setMutationCSN(mutation_id, csn); +} + +CancellationCode StorageObjectStorageCluster::killPartMoveToShard(const UUID & task_uuid) +{ + if (pure_storage) + return pure_storage->killPartMoveToShard(task_uuid); + return IStorageCluster::killPartMoveToShard(task_uuid); +} + +void StorageObjectStorageCluster::startup() +{ + if (pure_storage) + { + pure_storage->startup(); + return; + } + IStorageCluster::startup(); +} + +void StorageObjectStorageCluster::shutdown(bool is_drop) +{ + if (pure_storage) + { + pure_storage->shutdown(is_drop); + return; + } + IStorageCluster::shutdown(is_drop); +} + +void StorageObjectStorageCluster::flushAndPrepareForShutdown() +{ + if (pure_storage) + { + pure_storage->flushAndPrepareForShutdown(); + return; + } + IStorageCluster::flushAndPrepareForShutdown(); +} + +ActionLock StorageObjectStorageCluster::getActionLock(StorageActionBlockType action_type) +{ + if (pure_storage) + return pure_storage->getActionLock(action_type); + return IStorageCluster::getActionLock(action_type); +} + +void StorageObjectStorageCluster::onActionLockRemove(StorageActionBlockType action_type) +{ + if (pure_storage) + { + pure_storage->onActionLockRemove(action_type); + return; + } + IStorageCluster::onActionLockRemove(action_type); +} + +bool StorageObjectStorageCluster::supportsDelete() const +{ + if (pure_storage) + return pure_storage->supportsDelete(); + return IStorageCluster::supportsDelete(); +} + +bool StorageObjectStorageCluster::supportsParallelInsert() const +{ + if (pure_storage) + return pure_storage->supportsParallelInsert(); + return IStorageCluster::supportsParallelInsert(); +} + +bool StorageObjectStorageCluster::prefersLargeBlocks() const +{ + if (pure_storage) + return pure_storage->prefersLargeBlocks(); + return IStorageCluster::prefersLargeBlocks(); +} + +bool StorageObjectStorageCluster::supportsPartitionBy() const +{ + if (pure_storage) + return pure_storage->supportsPartitionBy(); + return IStorageCluster::supportsPartitionBy(); +} + +bool StorageObjectStorageCluster::supportsSubcolumns() const +{ + if (pure_storage) + return pure_storage->supportsSubcolumns(); + return IStorageCluster::supportsSubcolumns(); +} + +bool StorageObjectStorageCluster::supportsTrivialCountOptimization(const StorageSnapshotPtr & snapshot, ContextPtr context) const +{ + if (pure_storage) + return pure_storage->supportsTrivialCountOptimization(snapshot, context); + return IStorageCluster::supportsTrivialCountOptimization(snapshot, context); +} + +bool StorageObjectStorageCluster::supportsPrewhere() const +{ + if (pure_storage) + return pure_storage->supportsPrewhere(); + return IStorageCluster::supportsPrewhere(); +} + +bool StorageObjectStorageCluster::canMoveConditionsToPrewhere() const +{ + if (pure_storage) + return pure_storage->canMoveConditionsToPrewhere(); + return IStorageCluster::canMoveConditionsToPrewhere(); +} + +std::optional StorageObjectStorageCluster::supportedPrewhereColumns() const +{ + if (pure_storage) + return pure_storage->supportedPrewhereColumns(); + return IStorageCluster::supportedPrewhereColumns(); +} + +IStorageCluster::ColumnSizeByName StorageObjectStorageCluster::getColumnSizes() const +{ + if (pure_storage) + return pure_storage->getColumnSizes(); + return IStorageCluster::getColumnSizes(); +} + +bool StorageObjectStorageCluster::parallelizeOutputAfterReading(ContextPtr context) const +{ + if (pure_storage) + return pure_storage->parallelizeOutputAfterReading(context); + return IStorageCluster::parallelizeOutputAfterReading(context); +} + +Pipe StorageObjectStorageCluster::executeCommand(const String & command_name, const ASTPtr & args, ContextPtr context) +{ + if (pure_storage) + return pure_storage->executeCommand(command_name, args, context); + return IStorageCluster::executeCommand(command_name, args, context); +} + +bool StorageObjectStorageCluster::supportsImport(ContextPtr context) const +{ + if (pure_storage) + return pure_storage->supportsImport(context); + return IStorageCluster::supportsImport(context); +} + +SinkToStoragePtr StorageObjectStorageCluster::import( + const std::string & file_name, + Block & block_with_partition_values, + const std::function & new_file_path_callback, + MergeTreePartExportFileAlreadyExistsPolicy file_already_exists_policy, + std::size_t max_bytes_per_file, + std::size_t max_rows_per_file, + const std::optional & iceberg_metadata_json_string, + const std::optional & format_settings_, + ContextPtr context) +{ + if (pure_storage) + return pure_storage->import( + file_name, + block_with_partition_values, + new_file_path_callback, + file_already_exists_policy, + max_bytes_per_file, + max_rows_per_file, + iceberg_metadata_json_string, + format_settings_, + context); + return IStorageCluster::import( + file_name, + block_with_partition_values, + new_file_path_callback, + file_already_exists_policy, + max_bytes_per_file, + max_rows_per_file, + iceberg_metadata_json_string, + format_settings_, + context); +} + +IStorage::ExportPartitionCommitInfo StorageObjectStorageCluster::commitExportPartitionTransaction( + const String & transaction_id, + const String & partition_id, + const Strings & exported_paths, + const IcebergCommitExportPartitionArguments & iceberg_commit_export_partition_arguments, + ContextPtr local_context) +{ + if (pure_storage) + { + return pure_storage->commitExportPartitionTransaction( + transaction_id, + partition_id, + exported_paths, + iceberg_commit_export_partition_arguments, + local_context + ); + } + return IStorageCluster::commitExportPartitionTransaction( + transaction_id, + partition_id, + exported_paths, + iceberg_commit_export_partition_arguments, + local_context + ); +} + +} diff --git a/src/Storages/ObjectStorage/StorageObjectStorageCluster.h b/src/Storages/ObjectStorage/StorageObjectStorageCluster.h index 9815b14f6a17..ac848e9a77db 100644 --- a/src/Storages/ObjectStorage/StorageObjectStorageCluster.h +++ b/src/Storages/ObjectStorage/StorageObjectStorageCluster.h @@ -18,26 +18,108 @@ class StorageObjectStorageCluster : public IStorageCluster const ColumnsDescription & columns_in_table_or_function_definition, const ConstraintsDescription & constraints_, const ASTPtr & partition_by, + const ASTPtr & order_by, ContextPtr context_, + const String & comment_, + std::optional format_settings_, + LoadingStrictnessLevel mode_, + std::shared_ptr catalog, + bool if_not_exists, + bool is_datalake_query, bool is_table_function_ = false, - std::optional format_settings_ = std::nullopt, - std::shared_ptr catalog_ = nullptr); + bool lazy_init = false); std::string getName() const override; + bool supportsImport(ContextPtr context) const override; + + SinkToStoragePtr import( + const std::string & file_name, + Block & block_with_partition_values, + const std::function & new_file_path_callback, + MergeTreePartExportFileAlreadyExistsPolicy file_already_exists_policy, + std::size_t max_bytes_per_file, + std::size_t max_rows_per_file, + const std::optional & iceberg_metadata_json_string, + const std::optional & format_settings_, + ContextPtr context) override; + + ExportPartitionCommitInfo commitExportPartitionTransaction( + const String & transaction_id, + const String & partition_id, + const Strings & exported_paths, + const IcebergCommitExportPartitionArguments & iceberg_commit_export_partition_arguments, + ContextPtr local_context) override; + + RemoteQueryExecutor::Extension getTaskIteratorExtension( + const ActionsDAG::Node * predicate, + const ActionsDAG * filter, + const ContextPtr & context, + ClusterPtr cluster, + StorageMetadataPtr storage_metadata_snapshot) const override; + + String getPathSample(ContextPtr context); + + std::optional totalRows(ContextPtr query_context) const override; + std::optional totalBytes(ContextPtr query_context) const override; + void setClusterNameInSettings(bool cluster_name_in_settings_) { cluster_name_in_settings = cluster_name_in_settings_; } + + String getClusterName(ContextPtr context) const override; + + QueryProcessingStage::Enum getQueryProcessingStage(ContextPtr, QueryProcessingStage::Enum, const StorageSnapshotPtr &, SelectQueryInfo &) const override; + + /// Direct inserts (e.g. `INSERT ... VALUES`) are written from the initiator even when a cluster is set; + /// only `INSERT ... SELECT` is distributed, see `distributedWrite`. SinkToStoragePtr write( const ASTPtr & query, const StorageMetadataPtr & metadata_snapshot, ContextPtr context, bool async_insert) override; - bool isDataLake() const override { return configuration->isDataLakeConfiguration(); } + std::optional distributedWrite( + const ASTInsertQuery & query, + ContextPtr context) override; - bool isObjectStorage() const override { return true; } + void drop() override; - bool supportsParallelInsert() const override; + void dropInnerTableIfAny(bool sync, ContextPtr context) override; - bool supportsDelete() const override; + void truncate( + const ASTPtr & query, + const StorageMetadataPtr & metadata_snapshot, + ContextPtr local_context, + TableExclusiveLockHolder &) override; + + void checkTableCanBeRenamed(const StorageID & new_name) const override; + + void rename(const String & new_path_to_table_data, const StorageID & new_table_id) override; + + void renameInMemory(const StorageID & new_table_id) override; + + void alter(const AlterCommands & params, ContextPtr context, AlterLockHolder & alter_lock_holder) override; + + void addInferredEngineArgsToCreateQuery(ASTs & args, const ContextPtr & context) const override; + + std::shared_ptr getExternalMetadata(ContextPtr query_context); + + std::shared_ptr getCatalog() const { return pure_storage ? pure_storage->getCatalog() : nullptr; } + + StorageMetadataHandle getInMemoryMetadataPtr(ContextPtr context, bool bypass_metadata_cache) const override; + + void checkAlterIsPossible(const AlterCommands & commands, ContextPtr context) const override; + + void checkMutationIsPossible(const MutationCommands & commands, const Settings & settings) const override; + + Pipe alterPartition( + const StorageMetadataPtr & metadata_snapshot, + const PartitionCommands & commands, + ContextPtr context) override; + + void checkAlterPartitionIsPossible( + const PartitionCommands & commands, + const StorageMetadataPtr & metadata_snapshot, + const Settings & settings, + ContextPtr context) const override; bool optimize( const ASTPtr & query, @@ -49,42 +131,113 @@ class StorageObjectStorageCluster : public IStorageCluster bool cleanup, ContextPtr context) override; - void mutate(const MutationCommands & commands, ContextPtr context) override; - void checkMutationIsPossible(const MutationCommands & commands, const Settings & settings) const override; + QueryPipeline updateLightweight(const MutationCommands & commands, ContextPtr context) override; - void alter(const AlterCommands & params, ContextPtr context, AlterLockHolder & alter_lock_holder) override; - void checkAlterIsPossible(const AlterCommands & commands, ContextPtr context) const override; + void mutate(const MutationCommands & commands, ContextPtr context) override; Pipe executeCommand(const String & command_name, const ASTPtr & args, ContextPtr context) override; - void drop() override; + CancellationCode killMutation(const String & mutation_id) override; - RemoteQueryExecutor::Extension getTaskIteratorExtension( - const ActionsDAG::Node * predicate, - const ActionsDAG * filter, - const ContextPtr & context, - ClusterPtr cluster, - StorageMetadataPtr storage_metadata_snapshot) const override; + void waitForMutation(const String & mutation_id, bool wait_for_another_mutation) override; - String getPathSample(ContextPtr context); + void setMutationCSN(const String & mutation_id, UInt64 csn) override; - std::optional totalRows(ContextPtr query_context) const override; - std::optional totalBytes(ContextPtr query_context) const override; + CancellationCode killPartMoveToShard(const UUID & task_uuid) override; + + void startup() override; + + void shutdown(bool is_drop = false) override; + + void flushAndPrepareForShutdown() override; + + ActionLock getActionLock(StorageActionBlockType action_type) override; + + void onActionLockRemove(StorageActionBlockType action_type) override; void updateExternalDynamicMetadataIfExists(ContextPtr query_context) override; + bool supportsDelete() const override; + + bool supportsParallelInsert() const override; + + bool prefersLargeBlocks() const override; + + bool supportsPartitionBy() const override; + + bool supportsSubcolumns() const override; + + bool supportsTrivialCountOptimization(const StorageSnapshotPtr &, ContextPtr) const override; + + /// Things required for PREWHERE. + bool supportsPrewhere() const override; + bool canMoveConditionsToPrewhere() const override; + std::optional supportedPrewhereColumns() const override; + ColumnSizeByName getColumnSizes() const override; + + bool appliesRowLevelFilterInRead(ContextPtr context) const override; + + bool parallelizeOutputAfterReading(ContextPtr context) const override; + + bool isObjectStorage() const override { return true; } + + bool isDataLake() const override { return configuration->isDataLakeConfiguration(); } + + bool isIcebergStorage() const { return configuration->isIcebergConfiguration(); } + private: void updateQueryToSendIfNeeded( ASTPtr & query, const StorageSnapshotPtr & storage_snapshot, - const ContextPtr & context) override; + const ContextPtr & context, + bool make_cluster_function) override; + + bool isClusterSupported() const override; + + /// Whether this query is served by `pure_storage` instead of being distributed over a cluster. + bool readsFromPureStorage(ContextPtr context) const; + + void readFallBackToPure( + QueryPlan & query_plan, + const Names & column_names, + const StorageSnapshotPtr & storage_snapshot, + SelectQueryInfo & query_info, + ContextPtr context, + QueryProcessingStage::Enum processed_stage, + size_t max_block_size, + size_t num_streams) override; + + SinkToStoragePtr writeFallBackToPure( + const ASTPtr & query, + const StorageMetadataPtr & metadata_snapshot, + ContextPtr context, + bool async_insert) override; + + /* + In case the table was created with `object_storage_cluster` setting, + modify the AST query object so that it uses the table function implementation + by mapping the engine name to table function name and setting `object_storage_cluster`. + For table like + CREATE TABLE table ENGINE=S3(...) SETTINGS object_storage_cluster='cluster' + coverts request + SELECT * FROM table + to + SELECT * FROM s3(...) SETTINGS object_storage_cluster='cluster' + to make distributed request over cluster 'cluster'. + Returns true if cluster name was added to settings. + */ + bool updateQueryForDistributedEngineIfNeeded(ASTPtr & query, ContextPtr context, bool make_cluster_function); const String engine_name; - const StorageObjectStorageConfigurationPtr configuration; + StorageObjectStorageConfigurationPtr configuration; const ObjectStoragePtr object_storage; - const std::optional format_settings; - const std::shared_ptr catalog; - NamesAndTypesList hive_partition_columns_to_read_from_file_path; + bool cluster_name_in_settings; + + /// non-clustered storage to fall back on pure realisation if needed + std::shared_ptr pure_storage; + + /// Set only in the constructor when hive partitioning detection is deferred to the first use. + bool hive_partitioning_sample_path_deferred = false; }; } diff --git a/src/Storages/ObjectStorage/StorageObjectStorageConfiguration.cpp b/src/Storages/ObjectStorage/StorageObjectStorageConfiguration.cpp index fafc68825667..be6ca8128d5a 100644 --- a/src/Storages/ObjectStorage/StorageObjectStorageConfiguration.cpp +++ b/src/Storages/ObjectStorage/StorageObjectStorageConfiguration.cpp @@ -96,25 +96,24 @@ bool StorageObjectStorageConfiguration::shouldReloadSchemaForConsistency(Context void StorageObjectStorageConfiguration::initialize( - StorageObjectStorageConfiguration & configuration_to_initialize, ASTs & engine_args, ContextPtr local_context, bool with_table_structure, const StorageID * table_id) { std::string disk_name; - if (configuration_to_initialize.isDataLakeConfiguration()) + if (isDataLakeConfiguration()) { - const auto & storage_settings = configuration_to_initialize.getDataLakeSettings(); + const auto & storage_settings = getDataLakeSettings(); disk_name = storage_settings[DataLakeStorageSetting::disk].changed ? storage_settings[DataLakeStorageSetting::disk].value : ""; } if (!disk_name.empty()) - configuration_to_initialize.fromDisk(disk_name, engine_args, local_context, with_table_structure); + fromDisk(disk_name, engine_args, local_context, with_table_structure); else if (auto named_collection = tryGetNamedCollectionWithOverrides(engine_args, local_context, true, nullptr, table_id)) { - configuration_to_initialize.fromNamedCollection(*named_collection, local_context); + fromNamedCollection(*named_collection, local_context); /// A base-URL setting (e.g. `s3_base`) rewrote a relative URL coming from the named /// collection. Materialize the resolved URL back into the engine args as a `url='...'` @@ -122,56 +121,66 @@ void StorageObjectStorageConfiguration::initialize( /// restart) does not depend on the value of the setting at attach time. /// `skip_userinfo=true` keeps credentials that may originate from the base setting /// out of the persisted arguments. - if (!configuration_to_initialize.url_overridden_by_base_setting.empty()) + if (!url_overridden_by_base_setting.empty()) StorageURL::overrideURLInEngineArgs( - engine_args, configuration_to_initialize.url_overridden_by_base_setting, local_context, /*skip_userinfo=*/ true); + engine_args, url_overridden_by_base_setting, local_context, /*skip_userinfo=*/ true); } else - configuration_to_initialize.fromAST(engine_args, local_context, with_table_structure); + fromAST(engine_args, local_context, with_table_structure); - if (configuration_to_initialize.isNamespaceWithGlobs()) + if (isNamespaceWithGlobs()) throw Exception(ErrorCodes::BAD_ARGUMENTS, - "Expression can not have wildcards inside {} name", configuration_to_initialize.getNamespaceType()); + "Expression can not have wildcards inside {} name", getNamespaceType()); - if (configuration_to_initialize.isDataLakeConfiguration()) + if (isDataLakeConfiguration()) { - if (configuration_to_initialize.partition_strategy_type != PartitionStrategyFactory::StrategyType::NONE) + if (getPartitionStrategyType() != PartitionStrategyFactory::StrategyType::NONE) { throw Exception(ErrorCodes::BAD_ARGUMENTS, "The `partition_strategy` argument is incompatible with data lakes"); } } - else if (!configuration_to_initialize.partition_strategy_was_set - && configuration_to_initialize.partition_strategy_type == PartitionStrategyFactory::StrategyType::NONE - && configuration_to_initialize.getRawPath().hasPartitionWildcard() + else if (!partition_strategy_was_set + && partition_strategy_type == PartitionStrategyFactory::StrategyType::NONE + && getRawPath().hasPartitionWildcard() && local_context->getSettingsRef()[Setting::file_like_engine_default_partition_strategy].value == FileLikeEngineDefaultPartitionStrategy::WILDCARD) { /// Backwards compatibility: promote to WILDCARD only when it is the effective default strategy. - configuration_to_initialize.partition_strategy_type = PartitionStrategyFactory::StrategyType::WILDCARD; + setPartitionStrategyType(PartitionStrategyFactory::StrategyType::WILDCARD); } - if (configuration_to_initialize.format == "auto") + if (format == "auto") { - if (configuration_to_initialize.isDataLakeConfiguration()) + if (isDataLakeConfiguration()) { - configuration_to_initialize.format = "Parquet"; + format = "Parquet"; } else { - configuration_to_initialize.format + format = FormatFactory::instance() - .tryGetFormatFromFileName(configuration_to_initialize.isArchive() ? configuration_to_initialize.getPathInArchive() : configuration_to_initialize.getRawPath().path) + .tryGetFormatFromFileName(isArchive() ? getPathInArchive() : getRawPath().path) .value_or("auto"); } } else - FormatFactory::instance().checkFormatName(configuration_to_initialize.format); + FormatFactory::instance().checkFormatName(format); + + if (partition_strategy_type == PartitionStrategyFactory::StrategyType::HIVE) + { + file_path_generator = std::make_shared( + getRawPath().path, + format); + } + else + { + file_path_generator = std::make_shared(getRawPath().path); + } - /// It might be changed on `StorageObjectStorageConfiguration::initPartitionStrategy` /// We shouldn't set path for disk setup because path prefix is already set in used object_storage. if (disk_name.empty()) - configuration_to_initialize.read_path = configuration_to_initialize.getRawPath(); + read_path = file_path_generator->getPathForRead(); - configuration_to_initialize.initialized = true; + initialized = true; } String StorageObjectStorageConfiguration::computeSchemaHash(const ColumnsDescription & columns) @@ -193,6 +202,12 @@ void StorageObjectStorageConfiguration::setSchemaHash(const String & hash) boost::replace_all(path.path, SCHEMA_HASH_WILDCARD, schema_hash); setRawPath(path); setPaths({path}); + + /// `file_path_generator` was constructed before `setSchemaHash` ran and still + /// holds a copy of the raw path with the unreplaced `{_schema_hash}` placeholder. + /// `_schema_hash` is rejected for hive partitioning earlier, so the wildcard + /// generator is the only valid variant here. + file_path_generator = std::make_shared(path.path); } void StorageObjectStorageConfiguration::initPartitionStrategy(ASTPtr partition_by, const ColumnsDescription & columns, ContextPtr context) @@ -281,9 +296,26 @@ void StorageObjectStorageConfiguration::initPartitionStrategy(ASTPtr partition_b if (partition_strategy) { - read_path = partition_strategy->getPathForRead(getRawPath().path); LOG_DEBUG(getLogger("StorageObjectStorageConfiguration"), "Initialized partition strategy {}", magic_enum::enum_name(partition_strategy_type)); } + + /// `initialize()` picks the file path generator from the `partition_strategy_type` known at + /// parse time, which is before the strategy can be inferred here (a `PARTITION BY` without an + /// explicit `partition_strategy` resolves to `hive` by default). Rebuild the generator once the + /// effective strategy is known, otherwise every hive partition would be written to the raw path + /// and reads would not look into the partition directories. + if (partition_strategy_type == PartitionStrategyFactory::StrategyType::HIVE + && !std::dynamic_pointer_cast(file_path_generator)) + { + /// Keep a read path that does not come from the generator (e.g. set up from a disk) as is. + const bool read_path_derived_from_generator + = file_path_generator && read_path.path == file_path_generator->getPathForRead(); + + file_path_generator = std::make_shared(getRawPath().path, format); + + if (read_path_derived_from_generator) + read_path = Path{file_path_generator->getPathForRead()}; + } } const StorageObjectStorageConfiguration::Path & StorageObjectStorageConfiguration::getPathForRead() const @@ -293,17 +325,12 @@ const StorageObjectStorageConfiguration::Path & StorageObjectStorageConfiguratio StorageObjectStorageConfiguration::Path StorageObjectStorageConfiguration::getPathForWrite(const std::string & partition_id) const { - auto raw_path = getRawPath(); - - if (!schema_hash.empty()) - boost::replace_all(raw_path.path, SCHEMA_HASH_WILDCARD, schema_hash); - - if (!partition_strategy) - { - return raw_path; - } + return getPathForWrite(partition_id, /* filename_override */ ""); +} - return Path {partition_strategy->getPathForWrite(raw_path.path, partition_id)}; +StorageObjectStorageConfiguration::Path StorageObjectStorageConfiguration::getPathForWrite(const std::string & partition_id, const std::string & filename_override) const +{ + return Path {file_path_generator->getPathForWrite(partition_id, filename_override)}; } bool StorageObjectStorageConfiguration::Path::hasPartitionWildcard() const @@ -312,6 +339,11 @@ bool StorageObjectStorageConfiguration::Path::hasPartitionWildcard() const return path.contains(PARTITION_ID_WILDCARD); } +bool StorageObjectStorageConfiguration::Path::hasExportFilenameWildcard() const +{ + return path.find(ObjectStorageWildcardFilePathGenerator::FILE_WILDCARD) != String::npos; +} + bool StorageObjectStorageConfiguration::Path::hasSchemaHashWildcard() const { return path.contains(StorageObjectStorageConfiguration::SCHEMA_HASH_WILDCARD); diff --git a/src/Storages/ObjectStorage/StorageObjectStorageConfiguration.h b/src/Storages/ObjectStorage/StorageObjectStorageConfiguration.h index f903b18e2d47..936a42c37dc1 100644 --- a/src/Storages/ObjectStorage/StorageObjectStorageConfiguration.h +++ b/src/Storages/ObjectStorage/StorageObjectStorageConfiguration.h @@ -18,6 +18,7 @@ #include #include #include +#include namespace DB { @@ -78,6 +79,7 @@ class StorageObjectStorageConfiguration bool hasPartitionWildcard() const; bool hasSchemaHashWildcard() const; bool hasGlobsIgnorePlaceholders() const; + bool hasExportFilenameWildcard() const; bool hasGlobs() const; std::string cutGlobs(bool supports_partial_prefix) const; }; @@ -85,8 +87,7 @@ class StorageObjectStorageConfiguration using Paths = std::vector; /// Initialize configuration from either AST or NamedCollection. - static void initialize( - StorageObjectStorageConfiguration & configuration_to_initialize, + virtual void initialize( ASTs & engine_args, ContextPtr local_context, bool with_table_structure, @@ -109,11 +110,13 @@ class StorageObjectStorageConfiguration /// Raw URI, specified by a user. Used in permission check. virtual const String & getRawURI() const = 0; - const Path & getPathForRead() const; + virtual const Path & getPathForRead() const; + // Path used for writing, it should not be globbed and might contain a partition key - Path getPathForWrite(const std::string & partition_id = "") const; + virtual Path getPathForWrite(const std::string & partition_id = "") const; + virtual Path getPathForWrite(const std::string & partition_id, const std::string & filename_override) const; - void setPathForRead(const Path & path) + virtual void setPathForRead(const Path & path) { read_path = path; } @@ -135,10 +138,10 @@ class StorageObjectStorageConfiguration virtual void addStructureAndFormatToArgsIfNeeded( ASTs & args, const String & structure_, const String & format_, ContextPtr context, bool with_structure) = 0; - bool isNamespaceWithGlobs() const; + virtual bool isNamespaceWithGlobs() const; virtual bool isArchive() const { return false; } - bool isPathInArchiveWithGlobs() const; + virtual bool isPathInArchiveWithGlobs() const; virtual std::string getPathInArchive() const; virtual void check(ContextPtr context); @@ -184,9 +187,9 @@ class StorageObjectStorageConfiguration const PrepareReadingFromFormatHiveParams & hive_parameters); static String computeSchemaHash(const ColumnsDescription & columns); - void setSchemaHash(const String & hash); + virtual void setSchemaHash(const String & hash); - void initPartitionStrategy(ASTPtr partition_by, const ColumnsDescription & columns, ContextPtr context); + virtual void initPartitionStrategy(ASTPtr partition_by, const ColumnsDescription & columns, ContextPtr context); virtual std::optional getTableStateSnapshot(ContextPtr local_context) const; virtual std::unique_ptr buildStorageMetadataFromState(const DataLakeTableStateSnapshot & state, ContextPtr local_context) const; @@ -271,6 +274,49 @@ class StorageObjectStorageConfiguration throw Exception(ErrorCodes::NOT_IMPLEMENTED, "Method getDataLakeSettings() is not implemented for configuration type {}", getTypeName()); } + /// Create arguments for table function with path and access parameters + virtual ASTPtr createArgsWithAccessData() const + { + throw Exception(ErrorCodes::NOT_IMPLEMENTED, "Method createArgsWithAccessData is not supported by storage {}", getEngineName()); + } + + virtual void fromNamedCollection(const NamedCollection & collection, ContextPtr context) = 0; + virtual void fromAST(ASTs & args, ContextPtr context, bool with_structure) = 0; + virtual void fromDisk(const String & /*disk_name*/, ASTs & /*args*/, ContextPtr /*context*/, bool /*with_structure*/) + { + throw Exception(ErrorCodes::NOT_IMPLEMENTED, "method fromDisk is not implemented"); + } + + virtual ObjectStorageType extractDynamicStorageType(ASTs & /* args */, ContextPtr /* context */, ASTPtr * /* type_arg */, bool /* cluster_name_first */) const + { return ObjectStorageType::None; } + + virtual const String & getFormat() const { return format; } + virtual const String & getCompressionMethod() const { return compression_method; } + virtual const String & getStructure() const { return structure; } + + virtual PartitionStrategyFactory::StrategyType getPartitionStrategyType() const { return partition_strategy_type; } + virtual bool getPartitionColumnsInDataFile() const { return partition_columns_in_data_file; } + virtual std::shared_ptr getPartitionStrategy() const { return partition_strategy; } + + virtual void setFormat(const String & format_) { format = format_; } + virtual void setCompressionMethod(const String & compression_method_) { compression_method = compression_method_; } + virtual void setStructure(const String & structure_) { structure = structure_; } + + virtual void setPartitionStrategyType(PartitionStrategyFactory::StrategyType partition_strategy_type_) + { + partition_strategy_type = partition_strategy_type_; + } + virtual void setPartitionColumnsInDataFile(bool partition_columns_in_data_file_) + { + partition_columns_in_data_file = partition_columns_in_data_file_; + } + virtual void setPartitionStrategy(const std::shared_ptr & partition_strategy_) + { + partition_strategy = partition_strategy_; + } + + virtual void assertInitialized() const; + virtual ColumnMapperPtr getColumnMapperForObject(ObjectInfoPtr /**/) const { return nullptr; } virtual ColumnMapperPtr getColumnMapperForCurrentSchema(StorageMetadataPtr /**/, ContextPtr /**/) const { return nullptr; } @@ -330,6 +376,8 @@ class StorageObjectStorageConfiguration return 0; } + virtual bool isClusterSupported() const { return true; } + String format = "auto"; String compression_method = "auto"; String structure = "auto"; @@ -376,14 +424,6 @@ class StorageObjectStorageConfiguration void checkFormat() const; void initializeFromParsedArguments(const StorageParsedArguments & parsed_arguments); - virtual void fromNamedCollection(const NamedCollection & collection, ContextPtr context) = 0; - virtual void fromAST(ASTs & args, ContextPtr context, bool with_structure) = 0; - virtual void fromDisk(const String & /*disk_name*/, ASTs & /*args*/, ContextPtr /*context*/, bool /*with_structure*/) - { - throw Exception(ErrorCodes::NOT_IMPLEMENTED, "method fromDisk is not implemented"); - } - - void assertInitialized() const; bool initialized = false; String schema_hash; @@ -392,6 +432,8 @@ class StorageObjectStorageConfiguration // Path used for reading, by default it is the same as `getRawPath` // When using `partition_strategy=hive`, a recursive reading pattern will be appended `'table_root/**.parquet' Path read_path; + + std::shared_ptr file_path_generator; }; using StorageObjectStorageConfigurationPtr = std::shared_ptr; diff --git a/src/Storages/ObjectStorage/StorageObjectStorageSettings.h b/src/Storages/ObjectStorage/StorageObjectStorageSettings.h index 82b443b38fd6..543ccc2eb615 100644 --- a/src/Storages/ObjectStorage/StorageObjectStorageSettings.h +++ b/src/Storages/ObjectStorage/StorageObjectStorageSettings.h @@ -70,7 +70,17 @@ struct StorageObjectStorageSettings using StorageObjectStorageSettingsPtr = std::shared_ptr; +// clang-format off + +#define STORAGE_OBJECT_STORAGE_RELATED_SETTINGS(DECLARE, ALIAS) \ + DECLARE(String, object_storage_cluster, "", R"( +Cluster for distributed requests +)", 0) \ + +// clang-format on + #define LIST_OF_STORAGE_OBJECT_STORAGE_SETTINGS(M, ALIAS) \ + STORAGE_OBJECT_STORAGE_RELATED_SETTINGS(M, ALIAS) \ LIST_OF_ALL_FORMAT_SETTINGS(M, ALIAS) } diff --git a/src/Storages/ObjectStorage/StorageObjectStorageSink.cpp b/src/Storages/ObjectStorage/StorageObjectStorageSink.cpp index 68e7e7c64743..e710c3f7d277 100644 --- a/src/Storages/ObjectStorage/StorageObjectStorageSink.cpp +++ b/src/Storages/ObjectStorage/StorageObjectStorageSink.cpp @@ -141,14 +141,20 @@ size_t StorageObjectStorageSink::getFileSize() const return *result_file_size; } +size_t StorageObjectStorageSink::getWrittenBytes() const +{ + if (!write_buf) + throw Exception(ErrorCodes::LOGICAL_ERROR, "Buffer must be initialized before requesting written bytes"); + return write_buf->count(); +} + PartitionedStorageObjectStorageSink::PartitionedStorageObjectStorageSink( ObjectStoragePtr object_storage_, StorageObjectStorageConfigurationPtr configuration_, std::optional format_settings_, SharedHeader sample_block_, ContextPtr context_) - : PartitionedSink(configuration_->partition_strategy, context_, sample_block_) - , object_storage(object_storage_) + : object_storage(object_storage_) , configuration(configuration_) , query_settings(configuration_->getQuerySettings(context_)) , format_settings(format_settings_) @@ -182,10 +188,11 @@ SinkPtr PartitionedStorageObjectStorageSink::createSinkForPartition(const String file_path, object_storage, format_settings, - std::make_shared(partition_strategy->getFormatHeader()), + std::make_shared(configuration->getPartitionStrategy()->getFormatHeader()), context, - configuration->format, - configuration->compression_method); + configuration->getFormat(), + configuration->getCompressionMethod() + ); } } diff --git a/src/Storages/ObjectStorage/StorageObjectStorageSink.h b/src/Storages/ObjectStorage/StorageObjectStorageSink.h index a199d8b24ef4..6a31520d5d2d 100644 --- a/src/Storages/ObjectStorage/StorageObjectStorageSink.h +++ b/src/Storages/ObjectStorage/StorageObjectStorageSink.h @@ -11,6 +11,8 @@ using OutputFormatPtr = std::shared_ptr; class StorageObjectStorageSink final : public SinkToStorage { +friend class StorageObjectStorageImporterSink; + public: StorageObjectStorageSink( const std::string & path_, @@ -31,6 +33,8 @@ class StorageObjectStorageSink final : public SinkToStorage const String & getPath() const { return path; } + size_t getWrittenBytes() const; + size_t getFileSize() const; private: @@ -45,7 +49,7 @@ class StorageObjectStorageSink final : public SinkToStorage void cancelBuffers(); }; -class PartitionedStorageObjectStorageSink final : public PartitionedSink +class PartitionedStorageObjectStorageSink final : public PartitionedSink::SinkCreator { public: PartitionedStorageObjectStorageSink( diff --git a/src/Storages/ObjectStorage/StorageObjectStorageSource.cpp b/src/Storages/ObjectStorage/StorageObjectStorageSource.cpp index 392447b4f82b..1bed63dba006 100644 --- a/src/Storages/ObjectStorage/StorageObjectStorageSource.cpp +++ b/src/Storages/ObjectStorage/StorageObjectStorageSource.cpp @@ -48,6 +48,7 @@ #include #include #include +#include #include #include #include @@ -84,6 +85,8 @@ namespace ProfileEvents extern const Event ObjectStorageGlobFilteredObjects; extern const Event ObjectStoragePredicateFilteredObjects; extern const Event ObjectStorageReadObjects; + extern const Event ObjectStorageClusterProcessedTasks; + extern const Event ObjectStorageClusterWaitingMicroseconds; } namespace CurrentMetrics @@ -203,6 +206,8 @@ namespace Setting extern const SettingsUInt64 s3_path_filter_limit; extern const SettingsBool use_parquet_metadata_cache; extern const SettingsBool s3_validate_etag_on_read; + extern const SettingsBool use_object_storage_list_objects_cache; + extern const SettingsBool allow_experimental_iceberg_read_optimization; } static void logIcebergFileStats(const ObjectInfoPtr & object_info, const LoggerPtr & log) @@ -445,18 +450,52 @@ std::shared_ptr StorageObjectStorageSource::createFileIterator( // If paths contains a value, validate the extracted paths and use the key-based iterator // (even if the result is empty, indicating no scanning is required). if (!paths) + { + std::shared_ptr object_iterator = nullptr; + std::unique_ptr cache_ptr = nullptr; + + if (local_context->getSettingsRef()[Setting::use_object_storage_list_objects_cache] && object_storage->supportsListObjectsCache()) + { + auto & cache = ObjectStorageListObjectsCache::instance(); + ObjectStorageListObjectsCache::Key cache_key {object_storage->getDescription(), configuration->getNamespace(), configuration->getRawPath().cutGlobs(configuration->supportsPartialPathPrefix()), with_tags}; + + if (auto objects_info = cache.get(cache_key, /*filter_by_prefix=*/ false)) + { + /// suboptimal because of the recent upstream changes to the ObjectInfo structure + /// re-think this with more time and see if there is a more optimized approach + RelativePathsWithMetadata relative_path_with_metadata; + relative_path_with_metadata.reserve(objects_info->size()); + + for (const auto & object_info : *objects_info) + { + relative_path_with_metadata.emplace_back(std::make_shared(object_info->getPath(), object_info->getObjectMetadata())); + } + + object_iterator = std::make_shared(std::move(relative_path_with_metadata)); + } + else + { + cache_ptr = std::make_unique(cache, cache_key); + object_iterator = object_storage->iterate(configuration->getRawPath().cutGlobs(configuration->supportsPartialPathPrefix()), query_settings.list_object_keys_size, with_tags, std::nullopt); + } + } + else + { + object_iterator = object_storage->iterate(configuration->getRawPath().cutGlobs(configuration->supportsPartialPathPrefix()), query_settings.list_object_keys_size, with_tags, std::nullopt); + } + iterator = std::make_unique( - object_storage, + object_iterator, configuration, predicate, virtual_columns, hive_columns, local_context, is_archive ? nullptr : read_keys, - query_settings.list_object_keys_size, query_settings.throw_on_zero_files_match, - with_tags, - file_progress_callback); + file_progress_callback, + std::move(cache_ptr)); + } else { // Validate that extracted paths match the glob pattern to prevent scanning unallowed data @@ -681,6 +720,16 @@ Chunk StorageObjectStorageSource::generate() read_context, format_settings); + /// Not empty when allow_experimental_iceberg_read_optimization=true + /// and some columns were removed from read list as columns with constant values. + /// Restore data for these columns. + for (const auto & constant_column : reader.constant_columns_with_values) + { + chunk.addColumn(constant_column.first, + constant_column.second.name_and_type.type->createColumnConst( + chunk.getNumRows(), constant_column.second.value)); + } + if (read_from_format_info.requested_virtual_columns.contains("_headers")) { auto type = std::make_shared( @@ -905,7 +954,7 @@ void StorageObjectStorageSource::addNumRowsToCache(const ObjectInfo & object_inf { const auto cache_key = getKeyForSchemaCache( getUniqueStoragePathIdentifier(*configuration, object_info), - object_info.getFileFormat().value_or(configuration->format), + object_info.getFileFormat().value_or(configuration->getFormat()), format_settings, read_context); schema_cache.addNumRows(cache_key, num_rows); @@ -957,7 +1006,26 @@ StorageObjectStorageSource::ReaderHolder StorageObjectStorageSource::createReade { object_info = file_iterator->next(processor); - if (!object_info || object_info->getPath().empty()) + if (!object_info) + return {}; + + if (object_info->relative_path_with_metadata.getCommand().isValid()) + { + auto retry_after_us = object_info->relative_path_with_metadata.getCommand().getRetryAfterUs(); + if (retry_after_us.has_value()) + { + /// TODO: Make asyncronous waiting without sleep in thread + /// Now this sleep is on executor node in worker thread + /// Does not block query initiator + auto wait_time = std::min(Poco::Timestamp::TimeDiff(100000ul), retry_after_us.value()); + ProfileEvents::increment(ProfileEvents::ObjectStorageClusterWaitingMicroseconds, wait_time); + sleepForMicroseconds(wait_time); + continue; + } + object_info->relative_path_with_metadata.setFileMetaInfo(object_info->relative_path_with_metadata.getCommand().getFileMetaInfo()); + } + + if (object_info->getPath().empty()) return {}; if (!object_info->getObjectMetadata()) { @@ -1012,7 +1080,7 @@ StorageObjectStorageSource::ReaderHolder StorageObjectStorageSource::createReade continue; auto file_bucket_info = FormatFactory::instance().getFileBucketInfo( - object_info->getFileFormat().value_or(configuration->format)); + object_info->getFileFormat().value_or(configuration->getFormat())); if (file_bucket_info) { auto filtered = file_bucket_info->filterByMatchingRowGroups(matching_row_groups); @@ -1025,18 +1093,28 @@ StorageObjectStorageSource::ReaderHolder StorageObjectStorageSource::createReade break; } + ProfileEvents::increment(ProfileEvents::ObjectStorageClusterProcessedTasks); + QueryPipelineBuilder builder; std::shared_ptr source; std::unique_ptr read_buf; + std::optional rows_count_from_metadata; auto try_get_num_rows_from_cache = [&]() -> std::optional { + if (rows_count_from_metadata.has_value()) + { + /// Must be non negative here + size_t value = rows_count_from_metadata.value(); + return value; + } + if (!schema_cache) return std::nullopt; const auto cache_key = getKeyForSchemaCache( getUniqueStoragePathIdentifier(*configuration, *object_info), - object_info->getFileFormat().value_or(configuration->format), + object_info->getFileFormat().value_or(configuration->getFormat()), format_settings, context_); @@ -1059,6 +1137,144 @@ StorageObjectStorageSource::ReaderHolder StorageObjectStorageSource::createReade /// response. Skip the shortcut when `_headers` is requested so the real `GET` headers are used. const bool headers_requested = read_from_format_info.requested_virtual_columns.contains("_headers"); + /// List of columns with constant value in current file, and values + std::map constant_columns_with_values; + std::unordered_set constant_columns; + + NamesAndTypesList requested_columns_copy = read_from_format_info.requested_columns; + + std::unordered_map> requested_columns_list; + { + size_t column_index = 0; + for (const auto & column : requested_columns_copy) + requested_columns_list[column.getNameInStorage()] = std::make_pair(column_index++, column); + } + + if (context_->getSettingsRef()[Setting::allow_experimental_iceberg_read_optimization]) + { + auto file_meta_data = object_info->relative_path_with_metadata.getFileMetaInfo(); + if (file_meta_data.has_value()) + { + bool is_all_rows_count_equals = true; + for (const auto & column : file_meta_data.value()->columns_info) + { + if (is_all_rows_count_equals && column.second.rows_count.has_value()) + { + if (rows_count_from_metadata.has_value()) + { + if (column.second.rows_count.value() != rows_count_from_metadata.value()) + { + LOG_WARNING(log, "Inconsistent rows count for file {} in metadats, ignored", object_info->getPath()); + is_all_rows_count_equals = false; + rows_count_from_metadata = std::nullopt; + } + } + else if (column.second.rows_count.value() < 0) + { + LOG_WARNING(log, "Negative rows count for file {} in metadats, ignored", object_info->getPath()); + is_all_rows_count_equals = false; + rows_count_from_metadata = std::nullopt; + } + else + rows_count_from_metadata = column.second.rows_count; + } + + if (column.second.hyperrectangle.has_value()) + { + auto column_name = column.first; + + auto i_column = requested_columns_list.find(column_name); + if (i_column == requested_columns_list.end()) + continue; + + if (column.second.hyperrectangle.value().isPoint() && + (!column.second.nulls_count.has_value() || column.second.nulls_count.value() <= 0)) + { + /// isPoint() method checks before that left==right + constant_columns_with_values[i_column->second.first] = + ConstColumnWithValue{ + i_column->second.second, + column.second.hyperrectangle.value().left + }; + constant_columns.insert(column_name); + + LOG_DEBUG(log, "In file {} constant column '{}' type '{}' with value '{}'", + object_info->getPath(), + column_name, + i_column->second.second.type, + column.second.hyperrectangle.value().left.dump()); + } + else if (column.second.rows_count.has_value() && column.second.nulls_count.has_value() + && column.second.rows_count.value() == column.second.nulls_count.value() + && i_column->second.second.type->isNullable()) + { + constant_columns_with_values[i_column->second.first] = + ConstColumnWithValue{ + i_column->second.second, + Field() + }; + constant_columns.insert(column_name); + + LOG_DEBUG(log, "In file {} constant column '{}' type '{}' with value 'NULL'", + object_info->getPath(), + column_name, + i_column->second.second.type); + } + } + } + if (!file_meta_data.value()->columns_info.empty()) + { + for (const auto & column : requested_columns_list) + { + const auto & column_name = column.first; + + if (file_meta_data.value()->columns_info.contains(column_name)) + continue; + + if (!column.second.second.type->isNullable()) + continue; + + /// With View over Iceberg table we have someting like 'materialize(time)' as column_name + /// Simple cheap check + if (column_name.starts_with("materialize(") && column_name.ends_with(")")) + continue; + + /// Skip columns produced by prewhere or row-level filter expressions — + /// they are computed at read time, not stored in the file. + if (format_filter_info + && ((format_filter_info->prewhere_info && column_name == format_filter_info->prewhere_info->prewhere_column_name) + || (format_filter_info->row_level_filter && column_name == format_filter_info->row_level_filter->column_name))) + continue; + + /// Column is nullable and absent in file + constant_columns_with_values[column.second.first] = + ConstColumnWithValue{ + column.second.second, + Field() + }; + constant_columns.insert(column_name); + + LOG_DEBUG(log, "In file {} constant column '{}' type '{}' with value 'NULL'", + object_info->getPath(), + column_name, + column.second.second.type); + } + } + } + + if (!constant_columns.empty()) + { + size_t original_columns = requested_columns_copy.size(); + requested_columns_copy = requested_columns_copy.eraseNames(constant_columns); + if (requested_columns_copy.size() + constant_columns.size() != original_columns) + throw Exception(ErrorCodes::LOGICAL_ERROR, "Can't remove constant columns for file {} correct, fallback to read. Founded constant columns: [{}]", + object_info->getPath(), constant_columns); + if (requested_columns_copy.empty() + && (!format_filter_info || (!format_filter_info->row_level_filter && !format_filter_info->prewhere_info))) + need_only_count = true; + } + } + /// Equality-delete FilterTransform evaluates predicates against column values, but need_only_count /// emits default-filled chunks — so disable the fast path for equality deletes only. /// Position deletes and deletion vectors filter by row index (preserved on synthetic chunks); @@ -1091,10 +1307,12 @@ StorageObjectStorageSource::ReaderHolder StorageObjectStorageSource::createReade columns.emplace_back(type->createColumn(), type, name); builder.init(Pipe(std::make_shared( std::make_shared(columns), *num_rows_from_cache, max_block_size))); + if (!constant_columns.empty()) + configuration->addDeleteTransformers(object_info, builder, format_settings, parser_shared_resources, context_); } else { - const auto format_name = object_info->getFileFormat().value_or(configuration->format); + const auto format_name = object_info->getFileFormat().value_or(configuration->getFormat()); const bool input_format_does_not_read_file = Poco::toLower(format_name) == "one"; CompressionMethod compression_method = {}; @@ -1107,7 +1325,7 @@ StorageObjectStorageSource::ReaderHolder StorageObjectStorageSource::createReade else if (const auto * object_info_in_archive = dynamic_cast(object_info.get())) { ProfileEvents::increment(ProfileEvents::ObjectStorageReadObjects); - compression_method = chooseCompressionMethod(configuration->getPathInArchive(), configuration->compression_method); + compression_method = chooseCompressionMethod(configuration->getPathInArchive(), configuration->getCompressionMethod()); const auto & archive_reader = object_info_in_archive->archive_reader; read_buf = archive_reader->readFile(object_info_in_archive->path_in_archive, /*throw_on_not_found=*/true); } @@ -1152,7 +1370,7 @@ StorageObjectStorageSource::ReaderHolder StorageObjectStorageSource::createReade /// tables (e.g. Iceberg with Parquet + ORC files), table-level PREWHERE support /// may not match the individual file's format capabilities. /// See https://github.com/ClickHouse/ClickHouse/issues/96829 - const auto actual_format = object_info->getFileFormat().value_or(configuration->format); + const auto actual_format = object_info->getFileFormat().value_or(configuration->getFormat()); const bool format_supports_prewhere = FormatFactory::instance().checkIfFormatSupportsPrewhere(actual_format, context_, format_settings); @@ -1508,7 +1726,7 @@ StorageObjectStorageSource::ReaderHolder StorageObjectStorageSource::createReade /// from chunk read by IInputFormat. builder.addSimpleTransform([&](const SharedHeader & header) { - return std::make_shared(header, read_from_format_info.requested_columns); + return std::make_shared(header, requested_columns_copy); }); auto pipeline = std::make_unique(QueryPipelineBuilder::getPipeline(std::move(builder))); @@ -1517,7 +1735,12 @@ StorageObjectStorageSource::ReaderHolder StorageObjectStorageSource::createReade ProfileEvents::increment(ProfileEvents::EngineFileLikeReadFiles); return ReaderHolder( - object_info, std::move(read_buf), std::move(source), std::move(pipeline), std::move(current_reader)); + object_info, + std::move(read_buf), + std::move(source), + std::move(pipeline), + std::move(current_reader), + std::move(constant_columns_with_values)); } std::future StorageObjectStorageSource::createReaderAsync() @@ -1748,19 +1971,18 @@ std::unique_ptr createReadBuffer( } StorageObjectStorageSource::GlobIterator::GlobIterator( - ObjectStoragePtr object_storage_, - StorageObjectStorageConfigurationPtr configuration_, + const ObjectStorageIteratorPtr & object_storage_iterator_, + ConfigurationPtr configuration_, const ActionsDAG::Node * predicate, const NamesAndTypesList & virtual_columns_, const NamesAndTypesList & hive_columns_, ContextPtr context_, ObjectInfos * read_keys_, - size_t list_object_keys_size, bool throw_on_zero_files_match_, - bool with_tags, - std::function file_progress_callback_) + std::function file_progress_callback_, + std::unique_ptr list_cache_) : WithContext(context_) - , object_storage(object_storage_) + , object_storage_iterator(object_storage_iterator_) , configuration(configuration_) , virtual_columns(virtual_columns_) , hive_columns(hive_columns_) @@ -1769,6 +1991,7 @@ StorageObjectStorageSource::GlobIterator::GlobIterator( , read_keys(read_keys_) , local_context(context_) , file_progress_callback(file_progress_callback_) + , list_cache(std::move(list_cache_)) { const auto & reading_path = configuration->getPathForRead(); if (reading_path.hasGlobs()) @@ -1777,8 +2000,6 @@ StorageObjectStorageSource::GlobIterator::GlobIterator( const auto & key_with_globs = reading_path; const auto key_prefix = reading_path.cutGlobs(configuration->supportsPartialPathPrefix()); - object_storage_iterator = object_storage->iterate(key_prefix, list_object_keys_size, with_tags, std::nullopt); - matcher = std::make_unique(makeRegexpPatternFromGlobs(key_with_globs.path)); if (!matcher->ok()) { @@ -1844,6 +2065,10 @@ ObjectInfoPtr StorageObjectStorageSource::GlobIterator::nextUnlocked(size_t /* p auto result = object_storage_iterator->getCurrentBatchAndScheduleNext(); if (!result.has_value()) { + if (list_cache) + { + list_cache->set(std::move(object_list)); + } is_finished = true; LOG_DEBUG(log, "Listing finished: total_listed={}, glob_filtered={}, predicate_filtered={}", total_listed, total_glob_filtered, total_predicate_filtered); @@ -1862,6 +2087,11 @@ ObjectInfoPtr StorageObjectStorageSource::GlobIterator::nextUnlocked(size_t /* p listed_in_batch = new_batch.size(); + if (list_cache) + { + object_list.insert(object_list.end(), new_batch.begin(), new_batch.end()); + } + for (auto it = new_batch.begin(); it != new_batch.end();) { const auto path_for_matching = match_web_paths_only @@ -2024,12 +2254,14 @@ StorageObjectStorageSource::ReaderHolder::ReaderHolder( std::unique_ptr read_buf_, std::shared_ptr source_, std::unique_ptr pipeline_, - std::unique_ptr reader_) + std::unique_ptr reader_, + std::map && constant_columns_with_values_) : object_info(std::move(object_info_)) , read_buf(std::move(read_buf_)) , source(std::move(source_)) , pipeline(std::move(pipeline_)) , reader(std::move(reader_)) + , constant_columns_with_values(std::move(constant_columns_with_values_)) { } @@ -2043,6 +2275,7 @@ StorageObjectStorageSource::ReaderHolder::operator=(ReaderHolder && other) noexc source = std::move(other.source); read_buf = std::move(other.read_buf); object_info = std::move(other.object_info); + constant_columns_with_values = std::move(other.constant_columns_with_values); return *this; } @@ -2057,6 +2290,12 @@ StorageObjectStorageSource::ReadTaskIterator::ReadTaskIterator( , is_archive(is_archive_) , object_storage(object_storage_) { + if (!getContext()->isSwarmModeEnabled()) + { + LOG_DEBUG(getLogger("StorageObjectStorageSource"), "STOP SWARM MODE called, stop getting new tasks"); + return; + } + ThreadPool pool( CurrentMetrics::StorageObjectStorageThreads, CurrentMetrics::StorageObjectStorageThreadsActive, @@ -2104,6 +2343,12 @@ ObjectInfoPtr StorageObjectStorageSource::ReadTaskIterator::next(size_t) ObjectInfoPtr object_info; if (current_index >= buffer.size()) { + if (!getContext()->isSwarmModeEnabled()) + { + LOG_DEBUG(getLogger("StorageObjectStorageSource"), "STOP SWARM MODE called, stop getting new tasks"); + return nullptr; + } + auto task = callback(); if (auto query_status = getContext()->getProcessListElement()) diff --git a/src/Storages/ObjectStorage/StorageObjectStorageSource.h b/src/Storages/ObjectStorage/StorageObjectStorageSource.h index ec9b155d0860..3cf2b9788b23 100644 --- a/src/Storages/ObjectStorage/StorageObjectStorageSource.h +++ b/src/Storages/ObjectStorage/StorageObjectStorageSource.h @@ -12,6 +12,7 @@ #include #include #include +#include namespace DB { @@ -98,6 +99,12 @@ class StorageObjectStorageSource final : public ISource size_t total_files_read = 0; LoggerPtr log = getLogger("StorageObjectStorageSource"); + struct ConstColumnWithValue + { + NameAndTypePair name_and_type; + Field value; + }; + struct ReaderHolder : private boost::noncopyable { public: @@ -106,7 +113,8 @@ class StorageObjectStorageSource final : public ISource std::unique_ptr read_buf_, std::shared_ptr source_, std::unique_ptr pipeline_, - std::unique_ptr reader_); + std::unique_ptr reader_, + std::map && constant_columns_with_values_); ReaderHolder() = default; ReaderHolder(ReaderHolder && other) noexcept { *this = std::move(other); } @@ -126,6 +134,9 @@ class StorageObjectStorageSource final : public ISource std::shared_ptr source; std::unique_ptr pipeline; std::unique_ptr reader; + + public: + std::map constant_columns_with_values; }; ReaderHolder reader; @@ -198,18 +209,33 @@ class StorageObjectStorageSource::ReadTaskIterator : public IObjectIterator, pri class StorageObjectStorageSource::GlobIterator : public IObjectIterator, WithContext { public: + struct ListObjectsCacheWithKey + { + ListObjectsCacheWithKey(ObjectStorageListObjectsCache & cache_, const ObjectStorageListObjectsCache::Key & key_) : cache(cache_), key(key_) {} + + void set(ObjectStorageListObjectsCache::Value && value) const + { + cache.set(key, std::make_shared(std::move(value))); + } + + private: + ObjectStorageListObjectsCache & cache; + ObjectStorageListObjectsCache::Key key; + }; + + using ConfigurationPtr = std::shared_ptr; + GlobIterator( - ObjectStoragePtr object_storage_, - StorageObjectStorageConfigurationPtr configuration_, + const ObjectStorageIteratorPtr & object_storage_iterator_, + ConfigurationPtr configuration_, const ActionsDAG::Node * predicate, const NamesAndTypesList & virtual_columns_, const NamesAndTypesList & hive_columns_, ContextPtr context_, ObjectInfos * read_keys_, - size_t list_object_keys_size, bool throw_on_zero_files_match_, - bool with_tags, - std::function file_progress_callback_ = {}); + std::function file_progress_callback_ = {}, + std::unique_ptr list_cache_ = nullptr); ~GlobIterator() override = default; @@ -222,7 +248,7 @@ class StorageObjectStorageSource::GlobIterator : public IObjectIterator, WithCon void createFilterAST(const String & any_key); void fillBufferForKey(const std::string & uri_key); - const ObjectStoragePtr object_storage; + ObjectStorageIteratorPtr object_storage_iterator; const StorageObjectStorageConfigurationPtr configuration; const NamesAndTypesList virtual_columns; const NamesAndTypesList hive_columns; @@ -234,7 +260,6 @@ class StorageObjectStorageSource::GlobIterator : public IObjectIterator, WithCon ObjectInfos object_infos; ObjectInfos * read_keys; ExpressionActionsPtr filter_expr; - ObjectStorageIteratorPtr object_storage_iterator; bool recursive{false}; bool match_web_paths_only{false}; std::vector expanded_keys; @@ -252,6 +277,9 @@ class StorageObjectStorageSource::GlobIterator : public IObjectIterator, WithCon size_t total_listed = 0; size_t total_glob_filtered = 0; size_t total_predicate_filtered = 0; + + std::unique_ptr list_cache; + ObjectInfos object_list; }; class StorageObjectStorageSource::KeysIterator : public IObjectIterator @@ -277,7 +305,7 @@ class StorageObjectStorageSource::KeysIterator : public IObjectIterator const ObjectStoragePtr object_storage; const NamesAndTypesList virtual_columns; const std::function file_progress_callback; - const std::vector keys; + const Strings keys; std::atomic index = 0; const bool ignore_non_existent_files; const bool skip_object_metadata; diff --git a/src/Storages/ObjectStorage/StorageObjectStorageStableTaskDistributor.cpp b/src/Storages/ObjectStorage/StorageObjectStorageStableTaskDistributor.cpp index dd3b6a1544f3..abf8e1746e9b 100644 --- a/src/Storages/ObjectStorage/StorageObjectStorageStableTaskDistributor.cpp +++ b/src/Storages/ObjectStorage/StorageObjectStorageStableTaskDistributor.cpp @@ -3,13 +3,20 @@ #include #include +namespace ProfileEvents +{ + extern const Event ObjectStorageClusterSentToMatchedReplica; + extern const Event ObjectStorageClusterSentToNonMatchedReplica; +}; + namespace DB { namespace ErrorCodes { extern const int LOGICAL_ERROR; -} + extern const int CANNOT_READ_ALL_DATA; +}; namespace { @@ -27,29 +34,68 @@ String getSchedulingIdentifier(const ObjectInfoPtr & object_info, bool send_over StorageObjectStorageStableTaskDistributor::StorageObjectStorageStableTaskDistributor( std::shared_ptr iterator_, std::vector && ids_of_nodes_, - bool send_over_whole_archive_) + bool send_over_whole_archive_, + uint64_t lock_object_storage_task_distribution_ms_, + bool iceberg_read_optimization_enabled_) : iterator(std::move(iterator_)) , send_over_whole_archive(send_over_whole_archive_) , connection_to_files(ids_of_nodes_.size()) , ids_of_nodes(std::move(ids_of_nodes_)) + , lock_object_storage_task_distribution_us(lock_object_storage_task_distribution_ms_ * 1000) , iterator_exhausted(false) + , iceberg_read_optimization_enabled(iceberg_read_optimization_enabled_) { + Poco::Timestamp now; + size_t nodes = ids_of_nodes.size(); + for (size_t i = 0; i < nodes; ++i) + { + replica_to_files_to_be_processed[i] = std::list{}; + last_node_activity[i] = now; + } } ObjectInfoPtr StorageObjectStorageStableTaskDistributor::getNextTask(size_t number_of_current_replica) { LOG_TRACE(log, "Received request from replica {} looking for a file", number_of_current_replica); - // 1. Check pre-queued files first - if (auto file = getPreQueuedFile(number_of_current_replica)) - return file; + saveLastNodeActivity(number_of_current_replica); - // 2. Try to find a matching file from the iterator - if (auto file = getMatchingFileFromIterator(number_of_current_replica)) - return file; + { + std::lock_guard lock(mutex); + auto processed_file_list_ptr = replica_to_files_to_be_processed.find(number_of_current_replica); + if (processed_file_list_ptr == replica_to_files_to_be_processed.end()) + throw Exception( + ErrorCodes::LOGICAL_ERROR, + "Replica number {} was marked as lost, can't set task for it anymore", + number_of_current_replica + ); + } + // 1. Check pre-queued files first + auto file = getPreQueuedFile(number_of_current_replica); + // 2. Try to find a matching file from the iterator + if (!file) + file = getMatchingFileFromIterator(number_of_current_replica); // 3. Process unprocessed files if iterator is exhausted - return getAnyUnprocessedFile(number_of_current_replica); + if (!file) + file = getAnyUnprocessedFile(number_of_current_replica); + + if (file) + { + std::lock_guard lock(mutex); + auto processed_file_list_ptr = replica_to_files_to_be_processed.find(number_of_current_replica); + if (processed_file_list_ptr == replica_to_files_to_be_processed.end()) + { // It is possible that replica was lost after check in the begining of the method + auto file_identifier = getSchedulingIdentifier(file, send_over_whole_archive); + auto file_replica_idx = getReplicaForFile(file_identifier); + unprocessed_files.emplace(file_identifier, std::make_pair(file, file_replica_idx)); + connection_to_files[file_replica_idx].push_back(file); + } + else + processed_file_list_ptr->second.push_back(file); + } + + return file; } size_t StorageObjectStorageStableTaskDistributor::getReplicaForFile(const String & file_path) @@ -61,16 +107,27 @@ size_t StorageObjectStorageStableTaskDistributor::getReplicaForFile(const String return 0; /// Rendezvous hashing - size_t best_id = 0; - UInt64 best_weight = sipHash64(ids_of_nodes[0] + file_path); - for (size_t id = 1; id < nodes_count; ++id) + auto replica = replica_to_files_to_be_processed.begin(); + if (replica == replica_to_files_to_be_processed.end()) + throw Exception( + ErrorCodes::LOGICAL_ERROR, + "No active replicas, can't find best replica for file {}", + file_path + ); + + size_t best_id = replica->first; + UInt64 best_weight = sipHash64(ids_of_nodes[best_id] + file_path); + ++replica; + while (replica != replica_to_files_to_be_processed.end()) { + size_t id = replica->first; UInt64 weight = sipHash64(ids_of_nodes[id] + file_path); if (weight > best_weight) { best_weight = weight; best_id = id; } + ++replica; } return best_id; } @@ -108,6 +165,7 @@ ObjectInfoPtr StorageObjectStorageStableTaskDistributor::getPreQueuedFile(size_t number_of_current_replica ); + ProfileEvents::increment(ProfileEvents::ObjectStorageClusterSentToMatchedReplica); return next_file; } @@ -151,7 +209,24 @@ ObjectInfoPtr StorageObjectStorageStableTaskDistributor::getMatchingFileFromIter file_identifier = object_info->getIdentifier(); } - size_t file_replica_idx = getReplicaForFile(file_identifier); + if (iceberg_read_optimization_enabled) + { + auto file_meta_info = object_info->relative_path_with_metadata.getFileMetaInfo(); + if (file_meta_info.has_value()) + { + auto file_path = send_over_whole_archive ? object_info->getPathOrPathToArchiveIfArchive() : object_info->getPath(); + object_info->relative_path_with_metadata.command.setFilePath(file_path); + object_info->relative_path_with_metadata.command.setFileMetaInfo(file_meta_info.value()); + } + } + + size_t file_replica_idx; + + { + std::lock_guard lock(mutex); + file_replica_idx = getReplicaForFile(file_identifier); + } + if (file_replica_idx == number_of_current_replica) { LOG_TRACE( @@ -159,6 +234,7 @@ ObjectInfoPtr StorageObjectStorageStableTaskDistributor::getMatchingFileFromIter file_identifier, number_of_current_replica ); + ProfileEvents::increment(ProfileEvents::ObjectStorageClusterSentToMatchedReplica); return object_info; } LOG_TEST( @@ -172,7 +248,7 @@ ObjectInfoPtr StorageObjectStorageStableTaskDistributor::getMatchingFileFromIter // Queue file for its assigned replica { std::lock_guard lock(mutex); - unprocessed_files.emplace(file_identifier, object_info); + unprocessed_files.emplace(file_identifier, std::make_pair(object_info, file_replica_idx)); connection_to_files[file_replica_idx].push_back(object_info); } } @@ -182,26 +258,99 @@ ObjectInfoPtr StorageObjectStorageStableTaskDistributor::getMatchingFileFromIter ObjectInfoPtr StorageObjectStorageStableTaskDistributor::getAnyUnprocessedFile(size_t number_of_current_replica) { + /// Limit time of node activity to keep task in queue + Poco::Timestamp activity_limit; + Poco::Timestamp oldest_activity; + if (lock_object_storage_task_distribution_us > 0) + activity_limit -= lock_object_storage_task_distribution_us; + std::lock_guard lock(mutex); if (!unprocessed_files.empty()) { auto it = unprocessed_files.begin(); - auto next_file = it->second; - unprocessed_files.erase(it); - auto file_path = getSchedulingIdentifier(next_file, send_over_whole_archive); + while (it != unprocessed_files.end()) + { + auto number_of_matched_replica = it->second.second; + auto last_activity = last_node_activity.find(number_of_matched_replica); + if (lock_object_storage_task_distribution_us <= 0 // file deferring is turned off + || it->second.second == number_of_current_replica // file is matching with current replica + || last_activity == last_node_activity.end() // msut never be happen, last_activity is filled for each replica on start + || activity_limit > last_activity->second) // matched replica did not ask for a new files for a while + { + auto next_file = it->second.first; + unprocessed_files.erase(it); + + auto file_path = getSchedulingIdentifier(next_file, send_over_whole_archive); + LOG_TRACE( + log, + "Iterator exhausted. Assigning unprocessed file {} to replica {} from matched replica {}", + file_path, + number_of_current_replica, + number_of_matched_replica + ); + + ProfileEvents::increment(ProfileEvents::ObjectStorageClusterSentToNonMatchedReplica); + return next_file; + } + + oldest_activity = std::min(oldest_activity, last_activity->second); + ++it; + } + LOG_TRACE( log, - "Iterator exhausted. Assigning unprocessed file {} to replica {}", - file_path, - number_of_current_replica + "No unprocessed file for replica {}, need to retry after {} us", + number_of_current_replica, + oldest_activity - activity_limit ); - return next_file; + /// All unprocessed files owned by alive replicas with recenlty activity + /// Need to retry after (oldest_activity - activity_limit) microseconds + RelativePathWithMetadata::CommandInTaskResponse response; + response.setRetryAfterUs(oldest_activity - activity_limit); + return std::make_shared(response.toString()); } return {}; } +void StorageObjectStorageStableTaskDistributor::saveLastNodeActivity(size_t number_of_current_replica) +{ + Poco::Timestamp now; + std::lock_guard lock(mutex); + last_node_activity[number_of_current_replica] = now; +} + +void StorageObjectStorageStableTaskDistributor::rescheduleTasksFromReplica(size_t number_of_current_replica) +{ + LOG_INFO(log, "Replica {} is marked as lost, tasks are returned to queue", number_of_current_replica); + std::lock_guard lock(mutex); + + auto processed_file_list_ptr = replica_to_files_to_be_processed.find(number_of_current_replica); + if (processed_file_list_ptr == replica_to_files_to_be_processed.end()) + throw Exception( + ErrorCodes::LOGICAL_ERROR, + "Replica number {} was marked as lost already", + number_of_current_replica + ); + + if (replica_to_files_to_be_processed.size() < 2) + throw Exception( + ErrorCodes::CANNOT_READ_ALL_DATA, + "All replicas were marked as lost" + ); + + auto files = std::move(processed_file_list_ptr->second); + replica_to_files_to_be_processed.erase(number_of_current_replica); + for (const auto & file : files) + { + auto file_identifier = getSchedulingIdentifier(file, send_over_whole_archive); + auto file_replica_idx = getReplicaForFile(file_identifier); + unprocessed_files.emplace(file_identifier, std::make_pair(file, file_replica_idx)); + connection_to_files[file_replica_idx].push_back(file); + } +} + } diff --git a/src/Storages/ObjectStorage/StorageObjectStorageStableTaskDistributor.h b/src/Storages/ObjectStorage/StorageObjectStorageStableTaskDistributor.h index 02d3ba7a030f..0cd10ac7188e 100644 --- a/src/Storages/ObjectStorage/StorageObjectStorageStableTaskDistributor.h +++ b/src/Storages/ObjectStorage/StorageObjectStorageStableTaskDistributor.h @@ -4,7 +4,13 @@ #include #include #include +#include + +#include + #include +#include +#include #include #include #include @@ -18,26 +24,38 @@ class StorageObjectStorageStableTaskDistributor StorageObjectStorageStableTaskDistributor( std::shared_ptr iterator_, std::vector && ids_of_nodes_, - bool send_over_whole_archive_); + bool send_over_whole_archive_, + uint64_t lock_object_storage_task_distribution_ms_, + bool iceberg_read_optimization_enabled_); ObjectInfoPtr getNextTask(size_t number_of_current_replica); + /// Insert objects back to unprocessed files + void rescheduleTasksFromReplica(size_t number_of_current_replica); + private: size_t getReplicaForFile(const String & file_path); ObjectInfoPtr getPreQueuedFile(size_t number_of_current_replica); ObjectInfoPtr getMatchingFileFromIterator(size_t number_of_current_replica); ObjectInfoPtr getAnyUnprocessedFile(size_t number_of_current_replica); + void saveLastNodeActivity(size_t number_of_current_replica); + const std::shared_ptr iterator; const bool send_over_whole_archive; std::vector> connection_to_files; - std::unordered_map unprocessed_files; + std::unordered_map> unprocessed_files; std::vector ids_of_nodes; + std::unordered_map last_node_activity; + Poco::Timestamp::TimeDiff lock_object_storage_task_distribution_us; + std::unordered_map> replica_to_files_to_be_processed; + std::mutex mutex; bool iterator_exhausted = false; + bool iceberg_read_optimization_enabled = false; LoggerPtr log = getLogger("StorageClusterTaskDistributor"); }; diff --git a/src/Storages/ObjectStorage/Utils.cpp b/src/Storages/ObjectStorage/Utils.cpp index a3baa40beb1e..958d2adc3be7 100644 --- a/src/Storages/ObjectStorage/Utils.cpp +++ b/src/Storages/ObjectStorage/Utils.cpp @@ -72,14 +72,13 @@ std::optional checkAndGetNewFileOnInsertIfNeeded( void resolveSchemaAndFormat( ColumnsDescription & columns, - std::string & format, ObjectStoragePtr object_storage, - const StorageObjectStorageConfigurationPtr & configuration, + StorageObjectStorageConfigurationPtr & configuration, std::optional format_settings, std::string & sample_path, const ContextPtr & context) { - if (format == "auto") + if (configuration->getFormat() == "auto") { if (configuration->isDataLakeConfiguration()) { @@ -101,21 +100,23 @@ void resolveSchemaAndFormat( if (columns.empty()) { - if (format == "auto") + if (configuration->getFormat() == "auto") { + std::string format; std::tie(columns, format) = StorageObjectStorage::resolveSchemaAndFormatFromData( object_storage, configuration, format_settings, sample_path, context); + configuration->setFormat(format); } else { - chassert(!format.empty()); + chassert(!configuration->getFormat().empty()); columns = StorageObjectStorage::resolveSchemaFromData(object_storage, configuration, format_settings, sample_path, context); } } } - else if (format == "auto") + else if (configuration->getFormat() == "auto") { - format = StorageObjectStorage::resolveFormatFromData(object_storage, configuration, format_settings, sample_path, context); + configuration->setFormat(StorageObjectStorage::resolveFormatFromData(object_storage, configuration, format_settings, sample_path, context)); } validateSupportedColumns(columns, *configuration); diff --git a/src/Storages/ObjectStorage/Utils.h b/src/Storages/ObjectStorage/Utils.h index 2e4566e61014..096b401f9852 100644 --- a/src/Storages/ObjectStorage/Utils.h +++ b/src/Storages/ObjectStorage/Utils.h @@ -18,9 +18,8 @@ std::optional checkAndGetNewFileOnInsertIfNeeded( void resolveSchemaAndFormat( ColumnsDescription & columns, - std::string & format, ObjectStoragePtr object_storage, - const StorageObjectStorageConfigurationPtr & configuration, + StorageObjectStorageConfigurationPtr & configuration, std::optional format_settings, std::string & sample_path, const ContextPtr & context); diff --git a/src/Storages/ObjectStorage/registerStorageObjectStorage.cpp b/src/Storages/ObjectStorage/registerStorageObjectStorage.cpp index 6b91ad4f349f..8d901f42fb2f 100644 --- a/src/Storages/ObjectStorage/registerStorageObjectStorage.cpp +++ b/src/Storages/ObjectStorage/registerStorageObjectStorage.cpp @@ -16,6 +16,7 @@ #include #include #include +#include #include #include #include @@ -46,11 +47,20 @@ namespace // LocalObjectStorage is only supported for Iceberg Datalake operations where Avro format is required. For regular file access, use FileStorage instead. #if USE_AWS_S3 || USE_AZURE_BLOB_STORAGE || USE_HDFS || USE_AVRO -std::shared_ptr +StoragePtr createStorageObjectStorage(const StorageFactory::Arguments & args, StorageObjectStorageConfigurationPtr configuration) { const auto context = args.getLocalContext(); - StorageObjectStorageConfiguration::initialize(*configuration, args.engine_args, context, false, &args.table_id); + + std::string cluster_name; + + if (args.storage_def->settings) + { + if (const auto * value = args.storage_def->settings->changes.tryGet("object_storage_cluster")) + cluster_name = value->safeGet(); + } + + configuration->initialize(args.engine_args, context, false, &args.table_id); // Use format settings from global server context + settings from // the SETTINGS clause of the create query. Settings from current @@ -105,24 +115,26 @@ createStorageObjectStorage(const StorageFactory::Arguments & args, StorageObject && (args.table_id.table_name.ends_with("_s3") || args.table_id.table_name.ends_with("_s3queue"))) configuration->force_anonymous_load_fallback = true; - return std::make_shared( + return std::make_shared( + cluster_name, configuration, // We only want to perform write actions (e.g. create a container in Azure) when the table is being created, // and we want to avoid it when we load the table after a server restart. configuration->createObjectStorage(context, /* is_readonly */ args.mode != LoadingStrictnessLevel::CREATE, std::nullopt), - context_copy, /// Use global context. args.table_id, args.columns, args.constraints, + partition_by, + order_by, + context_copy, /// Use global context. args.comment, format_settings, args.mode, configuration->getCatalog(context, args.table_id), args.query.if_not_exists, - /* is_datalake_query*/ false, - /* distributed_processing */ false, - partition_by, - order_by); + /* is_datalake_query */ false, + /* is_table_function */ false, + /* lazy_init */ false); } #endif @@ -1128,9 +1140,8 @@ void registerStorageIceberg(StorageFactory & factory) } } else -#if USE_AWS_S3 - configuration = std::make_shared(storage_settings); -#endif + configuration = std::make_shared(storage_settings); + if (configuration == nullptr) { throw Exception(ErrorCodes::BAD_ARGUMENTS, "This storage configuration is not available at this build"); @@ -2179,7 +2190,7 @@ Data types supported in Paimon partition keys: void registerStorageDeltaLake(StorageFactory & factory); void registerStorageDeltaLake(StorageFactory & factory) { -#if USE_AWS_S3 +# if USE_AWS_S3 factory.registerStorage( DeltaLakeDefinition::storage_engine_name, [&](const StorageFactory::Arguments & args) diff --git a/src/Storages/ObjectStorage/tests/gtest_rendezvous_hashing.cpp b/src/Storages/ObjectStorage/tests/gtest_rendezvous_hashing.cpp index b10d1825048e..1c4216c61329 100644 --- a/src/Storages/ObjectStorage/tests/gtest_rendezvous_hashing.cpp +++ b/src/Storages/ObjectStorage/tests/gtest_rendezvous_hashing.cpp @@ -126,7 +126,7 @@ TEST(RendezvousHashing, SingleNode) { auto iterator = makeIterator(); std::vector replicas = {"replica0", "replica1", "replica2", "replica3"}; - StorageObjectStorageStableTaskDistributor distributor(iterator, std::move(replicas), false); + StorageObjectStorageStableTaskDistributor distributor(iterator, std::move(replicas), false, 0, false); std::vector paths; ASSERT_TRUE(extractNForReplica(distributor, paths, 0, 10)); ASSERT_TRUE(checkHead(paths, {6})); @@ -135,7 +135,7 @@ TEST(RendezvousHashing, SingleNode) { auto iterator = makeIterator(); std::vector replicas = {"replica0", "replica1", "replica2", "replica3"}; - StorageObjectStorageStableTaskDistributor distributor(iterator, std::move(replicas), false); + StorageObjectStorageStableTaskDistributor distributor(iterator, std::move(replicas), false, 0, false); std::vector paths; ASSERT_TRUE(extractNForReplica(distributor, paths, 1, 10)); ASSERT_TRUE(checkHead(paths, {0, 2, 4})); @@ -144,7 +144,7 @@ TEST(RendezvousHashing, SingleNode) { auto iterator = makeIterator(); std::vector replicas = {"replica0", "replica1", "replica2", "replica3"}; - StorageObjectStorageStableTaskDistributor distributor(iterator, std::move(replicas), false); + StorageObjectStorageStableTaskDistributor distributor(iterator, std::move(replicas), false, 0, false); std::vector paths; ASSERT_TRUE(extractNForReplica(distributor, paths, 2, 10)); ASSERT_TRUE(checkHead(paths, {1, 5, 7, 8})); @@ -153,7 +153,7 @@ TEST(RendezvousHashing, SingleNode) { auto iterator = makeIterator(); std::vector replicas = {"replica0", "replica1", "replica2", "replica3"}; - StorageObjectStorageStableTaskDistributor distributor(iterator, std::move(replicas), false); + StorageObjectStorageStableTaskDistributor distributor(iterator, std::move(replicas), false, 0, false); std::vector paths; ASSERT_TRUE(extractNForReplica(distributor, paths, 3, 10)); ASSERT_TRUE(checkHead(paths, {3, 9})); @@ -164,7 +164,7 @@ TEST(RendezvousHashing, MultipleNodes) { auto iterator = makeIterator(); std::vector replicas = {"replica0", "replica1", "replica2", "replica3"}; - StorageObjectStorageStableTaskDistributor distributor(iterator, std::move(replicas), false); + StorageObjectStorageStableTaskDistributor distributor(iterator, std::move(replicas), false, 0, false); { std::vector paths; @@ -196,7 +196,7 @@ TEST(RendezvousHashing, SingleNodeReducedCluster) { auto iterator = makeIterator(); std::vector replicas = {"replica2", "replica1"}; - StorageObjectStorageStableTaskDistributor distributor(iterator, std::move(replicas), false); + StorageObjectStorageStableTaskDistributor distributor(iterator, std::move(replicas), false, 0, false); std::vector paths; ASSERT_TRUE(extractNForReplica(distributor, paths, 0, 10)); ASSERT_TRUE(checkHead(paths, {1, 5, 6, 7, 8, 9})); @@ -205,7 +205,7 @@ TEST(RendezvousHashing, SingleNodeReducedCluster) { auto iterator = makeIterator(); std::vector replicas = {"replica2", "replica1"}; - StorageObjectStorageStableTaskDistributor distributor(iterator, std::move(replicas), false); + StorageObjectStorageStableTaskDistributor distributor(iterator, std::move(replicas), false, 0, false); std::vector paths; ASSERT_TRUE(extractNForReplica(distributor, paths, 1, 10)); ASSERT_TRUE(checkHead(paths, {0, 2, 3, 4})); @@ -216,7 +216,7 @@ TEST(RendezvousHashing, MultipleNodesReducedCluster) { auto iterator = makeIterator(); std::vector replicas = {"replica2", "replica1"}; - StorageObjectStorageStableTaskDistributor distributor(iterator, std::move(replicas), false); + StorageObjectStorageStableTaskDistributor distributor(iterator, std::move(replicas), false, 0, false); { std::vector paths; @@ -235,7 +235,7 @@ TEST(RendezvousHashing, MultipleNodesReducedClusterOneByOne) { auto iterator = makeIterator(); std::vector replicas = {"replica2", "replica1"}; - StorageObjectStorageStableTaskDistributor distributor(iterator, std::move(replicas), false); + StorageObjectStorageStableTaskDistributor distributor(iterator, std::move(replicas), false, 0, false); std::vector paths0; std::vector paths1; @@ -266,7 +266,7 @@ TEST(RendezvousHashing, DoesNotDeduplicateSamePathFromDifferentReadSources) auto iterator = std::make_shared(paths); std::vector replicas = {"replica0", "replica1", "replica2", "replica3"}; - StorageObjectStorageStableTaskDistributor distributor(iterator, std::move(replicas), false); + StorageObjectStorageStableTaskDistributor distributor(iterator, std::move(replicas), false, 0, false); std::vector> read_source_indices; ASSERT_TRUE(extractNForReplica(distributor, read_source_indices, 0, 2)); diff --git a/src/Storages/ObjectStorageQueue/StorageObjectStorageQueue.cpp b/src/Storages/ObjectStorageQueue/StorageObjectStorageQueue.cpp index 093276d481ac..22e9b05c5ed9 100644 --- a/src/Storages/ObjectStorageQueue/StorageObjectStorageQueue.cpp +++ b/src/Storages/ObjectStorageQueue/StorageObjectStorageQueue.cpp @@ -375,7 +375,7 @@ StorageObjectStorageQueue::StorageObjectStorageQueue( ColumnsDescription columns{columns_}; std::string sample_path; - resolveSchemaAndFormat(columns, configuration->format, object_storage, configuration, format_settings, sample_path, context_); + resolveSchemaAndFormat(columns, object_storage, configuration, format_settings, sample_path, context_); configuration->check(context_); bool is_path_with_hive_partitioning = false; @@ -432,7 +432,7 @@ StorageObjectStorageQueue::StorageObjectStorageQueue( zk_path, *queue_settings_, storage_metadata.getColumns(), - configuration_->format, + configuration_->getFormat(), context_, is_attach, log); @@ -613,7 +613,7 @@ void StorageObjectStorageQueue::renameInMemory(const StorageID & new_table_id) bool StorageObjectStorageQueue::supportsSubsetOfColumns(const ContextPtr & context_) const { - return FormatFactory::instance().checkIfFormatSupportsSubsetOfColumns(configuration->format, context_, format_settings); + return FormatFactory::instance().checkIfFormatSupportsSubsetOfColumns(configuration->getFormat(), context_, format_settings); } class ReadFromObjectStorageQueue : public SourceStepWithFilter diff --git a/src/Storages/ObjectStorageQueue/StorageObjectStorageQueue.h b/src/Storages/ObjectStorageQueue/StorageObjectStorageQueue.h index b82a747e157c..e51000bd26cb 100644 --- a/src/Storages/ObjectStorageQueue/StorageObjectStorageQueue.h +++ b/src/Storages/ObjectStorageQueue/StorageObjectStorageQueue.h @@ -62,7 +62,7 @@ class StorageObjectStorageQueue : public IStreamingStorage, WithContext void renameInMemory(const StorageID & new_table_id) override; - const auto & getFormatName() const { return configuration->format; } + const auto & getFormatName() const { return configuration->getFormat(); } const fs::path & getZooKeeperPath() const { return zk_path; } diff --git a/src/Storages/ObjectStorageQueue/registerQueueStorage.cpp b/src/Storages/ObjectStorageQueue/registerQueueStorage.cpp index 0ec97a2f1402..a5f6ad2d4a0f 100644 --- a/src/Storages/ObjectStorageQueue/registerQueueStorage.cpp +++ b/src/Storages/ObjectStorageQueue/registerQueueStorage.cpp @@ -53,7 +53,7 @@ StoragePtr createQueueStorage(const StorageFactory::Arguments & args) auto configuration = std::make_shared(); /// Parse with the create context so a `SETTINGS s3_allow_server_credentials_in_user_queries = 1` on the /// `CREATE` is honored (see `StorageS3Configuration::fromAST`); the processing context stays global below. - StorageObjectStorageConfiguration::initialize(*configuration, args.engine_args, args.getLocalContext(), false, &args.table_id); + configuration->initialize(args.engine_args, args.getLocalContext(), false, &args.table_id); // Use format settings from global server context + settings from // the SETTINGS clause of the create query. Settings from current diff --git a/src/Storages/PartitionCommands.cpp b/src/Storages/PartitionCommands.cpp index 593f4d895aad..c4d038f925c1 100644 --- a/src/Storages/PartitionCommands.cpp +++ b/src/Storages/PartitionCommands.cpp @@ -131,6 +131,37 @@ std::optional PartitionCommand::parse(const ASTAlterCommand * res.with_name = command_ast->with_name; return res; } + if (command_ast->type == ASTAlterCommand::EXPORT_PART) + { + PartitionCommand res; + res.type = EXPORT_PART; + res.partition = command_ast->partition->clone(); + res.part = command_ast->part; + res.to_database = command_ast->to_database; + res.to_table = command_ast->to_table; + if (command_ast->to_table_function) + { + res.to_table_function = command_ast->to_table_function->ptr(); + if (command_ast->partition_by_expr) + res.partition_by_expr = command_ast->partition_by_expr->clone(); + } + return res; + } + if (command_ast->type == ASTAlterCommand::EXPORT_PARTITION) + { + PartitionCommand res; + res.type = EXPORT_PARTITION; + res.partition = command_ast->partition->clone(); + res.to_database = command_ast->to_database; + res.to_table = command_ast->to_table; + if (command_ast->to_table_function) + { + res.to_table_function = command_ast->to_table_function->ptr(); + if (command_ast->partition_by_expr) + res.partition_by_expr = command_ast->partition_by_expr->clone(); + } + return res; + } return {}; } @@ -172,6 +203,10 @@ std::string PartitionCommand::typeToString() const return "UNFREEZE ALL"; case PartitionCommand::Type::REPLACE_PARTITION: return "REPLACE PARTITION"; + case PartitionCommand::Type::EXPORT_PART: + return "EXPORT PART"; + case PartitionCommand::Type::EXPORT_PARTITION: + return "EXPORT PARTITION"; default: throw Exception(ErrorCodes::LOGICAL_ERROR, "Uninitialized partition command"); } diff --git a/src/Storages/PartitionCommands.h b/src/Storages/PartitionCommands.h index 399a4b64c478..9c476d3c8e4f 100644 --- a/src/Storages/PartitionCommands.h +++ b/src/Storages/PartitionCommands.h @@ -32,6 +32,8 @@ struct PartitionCommand UNFREEZE_ALL_PARTITIONS, UNFREEZE_PARTITION, REPLACE_PARTITION, + EXPORT_PART, + EXPORT_PARTITION, }; Type type = UNKNOWN; @@ -49,10 +51,14 @@ struct PartitionCommand String from_table; bool replace = true; - /// For MOVE PARTITION + /// For MOVE PARTITION and EXPORT PART and EXPORT PARTITION String to_database; String to_table; + /// For EXPORT PART and EXPORT PARTITION with table functions + ASTPtr to_table_function; + ASTPtr partition_by_expr; + /// For FETCH PARTITION - path in ZK to the shard, from which to download the partition. String from_path; diff --git a/src/Storages/PartitionedSink.cpp b/src/Storages/PartitionedSink.cpp index 8af531716d4e..fbc41f6be4d3 100644 --- a/src/Storages/PartitionedSink.cpp +++ b/src/Storages/PartitionedSink.cpp @@ -26,10 +26,12 @@ namespace ErrorCodes PartitionedSink::PartitionedSink( std::shared_ptr partition_strategy_, + std::shared_ptr sink_creator_, ContextPtr context_, SharedHeader source_header_) : SinkToStorage(source_header_) , partition_strategy(partition_strategy_) + , sink_creator(sink_creator_) , context(context_) , source_header(source_header_) { @@ -41,7 +43,7 @@ SinkPtr PartitionedSink::getSinkForPartitionKey(std::string_view partition_key) auto it = partition_id_to_sink.find(partition_key); if (it == partition_id_to_sink.end()) { - auto sink = createSinkForPartition(std::string{partition_key}); + auto sink = sink_creator->createSinkForPartition(std::string{partition_key}); std::tie(it, std::ignore) = partition_id_to_sink.emplace(partition_key, sink); } diff --git a/src/Storages/PartitionedSink.h b/src/Storages/PartitionedSink.h index af2b88baad27..1f612076db92 100644 --- a/src/Storages/PartitionedSink.h +++ b/src/Storages/PartitionedSink.h @@ -16,10 +16,17 @@ namespace DB class PartitionedSink : public SinkToStorage { public: + struct SinkCreator + { + virtual ~SinkCreator() = default; + virtual SinkPtr createSinkForPartition(const String & partition_id) = 0; + }; + static constexpr auto PARTITION_ID_WILDCARD = "{_partition_id}"; PartitionedSink( std::shared_ptr partition_strategy_, + std::shared_ptr sink_creator_, ContextPtr context_, SharedHeader source_header_); @@ -33,16 +40,15 @@ class PartitionedSink : public SinkToStorage void onFinish() override; - virtual SinkPtr createSinkForPartition(const String & partition_id) = 0; - static void validatePartitionKey(const String & str, bool allow_slash); static String replaceWildcards(const String & haystack, const String & partition_id); + protected: std::shared_ptr partition_strategy; - private: + std::shared_ptr sink_creator; ContextPtr context; SharedHeader source_header; diff --git a/src/Storages/StorageDistributed.cpp b/src/Storages/StorageDistributed.cpp index 645ab311957d..0e4feb8d5e38 100644 --- a/src/Storages/StorageDistributed.cpp +++ b/src/Storages/StorageDistributed.cpp @@ -810,53 +810,6 @@ std::optional StorageDistributed::getOptimizedQueryP namespace { -class RewriteInToGlobalInVisitor : public InDepthQueryTreeVisitorWithContext -{ -public: - using Base = InDepthQueryTreeVisitorWithContext; - using Base::Base; - - void enterImpl(QueryTreeNodePtr & node) - { - if (auto * function_node = node->as(); function_node && isNameOfLocalInFunction(function_node->getFunctionName())) - { - auto * query = function_node->getArguments().getNodes()[1]->as(); - if (!query) - return; - bool no_replace = true; - for (const auto & table_node : extractTableExpressions(query->getJoinTreeNodeTyped(), false, true)) - { - const StorageDistributed * storage_distributed = nullptr; - if (const TableNode * table_node_typed = table_node->as()) - storage_distributed = typeid_cast(table_node_typed->getStorage().get()); - else if (const TableFunctionNode * table_function_node_typed = table_node->as()) - storage_distributed = typeid_cast(table_function_node_typed->getStorage().get()); - - if (!storage_distributed) - { - no_replace = false; - break; - } - } - if (no_replace) - return; - - auto result_function = std::make_shared(getGlobalInFunctionNameForLocalInFunctionName(function_node->getFunctionName())); - result_function->getArguments().getNodes() = std::move(function_node->getArguments().getNodes()); - resolveOrdinaryFunctionNodeByName(*result_function, result_function->getFunctionName(), getContext()); - node = result_function; - } - } - - static bool needChildVisit(QueryTreeNodePtr & parent, QueryTreeNodePtr &) - { - if (auto * function_node = parent->as(); function_node && function_node->getFunctionName().starts_with("global")) - return false; - - return true; - } -}; - bool rewriteJoinToGlobalJoinIfNeeded(QueryTreeNodePtr join_tree) { bool rewrite = false; @@ -912,6 +865,7 @@ QueryTreeNodePtr buildQueryTreeDistributed(SelectQueryInfo & query_info, auto table_function_node = std::make_shared(remote_table_function_node.getFunctionName()); table_function_node->getArgumentsNode() = remote_table_function_node.getArgumentsNode(); + table_function_node->setSettingsChanges(remote_table_function_node.getSettingsChanges()); if (table_expression_modifiers) table_function_node->setTableExpressionModifiers(*table_expression_modifiers); @@ -967,10 +921,7 @@ QueryTreeNodePtr buildQueryTreeDistributed(SelectQueryInfo & query_info, { auto & query_node = query_tree_to_modify->as(); if (query_node.hasWhere()) - { - RewriteInToGlobalInVisitor visitor(query_context); - visitor.visit(query_node.getWhere()); - } + rewriteInToGlobalIn(query_node.getWhere(), query_context); rewriteJoinToGlobalJoinIfNeeded(query_node.getJoinTreeNode()); } @@ -1546,7 +1497,8 @@ std::optional StorageDistributed::distributedWrite(const ASTInser } if (auto src_storage_cluster = std::dynamic_pointer_cast(src_storage)) { - return distributedWriteFromClusterStorage(*src_storage_cluster, query, local_context); + if (!src_storage_cluster->getClusterName(local_context).empty()) + return distributedWriteFromClusterStorage(*src_storage_cluster, query, local_context); } return {}; diff --git a/src/Storages/StorageFile.cpp b/src/Storages/StorageFile.cpp index 79e97e33900f..61cea1c2f7e5 100644 --- a/src/Storages/StorageFile.cpp +++ b/src/Storages/StorageFile.cpp @@ -2780,7 +2780,7 @@ class StorageFileSink final : public SinkToStorage, WithContext std::unique_lock lock; }; -class PartitionedStorageFileSink : public PartitionedSink +class PartitionedStorageFileSink : public PartitionedSink::SinkCreator { public: PartitionedStorageFileSink( @@ -2795,7 +2795,7 @@ class PartitionedStorageFileSink : public PartitionedSink const String format_name_, ContextPtr context_, int flags_) - : PartitionedSink(partition_strategy_, context_, std::make_shared(metadata_snapshot_->getSampleBlock())) + : partition_strategy(partition_strategy_) , path(path_) , metadata_snapshot(metadata_snapshot_) , table_name_for_log(table_name_for_log_) @@ -2811,11 +2811,12 @@ class PartitionedStorageFileSink : public PartitionedSink SinkPtr createSinkForPartition(const String & partition_id) override { - std::string filepath = partition_strategy->getPathForWrite(path, partition_id); + const auto file_path_generator = std::make_shared(path); + std::string filepath = file_path_generator->getPathForWrite(partition_id); fs::create_directories(fs::path(filepath).parent_path()); - validatePartitionKey(filepath, true); + PartitionedSink::validatePartitionKey(filepath, true); checkCreationIsAllowed(context, context->getUserFilesPath(), filepath, /*can_be_directory=*/ true); return std::make_shared( metadata_snapshot, @@ -2832,6 +2833,7 @@ class PartitionedStorageFileSink : public PartitionedSink } private: + std::shared_ptr partition_strategy; const String path; StorageMetadataPtr metadata_snapshot; String table_name_for_log; @@ -2883,7 +2885,7 @@ SinkToStoragePtr StorageFile::write( has_wildcards, /* partition_columns_in_data_file */true); - return std::make_shared( + auto sink_creator = std::make_shared( partition_strategy, metadata_snapshot, getStorageID().getNameForLogs(), @@ -2895,6 +2897,13 @@ SinkToStoragePtr StorageFile::write( format_name, context, flags); + + return std::make_shared( + partition_strategy, + sink_creator, + context, + std::make_shared(metadata_snapshot->getSampleBlock()) + ); } String path; @@ -2920,6 +2929,7 @@ SinkToStoragePtr StorageFile::write( String new_path; do { + new_path = path.substr(0, pos) + "." + std::to_string(index) + (pos == std::string::npos ? "" : path.substr(pos)); ++index; } diff --git a/src/Storages/StorageFileCluster.cpp b/src/Storages/StorageFileCluster.cpp index d8b6a06f2841..214c1bcaef2a 100644 --- a/src/Storages/StorageFileCluster.cpp +++ b/src/Storages/StorageFileCluster.cpp @@ -72,16 +72,23 @@ StorageFileCluster::StorageFileCluster( auto & storage_columns = storage_metadata.columns; + const auto sample_path = paths.empty() ? "" : paths.front(); + /// Not grabbing the file_columns because it is not necessary to do it here. std::tie(hive_partition_columns_to_read_from_file_path, std::ignore) = HivePartitioningUtils::setupHivePartitioningForFileURLLikeStorage( storage_columns, - paths.empty() ? "" : paths.front(), + sample_path, columns_.empty(), std::nullopt, context); storage_metadata.setConstraints(constraints_); - storage_metadata.setVirtuals(VirtualColumnUtils::getVirtualsForFileLikeStorage(storage_metadata.columns, context)); + storage_metadata.setVirtuals(VirtualColumnUtils::getVirtualsForFileLikeStorage( + storage_metadata.columns, + context, + std::nullopt, + PartitionStrategyFactory::StrategyType::NONE, + sample_path)); setInMemoryMetadata(storage_metadata); } @@ -105,7 +112,11 @@ void StorageFileCluster::updateBeforeRead(const ContextPtr & context) checkWriteAccessIfFilesAreRenamed(context); } -void StorageFileCluster::updateQueryToSendIfNeeded(DB::ASTPtr & query, const StorageSnapshotPtr & storage_snapshot, const DB::ContextPtr & context) +void StorageFileCluster::updateQueryToSendIfNeeded( + DB::ASTPtr & query, + const StorageSnapshotPtr & storage_snapshot, + const DB::ContextPtr & context, + bool /*make_cluster_function*/) { auto * table_function = extractTableFunctionFromSelectQuery(query); if (!table_function) @@ -119,6 +130,38 @@ void StorageFileCluster::updateQueryToSendIfNeeded(DB::ASTPtr & query, const Sto ); } +class FileTaskIterator : public TaskIterator +{ +public: + FileTaskIterator(const Strings & files, + std::optional archive_info, + const ActionsDAG::Node * predicate, + const NamesAndTypesList & virtual_columns, + const NamesAndTypesList & hive_partition_columns_to_read_from_file_path, + const ContextPtr & context, + bool distributed_processing = false) + : iterator(files + , archive_info + , predicate + , virtual_columns + , hive_partition_columns_to_read_from_file_path + , context + , distributed_processing) {} + + ~FileTaskIterator() override = default; + + ClusterFunctionReadTaskResponsePtr operator()(size_t /* number_of_current_replica */) const override + { + auto file = iterator.next(); + if (file.empty()) + return std::make_shared(); + return std::make_shared(std::move(file)); + } + +private: + mutable StorageFileSource::FilesIterator iterator; +}; + RemoteQueryExecutor::Extension StorageFileCluster::getTaskIteratorExtension( const ActionsDAG::Node * predicate, const ActionsDAG * /* filter */, const ContextPtr & context, ClusterPtr, StorageMetadataPtr metadata) const { @@ -126,15 +169,14 @@ RemoteQueryExecutor::Extension StorageFileCluster::getTaskIteratorExtension( /// through `IStorageCluster::read`, so this is the one place every path shares. checkWriteAccessIfFilesAreRenamed(context); - auto iterator = std::make_shared(paths, std::nullopt, predicate, metadata->virtuals.getSampleBlock(VirtualsKind::All, VirtualsMaterializationPlace::Reader).getNamesAndTypesList(), hive_partition_columns_to_read_from_file_path, context); - auto next_callback = [iter = std::move(iterator)](size_t) mutable -> ClusterFunctionReadTaskResponsePtr - { - auto file = iter->next(); - if (file.empty()) - return std::make_shared(); - return std::make_shared(std::move(file)); - }; - auto callback = std::make_shared(std::move(next_callback)); + auto callback = std::make_shared( + paths, + std::nullopt, + predicate, + metadata->virtuals.getSampleBlock(VirtualsKind::All, VirtualsMaterializationPlace::Reader).getNamesAndTypesList(), + getHivePartitionColumnsWithoutVirtuals(metadata), + context + ); return RemoteQueryExecutor::Extension{.task_iterator = std::move(callback)}; } diff --git a/src/Storages/StorageFileCluster.h b/src/Storages/StorageFileCluster.h index fab05dd2f645..d07cb6ac4d5a 100644 --- a/src/Storages/StorageFileCluster.h +++ b/src/Storages/StorageFileCluster.h @@ -36,13 +36,16 @@ class StorageFileCluster : public IStorageCluster StorageMetadataPtr) const override; private: - void updateQueryToSendIfNeeded(ASTPtr & query, const StorageSnapshotPtr & storage_snapshot, const ContextPtr & context) override; + void updateQueryToSendIfNeeded( + ASTPtr & query, + const StorageSnapshotPtr & storage_snapshot, + const ContextPtr & context, + bool /*make_cluster_function*/) override; void updateBeforeRead(const ContextPtr & context) override; Strings paths; String filename; String format_name; - NamesAndTypesList hive_partition_columns_to_read_from_file_path; }; } diff --git a/src/Storages/StorageMergeTree.cpp b/src/Storages/StorageMergeTree.cpp index f2afa34d2958..66650b8aeb23 100644 --- a/src/Storages/StorageMergeTree.cpp +++ b/src/Storages/StorageMergeTree.cpp @@ -159,6 +159,7 @@ namespace ErrorCodes extern const int PART_IS_TEMPORARILY_LOCKED; extern const int FAULT_INJECTED; extern const int INVALID_TRANSACTION; + extern const int INCOMPATIBLE_COLUMNS; } namespace ActionLocks @@ -260,7 +261,7 @@ void StorageMergeTree::startup() cleanup_thread.start(); background_operations_assignee.start(); background_streaming_assignee.start(); - startBackgroundMovesIfNeeded(); + startBackgroundMoves(); startOutdatedAndUnexpectedDataPartsLoadingTask(); startStatisticsCache(); } @@ -325,6 +326,11 @@ void StorageMergeTree::shutdown(bool) if (deduplication_log) deduplication_log->shutdown(); + + { + std::lock_guard lock(export_manifests_mutex); + export_manifests.clear(); + } } @@ -3686,12 +3692,6 @@ MutationCounters StorageMergeTree::getMutationCounters() const return mutation_counters; } -void StorageMergeTree::startBackgroundMovesIfNeeded() -{ - if (areBackgroundMovesNeeded()) - background_moves_assignee.start(); -} - std::unique_ptr StorageMergeTree::getDefaultSettings() const { return std::make_unique(getContext()->getMergeTreeSettings()); diff --git a/src/Storages/StorageMergeTree.h b/src/Storages/StorageMergeTree.h index 5cf26060411f..80a0b10c3ed3 100644 --- a/src/Storages/StorageMergeTree.h +++ b/src/Storages/StorageMergeTree.h @@ -326,8 +326,6 @@ class StorageMergeTree final : public MergeTreeData std::unique_ptr fillNewPartName(MutableDataPartPtr & part, DataPartsLock & lock); std::unique_ptr fillNewPartNameAndResetLevel(MutableDataPartPtr & part, DataPartsLock & lock); - void startBackgroundMovesIfNeeded() override; - BackupEntries backupMutations(UInt64 version, const String & data_path_in_backup) const; /// Attaches restored parts to the storage. diff --git a/src/Storages/StorageReplicatedMergeTree.cpp b/src/Storages/StorageReplicatedMergeTree.cpp index 1af170cec475..17c5cd176f43 100644 --- a/src/Storages/StorageReplicatedMergeTree.cpp +++ b/src/Storages/StorageReplicatedMergeTree.cpp @@ -7,6 +7,7 @@ #include #include +#include "Common/ZooKeeper/IKeeper.h" #include #include #include @@ -30,6 +31,7 @@ #include #include +#include #include #include #include @@ -74,12 +76,18 @@ #include #include #include +#include +#include +#include #include #include +#include #include #include #include #include +#include +#include #include #include @@ -128,6 +136,14 @@ #include #include +#include "Interpreters/StorageID.h" +#include "QueryPipeline/QueryPlanResourceHolder.h" +#include "Storages/ExportReplicatedMergeTreePartitionManifest.h" +#include "Storages/ExportReplicatedMergeTreePartitionTaskEntry.h" +#include +#include +#include +#include #include #include @@ -164,6 +180,15 @@ namespace ProfileEvents extern const Event ZooKeeperWatchTriggeredReplicatedMergeTreeLeaderElection; extern const Event ZooKeeperWatchTriggeredReplicatedMergeTreeReplicaSync; extern const Event ZooKeeperWatchTriggeredReplicatedMergeTreeMutations; + extern const Event ExportPartitionZooKeeperRequests; + extern const Event ExportPartitionZooKeeperGet; + extern const Event ExportPartitionZooKeeperGetChildren; + extern const Event ExportPartitionZooKeeperCreate; + extern const Event ExportPartitionZooKeeperSet; + extern const Event ExportPartitionZooKeeperRemove; + extern const Event ExportPartitionZooKeeperRemoveRecursive; + extern const Event ExportPartitionZooKeeperMulti; + extern const Event ExportPartitionZooKeeperExists; } namespace CurrentMetrics @@ -201,6 +226,32 @@ namespace Setting extern const SettingsInt64 replication_wait_for_inactive_replica_timeout; extern const SettingsUInt64 select_sequential_consistency; extern const SettingsBool update_sequential_consistency; + extern const SettingsBool allow_experimental_export_merge_tree_part; + extern const SettingsBool export_merge_tree_partition_force_export; + extern const SettingsUInt64 export_merge_tree_partition_retry_initial_backoff_seconds; + extern const SettingsUInt64 export_merge_tree_partition_retry_max_backoff_seconds; + extern const SettingsUInt64 export_merge_tree_partition_task_timeout_seconds; + extern const SettingsBool output_format_parallel_formatting; + extern const SettingsBool output_format_parquet_parallel_encoding; + extern const SettingsParquetCompression output_format_parquet_compression_method; + extern const SettingsUInt64 output_format_compression_level; + extern const SettingsUInt64 output_format_parquet_row_group_size; + extern const SettingsUInt64 output_format_parquet_row_group_size_bytes; + extern const SettingsMaxThreads max_threads; + extern const SettingsMergeTreePartExportFileAlreadyExistsPolicy export_merge_tree_part_file_already_exists_policy; + extern const SettingsUInt64 export_merge_tree_part_max_bytes_per_file; + extern const SettingsUInt64 export_merge_tree_part_max_rows_per_file; + extern const SettingsBool export_merge_tree_part_throw_on_pending_mutations; + extern const SettingsBool export_merge_tree_part_throw_on_pending_patch_parts; + extern const SettingsBool export_merge_tree_part_allow_lossy_cast; + extern const SettingsMergeTreePartExportSchemaMismatchMode export_merge_tree_part_schema_mismatch_mode; + extern const SettingsExportPartitionAllOnError export_merge_tree_partition_all_on_error; + extern const SettingsString export_merge_tree_part_filename_pattern; + extern const SettingsBool write_full_path_in_iceberg_metadata; + extern const SettingsBool allow_insert_into_iceberg; + extern const SettingsUInt64 iceberg_insert_max_bytes_in_data_file; + extern const SettingsUInt64 iceberg_insert_max_rows_in_data_file; + extern const SettingsTimezone iceberg_partition_timezone; } @@ -317,6 +368,15 @@ namespace ErrorCodes extern const int FAULT_INJECTED; extern const int CANNOT_FORGET_PARTITION; extern const int TIMEOUT_EXCEEDED; + extern const int INVALID_SETTING_VALUE; + extern const int PENDING_MUTATIONS_NOT_ALLOWED; + extern const int EXPORT_PARTITION_ALREADY_EXPORTED; + extern const int PARTITION_EXPORT_FAILED; +} + +namespace ServerSetting +{ + extern const ServerSettingsBool allow_experimental_export_merge_tree_partition; } namespace ActionLocks @@ -451,6 +511,7 @@ StorageReplicatedMergeTree::StorageReplicatedMergeTree( , merge_strategy_picker(*this) , queue(*this, merge_strategy_picker) , fetcher(*this) + , export_partition_manifests(std::make_unique()) , cleanup_thread(*this) , deduplication_hashes_cache(*this, "deduplication_hashes") , part_check_thread(*this) @@ -501,6 +562,31 @@ StorageReplicatedMergeTree::StorageReplicatedMergeTree( /// Will be activated by restarting thread. mutations_finalizing_task->deactivate(); + if (getContext()->getServerSettings()[ServerSetting::allow_experimental_export_merge_tree_partition]) + { + export_merge_tree_partition_manifest_updater = std::make_shared(*this); + + export_merge_tree_partition_task_scheduler = std::make_shared(*this); + + export_merge_tree_partition_updating_task = getContext()->getSchedulePool()->createTask( + getStorageID(), getStorageID().getFullTableName() + " (StorageReplicatedMergeTree::export_merge_tree_partition_updating_task)", [this] { exportMergeTreePartitionUpdatingTask(); }); + + export_merge_tree_partition_updating_task->deactivate(); + + export_merge_tree_partition_status_handling_task = getContext()->getSchedulePool()->createTask( + getStorageID(), getStorageID().getFullTableName() + " (StorageReplicatedMergeTree::export_merge_tree_partition_status_handling_task)", [this] { exportMergeTreePartitionStatusHandlingTask(); }); + + export_merge_tree_partition_status_handling_task->deactivate(); + + export_merge_tree_partition_watch_callback = export_merge_tree_partition_updating_task->getWatchCallback(); + + export_merge_tree_partition_select_task = getContext()->getSchedulePool()->createTask( + getStorageID(), getStorageID().getFullTableName() + " (StorageReplicatedMergeTree::export_merge_tree_partition_select_task)", [this] { selectPartsToExport(); }); + + export_merge_tree_partition_select_task->deactivate(); + } + + bool has_zookeeper = getContext()->hasZooKeeper() || getContext()->hasAuxiliaryZooKeeper(zookeeper_info.zookeeper_name); auto component_guard = Coordination::setCurrentComponent("StorageReplicatedMergeTree::StorageReplicatedMergeTree"); if (has_zookeeper) @@ -978,6 +1064,7 @@ void StorageReplicatedMergeTree::createNewZooKeeperNodesAttempt() const futures.push_back(zookeeper->asyncTryCreateNoThrow(zookeeper_path + "/quorum/last_part", String(), zkutil::CreateMode::Persistent)); futures.push_back(zookeeper->asyncTryCreateNoThrow(zookeeper_path + "/quorum/failed_parts", String(), zkutil::CreateMode::Persistent)); futures.push_back(zookeeper->asyncTryCreateNoThrow(zookeeper_path + "/mutations", String(), zkutil::CreateMode::Persistent)); + futures.push_back(zookeeper->asyncTryCreateNoThrow(zookeeper_path + "/exports", String(), zkutil::CreateMode::Persistent)); futures.push_back(zookeeper->asyncTryCreateNoThrow(zookeeper_path + "/quorum/parallel", String(), zkutil::CreateMode::Persistent)); @@ -1150,6 +1237,8 @@ bool StorageReplicatedMergeTree::createTableIfNotExistsAttempt(const StorageMeta zkutil::CreateMode::Persistent)); ops.emplace_back(zkutil::makeCreateRequest(zookeeper_path + "/mutations", "", zkutil::CreateMode::Persistent)); + ops.emplace_back(zkutil::makeCreateRequest(zookeeper_path + "/exports", "", + zkutil::CreateMode::Persistent)); /// And create first replica atomically. See also "createReplica" method that is used to create not the first replicas. @@ -4691,6 +4780,100 @@ void StorageReplicatedMergeTree::mutationsFinalizingTask() } } +void StorageReplicatedMergeTree::exportMergeTreePartitionUpdatingTask() +{ + auto component_guard = Coordination::setCurrentComponent("StorageReplicatedMergeTree::exportMergeTreePartitionUpdatingTask"); + try + { + export_merge_tree_partition_manifest_updater->poll(); + } + catch (const Coordination::Exception & e) + { + tryLogCurrentException(log, __PRETTY_FUNCTION__); + if (e.code == Coordination::Error::ZSESSIONEXPIRED) + { + LOG_DEBUG(log, "Export partition updating task: ZooKeeper session expired, waking up restarting thread"); + restarting_thread.wakeup(); + return; + } + } + catch (...) + { + tryLogCurrentException(log, __PRETTY_FUNCTION__); + } + + export_merge_tree_partition_updating_task->scheduleAfter(30 * 1000); +} + +void StorageReplicatedMergeTree::selectPartsToExport() +{ + auto component_guard = Coordination::setCurrentComponent("StorageReplicatedMergeTree::selectPartsToExport"); + + /// Default tick interval; may be shortened below if a part's back-off expires sooner. + static constexpr int64_t default_reschedule_ms = 1000 * 5; + int64_t reschedule_ms = default_reschedule_ms; + + try + { + if (parts_mover.moves_blocker.isCancelled()) + { + LOG_INFO(log, "Export partition select task: Moves are blocked, skipping"); + } + else + { + const auto earliest_backoff_retry = export_merge_tree_partition_task_scheduler->run(); + + /// If a part is only waiting on its back-off deadline and that deadline is sooner than + /// the default tick, wake up earlier so the retry is not delayed by up to a full tick. + if (earliest_backoff_retry) + { + const auto now = time(nullptr); + const int64_t until_ms = (static_cast(*earliest_backoff_retry) - static_cast(now)) * 1000; + reschedule_ms = std::clamp(until_ms, 0, default_reschedule_ms); + } + } + } + catch (...) + { + tryLogCurrentException(log, __PRETTY_FUNCTION__); + } + + export_merge_tree_partition_select_task->scheduleAfter(reschedule_ms); +} + +void StorageReplicatedMergeTree::exportMergeTreePartitionStatusHandlingTask() +{ + auto component_guard = Coordination::setCurrentComponent("StorageReplicatedMergeTree::exportMergeTreePartitionStatusHandlingTask"); + try + { + export_merge_tree_partition_manifest_updater->handleStatusChanges(); + } + catch (const Coordination::Exception & e) + { + tryLogCurrentException(log, __PRETTY_FUNCTION__); + if (e.code == Coordination::Error::ZSESSIONEXPIRED) + { + restarting_thread.wakeup(); + } + else + { + /// if an exception is thrown, we might have unprocessed status changes, so we need to schedule the task again + export_merge_tree_partition_status_handling_task->scheduleAfter(5000); + } + + return; + } + catch (...) + { + tryLogCurrentException(log, __PRETTY_FUNCTION__); + export_merge_tree_partition_status_handling_task->scheduleAfter(5000); + } +} + +std::vector StorageReplicatedMergeTree::getPartitionExportsInfo() const +{ + return export_merge_tree_partition_manifest_updater->getPartitionExportsInfo(); +} StorageReplicatedMergeTree::CreateMergeEntryResult StorageReplicatedMergeTree::createLogEntryToMergeParts( zkutil::ZooKeeperPtr & zookeeper, @@ -5995,7 +6178,7 @@ void StorageReplicatedMergeTree::startupImpl(bool from_attach_thread, const ZooK restarting_thread.start(true); }); - startBackgroundMovesIfNeeded(); + startBackgroundMoves(); part_moves_between_shards_orchestrator.start(); @@ -6094,6 +6277,13 @@ void StorageReplicatedMergeTree::partialShutdown() mutations_updating_task->deactivate(); mutations_finalizing_task->deactivate(); + if (getContext()->getServerSettings()[ServerSetting::allow_experimental_export_merge_tree_partition]) + { + export_merge_tree_partition_updating_task->deactivate(); + export_merge_tree_partition_select_task->deactivate(); + export_merge_tree_partition_status_handling_task->deactivate(); + } + cleanup_thread.stop(); deduplication_hashes_cache.stop(); part_check_thread.stop(); @@ -6166,6 +6356,14 @@ void StorageReplicatedMergeTree::shutdown(bool) /// Wait for all of them std::lock_guard lock(data_parts_exchange_ptr->rwlock); } + + export_partition_manifests.set(std::make_unique()); + + { + std::lock_guard lock(export_manifests_mutex); + export_manifests.clear(); + } + LOG_TRACE(log, "Shutdown finished"); } @@ -8418,6 +8616,358 @@ void StorageReplicatedMergeTree::fetchPartition( LOG_TRACE(log, "Fetch took {:.3f} sec. ({} tries)", watch.elapsedSeconds(), try_no); } +void StorageReplicatedMergeTree::exportPartitionToTable(const PartitionCommand & command, ContextPtr query_context) +{ + auto component_guard = Coordination::setCurrentComponent("StorageReplicatedMergeTree::exportPartitionToTable"); + if (!query_context->getServerSettings()[ServerSetting::allow_experimental_export_merge_tree_partition]) + { + throw Exception(ErrorCodes::SUPPORT_IS_DISABLED, + "Exporting merge tree partition is experimental. Set the server setting `allow_experimental_export_merge_tree_partition` to enable it (on all replicas).\n" + "If you are exporting to an Apache Iceberg table, you also need to enable the setting `allow_insert_into_iceberg` on the initiator (query, session or profile) - replicas inherit it from the scheduled task."); + } + + /// EXPORT PARTITION ALL: expand into one sub-call per active partition id. + /// Failure handling is controlled by `export_merge_tree_partition_all_on_error`. + if (const auto * partition_ast = command.partition->as(); partition_ast && partition_ast->all) + { + auto partition_id_set = getAllPartitionIds(); + if (partition_id_set.empty()) + throw Exception(ErrorCodes::BAD_ARGUMENTS, + "Table {} has no active partitions to export", + getStorageID().getNameForLogs()); + + /// Sort for deterministic ordering (so failure messages and tests are stable). + std::vector partition_ids(partition_id_set.begin(), partition_id_set.end()); + std::sort(partition_ids.begin(), partition_ids.end()); + + const auto & on_error_setting = query_context->getSettingsRef()[Setting::export_merge_tree_partition_all_on_error]; + const ExportPartitionAllOnError on_error = on_error_setting.value; + + LOG_INFO(log, "EXPORT PARTITION ALL: scheduling export for {} partitions, on_error={}", + partition_ids.size(), on_error_setting.toString()); + + std::vector> failures; /// (partition_id, message) + size_t skipped_conflicts = 0; + + for (const auto & partition_id : partition_ids) + { + PartitionCommand sub = command; + auto synthetic = make_intrusive(); + synthetic->setPartitionID(make_intrusive(partition_id)); + sub.partition = synthetic; + + try + { + exportPartitionToTable(sub, query_context); + } + catch (const Exception & e) + { + switch (on_error) + { + case ExportPartitionAllOnError::throw_first: + throw; + case ExportPartitionAllOnError::skip_conflicts: + if (e.code() == ErrorCodes::EXPORT_PARTITION_ALREADY_EXPORTED) + { + ++skipped_conflicts; + LOG_INFO(log, + "EXPORT PARTITION ALL: skipping partition {} (already exported / concurrent): {}", + partition_id, e.message()); + break; + } + throw; + case ExportPartitionAllOnError::collect: + LOG_WARNING(log, "EXPORT PARTITION ALL: partition {} failed: {}", + partition_id, e.message()); + failures.emplace_back(partition_id, e.message()); + break; + } + } + } + + if (!failures.empty()) + { + String aggregated = fmt::format( + "EXPORT PARTITION ALL: {}/{} partitions failed to schedule. Per-partition errors:", + failures.size(), partition_ids.size()); + for (const auto & [pid, msg] : failures) + aggregated += fmt::format("\n {}: {}", pid, msg); + throw Exception(ErrorCodes::PARTITION_EXPORT_FAILED, "{}", aggregated); + } + + if (skipped_conflicts > 0) + LOG_INFO(log, "EXPORT PARTITION ALL: skipped {} partitions due to existing exports", + skipped_conflicts); + + return; + } + + const auto dest_database = query_context->resolveDatabase(command.to_database); + const auto dest_table = command.to_table; + const auto dest_storage_id = StorageID(dest_database, dest_table); + auto dest_storage = DatabaseCatalog::instance().getTable({dest_database, dest_table}, query_context); + + if (dest_storage->getStorageID() == this->getStorageID()) + { + throw Exception(ErrorCodes::BAD_ARGUMENTS, "Exporting to the same table is not allowed"); + } + + if (!dest_storage->supportsImport(query_context)) + throw Exception(ErrorCodes::NOT_IMPLEMENTED, "Destination storage {} does not support MergeTree parts or uses unsupported partitioning", dest_storage->getName()); + + auto src_snapshot = getInMemoryMetadataPtr(query_context, false); + auto destination_snapshot = dest_storage->getInMemoryMetadataPtr(query_context, false); + + /// Positional CAST matching, like `INSERT INTO dest SELECT * FROM src`. + ExportPartitionUtils::verifyExportSchemaCastable( + src_snapshot, destination_snapshot, dest_storage->getStorageID(), query_context); + + zkutil::ZooKeeperPtr zookeeper = getZooKeeperAndAssertNotReadonly(); + + const String partition_id = getPartitionIDFromQuery(command.partition, query_context); + + const auto exports_path = fs::path(zookeeper_path) / "exports"; + + const auto export_key = partition_id + "_" + dest_storage_id.getQualifiedName().getFullName(); + + const auto partition_exports_path = fs::path(exports_path) / export_key; + + /// check if entry already exists + ProfileEvents::increment(ProfileEvents::ExportPartitionZooKeeperRequests); + ProfileEvents::increment(ProfileEvents::ExportPartitionZooKeeperExists); + if (zookeeper->exists(partition_exports_path)) + { + LOG_INFO(log, "Export with key {} is already exported or it is being exported", export_key); + + if (!query_context->getSettingsRef()[Setting::export_merge_tree_partition_force_export]) + { + throw Exception(ErrorCodes::EXPORT_PARTITION_ALREADY_EXPORTED, "Export with key {} already exported or it is being exported. Set `export_merge_tree_partition_force_export` to overwrite it.", export_key); + } + + LOG_INFO(log, "Overwriting export with key {}", export_key); + + /// Not putting in ops (same transaction) because we can't construct a "tryRemoveRecursive" request. + /// It is possible that the zk being used does not support RemoveRecursive requests. + /// It is ok for this to be non transactional. Worst case scenario an on-going export is going to be killed and a new task won't be scheduled. + ProfileEvents::increment(ProfileEvents::ExportPartitionZooKeeperRequests); + ProfileEvents::increment(ProfileEvents::ExportPartitionZooKeeperRemoveRecursive); + zookeeper->tryRemoveRecursive(partition_exports_path); + } + + Coordination::Requests ops; + + ops.emplace_back(zkutil::makeCreateRequest(partition_exports_path, "", zkutil::CreateMode::Persistent)); + + DataPartsVector parts; + + { + auto data_parts_lock = lockParts(); + parts = getDataPartsVectorInPartitionForInternalUsage(MergeTreeDataPartState::Active, partition_id, data_parts_lock); + } + + if (parts.empty()) + { + throw Exception(ErrorCodes::BAD_ARGUMENTS, "Partition {} doesn't exist", partition_id); + } + + const bool throw_on_pending_mutations = query_context->getSettingsRef()[Setting::export_merge_tree_part_throw_on_pending_mutations]; + const bool throw_on_pending_patch_parts = query_context->getSettingsRef()[Setting::export_merge_tree_part_throw_on_pending_patch_parts]; + + MergeTreeData::IMutationsSnapshot::Params mutations_snapshot_params + { + .metadata_version = src_snapshot->getMetadataVersion(), + .min_part_metadata_version = MergeTreeData::getPartsSnapshotInfo(parts).min_metadata_version, + .need_data_mutations = throw_on_pending_mutations, + .need_alter_mutations = throw_on_pending_mutations || throw_on_pending_patch_parts, + .need_patch_parts = throw_on_pending_patch_parts, + }; + + const auto mutations_snapshot = getMutationsSnapshot(mutations_snapshot_params); + + std::vector part_names; + for (const auto & part : parts) + { + const auto alter_conversions = getAlterConversionsForPart(part, mutations_snapshot, query_context); + + /// re-check `throw_on_pending_mutations` because `pending_mutations` might have been filled due to `throw_on_pending_patch_parts` + if (alter_conversions->hasMutations() && throw_on_pending_mutations) + { + throw Exception(ErrorCodes::PENDING_MUTATIONS_NOT_ALLOWED, + "Partition {} can not be exported because the part {} has pending mutations. Either wait for the mutations to be applied or set `export_merge_tree_part_throw_on_pending_mutations` to false", + partition_id, + part->name); + } + + if (alter_conversions->hasPatches()) + { + throw Exception(ErrorCodes::PENDING_MUTATIONS_NOT_ALLOWED, + "Partition {} can not be exported because the part {} has pending patch parts. Either wait for the patch parts to be applied or set `export_merge_tree_part_throw_on_pending_patch_parts` to false", + partition_id, + part->name); + } + + part_names.push_back(part->name); + } + + /// TODO arthur somehow check if the list of parts is updated "enough" + + ExportReplicatedMergeTreePartitionManifest manifest; + + manifest.transaction_id = toString(UUIDHelpers::generateV4()); + manifest.query_id = query_context->getCurrentQueryId(); + manifest.partition_id = partition_id; + manifest.destination_database = dest_database; + manifest.destination_table = dest_table; + manifest.source_replica = replica_name; + manifest.number_of_parts = part_names.size(); + manifest.parts = part_names; + manifest.create_time = time(nullptr); + manifest.retry_initial_backoff_seconds = query_context->getSettingsRef()[Setting::export_merge_tree_partition_retry_initial_backoff_seconds]; + manifest.retry_max_backoff_seconds = query_context->getSettingsRef()[Setting::export_merge_tree_partition_retry_max_backoff_seconds]; + manifest.task_timeout_seconds = query_context->getSettingsRef()[Setting::export_merge_tree_partition_task_timeout_seconds]; + manifest.max_threads = query_context->getSettingsRef()[Setting::max_threads]; + manifest.parallel_formatting = query_context->getSettingsRef()[Setting::output_format_parallel_formatting]; + manifest.parquet_parallel_encoding = query_context->getSettingsRef()[Setting::output_format_parquet_parallel_encoding]; + manifest.parquet_compression_method = query_context->getSettingsRef()[Setting::output_format_parquet_compression_method].toString(); + manifest.output_format_compression_level = query_context->getSettingsRef()[Setting::output_format_compression_level]; + manifest.parquet_row_group_size = query_context->getSettingsRef()[Setting::output_format_parquet_row_group_size]; + manifest.parquet_row_group_size_bytes = query_context->getSettingsRef()[Setting::output_format_parquet_row_group_size_bytes]; + manifest.max_bytes_per_file = query_context->getSettingsRef()[Setting::export_merge_tree_part_max_bytes_per_file]; + manifest.max_rows_per_file = query_context->getSettingsRef()[Setting::export_merge_tree_part_max_rows_per_file]; + + manifest.file_already_exists_policy = query_context->getSettingsRef()[Setting::export_merge_tree_part_file_already_exists_policy].value; + manifest.filename_pattern = query_context->getSettingsRef()[Setting::export_merge_tree_part_filename_pattern].value; + manifest.write_full_path_in_iceberg_metadata = query_context->getSettingsRef()[Setting::write_full_path_in_iceberg_metadata]; + manifest.allow_lossy_cast = query_context->getSettingsRef()[Setting::export_merge_tree_part_allow_lossy_cast]; + manifest.iceberg_partition_timezone = query_context->getSettingsRef()[Setting::iceberg_partition_timezone].toString(); + manifest.schema_mismatch_mode = query_context->getSettingsRef()[Setting::export_merge_tree_part_schema_mismatch_mode].value; + + if (dest_storage->isDataLake()) + { +#if USE_AVRO + auto * object_storage = dynamic_cast(dest_storage.get()); + auto * object_storage_cluster = dynamic_cast(dest_storage.get()); + + /// in theory this should never happen, but just in case + if (!object_storage && !object_storage_cluster) + { + throw Exception(ErrorCodes::BAD_ARGUMENTS, "Destination storage {} is not a StorageObjectStorage", dest_storage->getName()); + } + + std::shared_ptr iceberg_metadata; + if (object_storage) + iceberg_metadata = std::dynamic_pointer_cast(object_storage->getExternalMetadata(query_context)); + else if (object_storage_cluster) + iceberg_metadata = std::dynamic_pointer_cast(object_storage_cluster->getExternalMetadata(query_context)); + if (!iceberg_metadata) + { + throw Exception(ErrorCodes::BAD_ARGUMENTS, "Destination storage {} is a data lake but not an iceberg table", dest_storage->getName()); + } + + if (!query_context->getSettingsRef()[Setting::allow_insert_into_iceberg]) + { + throw Exception(ErrorCodes::SUPPORT_IS_DISABLED, + "Iceberg writes are experimental. " + "To allow its usage, enable the setting `allow_insert_into_iceberg` on the initiator (query, session or profile) - replicas inherit it from the scheduled task."); + } + + const auto metadata_object = iceberg_metadata->getMetadataJSON(query_context); + + ExportPartitionUtils::verifyIcebergPartitionCompatibility( + metadata_object, + src_snapshot, + destination_snapshot, + parts, + partition_id, + query_context); + + std::ostringstream oss; // STYLE_CHECK_ALLOW_STD_STRING_STREAM + oss.exceptions(std::ios::failbit); + metadata_object->stringify(oss); + manifest.iceberg_metadata_json = oss.str(); + + manifest.max_bytes_per_file = query_context->getSettingsRef()[Setting::iceberg_insert_max_bytes_in_data_file]; + manifest.max_rows_per_file = query_context->getSettingsRef()[Setting::iceberg_insert_max_rows_in_data_file]; + +#else + throw Exception(ErrorCodes::NOT_IMPLEMENTED, "Data lake export requires Avro support"); +#endif + } + else + { + ExportPartitionUtils::verifyPlainPartitionCompatibility( + src_snapshot, + destination_snapshot, + parts, + partition_id, + query_context); + } + + ops.emplace_back(zkutil::makeCreateRequest( + fs::path(partition_exports_path) / "metadata.json", + manifest.toJsonString(), + zkutil::CreateMode::Persistent)); + + /// Container for per-replica last_exception leaves; children are created lazily by the + /// first writer per replica (see ExportPartitionUtils::appendExceptionOps). + ops.emplace_back(zkutil::makeCreateRequest( + fs::path(partition_exports_path) / "last_exception", + "", + zkutil::CreateMode::Persistent)); + + ops.emplace_back(zkutil::makeCreateRequest( + fs::path(partition_exports_path) / "processing", + "", + zkutil::CreateMode::Persistent)); + + for (const auto & part : part_names) + { + ExportReplicatedMergeTreePartitionProcessingPartEntry entry; + entry.status = ExportReplicatedMergeTreePartitionProcessingPartEntry::Status::PENDING; + entry.part_name = part; + + ops.emplace_back(zkutil::makeCreateRequest( + fs::path(partition_exports_path) / "processing" / part, + entry.toJsonString(), + zkutil::CreateMode::Persistent)); + } + + ops.emplace_back(zkutil::makeCreateRequest( + fs::path(partition_exports_path) / "processed", + "", + zkutil::CreateMode::Persistent)); + + ops.emplace_back(zkutil::makeCreateRequest( + fs::path(partition_exports_path) / "locks", + "", + zkutil::CreateMode::Persistent)); + + /// status: IN_PROGRESS, COMPLETED, FAILED + ops.emplace_back(zkutil::makeCreateRequest( + fs::path(partition_exports_path) / "status", + "PENDING", + zkutil::CreateMode::Persistent)); + + ProfileEvents::increment(ProfileEvents::ExportPartitionZooKeeperRequests); + ProfileEvents::increment(ProfileEvents::ExportPartitionZooKeeperMulti); + Coordination::Responses responses; + Coordination::Error code = zookeeper->tryMulti(ops, responses); + + if (code != Coordination::Error::ZOK) + { + if (code == Coordination::Error::ZNODEEXISTS + && zkutil::getFailedOpIndex(code, responses) == 0) + { + /// Lost the race on the root export node. Current code already + /// validated (exists / expired / force) — so this is *always* a race. + throw Exception(ErrorCodes::EXPORT_PARTITION_ALREADY_EXPORTED, + "Export with key {} was created concurrently by another replica. Retry if needed", + export_key); + } + throw zkutil::KeeperException::fromPath(code, partition_exports_path); + } +} + void StorageReplicatedMergeTree::forgetPartition(const ASTPtr & partition, ContextPtr query_context) { @@ -9924,6 +10474,119 @@ CancellationCode StorageReplicatedMergeTree::killPartMoveToShard(const UUID & ta return part_moves_between_shards_orchestrator.killPartMoveToShard(task_uuid); } +CancellationCode StorageReplicatedMergeTree::killExportPartition(const String & transaction_id) +{ + /// Called from a query thread (KILL EXPORT PARTITION via InterpreterKillQueryQuery), which does not have a component set. + auto component_guard = Coordination::setCurrentComponent("StorageReplicatedMergeTree::killExportPartition"); + + /// KILL is serialized against the commit phase via commit_lock (see below), so a kill that + /// succeeds cannot be overwritten by a concurrent commit. + + auto try_set_status_to_killed = [this](const zkutil::ZooKeeperPtr & zk, const std::string & status_path) + { + /// Serialize against commit(): if a commit holds the lock, it is too late to cancel. + auto commit_lock = zkutil::EphemeralNodeHolder::tryCreate( + fs::path(status_path).parent_path() / "commit_lock", *zk, replica_name); + if (!commit_lock) + { + LOG_INFO(log, "Commit in progress, can not cancel export partition task"); + return CancellationCode::CancelCannotBeSent; + } + + Coordination::Stat stat; + std::string status_from_zk_string; + + if (!zk->tryGet(status_path, status_from_zk_string, &stat)) + { + /// found entry locally, but not in zk. It might have been deleted by another replica and we did not have time to update the local entry. + LOG_INFO(log, "Export partition task not found in zk, can not cancel it"); + return CancellationCode::CancelCannotBeSent; + } + + const auto status_from_zk = magic_enum::enum_cast(status_from_zk_string); + + if (!status_from_zk) + { + LOG_INFO(log, "Export partition task status is invalid, can not cancel it"); + return CancellationCode::CancelCannotBeSent; + } + + if (status_from_zk.value() != ExportReplicatedMergeTreePartitionTaskEntry::Status::PENDING) + { + LOG_INFO(log, "Export partition task is {}, can not cancel it", String(magic_enum::enum_name(status_from_zk.value()))); + return CancellationCode::CancelCannotBeSent; + } + + if (zk->trySet(status_path, String(magic_enum::enum_name(ExportReplicatedMergeTreePartitionTaskEntry::Status::KILLED)), stat.version) != Coordination::Error::ZOK) + { + LOG_INFO(log, "Status has been updated while trying to kill the export partition task, can not cancel it"); + return CancellationCode::CancelCannotBeSent; + } + + return CancellationCode::CancelSent; + }; + + const auto zk = getZooKeeper(); + + /// Read the published snapshot (shared_ptr copy, no lock, no ZooKeeper). The KILLED status set + /// below propagates back into the mirror via the status watch -> handleStatusChanges. + bool local_entry_found = false; + bool local_entry_pending = false; + std::string local_composite_key; + + if (const auto model = export_partition_manifests.get()) + { + const auto & by_transaction_id = model->get(); + const auto entry = by_transaction_id.find(transaction_id); + if (entry != by_transaction_id.end()) + { + local_entry_found = true; + local_entry_pending = entry->status == ExportReplicatedMergeTreePartitionTaskEntry::Status::PENDING; + local_composite_key = entry->getCompositeKey(); + } + } + + /// if we have the entry locally, no need to list from zk. we can save some requests. + if (local_entry_found) + { + LOG_INFO(log, "Export partition task found locally, trying to cancel it"); + /// found locally, no need to get children on zk + if (!local_entry_pending) + { + LOG_INFO(log, "Export partition task is not pending, can not cancel it"); + return CancellationCode::CancelCannotBeSent; + } + + return try_set_status_to_killed(zk, fs::path(zookeeper_path) / "exports" / local_composite_key / "status"); + } + else + { + LOG_INFO(log, "Export partition task not found locally, trying to find it on zk"); + /// for some reason, we don't have the entry locally. ls on zk to find the entry + const auto exports_path = fs::path(zookeeper_path) / "exports"; + + const auto export_keys = zk->getChildren(exports_path); + String export_key_to_be_cancelled; + + for (const auto & export_key : export_keys) + { + std::string metadata_json; + if (!zk->tryGet(fs::path(exports_path) / export_key / "metadata.json", metadata_json)) + continue; + const auto manifest = ExportReplicatedMergeTreePartitionManifest::fromJsonString(metadata_json); + if (manifest.transaction_id == transaction_id) + { + LOG_INFO(log, "Export partition task found on zk, trying to cancel it"); + return try_set_status_to_killed(zk, fs::path(exports_path) / export_key / "status"); + } + } + } + + LOG_INFO(log, "Export partition task not found, can not cancel it"); + + return CancellationCode::NotFound; +} + void StorageReplicatedMergeTree::getCommitPartOps( Coordination::Requests & ops, const DataPartPtr & part, @@ -10492,13 +11155,6 @@ MutationCounters StorageReplicatedMergeTree::getMutationCounters() const return queue.getMutationCounters(); } -void StorageReplicatedMergeTree::startBackgroundMovesIfNeeded() -{ - if (areBackgroundMovesNeeded()) - background_moves_assignee.start(); -} - - std::unique_ptr StorageReplicatedMergeTree::getDefaultSettings() const { return std::make_unique(getContext()->getReplicatedMergeTreeSettings()); diff --git a/src/Storages/StorageReplicatedMergeTree.h b/src/Storages/StorageReplicatedMergeTree.h index 3caa3cf223f5..dd5f3f397571 100644 --- a/src/Storages/StorageReplicatedMergeTree.h +++ b/src/Storages/StorageReplicatedMergeTree.h @@ -1,5 +1,6 @@ #pragma once +#include #include #include #include @@ -10,6 +11,9 @@ #include #include #include +#include +#include +#include #include #include #include @@ -32,6 +36,7 @@ #include #include #include +#include #include #include #include @@ -95,6 +100,8 @@ namespace DB class ZooKeeperWithFaultInjection; using ZooKeeperWithFaultInjectionPtr = std::shared_ptr; +struct ReplicatedPartitionExportInfo; + class StorageReplicatedMergeTree final : public MergeTreeData { public: @@ -375,6 +382,8 @@ class StorageReplicatedMergeTree final : public MergeTreeData using ShutdownDeadline = std::chrono::time_point; void waitForUniquePartsToBeFetchedByOtherReplicas(ShutdownDeadline shutdown_deadline); + std::vector getPartitionExportsInfo() const; + private: std::atomic_bool are_restoring_replica {false}; @@ -399,6 +408,8 @@ class StorageReplicatedMergeTree final : public MergeTreeData friend class MergeFromLogEntryTask; friend class MutateFromLogEntryTask; friend class ReplicatedMergeMutateTaskBase; + friend class ExportPartitionManifestUpdatingTask; + friend class ExportPartitionTaskScheduler; using MergeStrategyPicker = ReplicatedMergeTreeMergeStrategyPicker; using LogEntry = ReplicatedMergeTreeLogEntry; @@ -511,6 +522,21 @@ class StorageReplicatedMergeTree final : public MergeTreeData /// A task that marks finished mutations as done. BackgroundSchedulePoolTaskHolder mutations_finalizing_task; + BackgroundSchedulePoolTaskHolder export_merge_tree_partition_updating_task; + + /// mostly handle kill operations + BackgroundSchedulePoolTaskHolder export_merge_tree_partition_status_handling_task; + std::shared_ptr export_merge_tree_partition_manifest_updater; + + std::shared_ptr export_merge_tree_partition_task_scheduler; + + Coordination::WatchCallbackPtr export_merge_tree_partition_watch_callback; + + BackgroundSchedulePoolTaskHolder export_merge_tree_partition_select_task; + + /// Immutable snapshot republished after each writer batch (part_references stripped). Readers + /// (system table, scheduler, KILL) get() a consistent version with no lock and no ZooKeeper. + MultiVersion export_partition_manifests; /// A thread that removes old parts, log entries, and blocks. ReplicatedMergeTreeCleanupThread cleanup_thread; @@ -742,6 +768,14 @@ class StorageReplicatedMergeTree final : public MergeTreeData /// Checks if some mutations are done and marks them as done. void mutationsFinalizingTask(); + void selectPartsToExport(); + + /// update in-memory list of partition exports + void exportMergeTreePartitionUpdatingTask(); + + /// handle status changes for export partition tasks + void exportMergeTreePartitionStatusHandlingTask(); + /** Write the selected parts to merge into the log, * Call when merge_selecting_mutex is locked. * Returns false if any part is not in ZK. @@ -930,6 +964,7 @@ class StorageReplicatedMergeTree final : public MergeTreeData void movePartitionToTable(const StoragePtr & dest_table, const ASTPtr & partition, ContextPtr query_context) override; void movePartitionToShard(const ASTPtr & partition, bool move_part, const String & to, ContextPtr query_context) override; CancellationCode killPartMoveToShard(const UUID & task_uuid) override; + CancellationCode killExportPartition(const String & transaction_id) override; void fetchPartition( const ASTPtr & partition, const StorageMetadataPtr & metadata_snapshot, @@ -937,7 +972,8 @@ class StorageReplicatedMergeTree final : public MergeTreeData bool fetch_part, ContextPtr query_context) override; void forgetPartition(const ASTPtr & partition, ContextPtr query_context) override; - + + void exportPartitionToTable(const PartitionCommand &, ContextPtr) override; /// NOTE: there are no guarantees for concurrent merges. Dropping part can /// be concurrently merged into some covering part and dropPart will do @@ -969,8 +1005,6 @@ class StorageReplicatedMergeTree final : public MergeTreeData MutationsSnapshotPtr getMutationsSnapshot(const IMutationsSnapshot::Params & params) const override; - void startBackgroundMovesIfNeeded() override; - /// Attaches restored parts to the storage. void attachRestoredParts(MutableDataPartsVector && parts, const std::optional & zookeeper_retries_info) override; diff --git a/src/Storages/StorageURL.cpp b/src/Storages/StorageURL.cpp index 7f3e7f6dca7a..54fdd1959e3a 100644 --- a/src/Storages/StorageURL.cpp +++ b/src/Storages/StorageURL.cpp @@ -783,7 +783,7 @@ void StorageURLSink::cancelBuffers() write_buf->cancel(); } -class PartitionedStorageURLSink : public PartitionedSink +class PartitionedStorageURLSink : public PartitionedSink::SinkCreator { public: PartitionedStorageURLSink( @@ -797,7 +797,7 @@ class PartitionedStorageURLSink : public PartitionedSink const CompressionMethod compression_method_, const HTTPHeaderEntries & headers_, const String & http_method_) - : PartitionedSink(partition_strategy_, context_, std::make_shared(sample_block_)) + : partition_strategy(partition_strategy_) , uri(uri_) , format(format_) , format_settings(format_settings_) @@ -812,7 +812,8 @@ class PartitionedStorageURLSink : public PartitionedSink SinkPtr createSinkForPartition(const String & partition_id) override { - std::string partition_path = partition_strategy->getPathForWrite(uri, partition_id); + const auto file_path_generator = std::make_shared(uri); + std::string partition_path = file_path_generator->getPathForWrite(partition_id); context->getRemoteHostFilter().checkURL(Poco::URI(partition_path)); return std::make_shared( @@ -820,6 +821,7 @@ class PartitionedStorageURLSink : public PartitionedSink } private: + std::shared_ptr partition_strategy; const String uri; const String format; const std::optional format_settings; @@ -1509,7 +1511,7 @@ SinkToStoragePtr IStorageURLBase::write(const ASTPtr & query, const StorageMetad has_wildcards, /* partition_columns_in_data_file */true); - return std::make_shared( + auto sink_creator = std::make_shared( partition_strategy, uri, format_name, @@ -1520,6 +1522,8 @@ SinkToStoragePtr IStorageURLBase::write(const ASTPtr & query, const StorageMetad compression_method, headers, http_method); + + return std::make_shared(partition_strategy, sink_creator, context, std::make_shared(metadata_snapshot->getSampleBlock())); } return std::make_shared( @@ -2617,7 +2621,7 @@ void registerStorageURL(StorageFactory & factory) StorageURL::overrideURLInEngineArgs(object_storage_args, config.url, context, /*skip_userinfo=*/ false); auto configuration = std::make_shared(); - StorageObjectStorageConfiguration::initialize(*configuration, object_storage_args, context, /* with_table_structure */ false); + configuration->initialize(object_storage_args, context, /* with_table_structure */ false); /// Same contract as `createStorageObjectStorage`: only a user-issued `CREATE` applies the /// `file_like_engine_default_partition_strategy` default; ATTACH / startup / RESTORE must diff --git a/src/Storages/StorageURLCluster.cpp b/src/Storages/StorageURLCluster.cpp index e6378b222182..2a2437778614 100644 --- a/src/Storages/StorageURLCluster.cpp +++ b/src/Storages/StorageURLCluster.cpp @@ -109,7 +109,11 @@ StorageURLCluster::StorageURLCluster( setInMemoryMetadata(storage_metadata); } -void StorageURLCluster::updateQueryToSendIfNeeded(ASTPtr & query, const StorageSnapshotPtr & storage_snapshot, const ContextPtr & context) +void StorageURLCluster::updateQueryToSendIfNeeded( + ASTPtr & query, + const StorageSnapshotPtr & storage_snapshot, + const ContextPtr & context, + bool /*make_cluster_function*/) { auto * table_function = extractTableFunctionFromSelectQuery(query); if (!table_function) @@ -138,20 +142,42 @@ void StorageURLCluster::updateQueryToSendIfNeeded(ASTPtr & query, const StorageS } } -RemoteQueryExecutor::Extension StorageURLCluster::getTaskIteratorExtension( - const ActionsDAG::Node * predicate, const ActionsDAG * /* filter */, const ContextPtr & context, ClusterPtr, StorageMetadataPtr metadata) const +class UrlTaskIterator : public TaskIterator { - auto iterator = std::make_shared( - uri, context->getSettingsRef()[Setting::glob_expansion_max_elements], predicate, metadata->virtuals.getSampleBlock(VirtualsKind::All, VirtualsMaterializationPlace::Reader).getNamesAndTypesList(), hive_partition_columns_to_read_from_file_path, context); - - auto next_callback = [iter = std::move(iterator)](size_t) mutable -> ClusterFunctionReadTaskResponsePtr +public: + UrlTaskIterator(const String & uri, + size_t max_addresses, + const ActionsDAG::Node * predicate, + const NamesAndTypesList & virtual_columns, + const NamesAndTypesList & hive_partition_columns_to_read_from_file_path, + const ContextPtr & context) + : iterator(uri, max_addresses, predicate, virtual_columns, hive_partition_columns_to_read_from_file_path, context) {} + + ~UrlTaskIterator() override = default; + + ClusterFunctionReadTaskResponsePtr operator()(size_t /* number_of_current_replica */) const override { - auto url = iter->next(); + auto url = iterator.next(); if (url.empty()) return std::make_shared(); return std::make_shared(std::move(url)); - }; - auto callback = std::make_shared(std::move(next_callback)); + } + +private: + mutable StorageURLSource::DisclosedGlobIterator iterator; +}; + +RemoteQueryExecutor::Extension StorageURLCluster::getTaskIteratorExtension( + const ActionsDAG::Node * predicate, const ActionsDAG * /* filter */, const ContextPtr & context, ClusterPtr, StorageMetadataPtr metadata) const +{ + auto callback = std::make_shared( + uri, + context->getSettingsRef()[Setting::glob_expansion_max_elements], + predicate, + metadata->virtuals.getSampleBlock(VirtualsKind::All, VirtualsMaterializationPlace::Reader).getNamesAndTypesList(), + getHivePartitionColumnsWithoutVirtuals(metadata), + context + ); return RemoteQueryExecutor::Extension{.task_iterator = std::move(callback)}; } diff --git a/src/Storages/StorageURLCluster.h b/src/Storages/StorageURLCluster.h index bee90c4ba7b0..5dd81d755f65 100644 --- a/src/Storages/StorageURLCluster.h +++ b/src/Storages/StorageURLCluster.h @@ -38,11 +38,14 @@ class StorageURLCluster : public IStorageCluster StorageMetadataPtr) const override; private: - void updateQueryToSendIfNeeded(ASTPtr & query, const StorageSnapshotPtr & storage_snapshot, const ContextPtr & context) override; + void updateQueryToSendIfNeeded( + ASTPtr & query, + const StorageSnapshotPtr & storage_snapshot, + const ContextPtr & context, + bool /*make_cluster_function*/) override; String uri; String format_name; - NamesAndTypesList hive_partition_columns_to_read_from_file_path; }; diff --git a/src/Storages/System/StorageSystemExports.cpp b/src/Storages/System/StorageSystemExports.cpp new file mode 100644 index 000000000000..1bac86870712 --- /dev/null +++ b/src/Storages/System/StorageSystemExports.cpp @@ -0,0 +1,71 @@ +#include +#include +#include +#include +#include +#include +#include +#include +#include + +namespace DB +{ + +ColumnsDescription StorageSystemExports::getColumnsDescription() +{ + return ColumnsDescription + { + {"source_database", std::make_shared(), "Name of the source database."}, + {"source_table", std::make_shared(), "Name of the source table."}, + {"destination_database", std::make_shared(), "Name of the destination database."}, + {"destination_table", std::make_shared(), "Name of the destination table."}, + {"create_time", std::make_shared(), "Date and time when the export command was received in the server."}, + {"part_name", std::make_shared(), "Name of the part"}, + {"query_id", std::make_shared(), "Query ID of the export operation."}, + {"destination_file_paths", std::make_shared(std::make_shared()), "File paths where the part is being exported."}, + {"elapsed", std::make_shared(), "The time elapsed (in seconds) since the export started."}, + {"rows_read", std::make_shared(), "The number of rows read from the exported part."}, + {"total_rows_to_read", std::make_shared(), "The total number of rows to read from the exported part."}, + {"total_size_bytes_compressed", std::make_shared(), "The total size of the compressed data in the exported part."}, + {"total_size_bytes_uncompressed", std::make_shared(), "The total size of the uncompressed data in the exported part."}, + {"bytes_read_uncompressed", std::make_shared(), "The number of uncompressed bytes read from the exported part."}, + {"memory_usage", std::make_shared(), "Current memory usage in bytes for the export operation."}, + {"peak_memory_usage", std::make_shared(), "Peak memory usage in bytes during the export operation."}, + }; +} + +void StorageSystemExports::fillData(MutableColumns & res_columns, ContextPtr context, const ActionsDAG::Node *, std::vector) const +{ + const auto access = context->getAccess(); + const bool check_access_for_tables = !access->isGranted(AccessType::SHOW_TABLES); + + for (const auto & export_info : context->getExportsList().get()) + { + if (check_access_for_tables && !access->isGranted(AccessType::SHOW_TABLES, export_info.source_database, export_info.source_table)) + continue; + + size_t i = 0; + res_columns[i++]->insert(export_info.source_database); + res_columns[i++]->insert(export_info.source_table); + res_columns[i++]->insert(export_info.destination_database); + res_columns[i++]->insert(export_info.destination_table); + res_columns[i++]->insert(export_info.create_time); + res_columns[i++]->insert(export_info.part_name); + res_columns[i++]->insert(export_info.query_id); + Array destination_file_paths_array; + destination_file_paths_array.reserve(export_info.destination_file_paths.size()); + for (const auto & file_path : export_info.destination_file_paths) + destination_file_paths_array.push_back(file_path); + res_columns[i++]->insert(destination_file_paths_array); + res_columns[i++]->insert(export_info.elapsed); + res_columns[i++]->insert(export_info.rows_read); + res_columns[i++]->insert(export_info.total_rows_to_read); + res_columns[i++]->insert(export_info.total_size_bytes_compressed); + res_columns[i++]->insert(export_info.total_size_bytes_uncompressed); + res_columns[i++]->insert(export_info.bytes_read_uncompressed); + res_columns[i++]->insert(export_info.memory_usage); + res_columns[i++]->insert(export_info.peak_memory_usage); + } +} + +} diff --git a/src/Storages/System/StorageSystemExports.h b/src/Storages/System/StorageSystemExports.h new file mode 100644 index 000000000000..e13fbfa26aaa --- /dev/null +++ b/src/Storages/System/StorageSystemExports.h @@ -0,0 +1,25 @@ +#pragma once + +#include + + +namespace DB +{ + +class Context; + + +class StorageSystemExports final : public IStorageSystemOneBlock +{ +public: + std::string getName() const override { return "SystemExports"; } + + static ColumnsDescription getColumnsDescription(); + +protected: + using IStorageSystemOneBlock::IStorageSystemOneBlock; + + void fillData(MutableColumns & res_columns, ContextPtr context, const ActionsDAG::Node *, std::vector) const override; +}; + +} diff --git a/src/Storages/System/StorageSystemIcebergFiles.cpp b/src/Storages/System/StorageSystemIcebergFiles.cpp index c428bd784e6f..92de0c6eb2c9 100644 --- a/src/Storages/System/StorageSystemIcebergFiles.cpp +++ b/src/Storages/System/StorageSystemIcebergFiles.cpp @@ -25,6 +25,7 @@ #include #include #include +#include #include @@ -221,13 +222,33 @@ class SystemIcebergFilesSource : public ISource if (!lock) return false; + /// Object storage tables created by the storage factory are `StorageObjectStorageCluster` + /// (it falls back to plain object storage reads when no cluster is configured), while some + /// code paths still produce a plain `StorageObjectStorage`. Handle both. auto * object_storage_table = dynamic_cast(storage.get()); - if (!object_storage_table || !object_storage_table->isIcebergStorage()) + auto * object_storage_cluster_table = dynamic_cast(storage.get()); + + if (object_storage_table) + { + if (!object_storage_table->isIcebergStorage()) + return false; + } + else if (object_storage_cluster_table) + { + if (!object_storage_cluster_table->isIcebergStorage()) + return false; + } + else + { return false; + } try { - auto iceberg_metadata = std::dynamic_pointer_cast(object_storage_table->getExternalMetadata(context_copy)); + auto external_metadata = object_storage_table + ? object_storage_table->getExternalMetadata(context_copy) + : object_storage_cluster_table->getExternalMetadata(context_copy); + auto iceberg_metadata = std::dynamic_pointer_cast(external_metadata); if (!iceberg_metadata) return false; diff --git a/src/Storages/System/StorageSystemIcebergHistory.cpp b/src/Storages/System/StorageSystemIcebergHistory.cpp index e9cd3b66eb28..8bb30b0b8ec4 100644 --- a/src/Storages/System/StorageSystemIcebergHistory.cpp +++ b/src/Storages/System/StorageSystemIcebergHistory.cpp @@ -23,7 +23,7 @@ #include #include #include -#include +#include #include #include #include @@ -94,7 +94,7 @@ void StorageSystemIcebergHistory::fillData( if (!access->isGranted(AccessType::SHOW_TABLES)) return; - auto add_history_record = [&](const String & database_name, const String & table_name, StorageObjectStorage * object_storage) + auto add_history_record = [&](const String & database_name, const String & table_name, StorageObjectStorageCluster * object_storage) { if (!access->isGranted(AccessType::SHOW_TABLES, database_name, table_name)) return; @@ -202,7 +202,7 @@ void StorageSystemIcebergHistory::fillData( // Table was dropped while acquiring the lock, skipping table continue; - if (auto * object_storage_table = dynamic_cast(storage.get())) + if (auto * object_storage_table = dynamic_cast(storage.get())) { add_history_record(database_name, table_name, object_storage_table); } diff --git a/src/Storages/System/StorageSystemReplicatedPartitionExports.cpp b/src/Storages/System/StorageSystemReplicatedPartitionExports.cpp new file mode 100644 index 000000000000..6370e0f99433 --- /dev/null +++ b/src/Storages/System/StorageSystemReplicatedPartitionExports.cpp @@ -0,0 +1,205 @@ +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include "Columns/ColumnString.h" +#include "Storages/VirtualColumnUtils.h" + + +namespace DB +{ + +ColumnsDescription StorageSystemReplicatedPartitionExports::getColumnsDescription() +{ + auto last_exception_tuple = std::make_shared( + DataTypes{ + std::make_shared(), + std::make_shared(), + std::make_shared(), + std::make_shared(), + std::make_shared(), + }, + Names{"replica", "message", "part", "time", "count"}); + + auto backoff_tuple = std::make_shared( + DataTypes{ + std::make_shared(), + std::make_shared(), + std::make_shared(), + }, + Names{"part", "attempts", "next_retry_time"}); + + return ColumnsDescription + { + {"source_database", std::make_shared(), "Name of the source database."}, + {"source_table", std::make_shared(), "Name of the source table."}, + {"destination_database", std::make_shared(), "Name of the destination database."}, + {"destination_table", std::make_shared(), "Name of the destination table."}, + {"create_time", std::make_shared(), "Date and time when the export command was submitted"}, + {"partition_id", std::make_shared(), "ID of the partition"}, + {"transaction_id", std::make_shared(), "ID of the transaction."}, + {"query_id", std::make_shared(), "Query ID of the export operation."}, + {"source_replica", std::make_shared(), "Name of the source replica."}, + {"parts", std::make_shared(std::make_shared()), "List of part names to be exported."}, + {"parts_count", std::make_shared(), "Number of parts in the export."}, + {"parts_to_do", std::make_shared(), "Number of parts pending to be exported."}, + {"status", std::make_shared(), "Status of the export."}, + {"last_exception_per_replica", std::make_shared(last_exception_tuple), + "Per-replica last exception entries. Each tuple records the most recent exception observed by that replica plus a best-effort within-replica count. Empty array if no replica has reported an exception for this task."}, + {"exception_count", std::make_shared(), + "Sum of per-replica exception counts. Each replica owns its own count, so the sum is exact w.r.t. the in-memory snapshot; within-replica updates remain best-effort and may under-count by one under concurrent failures."}, + {"destination_file_paths", std::make_shared(std::make_shared(), std::make_shared(std::make_shared())), + "Per-part destination file paths written to the destination object storage. Keyed by part name; values are the file paths produced by exporting that part. Mirrored from ZooKeeper on every poll while PENDING; partial during in-flight tasks. When the in-memory mirror could not fully refresh from Keeper (or a processed leaf is unreadable), the map may contain the key and/or path value '' for the affected part (or for the whole field if listing processed leaves failed); replaced on the next successful poll."}, + {"committed_metadata_file", std::make_shared(), + "For Iceberg destinations: path of the new metadata JSON file written at commit time. Empty for non-Iceberg destinations and for tasks that have not committed yet. May also be empty if the committing replica crashed between writing the object-storage files and persisting commit_info. If the export was already committed by a previous run (detected via the transaction id stored in the snapshot summary), this column holds a human-readable note instead of a path since the original committer's paths are not trivially recoverable."}, + {"committed_manifest_list", std::make_shared(), + "For Iceberg destinations: path of the manifest list file (snap-*.avro) referenced by the new snapshot. Empty under the same conditions as committed_metadata_file."}, + {"committed_manifest_file", std::make_shared(), + "For Iceberg destinations: path of the manifest file referenced by committed_manifest_list. Empty under the same conditions as committed_metadata_file."}, + {"committed_marker_file", std::make_shared(), + "For plain object storage destinations: path of the per-transaction commit marker file written by the destination. Empty for Iceberg destinations and for tasks that have not committed yet."}, + {"local_backoff_per_part", std::make_shared(backoff_tuple), + "Per-part retry back-off local to this replica: parts currently waiting before their next attempt, with attempt count and the next eligible time. Not shared across replicas; empty if no part is backing off."}, + }; +} + +void StorageSystemReplicatedPartitionExports::fillData(MutableColumns & res_columns, ContextPtr context, const ActionsDAG::Node * predicate, std::vector) const +{ + const auto access = context->getAccess(); + const bool check_access_for_databases = !access->isGranted(AccessType::SHOW_TABLES); + + std::map> replicated_merge_tree_tables; + for (const auto & db : DatabaseCatalog::instance().getDatabases(GetDatabasesOptions{.with_datalake_catalogs = false})) + { + /// skip data lakes + if (db.second->isExternal()) + continue; + + const bool check_access_for_tables = check_access_for_databases && !access->isGranted(AccessType::SHOW_TABLES, db.first); + + for (auto iterator = db.second->getTablesIterator(context); iterator->isValid(); iterator->next()) + { + const auto & table = iterator->table(); + if (!table) + continue; + + StorageReplicatedMergeTree * table_replicated = dynamic_cast(table.get()); + if (!table_replicated) + continue; + + if (check_access_for_tables && !access->isGranted(AccessType::SHOW_TABLES, db.first, iterator->name())) + continue; + + replicated_merge_tree_tables[db.first][iterator->name()] = table; + } + } + + MutableColumnPtr col_database_mut = ColumnString::create(); + MutableColumnPtr col_table_mut = ColumnString::create(); + + for (auto & db : replicated_merge_tree_tables) + { + for (auto & table : db.second) + { + col_database_mut->insert(db.first); + col_table_mut->insert(table.first); + } + } + + ColumnPtr col_database = std::move(col_database_mut); + ColumnPtr col_table = std::move(col_table_mut); + + /// Determine what tables are needed by the conditions in the query. + { + Block filtered_block + { + { col_database, std::make_shared(), "database" }, + { col_table, std::make_shared(), "table" }, + }; + + VirtualColumnUtils::filterBlockWithPredicate(predicate, filtered_block, context); + + if (!filtered_block.rows()) + return; + + col_database = filtered_block.getByName("database").column; + col_table = filtered_block.getByName("table").column; + } + + for (size_t i_storage = 0; i_storage < col_database->size(); ++i_storage) + { + const auto database = (*col_database)[i_storage].safeGet(); + const auto table = (*col_table)[i_storage].safeGet(); + + std::vector partition_exports_info; + { + const IStorage * storage = replicated_merge_tree_tables[database][table].get(); + if (const auto * replicated_merge_tree = dynamic_cast(storage)) + partition_exports_info = replicated_merge_tree->getPartitionExportsInfo(); + } + + for (const ReplicatedPartitionExportInfo & info : partition_exports_info) + { + std::size_t i = 0; + res_columns[i++]->insert(database); + res_columns[i++]->insert(table); + res_columns[i++]->insert(info.destination_database); + res_columns[i++]->insert(info.destination_table); + res_columns[i++]->insert(info.create_time); + res_columns[i++]->insert(info.partition_id); + res_columns[i++]->insert(info.transaction_id); + res_columns[i++]->insert(info.query_id); + res_columns[i++]->insert(info.source_replica); + Array parts_array; + parts_array.reserve(info.parts.size()); + for (const auto & part : info.parts) + parts_array.push_back(part); + res_columns[i++]->insert(parts_array); + res_columns[i++]->insert(info.parts_count); + res_columns[i++]->insert(info.parts_to_do); + res_columns[i++]->insert(info.status); + + Array per_replica; + per_replica.reserve(info.last_exception_per_replica.size()); + for (const auto & ex : info.last_exception_per_replica) + per_replica.push_back(Tuple{ex.replica, ex.message, ex.part, ex.time, ex.count}); + res_columns[i++]->insert(per_replica); + res_columns[i++]->insert(info.exception_count); + + Map destination_paths_map; + destination_paths_map.reserve(info.destination_file_paths_per_part.size()); + for (const auto & [part_name, paths] : info.destination_file_paths_per_part) + { + Array paths_array; + paths_array.reserve(paths.size()); + for (const auto & path : paths) + paths_array.push_back(path); + destination_paths_map.emplace_back(Tuple{part_name, std::move(paths_array)}); + } + res_columns[i++]->insert(std::move(destination_paths_map)); + + res_columns[i++]->insert(info.committed_metadata_file); + res_columns[i++]->insert(info.committed_manifest_list); + + res_columns[i++]->insert(info.committed_manifest_file); + + res_columns[i++]->insert(info.committed_marker_file); + + Array backoff_array; + backoff_array.reserve(info.backoff_per_part.size()); + for (const auto & b : info.backoff_per_part) + backoff_array.push_back(Tuple{b.part, b.attempts, b.next_retry_time}); + res_columns[i++]->insert(backoff_array); + } + } +} + +} diff --git a/src/Storages/System/StorageSystemReplicatedPartitionExports.h b/src/Storages/System/StorageSystemReplicatedPartitionExports.h new file mode 100644 index 000000000000..768516c0a2ea --- /dev/null +++ b/src/Storages/System/StorageSystemReplicatedPartitionExports.h @@ -0,0 +1,76 @@ +#pragma once + +#include +#include + +namespace DB +{ + +class Context; + +struct ReplicatedPartitionExportInfo +{ + String destination_database; + String destination_table; + String partition_id; + String transaction_id; + String query_id; + time_t create_time; + String source_replica; + size_t parts_count; + size_t parts_to_do; + std::vector parts; + String status; + /// One entry per replica that has recorded at least one exception for this task. + /// Sourced verbatim from the in-memory mirror; no ZooKeeper traffic. + std::vector last_exception_per_replica; + /// Sum of per-replica counts. Each replica owns its own count, so cross-replica + /// updates do not race; the sum is exact w.r.t. the in-memory snapshot. Within a + /// single replica the count is best-effort (concurrent failing writers may under- + /// count by one), matching the documented column semantics. + size_t exception_count = 0; + + /// Per-part destination file paths, keyed by part name. Mirrors the + /// /processed//paths_in_destination data from ZooKeeper. + /// Empty until parts complete; partial during PENDING. May contain + /// "" when a Keeper refresh was incomplete or a + /// processed leaf could not be parsed. + std::map> destination_file_paths_per_part; + + /// Iceberg commit-time paths surfaced from /commit_info. + /// All empty for non-Iceberg destinations or before commit lands. + String committed_metadata_file; + String committed_manifest_list; + String committed_manifest_file; + + /// Plain object storage commit marker file surfaced from + /// /commit_info. Empty for Iceberg destinations or before + /// commit lands. + String committed_marker_file; + + struct PartBackoffEntry + { + String part; + size_t attempts = 0; + time_t next_retry_time = 0; + }; + + /// Parts of this task currently backing off (local to this replica). Empty if none. + std::vector backoff_per_part; +}; + +class StorageSystemReplicatedPartitionExports final : public IStorageSystemOneBlock +{ +public: + + std::string getName() const override { return "SystemReplicatedPartitionExports"; } + + static ColumnsDescription getColumnsDescription(); + +protected: + using IStorageSystemOneBlock::IStorageSystemOneBlock; + + void fillData(MutableColumns & res_columns, ContextPtr context, const ActionsDAG::Node *, std::vector) const override; +}; + +} diff --git a/src/Storages/System/StorageSystemTables.cpp b/src/Storages/System/StorageSystemTables.cpp index 4e2b8faedca9..c4cc06fec8be 100644 --- a/src/Storages/System/StorageSystemTables.cpp +++ b/src/Storages/System/StorageSystemTables.cpp @@ -32,6 +32,9 @@ #include #include #include +#include +#include +#include #include #include #include @@ -321,6 +324,23 @@ ColumnPtr getFilteredTables( } +namespace +{ + +/// Returns data lake metadata (Iceberg, DeltaLake, ...) of the table, if it has any. +/// Object storage table engines are instantiated as StorageObjectStorageCluster, which is not derived +/// from StorageObjectStorage, so both storage types have to be handled here. +std::shared_ptr tryGetDataLakeMetadata(const StoragePtr & table, ContextPtr context) +{ + if (auto * object_storage = dynamic_cast(table.get())) + return object_storage->getExternalMetadata(context); + if (auto * object_storage_cluster = dynamic_cast(table.get())) + return object_storage_cluster->getExternalMetadata(context); + return nullptr; +} + +} + StorageSystemTables::StorageSystemTables(const StorageID & table_id_) : StorageWithCommonVirtualColumns(table_id_) { @@ -869,18 +889,64 @@ class TablesBlockSource final : public ISource ASTPtr expression_ptr; if (columns_mask[src_index++]) { - if (metadata_snapshot && (expression_ptr = metadata_snapshot->getPartitionKeyAST())) - res_columns[res_index++]->insert(format({context, *expression_ptr})); - else - res_columns[res_index++]->insertDefault(); + bool inserted = false; + + try + { + // Extract from specific DataLake metadata if suitable + if (auto dl_meta = tryGetDataLakeMetadata(table, context)) + { + if (auto p = dl_meta->partitionKey(context); p.has_value()) + { + res_columns[res_index++]->insert(*p); + inserted = true; + } + } + } + catch (const Exception &) + { + /// Failed to get info. It's not critical, just log it. + tryLogCurrentException("StorageSystemTables"); + } + + if (!inserted) + { + if (metadata_snapshot && (expression_ptr = metadata_snapshot->getPartitionKeyAST())) + res_columns[res_index++]->insert(format({context, *expression_ptr})); + else + res_columns[res_index++]->insertDefault(); + } } if (columns_mask[src_index++]) { - if (metadata_snapshot && (expression_ptr = metadata_snapshot->getSortingKey().expression_list_ast)) - res_columns[res_index++]->insert(format({context, *expression_ptr})); - else - res_columns[res_index++]->insertDefault(); + bool inserted = false; + + try + { + // Extract from specific DataLake metadata if suitable + if (auto dl_meta = tryGetDataLakeMetadata(table, context)) + { + if (auto p = dl_meta->sortingKey(context); p.has_value()) + { + res_columns[res_index++]->insert(*p); + inserted = true; + } + } + } + catch (const Exception &) + { + /// Failed to get info. It's not critical, just log it. + tryLogCurrentException("StorageSystemTables"); + } + + if (!inserted) + { + if (metadata_snapshot && (expression_ptr = metadata_snapshot->getSortingKey().expression_list_ast)) + res_columns[res_index++]->insert(format({context, *expression_ptr})); + else + res_columns[res_index++]->insertDefault(); + } } if (columns_mask[src_index++]) diff --git a/src/Storages/System/attachSystemTables.cpp b/src/Storages/System/attachSystemTables.cpp index 594e25db3e9d..f9ed0ea8b750 100644 --- a/src/Storages/System/attachSystemTables.cpp +++ b/src/Storages/System/attachSystemTables.cpp @@ -1,11 +1,12 @@ #include +#include #include "config.h" #include #include #include #include - +#include #include #include #include @@ -40,6 +41,7 @@ #include #include #include +#include #include #include #include @@ -144,6 +146,7 @@ # include #endif #include +#include #include @@ -175,6 +178,11 @@ namespace ErrorCodes extern const int TABLE_ALREADY_EXISTS; } +namespace ServerSetting +{ + extern const ServerSettingsBool allow_experimental_export_merge_tree_partition; +} + void attachSystemTablesServer(ContextPtr context, IDatabase & system_database, bool has_zookeeper, [[maybe_unused]] bool has_keeper_server) { auto component_guard = Coordination::setCurrentComponent("attachSystemTablesServer"); @@ -270,6 +278,11 @@ void attachSystemTablesServer(ContextPtr context, IDatabase & system_database, b attach(context, system_database, "dimensional_metrics", "Contains dimensional metrics, which have multiple dimensions (labels) to provide more granular information. For example, counting failed merges by their error code. This table is always up to date."); attach(context, system_database, "merges", "Contains a list of merges currently executing merges of MergeTree tables and their progress. Each merge operation is represented by a single row."); attach(context, system_database, "moves", "Contains information about in-progress data part moves of MergeTree tables. Each data part movement is represented by a single row."); + attach(context, system_database, "exports", "Contains a list of exports currently executing exports of MergeTree tables and their progress. Each export operation is represented by a single row."); + if (context->getServerSettings()[ServerSetting::allow_experimental_export_merge_tree_partition]) + { + attach(context, system_database, "replicated_partition_exports", "Contains a list of partition exports of ReplicatedMergeTree tables and their progress. Each export operation is represented by a single row."); + } attach(context, system_database, "mutations", "Contains a list of mutations and their progress. Each mutation command is represented by a single row."); attachNoDescription(context, system_database, "replicas", "Contains information and status of all table replicas on current server. Each replica is represented by a single row."); attachNoDescription(context, system_database, "database_replicas", "Contains information and status of all database replicas on current server. Each database replica is represented by a single row."); diff --git a/src/Storages/buildQueryTreeForShard.cpp b/src/Storages/buildQueryTreeForShard.cpp index 9c6692420422..a253873a4949 100644 --- a/src/Storages/buildQueryTreeForShard.cpp +++ b/src/Storages/buildQueryTreeForShard.cpp @@ -66,6 +66,7 @@ namespace Setting extern const SettingsBool enable_add_distinct_to_in_subqueries; extern const SettingsInt64 optimize_const_name_size; extern const SettingsOverflowMode transfer_overflow_mode; + extern const SettingsObjectStorageClusterJoinMode object_storage_cluster_join_mode; } namespace ErrorCodes @@ -291,9 +292,10 @@ class DistributedProductModeRewriteInJoinVisitor : public InDepthQueryTreeVisito using Base = InDepthQueryTreeVisitorWithContext; using Base::Base; - explicit DistributedProductModeRewriteInJoinVisitor(const ContextPtr & context_, bool allow_global_join_for_right_table_) + explicit DistributedProductModeRewriteInJoinVisitor(const ContextPtr & context_, bool allow_global_join_for_right_table_, bool find_cross_join_) : Base(context_) , allow_global_join_for_right_table(allow_global_join_for_right_table_) + , find_cross_join(find_cross_join_) {} struct InFunctionOrJoin @@ -330,9 +332,11 @@ class DistributedProductModeRewriteInJoinVisitor : public InDepthQueryTreeVisito { auto * function_node = node->as(); auto * join_node = node->as(); + CrossJoinNode * cross_join_node = find_cross_join ? node->as() : nullptr; if ((function_node && isNameOfGlobalInFunction(function_node->getFunctionName())) || - (join_node && join_node->getLocality() == JoinLocality::Global)) + (join_node && join_node->getLocality() == JoinLocality::Global) || + cross_join_node) { InFunctionOrJoin in_function_or_join_entry; in_function_or_join_entry.query_node = node; @@ -396,7 +400,9 @@ class DistributedProductModeRewriteInJoinVisitor : public InDepthQueryTreeVisito replacement_table_expression->setTableExpressionModifiers(*table_expression_modifiers); replacement_map.emplace(&table_node_typed, std::move(replacement_table_expression)); } - else if ((distributed_product_mode == DistributedProductMode::GLOBAL || getSettings()[Setting::prefer_global_in_and_join]) && + else if ((distributed_product_mode == DistributedProductMode::GLOBAL || + getSettings()[Setting::prefer_global_in_and_join] || + (find_cross_join && getSettings()[Setting::object_storage_cluster_join_mode] == ObjectStorageClusterJoinMode::GLOBAL)) && !in_function_or_join_stack.empty()) { auto * in_or_join_node_to_modify = in_function_or_join_stack.back().query_node.get(); @@ -431,6 +437,8 @@ class DistributedProductModeRewriteInJoinVisitor : public InDepthQueryTreeVisito IQueryTreeNode::ReplacementMap replacement_map; std::vector global_in_or_join_nodes; bool allow_global_join_for_right_table = false; + + bool find_cross_join = false; }; /** Replaces large constant values with `__getScalar` function calls to avoid @@ -901,14 +909,18 @@ void inlineAliasColumns(QueryTreeNodePtr & query_tree_to_modify) inlineAliasColumnsImpl(query_tree_to_modify); } -QueryTreeNodePtr buildQueryTreeForShard(const PlannerContextPtr & planner_context, QueryTreeNodePtr query_tree_to_modify, bool allow_global_join_for_right_table) +QueryTreeNodePtr buildQueryTreeForShard( + const PlannerContextPtr & planner_context, + QueryTreeNodePtr query_tree_to_modify, + bool allow_global_join_for_right_table, + bool find_cross_join) { CollectColumnSourceToColumnsVisitor collect_column_source_to_columns_visitor; collect_column_source_to_columns_visitor.visit(query_tree_to_modify); const auto & column_source_to_columns = collect_column_source_to_columns_visitor.getColumnSourceToColumns(); - DistributedProductModeRewriteInJoinVisitor visitor(planner_context->getQueryContext(), allow_global_join_for_right_table); + DistributedProductModeRewriteInJoinVisitor visitor(planner_context->getQueryContext(), allow_global_join_for_right_table, find_cross_join); visitor.visit(query_tree_to_modify); auto replacement_map = visitor.getReplacementMap(); @@ -971,6 +983,42 @@ QueryTreeNodePtr buildQueryTreeForShard(const PlannerContextPtr & planner_contex replacement_map.emplace(join_table_expression.get(), std::move(temporary_table_expression_node)); continue; } + if (auto * cross_join_node = global_in_or_join_node.query_node->as()) + { + auto tables_count = cross_join_node->getTableExpressions().size(); + for (size_t i = 1; i < tables_count; ++i) + { + TableExpressionNodePtr join_table_expression = cross_join_node->getTableExpressionTypedAt(i); + + auto subquery_node = getSubqueryFromTableExpression(join_table_expression, column_source_to_columns, planner_context->getQueryContext()); + + auto temporary_table_expression_node = executeSubqueryNode(subquery_node, + planner_context->getMutableQueryContext(), + global_in_or_join_node.subquery_depth); + temporary_table_expression_node->setAlias(join_table_expression->getAlias()); + + std::vector descendants_to_map; + for (const auto & child : join_table_expression->getChildren()) + if (child) + descendants_to_map.push_back(child.get()); + + while (!descendants_to_map.empty()) + { + const auto * descendant = descendants_to_map.back(); + descendants_to_map.pop_back(); + + if (const auto * ptr = descendant->asTableExpression()) + replacement_map.emplace(ptr, temporary_table_expression_node); + + for (const auto & child : descendant->getChildren()) + if (child) + descendants_to_map.push_back(child.get()); + } + + replacement_map.emplace(join_table_expression.get(), std::move(temporary_table_expression_node)); + } + continue; + } if (auto * in_function_node = global_in_or_join_node.query_node->as()) { auto & in_function_subquery_node = in_function_node->getArguments().getNodes().at(1); @@ -1132,7 +1180,7 @@ class RewriteJoinToGlobalJoinVisitor : public InDepthQueryTreeVisitorWithContext { if (auto * join_node = node->as()) { - bool prefer_local_join = getContext()->getSettingsRef()[Setting::parallel_replicas_prefer_local_join]; + bool prefer_local_join = !force_prefer_local_join && getContext()->getSettingsRef()[Setting::parallel_replicas_prefer_local_join]; bool should_use_global_join = !prefer_local_join || !materializedSideCanStayLocal(*join_node); if (should_use_global_join) join_node->setLocality(JoinLocality::Global); @@ -1147,11 +1195,79 @@ class RewriteJoinToGlobalJoinVisitor : public InDepthQueryTreeVisitorWithContext return true; } + + void setForcePreferLocalJoin(bool force_prefer_local_join_) { force_prefer_local_join = force_prefer_local_join_; } + +private: + bool force_prefer_local_join = false; }; -void rewriteJoinToGlobalJoin(QueryTreeNodePtr query_tree_to_modify, ContextPtr context) +void rewriteJoinToGlobalJoin(QueryTreeNodePtr query_tree_to_modify, ContextPtr context, bool force_prefer_local_join) { RewriteJoinToGlobalJoinVisitor visitor(context); + visitor.setForcePreferLocalJoin(force_prefer_local_join); + visitor.visit(query_tree_to_modify); +} + +class RewriteInToGlobalInVisitor : public InDepthQueryTreeVisitorWithContext +{ +public: + using Base = InDepthQueryTreeVisitorWithContext; + using Base::Base; + + void enterImpl(QueryTreeNodePtr & node) + { + if (auto * function_node = node->as(); function_node && isNameOfLocalInFunction(function_node->getFunctionName())) + { + auto * query = function_node->getArguments().getNodes()[1]->as(); + if (!query) + return; + if (!rewrite_for_distributed) + { + bool no_replace = true; + for (const auto & table_node : extractTableExpressions(query->getJoinTreeNodeTyped(), false, true)) + { + const StorageDistributed * storage_distributed = nullptr; + if (const TableNode * table_node_typed = table_node->as()) + storage_distributed = typeid_cast(table_node_typed->getStorage().get()); + else if (const TableFunctionNode * table_function_node_typed = table_node->as()) + storage_distributed = typeid_cast(table_function_node_typed->getStorage().get()); + + if (!storage_distributed) + { + no_replace = false; + break; + } + } + if (no_replace) + return; + } + + auto result_function = std::make_shared(getGlobalInFunctionNameForLocalInFunctionName(function_node->getFunctionName())); + result_function->getArguments().getNodes() = std::move(function_node->getArguments().getNodes()); + resolveOrdinaryFunctionNodeByName(*result_function, result_function->getFunctionName(), getContext()); + node = result_function; + } + } + + static bool needChildVisit(QueryTreeNodePtr & parent, QueryTreeNodePtr &) + { + if (auto * function_node = parent->as(); function_node && function_node->getFunctionName().starts_with("global")) + return false; + + return true; + } + + void setRewriteForDistributed(bool rewrite_for_distributed_) { rewrite_for_distributed = rewrite_for_distributed_; } + +private: + bool rewrite_for_distributed = false; +}; + +void rewriteInToGlobalIn(QueryTreeNodePtr & query_tree_to_modify, ContextPtr context, bool rewrite_for_distributed) +{ + RewriteInToGlobalInVisitor visitor(context); + visitor.setRewriteForDistributed(rewrite_for_distributed); visitor.visit(query_tree_to_modify); } diff --git a/src/Storages/buildQueryTreeForShard.h b/src/Storages/buildQueryTreeForShard.h index 9132f6018ac3..66122c25be79 100644 --- a/src/Storages/buildQueryTreeForShard.h +++ b/src/Storages/buildQueryTreeForShard.h @@ -23,7 +23,11 @@ using ContextPtr = std::shared_ptr; class Block; -QueryTreeNodePtr buildQueryTreeForShard(const PlannerContextPtr & planner_context, QueryTreeNodePtr query_tree_to_modify, bool allow_global_join_for_right_table); +QueryTreeNodePtr buildQueryTreeForShard( + const PlannerContextPtr & planner_context, + QueryTreeNodePtr query_tree_to_modify, + bool allow_global_join_for_right_table, + bool find_cross_join = false); /** Replace every `ALIAS` column node with its defining expression, so the expression is evaluated on the shard/replica * that reads the real table instead of the column being resolved there as if it were physical. @@ -34,7 +38,8 @@ QueryTreeNodePtr buildQueryTreeForShard(const PlannerContextPtr & planner_contex */ void inlineAliasColumns(QueryTreeNodePtr & query_tree_to_modify); -void rewriteJoinToGlobalJoin(QueryTreeNodePtr query_tree_to_modify, ContextPtr context); +void rewriteJoinToGlobalJoin(QueryTreeNodePtr query_tree_to_modify, ContextPtr context, bool force_prefer_global_join = false); +void rewriteInToGlobalIn(QueryTreeNodePtr & query_tree_to_modify, ContextPtr context, bool rewrite_for_distributed = false); /** When a Distributed/parallel-replicas query is executed up to `WithMergeableState`, the shard's query tree has its * `ALIAS` columns inlined into their defining expressions. If several projection (or sort/group/...) items expand to the diff --git a/src/Storages/extractTableFunctionFromSelectQuery.cpp b/src/Storages/extractTableFunctionFromSelectQuery.cpp index 57302036c889..064f538eeae7 100644 --- a/src/Storages/extractTableFunctionFromSelectQuery.cpp +++ b/src/Storages/extractTableFunctionFromSelectQuery.cpp @@ -9,7 +9,7 @@ namespace DB { -ASTFunction * extractTableFunctionFromSelectQuery(ASTPtr & query) +ASTTableExpression * extractTableExpressionASTPtrFromSelectQuery(ASTPtr & query) { auto * select_query = query->as(); if (!select_query || !select_query->tables()) @@ -17,10 +17,36 @@ ASTFunction * extractTableFunctionFromSelectQuery(ASTPtr & query) auto * tables = select_query->tables()->as(); auto * table_expression = tables->children[0]->as()->table_expression->as(); - if (!table_expression->table_function) + return table_expression; +} + +ASTPtr extractTableFunctionASTPtrFromSelectQuery(ASTPtr & query) +{ + auto table_expression = extractTableExpressionASTPtrFromSelectQuery(query); + return table_expression ? table_expression->table_function : nullptr; +} + +ASTPtr extractTableASTPtrFromSelectQuery(ASTPtr & query) +{ + auto table_expression = extractTableExpressionASTPtrFromSelectQuery(query); + return table_expression ? table_expression->database_and_table_name : nullptr; +} + +ASTFunction * extractTableFunctionFromSelectQuery(ASTPtr & query) +{ + auto table_function_ast = extractTableFunctionASTPtrFromSelectQuery(query); + if (!table_function_ast) return nullptr; - return table_expression->table_function->as(); + return table_function_ast->as(); +} + +ASTExpressionList * extractTableFunctionArgumentsFromSelectQuery(ASTPtr & query) +{ + auto * table_function = extractTableFunctionFromSelectQuery(query); + if (!table_function) + return nullptr; + return table_function->arguments->as(); } } diff --git a/src/Storages/extractTableFunctionFromSelectQuery.h b/src/Storages/extractTableFunctionFromSelectQuery.h index c69cc7ce6c52..2a845477df82 100644 --- a/src/Storages/extractTableFunctionFromSelectQuery.h +++ b/src/Storages/extractTableFunctionFromSelectQuery.h @@ -1,12 +1,17 @@ #pragma once #include -#include #include +#include namespace DB { +struct ASTTableExpression; +ASTTableExpression * extractTableExpressionASTPtrFromSelectQuery(ASTPtr & query); +ASTPtr extractTableFunctionASTPtrFromSelectQuery(ASTPtr & query); +ASTPtr extractTableASTPtrFromSelectQuery(ASTPtr & query); ASTFunction * extractTableFunctionFromSelectQuery(ASTPtr & query); +ASTExpressionList * extractTableFunctionArgumentsFromSelectQuery(ASTPtr & query); } diff --git a/src/TableFunctions/CMakeLists.txt b/src/TableFunctions/CMakeLists.txt index ccdab2fc41b2..eb1c67d2018d 100644 --- a/src/TableFunctions/CMakeLists.txt +++ b/src/TableFunctions/CMakeLists.txt @@ -12,6 +12,7 @@ extract_into_parent_list(clickhouse_table_functions_sources dbms_sources ITableFunction.cpp TableFunctionView.cpp TableFunctionFactory.cpp + TableFunctionRemote.cpp ) extract_into_parent_list(clickhouse_table_functions_headers dbms_headers ITableFunction.h diff --git a/src/TableFunctions/ITableFunction.h b/src/TableFunctions/ITableFunction.h index f34337773e66..1d28c050b2b5 100644 --- a/src/TableFunctions/ITableFunction.h +++ b/src/TableFunctions/ITableFunction.h @@ -81,7 +81,7 @@ class ITableFunction : public std::enable_shared_from_this virtual bool supportsReadingSubsetOfColumns(const ContextPtr &) { return true; } - virtual bool canBeUsedToCreateTable() const { return true; } + virtual void validateUseToCreateTable() const {} /// The name of the named collection the table function arguments were resolved from, or an empty /// string. When a permanent table is created from the table function (`CREATE TABLE ... AS f(...)`), diff --git a/src/TableFunctions/ITableFunctionCluster.h b/src/TableFunctions/ITableFunctionCluster.h index 5345e1a0f0db..920f271f0535 100644 --- a/src/TableFunctions/ITableFunctionCluster.h +++ b/src/TableFunctions/ITableFunctionCluster.h @@ -16,6 +16,7 @@ namespace ErrorCodes extern const int NUMBER_OF_ARGUMENTS_DOESNT_MATCH; extern const int CLUSTER_DOESNT_EXIST; extern const int LOGICAL_ERROR; + extern const int BAD_ARGUMENTS; } /// Base class for *Cluster table functions that require cluster_name for the first argument. @@ -46,9 +47,13 @@ class ITableFunctionCluster : public Base throw Exception(ErrorCodes::LOGICAL_ERROR, "Unexpected table function name: {}", table_function->name); } - bool canBeUsedToCreateTable() const override { return false; } bool isClusterFunction() const override { return true; } + void validateUseToCreateTable() const override + { + throw Exception(ErrorCodes::BAD_ARGUMENTS, "Table function '{}' cannot be used to create a table", getName()); + } + protected: void parseArguments(const ASTPtr & ast, ContextPtr context) override { @@ -70,9 +75,11 @@ class ITableFunctionCluster : public Base /// Cluster name is always the first cluster_name = checkAndGetLiteralArgument(args[0], "cluster_name"); - - if (!context->tryGetCluster(cluster_name)) - throw Exception(ErrorCodes::CLUSTER_DOESNT_EXIST, "Requested cluster '{}' not found", cluster_name); + /// Remove check cluster existing here + /// In query like + /// remote('remote_host', xxxCluster('remote_cluster', ...)) + /// 'remote_cluster' can be defined only on 'remote_host' + /// If cluster not exists, query falls later /// Just cut the first arg (cluster_name) and try to parse other table function arguments as is args.erase(args.begin()); diff --git a/src/TableFunctions/TableFunctionEval.cpp b/src/TableFunctions/TableFunctionEval.cpp index d67d0277bb78..beb026e93956 100644 --- a/src/TableFunctions/TableFunctionEval.cpp +++ b/src/TableFunctions/TableFunctionEval.cpp @@ -66,7 +66,10 @@ class TableFunctionEval : public ITableFunction /// would be re-evaluated on every `ATTACH`, so such a table could fail to attach after a restart /// (the experimental setting might be disabled) or silently change if the expression depends on /// parameters, settings, or time. There is no stable persisted representation, so forbid it. - bool canBeUsedToCreateTable() const override { return false; } + void validateUseToCreateTable() const override + { + throw Exception(ErrorCodes::BAD_ARGUMENTS, "Table function '{}' cannot be used to create a table", getName()); + } private: StoragePtr executeImpl( diff --git a/src/TableFunctions/TableFunctionObjectStorage.cpp b/src/TableFunctions/TableFunctionObjectStorage.cpp index 3dc3147b132f..12c3a5d10a48 100644 --- a/src/TableFunctions/TableFunctionObjectStorage.cpp +++ b/src/TableFunctions/TableFunctionObjectStorage.cpp @@ -194,7 +194,7 @@ template ColumnsDescription TableFunctionObjectStorage< Definition, Configuration, is_data_lake>::getActualTableStructure(ContextPtr context, bool is_insert_query) const { - if (configuration->structure == "auto") + if (configuration->getStructure() == "auto") { configuration->check(context); auto storage = getObjectStorage(context, !is_insert_query); @@ -204,7 +204,6 @@ ColumnsDescription TableFunctionObjectStorage< ColumnsDescription columns; resolveSchemaAndFormat( columns, - configuration->format, std::move(storage), configuration, /* format_settings */std::nullopt, @@ -221,7 +220,7 @@ ColumnsDescription TableFunctionObjectStorage< return columns; } - return parseColumnsListFromString(configuration->structure, context); + return parseColumnsListFromString(configuration->getStructure(), context); } template @@ -235,8 +234,8 @@ StoragePtr TableFunctionObjectStorage:: chassert(configuration); ColumnsDescription columns; - if (configuration->structure != "auto") - columns = parseColumnsListFromString(configuration->structure, context); + if (configuration->getStructure() != "auto") + columns = parseColumnsListFromString(configuration->getStructure(), context); else if (!structure_hint.empty()) columns = structure_hint; else if (!cached_columns.empty()) @@ -249,11 +248,13 @@ StoragePtr TableFunctionObjectStorage:: /// Only use parallel replicas if the Cluster variant of this table function exists /// (e.g. `s3Cluster` for `s3`). Table functions without a Cluster variant (e.g. `paimonLocal`) /// cannot distribute work via task iterators, so distributing would just read all data on every replica. + /// `getName`, not `Definition::name`: with `TableFunctionObjectStorageClusterFallback` + /// the definition is the *Cluster one, while the invoked function is `s3`. const auto can_use_parallel_replicas = !parallel_replicas_cluster_name.empty() && query_settings[Setting::parallel_replicas_for_cluster_engines] && context->canUseTaskBasedParallelReplicas() && !context->isDistributed() - && TableFunctionFactory::instance().isTableFunctionName(String(name) + "Cluster"); + && TableFunctionFactory::instance().isTableFunctionName(getName() + "Cluster"); const auto is_secondary_query = context->getClientInfo().query_kind == ClientInfo::QueryKind::SECONDARY_QUERY; @@ -267,8 +268,15 @@ StoragePtr TableFunctionObjectStorage:: columns, ConstraintsDescription{}, partition_by, + /* order_by */ nullptr, context, - /* is_table_function */true); + /* comment */ String{}, + /* format_settings */ std::nullopt, /// No format_settings + /* mode */ LoadingStrictnessLevel::CREATE, + configuration->getCatalog(context, StorageID(getDatabaseName(), table_name)), + /* if_not_exists */ false, + /* is_datalake_query*/ false, + /* is_table_function */ true); storage->startup(); return storage; @@ -316,457 +324,7 @@ void registerTableFunctionObjectStorage(TableFunctionFactory & factory) { UNUSED(factory); #if USE_AWS_S3 - factory.registerFunction>( - {.description = R"DOCS_MD( -import { ExperimentalBadge } from "/snippets/components/ExperimentalBadge/ExperimentalBadge.jsx"; -import { CloudNotSupportedBadge } from "/snippets/components/CloudNotSupportedBadge/CloudNotSupportedBadge.jsx"; - -Provides a table-like interface to select/insert files in [Amazon S3](https://aws.amazon.com/s3/) and [Google Cloud Storage](https://cloud.google.com/storage/). This table function is similar to the [hdfs function](/reference/functions/table-functions/hdfs), but provides S3-specific features. - -If you have multiple replicas in your cluster, you can use the [s3Cluster function](/reference/functions/table-functions/s3Cluster) instead to parallelize inserts. - -When using the `s3 table function` with [`INSERT INTO...SELECT`](/reference/statements/insert-into#inserting-the-results-of-select), data is read and inserted in a streaming fashion. Only a few blocks of data reside in memory while the blocks are continuously read from S3 and pushed into the destination table. - -## Syntax {#syntax} - -```sql -s3(url [, NOSIGN | access_key_id, secret_access_key, [session_token]] [,format] [,structure] [,compression_method],[,headers], [,extra_credentials], [,partition_strategy], [,partition_columns_in_data_file]) -s3(named_collection[, option=value [,..]]) -``` - - -**GCS** - -The S3 Table Function integrates with Google Cloud Storage by using the GCS XML API and HMAC keys. See the [Google interoperability docs](https://cloud.google.com/storage/docs/interoperability) for more details about the endpoint and HMAC. - -For GCS, substitute your HMAC key and HMAC secret where you see `access_key_id` and `secret_access_key`. - - -**Parameters** - -`s3` table function supports the following plain parameters: - -| Parameter | Description | -|-----------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| -| `url` | Bucket url with path to file. Supports following wildcards in readonly mode: `*`, `**`, `?`, `{abc,def}` and `{N..M}` where `N`, `M` — numbers, `'abc'`, `'def'` — strings. For more information see [here](/reference/engines/table-engines/integrations/s3#wildcards-in-path). | -| `NOSIGN` | If this keyword is provided in place of credentials, all the requests will not be signed. | -| `access_key_id` and `secret_access_key` | Keys that specify credentials to use with given endpoint. Optional. | -| `session_token` | Session token to use with the given keys. Optional when passing keys. | -| `format` | The [format](/reference/formats/index) of the file. | -| `structure` | Structure of the table. Format `'column1_name column1_type, column2_name column2_type, ...'`. | -| `compression_method` | Parameter is optional. Supported values: `none`, `gzip` or `gz`, `brotli` or `br`, `xz` or `LZMA`, `zstd` or `zst`. By default, it will autodetect compression method by file extension. | -| `headers` | Parameter is optional. Allows headers to be passed in the S3 request. Pass in the format `headers(key=value)` e.g. `headers('x-amz-request-payer' = 'requester')`. | -| `partition_strategy` | Parameter is optional. Supported values: `wildcard` or `hive`. `wildcard` requires a `{_partition_id}` in the path, which is replaced with the partition key. `hive` does not allow wildcards, assumes the path is the table root, and generates Hive-style partitioned directories with Snowflake IDs as filenames and the file format as the extension. Without an explicit strategy, a path with `{_partition_id}` uses `wildcard`. A path with another glob uses no partition strategy and ignores `PARTITION BY`. A path without a glob uses `hive` when `file_like_engine_default_partition_strategy` is `hive`; otherwise it uses no partition strategy. | -| `partition_columns_in_data_file` | Parameter is optional. Only used with `hive` partition strategy. Tells ClickHouse whether to expect partition columns to be written in the data file. Defaults `false`. | -| `extra_credentials` | Parameter is optional. Used to pass a `role_arn` for role-based access in ClickHouse Cloud. See [Secure S3](/products/cloud/guides/data-sources/accessing-s3-data-securely) for configuration steps. | -| `storage_class_name` | Parameter is optional. Supported values: `STANDARD`, `REDUCED_REDUNDANCY`, `STANDARD_IA`, `ONEZONE_IA`, `INTELLIGENT_TIERING`, `GLACIER_IR`, `EXPRESS_ONEZONE`. Only S3 storage classes that allow immediate retrieval are supported (archival classes such as `GLACIER` and `DEEP_ARCHIVE` are not). Allows to specify [AWS S3 Intelligent Tiering](https://aws.amazon.com/s3/storage-classes/intelligent-tiering/). Defaults to `STANDARD`. | - - -**GCS** - -The GCS url is in this format as the endpoint for the Google XML API is different than the JSON API: - -```text - https://storage.googleapis.com/// -``` - -and not ~~https://storage.cloud.google.com~~. - - -Arguments can also be passed using [named collections](/concepts/features/configuration/server-config/named-collections). In this case `url`, `access_key_id`, `secret_access_key`, `format`, `structure`, `compression_method` work in the same way, and some extra parameters are supported: - -| Argument | Description | -|-------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| -| `filename` | appended to the url if specified. | -| `use_environment_credentials` | enabled by default, allows passing extra parameters using environment variables `AWS_CONTAINER_CREDENTIALS_RELATIVE_URI`, `AWS_CONTAINER_CREDENTIALS_FULL_URI`, `AWS_CONTAINER_AUTHORIZATION_TOKEN`, `AWS_EC2_METADATA_DISABLED`. | -| `no_sign_request` | disabled by default. | -| `expiration_window_seconds` | default value is 120. | - -## Returned value {#returned-value} - -A table with the specified structure for reading or writing data in the specified file. - -## Examples {#examples} - -Selecting the first 5 rows from the table from S3 file `https://datasets-documentation.s3.eu-west-3.amazonaws.com/aapl_stock.csv`: - -```sql -SELECT * -FROM s3( - 'https://datasets-documentation.s3.eu-west-3.amazonaws.com/aapl_stock.csv', - NOSIGN, - 'CSVWithNames' -) -LIMIT 5; -``` - -```response -┌───────Date─┬────Open─┬────High─┬─────Low─┬───Close─┬───Volume─┬─OpenInt─┐ -│ 1984-09-07 │ 0.42388 │ 0.42902 │ 0.41874 │ 0.42388 │ 23220030 │ 0 │ -│ 1984-09-10 │ 0.42388 │ 0.42516 │ 0.41366 │ 0.42134 │ 18022532 │ 0 │ -│ 1984-09-11 │ 0.42516 │ 0.43668 │ 0.42516 │ 0.42902 │ 42498199 │ 0 │ -│ 1984-09-12 │ 0.42902 │ 0.43157 │ 0.41618 │ 0.41618 │ 37125801 │ 0 │ -│ 1984-09-13 │ 0.43927 │ 0.44052 │ 0.43927 │ 0.43927 │ 57822062 │ 0 │ -└────────────┴─────────┴─────────┴─────────┴─────────┴──────────┴─────────┘ -``` - - -ClickHouse uses filename extensions to determine the format of the data. For example, we could have run the previous command without the `CSVWithNames`: - -```sql -SELECT * -FROM s3( - 'https://datasets-documentation.s3.eu-west-3.amazonaws.com/aapl_stock.csv', - NOSIGN -) -LIMIT 5; -``` - -ClickHouse also can determine the compression method of the file. For example, if the file was zipped up with a `.csv.gz` extension, ClickHouse would decompress the file automatically. - - - -Parquet files with names like `*.parquet.snappy` or `*.parquet.zstd` can confuse ClickHouse and cause `TOO_LARGE_COMPRESSED_BLOCK` or `ZSTD_DECODER_FAILED` errors. -This is because ClickHouse would attempt to read the entire file as Snappy or ZSTD-encoded data when, in fact, Parquet applies compression at the row-group and column level. - -Parquet metadata already specifies the per-column compression, and so the file extension is superfluous. -You can just use `compression_method = 'none'` in such cases: - -```sql -SELECT * -FROM s3( - 'https://.s3..amazonaws.com/path/to/my-data.parquet.snappy', - compression_format = 'none' -); -``` - - -## Usage {#usage} - -Suppose that we have several files with following URIs on S3: - -- 'https://clickhouse-public-datasets.s3.amazonaws.com/my-test-bucket-768/some_prefix/some_file_1.csv' -- 'https://clickhouse-public-datasets.s3.amazonaws.com/my-test-bucket-768/some_prefix/some_file_2.csv' -- 'https://clickhouse-public-datasets.s3.amazonaws.com/my-test-bucket-768/some_prefix/some_file_3.csv' -- 'https://clickhouse-public-datasets.s3.amazonaws.com/my-test-bucket-768/some_prefix/some_file_4.csv' -- 'https://clickhouse-public-datasets.s3.amazonaws.com/my-test-bucket-768/another_prefix/some_file_1.csv' -- 'https://clickhouse-public-datasets.s3.amazonaws.com/my-test-bucket-768/another_prefix/some_file_2.csv' -- 'https://clickhouse-public-datasets.s3.amazonaws.com/my-test-bucket-768/another_prefix/some_file_3.csv' -- 'https://clickhouse-public-datasets.s3.amazonaws.com/my-test-bucket-768/another_prefix/some_file_4.csv' - -Count the number of rows in files ending with numbers from 1 to 3: - -```sql -SELECT count(*) -FROM s3('https://datasets-documentation.s3.eu-west-3.amazonaws.com/my-test-bucket-768/{some,another}_prefix/some_file_{1..3}.csv', NOSIGN, 'CSV', 'column1 UInt32, column2 UInt32, column3 UInt32') -``` - -```text -┌─count()─┐ -│ 18 │ -└─────────┘ -``` - -Count the total amount of rows in all files in these two directories: - -```sql -SELECT count(*) -FROM s3('https://datasets-documentation.s3.eu-west-3.amazonaws.com/my-test-bucket-768/{some,another}_prefix/*', NOSIGN, 'CSV', 'column1 UInt32, column2 UInt32, column3 UInt32') -``` - -```text -┌─count()─┐ -│ 24 │ -└─────────┘ -``` - - -If your listing of files contains number ranges with leading zeros, use the construction with braces for each digit separately or use `?`. - - -Count the total number of rows in files named `file-1.csv`, ..., `file-4.csv`: - -```sql -SELECT count(*) -FROM s3('https://datasets-documentation.s3.eu-west-3.amazonaws.com/my-test-bucket-768/big_prefix/file-{1..4}.csv', NOSIGN, 'CSV', 'column1 UInt32, column2 UInt32, column3 UInt32'); -``` - -```text -┌─count()─┐ -│ 12 │ -└─────────┘ -``` - -Insert data into file `test-data.csv.gz`: - -```sql -INSERT INTO FUNCTION s3('https://clickhouse-public-datasets.s3.amazonaws.com/my-test-bucket-768/test-data.csv.gz', 'CSV', 'name String, value UInt32', 'gzip') -VALUES ('test-data', 1), ('test-data-2', 2); -``` - -Insert data into file `test-data.csv.gz` from existing table: - -```sql -INSERT INTO FUNCTION s3('https://clickhouse-public-datasets.s3.amazonaws.com/my-test-bucket-768/test-data.csv.gz', 'CSV', 'name String, value UInt32', 'gzip') -SELECT name, value FROM existing_table; -``` - -Glob ** can be used for recursive directory traversal. Consider the below example, it will fetch all files from `my-test-bucket-768` directory recursively: - -```sql -SELECT * FROM s3('https://clickhouse-public-datasets.s3.amazonaws.com/my-test-bucket-768/**', NOSIGN, 'CSV', 'name String, value UInt32', 'gzip'); -``` - -The below get data from all `test-data.csv.gz` files from any folder inside `my-test-bucket` directory recursively: - -```sql -SELECT * FROM s3('https://clickhouse-public-datasets.s3.amazonaws.com/my-test-bucket-768/**/test-data.csv.gz', NOSIGN, 'CSV', 'name String, value UInt32', 'gzip'); -``` - -Note. It is possible to specify custom URL mappers in the server configuration file. Example: -```sql -SELECT * FROM s3('s3://clickhouse-public-datasets/my-test-bucket-768/**/test-data.csv.gz', NOSIGN, 'CSV', 'name String, value UInt32', 'gzip'); -``` -The URL `'s3://clickhouse-public-datasets/my-test-bucket-768/**/test-data.csv.gz'` would be replaced to `'http://clickhouse-public-datasets.s3.amazonaws.com/my-test-bucket-768/**/test-data.csv.gz'` - -Custom mapper can be added into `config.xml`: -```xml - - - https://{bucket}.s3.amazonaws.com - - - https://{bucket}.storage.googleapis.com - - - https://{bucket}.oss.aliyuncs.com - - -``` - -For production use cases it is recommended to use [named collections](/concepts/features/configuration/server-config/named-collections). Here is the example: -```sql - -CREATE NAMED COLLECTION creds AS - access_key_id = '***', - secret_access_key = '***'; -SELECT count(*) -FROM s3(creds, url='https://s3-object-url.csv') -``` - -## Partitioned Write {#partitioned-write} - -### Partition Strategy {#partition-strategy} - -Supported for INSERT queries only. - -`wildcard`: Replaces the `{_partition_id}` wildcard in the file path with the actual partition key. It is selected by default when the path contains `{_partition_id}`. - -When no `partition_strategy` is set, a path with another glob uses no partition strategy and ignores `PARTITION BY`. A path without a glob uses `hive` when `file_like_engine_default_partition_strategy` is `hive`; otherwise it uses no partition strategy. - -`hive` implements hive style partitioning for reads & writes. It generates files using the following format: `//.`. - -**Example of `hive` partition strategy** - -```sql -INSERT INTO FUNCTION s3(s3_conn, filename='t_03363_function', format=Parquet, partition_strategy='hive') PARTITION BY (year, country) SELECT 2020 as year, 'Russia' as country, 1 as id; -``` - -```result -SELECT _path, * FROM s3(s3_conn, filename='t_03363_function/**.parquet'); - - ┌─_path──────────────────────────────────────────────────────────────────────┬─id─┬─country─┬─year─┐ -1. │ test/t_03363_function/year=2020/country=Russia/7351295896279887872.parquet │ 1 │ Russia │ 2020 │ - └────────────────────────────────────────────────────────────────────────────┴────┴─────────┴──────┘ -``` - -**Examples of `wildcard` partition strategy** - -1. Using partition ID in a key creates separate files: - -```sql -INSERT INTO TABLE FUNCTION - s3('http://bucket.amazonaws.com/my_bucket/file_{_partition_id}.csv', 'CSV', 'a String, b UInt32, c UInt32', partition_strategy='wildcard') - PARTITION BY a VALUES ('x', 2, 3), ('x', 4, 5), ('y', 11, 12), ('y', 13, 14), ('z', 21, 22), ('z', 23, 24); -``` -As a result, the data is written into three files: `file_x.csv`, `file_y.csv`, and `file_z.csv`. - -2. Using partition ID in a bucket name creates files in different buckets: - -```sql -INSERT INTO TABLE FUNCTION - s3('http://bucket.amazonaws.com/my_bucket_{_partition_id}/file.csv', 'CSV', 'a UInt32, b UInt32, c UInt32', partition_strategy='wildcard') - PARTITION BY a VALUES (1, 2, 3), (1, 4, 5), (10, 11, 12), (10, 13, 14), (20, 21, 22), (20, 23, 24); -``` -As a result, the data is written into three files in different buckets: `my_bucket_1/file.csv`, `my_bucket_10/file.csv`, and `my_bucket_20/file.csv`. - -## Accessing public buckets {#accessing-public-buckets} - -ClickHouse tries to fetch credentials from many different types of sources. -Sometimes, it can produce problems when accessing some buckets that are public causing the client to return `403` error code. -This issue can be avoided by using `NOSIGN` keyword, forcing the client to ignore all the credentials, and not sign the requests. - -```sql -SELECT * -FROM s3( - 'https://datasets-documentation.s3.eu-west-3.amazonaws.com/aapl_stock.csv', - NOSIGN, - 'CSVWithNames' -) -LIMIT 5; -``` - -## Using S3 credentials (ClickHouse Cloud) {#using-s3-credentials-clickhouse-cloud} - -For non-public buckets, users can pass an `aws_access_key_id` and `aws_secret_access_key` to the function. For example: - -```sql -SELECT count() FROM s3('https://datasets-documentation.s3.eu-west-3.amazonaws.com/mta/*.tsv', '', '','TSVWithNames') -``` - -This is appropriate for one-off accesses or in cases where credentials can easily be rotated. However, this is not recommended as a long-term solution for repeated access or where credentials are sensitive. In this case, we recommend users rely on role-based access. - -Role-based access for S3 in ClickHouse Cloud is documented [here](/products/cloud/guides/data-sources/accessing-s3-data-securely). - -Once configured, a `roleARN` can be passed to the s3 function via an `extra_credentials` parameter. For example: - -```sql -SELECT count() FROM s3('https://datasets-documentation.s3.eu-west-3.amazonaws.com/mta/*.tsv','CSVWithNames',extra_credentials(role_arn = 'arn:aws:iam::111111111111:role/ClickHouseAccessRole-001')) -``` - -An optional `external_id` can also be supplied alongside `role_arn`. It is passed as the `ExternalId` parameter of the AWS STS `AssumeRole` call and lets the role's trust policy require a shared secret, which mitigates the [confused deputy problem](https://docs.aws.amazon.com/IAM/latest/UserGuide/confused-deputy.html). For example: - -```sql -SELECT count() FROM s3('https://datasets-documentation.s3.eu-west-3.amazonaws.com/mta/*.tsv','CSVWithNames',extra_credentials(role_arn = 'arn:aws:iam::111111111111:role/ClickHouseAccessRole-001', external_id = 'my-external-id')) -``` - -Further examples can be found [here](/products/cloud/guides/data-sources/accessing-s3-data-securely#access-your-s3-bucket-with-the-clickhouseaccess-role) - -## Working with archives {#working-with-archives} - -Suppose that we have several archive files with following URIs on S3: - -- 'https://s3-us-west-1.amazonaws.com/umbrella-static/top-1m-2018-01-10.csv.zip' -- 'https://s3-us-west-1.amazonaws.com/umbrella-static/top-1m-2018-01-11.csv.zip' -- 'https://s3-us-west-1.amazonaws.com/umbrella-static/top-1m-2018-01-12.csv.zip' - -Extracting data from these archives is possible using ::. Globs can be used both in the url part as well as in the part after :: (responsible for the name of a file inside the archive). - -```sql -SELECT * -FROM s3( - 'https://s3-us-west-1.amazonaws.com/umbrella-static/top-1m-2018-01-1{0..2}.csv.zip :: *.csv', - NOSIGN -); -``` - - -ClickHouse supports three archive formats: -ZIP -TAR -7Z -While ZIP and TAR archives can be accessed from any supported storage location, 7Z archives can only be read from the local filesystem where ClickHouse is installed. - - -## Inserting Data {#inserting-data} - -Note that rows can only be inserted into new files. There are no merge cycles or file split operations. Once a file is written, subsequent inserts will fail. See more details [here](/integrations/connectors/data-ingestion/AWS/integrating-s3-with-clickhouse#inserting-data). - -## Virtual Columns {#virtual-columns} - -- `_path` — Path to the file. Type: `LowCardinality(String)`. In case of archive, shows path in a format: `"{path_to_archive}::{path_to_file_inside_archive}"` -- `_file` — Name of the file. Type: `LowCardinality(String)`. In case of archive shows name of the file inside the archive. -- `_size` — Size of the file in bytes. Type: `Nullable(UInt64)`. If the file size is unknown, the value is `NULL`. In case of archive shows uncompressed file size of the file inside the archive. -- `_time` — Last modified time of the file. Type: `Nullable(DateTime)`. If the time is unknown, the value is `NULL`. - -## use_hive_partitioning setting {#hive-style-partitioning} - -This is a hint for ClickHouse to parse hive style partitioned files upon reading time. It has no effect on writing. For symmetrical reads and writes, use the `partition_strategy` argument. - -When setting `use_hive_partitioning` is set to 1, ClickHouse will detect Hive-style partitioning in the path (`/name=value/`) and will allow to use partition columns as virtual columns in the query. These virtual columns will have the same names as in the partitioned path. - -**Example** - -```sql -SELECT * FROM s3('s3://data/path/date=*/country=*/code=*/*.parquet') WHERE date > '2020-01-01' AND country = 'Netherlands' AND code = 42; -``` - -## Accessing requester-pays buckets {#accessing-requester-pays-buckets} - -To access a requester-pays bucket, a header `x-amz-request-payer = requester` must be passed in any requests. This is achieved by passing the parameter `headers('x-amz-request-payer' = 'requester')` to the s3 function. For example: - -```sql -SELECT - count() AS num_rows, - uniqExact(_file) AS num_files -FROM s3('https://coiled-datasets-rp.s3.us-east-1.amazonaws.com/1trc/measurements-100*.parquet', 'AWS_ACCESS_KEY_ID', 'AWS_SECRET_ACCESS_KEY', headers('x-amz-request-payer' = 'requester')) - -┌───num_rows─┬─num_files─┐ -│ 1110000000 │ 111 │ -└────────────┴───────────┘ - -1 row in set. Elapsed: 3.089 sec. Processed 1.09 billion rows, 0.00 B (353.55 million rows/s., 0.00 B/s.) -Peak memory usage: 192.27 KiB. -``` - -## Resolving relative URLs {#resolving-relative-urls} - -The [s3_base](/reference/settings/session-settings/s3#s3_base) setting allows passing a relative URL to the `s3` function. When `s3_base` is set and the function argument has no scheme, it is resolved against the base URL per [RFC 3986](https://datatracker.ietf.org/doc/html/rfc3986), using the same rules as the [url_base](/reference/settings/session-settings/url#url_base) setting of the [url](/reference/functions/table-functions/url#resolving-relative-urls) function. Absolute URLs are passed through unchanged. - -The setting also applies to the [S3](/reference/engines/table-engines/integrations/s3) table engine and to the table functions sharing the `s3` configuration (`s3Cluster`, `gcs`, `oss`). For the `S3` table engine, the resolved URL is materialized into the stored table definition, so the table does not depend on the value of `s3_base` after creation. - -**Example** - -```sql -SET s3_base = 's3://clickhouse-public-datasets/'; -SELECT count() FROM s3('hits_compatible/hits.csv', NOSIGN); -``` - -## Storage Settings {#storage-settings} - -- [s3_truncate_on_insert](/reference/settings/session-settings/s3#s3_truncate_on_insert) - allows to truncate file before insert into it. Disabled by default. -- [s3_create_new_file_on_insert](/reference/settings/session-settings/s3#s3_create_new_file_on_insert) - allows to create a new file on each insert if format has suffix. Disabled by default. -- [s3_skip_empty_files](/reference/settings/session-settings/s3#s3_skip_empty_files) - allows to skip empty files while reading. Enabled by default. -- [s3_base](/reference/settings/session-settings/s3#s3_base) - base URL for resolving relative URLs passed to the `s3` function. Empty (disabled) by default. - -## Nested Avro Schemas {#nested-avro-schemas} - -When reading Avro files that contain **nested records** which diverge across files (for example, some files have an extra field inside a nested object), ClickHouse may return an error such as: - -> The number of leaves in record doesn't match the number of elements in tuple... - -This happens because ClickHouse expects all nested record structures to match the same schema. -To handle this scenario, you can: - -- Use `schema_inference_mode='union'` to merge different nested record schemas, or -- Manually align your nested structures and enable - `use_structure_from_insertion_table_in_table_functions=1`. - - -**Performance note** - -`schema_inference_mode='union'` may take longer on very large S3 datasets because it must scan each file to infer the schema. - - -**Example** -```sql -INSERT INTO data_stage -SELECT - id, - data -FROM s3('https://bucket-name/*.avro', 'Avro') -SETTINGS schema_inference_mode='union'; -``` - -## Related {#related} - -- [S3 engine](/reference/engines/table-engines/integrations/s3) -- [Integrating S3 with ClickHouse](/integrations/connectors/data-ingestion/AWS/integrating-s3-with-clickhouse) -)DOCS_MD", .category = FunctionDocumentation::Category::TableFunction}, - {.allow_readonly = false} - ); - - factory.registerFunction>( + factory.registerFunction>( {.description = R"DOCS_MD( Provides a table-like interface to `SELECT` and `INSERT` data from [Google Cloud Storage](https://cloud.google.com/storage/). Requires the [`Storage Object User` IAM role](https://cloud.google.com/storage/docs/access-control/iam-roles). @@ -984,7 +542,7 @@ As a result, the data is written into three files in different buckets: `my_buck {.allow_readonly = false} ); - factory.registerFunction>( + factory.registerFunction>( { .description=R"(The table function can be used to read the data stored on COSN.)", .examples{{COSNDefinition::name, "SELECT * FROM cosn(url, access_key_id, secret_access_key)", ""}}, @@ -993,7 +551,7 @@ As a result, the data is written into three files in different buckets: `my_buck {.allow_readonly = false} ); - factory.registerFunction>( + factory.registerFunction>( { .description=R"(The table function can be used to read the data stored on OSS.)", .examples{{OSSDefinition::name, "SELECT * FROM oss(url, access_key_id, secret_access_key)", ""}}, @@ -1002,1222 +560,71 @@ As a result, the data is written into three files in different buckets: `my_buck {.allow_readonly = false} ); #endif +} #if USE_AZURE_BLOB_STORAGE - factory.registerFunction>( - {.description = R"DOCS_MD( -import { ExperimentalBadge } from "/snippets/components/ExperimentalBadge/ExperimentalBadge.jsx"; -import { CloudNotSupportedBadge } from "/snippets/components/CloudNotSupportedBadge/CloudNotSupportedBadge.jsx"; +template class TableFunctionObjectStorage; +template class TableFunctionObjectStorage; +#endif -Provides a table-like interface to select/insert files in [Azure Blob Storage](https://azure.microsoft.com/en-us/products/storage/blobs). This table function is similar to the [s3 function](/reference/functions/table-functions/s3). +#if USE_AWS_S3 +template class TableFunctionObjectStorage; +template class TableFunctionObjectStorage; +template class TableFunctionObjectStorage; +template class TableFunctionObjectStorage; +template class TableFunctionObjectStorage; +#endif -## Syntax {#syntax} +#if USE_HDFS +template class TableFunctionObjectStorage; +template class TableFunctionObjectStorage; +#endif - - +#if USE_AVRO +template class TableFunctionObjectStorage; +#endif -Credentials are embedded in the connection string, so no separate `account_name`/`account_key` is needed: - -```sql -azureBlobStorage(connection_string, container_name, blobpath [, format, compression, partition_strategy, structure]) -``` - - - - -Requires `account_name` and `account_key` as separate arguments: - -```sql -azureBlobStorage(storage_account_url, container_name, blobpath, account_name, account_key [, format, compression, partition_strategy, structure]) -``` - - - - -See [Named Collections](#named-collections) below for the full list of supported keys: - -```sql -azureBlobStorage(named_collection[, option=value [,..]]) -``` - - - - -## Arguments {#arguments} - -| Argument | Description | -|----------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| -| `connection_string` | A connection string that includes embedded credentials (account name + account key or SAS token). When using this form, `account_name` and `account_key` should **not** be passed separately. See [Configure a connection string](https://learn.microsoft.com/en-us/azure/storage/common/storage-configure-connection-string?toc=%2Fazure%2Fstorage%2Fblobs%2Ftoc.json&bc=%2Fazure%2Fstorage%2Fblobs%2Fbreadcrumb%2Ftoc.json#configure-a-connection-string-for-an-azure-storage-account). | -| `storage_account_url` | The storage account endpoint URL, e.g. `https://myaccount.blob.core.windows.net/`. When using this form, you **must** also pass `account_name` and `account_key`. | -| `container_name` | Container name. | -| `blobpath` | File path. Supports the following wildcards in read-only mode: `*`, `**`, `?`, `{abc,def}` and `{N..M}` where `N`, `M` — numbers, `'abc'`, `'def'` — strings. | -| `account_name` | Storage account name. **Required** when using `storage_account_url` without SAS; must **not** be passed when using `connection_string`. | -| `account_key` | Storage account key. **Required** when using `storage_account_url` without SAS; must **not** be passed when using `connection_string`. | -| `format` | The [format](/reference/formats/index) of the file. | -| `compression` | Supported values: `none`, `gzip/gz`, `brotli/br`, `xz/LZMA`, `zstd/zst`. By default, it will autodetect compression by file extension (same as setting to `auto`). | -| `structure` | Structure of the table. Format `'column1_name column1_type, column2_name column2_type, ...'`. | -| `partition_strategy` | Optional. Supported values: `WILDCARD` or `HIVE`. `WILDCARD` requires a `{_partition_id}` in the path, which is replaced with the partition key. `HIVE` does not allow wildcards, assumes the path is the table root, and generates Hive-style partitioned directories with Snowflake IDs as filenames and the file format as the extension. Without an explicit strategy, a path with `{_partition_id}` uses `WILDCARD`. A path with another glob uses no partition strategy and ignores `PARTITION BY`. A path without a glob uses `HIVE` when `file_like_engine_default_partition_strategy` is `HIVE`; otherwise it uses no partition strategy. | -| `partition_columns_in_data_file` | Optional. Only used with `HIVE` partition strategy. Tells ClickHouse whether to expect partition columns to be written in the data file. Defaults `false`. | -| `extra_credentials` | Use `client_id` and `tenant_id` for authentication. If extra_credentials are provided, they are given priority over `account_name` and `account_key`. | - -## Named Collections {#named-collections} - -Arguments can also be passed using [named collections](/concepts/features/configuration/server-config/named-collections). In this case the following keys are supported: - -| Key | Required | Description | -|----------------------------------|----------|--------------------------------------------------------------------------------------------------------| -| `container` | Yes | Container name. Corresponds to the positional argument `container_name`. | -| `blob_path` | Yes | File path (with optional wildcards). Corresponds to the positional argument `blobpath`. | -| `connection_string` | No* | Connection string with embedded credentials. *Either `connection_string` or `storage_account_url` must be provided. | -| `storage_account_url` | No* | Storage account endpoint URL. *Either `connection_string` or `storage_account_url` must be provided. | -| `account_name` | No | Required when using `storage_account_url` | -| `account_key` | No | Required when using `storage_account_url` | -| `format` | No | File format. | -| `compression` | No | Compression type. | -| `structure` | No | Table structure. | -| `client_id` | No | Client ID for authentication. | -| `tenant_id` | No | Tenant ID for authentication. | - - -Named collection key names differ from positional function argument names: `container` (not `container_name`) and `blob_path` (not `blobpath`). - - -**Example:** - -```sql -CREATE NAMED COLLECTION azure_my_data AS - storage_account_url = 'https://myaccount.blob.core.windows.net/', - container = 'mycontainer', - blob_path = 'data/*.parquet', - account_name = 'myaccount', - account_key = 'mykey...==', - format = 'Parquet'; - -SELECT * -FROM azureBlobStorage(azure_my_data) -LIMIT 5; -``` - -You can also override named collection values at query time: - -```sql -SELECT * -FROM azureBlobStorage(azure_my_data, blob_path = 'other_data/*.csv', format = 'CSVWithNames') -LIMIT 5; -``` - -## Returned value {#returned-value} - -A table with the specified structure for reading or writing data in the specified file. - -## Examples {#examples} - -### Reading with `storage_account_url` form {#reading-with-storage-account-url} - -```sql -SELECT * -FROM azureBlobStorage( - 'https://myaccount.blob.core.windows.net/', - 'mycontainer', - 'data/*.parquet', - 'myaccount', - 'mykey...==', - 'Parquet' -) -LIMIT 5; -``` - -### Reading with `connection_string` form {#reading-with-connection-string} - -```sql -SELECT * -FROM azureBlobStorage( - 'DefaultEndpointsProtocol=https;AccountName=myaccount;AccountKey=mykey...==;EndPointSuffix=core.windows.net', - 'mycontainer', - 'data/*.csv', - 'CSVWithNames' -) -LIMIT 5; -``` - -### Writing with partitions {#writing-with-partitions} - -A path containing `{_partition_id}` implies the `WILDCARD` partition strategy. A path with another glob uses no partition strategy and ignores `PARTITION BY`. A path without a glob uses `HIVE` when `file_like_engine_default_partition_strategy` is `HIVE`; otherwise it uses no partition strategy. - -```sql -INSERT INTO TABLE FUNCTION azureBlobStorage( - 'DefaultEndpointsProtocol=https;AccountName=myaccount;AccountKey=mykey...==;EndPointSuffix=core.windows.net', - 'mycontainer', - 'test_{_partition_id}.csv', - 'CSV', - 'auto', - 'wildcard', - 'column1 UInt32, column2 UInt32, column3 UInt32' -) PARTITION BY column3 -VALUES (1, 2, 3), (3, 2, 1), (78, 43, 3); -``` - -Then read back a specific partition: - -```sql -SELECT * -FROM azureBlobStorage( - 'DefaultEndpointsProtocol=https;AccountName=myaccount;AccountKey=mykey...==;EndPointSuffix=core.windows.net', - 'mycontainer', - 'test_1.csv', - 'CSV', - 'auto', - 'column1 UInt32, column2 UInt32, column3 UInt32' -); -``` - -```response -┌─column1─┬─column2─┬─column3─┐ -│ 3 │ 2 │ 1 │ -└─────────┴─────────┴─────────┘ -``` - -## Virtual Columns {#virtual-columns} - -- `_path` — Path to the file. Type: `LowCardinality(String)`. -- `_file` — Name of the file. Type: `LowCardinality(String)`. -- `_size` — Size of the file in bytes. Type: `Nullable(UInt64)`. If the file size is unknown, the value is `NULL`. -- `_time` — Last modified time of the file. Type: `Nullable(DateTime)`. If the time is unknown, the value is `NULL`. - -## Partitioned Write {#partitioned-write} - -### Partition Strategy {#partition-strategy} - -Supported for INSERT queries only. - -`WILDCARD`: Replaces the `{_partition_id}` wildcard in the file path with the actual partition key. It is selected by default when the path contains `{_partition_id}`. - -When no `partition_strategy` is set, a path with another glob uses no partition strategy and ignores `PARTITION BY`. A path without a glob uses `HIVE` when `file_like_engine_default_partition_strategy` is `HIVE`; otherwise it uses no partition strategy. - -`HIVE` implements hive style partitioning for reads & writes. It generates files using the following format: `//.`. - -**Example of `HIVE` partition strategy** - -```sql -INSERT INTO TABLE FUNCTION azureBlobStorage( - azure_conf2, - storage_account_url = 'https://myaccount.blob.core.windows.net/', - container = 'cont', - blob_path = 'azure_table_root', - format = 'CSVWithNames', - compression = 'auto', - structure = 'year UInt16, country String, id Int32', - partition_strategy = 'hive' -) PARTITION BY (year, country) -VALUES (2020, 'Russia', 1), (2021, 'Brazil', 2); -``` - -```result -SELECT _path, * FROM azureBlobStorage( - azure_conf2, - storage_account_url = 'https://myaccount.blob.core.windows.net/', - container = 'cont', - blob_path = 'azure_table_root/**.csvwithnames' -) - - ┌─_path───────────────────────────────────────────────────────────────────────────┬─id─┬─year─┬─country─┐ -1. │ cont/azure_table_root/year=2021/country=Brazil/7351307847391293440.csvwithnames │ 2 │ 2021 │ Brazil │ -2. │ cont/azure_table_root/year=2020/country=Russia/7351307847378710528.csvwithnames │ 1 │ 2020 │ Russia │ - └─────────────────────────────────────────────────────────────────────────────────┴────┴──────┴─────────┘ -``` - -## use_hive_partitioning setting {#hive-style-partitioning} - -This is a hint for ClickHouse to parse hive style partitioned files upon reading time. It has no effect on writing. For symmetrical reads and writes, use the `partition_strategy` argument. - -When setting `use_hive_partitioning` is set to 1, ClickHouse will detect Hive-style partitioning in the path (`/name=value/`) and will allow to use partition columns as virtual columns in the query. These virtual columns will have the same names as in the partitioned path. - -**Example** - -Use virtual column, created with Hive-style partitioning - -```sql -SELECT * FROM azureBlobStorage(config, storage_account_url='...', container='...', blob_path='http://data/path/date=*/country=*/code=*/*.parquet') WHERE date > '2020-01-01' AND country = 'Netherlands' AND code = 42; -``` - -## Using Shared Access Signatures (SAS) {#using-shared-access-signatures-sas-sas-tokens} - -A Shared Access Signature (SAS) is a URI that grants restricted access to an Azure Storage container or file. Use it to provide time-limited access to storage account resources without sharing your storage account key. More details [here](https://learn.microsoft.com/en-us/rest/api/storageservices/delegate-access-with-shared-access-signature). - -The `azureBlobStorage` function supports Shared Access Signatures (SAS). - -A [Blob SAS token](https://learn.microsoft.com/en-us/azure/ai-services/translator/document-translation/how-to-guides/create-sas-tokens?tabs=Containers) contains all the information needed to authenticate the request, including the target blob, permissions, and validity period. To construct a blob URL, append the SAS token to the blob service endpoint. For example, if the endpoint is `https://clickhousedocstest.blob.core.windows.net/`, the request becomes: - -```sql -SELECT count() -FROM azureBlobStorage('BlobEndpoint=https://clickhousedocstest.blob.core.windows.net/;SharedAccessSignature=sp=r&st=2025-01-29T14:58:11Z&se=2025-01-29T22:58:11Z&spr=https&sv=2022-11-02&sr=c&sig=Ac2U0xl4tm%2Fp7m55IilWl1yHwk%2FJG0Uk6rMVuOiD0eE%3D', 'exampledatasets', 'example.csv') - -┌─count()─┐ -│ 10 │ -└─────────┘ - -1 row in set. Elapsed: 0.425 sec. -``` - -Alternatively, users can use the generated [Blob SAS URL](https://learn.microsoft.com/en-us/azure/ai-services/translator/document-translation/how-to-guides/create-sas-tokens?tabs=Containers): - -```sql -SELECT count() -FROM azureBlobStorage('https://clickhousedocstest.blob.core.windows.net/?sp=r&st=2025-01-29T14:58:11Z&se=2025-01-29T22:58:11Z&spr=https&sv=2022-11-02&sr=c&sig=Ac2U0xl4tm%2Fp7m55IilWl1yHwk%2FJG0Uk6rMVuOiD0eE%3D', 'exampledatasets', 'example.csv') - -┌─count()─┐ -│ 10 │ -└─────────┘ - -1 row in set. Elapsed: 0.153 sec. -``` - -## Related {#related} -- [AzureBlobStorage Table Engine](/reference/engines/table-engines/integrations/azureBlobStorage) -)DOCS_MD", .category = FunctionDocumentation::Category::TableFunction}, - {.allow_readonly = false} - ); -#endif -#if USE_HDFS - factory.registerFunction>( - {.description = R"DOCS_MD( -import { ExperimentalBadge } from "/snippets/components/ExperimentalBadge/ExperimentalBadge.jsx"; -import { CloudNotSupportedBadge } from "/snippets/components/CloudNotSupportedBadge/CloudNotSupportedBadge.jsx"; - -Creates a table from files in HDFS. This table function is similar to the [url](/reference/functions/table-functions/url) and [file](/reference/functions/table-functions/file) table functions. - -## Syntax {#syntax} - -```sql -hdfs(URI, format, structure) -``` - -## Arguments {#arguments} - -| Argument | Description | -|-----------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------| -| `URI` | The relative URI to the file in HDFS. Path to file support following globs in readonly mode: `*`, `?`, `{abc,def}` and `{N..M}` where `N`, `M` — numbers, `'abc', 'def'` — strings. | -| `format` | The [format](/reference/formats/index) of the file. | -| `structure`| Structure of the table. Format `'column1_name column1_type, column2_name column2_type, ...'`. | - -## Returned value {#returned-value} - -A table with the specified structure for reading or writing data in the specified file. - -**example** - -Table from `hdfs://hdfs1:9000/test` and selection of the first two rows from it: - -```sql -SELECT * -FROM hdfs('hdfs://hdfs1:9000/test', 'TSV', 'column1 UInt32, column2 UInt32, column3 UInt32') -LIMIT 2 -``` - -```text -┌─column1─┬─column2─┬─column3─┐ -│ 1 │ 2 │ 3 │ -│ 3 │ 2 │ 1 │ -└─────────┴─────────┴─────────┘ -``` - -## Globs in path {#globs-in-path} - -Paths may use globbing. Files must match the whole path pattern, not only the suffix or prefix. - -- `*` — Represents arbitrarily many characters except `/` but including the empty string. -- `**` — Represents all files inside a folder recursively. -- `?` — Represents an arbitrary single character. -- `{some_string,another_string,yet_another_one}` — Substitutes any of strings `'some_string', 'another_string', 'yet_another_one'`. The strings can contain the `/` symbol. -- `{N..M}` — Represents any number `>= N` and `<= M`. - -Constructions with `{}` are similar to the [remote](/reference/functions/table-functions/remote) and [file](/reference/functions/table-functions/file) table functions. - -**Example** - -1. Suppose that we have several files with following URIs on HDFS: - -- 'hdfs://hdfs1:9000/some_dir/some_file_1' -- 'hdfs://hdfs1:9000/some_dir/some_file_2' -- 'hdfs://hdfs1:9000/some_dir/some_file_3' -- 'hdfs://hdfs1:9000/another_dir/some_file_1' -- 'hdfs://hdfs1:9000/another_dir/some_file_2' -- 'hdfs://hdfs1:9000/another_dir/some_file_3' - -2. Query the amount of rows in these files: - -{/* */} - -```sql -SELECT count(*) -FROM hdfs('hdfs://hdfs1:9000/{some,another}_dir/some_file_{1..3}', 'TSV', 'name String, value UInt32') -``` - -3. Query the amount of rows in all files of these two directories: - -{/* */} - -```sql -SELECT count(*) -FROM hdfs('hdfs://hdfs1:9000/{some,another}_dir/*', 'TSV', 'name String, value UInt32') -``` - - -If your listing of files contains number ranges with leading zeros, use the construction with braces for each digit separately or use `?`. - - -**Example** - -Query the data from files named `file000`, `file001`, ... , `file999`: - -```sql -SELECT count(*) -FROM hdfs('hdfs://hdfs1:9000/big_dir/file{0..9}{0..9}{0..9}', 'CSV', 'name String, value UInt32') -``` - -## Virtual Columns {#virtual-columns} - -- `_path` — Path to the file. Type: `LowCardinality(String)`. -- `_file` — Name of the file. Type: `LowCardinality(String)`. -- `_size` — Size of the file in bytes. Type: `Nullable(UInt64)`. If the size is unknown, the value is `NULL`. -- `_time` — Last modified time of the file. Type: `Nullable(DateTime)`. If the time is unknown, the value is `NULL`. - -## use_hive_partitioning setting {#hive-style-partitioning} - -When setting `use_hive_partitioning` is set to 1, ClickHouse will detect Hive-style partitioning in the path (`/name=value/`) and will allow to use partition columns as virtual columns in the query. These virtual columns will have the same names as in the partitioned path. - -**Example** - -Use virtual column, created with Hive-style partitioning - -```sql -SELECT * FROM HDFS('hdfs://hdfs1:9000/data/path/date=*/country=*/code=*/*.parquet') WHERE date > '2020-01-01' AND country = 'Netherlands' AND code = 42; -``` - -## Storage Settings {#storage-settings} - -- [hdfs_truncate_on_insert](/reference/settings/session-settings/hdfs#hdfs_truncate_on_insert) - allows to truncate file before insert into it. Disabled by default. -- [hdfs_create_new_file_on_insert](/reference/settings/session-settings/hdfs#hdfs_create_new_file_on_insert) - allows to create a new file on each insert if format has suffix. Disabled by default. -- [hdfs_skip_empty_files](/reference/settings/session-settings/hdfs#hdfs_skip_empty_files) - allows to skip empty files while reading. Disabled by default. - -## Related {#related} - -- [Virtual columns](/reference/engines/table-engines/index#table_engines-virtual_columns) -)DOCS_MD", .category = FunctionDocumentation::Category::TableFunction}, - {.allow_readonly = false} - ); -#endif -} - -#if USE_AZURE_BLOB_STORAGE -template class TableFunctionObjectStorage; -template class TableFunctionObjectStorage; -#endif - -#if USE_AWS_S3 -template class TableFunctionObjectStorage; -template class TableFunctionObjectStorage; -template class TableFunctionObjectStorage; -template class TableFunctionObjectStorage; -template class TableFunctionObjectStorage; -#endif - -#if USE_HDFS -template class TableFunctionObjectStorage; -template class TableFunctionObjectStorage; -#endif - -#if USE_AVRO -template class TableFunctionObjectStorage; -#endif - -#if USE_AVRO && USE_AWS_S3 -template class TableFunctionObjectStorage; -template class TableFunctionObjectStorage; -#endif - -#if USE_AVRO && USE_AZURE_BLOB_STORAGE -template class TableFunctionObjectStorage; -#endif - -#if USE_AVRO && USE_HDFS -template class TableFunctionObjectStorage; -#endif - -#if USE_AVRO && USE_AWS_S3 -template class TableFunctionObjectStorage; -template class TableFunctionObjectStorage; -#endif - -#if USE_AVRO && USE_AZURE_BLOB_STORAGE -template class TableFunctionObjectStorage; -#endif - -#if USE_AVRO && USE_HDFS -template class TableFunctionObjectStorage; -#endif - -#if USE_PARQUET && USE_AWS_S3 && USE_DELTA_KERNEL_RS -template class TableFunctionObjectStorage; -template class TableFunctionObjectStorage; -#endif - -#if USE_PARQUET && USE_AZURE_BLOB_STORAGE && USE_DELTA_KERNEL_RS -template class TableFunctionObjectStorage; -#endif - -#if USE_AWS_S3 -template class TableFunctionObjectStorage; -#endif - -#if USE_AVRO -void registerTableFunctionIceberg(TableFunctionFactory & factory); -void registerTableFunctionIceberg(TableFunctionFactory & factory) -{ -#if USE_AWS_S3 - factory.registerFunction( - {.description = R"DOCS_MD( -Provides a table-like interface to Apache [Iceberg](https://iceberg.apache.org/) tables in Amazon S3, Azure, HDFS or locally stored. - -## Syntax {#syntax} - -```sql -icebergS3(url [, NOSIGN | access_key_id, secret_access_key, [session_token]] [,format] [,compression_method] [,extra_credentials]) -icebergS3(named_collection[, option=value [,..]]) - -icebergAzure(connection_string|storage_account_url, container_name, blobpath, [,account_name], [,account_key] [,format] [,compression_method]) -icebergAzure(named_collection[, option=value [,..]]) - -icebergHDFS(path_to_table, [,format] [,compression_method]) -icebergHDFS(named_collection[, option=value [,..]]) - -icebergLocal(path_to_table, [,format] [,compression_method]) -icebergLocal(named_collection[, option=value [,..]]) -``` - -## Arguments {#arguments} - -Description of the arguments coincides with description of arguments in table functions `s3`, `azureBlobStorage`, `HDFS` and `file` correspondingly. -`format` stands for the format of data files in the Iceberg table. - -For `icebergS3`, an optional `extra_credentials` parameter can be used to pass a `role_arn` for role-based access in ClickHouse Cloud. See [Secure S3](/products/cloud/guides/data-sources/accessing-s3-data-securely) for configuration steps. - -### Returned value {#returned-value} - -A table with the specified structure for reading data in the specified Iceberg table. - -### Example {#example} - -```sql -SELECT * FROM icebergS3('http://test.s3.amazonaws.com/clickhouse-bucket/test_table', 'test', 'test') -``` - - -ClickHouse supports reading v1 and v2 of the Iceberg format via the `icebergS3`, `icebergAzure`, `icebergHDFS` and `icebergLocal` table functions and `IcebergS3`, `IcebergAzure`, `IcebergHDFS` and `IcebergLocal` table engines. Support for v3 is partial; deletion vectors and manifest compaction aren't supported. - - -## Defining a named collection {#defining-a-named-collection} - -Here is an example of configuring a named collection for storing the URL and credentials: - -```xml - - - - http://test.s3.amazonaws.com/clickhouse-bucket/ - test - test - auto - auto - - - -``` - -```sql -SELECT * FROM icebergS3(iceberg_conf, filename = 'test_table') -DESCRIBE icebergS3(iceberg_conf, filename = 'test_table') -``` - -## Using a data catalog {#iceberg-writes-catalogs} - -Iceberg tables can also be used with various data catalogs, such as the [REST Catalog](https://iceberg.apache.org/rest-catalog-spec/), [AWS Glue Data Catalog](https://docs.aws.amazon.com/prescriptive-guidance/latest/serverless-etl-aws-glue/aws-glue-data-catalog.html) and [Unity Catalog](https://www.unitycatalog.io/). - - -When using a catalog, most users will want to use the `DataLakeCatalog` database engine, which connects ClickHouse to your catalog to discover your tables. You can use this database engine instead of manually creating individual tables with `IcebergS3` table engine. - - -To use them, create a table with the `IcebergS3` engine and provide the necessary settings. - -For example, using REST Catalog with MinIO storage: -```sql -CREATE TABLE `database_name.table_name` -ENGINE = IcebergS3( - 'http://minio:9000/warehouse-rest/table_name/', - 'minio_access_key', - 'minio_secret_key' -) -``` - -Or, using AWS Glue Data Catalog with S3: -```sql -CREATE TABLE `my_database.my_table` -ENGINE = IcebergS3( - 's3://my-data-bucket/warehouse/my_database/my_table/', - 'aws_access_key', - 'aws_secret_key' -) -``` - -## Schema Evolution {#schema-evolution} - -At the moment, with the help of CH, you can read iceberg tables, the schema of which has changed over time. We currently support reading tables where columns have been added and removed, and their order has changed. You can also change a column where a value is required to one where NULL is allowed. Additionally, we support permitted type casting for simple types, namely:   - -* int -> long -* float -> double -* decimal(P, S) -> decimal(P', S) where P' > P. - -Currently, it is not possible to change nested structures or the types of elements within arrays and maps. - -## Partition Pruning {#partition-pruning} - -ClickHouse supports partition pruning during SELECT queries for Iceberg tables, which helps optimize query performance by skipping irrelevant data files. To enable partition pruning, set `use_iceberg_partition_pruning = 1`. For more information about iceberg partition pruning address https://iceberg.apache.org/spec/#partitioning - -## Time Travel {#time-travel} - -ClickHouse supports time travel for Iceberg tables, allowing you to query historical data with a specific timestamp or snapshot ID. - -## Processing of tables with deleted rows {#deleted-rows} - -ClickHouse supports Iceberg tables with [position deletes](https://iceberg.apache.org/spec/#position-delete-files) and [equality deletes](https://iceberg.apache.org/spec/#equality-delete-files). Equality deletes are supported from v25.8. - -The following deletion method is **not supported**: -- [Deletion vectors](https://iceberg.apache.org/spec/#deletion-vectors) (introduced in v3) - -### Basic usage {#basic-usage} - - ```sql - SELECT * FROM example_table ORDER BY 1 - SETTINGS iceberg_timestamp_ms = 1714636800000 - ``` - - ```sql - SELECT * FROM example_table ORDER BY 1 - SETTINGS iceberg_snapshot_id = 3547395809148285433 - ``` - -Note: You cannot specify both `iceberg_timestamp_ms` and `iceberg_snapshot_id` parameters in the same query. - -### Important considerations {#important-considerations} - -* **Snapshots** are typically created when: -* New data is written to the table -* Some kind of data compaction is performed - -* **Schema changes typically don't create snapshots** - This leads to important behaviors when using time travel with tables that have undergone schema evolution. - -### Example scenarios {#example-scenarios} - -These scenarios use Spark to illustrate schema changes made by an external Iceberg writer. - -#### Scenario 1: Schema Changes Without New Snapshots {#scenario-1} - -Consider this sequence of operations: - - ```sql - -- Create a table with two columns - CREATE TABLE IF NOT EXISTS spark_catalog.db.time_travel_example ( - order_number bigint, - product_code string - ) - USING iceberg - OPTIONS ('format-version'='2') - -- - Insert data into the table - INSERT INTO spark_catalog.db.time_travel_example VALUES - (1, 'Mars') - - ts1 = now() // A piece of pseudo code - -- - Alter table to add a new column - ALTER TABLE spark_catalog.db.time_travel_example ADD COLUMN (price double) - - ts2 = now() - -- - Insert data into the table - INSERT INTO spark_catalog.db.time_travel_example VALUES (2, 'Venus', 100) - - ts3 = now() - -- - Query the table at each timestamp - SELECT * FROM spark_catalog.db.time_travel_example TIMESTAMP AS OF ts1; - -+------------+------------+ -|order_number|product_code| -+------------+------------+ -| 1| Mars| -+------------+------------+ - SELECT * FROM spark_catalog.db.time_travel_example TIMESTAMP AS OF ts2; - -+------------+------------+ -|order_number|product_code| -+------------+------------+ -| 1| Mars| -+------------+------------+ - - SELECT * FROM spark_catalog.db.time_travel_example TIMESTAMP AS OF ts3; - -+------------+------------+-----+ -|order_number|product_code|price| -+------------+------------+-----+ -| 1| Mars| NULL| -| 2| Venus|100.0| -+------------+------------+-----+ -``` - -Query results at different timestamps: - -* At ts1 & ts2: Only the original two columns appear -* At ts3: All three columns appear, with NULL for the price of the first row - -#### Scenario 2: Historical vs. Current Schema Differences {#scenario-2} - -A time travel query at a current moment might show a different schema than the current table: - -```sql --- Create a table - CREATE TABLE IF NOT EXISTS spark_catalog.db.time_travel_example_2 ( - order_number bigint, - product_code string - ) - USING iceberg - OPTIONS ('format-version'='2') - --- Insert initial data into the table - INSERT INTO spark_catalog.db.time_travel_example_2 VALUES (2, 'Venus'); - --- Alter table to add a new column - ALTER TABLE spark_catalog.db.time_travel_example_2 ADD COLUMN (price double); - - ts = now(); - --- Query the table at a current moment but using timestamp syntax - - SELECT * FROM spark_catalog.db.time_travel_example_2 TIMESTAMP AS OF ts; - - +------------+------------+ - |order_number|product_code| - +------------+------------+ - | 2| Venus| - +------------+------------+ - --- Query the table at a current moment - SELECT * FROM spark_catalog.db.time_travel_example_2; - +------------+------------+-----+ - |order_number|product_code|price| - +------------+------------+-----+ - | 2| Venus| NULL| - +------------+------------+-----+ -``` - -This happens because `ALTER TABLE` doesn't create a new snapshot but for the current table Spark takes value of `schema_id` from the latest metadata file, not a snapshot. - -#### Scenario 3: Historical vs. Current Schema Differences {#scenario-3} - -The second one is that while doing time travel you can't get state of table before any data was written to it: - -```sql --- Create a table - CREATE TABLE IF NOT EXISTS spark_catalog.db.time_travel_example_3 ( - order_number bigint, - product_code string - ) - USING iceberg - OPTIONS ('format-version'='2'); - - ts = now(); - --- Query the table at a specific timestamp - SELECT * FROM spark_catalog.db.time_travel_example_3 TIMESTAMP AS OF ts; -- Finises with error: Cannot find a snapshot older than ts. -``` - -In ClickHouse the behavior is consistent with Spark. You can mentally replace Spark Select queries with ClickHouse Select queries and it will work the same way. - -## Metadata File Resolution {#metadata-file-resolution} - -When using the `iceberg` table function in ClickHouse, the system needs to locate the correct metadata.json file that describes the Iceberg table structure. Here's how this resolution process works: - -### Candidate Search (in Priority Order) {#candidate-search} - -1. **Direct Path Specification**: -*If you set `iceberg_metadata_file_path`, the system will use this exact path by combining it with the Iceberg table directory path. -* When this setting is provided, all other resolution settings are ignored. - -2. **Table UUID Matching**: -*If `iceberg_metadata_table_uuid` is specified, the system will: - *Look only at `.metadata.json` files in the `metadata` directory - *Filter for files containing a `table-uuid` field matching your specified UUID (case-insensitive) - -3. **Default Search**: -*If neither of the above settings are provided, all `.metadata.json` files in the `metadata` directory become candidates - -### Selecting the Most Recent File {#most-recent-file} - -After identifying candidate files using the above rules, the system determines which one is the most recent: - -* If `iceberg_recent_metadata_file_by_last_updated_ms_field` is enabled: -* The file with the largest `last-updated-ms` value is selected - -* Otherwise: -* The file with the highest version number is selected -* (Version appears as `V` in filenames formatted as `V.metadata.json` or `V-uuid.metadata.json`) - -**Note**: All mentioned settings are table function settings (not global or query-level settings) and must be specified as shown below: - -```sql -SELECT * FROM iceberg('s3://bucket/path/to/iceberg_table', - SETTINGS iceberg_metadata_table_uuid = 'a90eed4c-f74b-4e5b-b630-096fb9d09021'); -``` - -**Note**: While Iceberg Catalogs typically handle metadata resolution, the `iceberg` table function in ClickHouse directly interprets files stored in S3 as Iceberg tables, which is why understanding these resolution rules is important. - -## Metadata cache {#metadata-cache} - -`Iceberg` table engine and table function support metadata cache storing the information of manifest files, manifest list and metadata json. The cache is stored in memory. This feature is controlled by setting `use_iceberg_metadata_files_cache`, which is enabled by default. - -## Aliases {#aliases} - -Table function `iceberg` is an alias to `icebergS3` now. - -## Virtual Columns {#virtual-columns} - -- `_path` — Path to the file. Type: `LowCardinality(String)`. -- `_file` — Name of the file. Type: `LowCardinality(String)`. -- `_size` — Size of the file in bytes. Type: `Nullable(UInt64)`. If the file size is unknown, the value is `NULL`. -- `_time` — Last modified time of the file. Type: `Nullable(DateTime)`. If the time is unknown, the value is `NULL`. -- `_etag` — The etag of the file. Type: `LowCardinality(String)`. If the etag is unknown, the value is `NULL`. - -## Writes into iceberg table {#writes-into-iceberg-table} - -Starting from version 25.7, ClickHouse supports modifications of Iceberg tables on writable storage backends. - -Before modifying or maintaining an Iceberg table, enable the [`allow_insert_into_iceberg` setting](/reference/settings/session-settings/allow#allow_insert_into_iceberg). Some operations require additional settings, as noted below: - -```sql -SET allow_insert_into_iceberg = 1; -``` - -### Creating table {#create-iceberg-table} - -To create a new standalone Iceberg table on a writable backend, use an Iceberg table engine and specify the schema explicitly. -Writes supports all data formats from iceberg specification, such as Parquet, Avro, ORC. - -### Example {#example-iceberg-writes-create} - -```sql -CREATE TABLE iceberg_writes_example -( - x Nullable(String), - y Nullable(Int32) -) -ENGINE = IcebergLocal('/home/scanhex12/iceberg_example/') -``` - -Note: To create a version hint file, enable the `iceberg_use_version_hint` setting. -If you want to compress the metadata.json file, specify the codec name in the `iceberg_metadata_compression_method` setting. - -### INSERT {#writes-inserts} - - - -After creating a new table, you can insert data using the usual ClickHouse syntax. - -### Example {#example-iceberg-writes-insert} - -```sql -INSERT INTO iceberg_writes_example VALUES ('Pavel', 777), ('Ivanov', 993); - -SELECT * -FROM iceberg_writes_example -FORMAT VERTICAL; - -Row 1: -────── -x: Pavel -y: 777 - -Row 2: -────── -x: Ivanov -y: 993 -``` - -### DELETE {#iceberg-writes-delete} - -Deleting extra rows in the merge-on-read format is also supported in ClickHouse. -This query will create a new snapshot with position delete files. - -### Example {#example-iceberg-writes-delete} - -```sql -ALTER TABLE iceberg_writes_example DELETE WHERE x != 'Ivanov'; - -SELECT * -FROM iceberg_writes_example -FORMAT VERTICAL; - -Row 1: -────── -x: Ivanov -y: 993 -``` - -### Schema evolution {#iceberg-writes-schema-evolution} - -ClickHouse allows you to add, drop, modify, or rename columns with simple types (non-tuple, non-array, non-map). - -### Example {#example-iceberg-writes-evolution} - -```sql -ALTER TABLE iceberg_writes_example MODIFY COLUMN y Nullable(Int64); -SHOW CREATE TABLE iceberg_writes_example; - - ┌─statement─────────────────────────────────────────────────┐ -1. │ CREATE TABLE default.iceberg_writes_example ↴│ - │↳( ↴│ - │↳ `x` Nullable(String), ↴│ - │↳ `y` Nullable(Int64) ↴│ - │↳) ↴│ - │↳ENGINE = IcebergLocal('/home/scanhex12/iceberg_example/') │ - └───────────────────────────────────────────────────────────┘ - -ALTER TABLE iceberg_writes_example ADD COLUMN z Nullable(Int32); -SHOW CREATE TABLE iceberg_writes_example; - - ┌─statement─────────────────────────────────────────────────┐ -1. │ CREATE TABLE default.iceberg_writes_example ↴│ - │↳( ↴│ - │↳ `x` Nullable(String), ↴│ - │↳ `y` Nullable(Int64), ↴│ - │↳ `z` Nullable(Int32) ↴│ - │↳) ↴│ - │↳ENGINE = IcebergLocal('/home/scanhex12/iceberg_example/') │ - └───────────────────────────────────────────────────────────┘ - -SELECT * -FROM iceberg_writes_example -FORMAT VERTICAL; - -Row 1: -────── -x: Ivanov -y: 993 -z: ᴺᵁᴸᴸ - -ALTER TABLE iceberg_writes_example DROP COLUMN z; -SHOW CREATE TABLE iceberg_writes_example; - ┌─statement─────────────────────────────────────────────────┐ -1. │ CREATE TABLE default.iceberg_writes_example ↴│ - │↳( ↴│ - │↳ `x` Nullable(String), ↴│ - │↳ `y` Nullable(Int64) ↴│ - │↳) ↴│ - │↳ENGINE = IcebergLocal('/home/scanhex12/iceberg_example/') │ - └───────────────────────────────────────────────────────────┘ - -SELECT * -FROM iceberg_writes_example -FORMAT VERTICAL; - -Row 1: -────── -x: Ivanov -y: 993 - -ALTER TABLE iceberg_writes_example RENAME COLUMN y TO value; -SHOW CREATE TABLE iceberg_writes_example; - - ┌─statement─────────────────────────────────────────────────┐ -1. │ CREATE TABLE default.iceberg_writes_example ↴│ - │↳( ↴│ - │↳ `x` Nullable(String), ↴│ - │↳ `value` Nullable(Int64) ↴│ - │↳) ↴│ - │↳ENGINE = IcebergLocal('/home/scanhex12/iceberg_example/') │ - └───────────────────────────────────────────────────────────┘ - -SELECT * -FROM iceberg_writes_example -FORMAT VERTICAL; - -Row 1: -────── -x: Ivanov -value: 993 -``` - -### Compaction {#iceberg-writes-compaction} - -ClickHouse supports compaction iceberg table. Currently, it can merge position delete files into data files while updating metadata. Previous snapshot IDs and timestamps remain unchanged, so the time-travel feature can still be used with the same values. - -How to use it: - -```sql -SET allow_experimental_iceberg_compaction = 1 - -OPTIMIZE TABLE iceberg_writes_example; - -SELECT * -FROM iceberg_writes_example -FORMAT VERTICAL; - -Row 1: -────── -x: Ivanov -y: 993 -``` - -### Expire Snapshots {#iceberg-expire-snapshots} - -Iceberg tables accumulate snapshots with each INSERT, DELETE, or UPDATE operation. Over time, this can lead to a large number of snapshots and associated data files. The `expire_snapshots` command removes old snapshots and cleans up data files that are no longer referenced by any retained snapshot. - -**Syntax:** - -```sql -ALTER TABLE iceberg_table EXECUTE expire_snapshots( - ['timestamp'] - [, expire_before = 'timestamp'] - [, retention_period = '3d'] - [, retain_last = 100] - [, snapshot_ids = [1, 2, 3, 4]] - [, dry_run = 1] -); -``` - -By default, which snapshots to keep is determined by the [retention policy](#iceberg-snapshot-retention-policy) (table properties `min-snapshots-to-keep`, `max-snapshot-age-ms`, and per-ref overrides). When `snapshot_ids` is specified, the retention policy is bypassed and only the listed snapshots are considered for expiration. - -**Arguments:** - -- `'timestamp'` (positional) or `expire_before = 'timestamp'` — a datetime string (e.g., `'2024-06-01 00:00:00'`) interpreted in the **server's timezone**. Acts as a safety fuse: snapshots whose `timestamp-ms` is at or after this value are protected from expiration, even if the retention policy would otherwise expire them. Can be combined with `snapshot_ids`, in which case listed snapshots at or newer than the timestamp are not expired. -- `retention_period = ''` — overrides the table-level `history.expire.max-snapshot-age-ms` for this invocation only. Snapshots older than this duration (measured from now) become candidates for expiration. The value is a duration string consisting of one or more `{number}{unit}` pairs concatenated together. Supported units: `y` (365 days), `w` (7 days), `d` (24 hours), `h` (60 minutes), `m` (60 seconds), `s` (1 second), `ms` (1 millisecond). Units can be combined, e.g. `'3d'`, `'12h'`, `'1d12h30m'`, `'500ms'`. -- `retain_last = N` — overrides the table-level `history.expire.min-snapshots-to-keep` for this invocation only. At least `N` snapshots are always retained regardless of age. -- `snapshot_ids = [id1, id2, ...]` — expires exactly the listed snapshot IDs (except snapshots referenced by current snapshot, branches, or tags). This mode bypasses the retention policy entirely and cannot be combined with `retention_period` or `retain_last`. -- `dry_run = 1` — computes what would be expired and returns metrics without writing new metadata or deleting files. - - -`retention_period` and `retain_last` override only the **table-level** retention defaults. Per-ref (branch/tag) retention overrides configured in the Iceberg table properties (e.g., `refs..min-snapshots-to-keep`) are never overridden — they always take effect as specified in the table metadata. - - -**Example:** - -```sql -SET allow_insert_into_iceberg = 1; - --- Create some snapshots by inserting data -INSERT INTO iceberg_table VALUES (1); -INSERT INTO iceberg_table VALUES (2); -INSERT INTO iceberg_table VALUES (3); - --- Expire using retention policy only -ALTER TABLE iceberg_table EXECUTE expire_snapshots(); - --- Expire with a safety fuse: protect snapshots newer than the timestamp (positional syntax) -ALTER TABLE iceberg_table EXECUTE expire_snapshots('2025-01-01 00:00:00'); - --- Same using the named argument form -ALTER TABLE iceberg_table EXECUTE expire_snapshots(expire_before = '2025-01-01 00:00:00'); - --- Override retention parameters for one execution -ALTER TABLE iceberg_table EXECUTE expire_snapshots(retention_period = '3d', retain_last = 10); - --- Expire explicit snapshots -ALTER TABLE iceberg_table EXECUTE expire_snapshots(snapshot_ids = [101, 102, 103]); - --- Dry-run preview (no metadata updates, no file deletes) -ALTER TABLE iceberg_table EXECUTE expire_snapshots(retention_period = '1d', dry_run = 1); -``` - -**Output:** - -The command returns a table with two columns (`metric_name String`, `metric_value Int64`) containing one row per metric. The metric names follow the [Iceberg spec](https://iceberg.apache.org/docs/latest/spark-procedures/#output): - -| metric_name | Description | -|---|---| -| `deleted_data_files_count` | Number of data files deleted | -| `deleted_position_delete_files_count` | Number of position delete files deleted | -| `deleted_equality_delete_files_count` | Number of equality delete files deleted | -| `deleted_manifest_files_count` | Number of manifest files deleted | -| `deleted_manifest_lists_count` | Number of manifest list files deleted | -| `deleted_statistics_files_count` | Number of statistics files deleted (always 0 currently) | -| `dry_run` | `1` for dry-run mode, `0` for normal execution | - -The command performs the following steps: - -1. Evaluates the retention policy (see below) to determine which snapshots must be preserved -2. If a timestamp argument was provided, additionally protects all snapshots at or newer than that timestamp -3. Expires snapshots that are neither retained by the policy nor protected by the timestamp fuse -4. Computes which files are exclusively associated with expired snapshots -5. In normal mode: generates new metadata without the expired snapshots -6. In normal mode: physically deletes unreachable manifest lists, manifest files, and data files -7. In `dry_run = 1` mode: skips steps 5 and 6 and only returns the calculated metrics - -#### Snapshot Retention Policy {#iceberg-snapshot-retention-policy} - -The `expire_snapshots` command respects the [Iceberg snapshot retention policy](https://iceberg.apache.org/spec/#snapshot-retention-policy). Retention is configured via Iceberg table properties and per-reference overrides: - -| Property | Scope | Default | Description | -|---|---|---|---| -| `history.expire.min-snapshots-to-keep` | Table | `iceberg_expire_default_min_snapshots_to_keep` (default `1`) | Minimum number of snapshots to keep in each branch's ancestor chain | -| `history.expire.max-snapshot-age-ms` | Table | `iceberg_expire_default_max_snapshot_age_ms` (default `432000000`, 5 days) | Maximum age (in ms) of snapshots to retain in a branch | -| `history.expire.max-ref-age-ms` | Table | `iceberg_expire_default_max_ref_age_ms` (default `∞`) | Maximum age (in ms) for a snapshot reference (branch or tag) before the reference itself is removed | - -Each snapshot reference (`refs` in the Iceberg metadata) can override these with per-ref fields: `min-snapshots-to-keep`, `max-snapshot-age-ms`, and `max-ref-age-ms`. - -**Retention evaluation:** - -- **For each branch** (including `main`): the ancestor chain is walked starting from the branch head. Snapshots are retained while either of these conditions is true: - - The snapshot is one of the first `min-snapshots-to-keep` in the chain - - The snapshot's age is within `max-snapshot-age-ms` (i.e., `now - timestamp-ms <= max-snapshot-age-ms`) -- **For tags**: the tagged snapshot is retained unless the tag has exceeded its `max-ref-age-ms`, in which case the tag reference is removed -- **Non-main references** whose age exceeds `max-ref-age-ms` are removed entirely (the `main` branch is never removed) -- **Dangling references** that point to non-existent snapshots are removed with a warning -- **The current snapshot is always preserved**, regardless of retention settings - -**Required privileges:** - -The `ALTER TABLE EXECUTE` privilege is required, which is a child of `ALTER TABLE` in the ClickHouse access control hierarchy. You can grant it specifically or via the parent: - -```sql --- Grant only EXECUTE permission -GRANT ALTER TABLE EXECUTE ON my_iceberg_table TO my_user; - --- Or grant all ALTER TABLE permissions (includes ALTER TABLE EXECUTE) -GRANT ALTER TABLE ON my_iceberg_table TO my_user; -``` - - -- Only Iceberg format version 2 tables are supported (v1 snapshots do not guarantee `manifest-list`, which is required to safely identify files for cleanup) -- The current snapshot is always preserved, even if it is older than the specified timestamp -- Requires the `allow_insert_into_iceberg` setting to be enabled -- Requires the `allow_experimental_expire_snapshots` setting to be enabled -- The catalog's own authorization (REST catalog auth, AWS Glue IAM, etc.) is enforced independently when ClickHouse updates the metadata - - -### Remove Orphan Files {#iceberg-remove-orphan-files} - -Orphan files are files on storage that are not referenced by any snapshot in the Iceberg table metadata. They accumulate from failed writes, partial cleanup after compaction, and interrupted operations, causing unbounded storage growth. The `remove_orphan_files` command identifies and removes these orphan files. - -**Syntax:** - -```sql --- Positional form: single unnamed older_than argument -ALTER TABLE iceberg_table EXECUTE remove_orphan_files('timestamp') - --- Named form -ALTER TABLE iceberg_table EXECUTE remove_orphan_files( - older_than = 'timestamp', - location = 'path', - dry_run = 0|1 -) - --- No arguments: use all defaults (older_than = 3 days ago) -ALTER TABLE iceberg_table EXECUTE remove_orphan_files() -``` - -**Parameters:** - -| Parameter | Type | Default | Description | -|---|---|---|---| -| `older_than` | `String` (timestamp) | 3 days ago (configurable via `iceberg_orphan_files_older_than_seconds`) | Only consider files with a last-modified time older than this timestamp as orphan candidates. Safety guard against deleting files from in-progress writes. | -| `location` | `String` | Table location | Restrict the scan to a specific subdirectory under the table location (e.g., `'data/'` or `'metadata/'`). | -| `dry_run` | `UInt64` | `0` | When `1`, identify orphan files and return the result summary without actually deleting anything. | +#if USE_AVRO +template class TableFunctionObjectStorage; +#endif -**Examples:** +#if USE_AVRO && USE_AWS_S3 +template class TableFunctionObjectStorage; +template class TableFunctionObjectStorage; +#endif -```sql --- Remove orphan files older than a specific timestamp -ALTER TABLE iceberg_table EXECUTE remove_orphan_files('2026-03-01 00:00:00'); - --- Dry run: preview which files would be deleted -ALTER TABLE iceberg_table EXECUTE remove_orphan_files(dry_run = 1); - --- Scan only the data directory -ALTER TABLE iceberg_table EXECUTE remove_orphan_files( - older_than = '2026-03-01 00:00:00', - location = 'data/' -); - --- Combine positional older_than with named arguments -ALTER TABLE iceberg_table EXECUTE remove_orphan_files( - '2026-03-01 00:00:00', - dry_run = 1 -); -``` +#if USE_AVRO && USE_AZURE_BLOB_STORAGE +template class TableFunctionObjectStorage; +#endif -**Output:** - -The command returns a table with `metric_name` and `metric_value` columns showing the count of deleted (or would-be-deleted in dry_run mode) files by category. File categories are classified using best-effort heuristics based on file naming conventions; files that do not match any specific pattern default to `deleted_data_files_count`: - -| metric_name | metric_value | -|---|---| -| deleted_data_files_count | 5 | -| deleted_position_delete_files_count | 2 | -| deleted_equality_delete_files_count | 0 | -| deleted_manifest_files_count | 3 | -| deleted_manifest_lists_count | 1 | -| deleted_metadata_files_count | 0 | -| deleted_statistics_files_count | 0 | -| skipped_missing_metadata_count | 0 | -| failed_deletions_count | 0 | - -**Settings:** - -| Setting | Type | Default | Description | -|---|---|---|---| -| `allow_iceberg_remove_orphan_files` | `Bool` | `false` | Gate setting to enable the feature (experimental). | -| `iceberg_orphan_files_older_than_seconds` | `UInt64` | `259200` (3 days) | Default `older_than` threshold in seconds when the argument is omitted. | - - -- **Requires Iceberg format version 2 (or higher).** Version 1 tables are rejected because they lack `manifest-list` pointers in snapshots, which are needed to safely determine the reachable file set. Running the command on a v1 table returns a `BAD_ARGUMENTS` error. -- Requires both `allow_insert_into_iceberg` and `allow_iceberg_remove_orphan_files` settings to be enabled -- It is recommended to run `expire_snapshots` before `remove_orphan_files` so that files uniquely referenced by expired snapshots are cleaned up first -- Use `dry_run = 1` to preview orphan files before deletion -- The `older_than` threshold protects against deleting files from in-progress writes — the default 3-day threshold provides a generous safety margin - +#if USE_AVRO && USE_HDFS +template class TableFunctionObjectStorage; +#endif -## See Also {#see-also} +#if USE_AVRO && USE_AWS_S3 +template class TableFunctionObjectStorage; +template class TableFunctionObjectStorage; +#endif -* [Iceberg engine](/reference/engines/table-engines/integrations/iceberg) -* [Iceberg cluster table function](/reference/functions/table-functions/icebergCluster) -)DOCS_MD", .category = FunctionDocumentation::Category::TableFunction}, - {.allow_readonly = false}); - factory.registerFunction( - {.description = R"(The table function can be used to read from and insert into an existing Iceberg table stored on S3 object storage.)", - .examples{{IcebergS3Definition::name, "SELECT * FROM icebergS3(url, access_key_id, secret_access_key)", ""}}, - .category = FunctionDocumentation::Category::TableFunction}, - {.allow_readonly = false}); +#if USE_AVRO && USE_AZURE_BLOB_STORAGE +template class TableFunctionObjectStorage; +#endif +#if USE_AVRO && USE_HDFS +template class TableFunctionObjectStorage; #endif -#if USE_AZURE_BLOB_STORAGE - factory.registerFunction( - {.description = R"(The table function can be used to read from and insert into an existing Iceberg table stored on Azure object storage.)", - .examples{{IcebergAzureDefinition::name, "SELECT * FROM icebergAzure(url, access_key_id, secret_access_key)", ""}}, - .category = FunctionDocumentation::Category::TableFunction}, - {.allow_readonly = false}); + +#if USE_PARQUET && USE_AWS_S3 && USE_DELTA_KERNEL_RS +template class TableFunctionObjectStorage; +template class TableFunctionObjectStorage; #endif -#if USE_HDFS - factory.registerFunction( - {.description = R"(The table function can be used to read the Iceberg table stored on HDFS virtual filesystem.)", - .examples{{IcebergHDFSDefinition::name, "SELECT * FROM icebergHDFS(url)", ""}}, - .category = FunctionDocumentation::Category::TableFunction}, - {.allow_readonly = false}); + +#if USE_PARQUET && USE_AZURE_BLOB_STORAGE && USE_DELTA_KERNEL_RS +template class TableFunctionObjectStorage; #endif - factory.registerFunction( - {.description = R"(The table function can be used to read from and insert into an existing Iceberg table stored locally.)", - .examples{{IcebergLocalDefinition::name, "SELECT * FROM icebergLocal(filename)", ""}}, - .category = FunctionDocumentation::Category::TableFunction}, - {.allow_readonly = false}); -} + +#if USE_AWS_S3 +template class TableFunctionObjectStorage; #endif @@ -2371,128 +778,6 @@ Data types supported in Paimon partition keys: void registerTableFunctionDeltaLake(TableFunctionFactory & factory); void registerTableFunctionDeltaLake(TableFunctionFactory & factory) { -#if USE_AWS_S3 - factory.registerFunction( - {.description = R"DOCS_MD( -Provides a table-like interface to [Delta Lake](https://github.com/delta-io/delta) tables in Amazon S3, Azure Blob Storage, or a locally mounted file system, supporting both reads and writes (from v25.10) - -## Syntax {#syntax} - -`deltaLake` is an alias of `deltaLakeS3` which is supported for compatibility. - -```sql -deltaLake(url [,aws_access_key_id, aws_secret_access_key] [,format] [,structure] [,compression] [,extra_credentials]) - -deltaLakeS3(url [,aws_access_key_id, aws_secret_access_key] [,format] [,structure] [,compression] [,extra_credentials]) - -deltaLakeAzure(connection_string|storage_account_url, container_name, blobpath, [,account_name], [,account_key] [,format] [,compression_method]) - -deltaLakeLocal(path, [,format]) -``` - -## Arguments {#arguments} - -The arguments for this table function are the same as for the `s3`, `azureBlobStorage`, `HDFS` and `file` table functions respectively. -The `format` argument stands for the format of data files in the Delta lake table. - -An optional `extra_credentials` parameter can be used to pass a `role_arn` for role-based access in ClickHouse Cloud. See [Secure S3](/products/cloud/guides/data-sources/accessing-s3-data-securely) for configuration steps. - -## Returned value {#returned-value} - -Returns a table with the specified structure for reading or writing data from/to the specified Delta Lake table. - -## Examples {#examples} - -### Reading data {#reading-data} - -Consider a table in S3 storage at `https://clickhouse-public-datasets.s3.amazonaws.com/delta_lake/hits/`. -To read data from the table in ClickHouse, run: - -```sql title="Query" -SELECT - URL, - UserAgent -FROM deltaLake('https://clickhouse-public-datasets.s3.amazonaws.com/delta_lake/hits/') -WHERE URL IS NOT NULL -LIMIT 2 -``` - -```response title="Response" -┌─URL───────────────────────────────────────────────────────────────────┬─UserAgent─┐ -│ http://auto.ria.ua/search/index.kz/jobinmoscow/detail/55089/hasimages │ 1 │ -│ http://auto.ria.ua/search/index.kz/jobinmoscow.ru/gosushi │ 1 │ -└───────────────────────────────────────────────────────────────────────┴───────────┘ -``` - -### Inserting data {#inserting-data} - -Consider a table in S3 storage at `s3://ch-docs-s3-bucket/people_10k/`. -Delta Lake writes are a Beta feature disabled by default. Enable them with the following (`allow_delta_lake_writes` is available from version 26.7; on earlier versions use `allow_experimental_delta_lake_writes`): - -```sql title="Query" -SET allow_delta_lake_writes=1 -``` - -Then write: - -```sql title="Query" -INSERT INTO TABLE FUNCTION deltaLake('s3://ch-docs-s3-bucket/people_10k/', '', '') VALUES (10001, 'John', 'Smith', 'Male', 30) -``` - -```response title="Response" -Query id: 09069b47-89fa-4660-9e42-3d8b1dde9b17 - -Ok. - -1 row in set. Elapsed: 3.426 sec. -``` - -You can confirm the insert worked by reading the table again: - -```sql title="Query" -SELECT * -FROM deltaLake('s3://ch-docs-s3-bucket/people_10k/', '', '') -WHERE (firstname = 'John') AND (lastname = 'Smith') -``` - -```response title="Response" -Query id: 65032944-bed6-4d45-86b3-a71205a2b659 - - ┌────id─┬─firstname─┬─lastname─┬─gender─┬─age─┐ -1. │ 10001 │ John │ Smith │ Male │ 30 │ - └───────┴───────────┴──────────┴────────┴─────┘ -``` - -## Virtual Columns {#virtual-columns} - -- `_path` — Path to the file. Type: `LowCardinality(String)`. -- `_file` — Name of the file. Type: `LowCardinality(String)`. -- `_size` — Size of the file in bytes. Type: `Nullable(UInt64)`. If the file size is unknown, the value is `NULL`. -- `_time` — Last modified time of the file. Type: `Nullable(DateTime)`. If the time is unknown, the value is `NULL`. -- `_etag` — The etag of the file. Type: `LowCardinality(String)`. If the etag is unknown, the value is `NULL`. - -## Related {#related} - -- [DeltaLake engine](/reference/engines/table-engines/integrations/deltalake) -- [DeltaLake cluster table function](/reference/functions/table-functions/deltalakeCluster) -)DOCS_MD", .category = FunctionDocumentation::Category::TableFunction}, - {.allow_readonly = false}); - - factory.registerFunction( - {.description = R"(The table function can be used to read the DeltaLake table stored on S3.)", - .examples{{DeltaLakeS3Definition::name, "SELECT * FROM deltaLakeS3(url, access_key_id, secret_access_key)", ""}}, - .category = FunctionDocumentation::Category::TableFunction}, - {.allow_readonly = false}); -#endif - -#if USE_AZURE_BLOB_STORAGE - factory.registerFunction( - {.description = R"(The table function can be used to read the DeltaLake table stored on Azure object store.)", - .examples{{DeltaLakeAzureDefinition::name, "SELECT * FROM deltaLakeAzure(connection_string|storage_account_url, container_name, blobpath, \"\n" - " \"[account_name, account_key, format, compression, structure])", ""}}, - .category = FunctionDocumentation::Category::TableFunction}, - {.allow_readonly = false}); -#endif // Register the new local Delta Lake table function factory.registerFunction( {.description = R"(The table function can be used to read the DeltaLake table stored locally.)", @@ -2502,68 +787,15 @@ Query id: 65032944-bed6-4d45-86b3-a71205a2b659 } #endif -#if USE_AWS_S3 -void registerTableFunctionHudi(TableFunctionFactory & factory); -void registerTableFunctionHudi(TableFunctionFactory & factory) -{ - factory.registerFunction( - {.description = R"DOCS_MD( -Provides a read-only table-like interface to Apache [Hudi](https://hudi.apache.org/) tables in Amazon S3. - -## Syntax {#syntax} - -```sql -hudi(url [,aws_access_key_id, aws_secret_access_key] [,format] [,structure] [,compression] [,extra_credentials]) -``` - -## Arguments {#arguments} - -| Argument | Description | -|----------------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| -| `url` | Bucket url with the path to an existing Hudi table in S3. | -| `aws_access_key_id`, `aws_secret_access_key` | Long-term credentials for the [AWS](https://aws.amazon.com/) account user. You can use these to authenticate your requests. These parameters are optional. If credentials are not specified, they are used from the ClickHouse configuration. For more information see [Using S3 for Data Storage](/reference/engines/table-engines/mergetree-family/mergetree#table_engine-mergetree-s3). | -| `format` | The [format](/reference/formats/index) of the file. | -| `structure` | Structure of the table. Format `'column1_name column1_type, column2_name column2_type, ...'`. | -| `compression` | Parameter is optional. Supported values: `none`, `gzip/gz`, `brotli/br`, `xz/LZMA`, `zstd/zst`. By default, compression will be autodetected by the file extension. | -| `extra_credentials` | Parameter is optional. Used to pass a `role_arn` for role-based access in ClickHouse Cloud. See [Secure S3](/products/cloud/guides/data-sources/accessing-s3-data-securely) for configuration steps. | - -## Returned value {#returned-value} - -A table with the specified structure for reading data in the specified Hudi table in S3. - -## Virtual Columns {#virtual-columns} - -- `_path` — Path to the file. Type: `LowCardinality(String)`. -- `_file` — Name of the file. Type: `LowCardinality(String)`. -- `_size` — Size of the file in bytes. Type: `Nullable(UInt64)`. If the file size is unknown, the value is `NULL`. -- `_time` — Last modified time of the file. Type: `Nullable(DateTime)`. If the time is unknown, the value is `NULL`. -- `_etag` — The etag of the file. Type: `LowCardinality(String)`. If the etag is unknown, the value is `NULL`. - -## Related {#related} - -- [Hudi engine](/reference/engines/table-engines/integrations/hudi) -- [Hudi cluster table function](/reference/functions/table-functions/hudiCluster) -)DOCS_MD", .category = FunctionDocumentation::Category::TableFunction}, - {.allow_readonly = false}); -} -#endif - void registerDataLakeTableFunctions(TableFunctionFactory & factory) { UNUSED(factory); -#if USE_AVRO - registerTableFunctionIceberg(factory); -#endif - -#if USE_AVRO - registerTableFunctionPaimon(factory); -#endif #if USE_PARQUET && USE_DELTA_KERNEL_RS registerTableFunctionDeltaLake(factory); #endif -#if USE_AWS_S3 - registerTableFunctionHudi(factory); +#if USE_AVRO + registerTableFunctionPaimon(factory); #endif } } diff --git a/src/TableFunctions/TableFunctionObjectStorage.h b/src/TableFunctions/TableFunctionObjectStorage.h index 24e6597779a6..a15395d11e68 100644 --- a/src/TableFunctions/TableFunctionObjectStorage.h +++ b/src/TableFunctions/TableFunctionObjectStorage.h @@ -25,10 +25,12 @@ struct S3StorageSettings; struct AzureStorageSettings; struct HDFSStorageSettings; -template +template class TableFunctionObjectStorage : public ITableFunction { public: + using Configuration = StorageConfiguration; + static constexpr auto name = Definition::name; using Settings = typename std::conditional_t< is_data_lake, @@ -37,15 +39,16 @@ class TableFunctionObjectStorage : public ITableFunction String getName() const override { return name; } - bool hasStaticStructure() const override { return configuration->structure != "auto"; } + bool hasStaticStructure() const override { return configuration->getStructure() != "auto"; } - bool needStructureHint() const override { return configuration->structure == "auto"; } + bool needStructureHint() const override { return configuration->getStructure() == "auto"; } void setStructureHint(const ColumnsDescription & structure_hint_) override { structure_hint = structure_hint_; } bool supportsReadingSubsetOfColumns(const ContextPtr & context) override { - return configuration->format != "auto" && FormatFactory::instance().checkIfFormatSupportsSubsetOfColumns(configuration->format, context); + return configuration->getFormat() != "auto" + && FormatFactory::instance().checkIfFormatSupportsSubsetOfColumns(configuration->getFormat(), context); } NameSet getVirtualsToCheckBeforeUsingStructureHint() const override @@ -55,7 +58,7 @@ class TableFunctionObjectStorage : public ITableFunction virtual void parseArgumentsImpl(ASTs & args, const ContextPtr & context) { - StorageObjectStorageConfiguration::initialize(*getConfiguration(context), args, context, true); + getConfiguration(context)->initialize(args, context, true); } static void updateStructureAndFormatArgumentsIfNeeded( @@ -67,8 +70,8 @@ class TableFunctionObjectStorage : public ITableFunction if constexpr (is_data_lake) { Configuration configuration(createEmptySettings()); - if (configuration.format == "auto") - configuration.format = "Parquet"; /// Default format of data lakes. + if (configuration.getFormat() == "auto") + configuration.setFormat("Parquet"); /// Default format of data lakes. configuration.addStructureAndFormatToArgsIfNeeded(args, structure, format, context, /*with_structure=*/true); } @@ -110,21 +113,22 @@ class TableFunctionObjectStorage : public ITableFunction }; #if USE_AWS_S3 -using TableFunctionS3 = TableFunctionObjectStorage; +using TableFunctionS3 = TableFunctionObjectStorage; #endif #if USE_AZURE_BLOB_STORAGE -using TableFunctionAzureBlob = TableFunctionObjectStorage; +using TableFunctionAzureBlob = TableFunctionObjectStorage; #endif #if USE_HDFS -using TableFunctionHDFS = TableFunctionObjectStorage; +using TableFunctionHDFS = TableFunctionObjectStorage; #endif #if USE_AVRO +using TableFunctionIceberg = TableFunctionObjectStorage; + # if USE_AWS_S3 -using TableFunctionIceberg = TableFunctionObjectStorage; using TableFunctionIcebergS3 = TableFunctionObjectStorage; # endif # if USE_AZURE_BLOB_STORAGE @@ -149,13 +153,13 @@ using TableFunctionPaimonHDFS = TableFunctionObjectStorage; #endif #if USE_PARQUET && USE_DELTA_KERNEL_RS -#if USE_AWS_S3 +# if USE_AWS_S3 using TableFunctionDeltaLake = TableFunctionObjectStorage; using TableFunctionDeltaLakeS3 = TableFunctionObjectStorage; -#endif -#if USE_AZURE_BLOB_STORAGE +# endif +# if USE_AZURE_BLOB_STORAGE using TableFunctionDeltaLakeAzure = TableFunctionObjectStorage; -#endif +# endif // New alias for local Delta Lake table function using TableFunctionDeltaLakeLocal = TableFunctionObjectStorage; #endif diff --git a/src/TableFunctions/TableFunctionObjectStorageCluster.cpp b/src/TableFunctions/TableFunctionObjectStorageCluster.cpp index d7541153b8a2..c2c5049df4d1 100644 --- a/src/TableFunctions/TableFunctionObjectStorageCluster.cpp +++ b/src/TableFunctions/TableFunctionObjectStorageCluster.cpp @@ -31,8 +31,9 @@ StoragePtr TableFunctionObjectStorageClusterstructure != "auto") - columns = parseColumnsListFromString(configuration->structure, context); + + if (configuration->getStructure() != "auto") + columns = parseColumnsListFromString(configuration->getStructure(), context); else if (!Base::structure_hint.empty()) columns = Base::structure_hint; else if (!cached_columns.empty()) @@ -79,8 +80,16 @@ StoragePtr TableFunctionObjectStorageClusterstartup(); @@ -319,7 +328,6 @@ void registerTableFunctionIcebergCluster(TableFunctionFactory & factory) {.allow_readonly = false} ); -#if USE_AWS_S3 factory.registerFunction( {.description = R"DOCS_MD( This is an extension to the [iceberg](/reference/functions/table-functions/iceberg) table function. @@ -371,37 +379,35 @@ SELECT * FROM icebergS3Cluster('cluster_simple', 'http://test.s3.amazonaws.com/c {.allow_readonly = false} ); +# if USE_AWS_S3 factory.registerFunction( { .description = R"(The table function can be used to read the Iceberg table stored on S3 object store in parallel for many nodes in a specified cluster.)", .examples{{IcebergS3ClusterDefinition::name, "SELECT * FROM icebergS3Cluster(cluster, url, [, NOSIGN | access_key_id, secret_access_key, [session_token]], format, [,compression])", ""}}, .category = FunctionDocumentation::Category::TableFunction }, - {.allow_readonly = false} - ); -#endif + {.allow_readonly = false}); +# endif -#if USE_AZURE_BLOB_STORAGE +# if USE_AZURE_BLOB_STORAGE factory.registerFunction( { .description = R"(The table function can be used to read the Iceberg table stored on Azure object store in parallel for many nodes in a specified cluster.)", .examples{{IcebergAzureClusterDefinition::name, "SELECT * FROM icebergAzureCluster(cluster, connection_string|storage_account_url, container_name, blobpath, [account_name, account_key, format, compression])", ""}}, .category = FunctionDocumentation::Category::TableFunction }, - {.allow_readonly = false} - ); -#endif + {.allow_readonly = false}); +# endif -#if USE_HDFS +# if USE_HDFS factory.registerFunction( { .description = R"(The table function can be used to read the Iceberg table stored on HDFS virtual filesystem in parallel for many nodes in a specified cluster.)", .examples{{IcebergHDFSClusterDefinition::name, "SELECT * FROM icebergHDFSCluster(cluster, uri, [format], [structure], [compression_method])", ""}}, .category = FunctionDocumentation::Category::TableFunction }, - {.allow_readonly = false} - ); -#endif + {.allow_readonly = false}); +# endif } void registerTableFunctionPaimonCluster(TableFunctionFactory & factory); diff --git a/src/TableFunctions/TableFunctionObjectStorageCluster.h b/src/TableFunctions/TableFunctionObjectStorageCluster.h index 58acf48d4f2a..26faefc2c5c2 100644 --- a/src/TableFunctions/TableFunctionObjectStorageCluster.h +++ b/src/TableFunctions/TableFunctionObjectStorageCluster.h @@ -12,8 +12,6 @@ namespace DB class Context; -class StorageS3Settings; -class StorageAzureBlobSettings; class StorageS3Configuration; class StorageAzureConfiguration; @@ -47,21 +45,25 @@ class TableFunctionObjectStorageCluster : public ITableFunctionClusterstructure != "auto"; } - bool needStructureHint() const override { return Base::getConfiguration(getQueryOrGlobalContext())->structure == "auto"; } + bool hasStaticStructure() const override { return Base::getConfiguration(getQueryOrGlobalContext())->getStructure() != "auto"; } + bool needStructureHint() const override { return Base::getConfiguration(getQueryOrGlobalContext())->getStructure() == "auto"; } void setStructureHint(const ColumnsDescription & structure_hint_) override { Base::structure_hint = structure_hint_; } }; #if USE_AWS_S3 -using TableFunctionS3Cluster = TableFunctionObjectStorageCluster; +using TableFunctionS3Cluster = TableFunctionObjectStorageCluster; #endif #if USE_AZURE_BLOB_STORAGE -using TableFunctionAzureBlobCluster = TableFunctionObjectStorageCluster; +using TableFunctionAzureBlobCluster = TableFunctionObjectStorageCluster; #endif #if USE_HDFS -using TableFunctionHDFSCluster = TableFunctionObjectStorageCluster; +using TableFunctionHDFSCluster = TableFunctionObjectStorageCluster; +#endif + +#if USE_AVRO +using TableFunctionIcebergCluster = TableFunctionObjectStorageCluster; #endif #if USE_AVRO @@ -70,7 +72,6 @@ using TableFunctionIcebergLocalCluster = TableFunctionObjectStorageCluster; -using TableFunctionIcebergCluster = TableFunctionObjectStorageCluster; #endif #if USE_AVRO && USE_AZURE_BLOB_STORAGE @@ -95,7 +96,7 @@ using TableFunctionPaimonHDFSCluster = TableFunctionObjectStorageCluster; using TableFunctionDeltaLakeS3Cluster = TableFunctionObjectStorageCluster; #endif diff --git a/src/TableFunctions/TableFunctionObjectStorageClusterFallback.cpp b/src/TableFunctions/TableFunctionObjectStorageClusterFallback.cpp new file mode 100644 index 000000000000..bc3d7237f134 --- /dev/null +++ b/src/TableFunctions/TableFunctionObjectStorageClusterFallback.cpp @@ -0,0 +1,470 @@ +#include +#include +#include +#include +#include +#include + +namespace DB +{ + +namespace Setting +{ + extern const SettingsString object_storage_cluster; + extern const SettingsBool object_storage_remote_initiator; + extern const SettingsString object_storage_remote_initiator_cluster; +} + +namespace ErrorCodes +{ + extern const int NUMBER_OF_ARGUMENTS_DOESNT_MATCH; + extern const int BAD_ARGUMENTS; +} + +struct S3ClusterFallbackDefinition +{ + static constexpr auto name = "s3"; + static constexpr auto storage_engine_name = "S3"; + static constexpr auto storage_engine_cluster_name = "S3Cluster"; +}; + +struct AzureClusterFallbackDefinition +{ + static constexpr auto name = "azureBlobStorage"; + static constexpr auto storage_engine_name = "Azure"; + static constexpr auto storage_engine_cluster_name = "AzureBlobStorageCluster"; +}; + +struct HDFSClusterFallbackDefinition +{ + static constexpr auto name = "hdfs"; + static constexpr auto storage_engine_name = "HDFS"; + static constexpr auto storage_engine_cluster_name = "HDFSCluster"; +}; + +struct IcebergClusterFallbackDefinition +{ + static constexpr auto name = "iceberg"; + static constexpr auto storage_engine_name = "UNDEFINED"; + static constexpr auto storage_engine_cluster_name = "IcebergCluster"; +}; + +struct IcebergS3ClusterFallbackDefinition +{ + static constexpr auto name = "icebergS3"; + static constexpr auto storage_engine_name = "S3"; + static constexpr auto storage_engine_cluster_name = "IcebergS3Cluster"; +}; + +struct IcebergAzureClusterFallbackDefinition +{ + static constexpr auto name = "icebergAzure"; + static constexpr auto storage_engine_name = "Azure"; + static constexpr auto storage_engine_cluster_name = "IcebergAzureCluster"; +}; + +struct IcebergHDFSClusterFallbackDefinition +{ + static constexpr auto name = "icebergHDFS"; + static constexpr auto storage_engine_name = "HDFS"; + static constexpr auto storage_engine_cluster_name = "IcebergHDFSCluster"; +}; + +struct IcebergLocalClusterFallbackDefinition +{ + static constexpr auto name = "icebergLocal"; + static constexpr auto storage_engine_name = "Local"; + static constexpr auto storage_engine_cluster_name = "IcebergLocalCluster"; +}; + +struct DeltaLakeClusterFallbackDefinition +{ + static constexpr auto name = "deltaLake"; + static constexpr auto storage_engine_name = "S3"; + static constexpr auto storage_engine_cluster_name = "DeltaLakeS3Cluster"; +}; + +struct DeltaLakeS3ClusterFallbackDefinition +{ + static constexpr auto name = "deltaLakeS3"; + static constexpr auto storage_engine_name = "S3"; + static constexpr auto storage_engine_cluster_name = "DeltaLakeS3Cluster"; +}; + +struct DeltaLakeAzureClusterFallbackDefinition +{ + static constexpr auto name = "deltaLakeAzure"; + static constexpr auto storage_engine_name = "Azure"; + static constexpr auto storage_engine_cluster_name = "DeltaLakeAzureCluster"; +}; + +struct HudiClusterFallbackDefinition +{ + static constexpr auto name = "hudi"; + static constexpr auto storage_engine_name = "S3"; + static constexpr auto storage_engine_cluster_name = "HudiS3Cluster"; +}; + +template +void TableFunctionObjectStorageClusterFallback::parseArgumentsImpl(ASTs & args, const ContextPtr & context) +{ + if (args.empty()) + throw Exception( + ErrorCodes::NUMBER_OF_ARGUMENTS_DOESNT_MATCH, + "The function {} should have arguments. The first argument must be the cluster name and the rest are the arguments of " + "corresponding table function", + getName()); + + const auto & settings = context->getSettingsRef(); + + is_cluster_function = !settings[Setting::object_storage_cluster].value.empty() && typename Base::Configuration().isClusterSupported(); + // Remote initiator requires 'object_storage_cluster' or 'object_storage_remote_initiator_cluster' + if (settings[Setting::object_storage_remote_initiator]) + { + if (settings[Setting::object_storage_cluster].value.empty() + && settings[Setting::object_storage_remote_initiator_cluster].value.empty()) + { + throw Exception(ErrorCodes::BAD_ARGUMENTS, + "Setting 'object_storage_remote_initiator' can be used only with 'object_storage_remote_initiator_cluster', 'object_storage_cluster', or cluster name in arguments"); + } + + is_remote = true; + } + + if (is_cluster_function) + { + /// Name may be empty, but cluster workaround may be used in remote initiator case + ASTPtr cluster_name_arg = make_intrusive(settings[Setting::object_storage_cluster].value); + args.insert(args.begin(), cluster_name_arg); + BaseCluster::parseArgumentsImpl(args, context); + args.erase(args.begin()); + } + else + BaseSimple::parseArgumentsImpl(args, context); // NOLINT(bugprone-parent-virtual-call) +} + +template +StoragePtr TableFunctionObjectStorageClusterFallback::executeImpl( + const ASTPtr & ast_function, + ContextPtr context, + const std::string & table_name, + ColumnsDescription cached_columns, + bool is_insert_query) const +{ + if (is_cluster_function || is_remote) + { + auto result = BaseCluster::executeImpl(ast_function, context, table_name, cached_columns, is_insert_query); + if (auto storage = typeid_cast>(result)) + storage->setClusterNameInSettings(true); + return result; + } + else + return BaseSimple::executeImpl(ast_function, context, table_name, cached_columns, is_insert_query); // NOLINT(bugprone-parent-virtual-call) +} + +template +void TableFunctionObjectStorageClusterFallback::validateUseToCreateTable() const +{ + if (is_cluster_function || is_remote) + throw Exception( + ErrorCodes::BAD_ARGUMENTS, + "Table function '{}' cannot be used to create a table in cluster mode or with remote initiator", + getName()); +} + +#if USE_AWS_S3 +using TableFunctionS3ClusterFallback = TableFunctionObjectStorageClusterFallback; +#endif + +#if USE_AZURE_BLOB_STORAGE +using TableFunctionAzureClusterFallback = TableFunctionObjectStorageClusterFallback; +#endif + +#if USE_HDFS +using TableFunctionHDFSClusterFallback = TableFunctionObjectStorageClusterFallback; +#endif + +#if USE_AVRO +using TableFunctionIcebergClusterFallback = TableFunctionObjectStorageClusterFallback; +using TableFunctionIcebergLocalClusterFallback = TableFunctionObjectStorageClusterFallback; +#endif + +#if USE_AVRO && USE_AWS_S3 +using TableFunctionIcebergS3ClusterFallback = TableFunctionObjectStorageClusterFallback; +#endif + +#if USE_AVRO && USE_AZURE_BLOB_STORAGE +using TableFunctionIcebergAzureClusterFallback = TableFunctionObjectStorageClusterFallback; +#endif + +#if USE_AVRO && USE_HDFS +using TableFunctionIcebergHDFSClusterFallback = TableFunctionObjectStorageClusterFallback; +#endif + +#if USE_AWS_S3 && USE_PARQUET && USE_DELTA_KERNEL_RS +using TableFunctionDeltaLakeClusterFallback = TableFunctionObjectStorageClusterFallback; +using TableFunctionDeltaLakeS3ClusterFallback = TableFunctionObjectStorageClusterFallback; +#endif + +#if USE_AZURE_BLOB_STORAGE && USE_PARQUET && USE_DELTA_KERNEL_RS +using TableFunctionDeltaLakeAzureClusterFallback = TableFunctionObjectStorageClusterFallback; +#endif + +#if USE_AWS_S3 +using TableFunctionHudiClusterFallback = TableFunctionObjectStorageClusterFallback; +#endif + +void registerTableFunctionObjectStorageClusterFallback(TableFunctionFactory & factory) +{ + UNUSED(factory); +#if USE_AWS_S3 + factory.registerFunction( + { + .description=R"(The table function can be used to read the data stored on S3 in parallel for many nodes in a specified cluster or from single node.)", + .examples{ + {"s3", "SELECT * FROM s3(url, format, structure)", ""}, + {"s3", "SELECT * FROM s3(url, format, structure) SETTINGS object_storage_cluster='cluster'", ""} + }, + .category = FunctionDocumentation::Category::TableFunction + }, + {.allow_readonly = false} + ); +#endif + +#if USE_AZURE_BLOB_STORAGE + factory.registerFunction( + { + .description=R"(The table function can be used to read the data stored on Azure Blob Storage in parallel for many nodes in a specified cluster or from single node.)", + .examples{ + { + "azureBlobStorage", + "SELECT * FROM azureBlobStorage(connection_string|storage_account_url, container_name, blobpath, " + "[account_name, account_key, format, compression, structure])", "" + }, + { + "azureBlobStorage", + "SELECT * FROM azureBlobStorage(connection_string|storage_account_url, container_name, blobpath, " + "[account_name, account_key, format, compression, structure]) " + "SETTINGS object_storage_cluster='cluster'", "" + }, + }, + .category = FunctionDocumentation::Category::TableFunction + }, + {.allow_readonly = false} + ); +#endif + +#if USE_HDFS + factory.registerFunction( + { + .description=R"(The table function can be used to read the data stored on HDFS virtual filesystem in parallel for many nodes in a specified cluster or from single node.)", + .examples{ + { + "hdfs", + "SELECT * FROM hdfs(url, format, compression, structure])", "" + }, + { + "hdfs", + "SELECT * FROM hdfs(url, format, compression, structure]) " + "SETTINGS object_storage_cluster='cluster'", "" + }, + }, + .category = FunctionDocumentation::Category::TableFunction + }, + {.allow_readonly = false} + ); +#endif + +#if USE_AVRO + factory.registerFunction( + { + .description=R"(The table function can be used to read the Iceberg table stored on different object store in parallel for many nodes in a specified cluster or from single node.)", + .examples{ + { + "iceberg", + "SELECT * FROM iceberg(url, access_key_id, secret_access_key, storage_type='s3')", "" + }, + { + "iceberg", + "SELECT * FROM iceberg(url, access_key_id, secret_access_key, storage_type='s3') " + "SETTINGS object_storage_cluster='cluster'", "" + }, + { + "iceberg", + "SELECT * FROM iceberg(url, access_key_id, secret_access_key, storage_type='azure')", "" + }, + { + "iceberg", + "SELECT * FROM iceberg(url, storage_type='hdfs') SETTINGS object_storage_cluster='cluster'", "" + }, + }, + .category = FunctionDocumentation::Category::TableFunction + }, + {.allow_readonly = false} + ); + + factory.registerFunction( + { + .description=R"(The table function can be used to read the Iceberg table stored on shared disk in parallel for many nodes in a specified cluster or from single node.)", + .examples{ + { + "icebergLocal", + "SELECT * FROM icebergLocal(filename)", "" + }, + { + "icebergLocal", + "SELECT * FROM icebergLocal(filename) " + "SETTINGS object_storage_cluster='cluster'", "" + }, + }, + .category = FunctionDocumentation::Category::TableFunction + }, + {.allow_readonly = false} + ); +#endif + +#if USE_AVRO && USE_AWS_S3 + factory.registerFunction( + { + .description=R"(The table function can be used to read the Iceberg table stored on S3 object store in parallel for many nodes in a specified cluster or from single node.)", + .examples{ + { + "icebergS3", + "SELECT * FROM icebergS3(url, access_key_id, secret_access_key)", "" + }, + { + "icebergS3", + "SELECT * FROM icebergS3(url, access_key_id, secret_access_key) " + "SETTINGS object_storage_cluster='cluster'", "" + }, + }, + .category = FunctionDocumentation::Category::TableFunction + }, + {.allow_readonly = false} + ); +#endif + +#if USE_AVRO && USE_AZURE_BLOB_STORAGE + factory.registerFunction( + { + .description=R"(The table function can be used to read the Iceberg table stored on Azure object store in parallel for many nodes in a specified cluster or from single node.)", + .examples{ + { + "icebergAzure", + "SELECT * FROM icebergAzure(url, access_key_id, secret_access_key)", "" + }, + { + "icebergAzure", + "SELECT * FROM icebergAzure(url, access_key_id, secret_access_key) " + "SETTINGS object_storage_cluster='cluster'", "" + }, + }, + .category = FunctionDocumentation::Category::TableFunction + }, + {.allow_readonly = false} + ); +#endif + +#if USE_AVRO && USE_HDFS + factory.registerFunction( + { + .description=R"(The table function can be used to read the Iceberg table stored on HDFS virtual filesystem in parallel for many nodes in a specified cluster or from single node.)", + .examples{ + { + "icebergHDFS", + "SELECT * FROM icebergHDFS(url)", "" + }, + { + "icebergHDFS", + "SELECT * FROM icebergHDFS(url) SETTINGS object_storage_cluster='cluster'", "" + }, + }, + .category = FunctionDocumentation::Category::TableFunction + }, + {.allow_readonly = false} + ); +#endif + +#if USE_PARQUET && USE_DELTA_KERNEL_RS +# if USE_AWS_S3 + factory.registerFunction( + { + .description=R"(The table function can be used to read the DeltaLake table stored on object store in parallel for many nodes in a specified cluster or from single node.)", + .examples{ + { + "deltaLake", + "SELECT * FROM deltaLake(url, access_key_id, secret_access_key)", "" + }, + { + "deltaLake", + "SELECT * FROM deltaLake(url, access_key_id, secret_access_key) " + "SETTINGS object_storage_cluster='cluster'", "" + }, + }, + .category = FunctionDocumentation::Category::TableFunction + }, + {.allow_readonly = false} + ); + factory.registerFunction( + { + .description=R"(The table function can be used to read the DeltaLake table stored on object store in parallel for many nodes in a specified cluster or from single node.)", + .examples{ + { + "deltaLakeS3", + "SELECT * FROM deltaLakeS3(url, access_key_id, secret_access_key)", "" + }, + { + "deltaLakeS3", + "SELECT * FROM deltaLakeS3(url, access_key_id, secret_access_key) " + "SETTINGS object_storage_cluster='cluster'", "" + }, + }, + .category = FunctionDocumentation::Category::TableFunction + }, + {.allow_readonly = false} + ); +# endif +# if USE_AZURE_BLOB_STORAGE + factory.registerFunction( + { + .description=R"(The table function can be used to read the DeltaLake table stored on object store in parallel for many nodes in a specified cluster or from single node.)", + .examples{ + { + "deltaLakeAzure", + "SELECT * FROM deltaLakeAzure(url, access_key_id, secret_access_key)", "" + }, + { + "deltaLakeAzure", + "SELECT * FROM deltaLakeAzure(url, access_key_id, secret_access_key) " + "SETTINGS object_storage_cluster='cluster'", "" + }, + }, + .category = FunctionDocumentation::Category::TableFunction + }, + {.allow_readonly = false} + ); +# endif +#endif + +#if USE_AWS_S3 + factory.registerFunction( + { + .description=R"(The table function can be used to read the Hudi table stored on object store in parallel for many nodes in a specified cluster or from single node.)", + .examples{ + { + "hudi", + "SELECT * FROM hudi(url, access_key_id, secret_access_key)", "" + }, + { + "hudi", + "SELECT * FROM hudi(url, access_key_id, secret_access_key) SETTINGS object_storage_cluster='cluster'", "" + }, + }, + .category = FunctionDocumentation::Category::TableFunction + }, + {.allow_readonly = false} + ); +#endif +} + +} diff --git a/src/TableFunctions/TableFunctionObjectStorageClusterFallback.h b/src/TableFunctions/TableFunctionObjectStorageClusterFallback.h new file mode 100644 index 000000000000..a21cc963d4c0 --- /dev/null +++ b/src/TableFunctions/TableFunctionObjectStorageClusterFallback.h @@ -0,0 +1,50 @@ +#pragma once +#include "config.h" +#include + +namespace DB +{ + +/** +* Class implementing s3/hdfs/azureBlobStorage(...) table functions, +* which allow to use simple or distributed function variant based on settings. +* If setting `object_storage_cluster` is empty, +* simple single-host variant is used, if setting not empty, cluster variant is used. +* `SELECT * FROM s3('s3://...', ...) SETTINGS object_storage_cluster='cluster'` +* is equal to +* `SELECT * FROM s3Cluster('cluster', 's3://...', ...)` +*/ + +template +class TableFunctionObjectStorageClusterFallback : public Base +{ +public: + using BaseCluster = Base; + using BaseSimple = BaseCluster::Base; + + static constexpr auto name = Definition::name; + + String getName() const override { return name; } + + void validateUseToCreateTable() const override; + +private: + const char * getStorageEngineName() const override + { + return is_cluster_function ? Definition::storage_engine_cluster_name : Definition::storage_engine_name; + } + + StoragePtr executeImpl( + const ASTPtr & ast_function, + ContextPtr context, + const std::string & table_name, + ColumnsDescription cached_columns, + bool is_insert_query) const override; + + void parseArgumentsImpl(ASTs & args, const ContextPtr & context) override; + + bool is_cluster_function = false; + bool is_remote = false; +}; + +} diff --git a/src/TableFunctions/TableFunctionRemote.cpp b/src/TableFunctions/TableFunctionRemote.cpp index b34c0f2a5eb2..5729742d0c82 100644 --- a/src/TableFunctions/TableFunctionRemote.cpp +++ b/src/TableFunctions/TableFunctionRemote.cpp @@ -90,6 +90,9 @@ StoragePtr TableFunctionRemote::executeImpl(const ASTPtr & /*ast_function*/, Con ColumnsDescription TableFunctionRemote::getActualTableStructure(ContextPtr context, bool /*is_insert_query*/) const { + if (!remote_table_columns.empty()) + return remote_table_columns; + chassert(cluster); return getStructureOfRemoteTable(*cluster, remote_table_id, context, remote_table_function_ptr); } diff --git a/src/TableFunctions/TableFunctionRemote.h b/src/TableFunctions/TableFunctionRemote.h index 2b9ced6a4180..d40e39eb544e 100644 --- a/src/TableFunctions/TableFunctionRemote.h +++ b/src/TableFunctions/TableFunctionRemote.h @@ -27,6 +27,10 @@ class TableFunctionRemote : public ITableFunction bool needStructureConversion() const override { return false; } + void setRemoteTableFunction(ASTPtr remote_table_function_ptr_) { remote_table_function_ptr = remote_table_function_ptr_; } + + void setActualTableStructure(ColumnsDescription remote_table_columns_) { remote_table_columns = remote_table_columns_; } + private: StoragePtr executeImpl(const ASTPtr & ast_function, ContextPtr context, const std::string & table_name, ColumnsDescription cached_columns, bool is_insert_query) const override; @@ -43,6 +47,7 @@ class TableFunctionRemote : public ITableFunction StorageID remote_table_id = StorageID::createEmpty(); ASTPtr remote_table_function_ptr; ASTPtr sharding_key = nullptr; + ColumnsDescription remote_table_columns; /// Changes from a SETTINGS clause among the arguments, applied to the `DistributedSettings` /// of the created `StorageDistributed`, e.g. SETTINGS skip_unavailable_shards = 1. diff --git a/src/TableFunctions/TableFunctionURL.cpp b/src/TableFunctions/TableFunctionURL.cpp index b58a0b842505..e18fe1bc990b 100644 --- a/src/TableFunctions/TableFunctionURL.cpp +++ b/src/TableFunctions/TableFunctionURL.cpp @@ -362,7 +362,7 @@ StoragePtr TableFunctionURL::getStorage( auto object_storage_configuration = std::make_shared(); auto engine_args = makeWebObjectStorageEngineArgs(source, format_, structure, compression_method_, configuration.headers); - StorageObjectStorageConfiguration::initialize(*object_storage_configuration, engine_args, context, /* with_table_structure */ true); + object_storage_configuration->initialize(engine_args, context, /* with_table_structure */ true); ObjectStoragePtr object_storage = object_storage_configuration->createObjectStorage(context, /* is_readonly */ true, std::nullopt); @@ -418,9 +418,12 @@ ColumnsDescription TableFunctionURL::getActualTableStructure(ContextPtr context, { checkExperimentalURLWildcardFromIndexPages(context); - auto object_storage_configuration = std::make_shared(); + /// Note: the base-class pointer type is required because + /// StorageObjectStorage::resolveSchemaAndFormatFromData() takes a non-const + /// reference to StorageObjectStorageConfigurationPtr. + StorageObjectStorageConfigurationPtr object_storage_configuration = std::make_shared(); auto engine_args = makeWebObjectStorageEngineArgs(filename, format, structure, compression_method, configuration.headers); - StorageObjectStorageConfiguration::initialize(*object_storage_configuration, engine_args, context, /* with_table_structure */ true); + object_storage_configuration->initialize(engine_args, context, /* with_table_structure */ true); object_storage_configuration->check(context); auto object_storage = object_storage_configuration->createObjectStorage(context, /* is_readonly */ true, std::nullopt); diff --git a/src/TableFunctions/registerTableFunctions.cpp b/src/TableFunctions/registerTableFunctions.cpp index f4d5e81e7ad3..7cd574c3232e 100644 --- a/src/TableFunctions/registerTableFunctions.cpp +++ b/src/TableFunctions/registerTableFunctions.cpp @@ -76,6 +76,7 @@ void registerTableFunctions() registerTableFunctionObjectStorage(factory); registerTableFunctionObjectStorageCluster(factory); registerDataLakeTableFunctions(factory); + registerTableFunctionObjectStorageClusterFallback(factory); registerDataLakeClusterTableFunctions(factory); #if USE_YTSAURUS diff --git a/src/TableFunctions/registerTableFunctions.h b/src/TableFunctions/registerTableFunctions.h index df7dc7acf8f3..039797dcb729 100644 --- a/src/TableFunctions/registerTableFunctions.h +++ b/src/TableFunctions/registerTableFunctions.h @@ -77,6 +77,7 @@ void registerTableFunctionExplain(TableFunctionFactory & factory); void registerTableFunctionObjectStorage(TableFunctionFactory & factory); void registerTableFunctionObjectStorageCluster(TableFunctionFactory & factory); void registerDataLakeTableFunctions(TableFunctionFactory & factory); +void registerTableFunctionObjectStorageClusterFallback(TableFunctionFactory & factory); void registerDataLakeClusterTableFunctions(TableFunctionFactory & factory); void registerTableFunctionTimeSeries(TableFunctionFactory & factory); diff --git a/tests/config/config.d/allow_experimental_export_merge_tree_partition.xml b/tests/config/config.d/allow_experimental_export_merge_tree_partition.xml new file mode 100644 index 000000000000..514cd710836a --- /dev/null +++ b/tests/config/config.d/allow_experimental_export_merge_tree_partition.xml @@ -0,0 +1,3 @@ + + 1 + diff --git a/tests/config/install.sh b/tests/config/install.sh index dfcb22bc63a3..350ec33d58ba 100755 --- a/tests/config/install.sh +++ b/tests/config/install.sh @@ -118,6 +118,7 @@ ln -sf $SRC_PATH/config.d/predicate_statistics_log.xml $DEST_SERVER_PATH/config. ln -sf $SRC_PATH/config.d/custom_settings_prefixes.xml $DEST_SERVER_PATH/config.d/ ln -sf $SRC_PATH/config.d/database_catalog_drop_table_concurrency.xml $DEST_SERVER_PATH/config.d/ ln -sf $SRC_PATH/config.d/enable_access_control_improvements.xml $DEST_SERVER_PATH/config.d/ +ln -sf $SRC_PATH/config.d/allow_experimental_export_merge_tree_partition.xml $DEST_SERVER_PATH/config.d/ ln -sf $SRC_PATH/config.d/macros.xml $DEST_SERVER_PATH/config.d/ ln -sf $SRC_PATH/config.d/secure_ports.xml $DEST_SERVER_PATH/config.d/ ln -sf $SRC_PATH/config.d/clusters.xml $DEST_SERVER_PATH/config.d/ diff --git a/tests/integration/helpers/cluster.py b/tests/integration/helpers/cluster.py index ff43d63d20b4..f0db60719152 100644 --- a/tests/integration/helpers/cluster.py +++ b/tests/integration/helpers/cluster.py @@ -745,6 +745,9 @@ def __init__( self.minio_secret_key = minio_secret_key self.spark_session = None + self.spark_iceberg_external_port = 8080 + self.spark_iceberg_external_port_2 = 10002 + self.spark_iceberg_external_port_3 = 10003 self.with_iceberg_catalog = False self._iceberg_rest_catalog_port = None self._iceberg_minio_port = None @@ -970,6 +973,8 @@ def __init__( self._letsencrypt_pebble_api_port = 14000 self._letsencrypt_pebble_management_port = 15000 + self.iceberg_rest_external_port = 8182 + self.docker_client: docker.DockerClient = None self.is_up = False self.env = os.environ.copy() @@ -1902,6 +1907,10 @@ def setup_hms_catalog_cmd(self, instance, env_variables, docker_compose_yml_dir) def setup_iceberg_catalog_cmd( self, instance, env_variables, docker_compose_yml_dir, extra_parameters=None ): + env_variables["ICEBERG_REST_EXTERNAL_PORT"] = str(self.iceberg_rest_external_port) + env_variables["SPARK_ICEBERG_EXTERNAL_PORT"] = str(self.spark_iceberg_external_port) + env_variables["SPARK_ICEBERG_EXTERNAL_PORT_2"] = str(self.spark_iceberg_external_port_2) + env_variables["SPARK_ICEBERG_EXTERNAL_PORT_3"] = str(self.spark_iceberg_external_port_3) self.with_iceberg_catalog = True file_name = "docker_compose_iceberg_rest_catalog.yml" if extra_parameters is not None and extra_parameters["docker_compose_file_name"] != "": diff --git a/tests/integration/helpers/export_partition_helpers.py b/tests/integration/helpers/export_partition_helpers.py new file mode 100644 index 000000000000..04c9cb244757 --- /dev/null +++ b/tests/integration/helpers/export_partition_helpers.py @@ -0,0 +1,212 @@ +""" +Shared helpers for export-partition and export-part integration tests. + +Centralises wait-polling, table creation, and partition helpers that were +previously duplicated across multiple test modules. +""" + +import time +import uuid + + +MINIO_USER = "minio" +MINIO_PASS = "ClickHouse_Minio_P@ssw0rd" + + +def wait_for_export_status( + node, + source_table, + dest_table, + partition_id, + expected_status="COMPLETED", + timeout=60, + poll_interval=0.5, +): + """Poll system.replicated_partition_exports until status matches. + + *dest_table* may be ``None`` to skip filtering by destination table + (useful for catalog-based tests where the destination is a database-qualified path). + """ + start_time = time.time() + last_status = None + while time.time() - start_time < timeout: + dest_filter = ( + f" AND destination_table = '{dest_table}'" if dest_table else "" + ) + status = node.query( + f"SELECT status FROM system.replicated_partition_exports" + f" WHERE source_table = '{source_table}'" + f"{dest_filter}" + f" AND partition_id = '{partition_id}'" + ).strip() + + last_status = status + if status and status == expected_status: + return status + + time.sleep(poll_interval) + + raise TimeoutError( + f"Export status did not reach '{expected_status}' within {timeout}s. " + f"Last status: '{last_status}'" + ) + + +def wait_for_export_to_start( + node, + source_table, + dest_table, + partition_id, + timeout=10, + poll_interval=0.2, +): + """Poll until at least one row exists in system.replicated_partition_exports.""" + start_time = time.time() + while time.time() - start_time < timeout: + count = node.query( + f"SELECT count() FROM system.replicated_partition_exports" + f" WHERE source_table = '{source_table}'" + f" AND destination_table = '{dest_table}'" + f" AND partition_id = '{partition_id}'" + ).strip() + + if count != "0": + return True + + time.sleep(poll_interval) + + raise TimeoutError( + f"Export of partition {partition_id!r} did not start within {timeout}s." + ) + + +def wait_for_exception_count( + node, + source_table, + dest_table, + partition_id, + min_exception_count=1, + timeout=60, + poll_interval=0.5, +): + """Wait for exception_count to reach at least *min_exception_count*. + + The default timeout is intentionally larger than one manifest-updater poll + cycle (~30s, see StorageReplicatedMergeTree::exportMergeTreePartitionUpdatingTask). + system.replicated_partition_exports is served from the in-memory mirror, which + is refreshed on (a) the periodic poll tick and (b) status changes. While the + task is still PENDING (e.g. transient part-export failures with a generous + max_retries), no status watch fires, so newly written per-replica exception + leaves only become visible on the next poll. Allow at least one full cycle + plus headroom so the test is not racing the cadence. + """ + start_time = time.time() + last_exception_count = None + while time.time() - start_time < timeout: + exception_count_str = node.query( + f"SELECT exception_count FROM system.replicated_partition_exports" + f" WHERE source_table = '{source_table}'" + f" AND destination_table = '{dest_table}'" + f" AND partition_id = '{partition_id}'" + ).strip() + + if exception_count_str: + exception_count = int(exception_count_str) + last_exception_count = exception_count + if exception_count >= min_exception_count: + return exception_count + + time.sleep(poll_interval) + + raise TimeoutError( + f"Exception count did not reach {min_exception_count} within {timeout}s. " + f"Last exception_count: {last_exception_count if last_exception_count is not None else 'N/A'}" + ) + + +# -- block-number settings are needed for patch parts support +_BLOCK_SETTINGS = ( + "enable_block_number_column = 1, enable_block_offset_column = 1" +) + + +def make_rmt( + node, + name, + columns, + partition_by, + replica_name="r1", + order_by="tuple()", + extra_settings="", +): + """Create a ReplicatedMergeTree table with block-number settings.""" + settings = f"{_BLOCK_SETTINGS}, {extra_settings}" if extra_settings else _BLOCK_SETTINGS + node.query( + f""" + CREATE TABLE {name} ({columns}) + ENGINE = ReplicatedMergeTree('/clickhouse/tables/{name}', '{replica_name}') + PARTITION BY {partition_by} + ORDER BY {order_by} + SETTINGS {settings} + """ + ) + + +def make_mt( + node, + name, + columns, + partition_by, + order_by="tuple()", +): + """Create a MergeTree table with block-number settings.""" + node.query( + f""" + CREATE TABLE {name} ({columns}) + ENGINE = MergeTree() + PARTITION BY {partition_by} + ORDER BY {order_by} + SETTINGS {_BLOCK_SETTINGS} + """ + ) + + +def make_iceberg_s3( + node, + name, + columns, + partition_by="", + url=None, + s3_retry_attempts=3, + if_not_exists=False, +): + """Create an IcebergS3 table at a MinIO prefix. + + *url* defaults to ``http://minio1:9001/root/data/{name}/``. + """ + if url is None: + url = f"http://minio1:9001/root/data/{name}/" + ine = "IF NOT EXISTS " if if_not_exists else "" + pclause = f"PARTITION BY {partition_by}" if partition_by else "" + node.query( + f""" + CREATE TABLE {ine}{name} ({columns}) + ENGINE = IcebergS3('{url}', '{MINIO_USER}', '{MINIO_PASS}') + {pclause} + SETTINGS s3_retry_attempts = {s3_retry_attempts} + """ + ) + + +def first_partition_id(node, table): + """Return the partition_id of the first active part of *table*.""" + return node.query( + f"SELECT partition_id FROM system.parts" + f" WHERE database = currentDatabase() AND table = '{table}' AND active" + f" ORDER BY name LIMIT 1" + ).strip() + + +def unique_suffix(): + """Return a UUID with hyphens replaced by underscores, suitable for table names.""" + return str(uuid.uuid4()).replace("-", "_") diff --git a/tests/integration/helpers/iceberg_export_stats.py b/tests/integration/helpers/iceberg_export_stats.py new file mode 100644 index 000000000000..45a997922099 --- /dev/null +++ b/tests/integration/helpers/iceberg_export_stats.py @@ -0,0 +1,179 @@ +"""Shared helpers for verifying Iceberg per-file column statistics produced by +``EXPORT PART`` / ``EXPORT PARTITION``. + +Both the MergeTree and ReplicatedMergeTree export test modules drive the same +schema and expected stats shape (see ``assert_exported_stats``), so the +assertions, the manifest-entry reader, and the small byte/int decoders live +here instead of being duplicated in each test module. + +The only ClickHouse-side prerequisite is that ``system.iceberg_metadata_log`` +is enabled on the node: point the test cluster at +``configs/config.d/metadata_log.xml`` (shipped next to each test) and run the +probing SELECT with ``SETTINGS iceberg_metadata_log_level = 'manifest_file_entry'``. +""" + +import json + +from helpers.iceberg_utils import get_bound_for_column + + +# Iceberg assigns field ids positionally (starting at 1) to the non-partition +# columns in declaration order; partition-source columns share the same ids and +# partition transform outputs live in a separate 1000+ namespace. For the +# schema used by the two export-stats tests (id Int32, name String, +# tag Nullable(String), year Int32) this yields the mapping below. +STATS_FIELD_IDS = {"id": 1, "name": 2, "tag": 3} + + +def decode_int_bound(raw): + """Decode a JSON-serialized Iceberg integer bound (little-endian signed bytes). + + ClickHouse's Iceberg writer currently dumps integer bounds using the + underlying ``Field`` storage width (8 bytes for Int32/Int64/Date/...). Some + writers produce the spec-correct 4-byte Int32 encoding. Accept both. + """ + data = raw.encode("latin-1") + assert len(data) in (4, 8), f"Unexpected bound width {len(data)}: {raw!r}" + return int.from_bytes(data, "little", signed=True) + + +def get_int_for_column(m, column_id): + """Look up a value for ``column_id`` in an Iceberg integer map serialized as + either a dict ``{str(column_id): value}`` or a list of ``{key, value}`` records. + + Unlike :func:`helpers.iceberg_utils.get_bound_for_column`, this helper does + not try to unescape the value, so it works for numeric columns like + ``column_sizes`` and ``null_value_counts`` where the raw value is an int + (or a quoted int64 string). + """ + if m is None: + return None + value = None + if isinstance(m, dict): + value = m.get(str(column_id)) + elif isinstance(m, list): + for item in m: + if isinstance(item, dict) and item.get("key") == column_id: + value = item.get("value") + break + if value is None: + return None + return int(value) if isinstance(value, str) else value + + +def fetch_manifest_entries(node, query_id): + """Read JSON manifest-file entries emitted for ``query_id`` into + ``system.iceberg_metadata_log``. + + The outer ``FORMAT JSONEachRow`` is required: the default TSV format + escapes backslashes in the ``content`` string, which would double-encode + the inner ``\\uXXXX`` sequences coming from Iceberg bytes-bounds. + """ + node.query("SYSTEM FLUSH LOGS") + raw = node.query( + f""" + SELECT DISTINCT content + FROM system.iceberg_metadata_log + WHERE query_id = '{query_id}' + AND content_type = 'ManifestFileEntry' + AND content != '' + FORMAT JSONEachRow + """ + ) + entries = [] + for line in raw.strip().split("\n"): + if not line: + continue + outer = json.loads(line) + content = outer.get("content") + if content: + entries.append(json.loads(content)) + return entries + + +def assert_exported_stats(entries): + """Assert that at least one manifest entry describes the exported 2020 data file. + + Expected shape (three rows, one NULL in ``tag``): + + * ``record_count = 3`` + * ``file_size_in_bytes > 0`` + * ``column_sizes[id|name|tag] > 0`` + * ``null_value_counts = {id: 0, name: 0, tag: 1}`` + * ``lower_bounds = {id: 1, name: "aaa", tag: "x"}`` + * ``upper_bounds = {id: 3, name: "zzz", tag: "y"}`` + """ + assert entries, "No ManifestFileEntry rows recorded in system.iceberg_metadata_log" + + id_fid = STATS_FIELD_IDS["id"] + name_fid = STATS_FIELD_IDS["name"] + tag_fid = STATS_FIELD_IDS["tag"] + + matched = False + for entry in entries: + data_file = entry.get("data_file") or {} + # Skip manifest entries that explicitly mark themselves as deletes; data + # entries either omit `content` (v1 manifest) or set it to 0. + if data_file.get("content", 0) not in (0, None): + continue + + record_count = data_file.get("record_count") + if record_count != 3: + continue + + file_size = data_file.get("file_size_in_bytes") + assert file_size and file_size > 0, ( + f"Expected positive file_size_in_bytes, got {file_size!r}" + ) + + for field in ("id", "name", "tag"): + fid = STATS_FIELD_IDS[field] + size = get_int_for_column(data_file.get("column_sizes"), fid) + assert size is not None, ( + f"column_sizes missing entry for field_id={fid} ({field})" + ) + assert size > 0, f"column_sizes[{field}] expected > 0, got {size!r}" + + null_counts = data_file.get("null_value_counts") + assert get_int_for_column(null_counts, id_fid) == 0, ( + f"Expected 0 nulls in id, got null_value_counts={null_counts!r}" + ) + assert get_int_for_column(null_counts, name_fid) == 0, ( + f"Expected 0 nulls in name, got null_value_counts={null_counts!r}" + ) + assert get_int_for_column(null_counts, tag_fid) == 1, ( + f"Expected 1 null in tag (one NULL was inserted), " + f"got null_value_counts={null_counts!r}" + ) + + lower = data_file.get("lower_bounds") + upper = data_file.get("upper_bounds") + + assert decode_int_bound(get_bound_for_column(lower, id_fid)) == 1, ( + f"lower_bounds[id] expected 1, got {get_bound_for_column(lower, id_fid)!r}" + ) + assert decode_int_bound(get_bound_for_column(upper, id_fid)) == 3, ( + f"upper_bounds[id] expected 3, got {get_bound_for_column(upper, id_fid)!r}" + ) + + assert get_bound_for_column(lower, name_fid) == "aaa", ( + f"lower_bounds[name] expected 'aaa', got {get_bound_for_column(lower, name_fid)!r}" + ) + assert get_bound_for_column(upper, name_fid) == "zzz", ( + f"upper_bounds[name] expected 'zzz', got {get_bound_for_column(upper, name_fid)!r}" + ) + + assert get_bound_for_column(lower, tag_fid) == "x", ( + f"lower_bounds[tag] expected 'x' (nulls are skipped), got {get_bound_for_column(lower, tag_fid)!r}" + ) + assert get_bound_for_column(upper, tag_fid) == "y", ( + f"upper_bounds[tag] expected 'y' (nulls are skipped), got {get_bound_for_column(upper, tag_fid)!r}" + ) + + matched = True + break + + assert matched, ( + f"No data-file manifest entry with record_count=3 was found. " + f"Parsed {len(entries)} entr(y|ies) but none matched." + ) diff --git a/tests/integration/helpers/iceberg_utils.py b/tests/integration/helpers/iceberg_utils.py index ab8524ff91e0..8f7d13e25c9c 100644 --- a/tests/integration/helpers/iceberg_utils.py +++ b/tests/integration/helpers/iceberg_utils.py @@ -237,8 +237,12 @@ def get_creation_expression( table_function=False, use_version_hint=False, run_on_cluster=False, + object_storage_cluster=False, explicit_metadata_path="", additional_settings = [], + storage_type_as_arg=False, + storage_type_in_named_collection=False, + cluster_name_as_literal=True, **kwargs, ): settings_array = list(additional_settings) @@ -249,6 +253,9 @@ def get_creation_expression( if use_version_hint: settings_array.append("iceberg_use_version_hint = true") + if object_storage_cluster: + settings_array.append(f"object_storage_cluster = '{object_storage_cluster}'") + if partition_by: partition_by = "PARTITION BY " + partition_by @@ -265,6 +272,24 @@ def get_creation_expression( else: settings_expression = "" + cluster_name = "'cluster_simple'" if cluster_name_as_literal else "cluster_simple" + + storage_arg = storage_type + engine_part = "" + if (storage_type_in_named_collection): + storage_arg += "_with_type" + elif (storage_type_as_arg): + storage_arg += f", storage_type='{storage_type}'" + else: + if (storage_type == "s3"): + engine_part = "S3" + elif (storage_type == "azure"): + engine_part = "Azure" + elif (storage_type == "hdfs"): + engine_part = "HDFS" + elif (storage_type == "local"): + engine_part = "Local" + if_not_exists_prefix = "" if if_not_exists: if_not_exists_prefix = "IF NOT EXISTS" @@ -277,16 +302,16 @@ def get_creation_expression( if run_on_cluster: assert table_function - return f"icebergS3Cluster('cluster_simple', s3, filename = 'var/lib/clickhouse/user_files/iceberg_data/default/{table_name}/', format={format}, url = 'http://minio1:9001/{bucket}/')" + return f"iceberg{engine_part}Cluster({cluster_name}, {storage_arg}, filename = 'var/lib/clickhouse/user_files/iceberg_data/default/{table_name}/', format={format}, url = 'http://minio1:9001/{bucket}/')" else: if table_function: - return f"icebergS3(s3, filename = 'var/lib/clickhouse/user_files/iceberg_data/default/{table_name}/', format={format}, url = 'http://minio1:9001/{bucket}/')" + return f"iceberg{engine_part}({storage_arg}, filename = 'var/lib/clickhouse/user_files/iceberg_data/default/{table_name}/', format={format}, url = 'http://minio1:9001/{bucket}/')" else: return ( f""" DROP TABLE IF EXISTS {table_name}; CREATE TABLE {if_not_exists_prefix} {table_name} {schema} - ENGINE=IcebergS3(s3, filename = 'var/lib/clickhouse/user_files/iceberg_data/default/{table_name}/', format={format}, url = 'http://minio1:9001/{bucket}/') + ENGINE=Iceberg{engine_part}({storage_arg}, filename = 'var/lib/clickhouse/user_files/iceberg_data/default/{table_name}/', format={format}, url = 'http://minio1:9001/{bucket}/') {order_by} {partition_by} {settings_expression}; @@ -297,19 +322,19 @@ def get_creation_expression( if run_on_cluster: assert table_function return f""" - icebergAzureCluster('cluster_simple', azure, container = '{cluster.azure_container_name}', storage_account_url = '{cluster.env_variables["AZURITE_STORAGE_ACCOUNT_URL"]}', blob_path = '/var/lib/clickhouse/user_files/iceberg_data/default/{table_name}/', format={format}) + iceberg{engine_part}Cluster({cluster_name}, {storage_arg}, container = '{cluster.azure_container_name}', storage_account_url = '{cluster.env_variables["AZURITE_STORAGE_ACCOUNT_URL"]}', blob_path = '/var/lib/clickhouse/user_files/iceberg_data/default/{table_name}/', format={format}) """ else: if table_function: return f""" - icebergAzure(azure, container = '{cluster.azure_container_name}', storage_account_url = '{cluster.env_variables["AZURITE_STORAGE_ACCOUNT_URL"]}', blob_path = '/var/lib/clickhouse/user_files/iceberg_data/default/{table_name}/', format={format}) + iceberg{engine_part}({storage_arg}, container = '{cluster.azure_container_name}', storage_account_url = '{cluster.env_variables["AZURITE_STORAGE_ACCOUNT_URL"]}', blob_path = '/var/lib/clickhouse/user_files/iceberg_data/default/{table_name}/', format={format}) """ else: return ( f""" DROP TABLE IF EXISTS {table_name}; CREATE TABLE {if_not_exists_prefix} {table_name} {schema} - ENGINE=IcebergAzure(azure, container = {cluster.azure_container_name}, storage_account_url = '{cluster.env_variables["AZURITE_STORAGE_ACCOUNT_URL"]}', blob_path = '/var/lib/clickhouse/user_files/iceberg_data/default/{table_name}/', format={format}) + ENGINE=Iceberg{engine_part}({storage_arg}, container = {cluster.azure_container_name}, storage_account_url = '{cluster.env_variables["AZURITE_STORAGE_ACCOUNT_URL"]}', blob_path = '/var/lib/clickhouse/user_files/iceberg_data/default/{table_name}/', format={format}) {order_by} {partition_by} {settings_expression} @@ -320,19 +345,19 @@ def get_creation_expression( if run_on_cluster: assert table_function return f""" - icebergLocalCluster('cluster_simple', local, path = '/var/lib/clickhouse/user_files/iceberg_data/default/{table_name}', format={format}) + iceberg{engine_part}Cluster({cluster_name}, {storage_arg}, path = '/var/lib/clickhouse/user_files/iceberg_data/default/{table_name}/', format={format}) """ else: if table_function: return f""" - icebergLocal(local, path = '/var/lib/clickhouse/user_files/iceberg_data/default/{table_name}', format={format}) + iceberg{engine_part}({storage_arg}, path = '/var/lib/clickhouse/user_files/iceberg_data/default/{table_name}', format={format}) """ else: return ( f""" DROP TABLE IF EXISTS {table_name}; CREATE TABLE {if_not_exists_prefix} {table_name} {schema} - ENGINE=IcebergLocal(local, path = '/var/lib/clickhouse/user_files/iceberg_data/default/{table_name}', format={format}) + ENGINE=Iceberg{engine_part}({storage_arg}, path = '/var/lib/clickhouse/user_files/iceberg_data/default/{table_name}/', format={format}) {order_by} {partition_by} {settings_expression} @@ -449,10 +474,11 @@ def create_iceberg_table( format="Parquet", order_by="", settings=None, + object_storage_cluster=False, **kwargs, ): node.query( - get_creation_expression(storage_type, table_name, cluster, schema, format_version, partition_by, if_not_exists, compression_method, format, order_by, run_on_cluster=run_on_cluster, **kwargs), + get_creation_expression(storage_type, table_name, cluster, schema, format_version, partition_by, if_not_exists, compression_method, format, order_by, run_on_cluster=run_on_cluster, object_storage_cluster=object_storage_cluster, **kwargs), settings=settings, ) diff --git a/tests/integration/test_cluster_discovery/config/config_discovery_disabled.xml b/tests/integration/test_cluster_discovery/config/config_discovery_disabled.xml new file mode 100644 index 000000000000..194579f950bc --- /dev/null +++ b/tests/integration/test_cluster_discovery/config/config_discovery_disabled.xml @@ -0,0 +1,5 @@ + + 0 + + + diff --git a/tests/integration/test_cluster_discovery/config/config_discovery_disabled_with_path.xml b/tests/integration/test_cluster_discovery/config/config_discovery_disabled_with_path.xml new file mode 100644 index 000000000000..0442acdc7d40 --- /dev/null +++ b/tests/integration/test_cluster_discovery/config/config_discovery_disabled_with_path.xml @@ -0,0 +1,11 @@ + + + 0 + + + + /clickhouse/discovery/test_enable_allow_only + + + + diff --git a/tests/integration/test_cluster_discovery/config/config_reload_discovery.xml b/tests/integration/test_cluster_discovery/config/config_reload_discovery.xml new file mode 100644 index 000000000000..acb57ca89084 --- /dev/null +++ b/tests/integration/test_cluster_discovery/config/config_reload_discovery.xml @@ -0,0 +1,12 @@ + + 1 + + + + /clickhouse/discovery/test_reload_cluster + user1 + password123 + + + + diff --git a/tests/integration/test_cluster_discovery/test_config_reload.py b/tests/integration/test_cluster_discovery/test_config_reload.py new file mode 100644 index 000000000000..3df61176f3f3 --- /dev/null +++ b/tests/integration/test_cluster_discovery/test_config_reload.py @@ -0,0 +1,1060 @@ +import time + +import pytest + +from helpers.cluster import ClickHouseCluster + +from .common import check_on_cluster + +cluster = ClickHouseCluster(__file__) + +nodes = { + "node0": cluster.add_instance( + "node0", + main_configs=["config/config_reload_discovery.xml"], + user_configs=["config/users.d/users_with_pwd.xml"], + stay_alive=True, + with_zookeeper=True, + ), + "node1": cluster.add_instance( + "node1", + main_configs=["config/config_reload_discovery.xml"], + user_configs=["config/users.d/users_with_pwd.xml"], + stay_alive=True, + with_zookeeper=True, + ), +} + +CONFIG_PATH = "/etc/clickhouse-server/config.d/config_reload_discovery.xml" + +CONFIG_WITH_PWD = """ + + 1 + + + + /clickhouse/discovery/test_reload_cluster + user1 + password123 + + + + +""" + +CONFIG_WITH_WRONG_PWD = """ + + 1 + + + + /clickhouse/discovery/test_reload_cluster + user1 + wrongpass1234 + + + + +""" + +CONFIG_PASSWORD_AND_SECRET = """ + + 1 + + + + /clickhouse/discovery/test_reload_cluster + user1 + password123 + cluster_secret_value + + + + + + 127.0.0.1 + 9000 + + + + + +""" + +CONFIG_PASSWORD_AND_SECRET_ALLOW_OFF = """ + + 0 + + + + /clickhouse/discovery/test_reload_cluster + user1 + password123 + cluster_secret_value + + + + + + 127.0.0.1 + 9000 + + + + + +""" + +CONFIG_NO_DISCOVERY = """ + + 1 + + + +""" + +CONFIG_WITH_CLUSTER_B = """ + + 1 + + + + /clickhouse/discovery/test_reload_cluster_b + + + + +""" + +CONFIG_MULTICLUSTER_ROOT = """ + + 1 + + + + /clickhouse/discovery/test_reload_cluster + + + + + + /clickhouse/discovery + + + + +""" + +CONFIG_NO_MULTICLUSTER_ROOT = """ + + 1 + + + + /clickhouse/discovery/test_reload_cluster + + + + +""" + +CONFIG_PARTICIPANT = """ + + 1 + + + + /clickhouse/discovery/test_observer_transition + + + + +""" + +CONFIG_OBSERVER = """ + + 1 + + + + /clickhouse/discovery/test_observer_transition + + + + + +""" + +CONFIG_INVISIBLE = """ + + 1 + + + + /clickhouse/discovery/test_invisible_transition + + + + + +""" + +CONFIG_VISIBLE = """ + + 1 + + + + /clickhouse/discovery/test_invisible_transition + + + + +""" + + +@pytest.fixture(scope="module") +def start_cluster(): + try: + cluster.start() + yield cluster + finally: + cluster.shutdown() + + +def wait_cluster_query(node, cluster_name, password="passwordAbc", should_succeed=True, retries=10): + query = ( + f"SELECT sum(number) FROM clusterAllReplicas('{cluster_name}', numbers(3)) " + f"GROUP BY hostname()" + ) + last_error = "" + for retry in range(retries): + if should_succeed: + try: + result = node.query(query, password=password) + if result.count("\n") >= 2: + return result + except Exception as e: + last_error = str(e) + else: + try: + error = node.query_and_get_error(query, password=password) + if "Authentication failed" in error or error: + return error + except Exception as e: + last_error = str(e) + time.sleep(1 + retry) + raise AssertionError( + f"wait_cluster_query failed (should_succeed={should_succeed}): {last_error}" + ) + + +def reload_config_on_all(config_body): + for node in nodes.values(): + node.replace_config(CONFIG_PATH, config_body) + node.query("SYSTEM RELOAD CONFIG", password="passwordAbc") + + +def reload_config_on_node(node, config_body): + node.replace_config(CONFIG_PATH, config_body) + node.query("SYSTEM RELOAD CONFIG", password="passwordAbc") + + +def test_reload_static_discovery_ownership_transitions(start_cluster): + """Static ↔ discovery must not leave a stale Clusters::impl entry that shadows discovery.""" + cluster_name = "test_ownership_cluster" + config_static = f""" + + 1 + + <{cluster_name}> + + + 127.0.0.1 + 9000 + + + + + +""" + config_discovery = f""" + + 1 + + <{cluster_name}> + + /clickhouse/discovery/{cluster_name} + + + + +""" + + def host_names(): + return [ + node.query( + f"SELECT groupArray(host_name) FROM system.clusters WHERE cluster = '{cluster_name}'", + password="passwordAbc", + ).strip() + for node in nodes.values() + ] + + def wait_hosts_contain(needle, msg, retries=10): + for _ in range(retries): + hosts = host_names() + if all(needle in h for h in hosts): + return + time.sleep(1) + raise AssertionError(f"{msg}: {hosts}") + + def wait_cluster_absent(msg, retries=10): + for _ in range(retries): + counts = [ + int( + node.query( + f"SELECT count() FROM system.clusters WHERE cluster = '{cluster_name}'", + password="passwordAbc", + ) + ) + for node in nodes.values() + ] + if all(c == 0 for c in counts): + return + time.sleep(1) + raise AssertionError(f"{msg}: {counts}") + + # Static only — placeholder host must be visible. + reload_config_on_all(config_static) + wait_hosts_contain("127.0.0.1", "Static ownership not applied") + + # Static → discovery: discovery must win (not keep 127.0.0.1 from impl). + reload_config_on_all(config_discovery) + check_on_cluster( + list(nodes.values()), + len(nodes), + cluster_name=cluster_name, + what="count()", + msg="Discovery ownership not applied after static→discovery", + query_params={"password": "passwordAbc"}, + retries=6, + ) + for hosts in host_names(): + if "127.0.0.1" in hosts: + raise AssertionError( + f"Stale static Cluster still shadows discovery after reload: {hosts}" + ) + + # Discovery → static. + reload_config_on_all(config_static) + wait_hosts_contain("127.0.0.1", "Static ownership not restored after discovery→static") + + # Static → removed. + reload_config_on_all(CONFIG_NO_DISCOVERY) + wait_cluster_absent("Cluster still present after removal") + + # static → discovery → removed (skip return to static). + reload_config_on_all(config_static) + wait_hosts_contain("127.0.0.1", "Static ownership not applied before second discovery cycle") + reload_config_on_all(config_discovery) + check_on_cluster( + list(nodes.values()), + len(nodes), + cluster_name=cluster_name, + what="count()", + msg="Discovery ownership not applied on second cycle", + query_params={"password": "passwordAbc"}, + retries=6, + ) + reload_config_on_all(CONFIG_NO_DISCOVERY) + wait_cluster_absent("Cluster still present after discovery→removed") + + +def test_reload_discovery_credentials(start_cluster): + reload_config_on_all(CONFIG_WITH_PWD) + + check_on_cluster( + list(nodes.values()), + len(nodes), + cluster_name="test_reload_cluster", + what="count()", + msg="Wrong nodes count after credential config apply", + query_params={"password": "passwordAbc"}, + retries=6, + ) + + wait_cluster_query(nodes["node0"], "test_reload_cluster", should_succeed=True) + + reload_config_on_all(CONFIG_WITH_WRONG_PWD) + wait_cluster_query(nodes["node0"], "test_reload_cluster", should_succeed=False) + + reload_config_on_all(CONFIG_WITH_PWD) + wait_cluster_query(nodes["node0"], "test_reload_cluster", should_succeed=True) + + +def test_reload_invalid_discovery_does_not_partially_apply(start_cluster): + """Invalid discovery must fail the reload before Clusters / discovery diverge.""" + reload_config_on_all(CONFIG_WITH_PWD) + wait_cluster_query(nodes["node0"], "test_reload_cluster", should_succeed=True) + + for node in nodes.values(): + node.replace_config(CONFIG_PATH, CONFIG_PASSWORD_AND_SECRET) + error = node.query_and_get_error("SYSTEM RELOAD CONFIG", password="passwordAbc") + assert "password" in error and "secret" in error, error + + wait_cluster_query(nodes["node0"], "test_reload_cluster", should_succeed=True) + + for node in nodes.values(): + count = int( + node.query( + "SELECT count() FROM system.clusters WHERE cluster = 'test_partial_apply_marker'", + password="passwordAbc", + ) + ) + assert count == 0, "Static cluster from rejected config was partially applied" + + reload_config_on_all(CONFIG_WITH_PWD) + + +def test_reload_invalid_discovery_allow_off_does_not_partially_apply(start_cluster): + """Existing discovery must still validate when allow is turned off on reload.""" + reload_config_on_all(CONFIG_WITH_PWD) + wait_cluster_query(nodes["node0"], "test_reload_cluster", should_succeed=True) + + for node in nodes.values(): + node.replace_config(CONFIG_PATH, CONFIG_PASSWORD_AND_SECRET_ALLOW_OFF) + error = node.query_and_get_error("SYSTEM RELOAD CONFIG", password="passwordAbc") + assert "password" in error and "secret" in error, error + + wait_cluster_query(nodes["node0"], "test_reload_cluster", should_succeed=True) + + for node in nodes.values(): + count = int( + node.query( + "SELECT count() FROM system.clusters " + "WHERE cluster = 'test_partial_apply_marker_allow_off'", + password="passwordAbc", + ) + ) + assert count == 0, "Static cluster from rejected allow=0 config was partially applied" + + reload_config_on_all(CONFIG_WITH_PWD) + + +def test_reload_add_remove_discovery_cluster(start_cluster): + reload_config_on_all(CONFIG_NO_DISCOVERY) + time.sleep(2) + + for node in nodes.values(): + count = int( + node.query( + "SELECT count() FROM system.clusters WHERE cluster = 'test_reload_cluster_b'", + password="passwordAbc", + ) + ) + assert count == 0 + + reload_config_on_all(CONFIG_WITH_CLUSTER_B) + check_on_cluster( + list(nodes.values()), + len(nodes), + cluster_name="test_reload_cluster_b", + what="count()", + msg="Cluster was not added after config reload", + query_params={"password": "passwordAbc"}, + retries=6, + ) + + reload_config_on_all(CONFIG_NO_DISCOVERY) + for retry in range(10): + counts = [ + int( + node.query( + "SELECT count() FROM system.clusters WHERE cluster = 'test_reload_cluster_b'", + password="passwordAbc", + ) + ) + for node in nodes.values() + ] + if all(c == 0 for c in counts): + break + time.sleep(1) + else: + raise AssertionError(f"Cluster was not removed after config reload: {counts}") + + +def test_reload_remove_retries_failed_unregister(start_cluster): + """Failed Keeper unregister is retried; re-add of the same path must not be undone by that retry.""" + reload_config_on_all(CONFIG_WITH_PWD) + check_on_cluster( + list(nodes.values()), + len(nodes), + cluster_name="test_reload_cluster", + what="count()", + msg="Cluster not ready before unregister-retry test", + query_params={"password": "passwordAbc"}, + retries=6, + ) + + node0 = nodes["node0"] + node1 = nodes["node1"] + + def enable_unregister_failpoint(): + node0.query( + "SYSTEM ENABLE FAILPOINT cluster_discovery_unregister_fail", + password="passwordAbc", + ) + + def disable_unregister_failpoint(): + node0.query( + "SYSTEM DISABLE FAILPOINT cluster_discovery_unregister_fail", + password="passwordAbc", + ) + + def wait_local_cluster_gone(): + for _ in range(10): + count = int( + node0.query( + "SELECT count() FROM system.clusters WHERE cluster = 'test_reload_cluster'", + password="passwordAbc", + ) + ) + if count == 0: + return + time.sleep(1) + raise AssertionError("node0 still exposes removed discovery cluster after reload") + + # --- remove while unregister fails, then retry cleanup after failpoint is cleared --- + enable_unregister_failpoint() + try: + reload_config_on_node(node0, CONFIG_NO_DISCOVERY) + wait_local_cluster_gone() + + for _ in range(10): + hosts = int( + node1.query( + "SELECT count() FROM system.clusters WHERE cluster = 'test_reload_cluster'", + password="passwordAbc", + ) + ) + if hosts == len(nodes): + break + time.sleep(1) + else: + raise AssertionError( + "Expected node0 ephemeral to remain visible on node1 while unregister failpoint is on" + ) + finally: + disable_unregister_failpoint() + + for _ in range(20): + hosts = int( + node1.query( + "SELECT count() FROM system.clusters WHERE cluster = 'test_reload_cluster'", + password="passwordAbc", + ) + ) + if hosts == 1: + break + time.sleep(1) + else: + raise AssertionError( + f"node0 ephemeral was not cleaned up after unregister retry; hosts on node1={hosts}" + ) + + reload_config_on_all(CONFIG_WITH_PWD) + check_on_cluster( + list(nodes.values()), + len(nodes), + cluster_name="test_reload_cluster", + what="count()", + msg="Cluster not restored before remove/re-add unregister test", + query_params={"password": "passwordAbc"}, + retries=6, + ) + + # --- remove, re-add same path while unregister still failing; pending retry must not drop the node --- + enable_unregister_failpoint() + try: + reload_config_on_node(node0, CONFIG_NO_DISCOVERY) + wait_local_cluster_gone() + + reload_config_on_node(node0, CONFIG_WITH_PWD) + check_on_cluster( + [node0, node1], + len(nodes), + cluster_name="test_reload_cluster", + what="count()", + msg="Cluster was not restored on node0 after re-add", + query_params={"password": "passwordAbc"}, + retries=6, + ) + finally: + disable_unregister_failpoint() + + for _ in range(15): + hosts = int( + node1.query( + "SELECT count() FROM system.clusters WHERE cluster = 'test_reload_cluster'", + password="passwordAbc", + ) + ) + if hosts != len(nodes): + raise AssertionError( + f"Pending unregister deleted re-registered ephemeral; hosts on node1={hosts}" + ) + time.sleep(1) + + reload_config_on_all(CONFIG_WITH_PWD) + + +def test_reload_add_remove_multicluster_root(start_cluster): + reload_config_on_all(CONFIG_MULTICLUSTER_ROOT) + + check_on_cluster( + list(nodes.values()), + len(nodes), + cluster_name="test_reload_cluster", + what="count()", + msg="Static discovery cluster missing", + query_params={"password": "passwordAbc"}, + retries=6, + ) + + # Observer root should discover the static cluster under /clickhouse/discovery + for retry in range(15): + counts = [ + int( + node.query( + "SELECT count() FROM system.clusters WHERE cluster = 'test_reload_cluster'", + password="passwordAbc", + ) + ) + for node in nodes.values() + ] + if all(c == len(nodes) for c in counts): + break + time.sleep(1) + + reload_config_on_all(CONFIG_NO_MULTICLUSTER_ROOT) + # Static cluster must remain after multicluster root removal + check_on_cluster( + list(nodes.values()), + len(nodes), + cluster_name="test_reload_cluster", + what="count()", + msg="Static cluster disappeared after multicluster root removal", + query_params={"password": "passwordAbc"}, + retries=6, + ) + + reload_config_on_all(CONFIG_MULTICLUSTER_ROOT) + check_on_cluster( + list(nodes.values()), + len(nodes), + cluster_name="test_reload_cluster", + what="count()", + msg="Static cluster missing after restoring multicluster root", + query_params={"password": "passwordAbc"}, + retries=6, + ) + + +def test_reload_participant_to_observer_unregisters(start_cluster): + """Participant -> observer reload must remove this node's ephemeral ZK registration.""" + reload_config_on_all(CONFIG_PARTICIPANT) + + check_on_cluster( + list(nodes.values()), + len(nodes), + cluster_name="test_observer_transition", + what="count()", + msg="Both participants should be visible before observer transition", + query_params={"password": "passwordAbc"}, + retries=6, + ) + + reload_config_on_node(nodes["node0"], CONFIG_OBSERVER) + + # node0 must disappear from node1's view without waiting for ZK session expiry. + for retry in range(15): + hosts = ( + nodes["node1"] + .query( + "SELECT host_name FROM system.clusters " + "WHERE cluster = 'test_observer_transition' ORDER BY host_name", + password="passwordAbc", + ) + .strip() + .split("\n") + ) + hosts = [h for h in hosts if h] + if hosts == ["node1"]: + break + time.sleep(1) + else: + raise AssertionError( + f"node0 still advertised after observer reload; hosts on node1: {hosts}" + ) + + # Observer still sees the remaining participant. + check_on_cluster( + [nodes["node0"]], + 1, + cluster_name="test_observer_transition", + what="count()", + msg="Observer should still see the remaining participant", + query_params={"password": "passwordAbc"}, + retries=6, + ) + + +def test_reload_invisible_to_visible_populates_cluster(start_cluster): + """Invisible -> visible reload must upsert and publish nodes promptly.""" + reload_config_on_all(CONFIG_INVISIBLE) + + for retry in range(10): + counts = [ + int( + node.query( + "SELECT count() FROM system.clusters " + "WHERE cluster = 'test_invisible_transition'", + password="passwordAbc", + ) + ) + for node in nodes.values() + ] + if all(c == 0 for c in counts): + break + time.sleep(1) + else: + raise AssertionError( + f"Invisible cluster should not appear in system.clusters: {counts}" + ) + + reload_config_on_all(CONFIG_VISIBLE) + + check_on_cluster( + list(nodes.values()), + len(nodes), + cluster_name="test_invisible_transition", + what="count()", + msg="Cluster did not appear after becoming visible", + query_params={"password": "passwordAbc"}, + retries=6, + ) + + reload_config_on_all(CONFIG_INVISIBLE) + + for retry in range(15): + counts = [ + int( + node.query( + "SELECT count() FROM system.clusters " + "WHERE cluster = 'test_invisible_transition'", + password="passwordAbc", + ) + ) + for node in nodes.values() + ] + if all(c == 0 for c in counts): + break + time.sleep(1) + else: + raise AssertionError( + f"Cluster still visible after invisible reload: {counts}" + ) + + +def test_reload_static_replaces_dynamic_same_name(start_cluster): + """Static for a name already discovered via multicluster must replace it cleanly.""" + config_participant = """ + + 1 + + + + /clickhouse/discovery/test_collision_cluster + + + + +""" + config_multicluster_observer = """ + + 1 + + + + + /clickhouse/discovery + + + + +""" + config_static_observer = """ + + 1 + + + + /clickhouse/discovery/test_collision_cluster + + + + + +""" + config_static_and_multicluster = """ + + 1 + + + + /clickhouse/discovery/test_collision_cluster + + + + + + + /clickhouse/discovery + + + + +""" + + reload_config_on_node(nodes["node1"], config_participant) + reload_config_on_node(nodes["node0"], config_multicluster_observer) + + check_on_cluster( + [nodes["node0"]], + 1, + cluster_name="test_collision_cluster", + what="count()", + msg="Observer should discover dynamic test_collision_cluster", + query_params={"password": "passwordAbc"}, + retries=6, + ) + + # Replace dynamic discovery with static config of the same name. + reload_config_on_node(nodes["node0"], config_static_observer) + + check_on_cluster( + [nodes["node0"]], + 1, + cluster_name="test_collision_cluster", + what="count()", + msg="Static observer should still see the participant after replacing dynamic", + query_params={"password": "passwordAbc"}, + retries=6, + ) + + # Static + multicluster: static shadows the dynamic name. Removing only the static entry + # must rescan roots so the dynamic cluster reappears without waiting for force refresh. + reload_config_on_node(nodes["node0"], config_static_and_multicluster) + check_on_cluster( + [nodes["node0"]], + 1, + cluster_name="test_collision_cluster", + what="count()", + msg="Static+multicluster observer should see the participant", + query_params={"password": "passwordAbc"}, + retries=6, + ) + + reload_config_on_node(nodes["node0"], config_multicluster_observer) + check_on_cluster( + [nodes["node0"]], + 1, + cluster_name="test_collision_cluster", + what="count()", + msg="Dynamic cluster must reappear after static shadow is removed", + query_params={"password": "passwordAbc"}, + retries=6, + ) + + +def _registration_config(hostname, shard): + return f""" + + 1 + + + + /clickhouse/discovery/test_registration_reload + {hostname} + {shard} + + + + +""" + + +def _registration_rows(node): + return node.query( + "SELECT host_name, shard_num FROM system.clusters " + "WHERE cluster = 'test_registration_reload' ORDER BY host_name, shard_num " + "FORMAT TSV", + password="passwordAbc", + ).strip() + + +def test_reload_my_hostname_and_shard_updates_local_and_peer(start_cluster): + """Registration field reload must refresh payloads locally and on peers without membership churn.""" + reload_config_on_node(nodes["node0"], _registration_config("reg-host-node0", 1)) + reload_config_on_node(nodes["node1"], _registration_config("reg-host-node1", 1)) + + expected_initial = "reg-host-node0\t1\nreg-host-node1\t1" + for retry in range(15): + rows = {_registration_rows(node) for node in nodes.values()} + if rows == {expected_initial}: + break + time.sleep(1) + else: + raise AssertionError(f"Initial registration view not ready: {rows}") + + # Change hostname and shard on node1 only; UUID set stays the same. + reload_config_on_node(nodes["node1"], _registration_config("reg-host-node1-renamed", 2)) + + expected_updated = "reg-host-node0\t1\nreg-host-node1-renamed\t2" + for retry in range(15): + rows = {_registration_rows(node) for node in nodes.values()} + if rows == {expected_updated}: + break + time.sleep(1) + else: + raise AssertionError( + f"Hostname/shard reload did not propagate to local and peer system.clusters: {rows}" + ) + + +def test_keeper_exception_after_wait_restores_retry_signal(start_cluster): + """A one-shot Keeper throw after Flags::wait must not leave peer updates stuck forever.""" + reload_config_on_node(nodes["node0"], _registration_config("reg-host-node0", 1)) + reload_config_on_node(nodes["node1"], _registration_config("reg-host-node1", 1)) + + expected_initial = "reg-host-node0\t1\nreg-host-node1\t1" + for retry in range(15): + rows = {_registration_rows(node) for node in nodes.values()} + if rows == {expected_initial}: + break + time.sleep(1) + else: + raise AssertionError(f"Initial registration view not ready: {rows}") + + node0 = nodes["node0"] + node0.query( + "SYSTEM ENABLE FAILPOINT cluster_discovery_retry_signal_fail", + password="passwordAbc", + ) + + # Peer-only registration change: node0 is woken by the Keeper children watch, not by a + # local config reload. Without restoring flags/wake after the failpoint throw, node0 would + # wait until an unrelated event (or never) to see the new payload. + reload_config_on_node(nodes["node1"], _registration_config("reg-host-node1-renamed", 2)) + + expected_updated = "reg-host-node0\t1\nreg-host-node1-renamed\t2" + for retry in range(20): + rows = _registration_rows(node0) + if rows == expected_updated: + break + time.sleep(1) + else: + raise AssertionError( + f"node0 did not recover peer registration update after one-shot Keeper failpoint; " + f"got {rows!r}" + ) + + # Peer that did not hit the failpoint must also converge. + for retry in range(15): + rows = {_registration_rows(node) for node in nodes.values()} + if rows == {expected_updated}: + break + time.sleep(1) + else: + raise AssertionError(f"Cluster views did not converge after retry-signal recovery: {rows}") + + +def _shared_path_aliases_config(include_alias_a, alias_a_observer=False): + alias_a = "" + if include_alias_a: + observer = "\n " if alias_a_observer else "" + alias_a = f""" + + + /clickhouse/discovery/test_shared_path_aliases{observer} + + """ + return f""" + + 1 + {alias_a} + + + /clickhouse/discovery/test_shared_path_aliases + + + + +""" + + +def _alias_b_host_count(node): + return int( + node.query( + "SELECT count() FROM system.clusters WHERE cluster = 'alias_b'", + password="passwordAbc", + ) + ) + + +def test_reload_remove_shared_path_alias_keeps_peer_membership(start_cluster): + """Removing one participant alias must not delete the shared ephemeral while another remains.""" + # node0: two aliases on one Keeper path; node1: only the retained alias. + reload_config_on_node(nodes["node0"], _shared_path_aliases_config(include_alias_a=True)) + reload_config_on_node(nodes["node1"], _shared_path_aliases_config(include_alias_a=False)) + + for retry in range(15): + if _alias_b_host_count(nodes["node1"]) == len(nodes): + break + time.sleep(1) + else: + raise AssertionError("alias_b not ready with both nodes before shared-path alias remove") + + reload_config_on_node(nodes["node0"], _shared_path_aliases_config(include_alias_a=False)) + + # Membership must never transiently drop: the shared ephemeral must stay. + for _ in range(20): + hosts = _alias_b_host_count(nodes["node1"]) + if hosts != len(nodes): + raise AssertionError( + f"Peer lost membership after removing shared-path alias_a; alias_b hosts={hosts}" + ) + time.sleep(0.2) + + # Convert-to-observer on a restored alias_a must also keep the shared registration. + reload_config_on_node(nodes["node0"], _shared_path_aliases_config(include_alias_a=True)) + for retry in range(15): + if _alias_b_host_count(nodes["node1"]) == len(nodes): + break + time.sleep(1) + else: + raise AssertionError("alias_b not ready before shared-path alias observer convert") + + reload_config_on_node( + nodes["node0"], + _shared_path_aliases_config(include_alias_a=True, alias_a_observer=True), + ) + + for _ in range(20): + hosts = _alias_b_host_count(nodes["node1"]) + if hosts != len(nodes): + raise AssertionError( + f"Peer lost membership after converting shared-path alias_a to observer; " + f"alias_b hosts={hosts}" + ) + time.sleep(0.2) diff --git a/tests/integration/test_cluster_discovery/test_enable_after_startup.py b/tests/integration/test_cluster_discovery/test_enable_after_startup.py new file mode 100644 index 000000000000..56ecd27adfc1 --- /dev/null +++ b/tests/integration/test_cluster_discovery/test_enable_after_startup.py @@ -0,0 +1,126 @@ +import time + +import pytest + +from helpers.cluster import ClickHouseCluster + +from .common import check_on_cluster + +cluster = ClickHouseCluster(__file__) + +nodes = { + "node0": cluster.add_instance( + "node0", + main_configs=["config/config_discovery_disabled.xml"], + user_configs=["config/users.d/users_with_pwd.xml"], + stay_alive=True, + with_zookeeper=True, + ), + "node1": cluster.add_instance( + "node1", + main_configs=["config/config_discovery_disabled.xml"], + user_configs=["config/users.d/users_with_pwd.xml"], + stay_alive=True, + with_zookeeper=True, + ), +} + +CONFIG_PATH = "/etc/clickhouse-server/config.d/config_discovery_disabled.xml" + +CONFIG_ENABLED = """ + + 1 + + + + /clickhouse/discovery/test_enable_after_startup + + + + +""" + +CONFIG_BAD_AUX_KEEPER = """ + + 1 + + + + missing_aux_keeper:/clickhouse/discovery/test_pending_during_init + + + + +""" + +CONFIG_GOOD_AFTER_BAD_INIT = """ + + 1 + + + + /clickhouse/discovery/test_pending_during_init + + + + +""" + + +@pytest.fixture(scope="module") +def start_cluster(): + try: + cluster.start() + yield cluster + finally: + cluster.shutdown() + + +def test_reload_applies_while_initial_update_is_failing(start_cluster): + """Pending config must be consumed before retrying initialUpdate, or a fix reload is stuck.""" + for node in nodes.values(): + node.replace_config(CONFIG_PATH, CONFIG_BAD_AUX_KEEPER) + node.query("SYSTEM RELOAD CONFIG", password="passwordAbc") + + # Let the worker retry failed init on the obsolete auxiliary keeper. + time.sleep(2) + + for node in nodes.values(): + node.replace_config(CONFIG_PATH, CONFIG_GOOD_AFTER_BAD_INIT) + node.query("SYSTEM RELOAD CONFIG", password="passwordAbc") + + check_on_cluster( + list(nodes.values()), + len(nodes), + cluster_name="test_pending_during_init", + what="count()", + msg="Corrective reload was not applied while discovery init was failing", + query_params={"password": "passwordAbc"}, + retries=6, + ) + + +def test_enable_discovery_after_startup_starts_worker(start_cluster): + """Creating ClusterDiscovery on reload must start the worker without a process restart.""" + for node in nodes.values(): + count = int( + node.query( + "SELECT count() FROM system.clusters WHERE cluster = 'test_enable_after_startup'", + password="passwordAbc", + ) + ) + assert count == 0 + + for node in nodes.values(): + node.replace_config(CONFIG_PATH, CONFIG_ENABLED) + node.query("SYSTEM RELOAD CONFIG", password="passwordAbc") + + check_on_cluster( + list(nodes.values()), + len(nodes), + cluster_name="test_enable_after_startup", + what="count()", + msg="Discovery cluster missing after enabling feature post-startup", + query_params={"password": "passwordAbc"}, + retries=6, + ) diff --git a/tests/integration/test_cluster_discovery/test_enable_allow_only.py b/tests/integration/test_cluster_discovery/test_enable_allow_only.py new file mode 100644 index 000000000000..e3235bbbb3e7 --- /dev/null +++ b/tests/integration/test_cluster_discovery/test_enable_allow_only.py @@ -0,0 +1,152 @@ +import time + +import pytest + +from helpers.cluster import ClickHouseCluster + +from .common import check_on_cluster + +cluster = ClickHouseCluster(__file__) + +nodes = { + "node0": cluster.add_instance( + "node0", + main_configs=["config/config_discovery_disabled_with_path.xml"], + user_configs=["config/users.d/users_with_pwd.xml"], + stay_alive=True, + with_zookeeper=True, + ), + "node1": cluster.add_instance( + "node1", + main_configs=["config/config_discovery_disabled_with_path.xml"], + user_configs=["config/users.d/users_with_pwd.xml"], + stay_alive=True, + with_zookeeper=True, + ), +} + +CONFIG_PATH = "/etc/clickhouse-server/config.d/config_discovery_disabled_with_path.xml" + +CONFIG_ALLOW_DISABLED = """ + + 0 + + + + /clickhouse/discovery/test_enable_allow_only + + + + +""" + +CONFIG_ALLOW_ENABLED = """ + + 1 + + + + /clickhouse/discovery/test_enable_allow_only + + + + +""" + + +@pytest.fixture(scope="module") +def start_cluster(): + try: + cluster.start() + yield cluster + finally: + cluster.shutdown() + + +def _cluster_host_count(node): + return int( + node.query( + "SELECT count() FROM system.clusters WHERE cluster = 'test_enable_allow_only'", + password="passwordAbc", + ) + ) + + +def test_enable_allow_flag_only_starts_worker(start_cluster): + """Flipping only allow_experimental_cluster_discovery must start discovery (remote_servers unchanged).""" + for node in nodes.values(): + node.replace_config(CONFIG_PATH, CONFIG_ALLOW_DISABLED) + node.query("SYSTEM RELOAD CONFIG", password="passwordAbc") + + for _ in range(15): + if all(_cluster_host_count(node) == 0 for node in nodes.values()): + break + time.sleep(1) + else: + raise AssertionError("Discovery cluster still published after allow=0 baseline") + + for node in nodes.values(): + node.replace_config(CONFIG_PATH, CONFIG_ALLOW_ENABLED) + node.query("SYSTEM RELOAD CONFIG", password="passwordAbc") + + check_on_cluster( + list(nodes.values()), + len(nodes), + cluster_name="test_enable_allow_only", + what="count()", + msg="Discovery cluster missing after allow-flag-only reload", + query_params={"password": "passwordAbc"}, + retries=6, + ) + + +def test_disable_allow_flag_only_stops_discovery(start_cluster): + """allow 1 → 0 must unregister and unpublish; 0 → 1 must restore without changing remote_servers.""" + for node in nodes.values(): + node.replace_config(CONFIG_PATH, CONFIG_ALLOW_ENABLED) + node.query("SYSTEM RELOAD CONFIG", password="passwordAbc") + + check_on_cluster( + list(nodes.values()), + len(nodes), + cluster_name="test_enable_allow_only", + what="count()", + msg="Discovery cluster not ready before allow-disable test", + query_params={"password": "passwordAbc"}, + retries=6, + ) + + node0 = nodes["node0"] + node1 = nodes["node1"] + + node0.replace_config(CONFIG_PATH, CONFIG_ALLOW_DISABLED) + node0.query("SYSTEM RELOAD CONFIG", password="passwordAbc") + + for _ in range(15): + if _cluster_host_count(node0) == 0: + break + time.sleep(1) + else: + raise AssertionError("node0 still publishes discovery cluster after allow=0 reload") + + for _ in range(15): + if _cluster_host_count(node1) == 1: + break + time.sleep(1) + else: + raise AssertionError( + f"node1 still sees node0 in Keeper after allow=0 on node0; hosts={_cluster_host_count(node1)}" + ) + + node0.replace_config(CONFIG_PATH, CONFIG_ALLOW_ENABLED) + node0.query("SYSTEM RELOAD CONFIG", password="passwordAbc") + + check_on_cluster( + list(nodes.values()), + len(nodes), + cluster_name="test_enable_allow_only", + what="count()", + msg="Discovery cluster missing after allow 0 → 1 reload", + query_params={"password": "passwordAbc"}, + retries=6, + ) diff --git a/tests/integration/test_database_delta/test.py b/tests/integration/test_database_delta/test.py index 09de0ae73e11..6868e0568ba2 100644 --- a/tests/integration/test_database_delta/test.py +++ b/tests/integration/test_database_delta/test.py @@ -13,6 +13,9 @@ UC_LOG = "/var/lib/clickhouse/user_files/unitycatalog/uc.log" +CATALOG_NAME = "unity_catalog_test_db" + + def start_unity_catalog(node): node.exec_in_container( [ @@ -1031,3 +1034,42 @@ def test_varchar_char_types_via_unity_catalog(started_cluster, use_delta_kernel) .strip() ) assert row == "1\thello varchar\thello char" + + +def test_namespace_filter(started_cluster): + node = started_cluster.instances["node1"] + + # Use the same table name in all namespaces + table_name = f"table_{uuid.uuid4()}".replace("-", "_") + namespace_prefix = f"namespace_{uuid.uuid4()}_".replace("-", "_") + + + def create_namespace(suffix): + namespace = f"{namespace_prefix}{suffix}" + execute_spark_query( + node, f"CREATE SCHEMA {namespace}" + ) + execute_spark_query( + node, f"CREATE TABLE {namespace}.{table_name} (col1 int, col2 double) using Delta location '/var/lib/clickhouse/user_files/tmp/{namespace}/{table_name}'" + ) + + create_namespace("alpha"); + create_namespace("bravo"); + + node.query( + f""" + drop database if exists {CATALOG_NAME}; + create database {CATALOG_NAME} + engine DataLakeCatalog('http://localhost:8080/api/2.1/unity-catalog') + settings warehouse = 'unity', catalog_type='unity', vended_credentials=false, namespaces = '{namespace_prefix}alpha' + """, + settings={"allow_database_unity_catalog": "1"}, + ) + + assert node.query(f"SELECT name FROM system.tables WHERE database='{CATALOG_NAME}' ORDER BY name", settings={"show_data_lake_catalogs_in_system_tables": 1}) == TSV( + [ + [f"{namespace_prefix}alpha.{table_name}"], + ]) + + assert node.query(f"SELECT count() FROM {CATALOG_NAME}.`{namespace_prefix}alpha.{table_name}`") == "0\n" + assert "is filtered by `namespaces` database parameter." in node.query_and_get_error(f"SELECT count() FROM {CATALOG_NAME}.`{namespace_prefix}bravo.{table_name}`") diff --git a/tests/integration/test_database_glue/test.py b/tests/integration/test_database_glue/test.py index 4ca41b7ac988..e0d32c159cda 100644 --- a/tests/integration/test_database_glue/test.py +++ b/tests/integration/test_database_glue/test.py @@ -16,6 +16,7 @@ from pyiceberg.table.sorting import SortField, SortOrder from pyiceberg.transforms import DayTransform, IdentityTransform from helpers.config_cluster import minio_access_key, minio_secret_key +from helpers.test_tools import TSV import decimal from pyiceberg.types import ( DoubleType, @@ -1302,6 +1303,45 @@ def test_check_database(started_cluster): "SYSTEM DISABLE FAILPOINT check_database_datalake_negative" ) + +def test_namespace_filter(started_cluster): + node = started_cluster.instances["node1"] + + # Use the same table name in all namespaces + table_name = f"table_{uuid.uuid4()}" + table2_name = f"table2_{uuid.uuid4()}" + namespace_prefix = f"namespace_{uuid.uuid4()}_" + + catalog = load_catalog_impl(started_cluster) + + def create_namespace(suffix): + namespace = f"{namespace_prefix}{suffix}" + catalog.create_namespace(namespace) + create_table(catalog, namespace, table_name, DEFAULT_SCHEMA, PartitionSpec(), DEFAULT_SORT_ORDER) + + create_namespace("alpha"); + create_namespace("bravo"); + + create_clickhouse_glue_database(started_cluster, node, CATALOG_NAME, + additional_settings={ + "namespaces": f"{namespace_prefix}alpha" + }) + + assert node.query(f"SELECT name FROM system.tables WHERE database='{CATALOG_NAME}' ORDER BY name", settings={"show_data_lake_catalogs_in_system_tables": 1}) == TSV( + [ + [f"{namespace_prefix}alpha.{table_name}"], + ]) + + assert node.query(f"SELECT count() FROM {CATALOG_NAME}.`{namespace_prefix}alpha.{table_name}`") == "0\n" + assert "is filtered by `namespaces` database parameter." in node.query_and_get_error(f"SELECT count() FROM {CATALOG_NAME}.`{namespace_prefix}bravo.{table_name}`") + + node.query(f"CREATE TABLE {CATALOG_NAME}.`{namespace_prefix}alpha.{table2_name}` (x String) ENGINE = IcebergS3('http://minio1:9001/warehouse-glue/{namespace_prefix}alpha/a1/{table2_name}/', '{minio_access_key}', '{minio_secret_key}')") + assert "is filtered by `namespaces` database parameter." in node.query_and_get_error(f"CREATE TABLE {CATALOG_NAME}.`{namespace_prefix}bravo.{table2_name}` (x String) ENGINE = IcebergS3('http://minio1:9001/warehouse-glue/{namespace_prefix}bravo/{table2_name}/', '{minio_access_key}', '{minio_secret_key}')") + + node.query(f"DROP TABLE {CATALOG_NAME}.`{namespace_prefix}alpha.{table_name}`") + assert "is filtered by `namespaces` database parameter." in node.query_and_get_error(f"DROP TABLE {CATALOG_NAME}.`{namespace_prefix}bravo.{table_name}`") + + def test_sts_smoke(started_cluster): """Test that STS authentication works with Glue catalog using role_arn and role_session_name""" node = started_cluster.instances["node1"] diff --git a/tests/integration/test_database_iceberg/configs/iceberg_partition_timezone.xml b/tests/integration/test_database_iceberg/configs/iceberg_partition_timezone.xml new file mode 100644 index 000000000000..40aebd33c515 --- /dev/null +++ b/tests/integration/test_database_iceberg/configs/iceberg_partition_timezone.xml @@ -0,0 +1,7 @@ + + + + UTC + + + diff --git a/tests/integration/test_database_iceberg/configs/timezone.xml b/tests/integration/test_database_iceberg/configs/timezone.xml new file mode 100644 index 000000000000..269e52ef2247 --- /dev/null +++ b/tests/integration/test_database_iceberg/configs/timezone.xml @@ -0,0 +1,3 @@ + + Asia/Istanbul + \ No newline at end of file diff --git a/tests/integration/test_database_iceberg/test.py b/tests/integration/test_database_iceberg/test.py index 8c421057eaca..909e4f4661f5 100644 --- a/tests/integration/test_database_iceberg/test.py +++ b/tests/integration/test_database_iceberg/test.py @@ -16,23 +16,26 @@ from avro.datafile import DataFileReader, DataFileWriter from avro.io import DatumReader, DatumWriter from pyiceberg.catalog import load_catalog -from pyiceberg.partitioning import PartitionField, PartitionSpec +from pyiceberg.partitioning import PartitionField, PartitionSpec, UNPARTITIONED_PARTITION_SPEC from pyiceberg.schema import Schema from pyiceberg.table.sorting import SortField, SortOrder from pyiceberg.transforms import DayTransform, IdentityTransform from pyiceberg.types import ( DoubleType, + LongType, NestedField, StringType, StructType, TimestampType, TimestamptzType ) +from pyiceberg.table.sorting import UNSORTED_SORT_ORDER from helpers.cluster import ClickHouseCluster from helpers.config_cluster import minio_secret_key, minio_access_key from helpers.client import QueryRuntimeException from helpers.s3_tools import get_file_contents, list_s3_objects +from helpers.test_tools import TSV BASE_URL = "http://rest:8181/v1" @@ -70,6 +73,9 @@ DEFAULT_SORT_ORDER = SortOrder(SortField(source_id=2, transform=IdentityTransform())) +AVAILABLE_ENGINES = ["DataLakeCatalog", "Iceberg"] + + def list_namespaces(started_cluster): base_url_local = f"http://localhost:{started_cluster.iceberg_rest_catalog_port}/v1" response = requests.get(f"{base_url_local}/namespaces") @@ -121,7 +127,7 @@ def generate_record(): def create_clickhouse_iceberg_database( - started_cluster, node, name, additional_settings={} + started_cluster, node, name, additional_settings={}, engine='DataLakeCatalog' ): settings = { "catalog_type": "rest", @@ -134,7 +140,7 @@ def create_clickhouse_iceberg_database( node.query( f""" DROP DATABASE IF EXISTS {name}; -CREATE DATABASE {name} ENGINE = DataLakeCatalog('{BASE_URL}', 'minio', '{minio_secret_key}') +CREATE DATABASE {name} ENGINE = {engine}('{BASE_URL}', 'minio', '{minio_secret_key}') SETTINGS {",".join((k+"="+repr(v) for k, v in settings.items()))} """, settings={ @@ -205,6 +211,7 @@ def started_cluster(): user_configs=[], stay_alive=True, with_iceberg_catalog=True, + with_zookeeper=True, ) logging.info("Starting cluster...") @@ -219,7 +226,8 @@ def started_cluster(): cluster.shutdown() -def test_list_tables(started_cluster): +@pytest.mark.parametrize("engine", AVAILABLE_ENGINES) +def test_list_tables(started_cluster, engine): node = started_cluster.instances["node1"] root_namespace = f"clickhouse_{uuid.uuid4()}" @@ -250,7 +258,7 @@ def test_list_tables(started_cluster): for namespace in [namespace_1, namespace_2]: assert len(catalog.list_tables(namespace)) == 0 - create_clickhouse_iceberg_database(started_cluster, node, CATALOG_NAME) + create_clickhouse_iceberg_database(started_cluster, node, CATALOG_NAME, engine=engine) tables_list = "" for table in namespace_1_tables: @@ -461,7 +469,8 @@ def test_check_database(started_cluster): ) -def test_many_namespaces(started_cluster): +@pytest.mark.parametrize("engine", AVAILABLE_ENGINES) +def test_many_namespaces(started_cluster, engine): node = started_cluster.instances["node1"] root_namespace_1 = f"A_{uuid.uuid4()}" root_namespace_2 = f"B_{uuid.uuid4()}" @@ -482,7 +491,7 @@ def test_many_namespaces(started_cluster): for table in tables: create_table(catalog, namespace, table) - create_clickhouse_iceberg_database(started_cluster, node, CATALOG_NAME) + create_clickhouse_iceberg_database(started_cluster, node, CATALOG_NAME, engine=engine) for namespace in namespaces: for table in tables: @@ -494,7 +503,8 @@ def test_many_namespaces(started_cluster): ) -def test_select(started_cluster): +@pytest.mark.parametrize("engine", AVAILABLE_ENGINES) +def test_select(started_cluster, engine): node = started_cluster.instances["node1"] test_ref = f"test_list_tables_{uuid.uuid4()}" @@ -522,7 +532,7 @@ def test_select(started_cluster): df = pa.Table.from_pylist(data) table.append(df) - create_clickhouse_iceberg_database(started_cluster, node, CATALOG_NAME) + create_clickhouse_iceberg_database(started_cluster, node, CATALOG_NAME, engine=engine) expected = DEFAULT_CREATE_TABLE.format(CATALOG_NAME, namespace, table_name) assert expected == node.query( @@ -549,7 +559,8 @@ def test_select(started_cluster): node.restart_clickhouse() -def test_hide_sensitive_info(started_cluster): +@pytest.mark.parametrize("engine", AVAILABLE_ENGINES) +def test_hide_sensitive_info(started_cluster, engine): node = started_cluster.instances["node1"] test_ref = f"test_hide_sensitive_info_{uuid.uuid4()}" @@ -573,7 +584,7 @@ def check_secret_hidden(secret, additional_settings): node.query(f"DROP DATABASE IF EXISTS {CATALOG_NAME}") try: node.query( - f"""CREATE DATABASE {CATALOG_NAME} ENGINE = DataLakeCatalog('{BASE_URL}', 'minio', '{minio_secret_key}') + f"""CREATE DATABASE {CATALOG_NAME} ENGINE = {engine}('{BASE_URL}', 'minio', '{minio_secret_key}') SETTINGS {",".join((k + "=" + repr(v) for k, v in settings.items()))}""", settings={ "allow_database_iceberg": 1, @@ -677,7 +688,8 @@ def test_no_secrets_in_logs(started_cluster): assert minio_secret_key not in val -def test_tables_with_same_location(started_cluster): +@pytest.mark.parametrize("engine", AVAILABLE_ENGINES) +def test_tables_with_same_location(started_cluster, engine): node = started_cluster.instances["node1"] test_ref = f"test_tables_with_same_location_{uuid.uuid4()}" @@ -708,7 +720,7 @@ def record(key): df = pa.Table.from_pylist(data) table_2.append(df) - create_clickhouse_iceberg_database(started_cluster, node, CATALOG_NAME) + create_clickhouse_iceberg_database(started_cluster, node, CATALOG_NAME, engine=engine) assert 'aaa\naaa\naaa' == node.query(f"SELECT symbol FROM {CATALOG_NAME}.`{namespace}.{table_name}`").strip() assert 'bbb\nbbb\nbbb' == node.query(f"SELECT symbol FROM {CATALOG_NAME}.`{namespace}.{table_name_2}`").strip() @@ -853,6 +865,52 @@ def test_timestamps(started_cluster): assert node.query(f"SHOW CREATE TABLE {CATALOG_NAME}.`{root_namespace}.{table_name}`") == f"CREATE TABLE {CATALOG_NAME}.`{root_namespace}.{table_name}`\\n(\\n `timestamp` Nullable(DateTime64(6)),\\n `timestamptz` Nullable(DateTime64(6, \\'UTC\\'))\\n)\\nENGINE = Iceberg(\\'http://minio1:9001/warehouse-rest/data/\\', \\'minio\\', \\'[HIDDEN]\\')\n" assert node.query(f"SELECT * FROM {CATALOG_NAME}.`{root_namespace}.{table_name}`") == "2024-01-01 12:00:00.000000\t2024-01-01 12:00:00.000000\n" + # Berlin - UTC+1 at winter + # Istanbul - UTC+3 at winter + + # 'UTC' is default value, responce is equal to query above + assert node.query(f""" + SELECT * FROM {CATALOG_NAME}.`{root_namespace}.{table_name}` + SETTINGS iceberg_timezone_for_timestamptz='UTC' + """) == "2024-01-01 12:00:00.000000\t2024-01-01 12:00:00.000000\n" + # Timezone from setting + assert node.query(f""" + SELECT * FROM {CATALOG_NAME}.`{root_namespace}.{table_name}` + SETTINGS iceberg_timezone_for_timestamptz='Europe/Berlin' + """) == "2024-01-01 12:00:00.000000\t2024-01-01 13:00:00.000000\n" + # Empty value means session timezone, by default it is 'UTC' too + assert node.query(f""" + SELECT * FROM {CATALOG_NAME}.`{root_namespace}.{table_name}` + SETTINGS iceberg_timezone_for_timestamptz='' + """) == "2024-01-01 12:00:00.000000\t2024-01-01 12:00:00.000000\n" + # If session timezone is used, `timestamptz` does not changed, 'UTC' by default + assert node.query(f""" + SELECT * FROM {CATALOG_NAME}.`{root_namespace}.{table_name}` + SETTINGS session_timezone='Asia/Istanbul' + """) == "2024-01-01 15:00:00.000000\t2024-01-01 12:00:00.000000\n" + # Setiing `iceberg_timezone_for_timestamptz` does not affect `timestamp` column + assert node.query(f""" + SELECT * FROM {CATALOG_NAME}.`{root_namespace}.{table_name}` + SETTINGS session_timezone='Asia/Istanbul', iceberg_timezone_for_timestamptz='Europe/Berlin' + """) == "2024-01-01 15:00:00.000000\t2024-01-01 13:00:00.000000\n" + # Empty value, used non-default session timezone + assert node.query(f""" + SELECT * FROM {CATALOG_NAME}.`{root_namespace}.{table_name}` + SETTINGS session_timezone='Asia/Istanbul', iceberg_timezone_for_timestamptz='' + """) == "2024-01-01 15:00:00.000000\t2024-01-01 15:00:00.000000\n" + # Invalid timezone + assert "Invalid time zone: Foo/Bar" in node.query_and_get_error(f""" + SELECT * FROM {CATALOG_NAME}.`{root_namespace}.{table_name}` + SETTINGS iceberg_timezone_for_timestamptz='Foo/Bar' + """) + + assert node.query(f"SHOW CREATE TABLE {CATALOG_NAME}.`{root_namespace}.{table_name}` SETTINGS iceberg_timezone_for_timestamptz='UTC'") == f"CREATE TABLE {CATALOG_NAME}.`{root_namespace}.{table_name}`\\n(\\n `timestamp` Nullable(DateTime64(6)),\\n `timestamptz` Nullable(DateTime64(6, \\'UTC\\'))\\n)\\nENGINE = Iceberg(\\'http://minio1:9001/warehouse-rest/data/\\', \\'minio\\', \\'[HIDDEN]\\')\n" + assert node.query(f"SHOW CREATE TABLE {CATALOG_NAME}.`{root_namespace}.{table_name}` SETTINGS iceberg_timezone_for_timestamptz='Europe/Berlin'") == f"CREATE TABLE {CATALOG_NAME}.`{root_namespace}.{table_name}`\\n(\\n `timestamp` Nullable(DateTime64(6)),\\n `timestamptz` Nullable(DateTime64(6, \\'Europe/Berlin\\'))\\n)\\nENGINE = Iceberg(\\'http://minio1:9001/warehouse-rest/data/\\', \\'minio\\', \\'[HIDDEN]\\')\n" + + assert node.query(f"SELECT timezoneOf(timestamptz) FROM {CATALOG_NAME}.`{root_namespace}.{table_name}` LIMIT 1") == "UTC\n" + assert node.query(f"SELECT timezoneOf(timestamptz) FROM {CATALOG_NAME}.`{root_namespace}.{table_name}` LIMIT 1 SETTINGS iceberg_timezone_for_timestamptz='UTC'") == "UTC\n" + assert node.query(f"SELECT timezoneOf(timestamptz) FROM {CATALOG_NAME}.`{root_namespace}.{table_name}` LIMIT 1 SETTINGS iceberg_timezone_for_timestamptz='Europe/Berlin'") == "Europe/Berlin\n" + def test_insert(started_cluster): node = started_cluster.instances["node1"] @@ -2004,6 +2062,224 @@ def test_iceberg_file_progress_callback(started_cluster): ) +def test_namespace_filter(started_cluster): + node = started_cluster.instances["node1"] + + # Use the same table name in all namespaces + table_name = f"table_{uuid.uuid4()}" + table2_name = f"table2_{uuid.uuid4()}" + namespace_prefix = f"namespace_{uuid.uuid4()}_" + + catalog = load_catalog_impl(started_cluster) + + def create_namespace(suffix): + namespace = f"{namespace_prefix}{suffix}" + catalog.create_namespace(namespace) + create_table(catalog, namespace, table_name, DEFAULT_SCHEMA, PartitionSpec(), DEFAULT_SORT_ORDER) + + create_namespace("alpha"); + create_namespace("alpha.a1"); + create_namespace("alpha.a2"); + create_namespace("bravo"); + create_namespace("bravo.b1"); + create_namespace("charlie"); + create_namespace("charlie.c1"); + create_namespace("delta"); + create_namespace("delta.d1"); + create_namespace("delta.d2"); + create_namespace("echo"); + create_namespace("echo.e1"); + + create_clickhouse_iceberg_database(started_cluster, node, CATALOG_NAME, + additional_settings={ + "namespaces": f"{namespace_prefix}alpha,{namespace_prefix}alpha.a1,{namespace_prefix}bravo,{namespace_prefix}bravo.*,{namespace_prefix}charlie,{namespace_prefix}delta.d1,{namespace_prefix}echo.*" + }) + + assert node.query(f"SELECT name FROM system.tables WHERE database='{CATALOG_NAME}' ORDER BY name", settings={"show_data_lake_catalogs_in_system_tables": 1}) == TSV( + [ + [f"{namespace_prefix}alpha.a1.{table_name}"], + [f"{namespace_prefix}alpha.{table_name}"], + [f"{namespace_prefix}bravo.b1.{table_name}"], + [f"{namespace_prefix}bravo.{table_name}"], + [f"{namespace_prefix}charlie.{table_name}"], + [f"{namespace_prefix}delta.d1.{table_name}"], + [f"{namespace_prefix}echo.e1.{table_name}"], + ]) + + assert node.query(f"SELECT count() FROM {CATALOG_NAME}.`{namespace_prefix}alpha.{table_name}`") == "0\n" + assert node.query(f"SELECT count() FROM {CATALOG_NAME}.`{namespace_prefix}alpha.a1.{table_name}`") == "0\n" + assert "is filtered by `namespaces` database parameter." in node.query_and_get_error(f"SELECT count() FROM {CATALOG_NAME}.`{namespace_prefix}alpha.a2.{table_name}`") + assert node.query(f"SELECT count() FROM {CATALOG_NAME}.`{namespace_prefix}bravo.{table_name}`") == "0\n" + assert node.query(f"SELECT count() FROM {CATALOG_NAME}.`{namespace_prefix}bravo.b1.{table_name}`") == "0\n" + assert node.query(f"SELECT count() FROM {CATALOG_NAME}.`{namespace_prefix}charlie.{table_name}`") == "0\n" + assert "is filtered by `namespaces` database parameter." in node.query_and_get_error(f"SELECT count() FROM {CATALOG_NAME}.`{namespace_prefix}charlie.c1.{table_name}`") + assert "is filtered by `namespaces` database parameter." in node.query_and_get_error(f"SELECT count() FROM {CATALOG_NAME}.`{namespace_prefix}delta.{table_name}`") + assert node.query(f"SELECT count() FROM {CATALOG_NAME}.`{namespace_prefix}delta.d1.{table_name}`") == "0\n" + assert "is filtered by `namespaces` database parameter." in node.query_and_get_error(f"SELECT count() FROM {CATALOG_NAME}.`{namespace_prefix}delta.d2.{table_name}`") + assert "is filtered by `namespaces` database parameter." in node.query_and_get_error(f"SELECT count() FROM {CATALOG_NAME}.`{namespace_prefix}echo.{table_name}`") + assert node.query(f"SELECT count() FROM {CATALOG_NAME}.`{namespace_prefix}echo.e1.{table_name}`") == "0\n" + + node.query(f"CREATE TABLE {CATALOG_NAME}.`{namespace_prefix}alpha.{table2_name}` (x String) ENGINE = IcebergS3('http://minio1:9001/warehouse-rest/{namespace_prefix}alpha/{table2_name}/', '{minio_access_key}', '{minio_secret_key}')", + settings={ + "allow_database_iceberg": 1, + "write_full_path_in_iceberg_metadata": 1, + }, + ) + node.query(f"CREATE TABLE {CATALOG_NAME}.`{namespace_prefix}alpha.a1.{table2_name}` (x String) ENGINE = IcebergS3('http://minio1:9001/warehouse-rest/{namespace_prefix}alpha/a1/{table2_name}/', '{minio_access_key}', '{minio_secret_key}')", + settings={ + "allow_database_iceberg": 1, + "write_full_path_in_iceberg_metadata": 1, + }, + ) + assert "is filtered by `namespaces` database parameter." in node.query_and_get_error(f"CREATE TABLE {CATALOG_NAME}.`{namespace_prefix}alpha.a2.{table2_name}` (x String) ENGINE = IcebergS3('http://minio1:9001/warehouse-rest/{namespace_prefix}alpha/a2/{table2_name}/', '{minio_access_key}', '{minio_secret_key}')") + + node.query(f"DROP TABLE {CATALOG_NAME}.`{namespace_prefix}alpha.{table_name}`") + node.query(f"DROP TABLE {CATALOG_NAME}.`{namespace_prefix}alpha.a1.{table_name}`") + assert "is filtered by `namespaces` database parameter." in node.query_and_get_error(f"DROP TABLE {CATALOG_NAME}.`{namespace_prefix}alpha.a2.{table_name}`") + + +# TODO - turn on after merge alternative syntax +@pytest.mark.parametrize("join_mode", ["local", "global"]) +def _test_cluster_joins(started_cluster, join_mode): + node = started_cluster.instances["node1"] + + test_ref = f"test_join_tables_{uuid.uuid4()}" + table_name = f"{test_ref}_table" + table_name_2 = f"{test_ref}_table_2" + table_name_local = f"{test_ref}_table_local" + + root_namespace = f"{test_ref}_namespace" + + catalog = load_catalog_impl(started_cluster) + catalog.create_namespace(root_namespace) + + schema = Schema( + NestedField( + field_id=1, + name="tag", + field_type=LongType(), + required=False + ), + NestedField( + field_id=2, + name="name", + field_type=StringType(), + required=False, + ), + ) + table = create_table(catalog, root_namespace, table_name, schema, + partition_spec=UNPARTITIONED_PARTITION_SPEC, sort_order=UNSORTED_SORT_ORDER) + data = [{"tag": 1, "name": "John"}, {"tag": 2, "name": "Jack"}] + df = pa.Table.from_pylist(data) + table.append(df) + + schema2 = Schema( + NestedField( + field_id=1, + name="id", + field_type=LongType(), + required=False + ), + NestedField( + field_id=2, + name="second_name", + field_type=StringType(), + required=False, + ), + ) + table2 = create_table(catalog, root_namespace, table_name_2, schema2, + partition_spec=UNPARTITIONED_PARTITION_SPEC, sort_order=UNSORTED_SORT_ORDER) + data = [{"id": 1, "second_name": "Dow"}, {"id": 2, "second_name": "Sparrow"}] + df = pa.Table.from_pylist(data) + table2.append(df) + + node.query(f"CREATE TABLE `{table_name_local}` (id Int64, second_name String) ENGINE = Memory()") + node.query(f"INSERT INTO `{table_name_local}` VALUES (1, 'Silver'), (2, 'Black')") + + create_clickhouse_iceberg_database(started_cluster, node, CATALOG_NAME) + + res = node.query( + f""" + SELECT t1.name,t2.second_name + FROM {CATALOG_NAME}.`{root_namespace}.{table_name}` AS t1 + JOIN {CATALOG_NAME}.`{root_namespace}.{table_name_2}` AS t2 + ON t1.tag=t2.id + WHERE t1.tag < 10 AND t2.id < 20 + ORDER BY ALL + SETTINGS + object_storage_cluster='cluster_simple', + object_storage_cluster_join_mode='{join_mode}' + """ + ) + + assert res == "Jack\tSparrow\nJohn\tDow\n" + + res = node.query( + f""" + SELECT name + FROM {CATALOG_NAME}.`{root_namespace}.{table_name}` + WHERE tag in ( + SELECT id + FROM {CATALOG_NAME}.`{root_namespace}.{table_name_2}` + ) + ORDER BY ALL + SETTINGS + object_storage_cluster='cluster_simple', + object_storage_cluster_join_mode='{join_mode}' + """ + ) + + assert res == "Jack\nJohn\n" + + res = node.query( + f""" + SELECT t1.name,t2.second_name + FROM {CATALOG_NAME}.`{root_namespace}.{table_name}` AS t1 + JOIN `{table_name_local}` AS t2 + ON t1.tag=t2.id + WHERE t1.tag < 10 AND t2.id < 20 + ORDER BY ALL + SETTINGS + object_storage_cluster='cluster_simple', + object_storage_cluster_join_mode='{join_mode}' + """ + ) + + assert res == "Jack\tBlack\nJohn\tSilver\n" + + res = node.query( + f""" + SELECT name + FROM {CATALOG_NAME}.`{root_namespace}.{table_name}` + WHERE tag in ( + SELECT id + FROM `{table_name_local}` + ) + ORDER BY ALL + SETTINGS + object_storage_cluster='cluster_simple', + object_storage_cluster_join_mode='{join_mode}' + """ + ) + + assert res == "Jack\nJohn\n" + + res = node.query( + f""" + SELECT t1.name,t2.second_name + FROM {CATALOG_NAME}.`{root_namespace}.{table_name}` AS t1 + CROSS JOIN `{table_name_local}` AS t2 + WHERE t1.tag < 10 AND t2.id < 20 + ORDER BY ALL + SETTINGS + object_storage_cluster='cluster_simple', + object_storage_cluster_join_mode='{join_mode}' + """ + ) + + assert res == "Jack\tBlack\nJack\tSilver\nJohn\tBlack\nJohn\tSilver\n" + + def test_alter_database_settings_not_supported(started_cluster): node = started_cluster.instances["node1"] diff --git a/tests/integration/test_database_iceberg/test_partition_timezone.py b/tests/integration/test_database_iceberg/test_partition_timezone.py new file mode 100644 index 000000000000..e7743aefabbe --- /dev/null +++ b/tests/integration/test_database_iceberg/test_partition_timezone.py @@ -0,0 +1,186 @@ +import glob +import json +import logging +import os +import random +import time +import uuid +from datetime import datetime, timedelta + +import pyarrow as pa +import pytest +import requests +import urllib3 +import pytz +from minio import Minio +from pyiceberg.catalog import load_catalog +from pyiceberg.partitioning import PartitionField, PartitionSpec, UNPARTITIONED_PARTITION_SPEC +from pyiceberg.schema import Schema +from pyiceberg.table.sorting import SortField, SortOrder +from pyiceberg.transforms import DayTransform, IdentityTransform +from pyiceberg.types import ( + DoubleType, + LongType, + FloatType, + NestedField, + StringType, + StructType, + TimestampType, + TimestamptzType +) +from pyiceberg.table.sorting import UNSORTED_SORT_ORDER + +from helpers.cluster import ClickHouseCluster, ClickHouseInstance, is_arm +from helpers.config_cluster import minio_secret_key, minio_access_key +from helpers.s3_tools import get_file_contents, list_s3_objects, prepare_s3_bucket +from helpers.test_tools import TSV, csv_compare +from helpers.config_cluster import minio_secret_key + +BASE_URL = "http://rest:8181/v1" +# The REST catalog and MinIO containers are shared with the rest of the suite and their +# host ports are allocated dynamically (`iceberg_rest_catalog_port` / `minio_port`), so +# host-side URLs have to be derived from the cluster object. +WAREHOUSE_ENDPOINT = "http://minio1:9001/warehouse-rest" + + +def get_base_url_local_raw(cluster): + return f"http://localhost:{cluster.iceberg_rest_catalog_port}" + +CATALOG_NAME = "demo" + +DEFAULT_PARTITION_SPEC = PartitionSpec( + PartitionField( + source_id=1, field_id=1000, transform=DayTransform(), name="datetime_day" + ) +) +DEFAULT_SORT_ORDER = SortOrder(SortField(source_id=1, transform=DayTransform())) +DEFAULT_SCHEMA = Schema( + NestedField(field_id=1, name="datetime", field_type=TimestampType(), required=False), + NestedField(field_id=2, name="value", field_type=LongType(), required=False), +) + + +@pytest.fixture(scope="module") +def started_cluster(): + try: + cluster = ClickHouseCluster(__file__) + cluster.add_instance( + "node1", + main_configs=["configs/timezone.xml", "configs/cluster.xml"], + user_configs=["configs/iceberg_partition_timezone.xml"], + stay_alive=True, + with_iceberg_catalog=True, + with_zookeeper=True, + ) + + logging.info("Starting cluster...") + cluster.start() + + # TODO: properly wait for container + time.sleep(10) + + yield cluster + + finally: + cluster.shutdown() + + +def load_catalog_impl(started_cluster): + return load_catalog( + CATALOG_NAME, + **{ + "uri": get_base_url_local_raw(started_cluster), + "type": "rest", + "s3.endpoint": f"http://{started_cluster.minio_ip}:{started_cluster.minio_port}", + "s3.access-key-id": minio_access_key, + "s3.secret-access-key": minio_secret_key, + }, + ) + + +def create_table( + catalog, + namespace, + table, + schema=DEFAULT_SCHEMA, + partition_spec=DEFAULT_PARTITION_SPEC, + sort_order=DEFAULT_SORT_ORDER, +): + return catalog.create_table( + identifier=f"{namespace}.{table}", + schema=schema, + location=f"s3://warehouse-rest/data", + partition_spec=partition_spec, + sort_order=sort_order, + ) + + +def create_clickhouse_iceberg_database( + node, name, additional_settings={}, engine='DataLakeCatalog' +): + settings = { + "catalog_type": "rest", + "warehouse": "demo", + "storage_endpoint": WAREHOUSE_ENDPOINT, + } + + settings.update(additional_settings) + + node.query( + f""" +DROP DATABASE IF EXISTS {name}; +SET allow_database_iceberg=true; +SET write_full_path_in_iceberg_metadata=1; +CREATE DATABASE {name} ENGINE = {engine}('{BASE_URL}', 'minio', '{minio_secret_key}') +SETTINGS {",".join((k+"="+repr(v) for k, v in settings.items()))} + """ + ) + show_result = node.query(f"SHOW DATABASE {name}") + assert minio_secret_key not in show_result + assert "HIDDEN" in show_result + + +def test_partition_timezone(started_cluster): + catalog = load_catalog_impl(started_cluster) + namespace = f"timezone_ns_{uuid.uuid4()}" + table_name = f"tz_table__{uuid.uuid4()}" + catalog.create_namespace(namespace) + table = create_table( + catalog, + namespace, + table_name, + ) + + # catalog accept data in UTC + data = [{"datetime": datetime(2024, 1, 1, 20, 0), "value": 1}, # partition 20240101 + {"datetime": datetime(2024, 1, 1, 23, 0), "value": 2}, # partition 20240101 + {"datetime": datetime(2024, 1, 2, 2, 0), "value": 3}] # partition 20240102 + df = pa.Table.from_pylist(data) + table.append(df) + + node = started_cluster.instances["node1"] + create_clickhouse_iceberg_database(node, CATALOG_NAME) + + # server timezone is Asia/Istanbul (UTC+3) + assert node.query(f""" + SELECT datetime, value + FROM {CATALOG_NAME}.`{namespace}.{table_name}` + ORDER BY datetime + """, timeout=10) == TSV( + [ + ["2024-01-01 23:00:00.000000", 1], + ["2024-01-02 02:00:00.000000", 2], + ["2024-01-02 05:00:00.000000", 3], + ]) + + # partitioning works correctly + assert node.query(f""" + SELECT datetime, value + FROM {CATALOG_NAME}.`{namespace}.{table_name}` + WHERE datetime >= '2024-01-02 00:00:00' + ORDER BY datetime + """, timeout=10) == TSV( + [ + ["2024-01-02 02:00:00.000000", 2], + ["2024-01-02 05:00:00.000000", 3], + ]) diff --git a/tests/integration/test_export_merge_tree_part_to_iceberg/configs/config.d/metadata_log.xml b/tests/integration/test_export_merge_tree_part_to_iceberg/configs/config.d/metadata_log.xml new file mode 100644 index 000000000000..c1fece21745c --- /dev/null +++ b/tests/integration/test_export_merge_tree_part_to_iceberg/configs/config.d/metadata_log.xml @@ -0,0 +1,7 @@ + + + system + iceberg_metadata_log
+ 10 + + diff --git a/tests/integration/test_export_merge_tree_part_to_iceberg/test.py b/tests/integration/test_export_merge_tree_part_to_iceberg/test.py new file mode 100644 index 000000000000..e7cd8809ef8b --- /dev/null +++ b/tests/integration/test_export_merge_tree_part_to_iceberg/test.py @@ -0,0 +1,1187 @@ +""" +Integration tests for EXPORT PART to an IcebergS3 destination. + +These tests cover the data-movement path from a plain MergeTree table to an +IcebergS3 table using the single-part export operation: + + ALTER TABLE EXPORT PART '' TO TABLE + +Coverage: + test_export_part_basic_to_iceberg – simple (id, year) schema; data + part_log checks + test_export_part_all_iceberg_types – schema covering all major Iceberg data types + test_export_multiple_parts_to_iceberg – two parts from different partitions land together + test_export_part_with_year_transform_partition – toYearNumSinceEpoch() partition expression + test_export_part_with_bucket_partition – icebergBucket(N, col) partition expression + test_export_part_partition_key_mismatch_is_rejected – mismatched partition spec rejected synchronously + test_export_part_multi_column_partition_key_success – composite (a, b, c) partition key round-trips + test_export_part_partition_key_mismatch_variants_are_rejected (parametrized) – partition key column reordering, + cardinality mismatches, and transform-expression reordering between src/dst are all rejected synchronously +""" + +import logging +import time +from typing import NamedTuple + +import pytest + +from helpers.cluster import ClickHouseCluster +from helpers.export_partition_helpers import ( + first_partition_id, + make_iceberg_s3, + make_mt, + unique_suffix, +) +from helpers.iceberg_export_stats import ( + assert_exported_stats, + fetch_manifest_entries, +) + + +# --------------------------------------------------------------------------- +# Cluster fixture +# --------------------------------------------------------------------------- + + +@pytest.fixture(scope="module") +def cluster(): + try: + cluster = ClickHouseCluster(__file__) + cluster.add_instance( + "node1", + main_configs=["configs/config.d/metadata_log.xml"], + with_minio=True, + ) + logging.info("Starting cluster...") + cluster.start() + yield cluster + finally: + cluster.shutdown() + + +# --------------------------------------------------------------------------- +# Helpers +# --------------------------------------------------------------------------- + + +def get_part(node, table: str, partition_id: str) -> str: + """Return the name of the first active part of *table* in *partition_id*.""" + return node.query( + f"SELECT name FROM system.parts " + f"WHERE database = currentDatabase() AND table = '{table}' " + f"AND partition_id = '{partition_id}' AND active " + f"ORDER BY name LIMIT 1" + ).strip() + + +def export_part(node, table: str, part: str, dest: str, extra_settings: str = "") -> None: + settings = ( + "allow_experimental_export_merge_tree_part = 1, " + "allow_experimental_insert_into_iceberg = 1" + ) + if extra_settings: + settings += ", " + extra_settings + node.query( + f"ALTER TABLE {table} EXPORT PART '{part}' TO TABLE {dest} SETTINGS {settings}" + ) + + +def wait_for_export_part( + node, + table: str, + part: str, + timeout: int = 60, + poll_interval: float = 0.5, +) -> None: + """Poll system.part_log until an ExportPart event appears for *part*.""" + deadline = time.time() + timeout + while time.time() < deadline: + node.query("SYSTEM FLUSH LOGS") + count = node.query( + f"SELECT count() FROM system.part_log " + f"WHERE event_type = 'ExportPart' " + f"AND database = currentDatabase() " + f"AND table = '{table}' " + f"AND part_name = '{part}'" + ).strip() + if count != "0": + return + time.sleep(poll_interval) + raise TimeoutError( + f"ExportPart event for part {part!r} in table {table!r} " + f"did not appear in system.part_log within {timeout}s" + ) + + +def assert_part_log(node, table: str, part: str) -> None: + """Assert that system.part_log contains at least one ExportPart entry.""" + log_count = int( + node.query( + f"SELECT count() FROM system.part_log " + f"WHERE event_type = 'ExportPart' " + f"AND database = currentDatabase() " + f"AND table = '{table}' " + f"AND part_name = '{part}'" + ).strip() + ) + assert log_count >= 1, ( + f"Expected at least one ExportPart entry in system.part_log " + f"for part {part!r} in table {table!r}, found {log_count}" + ) + + +def wait_for_failed_export_part( + node, + table: str, + part: str, + timeout: int = 60, + poll_interval: float = 0.5, +) -> str: + """Poll system.part_log until a failed ExportPart event appears for *part*. + + Returns the exception text recorded on the log entry, which lets callers + assert on the specific runtime error that propagated from the export worker. + """ + deadline = time.time() + timeout + last_seen = "" + while time.time() < deadline: + node.query("SYSTEM FLUSH LOGS") + row = node.query( + f"SELECT error, exception FROM system.part_log " + f"WHERE event_type = 'ExportPart' " + f"AND database = currentDatabase() " + f"AND table = '{table}' " + f"AND part_name = '{part}' " + f"AND error != 0 " + f"ORDER BY event_time DESC LIMIT 1" + ).strip() + if row: + _error, exception = row.split("\t", 1) + return exception + last_seen = row + time.sleep(poll_interval) + raise TimeoutError( + f"Failed ExportPart event for part {part!r} in table {table!r} " + f"did not appear in system.part_log within {timeout}s (last row: {last_seen!r})" + ) + + +# --------------------------------------------------------------------------- +# Tests +# --------------------------------------------------------------------------- + + +def test_export_part_basic_to_iceberg(cluster): + """ + Basic happy path: export a single MergeTree part to an IcebergS3 table and + verify the row count, the content, and the system.part_log ExportPart entry. + """ + node = cluster.instances["node1"] + sfx = unique_suffix() + mt = f"mt_basic_{sfx}" + iceberg = f"iceberg_basic_{sfx}" + + make_mt(node, mt, "id Int32, year Int32", "year") + make_iceberg_s3(node, iceberg, "id Int32, year Int32", "year") + + node.query(f"INSERT INTO {mt} VALUES (1, 2020), (2, 2020), (3, 2020), (4, 2021)") + + part_2020 = get_part(node, mt, "2020") + export_part(node, mt, part_2020, iceberg) + wait_for_export_part(node, mt, part_2020) + + count = int(node.query(f"SELECT count() FROM {iceberg}").strip()) + assert count == 3, f"Expected 3 rows in Iceberg table after export, got {count}" + + result = node.query(f"SELECT id, year FROM {iceberg} ORDER BY id").strip() + assert result == "1\t2020\n2\t2020\n3\t2020", f"Unexpected exported data:\n{result}" + + assert_part_log(node, mt, part_2020) + + node.query(f"DROP TABLE IF EXISTS {mt} SYNC") + node.query(f"DROP TABLE IF EXISTS {iceberg}") + + +def test_export_part_all_iceberg_types(cluster): + """ + Export a part whose schema covers every ClickHouse type that getIcebergType() + in Utils.cpp maps to an Iceberg primitive (see the switch statement): + + Iceberg type ClickHouse column + ------------- -------------------------- + int id Int32 + long big_val Int64 + float f32 Float32 + double f64 Float64 + date event_dt Date + timestamp ts DateTime64(6) (DateTime / DateTime64 both → "timestamp") + string name String + uuid uid_val UUID + + Types not in the switch (Bool/UInt8, FixedString, Decimal) are intentionally + excluded — they throw BAD_ARGUMENTS from getIcebergType(). + + Verifies that every column round-trips correctly through the Iceberg layer + and that system.part_log records the ExportPart event. + """ + node = cluster.instances["node1"] + sfx = unique_suffix() + mt = f"mt_types_{sfx}" + iceberg = f"iceberg_types_{sfx}" + + columns = ( + "id Int32, " + "big_val Int64, " + "f32 Float32, " + "f64 Float64, " + "event_dt Date, " + "ts DateTime64(6), " + "name String, " + "uid_val UUID, " + "year Int32" + ) + + make_mt(node, mt, columns, "year", order_by="id") + make_iceberg_s3(node, iceberg, columns, "year") + + node.query( + f""" + INSERT INTO {mt} (id, big_val, f32, f64, event_dt, ts, name, uid_val, year) + VALUES ( + 1, + 9999999999999, + 3.14, + 2.718281828459045, + '2024-01-15', + '2024-01-15 12:30:45.123456', + 'hello iceberg', + '550e8400-e29b-41d4-a716-446655440000', + 2024 + ) + """ + ) + + part = get_part(node, mt, "2024") + export_part(node, mt, part, iceberg) + wait_for_export_part(node, mt, part) + + count = int(node.query(f"SELECT count() FROM {iceberg}").strip()) + assert count == 1, f"Expected 1 row in Iceberg table, got {count}" + + row = node.query( + f"SELECT id, big_val, name, year FROM {iceberg}" + ).strip() + assert "1" in row, f"id column missing/wrong: {row}" + assert "9999999999999" in row, f"big_val column missing/wrong: {row}" + assert "hello iceberg" in row, f"name column missing/wrong: {row}" + assert "2024" in row, f"year column missing/wrong: {row}" + + # Verify date round-trip + date_result = node.query(f"SELECT toString(event_dt) FROM {iceberg}").strip() + assert date_result == "2024-01-15", f"Date round-trip failed: {date_result!r}" + + # Verify timestamp round-trip (date component is sufficient; exact time format varies) + ts_result = node.query(f"SELECT ts FROM {iceberg}").strip() + assert "2024-01-15" in ts_result, f"Timestamp date component missing: {ts_result!r}" + + # Verify UUID round-trip + uid_result = node.query(f"SELECT toString(uid_val) FROM {iceberg}").strip() + assert uid_result == "550e8400-e29b-41d4-a716-446655440000", ( + f"UUID round-trip failed: {uid_result!r}" + ) + + assert_part_log(node, mt, part) + + node.query(f"DROP TABLE IF EXISTS {mt} SYNC") + node.query(f"DROP TABLE IF EXISTS {iceberg}") + + +def test_export_multiple_parts_to_iceberg(cluster): + """ + Export parts from two different partitions to the same Iceberg table and + verify that both land correctly without overwriting each other. + system.part_log must contain one ExportPart entry per exported part. + """ + node = cluster.instances["node1"] + sfx = unique_suffix() + mt = f"mt_multi_{sfx}" + iceberg = f"iceberg_multi_{sfx}" + + make_mt(node, mt, "id Int32, year Int32", "year") + make_iceberg_s3(node, iceberg, "id Int32, year Int32", "year") + + # Each INSERT creates a separate part per partition + node.query(f"INSERT INTO {mt} VALUES (1, 2020), (2, 2020)") + node.query(f"INSERT INTO {mt} VALUES (10, 2021), (11, 2021), (12, 2021)") + + part_2020 = get_part(node, mt, "2020") + part_2021 = get_part(node, mt, "2021") + + export_part(node, mt, part_2020, iceberg) + export_part(node, mt, part_2021, iceberg) + + wait_for_export_part(node, mt, part_2020) + wait_for_export_part(node, mt, part_2021) + + total = int(node.query(f"SELECT count() FROM {iceberg}").strip()) + assert total == 5, f"Expected 5 rows total (2+3), got {total}" + + count_2020 = int(node.query(f"SELECT count() FROM {iceberg} WHERE year = 2020").strip()) + count_2021 = int(node.query(f"SELECT count() FROM {iceberg} WHERE year = 2021").strip()) + assert count_2020 == 2, f"Expected 2 rows for year=2020, got {count_2020}" + assert count_2021 == 3, f"Expected 3 rows for year=2021, got {count_2021}" + + result_2020 = node.query( + f"SELECT id FROM {iceberg} WHERE year = 2020 ORDER BY id" + ).strip() + assert result_2020 == "1\n2", f"Unexpected 2020 rows: {result_2020}" + + result_2021 = node.query( + f"SELECT id FROM {iceberg} WHERE year = 2021 ORDER BY id" + ).strip() + assert result_2021 == "10\n11\n12", f"Unexpected 2021 rows: {result_2021}" + + assert_part_log(node, mt, part_2020) + assert_part_log(node, mt, part_2021) + + node.query(f"DROP TABLE IF EXISTS {mt} SYNC") + node.query(f"DROP TABLE IF EXISTS {iceberg}") + + +def test_export_part_with_year_transform_partition(cluster): + """ + Export a part from a MergeTree table partitioned by toYearNumSinceEpoch(event_date) + to an Iceberg table with the matching year-transform spec. + + Verifies that the Iceberg year-transform partition expression is accepted + and that all rows survive the round-trip intact. + """ + node = cluster.instances["node1"] + sfx = unique_suffix() + mt = f"mt_year_tf_{sfx}" + iceberg = f"iceberg_year_tf_{sfx}" + + cols = "id Int64, event_date Date" + partition_by = "toYearNumSinceEpoch(event_date)" + + make_mt(node, mt, cols, partition_by, order_by="id") + make_iceberg_s3(node, iceberg, cols, partition_by) + + node.query( + f"INSERT INTO {mt} VALUES " + f"(1, '2023-03-15'), (2, '2023-11-01'), (3, '2023-06-30')" + ) + + pid = first_partition_id(node, mt) + part = get_part(node, mt, pid) + + export_part(node, mt, part, iceberg) + wait_for_export_part(node, mt, part) + + count = int(node.query(f"SELECT count() FROM {iceberg}").strip()) + assert count == 3, f"Expected 3 rows, got {count}" + + result = node.query( + f"SELECT id, toString(event_date) FROM {iceberg} ORDER BY id" + ).strip() + assert "1\t2023-03-15" in result, f"Row 1 missing or incorrect:\n{result}" + assert "2\t2023-11-01" in result, f"Row 2 missing or incorrect:\n{result}" + assert "3\t2023-06-30" in result, f"Row 3 missing or incorrect:\n{result}" + + assert_part_log(node, mt, part) + + node.query(f"DROP TABLE IF EXISTS {mt} SYNC") + node.query(f"DROP TABLE IF EXISTS {iceberg}") + + +def test_export_part_partition_column_lossless_widening(cluster): + """A lossless widening of a partition column (year Int32 -> Int64) round-trips.""" + node = cluster.instances["node1"] + sfx = unique_suffix() + mt = f"mt_pcol_widening_{sfx}" + iceberg = f"iceberg_pcol_widening_{sfx}" + + make_mt(node, mt, "id Int32, year Int32", "year") + make_iceberg_s3(node, iceberg, "id Int32, year Int64", "year") + + node.query(f"INSERT INTO {mt} VALUES (1, 2020), (2, 2020), (3, 2020)") + + part_2020 = get_part(node, mt, "2020") + export_part(node, mt, part_2020, iceberg) + wait_for_export_part(node, mt, part_2020) + + count = int(node.query(f"SELECT count() FROM {iceberg}").strip()) + assert count == 3, f"Expected 3 rows in Iceberg table after export, got {count}" + + result = node.query( + f"SELECT id, toTypeName(year), year FROM {iceberg} ORDER BY id" + ).strip() + assert result == "1\tInt64\t2020\n2\tInt64\t2020\n3\tInt64\t2020", ( + f"Unexpected widened partition-column data:\n{result}" + ) + + assert_part_log(node, mt, part_2020) + + node.query(f"DROP TABLE IF EXISTS {mt} SYNC") + node.query(f"DROP TABLE IF EXISTS {iceberg}") + + +def test_export_part_partition_key_mismatch_is_rejected(cluster): + """ + EXPORT PART must synchronously reject (BAD_ARGUMENTS) when the source + MergeTree partition key does not match the destination Iceberg partition + spec. Export does not repartition data, so the two specs must agree on + every field (same source column by Iceberg field-id and same transform, + in the same order). + + Failing case: MergeTree PARTITION BY year, Iceberg PARTITION BY id. + The part must NOT land in the Iceberg table. + """ + node = cluster.instances["node1"] + sfx = unique_suffix() + mt = f"mt_pkey_mismatch_{sfx}" + iceberg = f"iceberg_pkey_mismatch_{sfx}" + + make_mt(node, mt, "id Int32, year Int32", "year") + make_iceberg_s3(node, iceberg, "id Int32, year Int32", "id") + + node.query(f"INSERT INTO {mt} VALUES (1, 2020), (2, 2020), (3, 2020)") + + part_2020 = get_part(node, mt, "2020") + + error = node.query_and_get_error( + f"ALTER TABLE {mt} EXPORT PART '{part_2020}' TO TABLE {iceberg} " + f"SETTINGS allow_experimental_export_merge_tree_part = 1, " + f"allow_experimental_insert_into_iceberg = 1" + ) + assert "BAD_ARGUMENTS" in error, ( + f"Expected BAD_ARGUMENTS for partition key mismatch, got: {error!r}" + ) + + count = int(node.query(f"SELECT count() FROM {iceberg}").strip()) + assert count == 0, ( + f"Expected 0 rows in Iceberg table after rejected export, got {count}" + ) + + node.query(f"DROP TABLE IF EXISTS {mt} SYNC") + node.query(f"DROP TABLE IF EXISTS {iceberg}") + + +class RejectedPartExportCase(NamedTuple): + src_columns: str + src_partition_by: str + dst_columns: str + dst_partition_by: str + insert_values: str + error_substrings: tuple = () + + +REJECTED_PART_EXPORT_CASES = [ + pytest.param( + RejectedPartExportCase( + src_columns="a Int32, b Int32", + src_partition_by="a", + dst_columns="b Int32, a Int32", + dst_partition_by="a", + insert_values="(1, 1), (1, 2)", + error_substrings=("partition key column",), + ), + id="same_partition_key_different_column_order_single_column", + ), + pytest.param( + RejectedPartExportCase( + src_columns="a Int32, b Int32, c Int32, val String", + src_partition_by="(a, b, c)", + dst_columns="c Int32, b Int32, a Int32, val String", + dst_partition_by="(a, b, c)", + insert_values="(1, 1, 1, 'x'), (1, 1, 1, 'y')", + error_substrings=("partition key column",), + ), + id="same_partition_key_different_column_order_multi_column", + ), + pytest.param( + RejectedPartExportCase( + src_columns="a Int32, b Int32, c Int32, val String", + src_partition_by="(a, b)", + dst_columns="a Int32, b Int32, c Int32, val String", + dst_partition_by="(a, b, c)", + insert_values="(1, 2, 3, 'x')", + error_substrings=("column 'c', which is not part of the source MergeTree partition key",), + ), + id="multi_column_partition_key_more_in_destination", + ), + pytest.param( + RejectedPartExportCase( + src_columns="other_id Int64, user_id Int64", + src_partition_by="icebergBucket(8, user_id)", + dst_columns="user_id Int64, other_id Int64", + dst_partition_by="icebergBucket(8, user_id)", + insert_values="(1, 42)", + error_substrings=("partition key column",), + ), + id="transform_partition_key_different_column_order", + ), +] + + +@pytest.mark.parametrize("case", REJECTED_PART_EXPORT_CASES) +def test_export_part_partition_key_mismatch_variants_are_rejected(cluster, case): + node = cluster.instances["node1"] + sfx = unique_suffix() + mt = f"mt_rejected_{sfx}" + iceberg = f"iceberg_rejected_{sfx}" + + make_mt(node, mt, case.src_columns, case.src_partition_by) + make_iceberg_s3(node, iceberg, case.dst_columns, case.dst_partition_by) + + node.query(f"INSERT INTO {mt} VALUES {case.insert_values}") + + pid = first_partition_id(node, mt) + part = get_part(node, mt, pid) + + error = node.query_and_get_error( + f"ALTER TABLE {mt} EXPORT PART '{part}' TO TABLE {iceberg} " + f"SETTINGS allow_experimental_export_merge_tree_part = 1, " + f"allow_experimental_insert_into_iceberg = 1" + ) + assert "BAD_ARGUMENTS" in error, f"Expected BAD_ARGUMENTS, got: {error!r}" + for substring in case.error_substrings: + assert substring in error, f"Expected {substring!r} in error, got: {error!r}" + + count = int(node.query(f"SELECT count() FROM {iceberg}").strip()) + assert count == 0, f"Expected 0 rows in Iceberg table after rejected export, got {count}" + + node.query(f"DROP TABLE IF EXISTS {mt} SYNC") + node.query(f"DROP TABLE IF EXISTS {iceberg}") + + +@pytest.mark.parametrize("dst_partition_by", ["(a, b, c)", "(c, b, a)", "(a, b)"]) +def test_export_part_multi_column_partition_key_success(cluster, dst_partition_by): + """The source key pins a, b and c, so any destination spec over those columns holds the whole + part in one partition, whatever order or subset of them it lists.""" + node = cluster.instances["node1"] + sfx = unique_suffix() + mt = f"mt_multi_pkey_ok_{sfx}" + iceberg = f"iceberg_multi_pkey_ok_{sfx}" + + cols = "a Int32, b Int32, c Int32, val String" + make_mt(node, mt, cols, "(a, b, c)") + make_iceberg_s3(node, iceberg, cols, dst_partition_by) + + node.query(f"INSERT INTO {mt} VALUES (1, 2, 3, 'x'), (1, 2, 3, 'y')") + + pid = first_partition_id(node, mt) + part = get_part(node, mt, pid) + export_part(node, mt, part, iceberg) + wait_for_export_part(node, mt, part) + + count = int(node.query(f"SELECT count() FROM {iceberg}").strip()) + assert count == 2, f"Expected 2 rows in Iceberg table after export, got {count}" + + result = node.query(f"SELECT a, b, c, val FROM {iceberg} ORDER BY val").strip() + assert result == "1\t2\t3\tx\n1\t2\t3\ty", f"Unexpected exported data:\n{result}" + + assert_part_log(node, mt, part) + + node.query(f"DROP TABLE IF EXISTS {mt} SYNC") + node.query(f"DROP TABLE IF EXISTS {iceberg}") + + +def test_export_part_with_bucket_partition(cluster): + """ + Export a part from a MergeTree table partitioned by icebergBucket(8, user_id) + to a matching Iceberg table. + + Verifies that the bucket partition expression is accepted for EXPORT PART and + that data lands correctly in the Iceberg bucket partition. + """ + node = cluster.instances["node1"] + sfx = unique_suffix() + mt = f"mt_bucket_{sfx}" + iceberg = f"iceberg_bucket_{sfx}" + + cols = "id Int64, user_id Int64, value String" + partition_by = "icebergBucket(8, user_id)" + + make_mt(node, mt, cols, partition_by) + make_iceberg_s3(node, iceberg, cols, partition_by) + + # Both rows go to the same bucket (user_id=42 → bucket 2 for N=8) + node.query(f"INSERT INTO {mt} VALUES (1, 42, 'hello'), (2, 42, 'world')") + + pid = first_partition_id(node, mt) + part = get_part(node, mt, pid) + + export_part(node, mt, part, iceberg) + wait_for_export_part(node, mt, part) + + count = int(node.query(f"SELECT count() FROM {iceberg}").strip()) + assert count == 2, f"Expected 2 rows in Iceberg table, got {count}" + + result = node.query( + f"SELECT id, user_id, value FROM {iceberg} ORDER BY id" + ).strip() + assert "1\t42\thello" in result, f"Row 1 missing or incorrect:\n{result}" + assert "2\t42\tworld" in result, f"Row 2 missing or incorrect:\n{result}" + + assert_part_log(node, mt, part) + + node.query(f"DROP TABLE IF EXISTS {mt} SYNC") + node.query(f"DROP TABLE IF EXISTS {iceberg}") + + +def test_export_part_writes_column_statistics(cluster): + """ + Export a MergeTree part that contains one NULL and verify that the resulting + Iceberg manifest entry carries accurate per-file column statistics: + record_count, file_size_in_bytes, column_sizes, null_value_counts, + and lower/upper bounds (Int32 + String + Nullable(String) mix). + """ + node = cluster.instances["node1"] + sfx = unique_suffix() + mt = f"mt_stats_{sfx}" + iceberg = f"iceberg_stats_{sfx}" + + columns = "id Int32, name String, tag Nullable(String), year Int32" + + make_mt(node, mt, columns, "year", order_by="id") + make_iceberg_s3(node, iceberg, columns, "year") + + node.query( + f""" + INSERT INTO {mt} (id, name, tag, year) VALUES + (1, 'aaa', 'x', 2020), + (2, 'mmm', NULL, 2020), + (3, 'zzz', 'y', 2020), + (4, 'kkk', 'z', 2021) + """ + ) + + part_2020 = get_part(node, mt, "2020") + export_part(node, mt, part_2020, iceberg) + wait_for_export_part(node, mt, part_2020) + + count = int(node.query(f"SELECT count() FROM {iceberg}").strip()) + assert count == 3, f"Expected 3 rows after export, got {count}" + + query_id = f"stats_part_{sfx}" + node.query( + f"SELECT * FROM {iceberg} ORDER BY id", + query_id=query_id, + settings={"iceberg_metadata_log_level": "manifest_file_entry"}, + ) + + entries = fetch_manifest_entries(node, query_id) + assert_exported_stats(entries) + + node.query(f"DROP TABLE IF EXISTS {mt} SYNC") + node.query(f"DROP TABLE IF EXISTS {iceberg}") + + +def test_export_part_column_count_mismatch_source_more_is_rejected(cluster): + """ + Source has 3 columns (id, year, extra), destination has 2 (id, year). + The ALTER must be rejected synchronously with NUMBER_OF_COLUMNS_DOESNT_MATCH + and the Iceberg table must remain empty. + """ + node = cluster.instances["node1"] + sfx = unique_suffix() + mt = f"mt_count_more_{sfx}" + iceberg = f"iceberg_count_more_{sfx}" + + make_mt(node, mt, "id Int32, year Int32, extra String", "year") + make_iceberg_s3(node, iceberg, "id Int32, year Int32", "year") + + node.query(f"INSERT INTO {mt} VALUES (1, 2020, 'foo'), (2, 2020, 'bar')") + part_2020 = get_part(node, mt, "2020") + + error = node.query_and_get_error( + f"ALTER TABLE {mt} EXPORT PART '{part_2020}' TO TABLE {iceberg} " + f"SETTINGS allow_experimental_export_merge_tree_part = 1, " + f"allow_experimental_insert_into_iceberg = 1" + ) + assert "NUMBER_OF_COLUMNS_DOESNT_MATCH" in error, ( + f"Expected NUMBER_OF_COLUMNS_DOESNT_MATCH for source>dest column count, " + f"got: {error!r}" + ) + + node.query(f"DROP TABLE IF EXISTS {mt} SYNC") + node.query(f"DROP TABLE IF EXISTS {iceberg}") + + +def test_export_part_column_count_mismatch_source_fewer_is_rejected(cluster): + """ + Source has 2 columns (id, year), destination has 3 (id, year, extra). + Same expected synchronous rejection as the source>dest case. + """ + node = cluster.instances["node1"] + sfx = unique_suffix() + mt = f"mt_count_fewer_{sfx}" + iceberg = f"iceberg_count_fewer_{sfx}" + + make_mt(node, mt, "id Int32, year Int32", "year") + make_iceberg_s3(node, iceberg, "id Int32, year Int32, extra String", "year") + + node.query(f"INSERT INTO {mt} VALUES (1, 2020), (2, 2020)") + part_2020 = get_part(node, mt, "2020") + + error = node.query_and_get_error( + f"ALTER TABLE {mt} EXPORT PART '{part_2020}' TO TABLE {iceberg} " + f"SETTINGS allow_experimental_export_merge_tree_part = 1, " + f"allow_experimental_insert_into_iceberg = 1" + ) + assert "NUMBER_OF_COLUMNS_DOESNT_MATCH" in error, ( + f"Expected NUMBER_OF_COLUMNS_DOESNT_MATCH for source Int32) succeeds once the user opts in via + export_merge_tree_part_allow_lossy_cast.""" + node = cluster.instances["node1"] + sfx = unique_suffix() + mt = f"mt_narrow_fit_{sfx}" + iceberg = f"iceberg_narrow_fit_{sfx}" + + make_mt(node, mt, "id Int64, year Int32", "year") + make_iceberg_s3(node, iceberg, "id Int32, year Int32", "year") + + node.query(f"INSERT INTO {mt} VALUES (1, 2020), (2, 2020)") + part_2020 = get_part(node, mt, "2020") + + export_part(node, mt, part_2020, iceberg, "export_merge_tree_part_allow_lossy_cast = 1") + wait_for_export_part(node, mt, part_2020) + + count = int(node.query(f"SELECT count() FROM {iceberg}").strip()) + assert count == 2, f"Expected 2 rows in Iceberg table after export, got {count}" + + result = node.query( + f"SELECT id, toTypeName(id), year FROM {iceberg} ORDER BY id" + ).strip() + assert result == "1\tInt32\t2020\n2\tInt32\t2020", ( + f"Unexpected narrowed data:\n{result}" + ) + + assert_part_log(node, mt, part_2020) + + node.query(f"DROP TABLE IF EXISTS {mt} SYNC") + node.query(f"DROP TABLE IF EXISTS {iceberg}") + + +def test_export_part_runtime_cast_failure_propagates_async(cluster): + """A String value that cannot be parsed as the destination Int32 passes the + synchronous lossy-cast gate (with export_merge_tree_part_allow_lossy_cast = 1) but + fails at runtime in the async worker; the failure surfaces in system.part_log and + Iceberg is left empty. + + (Integer overflow is not used because the internal cast uses CastType::nonAccurate, + which wraps rather than throwing.) + """ + node = cluster.instances["node1"] + sfx = unique_suffix() + mt = f"mt_runtime_cast_fail_{sfx}" + iceberg = f"iceberg_runtime_cast_fail_{sfx}" + + make_mt(node, mt, "id String, year Int32", "year") + make_iceberg_s3(node, iceberg, "id Int32, year Int32", "year") + + node.query(f"INSERT INTO {mt} VALUES ('not a number', 2020)") + part_2020 = get_part(node, mt, "2020") + + export_part(node, mt, part_2020, iceberg, "export_merge_tree_part_allow_lossy_cast = 1") + + exception = wait_for_failed_export_part(node, mt, part_2020) + assert exception, ( + f"Expected non-empty exception text on failed ExportPart entry, got {exception!r}" + ) + + count = int(node.query(f"SELECT count() FROM {iceberg}").strip()) + assert count == 0, ( + f"Expected 0 rows in Iceberg table after failed export, got {count}" + ) + + node.query(f"DROP TABLE IF EXISTS {mt} SYNC") + node.query(f"DROP TABLE IF EXISTS {iceberg}") + + +def test_export_part_tuple_subcolumn_partition_key_iceberg_rejected(cluster): + node = cluster.instances["node1"] + sfx = unique_suffix() + mt = f"mt_tuple_subcol_{sfx}" + iceberg = f"iceberg_tuple_subcol_{sfx}" + iceberg_partitioned = f"iceberg_tuple_subcol_part_{sfx}" + + create_error = node.query_and_get_error( + f"CREATE TABLE {iceberg_partitioned} (t Tuple(b Int32, a Int32), val String) " + f"ENGINE = IcebergS3('http://minio1:9001/root/data/{iceberg_partitioned}/', 'minio', 'ClickHouse_Minio_P@ssw0rd') " + f"PARTITION BY t.a" + ) + assert "Unknown field to partition" in create_error, ( + f"Expected Iceberg to reject the tuple subcolumn partition key at CREATE time, " + f"got: {create_error!r}" + ) + + make_mt(node, mt, "t Tuple(a Int32, b Int32), val String", "t.a") + make_iceberg_s3(node, iceberg, "t Tuple(b Int32, a Int32), val String", "val") + + node.query(f"INSERT INTO {mt} VALUES ((1, 99), 'x')") + + part = node.query( + f"SELECT name FROM system.parts WHERE database = currentDatabase() " + f"AND table = '{mt}' AND active ORDER BY name LIMIT 1" + ).strip() + + export_error = node.query_and_get_error( + f"ALTER TABLE {mt} EXPORT PART '{part}' TO TABLE {iceberg} " + f"SETTINGS allow_experimental_export_merge_tree_part = 1, " + f"allow_experimental_insert_into_iceberg = 1" + ) + assert "different Tuple element layout" in export_error, ( + f"The destination declares the elements of `t` in the opposite order, so the export " + f"of the tuple subcolumn partition key of {mt} has to be rejected, got: " + f"{export_error!r}" + ) + + count = int(node.query(f"SELECT count() FROM {iceberg}").strip()) + assert count == 0, f"Expected 0 rows in Iceberg table after rejected export, got {count}" + + node.query(f"DROP TABLE IF EXISTS {mt} SYNC") + node.query(f"DROP TABLE IF EXISTS {iceberg}") diff --git a/tests/integration/test_export_merge_tree_part_to_object_storage/__init__.py b/tests/integration/test_export_merge_tree_part_to_object_storage/__init__.py new file mode 100644 index 000000000000..e69de29bb2d1 diff --git a/tests/integration/test_export_merge_tree_part_to_object_storage/configs/named_collections.xml b/tests/integration/test_export_merge_tree_part_to_object_storage/configs/named_collections.xml new file mode 100644 index 000000000000..d46920b7ba88 --- /dev/null +++ b/tests/integration/test_export_merge_tree_part_to_object_storage/configs/named_collections.xml @@ -0,0 +1,9 @@ + + + + http://minio1:9001/root/data + minio + ClickHouse_Minio_P@ssw0rd + + + \ No newline at end of file diff --git a/tests/integration/test_export_merge_tree_part_to_object_storage/test.py b/tests/integration/test_export_merge_tree_part_to_object_storage/test.py new file mode 100644 index 000000000000..3ca7915e0811 --- /dev/null +++ b/tests/integration/test_export_merge_tree_part_to_object_storage/test.py @@ -0,0 +1,1013 @@ +import logging +import time +import uuid +from typing import NamedTuple + +import pytest + +from helpers.cluster import ClickHouseCluster +from helpers.network import PartitionManager + + +def skip_if_remote_database_disk_enabled(cluster): + """Skip test if any instance in the cluster has remote database disk enabled. + + Tests that block MinIO cannot run when remote database disk is enabled, + as the database metadata is stored on MinIO and blocking it would break the database. + """ + for instance in cluster.instances.values(): + if instance.with_remote_database_disk: + pytest.skip("Test cannot run with remote database disk enabled (db disk), as it blocks MinIO which stores database metadata") + + +@pytest.fixture(scope="module") +def cluster(): + try: + cluster = ClickHouseCluster(__file__) + cluster.add_instance( + "node1", + main_configs=["configs/named_collections.xml"], + with_minio=True, + ) + logging.info("Starting cluster...") + cluster.start() + yield cluster + finally: + cluster.shutdown() + + +def create_s3_table(node, s3_table): + node.query(f"CREATE TABLE {s3_table} (id UInt64, year UInt16) ENGINE = S3(s3_conn, filename='{s3_table}', format=Parquet, partition_strategy='hive') PARTITION BY year") + + +def wait_for_export_part(node, table, part, timeout=60, poll_interval=0.5): + deadline = time.time() + timeout + while time.time() < deadline: + node.query("SYSTEM FLUSH LOGS") + count = node.query( + f"SELECT count() FROM system.part_log " + f"WHERE event_type = 'ExportPart' " + f"AND table = '{table}' " + f"AND part_name = '{part}'" + ).strip() + if count != "0": + return + time.sleep(poll_interval) + raise TimeoutError( + f"ExportPart event for part {part!r} in table {table!r} did not appear within {timeout}s" + ) + + +def create_tables_and_insert_data(node, mt_table, s3_table): + # enable_block_number_column and enable_block_offset_column are needed for patch parts support + node.query(f"CREATE TABLE {mt_table} (id UInt64, year UInt16) ENGINE = MergeTree() PARTITION BY year ORDER BY tuple() SETTINGS enable_block_number_column = 1, enable_block_offset_column = 1") + node.query(f"INSERT INTO {mt_table} VALUES (1, 2020), (2, 2020), (3, 2020), (4, 2021)") + + create_s3_table(node, s3_table) + + +def test_drop_column_during_export_snapshot(cluster): + skip_if_remote_database_disk_enabled(cluster) + node = cluster.instances["node1"] + + postfix = str(uuid.uuid4()).replace("-", "_") + + mt_table = f"mutations_snapshot_mt_table_{postfix}" + s3_table = f"mutations_snapshot_s3_table_{postfix}" + + create_tables_and_insert_data(node, mt_table, s3_table) + + # Block traffic to/from MinIO to force upload errors and retries, following existing S3 tests style + minio_ip = cluster.minio_ip + minio_port = cluster.minio_port + + # Ensure export sees a consistent snapshot at start time even if we mutate the source later + with PartitionManager() as pm: + # Block responses from MinIO (source_port matches MinIO service) + pm_rule_reject_responses = { + "instance": node, + "destination": node.ip_address, + "protocol": "tcp", + "source_port": minio_port, + "action": "REJECT --reject-with tcp-reset", + } + pm.add_rule(pm_rule_reject_responses) + + # Block requests to MinIO (destination: MinIO, destination_port: minio_port) + pm_rule_reject_requests = { + "instance": node, + "destination": minio_ip, + "protocol": "tcp", + "destination_port": minio_port, + "action": "REJECT --reject-with tcp-reset", + } + pm.add_rule(pm_rule_reject_requests) + + # Start export of 2020 + node.query( + f"ALTER TABLE {mt_table} EXPORT PART '2020_1_1_0' TO TABLE {s3_table};" + ) + + # Drop a column that is required for the export + node.query(f"ALTER TABLE {mt_table} DROP COLUMN id") + + time.sleep(3) + # assert the mutation has been applied AND the data has not been exported yet + assert "Unknown expression identifier `id`" in node.query_and_get_error(f"SELECT id FROM {mt_table}"), "Column id is not removed" + + # Wait for export to finish and then verify destination still reflects the original snapshot (3 rows) + time.sleep(5) + assert node.query(f"SELECT count() FROM {s3_table} WHERE id >= 0") == '3\n', "Export did not preserve snapshot at start time after source mutation" + + +def test_add_column_during_export(cluster): + skip_if_remote_database_disk_enabled(cluster) + node = cluster.instances["node1"] + + postfix = str(uuid.uuid4()).replace("-", "_") + + mt_table = f"add_column_during_export_mt_table_{postfix}" + s3_table = f"add_column_during_export_s3_table_{postfix}" + + create_tables_and_insert_data(node, mt_table, s3_table) + + # Block traffic to/from MinIO to force upload errors and retries, following existing S3 tests style + minio_ip = cluster.minio_ip + minio_port = cluster.minio_port + + # Ensure export sees a consistent snapshot at start time even if we mutate the source later + with PartitionManager() as pm: + # Block responses from MinIO (source_port matches MinIO service) + pm_rule_reject_responses = { + "instance": node, + "destination": node.ip_address, + "protocol": "tcp", + "source_port": minio_port, + "action": "REJECT --reject-with tcp-reset", + } + pm.add_rule(pm_rule_reject_responses) + + # Block requests to MinIO (destination: MinIO, destination_port: minio_port) + pm_rule_reject_requests = { + "instance": node, + "destination": minio_ip, + "protocol": "tcp", + "destination_port": minio_port, + "action": "REJECT --reject-with tcp-reset", + } + pm.add_rule(pm_rule_reject_requests) + + # Start export of 2020 + node.query( + f"ALTER TABLE {mt_table} EXPORT PART '2020_1_1_0' TO TABLE {s3_table};" + ) + + node.query(f"ALTER TABLE {mt_table} ADD COLUMN id2 UInt64") + + time.sleep(3) + + # assert the mutation has been applied AND the data has not been exported yet + assert node.query(f"SELECT count(id2) FROM {mt_table}") == '4\n', "Column id2 is not added" + + # Wait for export to finish and then verify destination still reflects the original snapshot (3 rows) + time.sleep(5) + assert node.query(f"SELECT count() FROM {s3_table} WHERE id >= 0") == '3\n', "Export did not preserve snapshot at start time after source mutation" + assert "Unknown expression identifier `id2`" in node.query_and_get_error(f"SELECT id2 FROM {s3_table}"), "Column id2 is present in the exported data" + + +def test_pending_mutations_throw_before_export(cluster): + """Test that pending mutations before export throw an error with default settings.""" + node = cluster.instances["node1"] + + postfix = str(uuid.uuid4()).replace("-", "_") + + mt_table = f"pending_mutations_throw_mt_table_{postfix}" + s3_table = f"pending_mutations_throw_s3_table_{postfix}" + + create_tables_and_insert_data(node, mt_table, s3_table) + + node.query(f"SYSTEM STOP MERGES {mt_table}") + + node.query(f"ALTER TABLE {mt_table} UPDATE id = id + 100 WHERE year = 2020") + + mutations = node.query(f"SELECT count() FROM system.mutations WHERE table = '{mt_table}' AND is_done = 0") + assert mutations.strip() != '0', "Mutation should be pending" + + error = node.query_and_get_error( + f"ALTER TABLE {mt_table} EXPORT PART '2020_1_1_0' TO TABLE {s3_table} SETTINGS export_merge_tree_part_throw_on_pending_mutations=true" + ) + + assert "PENDING_MUTATIONS_NOT_ALLOWED" in error, f"Expected error about pending mutations, got: {error}" + + +def test_pending_mutations_skip_before_export(cluster): + """Test that pending mutations before export are skipped with throw_on_pending_mutations=false.""" + node = cluster.instances["node1"] + + postfix = str(uuid.uuid4()).replace("-", "_") + + mt_table = f"pending_mutations_skip_mt_table_{postfix}" + s3_table = f"pending_mutations_skip_s3_table_{postfix}" + + create_tables_and_insert_data(node, mt_table, s3_table) + + node.query(f"SYSTEM STOP MERGES {mt_table}") + + node.query(f"ALTER TABLE {mt_table} UPDATE id = id + 100 WHERE year = 2020") + + mutations = node.query(f"SELECT count() FROM system.mutations WHERE table = '{mt_table}' AND is_done = 0") + assert mutations.strip() != '0', "Mutation should be pending" + + node.query( + f"ALTER TABLE {mt_table} EXPORT PART '2020_1_1_0' TO TABLE {s3_table} " + f"SETTINGS export_merge_tree_part_throw_on_pending_mutations=false" + ) + + time.sleep(5) + + result = node.query(f"SELECT id FROM {s3_table} WHERE year = 2020 ORDER BY id") + assert "101" not in result and "102" not in result and "103" not in result, \ + "Export should contain original data before mutation" + assert "1\n2\n3" in result, "Export should contain original data" + + +def test_data_mutations_after_export_started(cluster): + """Test that mutations applied after export starts don't affect the exported data.""" + skip_if_remote_database_disk_enabled(cluster) + node = cluster.instances["node1"] + + postfix = str(uuid.uuid4()).replace("-", "_") + + mt_table = f"mutations_after_export_mt_table_{postfix}" + s3_table = f"mutations_after_export_s3_table_{postfix}" + + create_tables_and_insert_data(node, mt_table, s3_table) + + # Block traffic to MinIO to delay export + minio_ip = cluster.minio_ip + minio_port = cluster.minio_port + + with PartitionManager() as pm: + pm_rule_reject_responses = { + "instance": node, + "destination": node.ip_address, + "protocol": "tcp", + "source_port": minio_port, + "action": "REJECT --reject-with tcp-reset", + } + pm.add_rule(pm_rule_reject_responses) + + pm_rule_reject_requests = { + "instance": node, + "destination": minio_ip, + "protocol": "tcp", + "destination_port": minio_port, + "action": "REJECT --reject-with tcp-reset", + } + pm.add_rule(pm_rule_reject_requests) + + node.query( + f"ALTER TABLE {mt_table} EXPORT PART '2020_1_1_0' TO TABLE {s3_table} " + f"SETTINGS export_merge_tree_part_throw_on_pending_mutations=true" + ) + + node.query(f"ALTER TABLE {mt_table} UPDATE id = id + 100 WHERE year = 2020") + + time.sleep(5) + + result = node.query(f"SELECT id FROM {s3_table} WHERE year = 2020 ORDER BY id") + assert "1\n2\n3" in result, "Export should contain original data before mutation" + assert "101" not in result, "Export should not contain mutated data" + + +def test_pending_patch_parts_throw_before_export(cluster): + """Test that pending patch parts before export throw an error with default settings.""" + node = cluster.instances["node1"] + + postfix = str(uuid.uuid4()).replace("-", "_") + + mt_table = f"pending_patches_throw_mt_table_{postfix}" + s3_table = f"pending_patches_throw_s3_table_{postfix}" + + create_tables_and_insert_data(node, mt_table, s3_table) + + node.query(f"SYSTEM STOP MERGES {mt_table}") + + node.query(f"UPDATE {mt_table} SET id = id + 100 WHERE year = 2020") + + error = node.query_and_get_error( + f"ALTER TABLE {mt_table} EXPORT PART '2020_1_1_0' TO TABLE {s3_table}" + ) + + node.query(f"DROP TABLE {mt_table}") + + assert "PENDING_MUTATIONS_NOT_ALLOWED" in error or "pending patch parts" in error.lower(), \ + f"Expected error about pending patch parts, got: {error}" + + +def test_pending_patch_parts_skip_before_export(cluster): + """Test that pending patch parts before export are skipped with throw_on_pending_patch_parts=false.""" + node = cluster.instances["node1"] + + postfix = str(uuid.uuid4()).replace("-", "_") + + mt_table = f"pending_patches_skip_mt_table_{postfix}" + s3_table = f"pending_patches_skip_s3_table_{postfix}" + + create_tables_and_insert_data(node, mt_table, s3_table) + + node.query(f"SYSTEM STOP MERGES {mt_table}") + + node.query(f"UPDATE {mt_table} SET id = id + 100 WHERE year = 2020") + + node.query( + f"ALTER TABLE {mt_table} EXPORT PART '2020_1_1_0' TO TABLE {s3_table} " + f"SETTINGS export_merge_tree_part_throw_on_pending_patch_parts=false" + ) + + time.sleep(5) + + result = node.query(f"SELECT id FROM {s3_table} WHERE year = 2020 ORDER BY id") + assert "1\n2\n3" in result, "Export should contain original data before patch" + + node.query(f"DROP TABLE {mt_table}") + + +class RejectedPartExportCase(NamedTuple): + src_columns: str + src_partition_by: str + dst_columns: str + dst_partition_by: str + insert_values: str + error_substrings: tuple = () + partition_strategy: str = "hive" + + +REJECTED_PART_EXPORT_CASES = [ + pytest.param( + RejectedPartExportCase( + src_columns="a Int32, b Int32", + src_partition_by="a", + dst_columns="b Int32, a Int32", + dst_partition_by="a", + insert_values="(1, 1), (1, 2)", + error_substrings=( + "partition key column 'a' is at position 0 in the source table", + ), + ), + id="same_partition_key_different_column_order_single_column", + ), + pytest.param( + RejectedPartExportCase( + src_columns="a Int32, b Int32, c Int32, val String", + src_partition_by="(a, b, c)", + dst_columns="c Int32, b Int32, a Int32, val String", + dst_partition_by="(a, b, c)", + insert_values="(1, 1, 1, 'x'), (1, 1, 1, 'y')", + error_substrings=( + "partition key column 'a' is at position 0 in the source table", + ), + ), + id="same_partition_key_different_column_order_multi_column", + ), + pytest.param( + RejectedPartExportCase( + src_columns="a Int32, b Int32, c Int32, val String", + src_partition_by="(a, b)", + dst_columns="a Int32, b Int32, c Int32, val String", + dst_partition_by="(a, b, c)", + insert_values="(1, 2, 3, 'x')", + error_substrings=( + "column 'c', which is not part of the source MergeTree partition key", + ), + ), + id="multi_column_partition_key_more_in_destination", + ), + pytest.param( + RejectedPartExportCase( + src_columns="ts DateTime, category String, decoy DateTime, val String", + src_partition_by="(toYYYYMM(ts), category)", + dst_columns="decoy DateTime, category String, ts DateTime, val String", + dst_partition_by="(toYYYYMM(ts), category)", + insert_values=( + "('2024-03-05 15:00:00', 'category', " + "'2024-03-06 15:00:00', 'x')" + ), + error_substrings=( + "partition key column 'ts' is at position 0 in the source table", + ), + partition_strategy="wildcard", + ), + id="function_and_column_partition_key_owner_reordered", + ), + pytest.param( + RejectedPartExportCase( + src_columns=( + "t Tuple(ts DateTime, value Int32), category String, " + "decoy Tuple(ts DateTime, value Int32), val String" + ), + src_partition_by="(toYYYYMM(t.ts), category)", + dst_columns=( + "decoy Tuple(ts DateTime, value Int32), category String, " + "t Tuple(ts DateTime, value Int32), val String" + ), + dst_partition_by="(toYYYYMM(t.ts), category)", + insert_values=( + "(('2024-03-05 15:00:00', 1), 'category', " + "('2024-03-06 15:00:00', 2), 'x')" + ), + error_substrings=( + "partition key column 't' is at position 0 in the source table", + ), + partition_strategy="wildcard", + ), + id="function_over_subcolumn_partition_key_owner_reordered", + ), +] + + +@pytest.mark.parametrize("case", REJECTED_PART_EXPORT_CASES) +def test_export_part_partition_key_mismatch_variants_are_rejected(cluster, case): + skip_if_remote_database_disk_enabled(cluster) + node = cluster.instances["node1"] + + postfix = str(uuid.uuid4()).replace("-", "_") + mt_table = f"rejected_mt_table_{postfix}" + s3_table = f"rejected_s3_table_{postfix}" + + node.query(f""" + CREATE TABLE {mt_table} ({case.src_columns}) + ENGINE = MergeTree() + PARTITION BY {case.src_partition_by} + ORDER BY tuple() + SETTINGS enable_block_number_column = 1, enable_block_offset_column = 1 + """) + + filename = ( + f"{s3_table}/{{_partition_id}}/{{_file}}" + if case.partition_strategy == "wildcard" + else s3_table + ) + node.query(f""" + CREATE TABLE {s3_table} ({case.dst_columns}) + ENGINE = S3(s3_conn, filename='{filename}', format=Parquet, partition_strategy='{case.partition_strategy}') + PARTITION BY {case.dst_partition_by} + """) + + node.query(f"INSERT INTO {mt_table} VALUES {case.insert_values}") + + part_name = node.query( + f"SELECT name FROM system.parts WHERE database = currentDatabase() " + f"AND table = '{mt_table}' AND active ORDER BY name LIMIT 1" + ).strip() + + error = node.query_and_get_error(f"ALTER TABLE {mt_table} EXPORT PART '{part_name}' TO TABLE {s3_table}") + assert "BAD_ARGUMENTS" in error, f"Expected BAD_ARGUMENTS, got: {error}" + for substring in case.error_substrings: + assert substring in error, f"Expected {substring!r} in error, got: {error}" + + if case.partition_strategy == "hive": + count = int(node.query(f"SELECT count() FROM {s3_table}").strip()) + assert count == 0, ( + f"Expected 0 rows in destination after rejected export, got {count}" + ) + + +@pytest.mark.parametrize( + "dst_partition_by", + ["(a, b, c)", "(c, b, a)", "(a, b)"], + ids=["same", "reordered", "coarser"], +) +def test_export_part_multi_column_partition_key_success(cluster, dst_partition_by): + """The source key pins every column the destination partitions by, so the destination may + also name them in another order or leave some out: each destination expression is still + single-valued over a source partition.""" + skip_if_remote_database_disk_enabled(cluster) + node = cluster.instances["node1"] + + postfix = str(uuid.uuid4()).replace("-", "_") + mt_table = f"multi_pkey_ok_mt_table_{postfix}" + s3_table = f"multi_pkey_ok_s3_table_{postfix}" + + node.query(f""" + CREATE TABLE {mt_table} (a Int32, b Int32, c Int32, val String) + ENGINE = MergeTree() + PARTITION BY (a, b, c) + ORDER BY tuple() + SETTINGS enable_block_number_column = 1, enable_block_offset_column = 1 + """) + + node.query(f""" + CREATE TABLE {s3_table} (a Int32, b Int32, c Int32, val String) + ENGINE = S3(s3_conn, filename='{s3_table}', format=Parquet, partition_strategy='hive') + PARTITION BY {dst_partition_by} + """) + + node.query(f"INSERT INTO {mt_table} VALUES (1, 2, 3, 'x'), (1, 2, 3, 'y')") + + part_name = node.query( + f"SELECT name FROM system.parts WHERE database = currentDatabase() " + f"AND table = '{mt_table}' AND active ORDER BY name LIMIT 1" + ).strip() + + node.query(f"ALTER TABLE {mt_table} EXPORT PART '{part_name}' TO TABLE {s3_table}") + + time.sleep(5) + + count = int(node.query(f"SELECT count() FROM {s3_table}").strip()) + assert count == 2, f"Expected 2 rows in destination after export, got {count}" + + result = node.query(f"SELECT a, b, c, val FROM {s3_table} ORDER BY val").strip() + assert result == "1\t2\t3\tx\n1\t2\t3\ty", f"Unexpected exported data:\n{result}" + + +@pytest.mark.parametrize( + "owner_name, source_type, destination_type, partition_by, insert_value", + [ + pytest.param( + "t", + "Tuple(a Int32, b Int32)", + "Tuple(b Int32, a Int32)", + "t.a", + "(1, 99)", + id="named_subcolumn", + ), + pytest.param( + "t", + "Tuple(a Int32, b Int32)", + "Tuple(b Int32, a Int32)", + "tupleElement(t, 1)", + "(1, 99)", + id="positional_tuple_element", + ), + pytest.param( + "arr", + "Array(Tuple(a Int32, b Int32))", + "Array(Tuple(b Int32, a Int32))", + "tupleElement(arr[1], 'a')", + "[(1, 99)]", + id="tuple_nested_in_array", + ), + pytest.param( + "m", + "Map(String, Tuple(a Int32, b Int32))", + "Map(String, Tuple(b Int32, a Int32))", + "tupleElement(m['key'], 'a')", + "map('key', (1, 99))", + id="tuple_nested_in_map_value", + ), + ], +) +def test_export_part_tuple_fields_reordered_for_partition_key_is_rejected( + cluster, + owner_name, + source_type, + destination_type, + partition_by, + insert_value, +): + skip_if_remote_database_disk_enabled(cluster) + node = cluster.instances["node1"] + + postfix = str(uuid.uuid4()).replace("-", "_") + mt_table = f"reordered_tuple_mt_table_{postfix}" + s3_table = f"reordered_tuple_s3_table_{postfix}" + + node.query(f""" + CREATE TABLE {mt_table} ({owner_name} {source_type}, val String) + ENGINE = MergeTree() + PARTITION BY {partition_by} + ORDER BY tuple() + SETTINGS enable_block_number_column = 1, enable_block_offset_column = 1 + """) + + node.query(f""" + CREATE TABLE {s3_table} ({owner_name} {destination_type}, val String) + ENGINE = S3(s3_conn, filename='{s3_table}/{{_partition_id}}/{{_file}}', format=Parquet, partition_strategy='wildcard') + PARTITION BY {partition_by} + """) + + node.query(f"INSERT INTO {mt_table} VALUES ({insert_value}, 'x')") + + part_name = node.query( + f"SELECT name FROM system.parts WHERE database = currentDatabase() " + f"AND table = '{mt_table}' AND active ORDER BY name LIMIT 1" + ).strip() + + error = node.query_and_get_error( + f"ALTER TABLE {mt_table} EXPORT PART '{part_name}' TO TABLE {s3_table}" + ) + assert "BAD_ARGUMENTS" in error and "different Tuple element layout" in error, ( + f"Expected export to reject reordered named `Tuple` fields used by " + f"`PARTITION BY {partition_by}`, got: {error!r}" + ) + + +def test_export_part_subcolumn_partition_key_different_subcolumn_is_rejected(cluster): + skip_if_remote_database_disk_enabled(cluster) + node = cluster.instances["node1"] + + postfix = str(uuid.uuid4()).replace("-", "_") + mt_table = f"subcol_diff_subcol_mt_table_{postfix}" + s3_table = f"subcol_diff_subcol_s3_table_{postfix}" + + node.query(f""" + CREATE TABLE {mt_table} (a Tuple(b Int32, c Int32), val String) + ENGINE = MergeTree() + PARTITION BY a.b + ORDER BY tuple() + SETTINGS enable_block_number_column = 1, enable_block_offset_column = 1 + """) + + node.query(f""" + CREATE TABLE {s3_table} (a Tuple(b Int32, c Int32), val String) + ENGINE = S3(s3_conn, filename='{s3_table}/{{_partition_id}}/{{_file}}', format=Parquet, partition_strategy='wildcard') + PARTITION BY a.c + """) + + node.query(f"INSERT INTO {mt_table} VALUES ((1, 2), 'x')") + + part_name = node.query( + f"SELECT name FROM system.parts WHERE database = currentDatabase() " + f"AND table = '{mt_table}' AND active ORDER BY name LIMIT 1" + ).strip() + + error = node.query_and_get_error( + f"ALTER TABLE {mt_table} EXPORT PART '{part_name}' TO TABLE {s3_table}" + ) + assert ( + "BAD_ARGUMENTS" in error + and "column 'a.c', which is not part of the source MergeTree partition key" + in error + ), ( + f"Both tables declare `a` as the same Tuple(b Int32, c Int32) (so the column-cast " + f"check passes and the owner-name-only `partition_key_owner_columns` contains " + f"only `a`, so `verifyExportSchemaCastable` cannot distinguish `a.b` from " + f"`a.c`), but the source partitions by `a.b` while the destination partitions by " + f"`a.c`, which the source key does not pin, so the compatibility gate has to " + f"reject it; got: {error!r}" + ) + + +def test_export_part_tuple_subcolumn_partition_key_owner_column_reordered_is_rejected(cluster): + skip_if_remote_database_disk_enabled(cluster) + node = cluster.instances["node1"] + + postfix = str(uuid.uuid4()).replace("-", "_") + mt_table = f"tuple_subcol_owner_mt_table_{postfix}" + s3_table = f"tuple_subcol_owner_s3_table_{postfix}" + + node.query(f""" + CREATE TABLE {mt_table} (t Tuple(a Int32, b Int32), decoy Tuple(a Int32, b Int32), val String) + ENGINE = MergeTree() + PARTITION BY t.a + ORDER BY tuple() + SETTINGS enable_block_number_column = 1, enable_block_offset_column = 1 + """) + + node.query(f""" + CREATE TABLE {s3_table} (decoy Tuple(a Int32, b Int32), t Tuple(a Int32, b Int32), val String) + ENGINE = S3(s3_conn, filename='{s3_table}/{{_partition_id}}/{{_file}}', format=Parquet, partition_strategy='wildcard') + PARTITION BY t.a + """) + + node.query(f"INSERT INTO {mt_table} VALUES ((1, 100), (2, 200), 'x')") + + part_name = node.query( + f"SELECT name FROM system.parts WHERE database = currentDatabase() " + f"AND table = '{mt_table}' AND active ORDER BY name LIMIT 1" + ).strip() + + error = node.query_and_get_error( + f"ALTER TABLE {mt_table} EXPORT PART '{part_name}' TO TABLE {s3_table}" + ) + assert "BAD_ARGUMENTS" in error and "partition key column" in error, ( + f"Expected export to reject `t` and `decoy` swapping positions around the " + f"partition key column `t.a`, the same way a plain (non-tuple) partition key " + f"column position swap is rejected; got: {error!r}" + ) + + +def test_export_part_multiple_partition_key_subcolumns_with_same_owner_reordered_is_rejected(cluster): + skip_if_remote_database_disk_enabled(cluster) + node = cluster.instances["node1"] + + postfix = str(uuid.uuid4()).replace("-", "_") + mt_table = f"same_owner_subcolumns_mt_table_{postfix}" + s3_table = f"same_owner_subcolumns_s3_table_{postfix}" + + node.query(f""" + CREATE TABLE {mt_table} ( + t Tuple(a Int32, b Int32), + decoy Tuple(a Int32, b Int32), + val String + ) + ENGINE = MergeTree() + PARTITION BY (t.a, t.b) + ORDER BY tuple() + SETTINGS enable_block_number_column = 1, enable_block_offset_column = 1 + """) + + node.query(f""" + CREATE TABLE {s3_table} ( + decoy Tuple(a Int32, b Int32), + t Tuple(a Int32, b Int32), + val String + ) + ENGINE = S3(s3_conn, filename='{s3_table}/{{_partition_id}}/{{_file}}', format=Parquet, partition_strategy='wildcard') + PARTITION BY (t.a, t.b) + """) + + node.query(f"INSERT INTO {mt_table} VALUES ((1, 10), (2, 20), 'x')") + + part_name = node.query( + f"SELECT name FROM system.parts WHERE database = currentDatabase() " + f"AND table = '{mt_table}' AND active ORDER BY name LIMIT 1" + ).strip() + + error = node.query_and_get_error( + f"ALTER TABLE {mt_table} EXPORT PART '{part_name}' TO TABLE {s3_table}" + ) + assert "BAD_ARGUMENTS" in error and "partition key column 't'" in error, ( + f"Expected both `t.a` and `t.b` to resolve to the same top-level owner `t` " + f"and reject swapping `t` with `decoy`; got: {error!r}" + ) + + +def test_export_part_multi_level_subcolumn_partition_key_owner_reordered_is_rejected(cluster): + skip_if_remote_database_disk_enabled(cluster) + node = cluster.instances["node1"] + + postfix = str(uuid.uuid4()).replace("-", "_") + mt_table = f"nested_subcol_owner_mt_table_{postfix}" + s3_table = f"nested_subcol_owner_s3_table_{postfix}" + + node.query(f""" + CREATE TABLE {mt_table} ( + t Tuple(x Tuple(a Int32, b Int32), c Int32), + decoy Tuple(x Tuple(a Int32, b Int32), c Int32), + val String + ) + ENGINE = MergeTree() + PARTITION BY t.x.a + ORDER BY tuple() + SETTINGS enable_block_number_column = 1, enable_block_offset_column = 1 + """) + + node.query(f""" + CREATE TABLE {s3_table} ( + decoy Tuple(x Tuple(a Int32, b Int32), c Int32), + t Tuple(x Tuple(a Int32, b Int32), c Int32), + val String + ) + ENGINE = S3(s3_conn, filename='{s3_table}/{{_partition_id}}/{{_file}}', format=Parquet, partition_strategy='wildcard') + PARTITION BY t.x.a + """) + + node.query(f"INSERT INTO {mt_table} VALUES ((((1, 100), 1000)), (((2, 200), 2000)), 'x')") + + part_name = node.query( + f"SELECT name FROM system.parts WHERE database = currentDatabase() " + f"AND table = '{mt_table}' AND active ORDER BY name LIMIT 1" + ).strip() + + error = node.query_and_get_error( + f"ALTER TABLE {mt_table} EXPORT PART '{part_name}' TO TABLE {s3_table}" + ) + assert "BAD_ARGUMENTS" in error and "partition key column" in error, ( + f"Expected export to reject `t` and `decoy` swapping positions around the " + f"two-level-deep partition key column `t.x.a`. This only works if " + f"`getNameInStorage` resolves all the way to the top-level column `t`, not to " + f"the intermediate level `t.x`; got: {error!r}" + ) + + +def test_export_part_multiple_subcolumn_partition_keys_owner_reordered_is_rejected(cluster): + skip_if_remote_database_disk_enabled(cluster) + node = cluster.instances["node1"] + + postfix = str(uuid.uuid4()).replace("-", "_") + mt_table = f"multi_subcol_key_mt_table_{postfix}" + s3_table = f"multi_subcol_key_s3_table_{postfix}" + + node.query(f""" + CREATE TABLE {mt_table} ( + t Tuple(a Int32, x Int32), + u Tuple(b Int32, y Int32), + decoy Tuple(b Int32, y Int32), + val String + ) + ENGINE = MergeTree() + PARTITION BY (t.a, u.b) + ORDER BY tuple() + SETTINGS enable_block_number_column = 1, enable_block_offset_column = 1 + """) + + node.query(f""" + CREATE TABLE {s3_table} ( + t Tuple(a Int32, x Int32), + decoy Tuple(b Int32, y Int32), + u Tuple(b Int32, y Int32), + val String + ) + ENGINE = S3(s3_conn, filename='{s3_table}/{{_partition_id}}/{{_file}}', format=Parquet, partition_strategy='wildcard') + PARTITION BY (t.a, u.b) + """) + + node.query(f"INSERT INTO {mt_table} VALUES ((1, 10), (2, 20), (3, 30), 'x')") + + part_name = node.query( + f"SELECT name FROM system.parts WHERE database = currentDatabase() " + f"AND table = '{mt_table}' AND active ORDER BY name LIMIT 1" + ).strip() + + error = node.query_and_get_error( + f"ALTER TABLE {mt_table} EXPORT PART '{part_name}' TO TABLE {s3_table}" + ) + assert "BAD_ARGUMENTS" in error and "partition key column 'u'" in error, ( + f"`t` (owner of key part `t.a`) stays at position 0 on both sides, so the guard " + f"must independently catch `u` (owner of key part `u.b`) swapping positions " + f"with `decoy` — a partition key with two subcolumn-owning columns must have " + f"both validated, not just the first one encountered; got: {error!r}" + ) + + +def test_export_part_mixed_flat_and_subcolumn_partition_key_flat_part_reordered_is_rejected(cluster): + skip_if_remote_database_disk_enabled(cluster) + node = cluster.instances["node1"] + + postfix = str(uuid.uuid4()).replace("-", "_") + mt_table = f"mixed_key_mt_table_{postfix}" + s3_table = f"mixed_key_s3_table_{postfix}" + + node.query(f""" + CREATE TABLE {mt_table} (a Int32, t Tuple(b Int32, c Int32), decoy Int32, val String) + ENGINE = MergeTree() + PARTITION BY (a, t.b) + ORDER BY tuple() + SETTINGS enable_block_number_column = 1, enable_block_offset_column = 1 + """) + + node.query(f""" + CREATE TABLE {s3_table} (decoy Int32, t Tuple(b Int32, c Int32), a Int32, val String) + ENGINE = S3(s3_conn, filename='{s3_table}/{{_partition_id}}/{{_file}}', format=Parquet, partition_strategy='wildcard') + PARTITION BY (a, t.b) + """) + + node.query(f"INSERT INTO {mt_table} VALUES (1, (2, 3), 4, 'x')") + + part_name = node.query( + f"SELECT name FROM system.parts WHERE database = currentDatabase() " + f"AND table = '{mt_table}' AND active ORDER BY name LIMIT 1" + ).strip() + + error = node.query_and_get_error( + f"ALTER TABLE {mt_table} EXPORT PART '{part_name}' TO TABLE {s3_table}" + ) + assert "BAD_ARGUMENTS" in error and "partition key column 'a'" in error, ( + f"`t` (owner of key part `t.b`) stays at position 1 on both sides, so the guard " + f"must independently catch the plain, non-tuple key part `a` swapping positions " + f"with `decoy` — the pre-existing flat-column check and the new subcolumn-owner " + f"resolution must both keep working when combined in one `PARTITION BY` " + f"expression; got: {error!r}" + ) + + +def test_export_part_subcolumn_partition_key_owner_reordered_rejected_even_with_allow_lossy_cast(cluster): + skip_if_remote_database_disk_enabled(cluster) + node = cluster.instances["node1"] + + postfix = str(uuid.uuid4()).replace("-", "_") + mt_table = f"lossy_owner_mt_table_{postfix}" + s3_table = f"lossy_owner_s3_table_{postfix}" + + node.query(f""" + CREATE TABLE {mt_table} (t Tuple(a Int32, b Int32), decoy Tuple(a Int32, b Int32), val String) + ENGINE = MergeTree() + PARTITION BY t.a + ORDER BY tuple() + SETTINGS enable_block_number_column = 1, enable_block_offset_column = 1 + """) + + node.query(f""" + CREATE TABLE {s3_table} (decoy Tuple(a Int32, b Int32), t Tuple(a Int32, b Int32), val String) + ENGINE = S3(s3_conn, filename='{s3_table}/{{_partition_id}}/{{_file}}', format=Parquet, partition_strategy='wildcard') + PARTITION BY t.a + """) + + node.query(f"INSERT INTO {mt_table} VALUES ((1, 100), (2, 200), 'x')") + + part_name = node.query( + f"SELECT name FROM system.parts WHERE database = currentDatabase() " + f"AND table = '{mt_table}' AND active ORDER BY name LIMIT 1" + ).strip() + + error = node.query_and_get_error( + f"ALTER TABLE {mt_table} EXPORT PART '{part_name}' TO TABLE {s3_table} " + f"SETTINGS export_merge_tree_part_allow_lossy_cast = 1" + ) + assert "BAD_ARGUMENTS" in error and "partition key column" in error, ( + f"The partition-key position/name guard is checked before the " + f"`allow_lossy_cast` early-continue in verifyExportSchemaCastable, so setting " + f"`export_merge_tree_part_allow_lossy_cast = 1` must not suppress the rejection " + f"of `t`/`decoy` swapping positions around the partition key column `t.a`; " + f"got: {error!r}" + ) + + +def test_export_part_column_count_mismatch_source_more_is_rejected(cluster): + node = cluster.instances["node1"] + + postfix = str(uuid.uuid4()).replace("-", "_") + mt_table = f"count_more_mt_table_{postfix}" + s3_table = f"count_more_s3_table_{postfix}" + + node.query( + f"CREATE TABLE {mt_table} (id UInt64, year UInt16, extra String) " + f"ENGINE = MergeTree() PARTITION BY year ORDER BY tuple() " + f"SETTINGS enable_block_number_column = 1, enable_block_offset_column = 1" + ) + node.query(f"INSERT INTO {mt_table} VALUES (1, 2020, 'foo'), (2, 2020, 'bar')") + + create_s3_table(node=node, s3_table=s3_table) + + error = node.query_and_get_error( + f"ALTER TABLE {mt_table} EXPORT PART '2020_1_1_0' TO TABLE {s3_table}" + ) + assert "NUMBER_OF_COLUMNS_DOESNT_MATCH" in error, ( + f"Expected NUMBER_OF_COLUMNS_DOESNT_MATCH for source>dest column count, got: {error}" + ) + + count = int(node.query(f"SELECT count() FROM {s3_table}").strip()) + assert count == 0, f"Expected 0 rows in destination table after rejected export, got {count}" + + node.query(f"DROP TABLE {mt_table}") + node.query(f"DROP TABLE {s3_table}") + + +def test_export_part_column_count_mismatch_source_fewer_is_rejected(cluster): + node = cluster.instances["node1"] + + postfix = str(uuid.uuid4()).replace("-", "_") + mt_table = f"count_fewer_mt_table_{postfix}" + s3_table = f"count_fewer_s3_table_{postfix}" + + node.query( + f"CREATE TABLE {mt_table} (id UInt64, year UInt16) " + f"ENGINE = MergeTree() PARTITION BY year ORDER BY tuple() " + f"SETTINGS enable_block_number_column = 1, enable_block_offset_column = 1" + ) + node.query(f"INSERT INTO {mt_table} VALUES (1, 2020), (2, 2020)") + + node.query( + f"CREATE TABLE {s3_table} (id UInt64, year UInt16, extra String) " + f"ENGINE = S3(s3_conn, filename='{s3_table}', format=Parquet, partition_strategy='hive') " + f"PARTITION BY year" + ) + + error = node.query_and_get_error( + f"ALTER TABLE {mt_table} EXPORT PART '2020_1_1_0' TO TABLE {s3_table}" + ) + assert "NUMBER_OF_COLUMNS_DOESNT_MATCH" in error, ( + f"Expected NUMBER_OF_COLUMNS_DOESNT_MATCH for source + 1 + diff --git a/tests/integration/test_export_replicated_mt_partition_to_iceberg/configs/config.d/metadata_log.xml b/tests/integration/test_export_replicated_mt_partition_to_iceberg/configs/config.d/metadata_log.xml new file mode 100644 index 000000000000..c1fece21745c --- /dev/null +++ b/tests/integration/test_export_replicated_mt_partition_to_iceberg/configs/config.d/metadata_log.xml @@ -0,0 +1,7 @@ + + + system + iceberg_metadata_log
+ 10 + + diff --git a/tests/integration/test_export_replicated_mt_partition_to_iceberg/configs/users.d/profile.xml b/tests/integration/test_export_replicated_mt_partition_to_iceberg/configs/users.d/profile.xml new file mode 100644 index 000000000000..518f29708929 --- /dev/null +++ b/tests/integration/test_export_replicated_mt_partition_to_iceberg/configs/users.d/profile.xml @@ -0,0 +1,8 @@ + + + + 3 + + + + diff --git a/tests/integration/test_export_replicated_mt_partition_to_iceberg/test.py b/tests/integration/test_export_replicated_mt_partition_to_iceberg/test.py new file mode 100644 index 000000000000..13e9dbf833a3 --- /dev/null +++ b/tests/integration/test_export_replicated_mt_partition_to_iceberg/test.py @@ -0,0 +1,2990 @@ +import io +import json +import logging +import re +import time +from typing import NamedTuple + +import pytest +from avro.datafile import DataFileReader +from avro.io import DatumReader + +from helpers.cluster import ClickHouseCluster +from helpers.export_partition_helpers import ( + first_partition_id, + make_iceberg_s3, + make_rmt, + unique_suffix, + wait_for_exception_count, + wait_for_export_status, + wait_for_export_to_start, +) +from helpers.iceberg_export_stats import ( + assert_exported_stats, + fetch_manifest_entries, +) +from helpers.network import PartitionManager + + +@pytest.fixture(scope="module") +def cluster(): + try: + cluster = ClickHouseCluster(__file__) + cluster.add_instance( + "replica1", + main_configs=[ + "configs/allow_experimental_export_partition.xml", + "configs/config.d/metadata_log.xml", + ], + user_configs=["configs/users.d/profile.xml"], + with_minio=True, + stay_alive=True, + with_zookeeper=True, + keeper_required_feature_flags=["multi_read"], + ) + cluster.add_instance( + "replica2", + main_configs=[ + "configs/allow_experimental_export_partition.xml", + "configs/config.d/metadata_log.xml", + ], + user_configs=["configs/users.d/profile.xml"], + with_minio=True, + stay_alive=True, + with_zookeeper=True, + keeper_required_feature_flags=["multi_read"], + ) + logging.info("Starting cluster...") + cluster.start() + yield cluster + finally: + cluster.shutdown() + + +@pytest.fixture(autouse=True) +def drop_tables_after_test(cluster): + """Drop all tables in the default database after every test. + + Without this, ReplicatedMergeTree tables from completed tests remain alive and keep + running ZooKeeper background threads. With many tables alive simultaneously the + ZooKeeper session becomes overwhelmed and subsequent tests start seeing + operation-timeout / session-expired errors. + """ + yield + for instance_name, instance in cluster.instances.items(): + try: + tables_str = instance.query( + "SELECT name FROM system.tables WHERE database = 'default' FORMAT TabSeparated" + ).strip() + if not tables_str: + continue + for table in tables_str.split("\n"): + table = table.strip() + if table: + instance.query(f"DROP TABLE IF EXISTS default.`{table}` SYNC") + except Exception as e: + logging.warning( + f"drop_tables_after_test: cleanup failed on {instance_name}: {e}" + ) + + +# --------------------------------------------------------------------------- +# Helpers +# --------------------------------------------------------------------------- + +def create_replicated_mt(node, mt_table: str, replica_name: str): + make_rmt(node, mt_table, "id Int64, year Int32", "year", + replica_name=replica_name) + + +def create_iceberg_s3_table(node, iceberg_table: str, if_not_exists: bool = False, + s3_retry_attempts: int = 3): + """Create (or attach to an existing) IcebergS3 table at a per-test MinIO prefix.""" + make_iceberg_s3( + node, iceberg_table, "id Int64, year Int32", + partition_by="year", if_not_exists=if_not_exists, + s3_retry_attempts=s3_retry_attempts, + ) + + +def setup_tables(cluster, mt_table: str, iceberg_table: str, nodes: list | None = None, + s3_retry_attempts: int = 3): + """ + Create the ReplicatedMergeTree table on the given nodes, insert data on the first + node, wait for replication, then create the Iceberg destination table on each node. + + The Iceberg table is created on the first node (which initialises the S3 metadata). + Subsequent nodes attach to the same path with IF NOT EXISTS. + + `nodes` defaults to ["replica1", "replica2"]. + """ + if nodes is None: + nodes = ["replica1", "replica2"] + + instances = [cluster.instances[n] for n in nodes] + primary = instances[0] + + for i, instance in enumerate(instances): + create_replicated_mt(instance, mt_table, nodes[i]) + + primary.query(f"INSERT INTO {mt_table} VALUES (1, 2020), (2, 2020), (3, 2020), (4, 2021)") + for instance in instances[1:]: + instance.query(f"SYSTEM SYNC REPLICA {mt_table}") + + create_iceberg_s3_table(primary, iceberg_table, s3_retry_attempts=s3_retry_attempts) + for instance in instances[1:]: + create_iceberg_s3_table(instance, iceberg_table, if_not_exists=True, + s3_retry_attempts=s3_retry_attempts) + + +# --------------------------------------------------------------------------- +# Tests +# --------------------------------------------------------------------------- + +def test_export_partition_to_iceberg(cluster): + """ + Basic happy path: export a single partition and verify row count and content. + """ + node = cluster.instances["replica1"] + + uid = unique_suffix() + mt_table = f"mt_{uid}" + iceberg_table = f"iceberg_{uid}" + + setup_tables(cluster, mt_table, iceberg_table, nodes=["replica1"]) + + node.query( + f"ALTER TABLE {mt_table} EXPORT PARTITION ID '2020' TO TABLE {iceberg_table}", + settings={"allow_insert_into_iceberg": 1}, + ) + wait_for_export_status(node, mt_table, iceberg_table, "2020", "COMPLETED") + + count = int(node.query(f"SELECT count() FROM {iceberg_table}").strip()) + assert count == 3, f"Expected 3 rows in Iceberg table after export, got {count}" + + result = node.query(f"SELECT id, year FROM {iceberg_table} ORDER BY id").strip() + assert result == "1\t2020\n2\t2020\n3\t2020", ( + f"Unexpected data in Iceberg table:\n{result}" + ) + + +def _destination_paths_has_sync_failed_marker(node, source_table, dest_table, partition_id): + """True when destination_file_paths contains the Keeper sync-failed marker value.""" + result = node.query( + f"SELECT has(arrayFlatten(mapValues(destination_file_paths)), '')" + f" FROM system.replicated_partition_exports" + f" WHERE source_table = '{source_table}'" + f" AND destination_table = '{dest_table}'" + f" AND partition_id = '{partition_id}'" + ).strip() + return result == "1" + + +def wait_for_destination_paths_sync_failed_marker( + node, source_table, dest_table, partition_id, expect_marker, timeout=90, poll_interval=0.5 +): + """Wait until destination_file_paths does/does not contain the sync-failed marker. + + The in-memory mirror refreshes on the manifest-updater poll (~30s), so the + default timeout allows at least one full cycle plus headroom. + """ + start_time = time.time() + last = None + while time.time() - start_time < timeout: + last = _destination_paths_has_sync_failed_marker( + node, source_table, dest_table, partition_id + ) + if last == expect_marker: + return + time.sleep(poll_interval) + + raise TimeoutError( + f"destination_file_paths sync-failed marker did not become {expect_marker}" + f" within {timeout}s (last={last})" + ) + + +def test_export_two_partitions_to_iceberg(cluster): + """ + Export two partitions in a single ALTER TABLE statement and verify that both + land in the Iceberg table with correct row counts. + """ + node = cluster.instances["replica1"] + + uid = unique_suffix() + mt_table = f"mt_{uid}" + iceberg_table = f"iceberg_{uid}" + + setup_tables(cluster, mt_table, iceberg_table, nodes=["replica1"]) + + node.query( + f""" + ALTER TABLE {mt_table} + EXPORT PARTITION ID '2020' TO TABLE {iceberg_table}, + EXPORT PARTITION ID '2021' TO TABLE {iceberg_table} + """, + settings={"allow_insert_into_iceberg": 1}, + ) + + wait_for_export_status(node, mt_table, iceberg_table, "2020", "COMPLETED") + wait_for_export_status(node, mt_table, iceberg_table, "2021", "COMPLETED") + + count_2020 = int(node.query(f"SELECT count() FROM {iceberg_table} WHERE year = 2020").strip()) + count_2021 = int(node.query(f"SELECT count() FROM {iceberg_table} WHERE year = 2021").strip()) + + assert count_2020 == 3, f"Expected 3 rows for year=2020, got {count_2020}" + assert count_2021 == 1, f"Expected 1 row for year=2021, got {count_2021}" + + +def test_export_partition_all_to_iceberg(cluster): + """ + `ALTER TABLE ... EXPORT PARTITION ALL TO TABLE ...` schedules every active partition + in one statement and exercises the Iceberg-specific destination compatibility checks + (which are repeated per sub-call inside the loop). + """ + node = cluster.instances["replica1"] + + uid = unique_suffix() + mt_table = f"mt_{uid}" + iceberg_table = f"iceberg_{uid}" + + setup_tables(cluster, mt_table, iceberg_table, nodes=["replica1"]) + + node.query( + f"ALTER TABLE {mt_table} EXPORT PARTITION ALL TO TABLE {iceberg_table}", + settings={"allow_insert_into_iceberg": 1}, + ) + + wait_for_export_status(node, mt_table, iceberg_table, "2020", "COMPLETED") + wait_for_export_status(node, mt_table, iceberg_table, "2021", "COMPLETED") + + count_2020 = int(node.query(f"SELECT count() FROM {iceberg_table} WHERE year = 2020").strip()) + count_2021 = int(node.query(f"SELECT count() FROM {iceberg_table} WHERE year = 2021").strip()) + + assert count_2020 == 3, f"Expected 3 rows for year=2020, got {count_2020}" + assert count_2021 == 1, f"Expected 1 row for year=2021, got {count_2021}" + + +def test_failure_is_logged_in_system_table(cluster): + """ + When a part export fails with a non-retryable error the export must be marked + FAILED in system.replicated_partition_exports with a non-zero exception_count. + + Uses the export_part_non_retryable_throw failpoint (throws BAD_ARGUMENTS, a + denylisted code) so the task fails fast without consuming any timeout budget. + """ + node = cluster.instances["replica1"] + + uid = unique_suffix() + mt_table = f"mt_{uid}" + iceberg_table = f"iceberg_{uid}" + + setup_tables(cluster, mt_table, iceberg_table, nodes=["replica1"]) + + node.query("SYSTEM ENABLE FAILPOINT export_part_non_retryable_throw") + try: + node.query( + f"ALTER TABLE {mt_table} EXPORT PARTITION ID '2020' TO TABLE {iceberg_table}", + settings={"allow_insert_into_iceberg": 1}, + ) + + # short timeout to exercise the fast fail path for non retryable errors + wait_for_export_status(node, mt_table, iceberg_table, "2020", "FAILED", timeout=20) + finally: + node.query("SYSTEM DISABLE FAILPOINT export_part_non_retryable_throw") + + status = node.query( + f""" + SELECT status FROM system.replicated_partition_exports + WHERE source_table = '{mt_table}' + AND destination_table = '{iceberg_table}' + AND partition_id = '2020' + """ + ).strip() + assert status == "FAILED", f"Expected FAILED status, got: {status!r}" + + exception_count = int(node.query( + f""" + SELECT any(exception_count) FROM system.replicated_partition_exports + WHERE source_table = '{mt_table}' + AND destination_table = '{iceberg_table}' + AND partition_id = '2020' + """ + ).strip()) + assert exception_count > 0, "Expected non-zero exception_count in system.replicated_partition_exports" + + count = int(node.query(f"SELECT count() FROM {iceberg_table}").strip()) + assert count == 0, f"Expected 0 rows in Iceberg table after a failed export, got {count}" + + +def test_inject_short_living_failures(cluster): + """ + Transient S3 failures must not prevent the export from completing: after the + network is restored the export should retry and eventually land COMPLETED. + """ + node = cluster.instances["replica1"] + minio_ip = cluster.minio_ip + minio_port = cluster.minio_port + + uid = unique_suffix() + mt_table = f"mt_{uid}" + iceberg_table = f"iceberg_{uid}" + + setup_tables(cluster, mt_table, iceberg_table, nodes=["replica1"], + s3_retry_attempts=1) + + node.query(f"SYSTEM STOP MOVES {mt_table}") + + node.query(f"ALTER TABLE {mt_table} EXPORT PARTITION ID '2020' TO TABLE {iceberg_table} SETTINGS allow_insert_into_iceberg = 1") + + with PartitionManager() as pm: + pm.add_rule({ + "instance": node, + "destination": node.ip_address, + "protocol": "tcp", + "source_port": minio_port, + "action": "REJECT --reject-with tcp-reset", + }) + pm.add_rule({ + "instance": node, + "destination": minio_ip, + "protocol": "tcp", + "destination_port": minio_port, + "action": "REJECT --reject-with tcp-reset", + }) + + node.query(f"SYSTEM START MOVES {mt_table}") + + # Let at least one retry happen before restoring the network. + time.sleep(15) + + wait_for_export_status(node, mt_table, iceberg_table, "2020", "COMPLETED") + + count = int(node.query(f"SELECT count() FROM {iceberg_table} WHERE year = 2020").strip()) + assert count == 3, f"Expected 3 rows after retry, got {count}" + + status = node.query( + f""" + SELECT status FROM system.replicated_partition_exports + WHERE source_table = '{mt_table}' + AND destination_table = '{iceberg_table}' + AND partition_id = '2020' + """ + ).strip() + assert status == "COMPLETED", f"Expected COMPLETED in system table, got: {status!r}" + + exception_count = int(node.query( + f""" + SELECT exception_count FROM system.replicated_partition_exports + WHERE source_table = '{mt_table}' + AND destination_table = '{iceberg_table}' + AND partition_id = '2020' + """ + ).strip()) + assert exception_count >= 1, "Expected at least one transient exception to be recorded" + + +def test_export_partition_retryable_error_killed_on_timeout(cluster): + """ + A retryable part-export error (here FAULT_INJECTED via export_part_retryable_throw) + must NOT fail the task on a retry budget: there is no retry budget anymore, so the + part keeps retrying until the absolute task timeout fires and the task is KILLED. + """ + node = cluster.instances["replica1"] + + uid = unique_suffix() + mt_table = f"mt_{uid}" + iceberg_table = f"iceberg_{uid}" + + setup_tables(cluster, mt_table, iceberg_table, nodes=["replica1"]) + + node.query("SYSTEM ENABLE FAILPOINT export_part_retryable_throw") + try: + # Under the old budget model a small retry budget would fail the task after the + # first retry. With the new model there is no budget and only the 5s timeout fails it. + node.query( + f"ALTER TABLE {mt_table} EXPORT PARTITION ID '2020' TO TABLE {iceberg_table}" + f" SETTINGS export_merge_tree_partition_task_timeout_seconds = 5," + f" allow_insert_into_iceberg = 1" + ) + + # Give the scheduler time to attempt and fail the part several times. The old + # budget would already have transitioned the task to FAILED by now. + time.sleep(15) + status = node.query( + f"SELECT status FROM system.replicated_partition_exports" + f" WHERE source_table = '{mt_table}'" + f" AND destination_table = '{iceberg_table}'" + f" AND partition_id = '2020'" + ).strip() + assert status != "FAILED", ( + f"Retryable failures must not fail the task on a budget, got status {status!r}" + ) + + # The timeout (5s) is past; KILLED fires on the next manifest-updater poll cycle. + wait_for_export_status( + node, mt_table, iceberg_table, "2020", "KILLED", timeout=90 + ) + finally: + node.query("SYSTEM DISABLE FAILPOINT export_part_retryable_throw") + + exception_count = int(node.query( + f"SELECT any(exception_count) FROM system.replicated_partition_exports" + f" WHERE source_table = '{mt_table}'" + f" AND destination_table = '{iceberg_table}'" + f" AND partition_id = '2020'" + ).strip()) + assert exception_count > 0, "Expected at least one retryable exception to be recorded" + + count = int(node.query(f"SELECT count() FROM {iceberg_table}").strip()) + assert count == 0, f"Expected 0 rows in Iceberg table after a killed export, got {count}" + + +def test_export_partition_retryable_error_recovers_after_failpoint_cleared(cluster): + """ + A retryable part-export error must keep the task PENDING (not FAILED) while the + failure persists, applying a per-replica back-off between attempts. Once the + failure clears the export completes successfully — proving the back-off only + spaces retries out and never permanently blocks progress. + """ + node = cluster.instances["replica1"] + + uid = unique_suffix() + mt_table = f"mt_{uid}" + iceberg_table = f"iceberg_{uid}" + + setup_tables(cluster, mt_table, iceberg_table, nodes=["replica1"]) + + node.query("SYSTEM ENABLE FAILPOINT export_part_retryable_throw") + try: + node.query( + f"ALTER TABLE {mt_table} EXPORT PARTITION ID '2020' TO TABLE {iceberg_table}" + f" SETTINGS export_merge_tree_partition_retry_initial_backoff_seconds = 1," + f" export_merge_tree_partition_retry_max_backoff_seconds = 2," + f" allow_insert_into_iceberg = 1" + ) + + # Wait until at least one retryable failure has been recorded; the task must + # still be PENDING (retrying), never FAILED. + wait_for_exception_count(node, mt_table, iceberg_table, "2020", + min_exception_count=1, timeout=60) + status = node.query( + f"SELECT status FROM system.replicated_partition_exports" + f" WHERE source_table = '{mt_table}'" + f" AND destination_table = '{iceberg_table}'" + f" AND partition_id = '2020'" + ).strip() + assert status == "PENDING", ( + f"Retryable failures must keep the task PENDING, got status {status!r}" + ) + finally: + node.query("SYSTEM DISABLE FAILPOINT export_part_retryable_throw") + + # With the failpoint cleared the next retry succeeds and the export completes. + wait_for_export_status(node, mt_table, iceberg_table, "2020", "COMPLETED", timeout=90) + + count = int(node.query(f"SELECT count() FROM {iceberg_table} WHERE year = 2020").strip()) + assert count == 3, f"Expected 3 rows after recovery, got {count}" + + +def test_export_partition_local_backoff_does_not_block_other_replica(cluster): + """ + Back-off is per-replica and in-memory: a part that one replica keeps failing on + (and therefore puts into its local back-off) must NOT be prevented from being + exported by another replica. This is the whole reason the back-off is local + rather than distributed in ZooKeeper. + + replica1 is given a persistent *retryable* failure (export_part_retryable_throw) + and is the only replica scheduling at first (moves are stopped on replica2). Once + replica1 has recorded a failure and a local back-off entry, replica2's scheduler + is enabled. Because the failpoint stays active on replica1 the whole time, the + only way the export can reach COMPLETED is replica2 picking up the very part that + replica1 keeps failing — proving the back-off does not leak across replicas. + """ + replica1 = cluster.instances["replica1"] + replica2 = cluster.instances["replica2"] + + uid = unique_suffix() + mt_table = f"mt_{uid}" + iceberg_table = f"iceberg_{uid}" + + setup_tables(cluster, mt_table, iceberg_table, nodes=["replica1", "replica2"]) + + # Phase 1: only replica1 schedules. Stop the export scheduler on replica2 so the + # part is guaranteed to be attempted (and fail) on replica1 first. + replica2.query(f"SYSTEM STOP MOVES {mt_table}") + + replica1.query("SYSTEM ENABLE FAILPOINT export_part_retryable_throw") + try: + replica1.query( + f"ALTER TABLE {mt_table} EXPORT PARTITION ID '2020' TO TABLE {iceberg_table}" + f" SETTINGS export_merge_tree_partition_retry_initial_backoff_seconds = 1," + f" export_merge_tree_partition_retry_max_backoff_seconds = 2," + f" allow_insert_into_iceberg = 1" + ) + + # replica1 attempts the part, fails (retryable), and enters local back-off. + # The task must stay PENDING — there is no retry budget to fail it. + wait_for_exception_count(replica1, mt_table, iceberg_table, "2020", + min_exception_count=1, timeout=60) + + wait_for_export_status(replica1, mt_table, iceberg_table, "2020", "PENDING", timeout=60) + + # The back-off entry must be observable on replica1 (the failing replica). + deadline = time.time() + 90 + backoff_replica1 = "0" + while time.time() < deadline: + backoff_replica1 = replica1.query( + f"SELECT length(local_backoff_per_part) FROM system.replicated_partition_exports" + f" WHERE source_table = '{mt_table}'" + f" AND destination_table = '{iceberg_table}'" + f" AND partition_id = '2020'" + ).strip() + if backoff_replica1 not in ("", "0"): + break + time.sleep(0.5) + assert backoff_replica1 not in ("", "0"), ( + "Expected replica1 to carry a local back-off entry for the failing part, " + f"got {backoff_replica1!r}" + ) + + # ... and it must NOT have leaked to replica2, which never attempted the part. + # This is the core assertion: local back-off state is not shared across replicas. + backoff_replica2 = replica2.query( + f"SELECT length(local_backoff_per_part) FROM system.replicated_partition_exports" + f" WHERE source_table = '{mt_table}'" + f" AND destination_table = '{iceberg_table}'" + f" AND partition_id = '2020'" + ).strip() + + assert backoff_replica2 in ("", "0"), ( + f"replica2 must not carry replica1's local back-off, got {backoff_replica2!r}" + ) + + # Phase 2: enable replica2's scheduler. replica1 keeps failing (the failpoint + # is still active), so completion can only come from replica2 exporting the + # part that replica1 is backing off on. + replica2.query(f"SYSTEM START MOVES {mt_table}") + + wait_for_export_status(replica2, mt_table, iceberg_table, "2020", "COMPLETED", timeout=60) + finally: + replica1.query("SYSTEM DISABLE FAILPOINT export_part_retryable_throw") + + count = int(replica2.query(f"SELECT count() FROM {iceberg_table} WHERE year = 2020").strip()) + assert count == 3, f"Expected 3 rows after replica2 completed the export, got {count}" + + +def test_export_partition_scheduler_skipped_when_moves_stopped(cluster): + """ + Verify that selectPartsToExport() skips the scheduler entirely when moves + are stopped (moves_blocker guard at the top of the function). + + No ZK locks are acquired and no background tasks are submitted, so the + Iceberg table must remain empty across multiple scheduler cycles. Once moves + are re-enabled the export completes and rows appear in the Iceberg table. + """ + node = cluster.instances["replica1"] + + uid = unique_suffix() + mt_table = f"mt_{uid}" + iceberg_table = f"iceberg_{uid}" + + setup_tables(cluster, mt_table, iceberg_table, nodes=["replica1"]) + + node.query(f"SYSTEM STOP MOVES {mt_table}") + + node.query( + f"ALTER TABLE {mt_table} EXPORT PARTITION ID '2020' TO TABLE {iceberg_table}", + settings={"allow_insert_into_iceberg": 1}, + ) + + wait_for_export_to_start(node, mt_table, iceberg_table, "2020") + + # Wait for several scheduler cycles (each fires every 5 s). + # If the guard is absent the scheduler would run and rows would appear in the Iceberg table. + time.sleep(12) + + status = node.query( + f"SELECT status FROM system.replicated_partition_exports" + f" WHERE source_table = '{mt_table}' AND destination_table = '{iceberg_table}'" + f" AND partition_id = '2020'" + ).strip() + + assert status == "PENDING", f"Expected PENDING while moves are stopped, got '{status}'" + + count = int(node.query(f"SELECT count() FROM {iceberg_table} WHERE year = 2020").strip()) + assert count == 0, f"Expected 0 rows in Iceberg table while scheduler is skipped, got {count}" + + node.query(f"SYSTEM START MOVES {mt_table}") + + wait_for_export_status(node, mt_table, iceberg_table, "2020", "COMPLETED") + + count = int(node.query(f"SELECT count() FROM {iceberg_table} WHERE year = 2020").strip()) + assert count == 3, f"Expected 3 rows in Iceberg table after export completed, got {count}" + + +def test_export_partition_resumes_after_stop_moves(cluster): + """ + Verify that SYSTEM STOP MOVES before EXPORT PARTITION does not permanently + orphan the ZooKeeper part lock for Iceberg destinations. + + When moves are stopped the scheduler still picks parts up and submits them to + the background executor, but ExportPartTask::isCancelled() returns true (via + moves_blocker), causing QUERY_WAS_CANCELLED before any data is written. The + fix in handlePartExportFailure must release the ZK lock so the part is retried + once moves are restarted. + """ + node = cluster.instances["replica1"] + + uid = unique_suffix() + mt_table = f"mt_{uid}" + iceberg_table = f"iceberg_{uid}" + + setup_tables(cluster, mt_table, iceberg_table, nodes=["replica1"]) + + node.query(f"SYSTEM STOP MOVES {mt_table}") + + node.query( + f"ALTER TABLE {mt_table} EXPORT PARTITION ID '2020' TO TABLE {iceberg_table}" + f" SETTINGS allow_insert_into_iceberg = 1" + ) + + wait_for_export_to_start(node, mt_table, iceberg_table, "2020") + + # Give the scheduler enough time to attempt (and cancel) the part task at least once. + time.sleep(5) + + status = node.query( + f"SELECT status FROM system.replicated_partition_exports" + f" WHERE source_table = '{mt_table}' AND destination_table = '{iceberg_table}'" + f" AND partition_id = '2020'" + ).strip() + assert status == "PENDING", f"Expected PENDING while moves are stopped, got '{status}'" + + count = int(node.query(f"SELECT count() FROM {iceberg_table} WHERE year = 2020").strip()) + assert count == 0, f"Expected 0 rows in Iceberg table while moves are stopped, got {count}" + + node.query(f"SYSTEM START MOVES {mt_table}") + + wait_for_export_status(node, mt_table, iceberg_table, "2020", "COMPLETED") + + count = int(node.query(f"SELECT count() FROM {iceberg_table} WHERE year = 2020").strip()) + assert count == 3, f"Expected 3 rows in Iceberg table after export completed, got {count}" + + +def test_export_partition_resumes_after_stop_moves_during_export(cluster): + """ + Verify that SYSTEM STOP MOVES issued while an Iceberg export is actively + retrying (S3 blocked) does not permanently orphan the ZooKeeper part lock. + """ + node = cluster.instances["replica1"] + minio_ip = cluster.minio_ip + minio_port = cluster.minio_port + + uid = unique_suffix() + mt_table = f"mt_{uid}" + iceberg_table = f"iceberg_{uid}" + + setup_tables(cluster, mt_table, iceberg_table, nodes=["replica1"]) + + node.query(f"SYSTEM STOP MOVES {mt_table}") + + node.query( + f"ALTER TABLE {mt_table} EXPORT PARTITION ID '2020' TO TABLE {iceberg_table}" + f" SETTINGS allow_insert_into_iceberg = 1") + + wait_for_export_to_start(node, mt_table, iceberg_table, "2020") + + with PartitionManager() as pm: + pm.add_rule({ + "instance": node, + "destination": node.ip_address, + "protocol": "tcp", + "source_port": minio_port, + "action": "REJECT --reject-with tcp-reset", + }) + pm.add_rule({ + "instance": node, + "destination": minio_ip, + "protocol": "tcp", + "destination_port": minio_port, + "action": "REJECT --reject-with tcp-reset", + }) + + node.query(f"SYSTEM STOP MOVES {mt_table}") + + time.sleep(3) + + status = node.query( + f"SELECT status FROM system.replicated_partition_exports" + f" WHERE source_table = '{mt_table}' AND destination_table = '{iceberg_table}'" + f" AND partition_id = '2020'" + ).strip() + assert status == "PENDING", ( + f"Expected PENDING while moves are stopped and S3 is blocked, got '{status}'" + ) + + node.query(f"SYSTEM START MOVES {mt_table}") + + # MinIO is now unblocked; the next scheduler cycle should succeed. + wait_for_export_status(node, mt_table, iceberg_table, "2020", "COMPLETED") + + count = int(node.query(f"SELECT count() FROM {iceberg_table} WHERE year = 2020").strip()) + assert count == 3, f"Expected 3 rows in Iceberg table after export completed, got {count}" + + +def test_partition_transform_compatibility_accepted(cluster): + """ + Verify that EXPORT PARTITION is accepted (no BAD_ARGUMENTS) for every + supported transform when the MergeTree and Iceberg partition specs match. + + Cases covered: + 1. Compound identity (year, region), exported to a spec that lists the fields in reverse order + 2. Year transform – toYearNumSinceEpoch(event_date) + 3. Month transform – toMonthNumSinceEpoch(event_date) + 4. truncate[4] – icebergTruncate(4, category) + 5. bucket[8] – icebergBucket(8, user_id) + 6. Compound mixed – (toYearNumSinceEpoch(event_date), icebergBucket(16, user_id)) + """ + node = cluster.instances["replica1"] + uid = unique_suffix() + + def check_accepted(mt, iceberg, description): + pid = first_partition_id(node, mt) + node.query( + f"ALTER TABLE {mt} EXPORT PARTITION ID '{pid}' TO TABLE {iceberg}", + settings={"allow_insert_into_iceberg": 1}, + ) + return pid + + # 1. Compound identity, with the destination listing the fields in the opposite order: the + # source key pins both columns, so the partition is single-valued for either field order. + cols = "id Int64, year Int32, region String" + t = f"mt_acc_1_{uid}"; i = f"iceberg_acc_1_{uid}" + make_rmt(node, t, cols, "(year, region)") + node.query(f"INSERT INTO {t} VALUES (1, 2023, 'EU')") + make_iceberg_s3(node, i, cols, "(region, year)") + pid = check_accepted(t, i, "compound identity (year, region)") + wait_for_export_status(node, t, i, pid, "COMPLETED") + count = int(node.query(f"SELECT count() FROM {i}").strip()) + assert count == 1, f"[compound identity (year, region)] Expected 1 row in Iceberg table, got {count}" + result = node.query(f"SELECT id, year, region FROM {i}").strip() + assert result == "1\t2023\tEU", f"[compound identity (year, region)] Unexpected exported data:\n{result}" + + # 2. Year transform + cols = "id Int64, event_date Date" + t = f"mt_acc_2_{uid}"; i = f"iceberg_acc_2_{uid}" + make_rmt(node, t, cols, "toYearNumSinceEpoch(event_date)") + node.query(f"INSERT INTO {t} VALUES (1, '2020-06-15')") + make_iceberg_s3(node, i, cols, "toYearNumSinceEpoch(event_date)") + check_accepted(t, i, "year transform") + + # 3. Month transform + cols = "id Int64, event_date Date" + t = f"mt_acc_3_{uid}"; i = f"iceberg_acc_3_{uid}" + make_rmt(node, t, cols, "toMonthNumSinceEpoch(event_date)") + node.query(f"INSERT INTO {t} VALUES (1, '2020-06-15')") + make_iceberg_s3(node, i, cols, "toMonthNumSinceEpoch(event_date)") + check_accepted(t, i, "month transform") + + # 4. truncate[4] + cols = "id Int64, category String" + t = f"mt_acc_4_{uid}"; i = f"iceberg_acc_4_{uid}" + make_rmt(node, t, cols, "icebergTruncate(4, category)") + node.query(f"INSERT INTO {t} VALUES (1, 'clickhouse')") + make_iceberg_s3(node, i, cols, "icebergTruncate(4, category)") + check_accepted(t, i, "truncate[4]") + + # 5. bucket[8] + cols = "id Int64, user_id Int64" + t = f"mt_acc_5_{uid}"; i = f"iceberg_acc_5_{uid}" + make_rmt(node, t, cols, "icebergBucket(8, user_id)") + node.query(f"INSERT INTO {t} VALUES (1, 42)") + make_iceberg_s3(node, i, cols, "icebergBucket(8, user_id)") + check_accepted(t, i, "bucket[8]") + + # 6. Compound mixed: year(event_date) + bucket[16](user_id) + cols = "id Int64, event_date Date, user_id Int64" + t = f"mt_acc_6_{uid}"; i = f"iceberg_acc_6_{uid}" + make_rmt(node, t, cols, "(toYearNumSinceEpoch(event_date), icebergBucket(16, user_id))") + node.query(f"INSERT INTO {t} VALUES (1, '2021-03-01', 99)") + make_iceberg_s3(node, i, cols, "(toYearNumSinceEpoch(event_date), icebergBucket(16, user_id))") + check_accepted(t, i, "compound year+bucket[16]") + + +def test_partition_transform_compatibility_rejected(cluster): + """ + Verify that partition specs that cannot be exported are rejected with BAD_ARGUMENTS. + + Acceptance is data-dependent: a source partition must map to a single Iceberg partition. The + mismatch cases below therefore use data that makes the source partition span several + destination partitions (a single-row partition would be trivially single-valued and accepted). + + Cases covered: + 1. Transform mismatch on the same column: year-transform source vs identity destination, where + the year partition contains several distinct dates. + 2. Bucket count mismatch: bucket[8] vs bucket[16] (bucket is non-monotonic, always structural). + 3. Truncate width mismatch: truncate[4] source vs truncate[8] destination, with values sharing + the 4-char prefix but differing within the first 8 chars. + 4. Unsupported MergeTree expression (intDiv) vs identity, with one bucket spanning several years. + 5. Destination partitions by a column that is not in the source partition key. + """ + node = cluster.instances["replica1"] + uid = unique_suffix() + + def assert_rejected(mt, iceberg, description): + pid = first_partition_id(node, mt) + error = node.query_and_get_error( + f"ALTER TABLE {mt} EXPORT PARTITION ID '{pid}' TO TABLE {iceberg}", + settings={"allow_insert_into_iceberg": 1}, + ) + assert "BAD_ARGUMENTS" in error, ( + f"[{description}] Expected BAD_ARGUMENTS, got: {error!r}" + ) + + # 1. Transform mismatch: MergeTree year-transform, Iceberg identity on same Date col + cols = "id Int64, event_date Date" + t = f"mt_rej_1_{uid}"; i = f"iceberg_rej_1_{uid}" + make_rmt(node, t, cols, "toYearNumSinceEpoch(event_date)") + node.query(f"INSERT INTO {t} VALUES (1, '2020-01-01'), (2, '2020-12-31')") + make_iceberg_s3(node, i, cols, "event_date") # identity, not year-transform + assert_rejected(t, i, "year-transform source vs identity destination") + + # 2. Bucket count mismatch: bucket[8] vs bucket[16] + cols = "id Int64, user_id Int64" + t = f"mt_rej_2_{uid}"; i = f"iceberg_rej_2_{uid}" + make_rmt(node, t, cols, "icebergBucket(8, user_id)") + node.query(f"INSERT INTO {t} VALUES (1, 42)") + make_iceberg_s3(node, i, cols, "icebergBucket(16, user_id)") + assert_rejected(t, i, "bucket[8] vs bucket[16]") + + # 3. Truncate width mismatch: values share the 4-char prefix but differ within 8 chars. + cols = "id Int64, category String" + t = f"mt_rej_3_{uid}"; i = f"iceberg_rej_3_{uid}" + make_rmt(node, t, cols, "icebergTruncate(4, category)") + node.query(f"INSERT INTO {t} VALUES (1, 'clickhouse'), (2, 'clickfmt')") + make_iceberg_s3(node, i, cols, "icebergTruncate(8, category)") + assert_rejected(t, i, "truncate[4] source vs truncate[8] destination") + + # 4. Unsupported MergeTree expression vs identity: one intDiv bucket spans several years. + cols = "id Int64, year Int32" + t = f"mt_rej_4_{uid}"; i = f"iceberg_rej_4_{uid}" + make_rmt(node, t, cols, "intDiv(year, 100)") + node.query(f"INSERT INTO {t} VALUES (1, 2000), (2, 2099)") + make_iceberg_s3(node, i, cols, "year") + assert_rejected(t, i, "intDiv source vs identity destination") + + # 5. Destination partitions by a column absent from the source partition key. + cols = "id Int64, year Int32" + t = f"mt_rej_5_{uid}"; i = f"iceberg_rej_5_{uid}" + make_rmt(node, t, cols, "year") + node.query(f"INSERT INTO {t} VALUES (1, 2020)") + make_iceberg_s3(node, i, cols, "id") # identity on id, which the source does not partition by + assert_rejected(t, i, "destination partitions by a non-source-key column") + + +def test_partition_key_compatibility_check(cluster): + """ + Verify that EXPORT PARTITION throws BAD_ARGUMENTS synchronously when the + MergeTree partition key does not match the Iceberg table's partition spec, + and is accepted without error when the destination is satisfiable. + + Three cases: + 1. Column mismatch – MergeTree PARTITION BY year, Iceberg PARTITION BY id (must be rejected) + 2. Unpartitioned dst – MergeTree PARTITION BY year, Iceberg unpartitioned (accepted: the source is + flattened into the single empty Iceberg partition) + 3. Matching keys – both PARTITION BY year (must be accepted) + """ + node = cluster.instances["replica1"] + + uid = unique_suffix() + mt_table = f"mt_{uid}" + + create_replicated_mt(node, mt_table, "replica1") + node.query(f"INSERT INTO {mt_table} VALUES (1, 2020), (2, 2020), (3, 2021)") + node.query(f"SYSTEM SYNC REPLICA {mt_table}") + + # --- Case 1: Iceberg partitioned by 'id' but MergeTree by 'year' --- + iceberg_col_mismatch = f"iceberg_col_mismatch_{uid}" + node.query( + f""" + CREATE TABLE {iceberg_col_mismatch} + (id Int64, year Int32) + ENGINE = IcebergS3( + 'http://minio1:9001/root/data/{iceberg_col_mismatch}/', + 'minio', + 'ClickHouse_Minio_P@ssw0rd' + ) + PARTITION BY id SETTINGS s3_retry_attempts = 3 + """ + ) + error = node.query_and_get_error( + f"ALTER TABLE {mt_table} EXPORT PARTITION ID '2020' TO TABLE {iceberg_col_mismatch}", + settings={"allow_insert_into_iceberg": 1}, + ) + assert "BAD_ARGUMENTS" in error, ( + f"Expected BAD_ARGUMENTS for partition column mismatch, got: {error!r}" + ) + + # --- Case 2: Iceberg unpartitioned, MergeTree PARTITION BY year --- + # An unpartitioned Iceberg table has a single (empty) partition, so a partitioned source is + # flattened into it and the export is accepted; the partition-column values survive as data. + iceberg_unpartitioned = f"iceberg_unpartitioned_{uid}" + node.query( + f""" + CREATE TABLE {iceberg_unpartitioned} + (id Int64, year Int32) + ENGINE = IcebergS3( + 'http://minio1:9001/root/data/{iceberg_unpartitioned}/', + 'minio', + 'ClickHouse_Minio_P@ssw0rd' + ) + SETTINGS s3_retry_attempts = 3 + """ + ) + node.query( + f"ALTER TABLE {mt_table} EXPORT PARTITION ID '2020' TO TABLE {iceberg_unpartitioned}", + settings={"allow_insert_into_iceberg": 1}, + ) + wait_for_export_status(node, mt_table, iceberg_unpartitioned, "2020", "COMPLETED") + count = int(node.query(f"SELECT count() FROM {iceberg_unpartitioned}").strip()) + assert count == 2, f"Expected 2 rows in unpartitioned Iceberg table after export, got {count}" + result = node.query(f"SELECT id, year FROM {iceberg_unpartitioned} ORDER BY id").strip() + assert result == "1\t2020\n2\t2020", f"Unexpected data in unpartitioned Iceberg table:\n{result}" + + # --- Case 3: Matching partition keys (both PARTITION BY year) --- + iceberg_match = f"iceberg_match_{uid}" + node.query( + f""" + CREATE TABLE {iceberg_match} + (id Int64, year Int32) + ENGINE = IcebergS3( + 'http://minio1:9001/root/data/{iceberg_match}/', + 'minio', + 'ClickHouse_Minio_P@ssw0rd' + ) + PARTITION BY year SETTINGS s3_retry_attempts = 3 + """ + ) + # Should not raise — the check passes so the export is accepted synchronously + node.query( + f"ALTER TABLE {mt_table} EXPORT PARTITION ID '2020' TO TABLE {iceberg_match}", + settings={"allow_insert_into_iceberg": 1}, + ) + + +def test_partition_transform_equivalence_gate(cluster): + """ + The Iceberg partition-compatibility gate accepts a source partition key whose transform is + equivalent to (or finer than) the destination Iceberg transform when the exported partition is + provably single-valued for every destination field, and rejects it otherwise. Accept cases are + verified end-to-end (data + metadata); reject cases must throw BAD_ARGUMENTS synchronously. + """ + node = cluster.instances["replica1"] + dt = "id Int64, event_time DateTime" + yr = "id Int64, year Int32, region String" + + cases = [ + # toDate -> day: rows within one day map to a single Iceberg day partition. + {"name": "todate_day", "columns": dt, "source_key": "toDate(event_time)", + "dest_key": "toRelativeDayNum(event_time)", + "rows": "(1, '2024-03-05 01:00:00'), (2, '2024-03-05 20:00:00')", "expect_ok": True}, + # toYYYYMM -> month: different days of the same month map to a single month partition. + {"name": "toyyyymm_month", "columns": dt, "source_key": "toYYYYMM(event_time)", + "dest_key": "toMonthNumSinceEpoch(event_time)", + "rows": "(1, '2024-03-01 00:00:00'), (2, '2024-03-20 00:00:00')", "expect_ok": True}, + # toStartOfHour -> hour. + {"name": "startofhour_hour", "columns": dt, "source_key": "toStartOfHour(event_time)", + "dest_key": "toRelativeHourNum(event_time)", + "rows": "(1, '2024-03-05 12:00:00'), (2, '2024-03-05 12:59:00')", "expect_ok": True}, + # Finer source (day + country) into a day-partitioned destination: extra column allowed. + {"name": "finer_day", "columns": "id Int64, event_time DateTime, country String", + "source_key": "(toDate(event_time), country)", "dest_key": "toRelativeDayNum(event_time)", + "rows": "(1, '2024-03-05 01:00:00', 'US'), (2, '2024-03-05 20:00:00', 'US')", + "expect_ok": True}, + # Compound field order reversed: matching is by column; the destination defines tuple order. + {"name": "reversed_order", "columns": yr, "source_key": "(year, region)", + "dest_key": "(region, year)", "rows": "(1, 2020, 'EU')", "expect_ok": True, + "verify": [("region", "region"), ("year", "year")]}, + # Superset source: (year, region) into a year-only destination is finer, so accepted. + {"name": "superset", "columns": yr, "source_key": "(year, region)", "dest_key": "year", + "rows": "(1, 2020, 'EU')", "expect_ok": True, "verify": [("year", "year")]}, + # Coarser source: a month partition spans several days, so it cannot map to one day. + {"name": "coarser_day", "columns": dt, "source_key": "toYYYYMM(event_time)", + "dest_key": "toRelativeDayNum(event_time)", + "rows": "(1, '2024-03-01 00:00:00'), (2, '2024-03-20 00:00:00')", "expect_ok": False}, + # A hash is never monotonic, so min/max cannot prove anything about it, but an identity source key + # pins k within the partition and a bucket of a single value is a single bucket. + {"name": "bucket_from_identity_source", "columns": "id Int64, k Int64", "source_key": "k", + "dest_key": "icebergBucket(8, k)", "rows": "(1, 10), (2, 10)", "expect_ok": True, + "verify": [("k", "icebergBucket(8, k)")]}, + # The same bucket destination over a source key that does not pin k: nothing proves the rows of one + # source partition hash into the same bucket. + {"name": "bucket_needs_structural", "columns": "id Int64, k Int64", + "source_key": "intDiv(k, 100)", "dest_key": "icebergBucket(8, k)", + "rows": "(1, 10), (2, 20)", "expect_ok": False}, + # Identical expressions on a Nullable column: accepted structurally. The min/max proof refuses + # Nullable (a NULL forms its own destination partition and the endpoints cannot rule it out), + # so this only passes because the source already groups by exactly this transform. DateTime64(6) + # round-trips through the Iceberg schema unchanged, which the structural type check requires. + {"name": "nullable_exact_day", "columns": "id Int64, event_time Nullable(DateTime64(6))", + "source_key": "toRelativeDayNum(event_time)", "dest_key": "toRelativeDayNum(event_time)", + "rows": "(1, '2024-03-05 01:00:00'), (2, '2024-03-05 20:00:00')", + "source_settings": "allow_nullable_key = 1", "expect_ok": True}, + # Same, for identity, which is exempt from the structural type check. + {"name": "nullable_exact_identity", "columns": "id Int64, k Nullable(Int64)", + "source_key": "k", "dest_key": "k", "rows": "(1, 10), (2, 10)", + "source_settings": "allow_nullable_key = 1", "expect_ok": True, + "verify": [("k", "k")]}, + # A Nullable column without identical expressions falls to the min/max proof, which cannot see + # NULLs, so it is rejected. + {"name": "nullable_no_match", "columns": "id Int64, event_time Nullable(DateTime64(6))", + "source_key": "toYYYYMM(event_time)", "dest_key": "toRelativeDayNum(event_time)", + "rows": "(1, '2024-03-05 01:00:00'), (2, '2024-03-05 20:00:00')", + "source_settings": "allow_nullable_key = 1", "expect_ok": False}, + ] + run_partition_compat_cases(node, cases) + + +def test_partition_transform_granularity_matrix(cluster): + """ + Exercise the common ClickHouse temporal partition keys and the granularity relationships between + the source key and the destination Iceberg transform. Acceptance is data-dependent (a source + partition must be single-valued for every destination field), so a coarser source can still be + accepted when a particular partition does not actually repartition. Accept cases are verified + end-to-end (data + metadata); reject cases must throw BAD_ARGUMENTS. + """ + node = cluster.instances["replica1"] + dt = "id Int64, event_time DateTime" + same_day = "(1, '2024-03-05 01:00:00'), (2, '2024-03-05 20:00:00')" + same_month = "(1, '2024-03-01 00:00:00'), (2, '2024-03-20 00:00:00')" + same_year = "(1, '2024-03-05 00:00:00'), (2, '2024-09-10 00:00:00')" + + def case(name, source_key, dest_key, rows, expect_ok): + return {"name": name, "columns": dt, "source_key": source_key, "dest_key": dest_key, + "rows": rows, "expect_ok": expect_ok} + + cases = [ + # Common temporal keys at the same granularity as the destination transform. + case("startofmonth_month", "toStartOfMonth(event_time)", "toMonthNumSinceEpoch(event_time)", same_month, True), + case("yyyymmdd_day", "toYYYYMMDD(event_time)", "toRelativeDayNum(event_time)", same_day, True), + case("startofday_day", "toStartOfDay(event_time)", "toRelativeDayNum(event_time)", same_day, True), + case("toyear_year", "toYear(event_time)", "toYearNumSinceEpoch(event_time)", same_year, True), + case("startofyear_year", "toStartOfYear(event_time)", "toYearNumSinceEpoch(event_time)", same_year, True), + # Finer source into a coarser destination: a finer partition sits inside one coarser bucket. + case("day_into_month", "toDate(event_time)", "toMonthNumSinceEpoch(event_time)", same_day, True), + case("day_into_year", "toDate(event_time)", "toYearNumSinceEpoch(event_time)", same_day, True), + case("hour_into_day", "toStartOfHour(event_time)", "toRelativeDayNum(event_time)", + "(1, '2024-03-05 12:00:00'), (2, '2024-03-05 12:30:00')", True), + case("month_into_year", "toYYYYMM(event_time)", "toYearNumSinceEpoch(event_time)", same_month, True), + # Coarser source into a finer destination: the partition spans several destination buckets. + case("year_into_month", "toYear(event_time)", "toMonthNumSinceEpoch(event_time)", + "(1, '2020-01-15 00:00:00'), (2, '2020-06-15 00:00:00')", False), + case("year_into_day", "toYear(event_time)", "toRelativeDayNum(event_time)", + "(1, '2020-01-01 00:00:00'), (2, '2020-12-31 00:00:00')", False), + # Same coarse/fine pair, but this year partition holds a single day, so it does not + # repartition and is accepted - acceptance depends on the data, not the structure. + case("year_into_day_single_day", "toYear(event_time)", "toRelativeDayNum(event_time)", same_day, True), + # Weekly has no Iceberg equivalent: a week partition holding two days cannot map to one day. + case("week_into_day", "toMonday(event_time)", "toRelativeDayNum(event_time)", + "(1, '2024-03-05 00:00:00'), (2, '2024-03-07 00:00:00')", False), + ] + run_partition_compat_cases(node, cases) + + +def test_partition_multicolumn_subset(cluster): + """ + Destination partition columns must be a subset of the source partition-key columns. A wide + source whose partition key is a superset of the destination's is accepted (and its multi-column + data plus per-field metadata verified); a destination partitioning by a column absent from the + source partition key is rejected. + """ + node = cluster.instances["replica1"] + wide = "id Int64, event_time DateTime, region String, tenant Int32, v1 Float64, v2 String" + + cases = [ + # Destination partition columns {event_time, region} are a strict subset of the source's + # {event_time, region, tenant}: accepted, with multi-column data and per-field metadata. + {"name": "subset_ok", "columns": wide, + "source_key": "(toDate(event_time), region, tenant)", + "dest_key": "(toRelativeDayNum(event_time), region)", + "rows": "(1, '2024-03-05 01:00:00', 'US', 7, 1.5, 'a'), " + "(2, '2024-03-05 20:00:00', 'US', 7, 2.5, 'b')", + "expect_ok": True, + "verify": [("event_time", "toRelativeDayNum(event_time)"), ("region", "region")]}, + # Destination partitions by 'region', which is not in the source partition key: rejected. + {"name": "not_subset", "columns": "id Int64, event_time DateTime, region String", + "source_key": "toDate(event_time)", + "dest_key": "(toRelativeDayNum(event_time), region)", + "rows": "(1, '2024-03-05 01:00:00', 'US'), (2, '2024-03-05 20:00:00', 'EU')", + "expect_ok": False}, + ] + run_partition_compat_cases(node, cases) + + +def test_export_partition_todate_source_matches_day_metadata(cluster): + """ + End-to-end: a source partitioned by toDate(event_time) exports into a day-partitioned Iceberg + table through the min/max refinement, and the day value written to the Iceberg metadata matches + the exported data. + """ + node = cluster.instances["replica1"] + + uid = unique_suffix() + mt_table = f"mt_todate_{uid}" + iceberg_table = f"iceberg_todate_{uid}" + + make_rmt(node, mt_table, "id Int64, event_time DateTime", "toDate(event_time)", + replica_name="replica1") + node.query( + f"INSERT INTO {mt_table} VALUES " + f"(1, '2024-03-05 01:00:00'), (2, '2024-03-05 12:00:00'), (3, '2024-03-05 23:00:00')" + ) + make_iceberg_s3(node, iceberg_table, "id Int64, event_time DateTime", + partition_by="toRelativeDayNum(event_time)") + + pid = first_partition_id(node, mt_table) + node.query( + f"ALTER TABLE {mt_table} EXPORT PARTITION ID '{pid}' TO TABLE {iceberg_table}", + settings={"allow_insert_into_iceberg": 1}, + ) + wait_for_export_status(node, mt_table, iceberg_table, pid, "COMPLETED") + + count = int(node.query(f"SELECT count() FROM {iceberg_table}").strip()) + assert count == 3, f"Expected 3 rows after export, got {count}" + + expected_day = int(node.query( + f"SELECT DISTINCT toRelativeDayNum(event_time) FROM {iceberg_table}" + ).strip()) + + query_id = f"todate_{uid}" + node.query( + f"SELECT * FROM {iceberg_table}", + query_id=query_id, + settings={"iceberg_metadata_log_level": "manifest_file_entry"}, + ) + entries = fetch_manifest_entries(node, query_id) + partitions = _data_file_partition_records(entries) + assert partitions, "No data-file partition records found in manifest entries" + meta_days = {int(_partition_scalar(p, "event_time")) for p in partitions} + assert meta_days == {expected_day}, ( + f"Metadata day {meta_days} must equal toRelativeDayNum {expected_day}." + ) + + +def test_export_partition_day_source_into_year_metadata(cluster): + """ + End-to-end: a source partitioned by toDate(event_time) (finer) exports into a year-partitioned + Iceberg destination (coarser). The value written to the Iceberg metadata is the year computed by + the destination transform over the data, not the source day. + """ + node = cluster.instances["replica1"] + + uid = unique_suffix() + mt_table = f"mt_day_year_{uid}" + iceberg_table = f"iceberg_day_year_{uid}" + + make_rmt(node, mt_table, "id Int64, event_time DateTime", "toDate(event_time)", + replica_name="replica1") + node.query( + f"INSERT INTO {mt_table} VALUES " + f"(1, '2024-03-05 01:00:00'), (2, '2024-03-05 12:00:00'), (3, '2024-03-05 23:00:00')" + ) + make_iceberg_s3(node, iceberg_table, "id Int64, event_time DateTime", + partition_by="toYearNumSinceEpoch(event_time)") + + pid = first_partition_id(node, mt_table) + node.query( + f"ALTER TABLE {mt_table} EXPORT PARTITION ID '{pid}' TO TABLE {iceberg_table}", + settings={"allow_insert_into_iceberg": 1}, + ) + wait_for_export_status(node, mt_table, iceberg_table, pid, "COMPLETED") + + count = int(node.query(f"SELECT count() FROM {iceberg_table}").strip()) + assert count == 3, f"Expected 3 rows after export, got {count}" + + expected_year = int(node.query( + f"SELECT DISTINCT toYearNumSinceEpoch(event_time) FROM {iceberg_table}" + ).strip()) + + query_id = f"day_year_{uid}" + node.query( + f"SELECT * FROM {iceberg_table}", + query_id=query_id, + settings={"iceberg_metadata_log_level": "manifest_file_entry"}, + ) + entries = fetch_manifest_entries(node, query_id) + partitions = _data_file_partition_records(entries) + assert partitions, "No data-file partition records found in manifest entries" + meta_years = {int(_partition_scalar(p, "event_time")) for p in partitions} + assert meta_years == {expected_year}, ( + f"Metadata year {meta_years} must equal toYearNumSinceEpoch {expected_year}." + ) + + +def test_export_partition_lossy_cast_dynamic_accept(cluster): + """ + A lossy Int64 -> Int32 partition-column cast is accepted by the dynamic proof when the + partition's values fit the destination type and map to a single Iceberg bucket. Source and + destination use different truncate widths, so the field is proven via min/max rather than a + structural match. + """ + node = cluster.instances["replica1"] + + uid = unique_suffix() + mt_table = f"mt_lossy_{uid}" + iceberg_table = f"iceberg_lossy_{uid}" + + make_rmt(node, mt_table, "id Int64, val Int64", "icebergTruncate(10, val)", + replica_name="replica1") + node.query(f"INSERT INTO {mt_table} VALUES (1, 100), (2, 109)") + make_iceberg_s3(node, iceberg_table, "id Int64, val Int32", + partition_by="icebergTruncate(1000000, val)") + + pid = first_partition_id(node, mt_table) + node.query( + f"ALTER TABLE {mt_table} EXPORT PARTITION ID '{pid}' TO TABLE {iceberg_table}", + settings={ + "allow_insert_into_iceberg": 1, + "export_merge_tree_part_allow_lossy_cast": 1, + }, + ) + wait_for_export_status(node, mt_table, iceberg_table, pid, "COMPLETED") + assert int(node.query(f"SELECT count() FROM {iceberg_table}").strip()) == 2 + + +def test_export_data_files_are_not_cleaned_up_on_commit_failure(cluster): + """ + Verify that a commit failure does not delete the already-written data files. + `cleanup` only removes the manifest entry / manifest list, never the data files + (a peer replica might still commit the same transaction). This guards against + data loss / dangling references. + + The iceberg_writes_non_retry_cleanup failpoint throws BAD_ARGUMENTS while writing + the manifest entry, after the data files have been written. BAD_ARGUMENTS is a + non-retryable error code, so the task transitions to FAILED; we then confirm the + exported data files are still physically present in object storage by reading + them directly (the Iceberg manifests were removed by cleanup, so we glob the raw + parquet data files instead). + """ + node = cluster.instances["replica1"] + uid = unique_suffix() + mt_table = f"mt_{uid}" + iceberg_table = f"iceberg_{uid}" + setup_tables(cluster, mt_table, iceberg_table, nodes=["replica1"]) + + node.query("SYSTEM ENABLE FAILPOINT iceberg_writes_non_retry_cleanup") + try: + node.query( + f"ALTER TABLE {mt_table} EXPORT PARTITION ID '2020' TO TABLE {iceberg_table}", + settings={"allow_insert_into_iceberg": 1}, + ) + # BAD_ARGUMENTS from the commit phase is non-retryable -> the task fails fast. + wait_for_export_status(node, mt_table, iceberg_table, "2020", "FAILED", timeout=60) + finally: + node.query("SYSTEM DISABLE FAILPOINT iceberg_writes_non_retry_cleanup") + + # The data files were written before the commit failure; cleanup must have left + # them intact. Read them straight from object storage (bypassing the Iceberg + # metadata, which cleanup removed) and confirm all 3 exported rows survive. + rows = int(node.query( + f"SELECT count() FROM s3(" + f"'http://minio1:9001/root/data/{iceberg_table}/**.parquet', " + f"'minio', 'ClickHouse_Minio_P@ssw0rd', 'Parquet')" + ).strip()) + assert rows == 3, ( + f"Expected the 3 exported rows to still exist as data files after a failed " + f"commit (data files must not be cleaned up), got {rows}" + ) + + +def test_post_publish_exception_preserves_snapshot(cluster): + """ + Regression test for the post-publish exception-safety bug in + commitImportPartitionTransactionImpl. + + Before the fix, any exception thrown after the Iceberg snapshot was published + (e.g. from metadata-cache invalidation) would fall through to the outer + `catch (...)` and invoke `cleanup(false)`, which unconditionally removed the + manifest entry and manifest list referenced by the just-published snapshot. + A subsequent read would then fail because the live snapshot points to deleted + files. + + The failpoint `iceberg_writes_post_publish_throw` is placed inside the + post-publish region (after both the metadata file is written and + `published = true` is set). With the fix in place: + - the commit stays durable (snapshot is readable, manifests are intact); + - the export is marked COMPLETED because the outer `catch (...)` sees + `published == true` and returns the populated commit info with the real + paths produced by this attempt (no retry needed); + - all exported rows are visible through the Iceberg table. + """ + node = cluster.instances["replica1"] + uid = unique_suffix() + mt_table = f"mt_{uid}" + iceberg_table = f"iceberg_{uid}" + setup_tables(cluster, mt_table, iceberg_table, nodes=["replica1"]) + + node.query("SYSTEM ENABLE FAILPOINT iceberg_writes_post_publish_throw") + + node.query( + f"ALTER TABLE {mt_table} EXPORT PARTITION ID '2020' TO TABLE {iceberg_table}", + settings={"allow_insert_into_iceberg": 1}, + ) + wait_for_export_status(node, mt_table, iceberg_table, "2020", "COMPLETED") + + count = int(node.query(f"SELECT count() FROM {iceberg_table} WHERE year = 2020").strip()) + assert count == 3, ( + f"Snapshot must remain readable after a post-publish exception, " + f"expected 3 rows but got {count} (manifest files likely deleted by " + f"over-broad cleanup)" + ) + + result = node.query( + f"SELECT id, year FROM {iceberg_table} WHERE year = 2020 ORDER BY id" + ).strip() + assert result == "1\t2020\n2\t2020\n3\t2020", ( + f"Unexpected data after post-publish exception recovery:\n{result}" + ) + + # After a post-publish exception the catch handler with published==true returns + # the populated commit info (real metadata / manifest list / manifest file paths). + # ExportPartitionUtils::commit persists it to the commit_info znode, so the system + # table should show a real metadata path here, not the already-committed sentinel. + committed_metadata_file = node.query( + f""" + SELECT committed_metadata_file FROM system.replicated_partition_exports + WHERE source_table = '{mt_table}' + AND destination_table = '{iceberg_table}' + AND partition_id = '2020' + """ + ).strip() + assert committed_metadata_file, ( + "committed_metadata_file should be populated after a successful post-publish-catch return" + ) + assert not committed_metadata_file.startswith("<"), ( + f"committed_metadata_file should be a real metadata path, got the already-committed sentinel: {committed_metadata_file!r}" + ) + assert committed_metadata_file.endswith(".metadata.json"), ( + f"Expected a *.metadata.json path in committed_metadata_file, got: {committed_metadata_file!r}" + ) + + +def test_export_task_timeout_kills_stuck_pending_task(cluster): + """ + Verify that export_merge_tree_partition_task_timeout_seconds auto-kills a task + that remains PENDING past the deadline, transitioning it to KILLED with a + descriptive last_exception. + + The export_partition_commit_always_throw failpoint wedges the task in the + commit retry loop (REGULAR failpoint, fires on every commit attempt) with a + retryable error, so the task never fails on its own and the timeout branch in + tryCleanup is the actual mechanism under test. + """ + node = cluster.instances["replica1"] + uid = unique_suffix() + mt_table = f"mt_{uid}" + iceberg_table = f"iceberg_{uid}" + setup_tables(cluster, mt_table, iceberg_table, nodes=["replica1"]) + + node.query("SYSTEM ENABLE FAILPOINT export_partition_commit_always_throw") + + try: + node.query( + f"ALTER TABLE {mt_table} EXPORT PARTITION ID '2020' TO TABLE {iceberg_table}" + f" SETTINGS export_merge_tree_partition_task_timeout_seconds = 5," + f" allow_insert_into_iceberg = 1" + ) + + # Timeout budget must cover: the 5s task timeout + one manifest-updating + # poll cycle (~30s) + watch propagation. 90s is safe. + wait_for_export_status( + node, mt_table, iceberg_table, "2020", + expected_status="KILLED", + timeout=90, + ) + + # The KILL transition writes a per-replica last_exception leaf in the same + # ZK multi as the status flip; handleStatusChanges then mirrors it into + # memory together with the status. Poll briefly to allow that watch -> + # mirror hop. We use arrayJoin to flatten the per-replica array column; + # any replica reporting the timeout reason is sufficient. + deadline = time.time() + 30 + last_exception = "" + while time.time() < deadline: + last_exception = node.query( + f""" + SELECT arrayStringConcat( + arrayMap(x -> x.message, last_exception_per_replica), + '\\n' + ) + FROM system.replicated_partition_exports + WHERE source_table = '{mt_table}' + AND destination_table = '{iceberg_table}' + AND partition_id = '2020' + """ + ).strip() + if "timed out" in last_exception: + break + time.sleep(0.5) + assert "timed out" in last_exception, ( + f"Expected last_exception_per_replica column to mention the timeout reason, got: {last_exception!r}" + ) + finally: + node.query("SYSTEM DISABLE FAILPOINT export_partition_commit_always_throw") + + +def setup_stats_tables(node, mt_table: str, iceberg_table: str): + """Local variant of setup_tables using the wider schema with a Nullable column.""" + columns = "id Int32, name String, tag Nullable(String), year Int32" + + make_rmt( + node, mt_table, columns, "year", + order_by="id", replica_name="replica1", + ) + node.query( + f""" + INSERT INTO {mt_table} (id, name, tag, year) VALUES + (1, 'aaa', 'x', 2020), + (2, 'mmm', NULL, 2020), + (3, 'zzz', 'y', 2020), + (4, 'kkk', 'z', 2021) + """ + ) + + make_iceberg_s3(node, iceberg_table, columns, partition_by="year") + + +def test_export_partition_writes_column_statistics(cluster): + """ + Export a whole partition (EXPORT PARTITION ID '2020') that contains one NULL + and verify that the resulting Iceberg manifest entry carries accurate per-file + column statistics: record_count, file_size_in_bytes, column_sizes, + null_value_counts, and lower/upper bounds. + """ + node = cluster.instances["replica1"] + + uid = unique_suffix() + mt_table = f"mt_stats_{uid}" + iceberg_table = f"iceberg_stats_{uid}" + + setup_stats_tables(node, mt_table, iceberg_table) + + node.query( + f"ALTER TABLE {mt_table} EXPORT PARTITION ID '2020' TO TABLE {iceberg_table}", + settings={"allow_insert_into_iceberg": 1}, + ) + wait_for_export_status(node, mt_table, iceberg_table, "2020", "COMPLETED") + + count = int(node.query(f"SELECT count() FROM {iceberg_table}").strip()) + assert count == 3, f"Expected 3 rows in Iceberg table after export, got {count}" + + query_id = f"stats_partition_{uid}" + node.query( + f"SELECT * FROM {iceberg_table} ORDER BY id", + query_id=query_id, + settings={"iceberg_metadata_log_level": "manifest_file_entry"}, + ) + + entries = fetch_manifest_entries(node, query_id) + assert_exported_stats(entries) + + +def test_export_partition_column_count_mismatch_source_more_is_rejected(cluster): + """ + Source has 3 columns (id, year, extra), destination has 2 (id, year). + The ALTER must be rejected synchronously with NUMBER_OF_COLUMNS_DOESNT_MATCH, + nothing must be scheduled in system.replicated_partition_exports, and the + Iceberg table must remain empty. + """ + node = cluster.instances["replica1"] + + uid = unique_suffix() + mt_table = f"mt_count_more_{uid}" + iceberg_table = f"iceberg_count_more_{uid}" + + make_rmt(node, mt_table, "id Int64, year Int32, extra String", "year", + replica_name="replica1") + node.query(f"INSERT INTO {mt_table} VALUES (1, 2020, 'foo'), (2, 2020, 'bar')") + + make_iceberg_s3(node, iceberg_table, "id Int64, year Int32", partition_by="year") + + error = node.query_and_get_error( + f"ALTER TABLE {mt_table} EXPORT PARTITION ID '2020' TO TABLE {iceberg_table}", + settings={"allow_insert_into_iceberg": 1}, + ) + assert "NUMBER_OF_COLUMNS_DOESNT_MATCH" in error, ( + f"Expected NUMBER_OF_COLUMNS_DOESNT_MATCH for source>dest column count, " + f"got: {error!r}" + ) + + rows_in_system_view = node.query( + f"SELECT count() FROM system.replicated_partition_exports " + f"WHERE source_table = '{mt_table}' " + f" AND destination_table = '{iceberg_table}' " + f" AND partition_id = '2020'" + ).strip() + assert rows_in_system_view == "0", ( + f"Expected no row in system.replicated_partition_exports after a " + f"synchronously-rejected export, got {rows_in_system_view}." + ) + + count = int(node.query(f"SELECT count() FROM {iceberg_table}").strip()) + assert count == 0, ( + f"Expected 0 rows in Iceberg table after rejected export, got {count}" + ) + + +def test_export_partition_column_count_mismatch_source_fewer_is_rejected(cluster): + """ + Source has 2 columns (id, year), destination has 3 (id, year, extra). + Same expected synchronous rejection as the source>dest case. + """ + node = cluster.instances["replica1"] + + uid = unique_suffix() + mt_table = f"mt_count_fewer_{uid}" + iceberg_table = f"iceberg_count_fewer_{uid}" + + make_rmt(node, mt_table, "id Int64, year Int32", "year", replica_name="replica1") + node.query(f"INSERT INTO {mt_table} VALUES (1, 2020), (2, 2020)") + + make_iceberg_s3(node, iceberg_table, "id Int64, year Int32, extra String", + partition_by="year") + + error = node.query_and_get_error( + f"ALTER TABLE {mt_table} EXPORT PARTITION ID '2020' TO TABLE {iceberg_table}", + settings={"allow_insert_into_iceberg": 1}, + ) + assert "NUMBER_OF_COLUMNS_DOESNT_MATCH" in error, ( + f"Expected NUMBER_OF_COLUMNS_DOESNT_MATCH for source Int64) and the + partition column (year Int32 -> Int64) round-trips.""" + node = cluster.instances["replica1"] + + uid = unique_suffix() + mt_table = f"mt_widen_{uid}" + iceberg_table = f"iceberg_widen_{uid}" + + make_rmt(node, mt_table, "id Int32, year Int32", "year", replica_name="replica1") + node.query(f"INSERT INTO {mt_table} VALUES (1, 2020), (2, 2020)") + + make_iceberg_s3(node, iceberg_table, "id Int64, year Int64", partition_by="year") + + node.query( + f"ALTER TABLE {mt_table} EXPORT PARTITION ID '2020' TO TABLE {iceberg_table}", + settings={"allow_insert_into_iceberg": 1}, + ) + wait_for_export_status(node, mt_table, iceberg_table, "2020", "COMPLETED") + + count = int(node.query(f"SELECT count() FROM {iceberg_table}").strip()) + assert count == 2, f"Expected 2 rows in Iceberg table after export, got {count}" + + result = node.query( + f"SELECT id, toTypeName(id), year, toTypeName(year) FROM {iceberg_table} ORDER BY id" + ).strip() + assert result == "1\tInt64\t2020\tInt64\n2\tInt64\t2020\tInt64", ( + f"Unexpected widened data:\n{result}" + ) + + +def test_export_partition_with_castable_narrowing_values_fit(cluster): + """A lossy narrowing (id Int64 -> Int32) succeeds once the user opts in via + export_merge_tree_part_allow_lossy_cast.""" + node = cluster.instances["replica1"] + + uid = unique_suffix() + mt_table = f"mt_narrow_fit_{uid}" + iceberg_table = f"iceberg_narrow_fit_{uid}" + + make_rmt(node, mt_table, "id Int64, year Int32", "year", replica_name="replica1") + node.query(f"INSERT INTO {mt_table} VALUES (1, 2020), (2, 2020)") + + make_iceberg_s3(node, iceberg_table, "id Int32, year Int32", partition_by="year") + + node.query( + f"ALTER TABLE {mt_table} EXPORT PARTITION ID '2020' TO TABLE {iceberg_table}", + settings={ + "allow_insert_into_iceberg": 1, + "export_merge_tree_part_allow_lossy_cast": 1, + }, + ) + wait_for_export_status(node, mt_table, iceberg_table, "2020", "COMPLETED") + + count = int(node.query(f"SELECT count() FROM {iceberg_table}").strip()) + assert count == 2, f"Expected 2 rows in Iceberg table after export, got {count}" + + result = node.query( + f"SELECT id, toTypeName(id), year FROM {iceberg_table} ORDER BY id" + ).strip() + assert result == "1\tInt32\t2020\n2\tInt32\t2020", ( + f"Unexpected narrowed data:\n{result}" + ) + + +def test_export_partition_lossy_cast_rejected_without_optin(cluster): + """A lossy narrowing (id Int64 -> Int32) is rejected synchronously with + INCOMPATIBLE_COLUMNS unless export_merge_tree_part_allow_lossy_cast is set.""" + node = cluster.instances["replica1"] + + uid = unique_suffix() + mt_table = f"mt_lossy_reject_{uid}" + iceberg_table = f"iceberg_lossy_reject_{uid}" + + make_rmt(node, mt_table, "id Int64, year Int32", "year", replica_name="replica1") + node.query(f"INSERT INTO {mt_table} VALUES (1, 2020)") + + make_iceberg_s3(node, iceberg_table, "id Int32, year Int32", partition_by="year") + + error = node.query_and_get_error( + f"ALTER TABLE {mt_table} EXPORT PARTITION ID '2020' TO TABLE {iceberg_table} " + f"SETTINGS allow_insert_into_iceberg = 1" + ) + assert "INCOMPATIBLE_COLUMNS" in error, f"Expected INCOMPATIBLE_COLUMNS, got: {error!r}" + assert "lossy cast" in error, f"Expected 'lossy cast' in error, got: {error!r}" + + count = int(node.query(f"SELECT count() FROM {iceberg_table}").strip()) + assert count == 0, f"Expected no rows after a rejected export, got {count}" + + +def test_export_partition_runtime_cast_failure_propagates_async(cluster): + """A String value that cannot be parsed as the destination Int32 passes the + synchronous lossy-cast gate (with export_merge_tree_part_allow_lossy_cast = 1) but + fails at runtime in the async worker with CANNOT_PARSE_TEXT. That is a deterministic + value-conversion error on the part's immutable data — retrying the same part can + never succeed — so it is classified as non-retryable and fails the whole task fast, + without waiting for the absolute task timeout, leaving Iceberg empty. + + The task timeout is left at its large default, so reaching FAILED quickly proves the + transition is driven by error classification rather than by a timeout. + + (Integer overflow is not used because the internal cast uses CastType::nonAccurate, + which wraps rather than throwing.) + """ + node = cluster.instances["replica1"] + + uid = unique_suffix() + mt_table = f"mt_runtime_cast_fail_{uid}" + iceberg_table = f"iceberg_runtime_cast_fail_{uid}" + + make_rmt(node, mt_table, "id String, year Int32", "year", replica_name="replica1") + node.query(f"INSERT INTO {mt_table} VALUES ('not a number', 2020)") + + make_iceberg_s3(node, iceberg_table, "id Int32, year Int32", partition_by="year") + + node.query( + f"ALTER TABLE {mt_table} EXPORT PARTITION ID '2020' TO TABLE {iceberg_table} " + f"SETTINGS allow_insert_into_iceberg = 1, export_merge_tree_part_allow_lossy_cast = 1" + ) + + # The runtime parse error (CANNOT_PARSE_TEXT) is non-retryable, so the task fails fast. + # No short timeout is set; FAILED within this window can only come from the + # non-retryable classification, not from the (default, ~1 day) task timeout. + wait_for_export_status(node, mt_table, iceberg_table, "2020", "FAILED", timeout=60) + + exception_count = int(node.query( + f"SELECT any(exception_count) FROM system.replicated_partition_exports " + f"WHERE source_table = '{mt_table}' " + f" AND destination_table = '{iceberg_table}' " + f" AND partition_id = '2020'" + ).strip()) + assert exception_count > 0, ( + "Expected non-zero exception_count after a failed runtime cast" + ) + + count = int(node.query(f"SELECT count() FROM {iceberg_table}").strip()) + assert count == 0, ( + f"Expected 0 rows in Iceberg table after failed export, got {count}" + ) + + +def test_export_partition_all_iceberg_types(cluster): + """Every getIcebergType-supported type round-trips through an EXPORT PARTITION: + scalars use narrower source types (explicit lossless widening CASTs), plus + Array/Map/Tuple nested columns.""" + node = cluster.instances["replica1"] + + uid = unique_suffix() + mt_table = f"mt_all_types_{uid}" + iceberg_table = f"iceberg_all_types_{uid}" + + # Scalar source types are strictly narrower than the destination; the export inserts + # a positional widening CAST per column (Int8->Int16, UInt32->UInt64, ...). Nested + # columns keep the same type on both sides. + source_columns = ( + "i16 Int8, u16 UInt8, u32 UInt16, u64 UInt32, " + "id Int16, big Int32, f32 Float32, f64 Float64, " + "d Date, d32 Date32, dt DateTime, dt64 DateTime64(6), " + "s String, uid UUID, " + "arr Array(Int32), m Map(String, Int64), tup Tuple(a Int32, b String), " + "year Int32" + ) + dest_columns = ( + "i16 Int16, u16 UInt16, u32 UInt32, u64 UInt64, " + "id Int32, big Int64, f32 Float32, f64 Float64, " + "d Date, d32 Date32, dt DateTime, dt64 DateTime64(6), " + "s String, uid UUID, " + "arr Array(Int32), m Map(String, Int64), tup Tuple(a Int32, b String), " + "year Int32" + ) + + make_rmt(node, mt_table, source_columns, "year", replica_name="replica1") + make_iceberg_s3(node, iceberg_table, dest_columns, partition_by="year") + + node.query( + f""" + INSERT INTO {mt_table} + (i16, u16, u32, u64, id, big, f32, f64, d, d32, dt, dt64, s, uid, arr, m, tup, year) + VALUES ( + -100, 200, 50000, 4000000000, + 12345, 1000000000, 3.14, 2.718281828459045, + '2024-01-15', '2024-01-15', '2024-01-15 12:30:45', '2024-01-15 12:30:45.123456', + 'hello iceberg', '550e8400-e29b-41d4-a716-446655440000', + [1, 2, 3], {{'a': 10, 'b': 20}}, (7, 'seven'), 2024 + ) + """ + ) + + node.query( + f"ALTER TABLE {mt_table} EXPORT PARTITION ID '2024' TO TABLE {iceberg_table}", + settings={"allow_insert_into_iceberg": 1}, + ) + wait_for_export_status(node, mt_table, iceberg_table, "2024", "COMPLETED") + + count = int(node.query(f"SELECT count() FROM {iceberg_table}").strip()) + assert count == 1, f"Expected 1 row in Iceberg table, got {count}" + + result = node.query( + f""" + SELECT + i16, u16, u32, u64, id, big, + toString(d), toString(d32), toString(dt), + s, toString(uid), + arr, m['a'], m['b'], tup.a, tup.b, year + FROM {iceberg_table} + """ + ).strip() + expected = "\t".join([ + "-100", "200", "50000", "4000000000", + "12345", "1000000000", + "2024-01-15", "2024-01-15", "2024-01-15 12:30:45.000000", + "hello iceberg", "550e8400-e29b-41d4-a716-446655440000", + "[1,2,3]", "10", "20", "7", "seven", "2024", + ]) + assert result == expected, f"Unexpected round-trip data:\n{result!r}\nexpected:\n{expected!r}" + + # Floats compared with a tolerance to avoid formatting flakiness. + floats_ok = node.query( + f"SELECT abs(f32 - 3.14) < 1e-4 AND abs(f64 - 2.718281828459045) < 1e-12 FROM {iceberg_table}" + ).strip() + assert floats_ok == "1", f"Float round-trip outside tolerance: {floats_ok!r}" + + # DateTime64 sub-second component: assert the date part is preserved (exact format varies). + ts_result = node.query(f"SELECT dt64 FROM {iceberg_table}").strip() + assert "2024-01-15" in ts_result, f"DateTime64 date component missing: {ts_result!r}" + + +def test_export_partition_all_iceberg_types_lossy(cluster): + """Lossy narrowing casts across types succeed with the opt-in flag: values that + fit round-trip, Float64 -> Float32 loses precision, and Nullable columns carry + both NULL and non-NULL (the latter via a lossy Nullable(Int64) -> Nullable(Int32)).""" + node = cluster.instances["replica1"] + + uid = unique_suffix() + mt_table = f"mt_lossy_types_{uid}" + iceberg_table = f"iceberg_lossy_types_{uid}" + + # Each source column is wider than the destination, so the export inserts a lossy + # narrowing CAST (allowed only because export_merge_tree_part_allow_lossy_cast=1). + # Int8/UInt8 are not Iceberg-representable, so the narrowest integer dest is Int16. + source_columns = ( + "big Int64, ubig UInt64, mid Int32, " + "f Float64, dt DateTime64(6), d Date32, " + "opt_s Nullable(String), opt_i Nullable(Int64), year Int32" + ) + dest_columns = ( + "big Int32, ubig UInt32, mid Int16, " + "f Float32, dt DateTime, d Date, " + "opt_s Nullable(String), opt_i Nullable(Int32), year Int32" + ) + + make_rmt(node, mt_table, source_columns, "year", replica_name="replica1") + make_iceberg_s3(node, iceberg_table, dest_columns, partition_by="year") + + # Values chosen to fit the destination types (the async cast wraps on overflow + # rather than throwing, so out-of-range values would silently corrupt instead). + # opt_s is NULL and opt_i is set, covering both nullable paths in one row. + node.query( + f""" + INSERT INTO {mt_table} (big, ubig, mid, f, dt, d, opt_s, opt_i, year) + VALUES ( + 1000000, 2000000000, 30000, + 2.718281828459045, '2024-01-15 12:30:45.123456', '2024-01-15', + NULL, 100, 2024 + ) + """ + ) + + node.query( + f"ALTER TABLE {mt_table} EXPORT PARTITION ID '2024' TO TABLE {iceberg_table}", + settings={ + "allow_insert_into_iceberg": 1, + "export_merge_tree_part_allow_lossy_cast": 1, + }, + ) + wait_for_export_status(node, mt_table, iceberg_table, "2024", "COMPLETED") + + count = int(node.query(f"SELECT count() FROM {iceberg_table}").strip()) + assert count == 1, f"Expected 1 row in Iceberg table, got {count}" + + result = node.query( + f"SELECT big, ubig, mid, toString(d), toString(dt), opt_s, opt_i, year FROM {iceberg_table}" + ).strip() + expected = "\t".join([ + "1000000", "2000000000", "30000", + "2024-01-15", "2024-01-15 12:30:45.000000", "\\N", "100", "2024", + ]) + assert result == expected, f"Unexpected lossy round-trip data:\n{result!r}\nexpected:\n{expected!r}" + + # Float64 -> Float32 stays within Float32 precision but is no longer exact. + f_checks = node.query( + f"SELECT abs(f - 2.718281828459045) < 1e-6, abs(f - 2.718281828459045) > 1e-9 FROM {iceberg_table}" + ).strip() + assert f_checks == "1\t1", f"Expected Float32 precision loss within tolerance, got: {f_checks!r}" + + +def _data_file_partition_records(entries): + """Partition dicts of the non-delete data files described by manifest entries.""" + records = [] + for entry in entries: + data_file = entry.get("data_file") or {} + if data_file.get("content", 0) not in (0, None): + continue + partition = data_file.get("partition") + if partition is not None: + records.append(partition) + return records + + +def _partition_scalar(partition, field): + """Read a partition field value, tolerating an Avro-union ``{type: value}`` wrapper.""" + value = partition.get(field) + if isinstance(value, dict): + assert len(value) == 1, f"Unexpected partition union shape for {field!r}: {value!r}" + value = next(iter(value.values())) + return value + + +def assert_iceberg_partition_metadata(node, iceberg_table, uid, fields): + """Assert every data-file partition record's field equals the single DISTINCT value of the + corresponding expression over the exported destination data. `fields` is a list of + (metadata_field_name, value_expr). String-normalized so integer transforms and identity + string/int fields compare uniformly.""" + query_id = f"verify_{uid}" + node.query( + f"SELECT * FROM {iceberg_table}", + query_id=query_id, + settings={"iceberg_metadata_log_level": "manifest_file_entry"}, + ) + entries = fetch_manifest_entries(node, query_id) + partitions = _data_file_partition_records(entries) + assert partitions, "No data-file partition records found in manifest entries" + for field_name, value_expr in fields: + expected = node.query( + f"SELECT DISTINCT toString({value_expr}) FROM {iceberg_table}" + ).strip() + got = {str(_partition_scalar(p, field_name)) for p in partitions} + assert got == {expected}, ( + f"metadata field {field_name!r} = {got}, expected {{{expected!r}}}" + ) + + +def run_partition_compat_cases(node, cases): + """Run partition-compatibility cases against the Iceberg export gate. + + Reject cases (``expect_ok=False``) are checked synchronously - the gate fires while scheduling, + so the ALTER throws immediately. Accept cases are dispatched together, then awaited, then their + data (full ordered row comparison against the exported source partition) and Iceberg partition + metadata are verified. Each case is a dict: name, columns, source_key, dest_key, rows, expect_ok, + and optional verify (list of (metadata_field_name, value_expr); defaults to + [("event_time", dest_key)]) and source_settings (extra MergeTree settings).""" + settings = {"allow_insert_into_iceberg": 1} + + def setup(case): + uid = unique_suffix() + mt_table = f"mt_{case['name']}_{uid}" + iceberg_table = f"iceberg_{case['name']}_{uid}" + make_rmt(node, mt_table, case["columns"], case["source_key"], replica_name="replica1", + extra_settings=case.get("source_settings", "")) + node.query(f"INSERT INTO {mt_table} VALUES {case['rows']}") + make_iceberg_s3(node, iceberg_table, case["columns"], partition_by=case["dest_key"]) + pid = first_partition_id(node, mt_table) + return uid, mt_table, iceberg_table, pid + + for case in cases: + if case["expect_ok"]: + continue + _uid, mt_table, iceberg_table, pid = setup(case) + error = node.query_and_get_error( + f"ALTER TABLE {mt_table} EXPORT PARTITION ID '{pid}' TO TABLE {iceberg_table}", + settings=settings, + ) + assert "BAD_ARGUMENTS" in error, f"{case['name']}: expected BAD_ARGUMENTS, got: {error!r}" + + dispatched = [] + for case in cases: + if not case["expect_ok"]: + continue + uid, mt_table, iceberg_table, pid = setup(case) + node.query( + f"ALTER TABLE {mt_table} EXPORT PARTITION ID '{pid}' TO TABLE {iceberg_table}", + settings=settings, + ) + dispatched.append((case, uid, mt_table, iceberg_table, pid)) + + for case, uid, mt_table, iceberg_table, pid in dispatched: + wait_for_export_status(node, mt_table, iceberg_table, pid, "COMPLETED") + + for case, uid, mt_table, iceberg_table, pid in dispatched: + # Export is a positional cast into the destination schema, so verify the destination equals + # the source cast into the destination column types. Normalizing to the destination types + # tolerates legitimate Iceberg type promotion (e.g. DateTime is stored as a microsecond + # timestamp and returns as DateTime64(6)) while preserving destination precision, so a + # spurious sub-second value would still surface as a mismatch. + col_defs = node.query( + f"SELECT name, type FROM system.columns " + f"WHERE database = currentDatabase() AND table = '{iceberg_table}' ORDER BY position" + ).strip().split("\n") + projection = ", ".join( + f"CAST({name} AS {ctype})" for name, ctype in (c.split("\t") for c in col_defs) + ) + src = node.query(f"SELECT {projection} FROM {mt_table} ORDER BY id") + dst = node.query(f"SELECT {projection} FROM {iceberg_table} ORDER BY id") + assert src == dst, f"{case['name']}: destination rows differ from source" + fields = case.get("verify") or [("event_time", case["dest_key"])] + assert_iceberg_partition_metadata(node, iceberg_table, f"{case['name']}_{uid}", fields) + + +def test_export_partition_bucket_type_change_rejected(cluster): + """A bucket[N] partition column whose type changes (Int64 -> String) is rejected. The source + hashLong grouping differs from the destination murmur(String) grouping, so a single source bucket + can fan out across several destination buckets; bucket is not order-preserving, so this cannot be + proven dynamically and must be rejected. This previously slipped through the structural fast path, + which matched on transform name and width while ignoring the pre-transform cast.""" + node = cluster.instances["replica1"] + + uid = unique_suffix() + mt_table = f"mt_bucket_xform_{uid}" + iceberg_table = f"iceberg_bucket_xform_{uid}" + + make_rmt(node, mt_table, "id Int64, key Int64", "icebergBucket(16, key)", + replica_name="replica1") + node.query(f"INSERT INTO {mt_table} VALUES (1, 42), (2, 42)") + + make_iceberg_s3(node, iceberg_table, "id Int64, key String", + partition_by="icebergBucket(16, key)") + + pid = first_partition_id(node, mt_table) + error = node.query_and_get_error( + f"ALTER TABLE {mt_table} EXPORT PARTITION ID '{pid}' TO TABLE {iceberg_table}", + settings={"allow_insert_into_iceberg": 1}, + ) + assert "BAD_ARGUMENTS" in error, ( + f"Expected BAD_ARGUMENTS for a type-changing bucket transform, got: {error!r}" + ) + + +def test_export_partition_truncate_type_change_rejected(cluster): + """icebergTruncate with the same width but a changed column type (Int64 -> String) is rejected. + Truncate is numeric on integers (120..129 -> 120) but byte-wise on strings ('120'..'129' stay + distinct), so one source truncate bucket can map to several destination buckets. The structural + fast path must not accept it on matching transform name and width; the dynamic proof rejects it + because the endpoints do not collapse to a single destination value.""" + node = cluster.instances["replica1"] + + uid = unique_suffix() + mt_table = f"mt_trunc_xform_{uid}" + iceberg_table = f"iceberg_trunc_xform_{uid}" + + # 120 and 129 are one Int64 truncate[10] bucket (120) but two distinct string truncations. + make_rmt(node, mt_table, "id Int64, key Int64", "icebergTruncate(10, key)", + replica_name="replica1") + node.query(f"INSERT INTO {mt_table} VALUES (1, 120), (2, 129)") + + make_iceberg_s3(node, iceberg_table, "id Int64, key String", + partition_by="icebergTruncate(10, key)") + + pid = first_partition_id(node, mt_table) + error = node.query_and_get_error( + f"ALTER TABLE {mt_table} EXPORT PARTITION ID '{pid}' TO TABLE {iceberg_table}", + settings={"allow_insert_into_iceberg": 1}, + ) + assert "BAD_ARGUMENTS" in error, ( + f"Expected BAD_ARGUMENTS for a type-changing truncate transform, got: {error!r}" + ) + + +def test_export_partition_value_preserving_cast_not_order_preserving_rejected(cluster): + """Int64 -> String keeps every value, but not their order: 2 and 29 are the endpoints of the + source partition, yet the interior value 10 casts to a string that sorts outside them. The + endpoints truncate to '2' while 10 truncates to '1', so the partition spans two destination + buckets and must be rejected instead of being waved through as a lossless cast.""" + node = cluster.instances["replica1"] + + uid = unique_suffix() + mt_table = f"mt_cast_order_{uid}" + iceberg_table = f"iceberg_cast_order_{uid}" + + make_rmt(node, mt_table, "id Int64, k Int64", "intDiv(k, 100)", + replica_name="replica1") + node.query(f"INSERT INTO {mt_table} VALUES (1, 2), (2, 10), (3, 29)") + + make_iceberg_s3(node, iceberg_table, "id Int64, k String", + partition_by="icebergTruncate(1, k)") + + pid = first_partition_id(node, mt_table) + error = node.query_and_get_error( + f"ALTER TABLE {mt_table} EXPORT PARTITION ID '{pid}' TO TABLE {iceberg_table}", + settings={"allow_insert_into_iceberg": 1}, + ) + assert "BAD_ARGUMENTS" in error, ( + f"Expected BAD_ARGUMENTS for a non-order-preserving cast, got: {error!r}" + ) + + +def test_export_partition_order_preserving_cast_accepted(cluster): + """The same shape as the rejected case, but with all values sharing a digit count: Int64 -> + String is order-preserving over [20, 29], so the endpoints do bound the interior and the whole + source partition truncates to the single destination bucket '2'.""" + node = cluster.instances["replica1"] + + uid = unique_suffix() + mt_table = f"mt_cast_order_ok_{uid}" + iceberg_table = f"iceberg_cast_order_ok_{uid}" + + make_rmt(node, mt_table, "id Int64, k Int64", "intDiv(k, 100)", + replica_name="replica1") + node.query(f"INSERT INTO {mt_table} VALUES (1, 20), (2, 25), (3, 29)") + + make_iceberg_s3(node, iceberg_table, "id Int64, k String", + partition_by="icebergTruncate(1, k)") + + pid = first_partition_id(node, mt_table) + node.query( + f"ALTER TABLE {mt_table} EXPORT PARTITION ID '{pid}' TO TABLE {iceberg_table}", + settings={"allow_insert_into_iceberg": 1}, + ) + wait_for_export_status(node, mt_table, iceberg_table, pid, "COMPLETED") + + src = node.query(f"SELECT id, toString(k) FROM {mt_table} ORDER BY id").strip() + dst = node.query(f"SELECT id, k FROM {iceberg_table} ORDER BY id").strip() + assert src == dst, f"destination rows differ from source:\n{src}\n---\n{dst}" + + assert_iceberg_partition_metadata(node, iceberg_table, uid, [("k", "icebergTruncate(1, k)")]) + + +def test_export_partition_timezone_mismatch_rejected(cluster): + """A source partitioned by day in one timezone must not be treated as structurally identical to a + destination day computed in another timezone. The source uses Asia/Tokyo (UTC+9) and the + destination UTC; the exported part spans a UTC-day boundary while staying within one Tokyo day, so + it maps to two destination partitions and must be rejected.""" + node = cluster.instances["replica1"] + + uid = unique_suffix() + mt_table = f"mt_tzmismatch_{uid}" + iceberg_table = f"iceberg_tzmismatch_{uid}" + + make_rmt(node, mt_table, "id Int64, event_time DateTime('UTC')", + "toRelativeDayNum(event_time, 'Asia/Tokyo')", replica_name="replica1") + # Both instants are 2024-03-05 in Tokyo (UTC+9) but 2024-03-04 and 2024-03-05 in UTC. + node.query( + f"INSERT INTO {mt_table} VALUES (1, '2024-03-04 16:00:00'), (2, '2024-03-05 10:00:00')" + ) + + make_iceberg_s3(node, iceberg_table, "id Int64, event_time DateTime('UTC')", + partition_by="toRelativeDayNum(event_time)") + + pid = first_partition_id(node, mt_table) + error = node.query_and_get_error( + f"ALTER TABLE {mt_table} EXPORT PARTITION ID '{pid}' TO TABLE {iceberg_table}", + settings={"allow_insert_into_iceberg": 1, "iceberg_partition_timezone": "UTC"}, + ) + assert "BAD_ARGUMENTS" in error, ( + f"Expected BAD_ARGUMENTS for a source/destination timezone mismatch, got: {error!r}" + ) + + +def test_export_partition_column_timezone_mismatch_rejected(cluster): + """The same mismatch as above, but with the timezone carried by the column type instead of the + partition expression. Both sides read `toRelativeDayNum(event_time)`, so the terms are identical and + only the types differ - and DateTime types with different timezones compare equal, so the structural + match must not be decided by type equality alone. The part stays within one Tokyo day while spanning + two UTC days, so it maps to two destination partitions and must be rejected. + + `iceberg_partition_timezone` is deliberately left unset: setting it stamps a timezone onto the + destination term, which alone makes the terms differ and hides what this test covers.""" + node = cluster.instances["replica1"] + + uid = unique_suffix() + mt_table = f"mt_coltz_{uid}" + iceberg_table = f"iceberg_coltz_{uid}" + + make_rmt(node, mt_table, "id Int64, event_time DateTime('Asia/Tokyo')", + "toRelativeDayNum(event_time)", replica_name="replica1") + # Both literals are 2024-03-05 in Tokyo (the column's timezone) but 2024-03-04 and 2024-03-05 in UTC. + node.query( + f"INSERT INTO {mt_table} VALUES (1, '2024-03-05 01:00:00'), (2, '2024-03-05 18:00:00')" + ) + + make_iceberg_s3(node, iceberg_table, "id Int64, event_time DateTime('UTC')", + partition_by="toRelativeDayNum(event_time)") + + pid = first_partition_id(node, mt_table) + error = node.query_and_get_error( + f"ALTER TABLE {mt_table} EXPORT PARTITION ID '{pid}' TO TABLE {iceberg_table}", + settings={"allow_insert_into_iceberg": 1}, + ) + assert "BAD_ARGUMENTS" in error, ( + f"Expected BAD_ARGUMENTS for a partition-column timezone mismatch, got: {error!r}" + ) + + +def test_export_partition_commit_uses_exported_parts_not_new_inserts(cluster): + """The deferred commit derives the Iceberg partition value only from the exact exported parts + recorded in the manifest, never from parts inserted/merged into the source partition after + scheduling. A month-partitioned source exports one day into a day-partitioned destination (a + data-dependent acceptance); while the commit is wedged, an earlier day is inserted and merged in, + so the only active part now spans both days with its min at the new day. The commit must still + stamp the exported day (the exported part is found among Outdated parts by name), not the merged-in + earlier day, so the metadata matches the exported data files.""" + node = cluster.instances["replica1"] + uid = unique_suffix() + mt_table = f"mt_commit_parts_{uid}" + iceberg_table = f"iceberg_commit_parts_{uid}" + + make_rmt(node, mt_table, "id Int64, event_date Date", "toYYYYMM(event_date)", replica_name="replica1") + node.query(f"INSERT INTO {mt_table} VALUES (1, '2024-03-20'), (2, '2024-03-20')") + make_iceberg_s3(node, iceberg_table, "id Int64, event_date Date", + partition_by="toRelativeDayNum(event_date)") + + exported_day = int(node.query("SELECT toRelativeDayNum(toDate('2024-03-20'))").strip()) + injected_day = int(node.query("SELECT toRelativeDayNum(toDate('2024-03-05'))").strip()) + + node.query("SYSTEM ENABLE FAILPOINT export_partition_commit_always_throw") + try: + node.query( + f"ALTER TABLE {mt_table} EXPORT PARTITION ID '202403' TO TABLE {iceberg_table}" + f" SETTINGS allow_insert_into_iceberg = 1" + ) + # The commit is attempted only after every part is exported, so a non-zero exception count + # means the data files are written and the commit is now wedged by the failpoint. + wait_for_exception_count(node, mt_table, iceberg_table, "202403", min_exception_count=1, timeout=90) + + # Insert an earlier day into the same month partition and merge: the merged active part spans + # both days with min = the injected (earlier) day, while the exported part becomes Outdated. + node.query(f"INSERT INTO {mt_table} VALUES (3, '2024-03-05')") + node.query(f"OPTIMIZE TABLE {mt_table} PARTITION ID '202403' FINAL") + finally: + node.query("SYSTEM DISABLE FAILPOINT export_partition_commit_always_throw") + + wait_for_export_status(node, mt_table, iceberg_table, "202403", "COMPLETED", timeout=90) + + # The exported data files hold only 2024-03-20; the metadata day must match them. + query_id = f"commit_parts_{uid}" + node.query( + f"SELECT * FROM {iceberg_table}", + query_id=query_id, + settings={"iceberg_metadata_log_level": "manifest_file_entry"}, + ) + entries = fetch_manifest_entries(node, query_id) + partitions = _data_file_partition_records(entries) + assert partitions, "No data-file partition records found in manifest entries" + meta_days = {int(_partition_scalar(p, "event_date")) for p in partitions} + assert meta_days == {exported_day}, ( + f"Metadata day {meta_days} must equal the exported day {exported_day} (2024-03-20), " + f"not the injected day {injected_day} (2024-03-05)." + ) + + assert int(node.query(f"SELECT count() FROM {iceberg_table}").strip()) == 2, ( + "Only the two exported rows must be present in the destination." + ) + + +def test_export_partition_month_transform_metadata_matches_data(cluster): + """A month-transform partition records a months-since-epoch value in metadata that + matches the value derived from the exported data, and a transform-filtered read + returns the rows.""" + node = cluster.instances["replica1"] + + uid = unique_suffix() + mt_table = f"mt_month_xform_{uid}" + iceberg_table = f"iceberg_month_xform_{uid}" + + make_rmt(node, mt_table, "id Int64, event_date Date", + "toMonthNumSinceEpoch(event_date)", replica_name="replica1") + node.query( + f"INSERT INTO {mt_table} VALUES " + f"(1, '2024-03-05'), (2, '2024-03-20'), (3, '2024-03-31')" + ) + + make_iceberg_s3(node, iceberg_table, "id Int64, event_date Date", + partition_by="toMonthNumSinceEpoch(event_date)") + + pid = first_partition_id(node, mt_table) + node.query( + f"ALTER TABLE {mt_table} EXPORT PARTITION ID '{pid}' TO TABLE {iceberg_table}", + settings={"allow_insert_into_iceberg": 1}, + ) + wait_for_export_status(node, mt_table, iceberg_table, pid, "COMPLETED") + + count = int(node.query(f"SELECT count() FROM {iceberg_table}").strip()) + assert count == 3, f"Expected 3 rows after export, got {count}" + + month_num = int(node.query( + f"SELECT DISTINCT toMonthNumSinceEpoch(event_date) FROM {iceberg_table}" + ).strip()) + + query_id = f"month_xform_{uid}" + node.query( + f"SELECT * FROM {iceberg_table}", + query_id=query_id, + settings={"iceberg_metadata_log_level": "manifest_file_entry"}, + ) + entries = fetch_manifest_entries(node, query_id) + partitions = _data_file_partition_records(entries) + assert partitions, "No data-file partition records found in manifest entries" + meta_values = {int(_partition_scalar(p, "event_date")) for p in partitions} + assert meta_values == {month_num}, ( + f"Metadata month {meta_values} must equal toMonthNumSinceEpoch over the data " + f"({month_num})." + ) + + filtered = int(node.query( + f"SELECT count() FROM {iceberg_table} " + f"WHERE toMonthNumSinceEpoch(event_date) = {month_num}" + ).strip()) + assert filtered == 3, f"Transform-filtered read expected 3 rows, got {filtered}" + + +def test_export_partition_identity_type_change_metadata_matches_data(cluster): + """An identity partition column whose type changes UInt16 -> String records the + destination String value in the Iceberg metadata, matching the exported data.""" + node = cluster.instances["replica1"] + + uid = unique_suffix() + mt_table = f"mt_identity_xform_{uid}" + iceberg_table = f"iceberg_identity_xform_{uid}" + + make_rmt(node, mt_table, "id Int32, year UInt16", "year", replica_name="replica1") + node.query(f"INSERT INTO {mt_table} VALUES (1, 2024), (2, 2024)") + + make_iceberg_s3(node, iceberg_table, "id Int32, year String", partition_by="year") + + pid = first_partition_id(node, mt_table) + node.query( + f"ALTER TABLE {mt_table} EXPORT PARTITION ID '{pid}' TO TABLE {iceberg_table}", + settings={"allow_insert_into_iceberg": 1}, + ) + wait_for_export_status(node, mt_table, iceberg_table, pid, "COMPLETED") + + count = int(node.query(f"SELECT count() FROM {iceberg_table}").strip()) + assert count == 2, f"Expected 2 rows after export, got {count}" + + data_year = node.query(f"SELECT DISTINCT year FROM {iceberg_table}").strip() + assert data_year == "2024", f"Expected exported year '2024' (String), got {data_year!r}" + + query_id = f"identity_xform_{uid}" + node.query( + f"SELECT * FROM {iceberg_table}", + query_id=query_id, + settings={"iceberg_metadata_log_level": "manifest_file_entry"}, + ) + entries = fetch_manifest_entries(node, query_id) + partitions = _data_file_partition_records(entries) + assert partitions, "No data-file partition records found in manifest entries" + meta_values = {str(_partition_scalar(p, "year")) for p in partitions} + assert meta_values == {"2024"}, ( + f"Metadata partition {meta_values} must equal the destination String value " + f"'2024' (not the source integer representation)." + ) + + +def test_export_partition_multicolumn_identity_metadata_matches_data(cluster): + """A multi-column identity partition (event_date Date, retention UInt64 -> Int64) + records per-column values in the Iceberg metadata that match the exported data.""" + node = cluster.instances["replica1"] + + uid = unique_suffix() + mt_table = f"mt_multicol_{uid}" + iceberg_table = f"iceberg_multicol_{uid}" + + # Iceberg has no unsigned types, so retention widens UInt64 -> Int64; the cast is + # not value-preserving per canBeSafelyCast, hence the lossy opt-in below. + make_rmt(node, mt_table, "id Int64, event_date Date, retention UInt64", + "(event_date, retention)", replica_name="replica1") + node.query( + f"INSERT INTO {mt_table} VALUES " + f"(1, '2024-03-05', 30), (2, '2024-03-05', 30), (3, '2024-03-05', 30)" + ) + + make_iceberg_s3(node, iceberg_table, "id Int64, event_date Date, retention Int64", + partition_by="(event_date, retention)") + + pid = first_partition_id(node, mt_table) + node.query( + f"ALTER TABLE {mt_table} EXPORT PARTITION ID '{pid}' TO TABLE {iceberg_table}", + settings={ + "allow_insert_into_iceberg": 1, + "export_merge_tree_part_allow_lossy_cast": 1, + }, + ) + wait_for_export_status(node, mt_table, iceberg_table, pid, "COMPLETED") + + count = int(node.query(f"SELECT count() FROM {iceberg_table}").strip()) + assert count == 3, f"Expected 3 rows after export, got {count}" + + data_retention = int(node.query( + f"SELECT DISTINCT retention FROM {iceberg_table}" + ).strip()) + assert data_retention == 30, f"Expected exported retention 30, got {data_retention}" + + days = int(node.query( + f"SELECT DISTINCT toInt64(event_date) FROM {iceberg_table}" + ).strip()) + + query_id = f"multicol_{uid}" + node.query( + f"SELECT * FROM {iceberg_table}", + query_id=query_id, + settings={"iceberg_metadata_log_level": "manifest_file_entry"}, + ) + entries = fetch_manifest_entries(node, query_id) + partitions = _data_file_partition_records(entries) + assert partitions, "No data-file partition records found in manifest entries" + + meta_dates = {int(_partition_scalar(p, "event_date")) for p in partitions} + assert meta_dates == {days}, ( + f"Metadata event_date {meta_dates} must equal days-since-epoch {days}." + ) + meta_retentions = {int(_partition_scalar(p, "retention")) for p in partitions} + assert meta_retentions == {30}, ( + f"Metadata retention {meta_retentions} must equal the exported value 30." + ) + + filtered = int(node.query( + f"SELECT count() FROM {iceberg_table} " + f"WHERE event_date = '2024-03-05' AND retention = 30" + ).strip()) + assert filtered == 3, f"Partition-filtered read expected 3 rows, got {filtered}" diff --git a/tests/integration/test_export_replicated_mt_partition_to_object_storage/__init__.py b/tests/integration/test_export_replicated_mt_partition_to_object_storage/__init__.py new file mode 100644 index 000000000000..e69de29bb2d1 diff --git a/tests/integration/test_export_replicated_mt_partition_to_object_storage/configs/allow_experimental_export_partition.xml b/tests/integration/test_export_replicated_mt_partition_to_object_storage/configs/allow_experimental_export_partition.xml new file mode 100644 index 000000000000..d931c6fb00db --- /dev/null +++ b/tests/integration/test_export_replicated_mt_partition_to_object_storage/configs/allow_experimental_export_partition.xml @@ -0,0 +1,3 @@ + + 1 + \ No newline at end of file diff --git a/tests/integration/test_export_replicated_mt_partition_to_object_storage/configs/disable_experimental_export_partition.xml b/tests/integration/test_export_replicated_mt_partition_to_object_storage/configs/disable_experimental_export_partition.xml new file mode 100644 index 000000000000..5379b8e892f0 --- /dev/null +++ b/tests/integration/test_export_replicated_mt_partition_to_object_storage/configs/disable_experimental_export_partition.xml @@ -0,0 +1,3 @@ + + 0 + \ No newline at end of file diff --git a/tests/integration/test_export_replicated_mt_partition_to_object_storage/configs/macros_shard1_replica1.xml b/tests/integration/test_export_replicated_mt_partition_to_object_storage/configs/macros_shard1_replica1.xml new file mode 100644 index 000000000000..bae1ce119255 --- /dev/null +++ b/tests/integration/test_export_replicated_mt_partition_to_object_storage/configs/macros_shard1_replica1.xml @@ -0,0 +1,6 @@ + + + shard1 + replica1 + + diff --git a/tests/integration/test_export_replicated_mt_partition_to_object_storage/configs/macros_shard2_replica1.xml b/tests/integration/test_export_replicated_mt_partition_to_object_storage/configs/macros_shard2_replica1.xml new file mode 100644 index 000000000000..fb9a587e736d --- /dev/null +++ b/tests/integration/test_export_replicated_mt_partition_to_object_storage/configs/macros_shard2_replica1.xml @@ -0,0 +1,6 @@ + + + shard2 + replica1 + + diff --git a/tests/integration/test_export_replicated_mt_partition_to_object_storage/configs/named_collections.xml b/tests/integration/test_export_replicated_mt_partition_to_object_storage/configs/named_collections.xml new file mode 100644 index 000000000000..d46920b7ba88 --- /dev/null +++ b/tests/integration/test_export_replicated_mt_partition_to_object_storage/configs/named_collections.xml @@ -0,0 +1,9 @@ + + + + http://minio1:9001/root/data + minio + ClickHouse_Minio_P@ssw0rd + + + \ No newline at end of file diff --git a/tests/integration/test_export_replicated_mt_partition_to_object_storage/configs/users.d/profile.xml b/tests/integration/test_export_replicated_mt_partition_to_object_storage/configs/users.d/profile.xml new file mode 100644 index 000000000000..518f29708929 --- /dev/null +++ b/tests/integration/test_export_replicated_mt_partition_to_object_storage/configs/users.d/profile.xml @@ -0,0 +1,8 @@ + + + + 3 + + + + diff --git a/tests/integration/test_export_replicated_mt_partition_to_object_storage/test.py b/tests/integration/test_export_replicated_mt_partition_to_object_storage/test.py new file mode 100644 index 000000000000..813e4987a592 --- /dev/null +++ b/tests/integration/test_export_replicated_mt_partition_to_object_storage/test.py @@ -0,0 +1,2541 @@ +import logging +import time +import uuid +from typing import NamedTuple + +import pytest + +from helpers.cluster import ClickHouseCluster +from helpers.export_partition_helpers import ( + first_partition_id, + make_rmt, + wait_for_exception_count, + wait_for_export_status, + wait_for_export_to_start, +) +from helpers.network import PartitionManager + + + +def skip_if_remote_database_disk_enabled(cluster): + """Skip test if any instance in the cluster has remote database disk enabled. + + Tests that block MinIO cannot run when remote database disk is enabled, + as the database metadata is stored on MinIO and blocking it would break the database. + """ + for instance in cluster.instances.values(): + if instance.with_remote_database_disk: + pytest.skip("Test cannot run with remote database disk enabled (db disk), as it blocks MinIO which stores database metadata") + + +@pytest.fixture(scope="module") +def cluster(): + try: + cluster = ClickHouseCluster(__file__) + cluster.add_instance( + "replica1", + main_configs=["configs/named_collections.xml", "configs/allow_experimental_export_partition.xml"], + user_configs=["configs/users.d/profile.xml"], + with_minio=True, + stay_alive=True, + with_zookeeper=True, + keeper_required_feature_flags=["multi_read"], + ) + cluster.add_instance( + "replica2", + main_configs=["configs/named_collections.xml", "configs/allow_experimental_export_partition.xml"], + user_configs=["configs/users.d/profile.xml"], + with_minio=True, + stay_alive=True, + with_zookeeper=True, + keeper_required_feature_flags=["multi_read"], + ) + # node that does not participate in the export, but will have visibility over the s3 table + cluster.add_instance( + "watcher_node", + main_configs=["configs/named_collections.xml"], + user_configs=[], + with_minio=True, + ) + cluster.add_instance( + "replica_with_export_disabled", + main_configs=["configs/named_collections.xml", "configs/disable_experimental_export_partition.xml"], + user_configs=["configs/users.d/profile.xml"], + with_minio=True, + stay_alive=True, + with_zookeeper=True, + keeper_required_feature_flags=["multi_read"], + ) + # Sharded instances for filename pattern tests + cluster.add_instance( + "shard1_replica1", + main_configs=["configs/named_collections.xml", "configs/allow_experimental_export_partition.xml", "configs/macros_shard1_replica1.xml"], + user_configs=["configs/users.d/profile.xml"], + with_minio=True, + stay_alive=True, + with_zookeeper=True, + keeper_required_feature_flags=["multi_read"], + ) + + cluster.add_instance( + "shard2_replica1", + main_configs=["configs/named_collections.xml", "configs/allow_experimental_export_partition.xml", "configs/macros_shard2_replica1.xml"], + user_configs=["configs/users.d/profile.xml"], + with_minio=True, + stay_alive=True, + with_zookeeper=True, + keeper_required_feature_flags=["multi_read"], + ) + logging.info("Starting cluster...") + cluster.start() + yield cluster + finally: + cluster.shutdown() + + +@pytest.fixture(autouse=True) +def drop_tables_after_test(cluster): + """Drop all tables in the default database after every test. + + Without this, ReplicatedMergeTree tables from completed tests remain alive and keep + running ZooKeeper background threads (merge selector, queue log, cleanup, export manifest + updater). With many tables alive simultaneously the ZooKeeper session becomes overwhelmed + and subsequent tests start seeing operation-timeout / session-expired errors. + """ + yield + for instance_name, instance in cluster.instances.items(): + try: + tables_str = instance.query( + "SELECT name FROM system.tables WHERE database = 'default' FORMAT TabSeparated" + ).strip() + if not tables_str: + continue + for table in tables_str.split('\n'): + table = table.strip() + if table: + instance.query(f"DROP TABLE IF EXISTS default.`{table}` SYNC") + except Exception as e: + logging.warning(f"drop_tables_after_test: cleanup failed on {instance_name}: {e}") + + +def create_s3_table(node, s3_table): + node.query(f"CREATE TABLE {s3_table} (id UInt64, year UInt16) ENGINE = S3(s3_conn, filename='{s3_table}', format=Parquet, partition_strategy='hive') PARTITION BY year") + + +def create_tables_and_insert_data(node, mt_table, s3_table, replica_name): + node.query(f"DROP TABLE IF EXISTS {mt_table} SYNC") + # enable_block_number_column and enable_block_offset_column are needed for patch parts support + node.query(f"CREATE TABLE {mt_table} (id UInt64, year UInt16) ENGINE = ReplicatedMergeTree('/clickhouse/tables/{mt_table}', '{replica_name}') PARTITION BY year ORDER BY tuple() SETTINGS enable_block_number_column = 1, enable_block_offset_column = 1") + node.query(f"INSERT INTO {mt_table} VALUES (1, 2020), (2, 2020), (3, 2020), (4, 2021)") + + create_s3_table(node, s3_table) + + +def create_sharded_tables_and_insert_data(node, mt_table, s3_table, replica_name): + """Create sharded ReplicatedMergeTree table with {shard} macro in ZooKeeper path.""" + node.query(f"CREATE TABLE {mt_table} (id UInt64, year UInt16) ENGINE = ReplicatedMergeTree('/clickhouse/tables/{{shard}}/{mt_table}', '{replica_name}') PARTITION BY year ORDER BY tuple()") + node.query(f"INSERT INTO {mt_table} VALUES (1, 2020), (2, 2020), (3, 2020), (4, 2021)") + + create_s3_table(node, s3_table) + + +def test_restart_nodes_during_export(cluster): + skip_if_remote_database_disk_enabled(cluster) + node = cluster.instances["replica1"] + node2 = cluster.instances["replica2"] + watcher_node = cluster.instances["watcher_node"] + + postfix = str(uuid.uuid4()).replace("-", "_") + mt_table = f"disaster_mt_table_{postfix}" + s3_table = f"disaster_s3_table_{postfix}" + + create_tables_and_insert_data(node, mt_table, s3_table, "replica1") + create_tables_and_insert_data(node2, mt_table, s3_table, "replica2") + create_s3_table(watcher_node, s3_table) + + # Block S3/MinIO requests to keep exports alive via retry mechanism + # This allows ZooKeeper operations to proceed quickly + minio_ip = cluster.minio_ip + minio_port = cluster.minio_port + + with PartitionManager() as pm: + # Block responses from MinIO (source_port matches MinIO service) + pm_rule_reject_responses_node1 = { + "instance": node, + "destination": node.ip_address, + "protocol": "tcp", + "source_port": minio_port, + "action": "REJECT --reject-with tcp-reset", + } + pm.add_rule(pm_rule_reject_responses_node1) + + pm_rule_reject_responses_node2 = { + "instance": node2, + "destination": node2.ip_address, + "protocol": "tcp", + "source_port": minio_port, + "action": "REJECT --reject-with tcp-reset", + } + pm.add_rule(pm_rule_reject_responses_node2) + + # Block requests to MinIO (destination: MinIO, destination_port: minio_port) + pm_rule_reject_requests_node1 = { + "instance": node, + "destination": minio_ip, + "protocol": "tcp", + "destination_port": minio_port, + "action": "REJECT --reject-with tcp-reset", + } + pm.add_rule(pm_rule_reject_requests_node1) + + pm_rule_reject_requests_node2 = { + "instance": node2, + "destination": minio_ip, + "protocol": "tcp", + "destination_port": minio_port, + "action": "REJECT --reject-with tcp-reset", + } + pm.add_rule(pm_rule_reject_requests_node2) + + export_queries = f""" + ALTER TABLE {mt_table} + EXPORT PARTITION ID '2020' TO TABLE {s3_table}; + ALTER TABLE {mt_table} + EXPORT PARTITION ID '2021' TO TABLE {s3_table}; + """ + + node.query(export_queries) + + # wait for the exports to start + wait_for_export_to_start(node, mt_table, s3_table, "2020") + wait_for_export_to_start(node, mt_table, s3_table, "2021") + + node.stop_clickhouse(kill=True) + node2.stop_clickhouse(kill=True) + + assert watcher_node.query(f"SELECT count() FROM {s3_table} where year = 2020") == '0\n', "Partition 2020 was written to S3 during network delay crash" + + assert watcher_node.query(f"SELECT count() FROM {s3_table} where year = 2021") == '0\n', "Partition 2021 was written to S3 during network delay crash" + + # start the nodes, they should finish the export + node.start_clickhouse() + node2.start_clickhouse() + + wait_for_export_status(node, mt_table, s3_table, "2020", "COMPLETED") + wait_for_export_status(node, mt_table, s3_table, "2021", "COMPLETED") + + assert node.query(f"SELECT count() FROM {s3_table} WHERE year = 2020") != f'0\n', "Export of partition 2020 did not resume after crash" + + assert node.query(f"SELECT count() FROM {s3_table} WHERE year = 2021") != f'0\n', "Export of partition 2021 did not resume after crash" + + +def test_kill_export(cluster): + skip_if_remote_database_disk_enabled(cluster) + node = cluster.instances["replica1"] + node2 = cluster.instances["replica2"] + watcher_node = cluster.instances["watcher_node"] + + postfix = str(uuid.uuid4()).replace("-", "_") + mt_table = f"kill_export_mt_table_{postfix}" + s3_table = f"kill_export_s3_table_{postfix}" + + create_tables_and_insert_data(node, mt_table, s3_table, "replica1") + create_tables_and_insert_data(node2, mt_table, s3_table, "replica2") + + # Block S3/MinIO requests to keep exports alive via retry mechanism + # This allows ZooKeeper operations (KILL) to proceed quickly + minio_ip = cluster.minio_ip + minio_port = cluster.minio_port + + with PartitionManager() as pm: + # Block responses from MinIO (source_port matches MinIO service) + pm_rule_reject_responses = { + "instance": node, + "destination": node.ip_address, + "protocol": "tcp", + "source_port": minio_port, + "action": "REJECT --reject-with tcp-reset", + } + pm.add_rule(pm_rule_reject_responses) + + # Block requests to MinIO (destination: MinIO, destination_port: minio_port) + pm_rule_reject_requests = { + "instance": node, + "destination": minio_ip, + "protocol": "tcp", + "destination_port": minio_port, + "action": "REJECT --reject-with tcp-reset", + } + pm.add_rule(pm_rule_reject_requests) + + # Block responses from MinIO for node2 + pm_rule_reject_responses_node2 = { + "instance": node2, + "destination": node2.ip_address, + "protocol": "tcp", + "source_port": minio_port, + "action": "REJECT --reject-with tcp-reset", + } + pm.add_rule(pm_rule_reject_responses_node2) + + # Block requests to MinIO from node2 + pm_rule_reject_requests_node2 = { + "instance": node2, + "destination": minio_ip, + "protocol": "tcp", + "destination_port": minio_port, + "action": "REJECT --reject-with tcp-reset", + } + pm.add_rule(pm_rule_reject_requests_node2) + + export_queries = f""" + ALTER TABLE {mt_table} + EXPORT PARTITION ID '2020' TO TABLE {s3_table}; + ALTER TABLE {mt_table} + EXPORT PARTITION ID '2021' TO TABLE {s3_table}; + """ + + node.query(export_queries) + + # Kill only 2020 while S3 is blocked - retry mechanism keeps exports alive + # ZooKeeper operations (KILL) proceed quickly since only S3 is blocked + node.query(f"KILL EXPORT PARTITION WHERE partition_id = '2020' and source_table = '{mt_table}' and destination_table = '{s3_table}'") + + # sleep for a while to let the kill to be processed + time.sleep(2) + + # wait for 2021 to finish + wait_for_export_status(node, mt_table, s3_table, "2021", "COMPLETED") + + # checking for the commit file because maybe the data file was too fast? + assert node.query(f"SELECT count() FROM s3(s3_conn, filename='{s3_table}/commit_2020_*', format=LineAsString)") == '0\n', "Partition 2020 was written to S3, it was not killed as expected" + assert node.query(f"SELECT count() FROM s3(s3_conn, filename='{s3_table}/commit_2021_*', format=LineAsString)") != f'0\n', "Partition 2021 was not written to S3, but it should have been" + + # check system.replicated_partition_exports for the export, status should be KILLED + assert node.query(f"SELECT status FROM system.replicated_partition_exports WHERE partition_id = '2020' and source_table = '{mt_table}' and destination_table = '{s3_table}'") == 'KILLED\n', "Partition 2020 was not killed as expected" + assert node.query(f"SELECT status FROM system.replicated_partition_exports WHERE partition_id = '2021' and source_table = '{mt_table}' and destination_table = '{s3_table}'") == 'COMPLETED\n', "Partition 2021 was not completed, this is unexpected" + + # check the data did not land on s3 + assert node.query(f"SELECT count() FROM {s3_table} WHERE year = 2020") == '0\n', "Partition 2020 was written to S3, it was not killed as expected" + + +def test_kill_export_resilient_to_status_handling_failure(cluster): + """KILL EXPORT PARTITION must eventually take effect even when the first + attempt to handle the ZK status-change event throws (simulated via a ONCE + failpoint). The re-queue + reschedule mechanism retries after ~5 s and + the second attempt succeeds because the ONCE failpoint has already fired.""" + skip_if_remote_database_disk_enabled(cluster) + node = cluster.instances["replica1"] + + postfix = str(uuid.uuid4()).replace("-", "_") + mt_table = f"kill_resilient_mt_{postfix}" + s3_table = f"kill_resilient_s3_{postfix}" + + create_tables_and_insert_data(node, mt_table, s3_table, "replica1") + + minio_ip = cluster.minio_ip + minio_port = cluster.minio_port + + with PartitionManager() as pm: + pm.add_rule({ + "instance": node, + "destination": node.ip_address, + "protocol": "tcp", + "source_port": minio_port, + "action": "REJECT --reject-with tcp-reset", + }) + + pm.add_rule({ + "instance": node, + "destination": minio_ip, + "protocol": "tcp", + "destination_port": minio_port, + "action": "REJECT --reject-with tcp-reset", + }) + + node.query( + f"ALTER TABLE {mt_table} EXPORT PARTITION ID '2020' TO TABLE {s3_table}" + ) + + node.query("SYSTEM ENABLE FAILPOINT export_partition_status_change_throw") + + node.query( + f"KILL EXPORT PARTITION WHERE partition_id = '2020'" + f" AND source_table = '{mt_table}' AND destination_table = '{s3_table}'") + + # sleep for a while to let the kill to be processed + time.sleep(5) + + # The ONCE failpoint makes the first handleStatusChanges() throw. + # The catch re-queues the key and scheduleAfter(5000) arms a retry. + # Wait up to 15 s (5 s retry delay + margin) for the kill to propagate. + wait_for_export_status(node, mt_table, s3_table, "2020", "KILLED", timeout=15) + + assert ( + node.query( + f"SELECT status FROM system.replicated_partition_exports" + f" WHERE partition_id = '2020'" + f" AND source_table = '{mt_table}'" + f" AND destination_table = '{s3_table}'" + ).strip() == "KILLED" + ), "Export was not killed — status change was lost after the injected failure" + + +def test_drop_source_table_during_export(cluster): + skip_if_remote_database_disk_enabled(cluster) + node = cluster.instances["replica1"] + # node2 = cluster.instances["replica2"] + watcher_node = cluster.instances["watcher_node"] + + postfix = str(uuid.uuid4()).replace("-", "_") + mt_table = f"drop_source_table_during_export_mt_table_{postfix}" + s3_table = f"drop_source_table_during_export_s3_table_{postfix}" + + create_tables_and_insert_data(node, mt_table, s3_table, "replica1") + # create_tables_and_insert_data(node2, mt_table, s3_table, "replica2") + create_s3_table(watcher_node, s3_table) + + # Block S3/MinIO requests to keep exports alive via retry mechanism + # This allows ZooKeeper operations (KILL) to proceed quickly + minio_ip = cluster.minio_ip + minio_port = cluster.minio_port + + with PartitionManager() as pm: + # Block responses from MinIO (source_port matches MinIO service) + pm_rule_reject_responses = { + "instance": node, + "destination": node.ip_address, + "protocol": "tcp", + "source_port": minio_port, + "action": "REJECT --reject-with tcp-reset", + } + pm.add_rule(pm_rule_reject_responses) + + # Block requests to MinIO (destination: MinIO, destination_port: minio_port) + pm_rule_reject_requests = { + "instance": node, + "destination": minio_ip, + "protocol": "tcp", + "destination_port": minio_port, + "action": "REJECT --reject-with tcp-reset", + } + pm.add_rule(pm_rule_reject_requests) + + export_queries = f""" + ALTER TABLE {mt_table} + EXPORT PARTITION ID '2020' TO TABLE {s3_table} SETTINGS s3_retry_attempts = 500; + ALTER TABLE {mt_table} + EXPORT PARTITION ID '2021' TO TABLE {s3_table} SETTINGS s3_retry_attempts = 500; + """ + + node.query(export_queries) + + wait_for_export_status(node, mt_table, s3_table, "2020", "PENDING") + wait_for_export_status(node, mt_table, s3_table, "2021", "PENDING") + + # This should kill the background operations and drop the table + node.query(f"DROP TABLE {mt_table}") + + # Sleep some time to let the export finish (assuming it was not properly cancelled) + time.sleep(10) + + assert node.query(f"SELECT count() FROM s3(s3_conn, filename='{s3_table}/commit_*', format=LineAsString)") == '0\n', "Background operations completed even with the table dropped" + + +def test_concurrent_exports_to_different_targets(cluster): + node = cluster.instances["replica1"] + + postfix = str(uuid.uuid4()).replace("-", "_") + mt_table = f"concurrent_diff_targets_mt_table_{postfix}" + s3_table_a = f"concurrent_diff_targets_s3_a_{postfix}" + s3_table_b = f"concurrent_diff_targets_s3_b_{postfix}" + + create_tables_and_insert_data(node, mt_table, s3_table_a, "replica1") + create_s3_table(node, s3_table_b) + + # Launch two exports of the same partition to two different S3 tables concurrently + with PartitionManager() as pm: + pm.add_network_delay(node, delay_ms=1000) + + node.query( + f"ALTER TABLE {mt_table} EXPORT PARTITION ID '2020' TO TABLE {s3_table_a}" + ) + node.query( + f"ALTER TABLE {mt_table} EXPORT PARTITION ID '2020' TO TABLE {s3_table_b}" + ) + + wait_for_export_status(node, mt_table, s3_table_a, "2020", "COMPLETED") + wait_for_export_status(node, mt_table, s3_table_b, "2020", "COMPLETED") + + # Both targets should receive the same data independently + assert node.query(f"SELECT count() FROM {s3_table_a} WHERE year = 2020") == '3\n', "First target did not receive expected rows" + assert node.query(f"SELECT count() FROM {s3_table_b} WHERE year = 2020") == '3\n', "Second target did not receive expected rows" + + # And both should have a commit marker + assert node.query( + f"SELECT count() FROM s3(s3_conn, filename='{s3_table_a}/commit_2020_*', format=LineAsString)" + ) != '0\n', "Commit file missing for first target" + assert node.query( + f"SELECT count() FROM s3(s3_conn, filename='{s3_table_b}/commit_2020_*', format=LineAsString)" + ) != '0\n', "Commit file missing for second target" + + +def test_failure_is_logged_in_system_table(cluster): + skip_if_remote_database_disk_enabled(cluster) + node = cluster.instances["replica1"] + + postfix = str(uuid.uuid4()).replace("-", "_") + mt_table = f"failure_is_logged_in_system_table_mt_table_{postfix}" + s3_table = f"failure_is_logged_in_system_table_s3_table_{postfix}" + + create_tables_and_insert_data(node, mt_table, s3_table, "replica1") + + # Block traffic to/from MinIO to force upload errors and retries, following existing S3 tests style + minio_ip = cluster.minio_ip + minio_port = cluster.minio_port + + with PartitionManager() as pm: + # Block responses from MinIO (source_port matches MinIO service) + pm_rule_reject_responses = { + "instance": node, + "destination": node.ip_address, + "protocol": "tcp", + "source_port": minio_port, + "action": "REJECT --reject-with tcp-reset", + } + pm.add_rule(pm_rule_reject_responses) + + # Also block requests to MinIO (destination: MinIO, destination_port: 9001) with REJECT to fail fast + pm_rule_reject_requests = { + "instance": node, + "destination": minio_ip, + "protocol": "tcp", + "destination_port": minio_port, + "action": "REJECT --reject-with tcp-reset", + } + pm.add_rule(pm_rule_reject_requests) + + # Blocked MinIO produces transient (retryable) S3 errors. There is no retry + # budget anymore, so the task keeps retrying and is only torn down once the + # absolute task timeout fires (transitioning to KILLED). Use a small timeout + # so the test does not wait for the default (a day). + node.query( + f"ALTER TABLE {mt_table} EXPORT PARTITION ID '2020' TO TABLE {s3_table}" + f" SETTINGS export_merge_tree_partition_task_timeout_seconds = 5;" + ) + + # Wait for the timeout to kill the stuck task. The KILL is a Keeper operation + # (MinIO being blocked does not affect it); the status mirror needs roughly one + # manifest-updater poll cycle (~30s) plus watch propagation on top of the 5s + # timeout, so allow a generous budget. + wait_for_export_status(node, mt_table, s3_table, "2020", "KILLED", timeout=90) + + # Network restored; verify the export is marked as KILLED in the system table + # Also verify we captured at least one exception and no commit file exists + status = node.query( + f""" + SELECT status FROM system.replicated_partition_exports + WHERE source_table = '{mt_table}' + AND destination_table = '{s3_table}' + AND partition_id = '2020' + """ + ) + + assert status.strip() == "KILLED", f"Expected KILLED status, got: {status!r}" + + exception_count = node.query( + f""" + SELECT any(exception_count) FROM system.replicated_partition_exports + WHERE source_table = '{mt_table}' + AND destination_table = '{s3_table}' + AND partition_id = '2020' + """ + ) + assert int(exception_count.strip()) > 0, "Expected non-zero exception_count in system.replicated_partition_exports" + + # No commit should have been produced for this partition + assert node.query( + f"SELECT count() FROM s3(s3_conn, filename='{s3_table}/commit_2020_*', format=LineAsString)" + ) == '0\n', "Commit file exists despite forced S3 failures" + + +def test_inject_short_living_failures(cluster): + skip_if_remote_database_disk_enabled(cluster) + node = cluster.instances["replica1"] + + postfix = str(uuid.uuid4()).replace("-", "_") + mt_table = f"inject_short_living_failures_mt_table_{postfix}" + s3_table = f"inject_short_living_failures_s3_table_{postfix}" + + create_tables_and_insert_data(node, mt_table, s3_table, "replica1") + + # Block traffic to/from MinIO to force upload errors and retries, following existing S3 tests style + minio_ip = cluster.minio_ip + minio_port = cluster.minio_port + + with PartitionManager() as pm: + # Block responses from MinIO (source_port matches MinIO service) + pm_rule_reject_responses = { + "instance": node, + "destination": node.ip_address, + "protocol": "tcp", + "source_port": minio_port, + "action": "REJECT --reject-with tcp-reset", + } + pm.add_rule(pm_rule_reject_responses) + + # Also block requests to MinIO (destination: MinIO, destination_port: 9001) with REJECT to fail fast + pm_rule_reject_requests = { + "instance": node, + "destination": minio_ip, + "protocol": "tcp", + "destination_port": minio_port, + "action": "REJECT --reject-with tcp-reset", + } + pm.add_rule(pm_rule_reject_requests) + + # Transient (retryable) failures never fail the task on a budget; it keeps + # retrying until the network is restored and the export completes. + node.query( + f"ALTER TABLE {mt_table} EXPORT PARTITION ID '2020' TO TABLE {s3_table};" + ) + + # wait for at least one exception to occur, but not enough to finish the export. + # Use the helper default (>= one manifest-updater poll cycle): system.replicated_partition_exports + # is served from the in-memory mirror, and while the task stays PENDING the mirror only + # picks up new exception leaves on the next poll tick (~30s) — see helper docstring. + wait_for_exception_count(node, mt_table, s3_table, "2020", min_exception_count=1) + + # wait for the export to finish + wait_for_export_status(node, mt_table, s3_table, "2020", "COMPLETED") + + # Assert the export succeeded + assert node.query(f"SELECT count() FROM {s3_table} WHERE year = 2020") == '3\n', "Export did not succeed" + assert node.query(f"SELECT count() FROM s3(s3_conn, filename='{s3_table}/commit_2020_*', format=LineAsString)") == '1\n', "Export did not succeed" + + # check system.replicated_partition_exports for the export + assert node.query( + f""" + SELECT status FROM system.replicated_partition_exports + WHERE source_table = '{mt_table}' + AND destination_table = '{s3_table}' + AND partition_id = '2020' + """ + ) == "COMPLETED\n", "Export should be marked as COMPLETED" + + exception_count = node.query( + f""" + SELECT exception_count FROM system.replicated_partition_exports + WHERE source_table = '{mt_table}' + AND destination_table = '{s3_table}' + AND partition_id = '2020' + """ + ) + assert int(exception_count.strip()) >= 1, "Expected at least one exception" + + +def test_export_partition_retry_backoff(cluster): + """Verify the per-replica in-memory exponential back-off between failed part exports. + + The back-off is local in-memory state (no ZooKeeper retry_count / next_retry_time + anymore), so it is not directly observable; instead we observe its effect. With a + large back-off, a part that keeps failing (object storage blocked) is parked for the + back-off window after its first failure and must NOT be retried on every ~5s + scheduler tick. We assert that exception_count stays low across a window that spans + several ticks. Once the network is restored and the back-off elapses, the export + completes (there is no retry budget to exhaust).""" + skip_if_remote_database_disk_enabled(cluster) + node = cluster.instances["replica1"] + + postfix = str(uuid.uuid4()).replace("-", "_") + mt_table = f"retry_backoff_mt_table_{postfix}" + s3_table = f"retry_backoff_s3_table_{postfix}" + + create_tables_and_insert_data(node, mt_table, s3_table, "replica1") + + # Large back-off so a single failed attempt parks the part well beyond the + # ~5s scheduler tick. Kept moderate so the export can still complete promptly + # once the network is restored. + initial_backoff_seconds = 30 + max_backoff_seconds = 30 + + minio_ip = cluster.minio_ip + minio_port = cluster.minio_port + + with PartitionManager() as pm: + # Block responses from MinIO (source_port matches MinIO service) + pm.add_rule({ + "instance": node, + "destination": node.ip_address, + "protocol": "tcp", + "source_port": minio_port, + "action": "REJECT --reject-with tcp-reset", + }) + # Also block requests to MinIO to fail fast + pm.add_rule({ + "instance": node, + "destination": minio_ip, + "protocol": "tcp", + "destination_port": minio_port, + "action": "REJECT --reject-with tcp-reset", + }) + + node.query( + f"ALTER TABLE {mt_table} EXPORT PARTITION ID '2020' TO TABLE {s3_table} " + f"SETTINGS export_merge_tree_partition_retry_initial_backoff_seconds = {initial_backoff_seconds}, " + f"export_merge_tree_partition_retry_max_backoff_seconds = {max_backoff_seconds}" + ) + + # Wait until the first failure is recorded. + count_after_first = wait_for_exception_count( + node, mt_table, s3_table, "2020", min_exception_count=1, timeout=60 + ) + + # While the part is backing off (~30s) it must not be retried again. Observe + # across a window that spans several scheduler ticks: without back-off the + # ~5s tick would add roughly five more failures, so a small increase proves + # the back-off is pacing retries. + time.sleep(25) + count_during_backoff = int(node.query( + f"SELECT exception_count FROM system.replicated_partition_exports" + f" WHERE source_table = '{mt_table}'" + f" AND destination_table = '{s3_table}'" + f" AND partition_id = '2020'" + ).strip()) + assert count_during_backoff - count_after_first <= 2, ( + f"exception_count jumped during the back-off window: " + f"{count_after_first} -> {count_during_backoff}; back-off was not applied" + ) + + # Network restored; once the back-off elapses the export should complete because + # there is no retry budget to exhaust. + wait_for_export_status(node, mt_table, s3_table, "2020", "COMPLETED", timeout=120) + assert node.query(f"SELECT count() FROM {s3_table} WHERE year = 2020") == "3\n", "Export did not succeed" + + +def test_export_partition_file_already_exists_policy(cluster): + node = cluster.instances["replica1"] + + postfix = str(uuid.uuid4()).replace("-", "_") + mt_table = f"export_partition_file_already_exists_policy_mt_table_{postfix}" + s3_table = f"export_partition_file_already_exists_policy_s3_table_{postfix}" + + create_tables_and_insert_data(node, mt_table, s3_table, "replica1") + + # stop merges so part names remain stable. it is important for the test. + node.query(f"SYSTEM STOP MERGES {mt_table}") + + # Export all parts + node.query( + f"ALTER TABLE {mt_table} EXPORT PARTITION ID '2020' TO TABLE {s3_table}", + ) + + # check system.replicated_partition_exports for the export + assert node.query( + f""" + SELECT status FROM system.replicated_partition_exports + WHERE source_table = '{mt_table}' + AND destination_table = '{s3_table}' + AND partition_id = '2020' + """ + ) == "COMPLETED\n", "Export should be marked as COMPLETED" + + # wait for the exports to finish + wait_for_export_status(node, mt_table, s3_table, "2020", "COMPLETED") + + # plain object storage destinations surface the commit marker file path via + # system.replicated_partition_exports.committed_marker_file + committed_marker_file = node.query( + f""" + SELECT committed_marker_file FROM system.replicated_partition_exports + WHERE source_table = '{mt_table}' + AND destination_table = '{s3_table}' + AND partition_id = '2020' + """ + ).strip() + # `committed_marker_file` is the absolute key in the bucket (same convention as + # `destination_file_paths`); it may carry the s3_conn URL's in-bucket prefix on + # top of the table's `filename` argument, so use a "contains" check that does + # not depend on knowing that prefix. + assert f"{s3_table}/commit_2020_" in committed_marker_file, \ + f"Expected committed_marker_file under {s3_table}/, got: {committed_marker_file!r}" + # Path relative to the `s3_conn` URL, derived from the absolute key without + # assuming a particular URL prefix. + marker_relative_path = committed_marker_file[committed_marker_file.index(f"{s3_table}/"):] + assert node.query( + f"SELECT count() FROM s3(s3_conn, filename='{marker_relative_path}', format=LineAsString)" + ) == '1\n', f"Commit marker file does not exist at {committed_marker_file!r}" + + # try to export the partition + node.query( + f"ALTER TABLE {mt_table} EXPORT PARTITION ID '2020' TO TABLE {s3_table} SETTINGS export_merge_tree_partition_force_export=1" + ) + + wait_for_export_status(node, mt_table, s3_table, "2020", "COMPLETED") + + assert node.query( + f""" + SELECT count() FROM system.replicated_partition_exports + WHERE source_table = '{mt_table}' + AND destination_table = '{s3_table}' + AND partition_id = '2020' + AND status = 'COMPLETED' + """ + ) == '1\n', "Expected the export to be marked as COMPLETED" + + # overwrite policy + node.query( + f"ALTER TABLE {mt_table} EXPORT PARTITION ID '2020' TO TABLE {s3_table} SETTINGS export_merge_tree_partition_force_export=1, export_merge_tree_part_file_already_exists_policy='overwrite'" + ) + + # wait for the export to finish + wait_for_export_status(node, mt_table, s3_table, "2020", "COMPLETED") + + # check system.replicated_partition_exports for the export + # ideally we would make sure the transaction id is different, but I do not have the time to do that now + assert node.query( + f""" + SELECT count() FROM system.replicated_partition_exports + WHERE source_table = '{mt_table}' + AND destination_table = '{s3_table}' + AND partition_id = '2020' + AND status = 'COMPLETED' + """ + ) == '1\n', "Expected the export to be marked as COMPLETED" + + # last but not least, let's try with the error policy. FILE_ALREADY_EXISTS is a + # non-retryable error (retrying always hits the same existing file), so the task + # fails fast without needing a retry budget. + node.query( + f"ALTER TABLE {mt_table} EXPORT PARTITION ID '2020' TO TABLE {s3_table} SETTINGS export_merge_tree_partition_force_export=1, export_merge_tree_part_file_already_exists_policy='error'", + ) + + # wait for the export to finish + wait_for_export_status(node, mt_table, s3_table, "2020", "FAILED") + + # check system.replicated_partition_exports for the export + assert node.query( + f""" + SELECT count() FROM system.replicated_partition_exports + WHERE source_table = '{mt_table}' + AND destination_table = '{s3_table}' + AND partition_id = '2020' + AND status = 'FAILED' + """ + ) == '1\n', "Expected the export to be marked as FAILED" + + +def export_transaction_id(node, mt_table, s3_table): + return node.query( + f""" + SELECT transaction_id FROM system.replicated_partition_exports + WHERE source_table = '{mt_table}' + AND destination_table = '{s3_table}' + AND partition_id = '2020' + """ + ).strip() + + +def wait_for_new_export_transaction(node, mt_table, s3_table, previous_transaction_id, timeout=60): + """Wait until the export entry carries a transaction id other than *previous_transaction_id*. + + A force re-export replaces the entry. Without this wait, the COMPLETED status of the export + being replaced can still be visible in the in-memory mirror and satisfy a status wait + immediately, before the new export has even started. + """ + start_time = time.time() + last_transaction_id = None + while time.time() - start_time < timeout: + last_transaction_id = export_transaction_id(node, mt_table, s3_table) + if last_transaction_id and last_transaction_id != previous_transaction_id: + return last_transaction_id + time.sleep(0.2) + + raise TimeoutError( + f"Export transaction id did not change from {previous_transaction_id!r} within {timeout}s. " + f"Last seen: {last_transaction_id!r}" + ) + + +def create_split_export_tables(node, mt_table, s3_table, replica_name): + """Create a source table whose part splits into one destination file per row on export. + + `export_merge_tree_part_max_rows_per_file` is evaluated once per chunk rather than per row + (see `MultiFileStorageObjectStorageSink::consume`), and `MergeTreeSequentialSource` emits one + chunk per index granule, so a part can only split at granule boundaries. With the default + granularity a small part is a single granule and never splits at all, hence + `index_granularity = 1`. `index_granularity_bytes = 0` disables adaptive granularity, which + would otherwise choose the granule size itself. + """ + node.query(f"DROP TABLE IF EXISTS {mt_table} SYNC") + node.query( + f"CREATE TABLE {mt_table} (id UInt64, year UInt16) " + f"ENGINE = ReplicatedMergeTree('/clickhouse/tables/{mt_table}', '{replica_name}') " + f"PARTITION BY year ORDER BY tuple() " + f"SETTINGS index_granularity = 1, index_granularity_bytes = 0" + ) + node.query(f"INSERT INTO {mt_table} VALUES (1, 2020), (2, 2020), (3, 2020), (4, 2021)") + + create_s3_table(node, s3_table) + + +def export_partition_split_into_files( + node, mt_table, s3_table, force=False, policy=None, previous_transaction_id=None +): + """Export partition 2020 with one row per destination file and wait for completion. + + Only splits per row for a table built by `create_split_export_tables`. + """ + settings = ["export_merge_tree_part_max_rows_per_file = 1"] + if force: + settings.append("export_merge_tree_partition_force_export = 1") + if policy: + settings.append(f"export_merge_tree_part_file_already_exists_policy = '{policy}'") + + node.query( + f"ALTER TABLE {mt_table} EXPORT PARTITION ID '2020' TO TABLE {s3_table} " + f"SETTINGS {', '.join(settings)}" + ) + + if previous_transaction_id is not None: + wait_for_new_export_transaction(node, mt_table, s3_table, previous_transaction_id) + + wait_for_export_status(node, mt_table, s3_table, "2020", "COMPLETED") + + +def recorded_export_paths(node, mt_table, s3_table): + """Destination file paths recorded for the exported parts, in the order the sink wrote them. + + Mirrors the `/processed//paths_in_destination` data in ZooKeeper, which + is what the commit phase turns into the partition commit marker. + """ + paths = node.query( + f""" + SELECT arrayJoin(arrayFlatten(mapValues(destination_file_paths))) + FROM system.replicated_partition_exports + WHERE source_table = '{mt_table}' + AND destination_table = '{s3_table}' + AND partition_id = '2020' + """ + ) + return [path for path in paths.splitlines() if path] + + +def partition_commit_marker_lines(node, mt_table, s3_table): + """Data-file paths listed inside the partition-level commit marker.""" + committed_marker_file = node.query( + f""" + SELECT committed_marker_file FROM system.replicated_partition_exports + WHERE source_table = '{mt_table}' + AND destination_table = '{s3_table}' + AND partition_id = '2020' + """ + ).strip() + assert f"{s3_table}/commit_2020_" in committed_marker_file, \ + f"Expected committed_marker_file under {s3_table}/, got: {committed_marker_file!r}" + + # Path relative to the `s3_conn` URL, derived from the absolute key without assuming the + # URL's in-bucket prefix. + marker_relative_path = committed_marker_file[committed_marker_file.index(f"{s3_table}/"):] + lines = node.query( + f"SELECT * FROM s3(s3_conn, filename='{marker_relative_path}', format=LineAsString)" + ) + return [line for line in lines.splitlines() if line] + + +def list_partition_directory(cluster, data_path): + """Object keys sitting next to *data_path*, split into data files and commit markers. + + The per-part commit marker is written by `MultiFileStorageObjectStorageSink::commit` in the + same directory as the data files, named `commit_`. + """ + directory = data_path.rsplit("/", 1)[0] + "/" + object_names = sorted( + obj.object_name + for obj in cluster.minio_client.list_objects( + cluster.minio_bucket, prefix=directory, recursive=True + ) + ) + data_files = [n for n in object_names if not n.rsplit("/", 1)[-1].startswith("commit_")] + markers = [n for n in object_names if n.rsplit("/", 1)[-1].startswith("commit_")] + return data_files, markers + + +def test_export_partition_skip_policy_reports_every_split_file(cluster): + """A `skip` re-export of an already-exported multi-file part must record every destination + file, not just the first one. + + The recorded list is what the commit phase turns into the partition commit marker, so + dropping the later split files from it misrepresents the export even though the data is all + there. + """ + node = cluster.instances["replica1"] + + postfix = str(uuid.uuid4()).replace("-", "_") + mt_table = f"skip_reports_all_files_mt_table_{postfix}" + s3_table = f"skip_reports_all_files_s3_table_{postfix}" + + create_split_export_tables(node, mt_table, s3_table, "replica1") + # The destination file name is derived from the part name, so part names have to stay stable + # across the two exports, otherwise the second one writes to fresh paths and skips nothing. + node.query(f"SYSTEM STOP MERGES {mt_table}") + + export_partition_split_into_files(node, mt_table, s3_table) + first_transaction_id = export_transaction_id(node, mt_table, s3_table) + + exported_paths = recorded_export_paths(node, mt_table, s3_table) + assert len(exported_paths) == 3, \ + f"Expected the 3-row partition to split into 3 files, got {exported_paths}" + assert len(partition_commit_marker_lines(node, mt_table, s3_table)) == 3 + + # Re-export. Every destination file is already there, so `skip` short-circuits the part -- + # but it must do so with the complete file list. + export_partition_split_into_files( + node, mt_table, s3_table, force=True, policy="skip", + previous_transaction_id=first_transaction_id, + ) + + skipped_paths = recorded_export_paths(node, mt_table, s3_table) + assert sorted(skipped_paths) == sorted(exported_paths), ( + f"Skipped re-export recorded {skipped_paths} instead of all 3 split files {exported_paths}" + ) + + committed = partition_commit_marker_lines(node, mt_table, s3_table) + assert len(committed) == 3, \ + f"Skipped re-export committed {len(committed)} path(s) instead of all 3 split files: {committed}" + + +def test_export_partition_skip_policy_reexports_incomplete_part(cluster): + """A part whose multi-file export was interrupted must be re-exported in full under `skip`. + + The first split file existing proves nothing on its own: only the per-part commit marker, + written after the last file is finalized, proves the part was fully exported. Removing the + trailing files together with the marker reproduces what an attempt that died mid-part leaves + behind, and the retry has to rewrite them -- the rows in those files are produced by no other + attempt. + """ + node = cluster.instances["replica1"] + + postfix = str(uuid.uuid4()).replace("-", "_") + mt_table = f"skip_reexports_partial_mt_table_{postfix}" + s3_table = f"skip_reexports_partial_s3_table_{postfix}" + + create_split_export_tables(node, mt_table, s3_table, "replica1") + node.query(f"SYSTEM STOP MERGES {mt_table}") + + export_partition_split_into_files(node, mt_table, s3_table) + first_transaction_id = export_transaction_id(node, mt_table, s3_table) + + written_in_order = recorded_export_paths(node, mt_table, s3_table) + assert len(written_in_order) == 3, \ + f"Expected the 3-row partition to split into 3 files, got {written_in_order}" + + data_files, markers = list_partition_directory(cluster, written_in_order[0]) + assert data_files == sorted(written_in_order), \ + f"Objects in the partition directory {data_files} do not match the recorded paths {written_in_order}" + assert len(markers) == 1, f"Expected one per-part commit marker, got {markers}" + + # Roll the destination back to "first file finalized, nothing else": drop the trailing files + # and the marker that would otherwise prove the part complete. + for key in written_in_order[1:] + markers: + cluster.minio_client.remove_object(cluster.minio_bucket, key) + + surviving_data_files, surviving_markers = list_partition_directory(cluster, written_in_order[0]) + assert surviving_data_files == [written_in_order[0]], \ + f"Expected only the first split file to remain, got {surviving_data_files}" + assert surviving_markers == [], \ + f"Expected the per-part commit marker to be gone, got {surviving_markers}" + + export_partition_split_into_files( + node, mt_table, s3_table, force=True, policy="skip", + previous_transaction_id=first_transaction_id, + ) + + data_files_after, markers_after = list_partition_directory(cluster, written_in_order[0]) + assert len(data_files_after) == 3, ( + f"Retry left the part partially exported: {data_files_after} " + f"(the interrupted attempt's missing files were never rewritten)" + ) + assert len(markers_after) == 1, \ + f"Retry did not rewrite the per-part commit marker: {markers_after}" + assert node.query(f"SELECT count() FROM {s3_table} WHERE year = 2020") == "3\n", \ + "Rows from the split files the interrupted attempt never wrote are missing from the destination" + assert len(partition_commit_marker_lines(node, mt_table, s3_table)) == 3 + + +def test_export_partition_feature_is_disabled(cluster): + replica_with_export_disabled = cluster.instances["replica_with_export_disabled"] + + postfix = str(uuid.uuid4()).replace("-", "_") + mt_table = f"export_partition_feature_is_disabled_mt_table_{postfix}" + s3_table = f"export_partition_feature_is_disabled_s3_table_{postfix}" + + create_tables_and_insert_data(replica_with_export_disabled, mt_table, s3_table, "replica1") + + error = replica_with_export_disabled.query_and_get_error(f"ALTER TABLE {mt_table} EXPORT PARTITION ID '2020' TO TABLE {s3_table};") + assert "experimental" in error, "Expected error about disabled feature" + + # make sure kill operation also throws + error = replica_with_export_disabled.query_and_get_error(f"KILL EXPORT PARTITION WHERE partition_id = '2020' and source_table = '{mt_table}' and destination_table = '{s3_table}'") + assert "experimental" in error, "Expected error about disabled feature" + + +def test_export_partition_permissions(cluster): + """Test that export partition validates permissions correctly: + - User needs ALTER permission on source table + - User needs INSERT permission on destination table + """ + node = cluster.instances["replica1"] + + postfix = str(uuid.uuid4()).replace("-", "_") + mt_table = f"permissions_mt_table_{postfix}" + s3_table = f"permissions_s3_table_{postfix}" + + # Create tables as default user + create_tables_and_insert_data(node, mt_table, s3_table, "replica1") + + # Create test users with specific permissions + node.query("CREATE USER IF NOT EXISTS user_no_alter IDENTIFIED WITH no_password") + node.query("CREATE USER IF NOT EXISTS user_no_insert IDENTIFIED WITH no_password") + node.query("CREATE USER IF NOT EXISTS user_with_permissions IDENTIFIED WITH no_password") + + # Grant basic access to all users + node.query(f"GRANT SELECT ON {mt_table} TO user_no_alter") + node.query(f"GRANT SELECT ON {s3_table} TO user_no_alter") + + # user_no_insert has ALTER on source but no INSERT on destination + node.query(f"GRANT ALTER ON {mt_table} TO user_no_insert") + node.query(f"GRANT SELECT ON {s3_table} TO user_no_insert") + + # user_with_permissions has both ALTER and INSERT + node.query(f"GRANT ALTER ON {mt_table} TO user_with_permissions") + node.query(f"GRANT INSERT ON {s3_table} TO user_with_permissions") + + # Test 1: User without ALTER permission should fail + error = node.query_and_get_error( + f"ALTER TABLE {mt_table} EXPORT PARTITION ID '2020' TO TABLE {s3_table}", + user="user_no_alter" + ) + + assert "ACCESS_DENIED" in error or "Not enough privileges" in error, \ + f"Expected ACCESS_DENIED error for user without ALTER, got: {error}" + + # Test 2: User with ALTER but without INSERT permission should fail + error = node.query_and_get_error( + f"ALTER TABLE {mt_table} EXPORT PARTITION ID '2020' TO TABLE {s3_table}", + user="user_no_insert" + ) + + assert "ACCESS_DENIED" in error or "Not enough privileges" in error, \ + f"Expected ACCESS_DENIED error for user without INSERT, got: {error}" + + # Test 3: User with both ALTER and INSERT should succeed + node.query( + f"ALTER TABLE {mt_table} EXPORT PARTITION ID '2020' TO TABLE {s3_table}", + user="user_with_permissions" + ) + + # Wait for export to complete + wait_for_export_status(node, mt_table, s3_table, "2020", "COMPLETED") + + # Verify the export succeeded + result = node.query(f"SELECT count() FROM {s3_table} WHERE year = 2020") + assert result.strip() == "3", f"Expected 3 rows exported, got: {result}" + + # Verify system table shows COMPLETED status + status = node.query( + f""" + SELECT status FROM system.replicated_partition_exports + WHERE source_table = '{mt_table}' + AND destination_table = '{s3_table}' + AND partition_id = '2020' + """ + ) + assert status.strip() == "COMPLETED", f"Expected COMPLETED status, got: {status}" + + +# assert multiple exports within a single query are executed. They all share the same query id +# and previously the transaction id was the query id, which would cause problems +def test_multiple_exports_within_a_single_query(cluster): + node = cluster.instances["replica1"] + + postfix = str(uuid.uuid4()).replace("-", "_") + mt_table = f"multiple_exports_within_a_single_query_mt_table_{postfix}" + s3_table = f"multiple_exports_within_a_single_query_s3_table_{postfix}" + + create_tables_and_insert_data(node, mt_table, s3_table, "replica1") + + node.query(f"ALTER TABLE {mt_table} EXPORT PARTITION ID '2020' TO TABLE {s3_table}, EXPORT PARTITION ID '2021' TO TABLE {s3_table};") + + wait_for_export_status(node, mt_table, s3_table, "2020", "COMPLETED") + wait_for_export_status(node, mt_table, s3_table, "2021", "COMPLETED") + + # assert the exports have been executed + assert node.query(f"SELECT count() FROM {s3_table} WHERE year = 2020") == '3\n', "Export did not succeed" + assert node.query(f"SELECT count() FROM {s3_table} WHERE year = 2021") == '1\n', "Export did not succeed" + + # check system.replicated_partition_exports for the exports + assert node.query( + f""" + SELECT status FROM system.replicated_partition_exports + WHERE source_table = '{mt_table}' + AND destination_table = '{s3_table}' + AND partition_id = '2020' + """ + ) == "COMPLETED\n", "Export should be marked as COMPLETED" + + assert node.query( + f""" + SELECT status FROM system.replicated_partition_exports + WHERE source_table = '{mt_table}' + AND destination_table = '{s3_table}' + AND partition_id = '2021' + """ + ) == "COMPLETED\n", "Export should be marked as COMPLETED" + + +def test_pending_mutations_throw_before_export_partition(cluster): + """Test that pending mutations before export partition throw an error.""" + node = cluster.instances["replica1"] + + postfix = str(uuid.uuid4()).replace("-", "_") + mt_table = f"pending_mutations_throw_partition_mt_table_{postfix}" + s3_table = f"pending_mutations_throw_partition_s3_table_{postfix}" + + create_tables_and_insert_data(node, mt_table, s3_table, "replica1") + + node.query(f"SYSTEM STOP MERGES {mt_table}") + + node.query(f"ALTER TABLE {mt_table} UPDATE id = id + 100 WHERE year = 2020") + + mutations = node.query(f"SELECT count() FROM system.mutations WHERE table = '{mt_table}' AND is_done = 0") + assert mutations.strip() != '0', "Mutation should be pending" + + error = node.query_and_get_error( + f"ALTER TABLE {mt_table} EXPORT PARTITION ID '2020' TO TABLE {s3_table} " + f"SETTINGS export_merge_tree_part_throw_on_pending_mutations=true" + ) + + assert "PENDING_MUTATIONS_NOT_ALLOWED" in error, f"Expected error about pending mutations, got: {error}" + + +def test_pending_mutations_skip_before_export_partition(cluster): + """Test that pending mutations before export partition are skipped with throw_on_pending_mutations=false.""" + node = cluster.instances["replica1"] + + postfix = str(uuid.uuid4()).replace("-", "_") + mt_table = f"pending_mutations_skip_partition_mt_table_{postfix}" + s3_table = f"pending_mutations_skip_partition_s3_table_{postfix}" + + create_tables_and_insert_data(node, mt_table, s3_table, "replica1") + + node.query(f"SYSTEM STOP MERGES {mt_table}") + + node.query(f"ALTER TABLE {mt_table} UPDATE id = id + 100 WHERE year = 2020") + + mutations = node.query(f"SELECT count() FROM system.mutations WHERE table = '{mt_table}' AND is_done = 0") + assert mutations.strip() != '0', "Mutation should be pending" + + node.query( + f"ALTER TABLE {mt_table} EXPORT PARTITION ID '2020' TO TABLE {s3_table} " + f"SETTINGS export_merge_tree_part_throw_on_pending_mutations=false" + ) + + wait_for_export_status(node, mt_table, s3_table, "2020", "COMPLETED") + + result = node.query(f"SELECT id FROM {s3_table} WHERE year = 2020 ORDER BY id") + assert "101" not in result and "102" not in result and "103" not in result, \ + "Export should contain original data before mutation" + assert "1\n2\n3" in result, "Export should contain original data" + + +def test_pending_patch_parts_throw_before_export_partition(cluster): + """Test that pending patch parts before export partition throw an error with default settings.""" + node = cluster.instances["replica1"] + + postfix = str(uuid.uuid4()).replace("-", "_") + mt_table = f"pending_patches_throw_partition_mt_table_{postfix}" + s3_table = f"pending_patches_throw_partition_s3_table_{postfix}" + + create_tables_and_insert_data(node, mt_table, s3_table, "replica1") + + node.query(f"SYSTEM STOP MERGES {mt_table}") + + node.query(f"UPDATE {mt_table} SET id = id + 100 WHERE year = 2020") + + error = node.query_and_get_error( + f"ALTER TABLE {mt_table} EXPORT PARTITION ID '2020' TO TABLE {s3_table}" + ) + + node.query(f"DROP TABLE {mt_table}") + + assert "PENDING_MUTATIONS_NOT_ALLOWED" in error or "pending patch parts" in error.lower(), \ + f"Expected error about pending patch parts, got: {error}" + + +def test_pending_patch_parts_skip_before_export_partition(cluster): + """Test that pending patch parts before export partition are skipped with throw_on_pending_patch_parts=false.""" + node = cluster.instances["replica1"] + + postfix = str(uuid.uuid4()).replace("-", "_") + mt_table = f"pending_patches_skip_partition_mt_table_{postfix}" + s3_table = f"pending_patches_skip_partition_s3_table_{postfix}" + + create_tables_and_insert_data(node, mt_table, s3_table, "replica1") + + node.query(f"SYSTEM STOP MERGES {mt_table}") + + node.query(f"UPDATE {mt_table} SET id = id + 100 WHERE year = 2020") + + node.query( + f"ALTER TABLE {mt_table} EXPORT PARTITION ID '2020' TO TABLE {s3_table} " + f"SETTINGS export_merge_tree_part_throw_on_pending_patch_parts=false" + ) + + wait_for_export_status(node, mt_table, s3_table, "2020", "COMPLETED") + + result = node.query(f"SELECT id FROM {s3_table} WHERE year = 2020 ORDER BY id") + assert "1\n2\n3" in result, "Export should contain original data before patch" + + node.query(f"DROP TABLE {mt_table}") + + +def test_mutations_after_export_partition_started(cluster): + """Test that mutations applied after export partition starts don't affect the exported data.""" + skip_if_remote_database_disk_enabled(cluster) + node = cluster.instances["replica1"] + + postfix = str(uuid.uuid4()).replace("-", "_") + mt_table = f"mutations_after_export_partition_mt_table_{postfix}" + s3_table = f"mutations_after_export_partition_s3_table_{postfix}" + + create_tables_and_insert_data(node, mt_table, s3_table, "replica1") + + # Block traffic to MinIO to delay export + minio_ip = cluster.minio_ip + minio_port = cluster.minio_port + + with PartitionManager() as pm: + pm_rule_reject_responses = { + "instance": node, + "destination": node.ip_address, + "protocol": "tcp", + "source_port": minio_port, + "action": "REJECT --reject-with tcp-reset", + } + pm.add_rule(pm_rule_reject_responses) + + pm_rule_reject_requests = { + "instance": node, + "destination": minio_ip, + "protocol": "tcp", + "destination_port": minio_port, + "action": "REJECT --reject-with tcp-reset", + } + pm.add_rule(pm_rule_reject_requests) + + node.query( + f"ALTER TABLE {mt_table} EXPORT PARTITION ID '2020' TO TABLE {s3_table} " + f"SETTINGS export_merge_tree_part_throw_on_pending_mutations=true" + ) + + # Wait for export to start + wait_for_export_to_start(node, mt_table, s3_table, "2020") + + node.query(f"ALTER TABLE {mt_table} UPDATE id = id + 100 WHERE year = 2020") + + wait_for_export_status(node, mt_table, s3_table, "2020", "COMPLETED") + + result = node.query(f"SELECT id FROM {s3_table} WHERE year = 2020 ORDER BY id") + assert "1\n2\n3" in result, "Export should contain original data before mutation" + assert "101" not in result, "Export should not contain mutated data" + + +def test_patch_parts_after_export_partition_started(cluster): + """Test that patch parts created after export partition starts don't affect the exported data.""" + skip_if_remote_database_disk_enabled(cluster) + node = cluster.instances["replica1"] + + postfix = str(uuid.uuid4()).replace("-", "_") + mt_table = f"patches_after_export_partition_mt_table_{postfix}" + s3_table = f"patches_after_export_partition_s3_table_{postfix}" + + create_tables_and_insert_data(node, mt_table, s3_table, "replica1") + + # Block traffic to MinIO to delay export + minio_ip = cluster.minio_ip + minio_port = cluster.minio_port + + with PartitionManager() as pm: + pm_rule_reject_responses = { + "instance": node, + "destination": node.ip_address, + "protocol": "tcp", + "source_port": minio_port, + "action": "REJECT --reject-with tcp-reset", + } + pm.add_rule(pm_rule_reject_responses) + + pm_rule_reject_requests = { + "instance": node, + "destination": minio_ip, + "protocol": "tcp", + "destination_port": minio_port, + "action": "REJECT --reject-with tcp-reset", + } + pm.add_rule(pm_rule_reject_requests) + + node.query( + f"ALTER TABLE {mt_table} EXPORT PARTITION ID '2020' TO TABLE {s3_table}" + ) + + # Wait for export to start + wait_for_export_to_start(node, mt_table, s3_table, "2020") + + node.query(f"UPDATE {mt_table} SET id = id + 100 WHERE year = 2020") + + wait_for_export_status(node, mt_table, s3_table, "2020", "COMPLETED") + + result = node.query(f"SELECT id FROM {s3_table} WHERE year = 2020 ORDER BY id") + assert "1\n2\n3" in result, "Export should contain original data before patch" + assert "101" not in result, "Export should not contain patched data" + + node.query(f"DROP TABLE {mt_table}") + + +def test_mutation_in_partition_clause(cluster): + """Test that mutations limited to specific partitions using IN PARTITION clause + allow exports of unaffected partitions to succeed.""" + node = cluster.instances["replica1"] + + postfix = str(uuid.uuid4()).replace("-", "_") + mt_table = f"mutation_in_partition_clause_mt_table_{postfix}" + s3_table = f"mutation_in_partition_clause_s3_table_{postfix}" + + create_tables_and_insert_data(node, mt_table, s3_table, "replica1") + + node.query(f"SYSTEM STOP MERGES {mt_table}") + + # Issue a mutation that uses IN PARTITION to limit it to partition 2020 + node.query(f"ALTER TABLE {mt_table} UPDATE id = id + 100 IN PARTITION '2020' WHERE year = 2020") + + # Verify mutation is pending for 2020 + mutations = node.query( + f"SELECT count() FROM system.mutations WHERE table = '{mt_table}' AND is_done = 0" + ) + assert mutations.strip() != '0', "Mutation should be pending" + + # Export of 2020 should fail (it has pending mutations) + error = node.query_and_get_error( + f"ALTER TABLE {mt_table} EXPORT PARTITION ID '2020' TO TABLE {s3_table} " + f"SETTINGS export_merge_tree_part_throw_on_pending_mutations=true" + ) + assert "PENDING_MUTATIONS_NOT_ALLOWED" in error, f"Expected error about pending mutations for partition 2020, got: {error}" + + # Export of 2021 should succeed (no mutations affecting it) + node.query( + f"ALTER TABLE {mt_table} EXPORT PARTITION ID '2021' TO TABLE {s3_table} " + f"SETTINGS export_merge_tree_part_throw_on_pending_mutations=true" + ) + + wait_for_export_status(node, mt_table, s3_table, "2021", "COMPLETED") + + result = node.query(f"SELECT id FROM {s3_table} WHERE year = 2021 ORDER BY id") + assert "4" in result, "Export of partition 2021 should contain original data" + + +def test_export_partition_with_mixed_computed_columns(cluster): + """Test export partition with ALIAS, MATERIALIZED, and EPHEMERAL columns.""" + node = cluster.instances["replica1"] + + postfix = str(uuid.uuid4()).replace("-", "_") + mt_table = f"mixed_computed_mt_table_{postfix}" + s3_table = f"mixed_computed_s3_table_{postfix}" + + node.query(f""" + CREATE TABLE {mt_table} ( + id UInt32, + value UInt32, + tag_input String EPHEMERAL, + doubled UInt64 ALIAS value * 2, + tripled UInt64 MATERIALIZED value * 3, + tag String DEFAULT upper(tag_input) + ) ENGINE = ReplicatedMergeTree('/clickhouse/tables/{mt_table}', 'replica1') + PARTITION BY id + ORDER BY id + SETTINGS index_granularity = 1 + """) + + # Create S3 destination table with regular columns (no EPHEMERAL) + node.query(f""" + CREATE TABLE {s3_table} ( + id UInt32, + value UInt32, + doubled UInt64, + tripled UInt64, + tag String + ) ENGINE = S3(s3_conn, filename='{s3_table}', format=Parquet, partition_strategy='hive') + PARTITION BY id + """) + + node.query(f"INSERT INTO {mt_table} (id, value, tag_input) VALUES (1, 5, 'test'), (1, 10, 'prod')") + + node.query(f"ALTER TABLE {mt_table} EXPORT PARTITION ID '1' TO TABLE {s3_table}") + + wait_for_export_status(node, mt_table, s3_table, "1", "COMPLETED") + + # Verify source data (ALIAS computed, EPHEMERAL not stored) + source_result = node.query(f"SELECT id, value, doubled, tripled, tag FROM {mt_table} ORDER BY value") + expected = "1\t5\t10\t15\tTEST\n1\t10\t20\t30\tPROD\n" + assert source_result == expected, f"Source table data mismatch. Expected:\n{expected}\nGot:\n{source_result}" + + dest_result = node.query(f"SELECT id, value, doubled, tripled, tag FROM {s3_table} ORDER BY value") + assert dest_result == expected, f"Exported data mismatch. Expected:\n{expected}\nGot:\n{dest_result}" + + status = node.query(f""" + SELECT status FROM system.replicated_partition_exports + WHERE source_table = '{mt_table}' + AND destination_table = '{s3_table}' + AND partition_id = '1' + """) + assert status.strip() == "COMPLETED", f"Expected COMPLETED status, got: {status}" + + +def test_sharded_export_partition_with_filename_pattern(cluster): + """Test that export partition with filename pattern prevents collisions in sharded setup.""" + shard1_r1 = cluster.instances["shard1_replica1"] + shard2_r1 = cluster.instances["shard2_replica1"] + watcher_node = cluster.instances["watcher_node"] + + postfix = str(uuid.uuid4()).replace("-", "_") + mt_table = f"sharded_mt_table_{postfix}" + s3_table = f"sharded_s3_table_{postfix}" + + # Create sharded tables on all shards with same partition data (same part names) + # Each shard uses different ZooKeeper path via {shard} macro + create_sharded_tables_and_insert_data(shard1_r1, mt_table, s3_table, "replica1") + create_sharded_tables_and_insert_data(shard2_r1, mt_table, s3_table, "replica1") + create_s3_table(watcher_node, s3_table) + + # Export partition from both shards with filename pattern including shard + # This should prevent filename collisions + shard1_r1.query( + f"ALTER TABLE {mt_table} EXPORT PARTITION ID '2020' TO TABLE {s3_table} " + f"SETTINGS export_merge_tree_part_filename_pattern = '{{part_name}}_{{shard}}_{{replica}}_{{checksum}}'" + ) + shard2_r1.query( + f"ALTER TABLE {mt_table} EXPORT PARTITION ID '2020' TO TABLE {s3_table} " + f"SETTINGS export_merge_tree_part_filename_pattern = '{{part_name}}_{{shard}}_{{replica}}_{{checksum}}'" + ) + + # Wait for exports to complete + wait_for_export_status(shard1_r1, mt_table, s3_table, "2020", "COMPLETED") + wait_for_export_status(shard2_r1, mt_table, s3_table, "2020", "COMPLETED") + + total_count = watcher_node.query(f"SELECT count() FROM {s3_table} WHERE year = 2020").strip() + assert total_count == "6", f"Expected 6 total rows (3 from each shard), got {total_count}" + + # Verify filenames contain shard information (check via S3 directly) + # Get all files from S3 - query from watcher_node since S3 is shared + files_shard1 = watcher_node.query( + f"SELECT _file FROM s3(s3_conn, filename='{s3_table}/**', format='One') WHERE _file LIKE '%shard1%' LIMIT 1" + ).strip() + files_shard2 = watcher_node.query( + f"SELECT _file FROM s3(s3_conn, filename='{s3_table}/**', format='One') WHERE _file LIKE '%shard2%' LIMIT 1" + ).strip() + + # Both shards should have files with their shard names + assert "shard1" in files_shard1 or files_shard1 == "", f"Expected shard1 in filenames, got: {files_shard1}" + assert "shard2" in files_shard2 or files_shard2 == "", f"Expected shard2 in filenames, got: {files_shard2}" + + +def test_export_partition_from_replicated_database_uses_db_shard_replica_macros(cluster): + """Test that {shard} and {replica} in the filename pattern are expanded from the + DatabaseReplicated identity, NOT from server config macros. + + replica1 has no / entries in its server config section. + Without the fix buildDestinationFilename() leaves macro_info.shard/replica unset, so + Macros::expand() falls through to the config-macros lookup and throws NO_ELEMENTS_IN_CONFIG. + With the fix the DatabaseReplicated shard_name / replica_name are injected into macro_info + before the expand call, and the pattern resolves correctly. + """ + + # The remote disk test suite sets the shard and replica macros in https://github.com/Altinity/ClickHouse/blob/bbabcaa96e8b7fe8f70ecd0bd4f76fb0f76f2166/tests/integration/helpers/cluster.py#L4356 + # When expanding the macros, the configured ones are preferred over the ones from the DatabaseReplicated definition. + # Therefore, this test fails. It is easier to skip it than to fix it. + skip_if_remote_database_disk_enabled(cluster) + + node = cluster.instances["replica1"] + watcher_node = cluster.instances["watcher_node"] + + postfix = str(uuid.uuid4()).replace("-", "_") + db_name = f"repdb_{postfix}" + table_name = "mt_table" + s3_table = f"s3_dbreplicated_{postfix}" + + # These values exist only in the DatabaseReplicated definition – they are NOT + # present anywhere in replica1's server config . + db_shard = "db_shard_x" + db_replica = "db_replica_y" + + node.query( + f"CREATE DATABASE {db_name} " + f"ENGINE = Replicated('/clickhouse/databases/{db_name}', '{db_shard}', '{db_replica}')") + + node.query(f""" + CREATE TABLE {db_name}.{table_name} + (id UInt64, year UInt16) + ENGINE = ReplicatedMergeTree() + PARTITION BY year ORDER BY tuple()""") + + node.query(f"INSERT INTO {db_name}.{table_name} VALUES (1, 2020), (2, 2020), (3, 2020)") + # Stop merges so part names stay stable during the test. + node.query(f"SYSTEM STOP MERGES {db_name}.{table_name}") + + node.query( + f"CREATE TABLE {s3_table} (id UInt64, year UInt16) " + f"ENGINE = S3(s3_conn, filename='{s3_table}', format=Parquet, partition_strategy='hive') " + f"PARTITION BY year") + + watcher_node.query( + f"CREATE TABLE {s3_table} (id UInt64, year UInt16) " + f"ENGINE = S3(s3_conn, filename='{s3_table}', format=Parquet, partition_strategy='hive') " + f"PARTITION BY year") + + # Export with {shard} and {replica} in the pattern. + # Before the fix: Macros::expand throws NO_ELEMENTS_IN_CONFIG because replica1 has + # no / server config macros. + # After the fix: DatabaseReplicated's shard_name/replica_name are wired into + # macro_info before the expand call, so this succeeds and produces the right names. + node.query( + f"ALTER TABLE {db_name}.{table_name} EXPORT PARTITION ID '2020' TO TABLE {s3_table} " + f"SETTINGS export_merge_tree_part_filename_pattern = " + f"'{{part_name}}_{{shard}}_{{replica}}_{{checksum}}'") + + # A FAILED status here almost certainly means the macro expansion threw + # NO_ELEMENTS_IN_CONFIG (i.e. the fix is missing or broken). + wait_for_export_status(node, table_name, s3_table, "2020", "COMPLETED") + + # Data should have landed in S3. + count = watcher_node.query(f"SELECT count() FROM {s3_table} WHERE year = 2020").strip() + assert count == "3", f"Expected 3 exported rows, got {count}" + + # The exported filename must contain the exact shard and replica names from the + # DatabaseReplicated definition, proving the fix injected them (not server config macros). + filename = watcher_node.query( + f"SELECT _file FROM s3(s3_conn, filename='{s3_table}/**/*.parquet', format='One') LIMIT 1" + ).strip() + + assert db_shard in filename, ( + f"Expected filename to contain DatabaseReplicated shard '{db_shard}', got: {filename!r}. " + "Suggests {shard} was not expanded from the DatabaseReplicated identity.") + + assert db_replica in filename, ( + f"Expected filename to contain DatabaseReplicated replica '{db_replica}', got: {filename!r}. " + "Suggests {replica} was not expanded from the DatabaseReplicated identity.") + + +def test_sharded_export_partition_default_pattern(cluster): + shard1_r1 = cluster.instances["shard1_replica1"] + shard2_r1 = cluster.instances["shard2_replica1"] + watcher_node = cluster.instances["watcher_node"] + + mt_table = "sharded_mt_table_default" + s3_table = "sharded_s3_table_default" + + # Create sharded tables with different ZooKeeper paths per shard + create_sharded_tables_and_insert_data(shard1_r1, mt_table, s3_table, "replica1") + create_sharded_tables_and_insert_data(shard2_r1, mt_table, s3_table, "replica1") + create_s3_table(watcher_node, s3_table) + + # Export with default pattern ({part_name}_{checksum}) - may cause collisions if parts have same name and the same checksum + shard1_r1.query( + f"ALTER TABLE {mt_table} EXPORT PARTITION ID '2020' TO TABLE {s3_table}" + ) + shard2_r1.query( + f"ALTER TABLE {mt_table} EXPORT PARTITION ID '2020' TO TABLE {s3_table}" + ) + + wait_for_export_status(shard1_r1, mt_table, s3_table, "2020", "COMPLETED") + wait_for_export_status(shard2_r1, mt_table, s3_table, "2020", "COMPLETED") + + # Both exports should complete (even if there are collisions, the overwrite policy handles it) + # S3 tables are shared, so query from watcher_node + total_count = watcher_node.query(f"SELECT count() FROM {s3_table} WHERE year = 2020").strip() + + # only one file with 3 rows should be present + assert int(total_count) == 3, f"Expected 3 rows, got {total_count}" + + +def test_export_partition_scheduler_skipped_when_moves_stopped(cluster): + node = cluster.instances["replica1"] + + uid = str(uuid.uuid4()).replace("-", "_") + mt_table = f"sched_skip_mt_{uid}" + s3_table = f"sched_skip_s3_{uid}" + + create_tables_and_insert_data(node, mt_table, s3_table, "replica1") + + node.query(f"SYSTEM STOP MOVES {mt_table}") + + node.query( + f"ALTER TABLE {mt_table} EXPORT PARTITION ID '2020' TO TABLE {s3_table}" + ) + + wait_for_export_to_start(node, mt_table, s3_table, "2020") + + # Wait for several scheduler cycles (each fires every 5 s). + # If the guard is missing the scheduler would run and data would land in S3. + time.sleep(10) + + status = node.query( + f"SELECT status FROM system.replicated_partition_exports" + f" WHERE source_table = '{mt_table}' AND destination_table = '{s3_table}'" + f" AND partition_id = '2020'" + ).strip() + + assert status == "PENDING", ( + f"Expected PENDING while moves are stopped, got '{status}'" + ) + + row_count = int(node.query(f"SELECT count() FROM {s3_table} WHERE year = 2020").strip()) + assert row_count == 0, ( + f"Expected 0 rows in S3 while scheduler is skipped, got {row_count}" + ) + + node.query(f"SYSTEM START MOVES {mt_table}") + + wait_for_export_status(node, mt_table, s3_table, "2020", "COMPLETED", timeout=60) + + row_count = int(node.query(f"SELECT count() FROM {s3_table} WHERE year = 2020").strip()) + assert row_count == 3, f"Expected 3 rows in S3 after export completed, got {row_count}" + + +def test_export_partition_resumes_after_stop_moves(cluster): + node = cluster.instances["replica1"] + + uid = str(uuid.uuid4()).replace("-", "_") + mt_table = f"stop_moves_before_mt_{uid}" + s3_table = f"stop_moves_before_s3_{uid}" + + create_tables_and_insert_data(node, mt_table, s3_table, "replica1") + + node.query(f"SYSTEM STOP MOVES {mt_table}") + + node.query( + f"ALTER TABLE {mt_table} EXPORT PARTITION ID '2020' TO TABLE {s3_table}" + ) + + wait_for_export_to_start(node, mt_table, s3_table, "2020") + + # Give the scheduler enough time to attempt (and cancel) the part task at + # least once, exercising the lock-release code path. + time.sleep(5) + + status = node.query( + f"SELECT status FROM system.replicated_partition_exports" + f" WHERE source_table = '{mt_table}' AND destination_table = '{s3_table}'" + f" AND partition_id = '2020'" + ).strip() + assert status == "PENDING", f"Expected PENDING while moves are stopped, got '{status}'" + + row_count = int(node.query(f"SELECT count() FROM {s3_table} WHERE year = 2020").strip()) + assert row_count == 0, f"Expected 0 rows in S3 while moves are stopped, got {row_count}" + + node.query(f"SYSTEM START MOVES {mt_table}") + + wait_for_export_status(node, mt_table, s3_table, "2020", "COMPLETED", timeout=60) + + row_count = int(node.query(f"SELECT count() FROM {s3_table} WHERE year = 2020").strip()) + assert row_count == 3, f"Expected 3 rows in S3 after export completed, got {row_count}" + + +def test_export_partition_resumes_after_stop_moves_during_export(cluster): + skip_if_remote_database_disk_enabled(cluster) + + node = cluster.instances["replica1"] + + uid = str(uuid.uuid4()).replace("-", "_") + mt_table = f"stop_moves_during_mt_{uid}" + s3_table = f"stop_moves_during_s3_{uid}" + + create_tables_and_insert_data(node, mt_table, s3_table, "replica1") + + minio_ip = cluster.minio_ip + minio_port = cluster.minio_port + + with PartitionManager() as pm: + pm.add_rule({ + "instance": node, + "destination": node.ip_address, + "protocol": "tcp", + "source_port": minio_port, + "action": "REJECT --reject-with tcp-reset", + }) + pm.add_rule({ + "instance": node, + "destination": minio_ip, + "protocol": "tcp", + "destination_port": minio_port, + "action": "REJECT --reject-with tcp-reset", + }) + + node.query( + f"ALTER TABLE {mt_table} EXPORT PARTITION ID '2020' TO TABLE {s3_table}" + ) + + wait_for_export_to_start(node, mt_table, s3_table, "2020") + + # Let the tasks start executing and failing against the blocked S3. + time.sleep(2) + + node.query(f"SYSTEM STOP MOVES {mt_table}") + + # Give the cancel callback time to fire and the lock-release path to run. + time.sleep(3) + + status = node.query( + f"SELECT status FROM system.replicated_partition_exports" + f" WHERE source_table = '{mt_table}' AND destination_table = '{s3_table}'" + f" AND partition_id = '2020'" + ).strip() + + assert status == "PENDING", ( + f"Expected PENDING while moves are stopped and S3 is blocked, got '{status}'" + ) + + node.query(f"SYSTEM START MOVES {mt_table}") + + # MinIO is now unblocked; the next scheduler cycle should succeed. + wait_for_export_status(node, mt_table, s3_table, "2020", "COMPLETED", timeout=60) + + row_count = int(node.query(f"SELECT count() FROM {s3_table} WHERE year = 2020").strip()) + assert row_count == 3, f"Expected 3 rows in S3 after export completed, got {row_count}" + + +def test_export_partition_all(cluster): + """Happy path for `ALTER TABLE ... EXPORT PARTITION ALL TO TABLE ...`. + + Schedules one export task per active partition in a single ALTER, then + verifies every partition lands in the destination S3 table. + """ + node = cluster.instances["replica1"] + + uid = str(uuid.uuid4()).replace("-", "_") + mt_table = f"export_all_mt_{uid}" + s3_table = f"export_all_s3_{uid}" + + node.query( + f"CREATE TABLE {mt_table} (id UInt64, year UInt16)" + f" ENGINE = ReplicatedMergeTree('/clickhouse/tables/{mt_table}', 'replica1')" + f" PARTITION BY year ORDER BY tuple()" + ) + node.query(f"INSERT INTO {mt_table} VALUES (1, 2020), (2, 2021), (3, 2022)") + create_s3_table(node, s3_table) + + node.query(f"ALTER TABLE {mt_table} EXPORT PARTITION ALL TO TABLE {s3_table}") + + for partition_id in ("2020", "2021", "2022"): + wait_for_export_status(node, mt_table, s3_table, partition_id, "COMPLETED", timeout=60) + + row_count = int(node.query(f"SELECT count() FROM {s3_table}").strip()) + assert row_count == 3, f"Expected 3 rows in S3 after EXPORT PARTITION ALL, got {row_count}" + + +def test_export_partition_partition_column_castable_type_mismatch(cluster): + """A lossy partition-column cast (year String -> UInt16) is rejected synchronously + when export_merge_tree_part_allow_lossy_cast is off, scheduling nothing.""" + skip_if_remote_database_disk_enabled(cluster) + node = cluster.instances["replica1"] + + postfix = str(uuid.uuid4()).replace("-", "_") + mt_table = f"pkey_cast_mismatch_partition_mt_{postfix}" + s3_table = f"pkey_cast_mismatch_partition_s3_{postfix}" + + # Source: year String; destination: year UInt16. PARTITION BY year on + # both sides — same AST text — to defeat the AST equivalence check. + node.query( + f"CREATE TABLE {mt_table} (id UInt64, year String) " + f"ENGINE = ReplicatedMergeTree('/clickhouse/tables/{mt_table}', 'replica1') " + f"PARTITION BY year " + f"ORDER BY tuple()" + ) + node.query( + f"CREATE TABLE {s3_table} (id UInt64, year UInt16) " + f"ENGINE = S3(s3_conn, filename='{s3_table}', " + f"format=Parquet, partition_strategy='hive') " + f"PARTITION BY year" + ) + + node.query( + f"INSERT INTO {mt_table} VALUES (1, '2020'), (2, '2020'), (3, '2020')" + ) + + # With a String partition column the partition_id is the SipHash of the + # value rather than the textual representation — look it up so we can + # reference the partition explicitly in EXPORT PARTITION ID and in + # subsequent system.replicated_partition_exports queries. + partition_id = node.query( + f"SELECT partition_id FROM system.parts " + f"WHERE database = currentDatabase() AND table = '{mt_table}' " + f" AND active " + f"ORDER BY name LIMIT 1" + ).strip() + assert partition_id, ( + "Expected one active part on the source table after INSERT; " + "system.parts returned nothing." + ) + + error = node.query_and_get_error( + f"ALTER TABLE {mt_table} EXPORT PARTITION ID '{partition_id}' " + f"TO TABLE {s3_table}" + ) + assert "INCOMPATIBLE_COLUMNS" in error, ( + f"Expected INCOMPATIBLE_COLUMNS for a lossy partition-column cast, " + f"got: {error!r}" + ) + assert "requires a lossy cast" in error and "'year'" in error, ( + f"Expected the error message to report the lossy cast on column " + f"'year', got: {error!r}" + ) + + # Nothing scheduled: no row in system.replicated_partition_exports. + rows_in_system_view = node.query( + f"SELECT count() FROM system.replicated_partition_exports " + f"WHERE source_table = '{mt_table}' " + f" AND destination_table = '{s3_table}' " + f" AND partition_id = '{partition_id}'" + ).strip() + assert rows_in_system_view == "0", ( + f"Expected no row in system.replicated_partition_exports after a " + f"synchronously-rejected export, got {rows_in_system_view}." + ) + + # Nothing written: no parquet file under any year=*/ partition prefix. + files_in_s3 = node.query( + f"SELECT count() FROM s3(s3_conn, " + f"filename='{s3_table}/year=*/*.parquet', format='One')" + ).strip() + assert files_in_s3 == "0", ( + f"Expected no Parquet files in S3 after a synchronously-rejected " + f"export, found {files_in_s3}." + ) + + +def test_export_partition_all_failure_modes(cluster): + """Cover the three values of `export_merge_tree_partition_all_on_error`. + + Set up an already-fully-exported source table, then re-run EXPORT PARTITION ALL + with each failure mode and assert the documented behavior. + """ + node = cluster.instances["replica1"] + + uid = str(uuid.uuid4()).replace("-", "_") + mt_table = f"export_all_modes_mt_{uid}" + s3_table = f"export_all_modes_s3_{uid}" + empty_mt = f"export_all_empty_mt_{uid}" + + node.query( + f"CREATE TABLE {mt_table} (id UInt64, year UInt16)" + f" ENGINE = ReplicatedMergeTree('/clickhouse/tables/{mt_table}', 'replica1')" + f" PARTITION BY year ORDER BY tuple()" + ) + node.query(f"INSERT INTO {mt_table} VALUES (1, 2020), (2, 2021), (3, 2022)") + create_s3_table(node, s3_table) + + # First run: schedule + wait for all partitions to complete. + node.query(f"ALTER TABLE {mt_table} EXPORT PARTITION ALL TO TABLE {s3_table}") + for partition_id in ("2020", "2021", "2022"): + wait_for_export_status(node, mt_table, s3_table, partition_id, "COMPLETED", timeout=60) + + # Empty table: throws BAD_ARGUMENTS (no active partitions). + node.query( + f"CREATE TABLE {empty_mt} (id UInt64, year UInt16)" + f" ENGINE = ReplicatedMergeTree('/clickhouse/tables/{empty_mt}', 'replica1')" + f" PARTITION BY year ORDER BY tuple()" + ) + error = node.query_and_get_error( + f"ALTER TABLE {empty_mt} EXPORT PARTITION ALL TO TABLE {s3_table}" + ) + assert "no active partitions to export" in error, ( + f"Expected 'no active partitions' error, got: {error}" + ) + + # throw_first (default): re-run aborts on the first conflicting partition. + error = node.query_and_get_error( + f"ALTER TABLE {mt_table} EXPORT PARTITION ALL TO TABLE {s3_table}" + f" SETTINGS export_merge_tree_partition_all_on_error = 'throw_first'" + ) + assert "EXPORT_PARTITION_ALREADY_EXPORTED" in error, ( + f"Expected EXPORT_PARTITION_ALREADY_EXPORTED in error, got: {error}" + ) + + # collect: aggregated PARTITION_EXPORT_FAILED message lists every conflicting partition. + error = node.query_and_get_error( + f"ALTER TABLE {mt_table} EXPORT PARTITION ALL TO TABLE {s3_table}" + f" SETTINGS export_merge_tree_partition_all_on_error = 'collect'" + ) + assert "PARTITION_EXPORT_FAILED" in error, ( + f"Expected PARTITION_EXPORT_FAILED in error, got: {error}" + ) + for partition_id in ("2020", "2021", "2022"): + assert partition_id in error, ( + f"Expected aggregated error to mention partition {partition_id}, got: {error}" + ) + + # skip_conflicts: succeeds silently because every partition conflicts and is skipped. + node.query( + f"ALTER TABLE {mt_table} EXPORT PARTITION ALL TO TABLE {s3_table}" + f" SETTINGS export_merge_tree_partition_all_on_error = 'skip_conflicts'" + ) + + +# ---- Partition-key compatibility gate (unified with the Iceberg gate) -------------------------- +# +# Plain (hive) object storage writes every row of a part to the single directory computed from the +# destination PARTITION BY, so each source partition must map to exactly one destination partition. +# The gate accepts equivalent or finer source keys (e.g. a source that adds partition columns on top +# of the destination's) and rejects source partitions that would span several destination partitions +# or that do not cover the destination partition column. Hive destinations partition by bare columns +# only, so these cases exercise the column-subset and single-value paths. + + +def _run_subset_accept(node, source_key): + """Export a source partitioned by *source_key* (a superset of the destination key ``year``) into a + hive destination partitioned by ``year``, then verify the full dataset, the hive directory layout, + and a round-trip back into MergeTree.""" + uid = str(uuid.uuid4()).replace("-", "_") + mt_table = f"subset_mt_{uid}" + s3_table = f"subset_s3_{uid}" + roundtrip = f"subset_roundtrip_{uid}" + + node.query( + f"CREATE TABLE {mt_table} (id UInt64, year UInt16, country String)" + f" ENGINE = ReplicatedMergeTree('/clickhouse/tables/{mt_table}', 'replica1')" + f" PARTITION BY {source_key} ORDER BY tuple()" + ) + node.query( + f"INSERT INTO {mt_table} VALUES (1, 2020, 'US'), (2, 2020, 'FR'), (3, 2021, 'US')" + ) + node.query( + f"CREATE TABLE {s3_table} (id UInt64, year UInt16, country String)" + f" ENGINE = S3(s3_conn, filename='{s3_table}', format=Parquet, partition_strategy='hive')" + f" PARTITION BY year" + ) + + partition_ids = node.query( + f"SELECT DISTINCT partition_id FROM system.parts" + f" WHERE database = currentDatabase() AND table = '{mt_table}' AND active" + ).strip().split("\n") + assert len(partition_ids) == 3, f"expected 3 source partitions, got {partition_ids}" + + node.query(f"ALTER TABLE {mt_table} EXPORT PARTITION ALL TO TABLE {s3_table}") + for pid in partition_ids: + wait_for_export_status(node, mt_table, s3_table, pid, "COMPLETED", timeout=90) + + src = node.query(f"SELECT id, year, country FROM {mt_table} ORDER BY id") + dst = node.query(f"SELECT id, year, country FROM {s3_table} ORDER BY id") + assert dst == src, f"destination rows differ from source:\nsrc={src!r}\ndst={dst!r}" + + # The destination partitions by year only: rows land in the year= hive directory. + rows_2020 = node.query( + f"SELECT count() FROM s3(s3_conn, filename='{s3_table}/year=2020/*.parquet', format='Parquet')" + ).strip() + rows_2021 = node.query( + f"SELECT count() FROM s3(s3_conn, filename='{s3_table}/year=2021/*.parquet', format='Parquet')" + ).strip() + assert rows_2020 == "2", f"expected 2 rows under year=2020, got {rows_2020}" + assert rows_2021 == "1", f"expected 1 row under year=2021, got {rows_2021}" + + node.query( + f"CREATE TABLE {roundtrip} (id UInt64, year UInt16, country String)" + f" ENGINE = ReplicatedMergeTree('/clickhouse/tables/{roundtrip}', 'replica1')" + f" PARTITION BY {source_key} ORDER BY tuple()" + ) + node.query(f"INSERT INTO {roundtrip} SELECT * FROM {s3_table}") + rt = node.query(f"SELECT id, year, country FROM {roundtrip} ORDER BY id") + assert rt == src, f"round-trip rows differ from source:\nsrc={src!r}\nrt={rt!r}" + + +def test_export_partition_multicolumn_subset_accepted(cluster): + """Source partitions by (year, country); destination by year only - a coarser key that is covered + by the source key, so every source partition has a single year and maps to exactly one destination + partition. Accepted (this was rejected as a partition-key mismatch before the plain gate was + unified with the Iceberg one).""" + node = cluster.instances["replica1"] + _run_subset_accept(node, "(year, country)") + + +def test_export_partition_subset_reversed_order_accepted(cluster): + """The subset match is order-independent: a source keyed by (country, year) still covers a + destination keyed by year.""" + node = cluster.instances["replica1"] + _run_subset_accept(node, "(country, year)") + + +def test_export_partition_coarser_source_rejected(cluster): + """Source partitions monthly (toYYYYMM(dt)); destination by the raw date. A single source part + holding two different days would map to two destination partitions, so the gate rejects the + export synchronously with BAD_ARGUMENTS and schedules nothing.""" + node = cluster.instances["replica1"] + + uid = str(uuid.uuid4()).replace("-", "_") + mt_table = f"coarser_mt_{uid}" + s3_table = f"coarser_s3_{uid}" + + node.query( + f"CREATE TABLE {mt_table} (id UInt64, dt Date)" + f" ENGINE = ReplicatedMergeTree('/clickhouse/tables/{mt_table}', 'replica1')" + f" PARTITION BY toYYYYMM(dt) ORDER BY tuple()" + ) + node.query(f"INSERT INTO {mt_table} VALUES (1, '2024-03-05'), (2, '2024-03-20')") + node.query( + f"CREATE TABLE {s3_table} (id UInt64, dt Date)" + f" ENGINE = S3(s3_conn, filename='{s3_table}', format=Parquet, partition_strategy='hive')" + f" PARTITION BY dt" + ) + + pid = first_partition_id(node, mt_table) + error = node.query_and_get_error( + f"ALTER TABLE {mt_table} EXPORT PARTITION ID '{pid}' TO TABLE {s3_table}" + ) + assert "BAD_ARGUMENTS" in error, f"expected BAD_ARGUMENTS, got: {error!r}" + + scheduled = node.query( + f"SELECT count() FROM system.replicated_partition_exports" + f" WHERE source_table = '{mt_table}' AND destination_table = '{s3_table}'" + ).strip() + assert scheduled == "0", f"expected nothing scheduled after a synchronous reject, got {scheduled}" + + +def test_export_partition_dest_column_not_in_source_key_rejected(cluster): + """Destination partitions by a column that is not part of the source partition key; the gate + rejects the export synchronously with BAD_ARGUMENTS naming the uncovered column.""" + node = cluster.instances["replica1"] + + uid = str(uuid.uuid4()).replace("-", "_") + mt_table = f"nocover_mt_{uid}" + s3_table = f"nocover_s3_{uid}" + + node.query( + f"CREATE TABLE {mt_table} (id UInt64, year UInt16, country String)" + f" ENGINE = ReplicatedMergeTree('/clickhouse/tables/{mt_table}', 'replica1')" + f" PARTITION BY year ORDER BY tuple()" + ) + node.query(f"INSERT INTO {mt_table} VALUES (1, 2020, 'US'), (2, 2020, 'FR')") + node.query( + f"CREATE TABLE {s3_table} (id UInt64, year UInt16, country String)" + f" ENGINE = S3(s3_conn, filename='{s3_table}', format=Parquet, partition_strategy='hive')" + f" PARTITION BY country" + ) + + pid = first_partition_id(node, mt_table) + error = node.query_and_get_error( + f"ALTER TABLE {mt_table} EXPORT PARTITION ID '{pid}' TO TABLE {s3_table}" + ) + assert "BAD_ARGUMENTS" in error, f"expected BAD_ARGUMENTS, got: {error!r}" + assert "country" in error, f"expected the error to name column 'country', got: {error!r}" + + +def test_export_partition_column_timezone_rendered_in_destination_zone(cluster): + """A hive partition value lives as text in the object path and is read back in the destination + column's time zone, so the export has to spell it the way the destination would. Spelling it in the + source's zone names a different instant and the row reads back shifted by the offset between the + two zones. INSERT SELECT into an identical table is the reference behavior.""" + node = cluster.instances["replica1"] + + uid = str(uuid.uuid4()).replace("-", "_") + mt_table = f"tz_mt_{uid}" + s3_export = f"tz_export_s3_{uid}" + s3_insert = f"tz_insert_s3_{uid}" + + node.query( + f"CREATE TABLE {mt_table} (id UInt64, ts DateTime('UTC'))" + f" ENGINE = ReplicatedMergeTree('/clickhouse/tables/{mt_table}', 'replica1')" + f" PARTITION BY toDate(ts) ORDER BY tuple()" + ) + node.query(f"INSERT INTO {mt_table} VALUES (1, '2024-03-05 15:00:00')") + for table in (s3_export, s3_insert): + node.query( + f"CREATE TABLE {table} (id UInt64, ts DateTime('Asia/Tokyo'))" + f" ENGINE = S3(s3_conn, filename='{table}', format=Parquet, partition_strategy='hive')" + f" PARTITION BY ts" + ) + + pid = first_partition_id(node, mt_table) + node.query(f"ALTER TABLE {mt_table} EXPORT PARTITION ID '{pid}' TO TABLE {s3_export}") + wait_for_export_status(node, mt_table, s3_export, pid, "COMPLETED", timeout=90) + + node.query(f"INSERT INTO {s3_insert} SELECT * FROM {mt_table}") + + source_instant = node.query(f"SELECT toUnixTimestamp(ts) FROM {mt_table}").strip() + exported_instant = node.query(f"SELECT toUnixTimestamp(ts) FROM {s3_export}").strip() + inserted_instant = node.query(f"SELECT toUnixTimestamp(ts) FROM {s3_insert}").strip() + assert exported_instant == source_instant, ( + f"the exported row moved in time: source {source_instant}, destination {exported_instant}" + ) + assert inserted_instant == source_instant, ( + f"INSERT SELECT must not move it either: source {source_instant}," + f" destination {inserted_instant}" + ) + + # 2024-03-05 15:00:00 UTC is 2024-03-06 00:00:00 in Tokyo. + exported_directory = node.query( + f"SELECT DISTINCT extract(_path, 'ts=[^/]*') FROM {s3_export}" + ).strip() + inserted_directory = node.query( + f"SELECT DISTINCT extract(_path, 'ts=[^/]*') FROM {s3_insert}" + ).strip() + assert exported_directory == "ts=2024-03-06 00:00:00", ( + f"unexpected hive directory: {exported_directory!r}" + ) + assert inserted_directory == exported_directory, ( + f"export and INSERT SELECT disagree on the partition directory:" + f" {exported_directory!r} vs {inserted_directory!r}" + ) + + +def create_wildcard_destination(node, table, columns, partition_key): + """A wildcard destination, the only partition strategy that accepts an expression as its + partition key: the hive strategy allows storage columns only.""" + node.query( + f"CREATE TABLE {table} ({columns})" + f" ENGINE = S3(s3_conn, filename='{table}/{{_partition_id}}/{{_file}}.parquet'," + f" format=Parquet, partition_strategy='wildcard')" + f" PARTITION BY {partition_key}" + ) + + +def test_export_partition_dest_argument_order_rejected(cluster): + """The destination key intDiv(x, 100) has to be validated as written. This source part holds + x in [201, 350], which covers the destination partitions 2 and 3, so the export must be rejected. + Reading the arguments in the reverse order would validate intDiv(100, x) instead, which is 0 at + both endpoints and would silently write both destination partitions into one directory.""" + node = cluster.instances["replica1"] + + uid = str(uuid.uuid4()).replace("-", "_") + mt_table = f"argorder_mt_{uid}" + s3_table = f"argorder_s3_{uid}" + + node.query( + f"CREATE TABLE {mt_table} (id UInt64, x UInt64)" + f" ENGINE = ReplicatedMergeTree('/clickhouse/tables/{mt_table}', 'replica1')" + f" PARTITION BY intDiv(x, 1000) ORDER BY tuple()" + ) + node.query(f"INSERT INTO {mt_table} VALUES (1, 201), (2, 350)") + create_wildcard_destination(node, s3_table, "id UInt64, x UInt64", "intDiv(x, 100)") + + pid = first_partition_id(node, mt_table) + error = node.query_and_get_error( + f"ALTER TABLE {mt_table} EXPORT PARTITION ID '{pid}' TO TABLE {s3_table}" + ) + assert "BAD_ARGUMENTS" in error, f"expected BAD_ARGUMENTS, got: {error!r}" + + +def test_export_partition_dest_finer_expression_single_partition_accepted(cluster): + """The same shape as the rejected case, with x in [100, 150]: the whole source partition maps to + the single destination partition 1, so it is accepted and every row lands in one directory. The + swapped-argument reading would refuse this one, since intDiv(100, 100) != intDiv(100, 150).""" + node = cluster.instances["replica1"] + + uid = str(uuid.uuid4()).replace("-", "_") + mt_table = f"argorder_ok_mt_{uid}" + s3_table = f"argorder_ok_s3_{uid}" + + node.query( + f"CREATE TABLE {mt_table} (id UInt64, x UInt64)" + f" ENGINE = ReplicatedMergeTree('/clickhouse/tables/{mt_table}', 'replica1')" + f" PARTITION BY intDiv(x, 1000) ORDER BY tuple()" + ) + node.query(f"INSERT INTO {mt_table} VALUES (1, 100), (2, 150)") + create_wildcard_destination(node, s3_table, "id UInt64, x UInt64", "intDiv(x, 100)") + + pid = first_partition_id(node, mt_table) + node.query(f"ALTER TABLE {mt_table} EXPORT PARTITION ID '{pid}' TO TABLE {s3_table}") + wait_for_export_status(node, mt_table, s3_table, pid, "COMPLETED", timeout=90) + + # A wildcard destination cannot be read as a table, so read the objects it wrote. + exported = f"s3(s3_conn, filename='{s3_table}/**/*.parquet', format='Parquet', structure='id UInt64, x UInt64')" + src = node.query(f"SELECT id, x FROM {mt_table} ORDER BY id") + dst = node.query(f"SELECT id, x FROM {exported} ORDER BY id") + assert dst == src, f"destination rows differ from source:\nsrc={src!r}\ndst={dst!r}" + + directories = node.query( + f"SELECT DISTINCT extract(_path, '{s3_table}/[^/]*') FROM {exported}" + ).strip() + assert directories == f"{s3_table}/1", f"unexpected destination directories: {directories!r}" + + +def test_export_partition_dest_nested_expression_accepted(cluster): + """A destination key that wraps the source key in a coarser transform - toYYYYMM(toDate(ts)) over + a source keyed by toDate(ts) - is a function of the source key, so every source partition sits + inside one destination partition whatever the data is.""" + node = cluster.instances["replica1"] + + uid = str(uuid.uuid4()).replace("-", "_") + mt_table = f"nested_mt_{uid}" + s3_table = f"nested_s3_{uid}" + + node.query( + f"CREATE TABLE {mt_table} (id UInt64, ts DateTime)" + f" ENGINE = ReplicatedMergeTree('/clickhouse/tables/{mt_table}', 'replica1')" + f" PARTITION BY toDate(ts) ORDER BY tuple()" + ) + node.query( + f"INSERT INTO {mt_table} VALUES (1, '2024-03-05 01:00:00'), (2, '2024-03-05 20:00:00')" + ) + create_wildcard_destination(node, s3_table, "id UInt64, ts DateTime", "toYYYYMM(toDate(ts))") + + pid = first_partition_id(node, mt_table) + node.query(f"ALTER TABLE {mt_table} EXPORT PARTITION ID '{pid}' TO TABLE {s3_table}") + wait_for_export_status(node, mt_table, s3_table, pid, "COMPLETED", timeout=90) + + exported = f"s3(s3_conn, filename='{s3_table}/**/*.parquet', format='Parquet', structure='id UInt64, ts DateTime')" + src = node.query(f"SELECT id, ts FROM {mt_table} ORDER BY id") + dst = node.query(f"SELECT id, ts FROM {exported} ORDER BY id") + assert dst == src, f"destination rows differ from source:\nsrc={src!r}\ndst={dst!r}" + + directories = node.query( + f"SELECT DISTINCT extract(_path, '{s3_table}/[^/]*') FROM {exported}" + ).strip() + assert directories == f"{s3_table}/202403", ( + f"unexpected destination directories: {directories!r}" + ) + + +def test_export_partition_dest_term_over_two_columns_rejected(cluster): + """A destination expression over two columns is only single-valued when the source key pins both. + This source pins b but only intDiv(a, 100), so a spans [10, 90] within one source partition and + intDiv(a + b, 100) takes both 0 and 1 there. Per-column min/max cannot bound such an expression, + so it is rejected; a source keyed by (a, b) would be accepted, since it pins both columns.""" + node = cluster.instances["replica1"] + + uid = str(uuid.uuid4()).replace("-", "_") + mt_table = f"twocol_mt_{uid}" + s3_table = f"twocol_s3_{uid}" + + node.query( + f"CREATE TABLE {mt_table} (id UInt64, a UInt64, b UInt64)" + f" ENGINE = ReplicatedMergeTree('/clickhouse/tables/{mt_table}', 'replica1')" + f" PARTITION BY (intDiv(a, 100), b) ORDER BY tuple()" + ) + node.query(f"INSERT INTO {mt_table} VALUES (1, 10, 20), (2, 90, 20)") + create_wildcard_destination( + node, s3_table, "id UInt64, a UInt64, b UInt64", "intDiv(a + b, 100)" + ) + + pid = first_partition_id(node, mt_table) + error = node.query_and_get_error( + f"ALTER TABLE {mt_table} EXPORT PARTITION ID '{pid}' TO TABLE {s3_table}" + ) + assert "BAD_ARGUMENTS" in error, f"expected BAD_ARGUMENTS, got: {error!r}" +class RejectedPartitionExportCase(NamedTuple): + src_columns: str + src_partition_by: str + dst_columns: str + dst_partition_by: str + insert_values: str + error_substrings: tuple = () + + +REJECTED_PARTITION_EXPORT_CASES = [ + pytest.param( + RejectedPartitionExportCase( + src_columns="a Int32, b Int32", + src_partition_by="a", + dst_columns="b Int32, a Int32", + dst_partition_by="a", + insert_values="(1, 1), (1, 2)", + error_substrings=("partition key column",), + ), + id="same_partition_key_different_column_order_single_column", + ), + pytest.param( + RejectedPartitionExportCase( + src_columns="a Int32, b Int32, c Int32, val String", + src_partition_by="(a, b, c)", + dst_columns="c Int32, b Int32, a Int32, val String", + dst_partition_by="(a, b, c)", + insert_values="(1, 1, 1, 'x'), (1, 1, 1, 'y')", + error_substrings=("partition key column",), + ), + id="same_partition_key_different_column_order_multi_column", + ), + pytest.param( + RejectedPartitionExportCase( + src_columns="a Int32, b Int32, c Int32, val String", + src_partition_by="(a, b)", + dst_columns="a Int32, b Int32, c Int32, val String", + dst_partition_by="(a, b, c)", + insert_values="(1, 2, 3, 'x')", + error_substrings=( + "column 'c', which is not part of the source MergeTree partition key", + ), + ), + id="multi_column_partition_key_more_in_destination", + ), +] + + +@pytest.mark.parametrize("case", REJECTED_PARTITION_EXPORT_CASES) +def test_export_partition_partition_key_mismatch_variants_are_rejected(cluster, case): + skip_if_remote_database_disk_enabled(cluster) + node = cluster.instances["replica1"] + + postfix = str(uuid.uuid4()).replace("-", "_") + mt_table = f"rejected_mt_table_{postfix}" + s3_table = f"rejected_s3_table_{postfix}" + + node.query(f""" + CREATE TABLE {mt_table} ({case.src_columns}) + ENGINE = ReplicatedMergeTree('/clickhouse/tables/{mt_table}', 'replica1') + PARTITION BY {case.src_partition_by} + ORDER BY tuple() + """) + + node.query(f""" + CREATE TABLE {s3_table} ({case.dst_columns}) + ENGINE = S3(s3_conn, filename='{s3_table}', format=Parquet, partition_strategy='hive') + PARTITION BY {case.dst_partition_by} + """) + + node.query(f"INSERT INTO {mt_table} VALUES {case.insert_values}") + + partition_id = node.query( + f"SELECT partition_id FROM system.parts WHERE database = currentDatabase() " + f"AND table = '{mt_table}' AND active ORDER BY name LIMIT 1" + ).strip() + + error = node.query_and_get_error(f"ALTER TABLE {mt_table} EXPORT PARTITION ID '{partition_id}' TO TABLE {s3_table}") + assert "BAD_ARGUMENTS" in error, f"Expected BAD_ARGUMENTS, got: {error}" + for substring in case.error_substrings: + assert substring in error, f"Expected {substring!r} in error, got: {error}" + + error_all = node.query_and_get_error(f"ALTER TABLE {mt_table} EXPORT PARTITION ALL TO TABLE {s3_table}") + assert "BAD_ARGUMENTS" in error_all, f"Expected BAD_ARGUMENTS, got: {error_all}" + + count = int(node.query(f"SELECT count() FROM {s3_table}").strip()) + assert count == 0, f"Expected 0 rows in destination after rejected export, got {count}" + + +@pytest.mark.parametrize( + "dst_partition_by", + ["(a, b, c)", "(c, b, a)", "(a, b)"], + ids=["same", "reordered", "coarser"], +) +def test_export_partition_multi_column_partition_key_success(cluster, dst_partition_by): + """The source key pins every column the destination partitions by, so the destination may + also name them in another order or leave some out: each destination expression is still + single-valued over a source partition.""" + skip_if_remote_database_disk_enabled(cluster) + node = cluster.instances["replica1"] + + postfix = str(uuid.uuid4()).replace("-", "_") + mt_table = f"multi_pkey_ok_mt_table_{postfix}" + s3_table = f"multi_pkey_ok_s3_table_{postfix}" + + node.query(f""" + CREATE TABLE {mt_table} (a Int32, b Int32, c Int32, val String) + ENGINE = ReplicatedMergeTree('/clickhouse/tables/{mt_table}', 'replica1') + PARTITION BY (a, b, c) + ORDER BY tuple() + """) + + node.query(f""" + CREATE TABLE {s3_table} (a Int32, b Int32, c Int32, val String) + ENGINE = S3(s3_conn, filename='{s3_table}', format=Parquet, partition_strategy='hive') + PARTITION BY {dst_partition_by} + """) + + node.query(f"INSERT INTO {mt_table} VALUES (1, 2, 3, 'x'), (1, 2, 3, 'y')") + + partition_id = node.query( + f"SELECT partition_id FROM system.parts WHERE database = currentDatabase() " + f"AND table = '{mt_table}' AND active ORDER BY name LIMIT 1" + ).strip() + + node.query(f"ALTER TABLE {mt_table} EXPORT PARTITION ID '{partition_id}' TO TABLE {s3_table}") + wait_for_export_status(node, mt_table, s3_table, partition_id, "COMPLETED") + + count = int(node.query(f"SELECT count() FROM {s3_table}").strip()) + assert count == 2, f"Expected 2 rows in destination after export, got {count}" + + result = node.query(f"SELECT a, b, c, val FROM {s3_table} ORDER BY val").strip() + assert result == "1\t2\t3\tx\n1\t2\t3\ty", f"Unexpected exported data:\n{result}" + + +def test_export_partition_multi_column_partition_key_success_all(cluster): + skip_if_remote_database_disk_enabled(cluster) + node = cluster.instances["replica1"] + + postfix = str(uuid.uuid4()).replace("-", "_") + mt_table = f"multi_pkey_ok_all_mt_table_{postfix}" + s3_table = f"multi_pkey_ok_all_s3_table_{postfix}" + + node.query(f""" + CREATE TABLE {mt_table} (a Int32, b Int32, c Int32, val String) + ENGINE = ReplicatedMergeTree('/clickhouse/tables/{mt_table}', 'replica1') + PARTITION BY (a, b, c) + ORDER BY tuple() + """) + + node.query(f""" + CREATE TABLE {s3_table} (a Int32, b Int32, c Int32, val String) + ENGINE = S3(s3_conn, filename='{s3_table}', format=Parquet, partition_strategy='hive') + PARTITION BY (a, b, c) + """) + + node.query(f"INSERT INTO {mt_table} VALUES (1, 2, 3, 'x'), (4, 5, 6, 'y')") + + partition_ids = node.query( + f"SELECT DISTINCT partition_id FROM system.parts WHERE database = currentDatabase() " + f"AND table = '{mt_table}' AND active ORDER BY partition_id" + ).strip().split("\n") + + node.query(f"ALTER TABLE {mt_table} EXPORT PARTITION ALL TO TABLE {s3_table}") + + for pid in partition_ids: + wait_for_export_status(node, mt_table, s3_table, pid, "COMPLETED") + + count = int(node.query(f"SELECT count() FROM {s3_table}").strip()) + assert count == 2, f"Expected 2 rows in destination after export, got {count}" + + result = node.query(f"SELECT a, b, c, val FROM {s3_table} ORDER BY val").strip() + assert result == "1\t2\t3\tx\n4\t5\t6\ty", f"Unexpected exported data:\n{result}" + + +def test_export_partition_schema_mismatch_mode_honored_by_non_initiating_replica(cluster): + replica1 = cluster.instances["replica1"] + replica2 = cluster.instances["replica2"] + + postfix = str(uuid.uuid4()).replace("-", "_") + mt_table = f"schema_mode_cross_replica_mt_{postfix}" + s3_table = f"schema_mode_cross_replica_s3_{postfix}" + + make_rmt(node=replica1, name=mt_table, columns="id UInt64, year UInt16, extra String", + partition_by="year", replica_name="replica1") + make_rmt(node=replica2, name=mt_table, columns="id UInt64, year UInt16, extra String", + partition_by="year", replica_name="replica2") + replica1.query(f"INSERT INTO {mt_table} VALUES (1, 2020, 'foo'), (2, 2020, 'bar'), (3, 2020, 'baz')") + replica2.query(f"SYSTEM SYNC REPLICA {mt_table}") + + create_s3_table(node=replica1, s3_table=s3_table) + create_s3_table(node=replica2, s3_table=s3_table) + + replica1.query(f"SYSTEM STOP MOVES {mt_table}") + + replica1.query( + f"ALTER TABLE {mt_table} EXPORT PARTITION ID '2020' TO TABLE {s3_table}" + f" SETTINGS export_merge_tree_part_schema_mismatch_mode = 'ignore_extra_source_columns_by_position'" + ) + + wait_for_export_status(node=replica1, source_table=mt_table, dest_table=s3_table, + partition_id="2020", expected_status="COMPLETED", timeout=60) + + count = int(replica1.query(f"SELECT count() FROM {s3_table}").strip()) + assert count == 3, f"Expected 3 rows in destination table after export, got {count}" + + result = replica1.query(f"SELECT id, year FROM {s3_table} ORDER BY id").strip() + assert result == "1\t2020\n2\t2020\n3\t2020", f"Unexpected data:\n{result}" + + replica1.query(f"SYSTEM START MOVES {mt_table}") diff --git a/tests/integration/test_file_cluster/test.py b/tests/integration/test_file_cluster/test.py index f8922cbb9a5d..9f843d864171 100644 --- a/tests/integration/test_file_cluster/test.py +++ b/tests/integration/test_file_cluster/test.py @@ -1,4 +1,5 @@ import logging +import uuid import pytest @@ -209,3 +210,60 @@ def test_format_detection(started_cluster): "select * from fileCluster('my_cluster', 'file_for_format_detection*', auto, 's String, i UInt32', auto) ORDER BY (i, s)" ) assert result == expected_result + + +def test_hive_partitioning_with_where_condition(started_cluster): + test_id = uuid.uuid4().hex[:8] + hive_glob = f"hive_file_cluster_{test_id}/date=*/data.csv" + + for node_name in ("s0_0_0", "s0_0_1", "s0_1_0"): + node = started_cluster.instances[node_name] + for i in range(1, 5): + node.query( + f""" + INSERT INTO TABLE FUNCTION file( + 'hive_file_cluster_{test_id}/date=2000-01-0{i}/data.csv', 'CSVWithNames', 'd UInt64') + SELECT number FROM numbers(10) + SETTINGS engine_file_truncate_on_insert=1 + """ + ) + + node = started_cluster.instances["s0_0_0"] + + result = node.query( + f""" + SELECT count() FROM file('{hive_glob}', 'CSVWithNames', 'd UInt64') + WHERE date='2000-01-02' + SETTINGS use_hive_partitioning=1 + """ + ) + assert result.strip() == "10" + + result = node.query( + f""" + SELECT date, d FROM file('{hive_glob}', 'CSVWithNames', 'd UInt64') + WHERE date='2000-01-02' + LIMIT 1 + SETTINGS use_hive_partitioning=1 + """ + ) + assert "2000-01-02" in result + + result = node.query( + f""" + SELECT count() FROM fileCluster('my_cluster', '{hive_glob}', 'CSVWithNames', 'd UInt64') + WHERE date='2000-01-02' + SETTINGS use_hive_partitioning=1 + """ + ) + assert result.strip() == "10" + + result = node.query( + f""" + SELECT date, d FROM fileCluster('my_cluster', '{hive_glob}', 'CSVWithNames', 'd UInt64') + WHERE date='2000-01-02' + LIMIT 1 + SETTINGS use_hive_partitioning=1 + """ + ) + assert "2000-01-02" in result diff --git a/tests/integration/test_mask_sensitive_info/test.py b/tests/integration/test_mask_sensitive_info/test.py index a9ffa2e37f80..49d9e69e0ea5 100644 --- a/tests/integration/test_mask_sensitive_info/test.py +++ b/tests/integration/test_mask_sensitive_info/test.py @@ -3,6 +3,7 @@ import string import pytest +import uuid from helpers.cluster import ClickHouseCluster from helpers.test_tools import TSV @@ -247,6 +248,8 @@ def test_create_table(): azure_account_name = "devstoreaccount1" azure_account_key = "Eby8vdM02xNOcqFlqUwJPLlmEtlCDXJ1OUzFT50uSRZ6IFsuFq2UVErCz4I6tq/K1SZFPTOtr/KBHBeksoGMGw==" + table_suffix = uuid.uuid4().hex + table_engines = [ f"MySQL('mysql80:3306', 'mysql_db', 'mysql_table', 'mysql_user', '{password}')", f"PostgreSQL('postgres1:5432', 'postgres_db', 'postgres_table', 'postgres_user', '{password}')", @@ -278,11 +281,13 @@ def test_create_table(): f"IcebergS3('http://minio1:9001/root/data/test11.csv.gz', 'minio', '{password}')", "DNS_ERROR", ), + ( + f"Iceberg(storage_type='s3', 'http://minio1:9001/root/data/test11.csv.gz', 'minio', '{password}')", + "DNS_ERROR", + ), f"AzureBlobStorage('{azure_conn_string}', 'cont', 'test_simple.csv', 'CSV')", f"AzureBlobStorage('{azure_conn_string}', 'cont', 'test_simple_1.csv', 'CSV', 'none')", - f"AzureBlobStorage('{azure_storage_account_url}', 'cont', 'test_simple_2.csv', '{azure_account_name}', '{azure_account_key}')", - f"AzureBlobStorage('{azure_storage_account_url}', 'cont', 'test_simple_3.csv', '{azure_account_name}', '{azure_account_key}', 'CSV')", - f"AzureBlobStorage('{azure_storage_account_url}', 'cont', 'test_simple_4.csv', '{azure_account_name}', '{azure_account_key}', 'CSV', 'none')", + f"AzureQueue('{azure_conn_string}', 'cont', '*', 'CSV') SETTINGS mode = 'unordered'", f"AzureQueue('{azure_conn_string}', 'cont', '*', 'CSV', 'none') SETTINGS mode = 'unordered'", f"AzureQueue('{azure_conn_string}', 'cont', '*', 'CSV') SETTINGS mode = 'unordered', after_processing = 'move', after_processing_move_connection_string = '{azure_conn_string}', after_processing_move_container = 'chprocessed'", @@ -290,6 +295,19 @@ def test_create_table(): f"AzureQueue('{azure_storage_account_url}', 'cont', '*', '{azure_account_name}', '{azure_account_key}', 'CSV') SETTINGS mode = 'unordered'", f"AzureQueue('{azure_storage_account_url}', 'cont', '*', '{azure_account_name}', '{azure_account_key}', 'CSV', 'none') SETTINGS mode = 'unordered'", "AzureBlobStorage('BlobEndpoint=https://my-endpoint/;SharedAccessSignature=sp=r&st=2025-09-29T14:58:11Z&se=2025-09-29T00:00:00Z&spr=https&sv=2022-11-02&sr=c&sig=SECRET%SECRET%SECRET%SECRET', 'exampledatasets', 'example.csv')", + f"AzureBlobStorage(named_collection_2, connection_string = '{azure_conn_string}', container = 'cont', blob_path = 'test_simple_7.csv', format = 'CSV')", + f"AzureBlobStorage(named_collection_2, storage_account_url = '{azure_storage_account_url}', container = 'cont', blob_path = 'test_simple_8.csv', account_name = '{azure_account_name}', account_key = '{azure_account_key}')", + f"AzureBlobStorage('{azure_storage_account_url}', 'cont', 'test_simple_3.csv', '{azure_account_name}', '{azure_account_key}')", + f"AzureBlobStorage('{azure_storage_account_url}', 'cont', 'test_simple_4.csv', '{azure_account_name}', '{azure_account_key}', 'CSV')", + f"AzureBlobStorage('{azure_storage_account_url}', 'cont', 'test_simple_5.csv', '{azure_account_name}', '{azure_account_key}', 'CSV', 'none')", + f"IcebergAzure('{azure_conn_string}', 'cont', 'test_simple_0_{table_suffix}.csv')", + f"IcebergAzure('{azure_storage_account_url}', 'cont', 'test_simple_1_{table_suffix}.csv', '{azure_account_name}', '{azure_account_key}')", + f"IcebergAzure(named_collection_2, connection_string = '{azure_conn_string}', container = 'cont', blob_path = 'test_simple_2_{table_suffix}.csv', format = 'CSV')", + f"IcebergAzure(named_collection_2, storage_account_url = '{azure_storage_account_url}', container = 'cont', blob_path = 'test_simple_3_{table_suffix}.csv', account_name = '{azure_account_name}', account_key = '{azure_account_key}')", + f"Iceberg(storage_type='azure', '{azure_conn_string}', 'cont', 'test_simple_4_{table_suffix}.csv')", + f"Iceberg(storage_type='azure', '{azure_storage_account_url}', 'cont', 'test_simple_5_{table_suffix}.csv', '{azure_account_name}', '{azure_account_key}')", + f"Iceberg(storage_type='azure', named_collection_2, connection_string = '{azure_conn_string}', container = 'cont', blob_path = 'test_simple_6_{table_suffix}.csv', format = 'CSV')", + f"Iceberg(storage_type='azure', named_collection_2, storage_account_url = '{azure_storage_account_url}', container = 'cont', blob_path = 'test_simple_7_{table_suffix}.csv', account_name = '{azure_account_name}', account_key = '{azure_account_key}')", f"S3('https://my-s3-endpoint/bucket/data.csv', 'myaccess', '{password}', 'CSV')", f"Kafka() SETTINGS kafka_broker_list = '127.0.0.1', kafka_topic_list = 'topic', kafka_group_name = 'group', kafka_format = 'JSONEachRow', kafka_security_protocol = 'sasl_ssl', kafka_sasl_mechanism = 'PLAIN', kafka_sasl_username = 'user', kafka_sasl_password = '{password}', format_avro_schema_registry_url = 'http://schema_user:{password}@'", f"Kafka() SETTINGS kafka_broker_list = '127.0.0.1', kafka_topic_list = 'topic', kafka_group_name = 'group', kafka_format = 'JSONEachRow', kafka_security_protocol = 'sasl_ssl', kafka_sasl_mechanism = 'PLAIN', kafka_sasl_username = 'user', kafka_sasl_password = '{password}', format_avro_schema_registry_url = 'http://schema_user:{password}@domain.com'", @@ -312,7 +330,7 @@ def test_create_table(): ] def make_test_case(i): - table_name = f"table{i}" + table_name = f"table{i}_{table_suffix}" table_engine = table_engines[i] error = None if isinstance(table_engine, tuple): @@ -331,18 +349,18 @@ def make_test_case(i): for toggle, secret in enumerate(["[HIDDEN]", password]): assert ( - node.query(f"SHOW CREATE TABLE table0 {show_secrets}={toggle}") - == "CREATE TABLE default.table0\\n(\\n `x` Int32\\n)\\n" + node.query(f"SHOW CREATE TABLE table0_{table_suffix} {show_secrets}={toggle}") + == f"CREATE TABLE default.table0_{table_suffix}\\n(\\n `x` Int32\\n)\\n" "ENGINE = MySQL(\\'mysql80:3306\\', \\'mysql_db\\', " f"\\'mysql_table\\', \\'mysql_user\\', \\'{secret}\\')\n" ) assert node.query( - f"SELECT create_table_query, engine_full FROM system.tables WHERE name = 'table0' {show_secrets}={toggle}" + f"SELECT create_table_query, engine_full FROM system.tables WHERE name = 'table0_{table_suffix}' {show_secrets}={toggle}" ) == TSV( [ [ - "CREATE TABLE default.table0 (`x` Int32) ENGINE = MySQL(\\'mysql80:3306\\', \\'mysql_db\\', " + f"CREATE TABLE default.table0_{table_suffix} (`x` Int32) ENGINE = MySQL(\\'mysql80:3306\\', \\'mysql_db\\', " f"\\'mysql_table\\', \\'mysql_user\\', \\'{secret}\\')", f"MySQL(\\'mysql80:3306\\', \\'mysql_db\\', \\'mysql_table\\', \\'mysql_user\\', \\'{secret}\\')", ], @@ -352,7 +370,7 @@ def make_test_case(i): create_table_statement_counter = 0 def generate_create_table_numbered(tail): nonlocal create_table_statement_counter - result = f"CREATE TABLE table{create_table_statement_counter} {tail}" + result = f"CREATE TABLE table{create_table_statement_counter}_{table_suffix} {tail}" create_table_statement_counter += 1 return result @@ -383,11 +401,9 @@ def generate_create_table_numbered(tail): generate_create_table_numbered("(`x` int) ENGINE = S3Queue('http://minio1:9001/root/data/', 'CSV') SETTINGS mode = 'ordered', after_processing = 'move', after_processing_move_uri = 'http://minio1:9001/chprocessed', after_processing_move_access_key_id = 'minio', after_processing_move_secret_access_key = '[HIDDEN]'"), generate_create_table_numbered("(`x` int) ENGINE = Iceberg('http://minio1:9001/root/data/test11.csv.gz', 'minio', '[HIDDEN]')"), generate_create_table_numbered("(`x` int) ENGINE = IcebergS3('http://minio1:9001/root/data/test11.csv.gz', 'minio', '[HIDDEN]')"), + generate_create_table_numbered("(`x` int) ENGINE = Iceberg(storage_type = 's3', 'http://minio1:9001/root/data/test11.csv.gz', 'minio', '[HIDDEN]')"), generate_create_table_numbered(f"(`x` int) ENGINE = AzureBlobStorage('{masked_azure_conn_string}', 'cont', 'test_simple.csv', 'CSV')"), generate_create_table_numbered(f"(`x` int) ENGINE = AzureBlobStorage('{masked_azure_conn_string}', 'cont', 'test_simple_1.csv', 'CSV', 'none')"), - generate_create_table_numbered(f"(`x` int) ENGINE = AzureBlobStorage('{azure_storage_account_url}', 'cont', 'test_simple_2.csv', '{azure_account_name}', '[HIDDEN]')"), - generate_create_table_numbered(f"(`x` int) ENGINE = AzureBlobStorage('{azure_storage_account_url}', 'cont', 'test_simple_3.csv', '{azure_account_name}', '[HIDDEN]', 'CSV')"), - generate_create_table_numbered(f"(`x` int) ENGINE = AzureBlobStorage('{azure_storage_account_url}', 'cont', 'test_simple_4.csv', '{azure_account_name}', '[HIDDEN]', 'CSV', 'none')"), generate_create_table_numbered(f"(`x` int) ENGINE = AzureQueue('{masked_azure_conn_string}', 'cont', '*', 'CSV') SETTINGS mode = 'unordered'"), generate_create_table_numbered(f"(`x` int) ENGINE = AzureQueue('{masked_azure_conn_string}', 'cont', '*', 'CSV', 'none') SETTINGS mode = 'unordered'"), generate_create_table_numbered(f"(`x` int) ENGINE = AzureQueue('{masked_azure_conn_string}', 'cont', '*', 'CSV') SETTINGS mode = 'unordered', after_processing = 'move', after_processing_move_connection_string = '{masked_azure_conn_string}', after_processing_move_container = 'chprocessed'",), @@ -395,6 +411,19 @@ def generate_create_table_numbered(tail): generate_create_table_numbered(f"(`x` int) ENGINE = AzureQueue('{azure_storage_account_url}', 'cont', '*', '{azure_account_name}', '[HIDDEN]', 'CSV') SETTINGS mode = 'unordered'"), generate_create_table_numbered(f"(`x` int) ENGINE = AzureQueue('{azure_storage_account_url}', 'cont', '*', '{azure_account_name}', '[HIDDEN]', 'CSV', 'none') SETTINGS mode = 'unordered'"), generate_create_table_numbered(f"(`x` int) ENGINE = AzureBlobStorage('{masked_sas_conn_string}', 'exampledatasets', 'example.csv')"), + generate_create_table_numbered(f"(`x` int) ENGINE = AzureBlobStorage(named_collection_2, connection_string = '{masked_azure_conn_string}', container = 'cont', blob_path = 'test_simple_7.csv', format = 'CSV')"), + generate_create_table_numbered(f"(`x` int) ENGINE = AzureBlobStorage(named_collection_2, storage_account_url = '{azure_storage_account_url}', container = 'cont', blob_path = 'test_simple_8.csv', account_name = '{azure_account_name}', account_key = '[HIDDEN]')"), + generate_create_table_numbered(f"(`x` int) ENGINE = AzureBlobStorage('{azure_storage_account_url}', 'cont', 'test_simple_3.csv', '{azure_account_name}', '[HIDDEN]')"), + generate_create_table_numbered(f"(`x` int) ENGINE = AzureBlobStorage('{azure_storage_account_url}', 'cont', 'test_simple_4.csv', '{azure_account_name}', '[HIDDEN]', 'CSV')"), + generate_create_table_numbered(f"(`x` int) ENGINE = AzureBlobStorage('{azure_storage_account_url}', 'cont', 'test_simple_5.csv', '{azure_account_name}', '[HIDDEN]', 'CSV', 'none')"), + generate_create_table_numbered(f"(`x` int) ENGINE = IcebergAzure('{masked_azure_conn_string}', 'cont', 'test_simple_0_{table_suffix}.csv')"), + generate_create_table_numbered(f"(`x` int) ENGINE = IcebergAzure('{azure_storage_account_url}', 'cont', 'test_simple_1_{table_suffix}.csv', '{azure_account_name}', '[HIDDEN]')"), + generate_create_table_numbered(f"(`x` int) ENGINE = IcebergAzure(named_collection_2, connection_string = '{masked_azure_conn_string}', container = 'cont', blob_path = 'test_simple_2_{table_suffix}.csv', format = 'CSV')"), + generate_create_table_numbered(f"(`x` int) ENGINE = IcebergAzure(named_collection_2, storage_account_url = '{azure_storage_account_url}', container = 'cont', blob_path = 'test_simple_3_{table_suffix}.csv', account_name = '{azure_account_name}', account_key = '[HIDDEN]')"), + generate_create_table_numbered(f"(`x` int) ENGINE = Iceberg(storage_type = 'azure', '{masked_azure_conn_string}', 'cont', 'test_simple_4_{table_suffix}.csv')"), + generate_create_table_numbered(f"(`x` int) ENGINE = Iceberg(storage_type = 'azure', '{azure_storage_account_url}', 'cont', 'test_simple_5_{table_suffix}.csv', '{azure_account_name}', '[HIDDEN]')"), + generate_create_table_numbered(f"(`x` int) ENGINE = Iceberg(storage_type = 'azure', named_collection_2, connection_string = '{masked_azure_conn_string}', container = 'cont', blob_path = 'test_simple_6_{table_suffix}.csv', format = 'CSV')"), + generate_create_table_numbered(f"(`x` int) ENGINE = Iceberg(storage_type = 'azure', named_collection_2, storage_account_url = '{azure_storage_account_url}', container = 'cont', blob_path = 'test_simple_7_{table_suffix}.csv', account_name = '{azure_account_name}', account_key = '[HIDDEN]')"), generate_create_table_numbered("(`x` int) ENGINE = S3('https://my-s3-endpoint/bucket/data.csv', 'myaccess', '[HIDDEN]', 'CSV')"), generate_create_table_numbered("(`x` int) ENGINE = Kafka SETTINGS kafka_broker_list = '127.0.0.1', kafka_topic_list = 'topic', kafka_group_name = 'group', kafka_format = 'JSONEachRow', kafka_security_protocol = 'sasl_ssl', kafka_sasl_mechanism = 'PLAIN', kafka_sasl_username = 'user', kafka_sasl_password = '[HIDDEN]', format_avro_schema_registry_url = 'http://schema_user:[HIDDEN]@'"), generate_create_table_numbered("(`x` int) ENGINE = Kafka SETTINGS kafka_broker_list = '127.0.0.1', kafka_topic_list = 'topic', kafka_group_name = 'group', kafka_format = 'JSONEachRow', kafka_security_protocol = 'sasl_ssl', kafka_sasl_mechanism = 'PLAIN', kafka_sasl_username = 'user', kafka_sasl_password = '[HIDDEN]', format_avro_schema_registry_url = 'http://schema_user:[HIDDEN]@domain.com'"), @@ -535,9 +564,22 @@ def test_table_functions(): f"azureBlobStorage(named_collection_2, connection_string = '{azure_conn_string}', container = 'cont', blob_path = 'test_simple_7.csv', format = 'CSV')", f"azureBlobStorage(named_collection_2, storage_account_url = '{azure_storage_account_url}', container = 'cont', blob_path = 'test_simple_8.csv', account_name = '{azure_account_name}', account_key = '{azure_account_key}')", f"iceberg('http://minio1:9001/root/data/test11.csv.gz', 'minio', '{password}')", - f"gcs('http://minio1:9001/root/data/test11.csv.gz', 'minio', '{password}')", + f"iceberg(named_collection_2, url = 'http://minio1:9001/root/data/test4.csv', access_key_id = 'minio', secret_access_key = '{password}')", f"icebergS3('http://minio1:9001/root/data/test11.csv.gz', 'minio', '{password}')", + f"icebergS3(named_collection_2, url = 'http://minio1:9001/root/data/test4.csv', access_key_id = 'minio', secret_access_key = '{password}')", + f"icebergAzure('{azure_conn_string}', 'cont', 'test_simple.csv')", + f"icebergAzure('{azure_storage_account_url}', 'cont', 'test_simple.csv', '{azure_account_name}', '{azure_account_key}')", f"icebergAzure('{azure_storage_account_url}', 'cont', 'test_simple_6.csv', '{azure_account_name}', '{azure_account_key}', 'CSV', 'none', 'auto')", + f"icebergAzure(named_collection_2, connection_string = '{azure_conn_string}', container = 'cont', blob_path = 'test_simple_7.csv', format = 'CSV')", + f"icebergAzure(named_collection_2, storage_account_url = '{azure_storage_account_url}', container = 'cont', blob_path = 'test_simple_8.csv', account_name = '{azure_account_name}', account_key = '{azure_account_key}')", + f"iceberg(storage_type='s3', 'http://minio1:9001/root/data/test11.csv.gz', 'minio', '{password}')", + f"iceberg(storage_type='s3', named_collection_2, url = 'http://minio1:9001/root/data/test4.csv', access_key_id = 'minio', secret_access_key = '{password}')", + f"iceberg(storage_type='azure', '{azure_conn_string}', 'cont', 'test_simple.csv')", + f"iceberg(storage_type='azure', '{azure_storage_account_url}', 'cont', 'test_simple.csv', '{azure_account_name}', '{azure_account_key}')", + f"iceberg(storage_type='azure', '{azure_storage_account_url}', 'cont', 'test_simple_6.csv', '{azure_account_name}', '{azure_account_key}', 'CSV', 'none', 'auto')", + f"iceberg(storage_type='azure', named_collection_2, connection_string = '{azure_conn_string}', container = 'cont', blob_path = 'test_simple_7.csv', format = 'CSV')", + f"iceberg(storage_type='azure', named_collection_2, storage_account_url = '{azure_storage_account_url}', container = 'cont', blob_path = 'test_simple_8.csv', account_name = '{azure_account_name}', account_key = '{azure_account_key}')", + f"gcs('http://minio1:9001/root/data/test11.csv.gz', 'minio', '{password}')", f"deltaLakeAzure('{azure_storage_account_url}', 'cont', 'test_simple_6.csv', '{azure_account_name}', '{azure_account_key}', 'CSV', 'none', 'auto')" if has_delta_lake else (f"deltaLakeAzure('{azure_storage_account_url}', 'cont', 'test_simple_6.csv', '{azure_account_name}', '{azure_account_key}', 'CSV', 'none', 'auto')", "UNKNOWN_FUNCTION"), f"hudi('http://minio1:9001/root/data/test7.csv', 'minio', '{password}')", f"arrowFlight('arrowflight1:5006', 'dataset', 'arrowflight_user', '{password}')", @@ -642,30 +684,43 @@ def make_test_case(i): f"CREATE TABLE tablefunc37 (`x` int) AS azureBlobStorage(named_collection_2, connection_string = '{masked_azure_conn_string}', container = 'cont', blob_path = 'test_simple_7.csv', format = 'CSV')", f"CREATE TABLE tablefunc38 (`x` int) AS azureBlobStorage(named_collection_2, storage_account_url = '{azure_storage_account_url}', container = 'cont', blob_path = 'test_simple_8.csv', account_name = '{azure_account_name}', account_key = '[HIDDEN]')", "CREATE TABLE tablefunc39 (`x` int) AS iceberg('http://minio1:9001/root/data/test11.csv.gz', 'minio', '[HIDDEN]')", - "CREATE TABLE tablefunc40 (`x` int) AS gcs('http://minio1:9001/root/data/test11.csv.gz', 'minio', '[HIDDEN]')", + "CREATE TABLE tablefunc40 (`x` int) AS iceberg(named_collection_2, url = 'http://minio1:9001/root/data/test4.csv', access_key_id = 'minio', secret_access_key = '[HIDDEN]')", "CREATE TABLE tablefunc41 (`x` int) AS icebergS3('http://minio1:9001/root/data/test11.csv.gz', 'minio', '[HIDDEN]')", - f"CREATE TABLE tablefunc42 (`x` int) AS icebergAzure('{azure_storage_account_url}', 'cont', 'test_simple_6.csv', '{azure_account_name}', '[HIDDEN]', 'CSV', 'none', 'auto')", - f"CREATE TABLE tablefunc43 (`x` int) AS deltaLakeAzure('{azure_storage_account_url}', 'cont', 'test_simple_6.csv', '{azure_account_name}', '[HIDDEN]', 'CSV', 'none', 'auto')", - "CREATE TABLE tablefunc44 (`x` int) AS hudi('http://minio1:9001/root/data/test7.csv', 'minio', '[HIDDEN]')", - "CREATE TABLE tablefunc45 (`x` int) AS arrowFlight('arrowflight1:5006', 'dataset', 'arrowflight_user', '[HIDDEN]')", - "CREATE TABLE tablefunc46 (`x` int) AS arrowFlight(named_collection_1, host = 'arrowflight1', port = 5006, dataset = 'dataset', username = 'arrowflight_user', password = '[HIDDEN]')", - "CREATE TABLE tablefunc47 (`x` int) AS arrowflight(named_collection_1, host = 'arrowflight1', port = 5006, dataset = 'dataset', username = 'arrowflight_user', password = '[HIDDEN]')", - "CREATE TABLE tablefunc48 (`x` int) AS url('https://username:[HIDDEN]@domain.com/path', 'CSV')", - "CREATE TABLE tablefunc49 (`x` int) AS redis('localhost', 'key', 'key Int64', 0, '[HIDDEN]')", - "CREATE TABLE tablefunc50 (`x` int) AS jdbc('[HIDDEN]', 'mydb', 'mytable')", - "CREATE TABLE tablefunc51 (`x` int) AS odbc('[HIDDEN]', 'mydb', 'mytable')", - "CREATE TABLE tablefunc52 (`x` int) AS jdbc('jdbc://user:[HIDDEN]@localhost:5432/mydb', 'mydb', 'mytable')", - "CREATE TABLE tablefunc53 (`x` int) AS odbc('odbc://user:[HIDDEN]@localhost:5432/mydb', 'mydb', 'mytable')", - "CREATE TABLE tablefunc54 (`x` int) AS jdbc(named_collection_1, datasource = '[HIDDEN]')", - "CREATE TABLE tablefunc55 (`x` int) AS odbc(named_collection_1, connection_settings = '[HIDDEN]')", - "CREATE TABLE tablefunc56 (`x` int) AS jdbc(named_collection_1, datasource = 'jdbc://user:[HIDDEN]@localhost:5432/mydb')", - "CREATE TABLE tablefunc57 (`x` int) AS odbc(named_collection_1, connection_settings = 'odbc://user:[HIDDEN]@localhost:5432/mydb')", - "CREATE TABLE tablefunc58 (`x` int) AS jdbc(named_collection_1, datasource = '[HIDDEN]', connection_settings = '[HIDDEN]')", - "CREATE TABLE tablefunc59 (`x` int) AS jdbc(named_collection_1, connection_settings = '[HIDDEN]', external_database = '[HIDDEN]', datasource = '[HIDDEN]')", - "CREATE TABLE tablefunc60 (`x` int) AS deltaLakeS3('http://minio1:9001/root/data/test11.csv.gz', 'minio', '[HIDDEN]')", - "CREATE TABLE tablefunc61 (`x` int) AS paimon('http://minio1:9001/root/data/test11.csv.gz', 'minio', '[HIDDEN]')", - "CREATE TABLE tablefunc62 (`x` int) AS paimonS3('http://minio1:9001/root/data/test11.csv.gz', 'minio', '[HIDDEN]')", - f"CREATE TABLE tablefunc63 (`x` int) AS paimonAzure('{azure_storage_account_url}', 'cont', 'test_simple_6.csv', '{azure_account_name}', '[HIDDEN]', 'CSV', 'none', 'auto')", + "CREATE TABLE tablefunc42 (`x` int) AS icebergS3(named_collection_2, url = 'http://minio1:9001/root/data/test4.csv', access_key_id = 'minio', secret_access_key = '[HIDDEN]')", + f"CREATE TABLE tablefunc43 (`x` int) AS icebergAzure('{masked_azure_conn_string}', 'cont', 'test_simple.csv')", + f"CREATE TABLE tablefunc44 (`x` int) AS icebergAzure('{azure_storage_account_url}', 'cont', 'test_simple.csv', '{azure_account_name}', '[HIDDEN]')", + f"CREATE TABLE tablefunc45 (`x` int) AS icebergAzure('{azure_storage_account_url}', 'cont', 'test_simple_6.csv', '{azure_account_name}', '[HIDDEN]', 'CSV', 'none', 'auto')", + f"CREATE TABLE tablefunc46 (`x` int) AS icebergAzure(named_collection_2, connection_string = '{masked_azure_conn_string}', container = 'cont', blob_path = 'test_simple_7.csv', format = 'CSV')", + f"CREATE TABLE tablefunc47 (`x` int) AS icebergAzure(named_collection_2, storage_account_url = '{azure_storage_account_url}', container = 'cont', blob_path = 'test_simple_8.csv', account_name = '{azure_account_name}', account_key = '[HIDDEN]')", + "CREATE TABLE tablefunc48 (`x` int) AS iceberg(storage_type = 's3', 'http://minio1:9001/root/data/test11.csv.gz', 'minio', '[HIDDEN]')", + "CREATE TABLE tablefunc49 (`x` int) AS iceberg(storage_type = 's3', named_collection_2, url = 'http://minio1:9001/root/data/test4.csv', access_key_id = 'minio', secret_access_key = '[HIDDEN]')", + f"CREATE TABLE tablefunc50 (`x` int) AS iceberg(storage_type = 'azure', '{masked_azure_conn_string}', 'cont', 'test_simple.csv')", + f"CREATE TABLE tablefunc51 (`x` int) AS iceberg(storage_type = 'azure', '{azure_storage_account_url}', 'cont', 'test_simple.csv', '{azure_account_name}', '[HIDDEN]')", + f"CREATE TABLE tablefunc52 (`x` int) AS iceberg(storage_type = 'azure', '{azure_storage_account_url}', 'cont', 'test_simple_6.csv', '{azure_account_name}', '[HIDDEN]', 'CSV', 'none', 'auto')", + f"CREATE TABLE tablefunc53 (`x` int) AS iceberg(storage_type = 'azure', named_collection_2, connection_string = '{masked_azure_conn_string}', container = 'cont', blob_path = 'test_simple_7.csv', format = 'CSV')", + f"CREATE TABLE tablefunc54 (`x` int) AS iceberg(storage_type = 'azure', named_collection_2, storage_account_url = '{azure_storage_account_url}', container = 'cont', blob_path = 'test_simple_8.csv', account_name = '{azure_account_name}', account_key = '[HIDDEN]')", + "CREATE TABLE tablefunc55 (`x` int) AS gcs('http://minio1:9001/root/data/test11.csv.gz', 'minio', '[HIDDEN]')", + f"CREATE TABLE tablefunc56 (`x` int) AS deltaLakeAzure('{azure_storage_account_url}', 'cont', 'test_simple_6.csv', '{azure_account_name}', '[HIDDEN]', 'CSV', 'none', 'auto')", + "CREATE TABLE tablefunc57 (`x` int) AS hudi('http://minio1:9001/root/data/test7.csv', 'minio', '[HIDDEN]')", + "CREATE TABLE tablefunc58 (`x` int) AS arrowFlight('arrowflight1:5006', 'dataset', 'arrowflight_user', '[HIDDEN]')", + "CREATE TABLE tablefunc59 (`x` int) AS arrowFlight(named_collection_1, host = 'arrowflight1', port = 5006, dataset = 'dataset', username = 'arrowflight_user', password = '[HIDDEN]')", + "CREATE TABLE tablefunc60 (`x` int) AS arrowflight(named_collection_1, host = 'arrowflight1', port = 5006, dataset = 'dataset', username = 'arrowflight_user', password = '[HIDDEN]')", + "CREATE TABLE tablefunc61 (`x` int) AS url('https://username:[HIDDEN]@domain.com/path', 'CSV')", + "CREATE TABLE tablefunc62 (`x` int) AS redis('localhost', 'key', 'key Int64', 0, '[HIDDEN]')", + "CREATE TABLE tablefunc63 (`x` int) AS jdbc('[HIDDEN]', 'mydb', 'mytable')", + "CREATE TABLE tablefunc64 (`x` int) AS odbc('[HIDDEN]', 'mydb', 'mytable')", + "CREATE TABLE tablefunc65 (`x` int) AS jdbc('jdbc://user:[HIDDEN]@localhost:5432/mydb', 'mydb', 'mytable')", + "CREATE TABLE tablefunc66 (`x` int) AS odbc('odbc://user:[HIDDEN]@localhost:5432/mydb', 'mydb', 'mytable')", + "CREATE TABLE tablefunc67 (`x` int) AS jdbc(named_collection_1, datasource = '[HIDDEN]')", + "CREATE TABLE tablefunc68 (`x` int) AS odbc(named_collection_1, connection_settings = '[HIDDEN]')", + "CREATE TABLE tablefunc69 (`x` int) AS jdbc(named_collection_1, datasource = 'jdbc://user:[HIDDEN]@localhost:5432/mydb')", + "CREATE TABLE tablefunc70 (`x` int) AS odbc(named_collection_1, connection_settings = 'odbc://user:[HIDDEN]@localhost:5432/mydb')", + "CREATE TABLE tablefunc71 (`x` int) AS jdbc(named_collection_1, datasource = '[HIDDEN]', connection_settings = '[HIDDEN]')", + "CREATE TABLE tablefunc72 (`x` int) AS jdbc(named_collection_1, connection_settings = '[HIDDEN]', external_database = '[HIDDEN]', datasource = '[HIDDEN]')", + "CREATE TABLE tablefunc73 (`x` int) AS deltaLakeS3('http://minio1:9001/root/data/test11.csv.gz', 'minio', '[HIDDEN]')", + "CREATE TABLE tablefunc74 (`x` int) AS paimon('http://minio1:9001/root/data/test11.csv.gz', 'minio', '[HIDDEN]')", + "CREATE TABLE tablefunc75 (`x` int) AS paimonS3('http://minio1:9001/root/data/test11.csv.gz', 'minio', '[HIDDEN]')", + f"CREATE TABLE tablefunc76 (`x` int) AS paimonAzure('{azure_storage_account_url}', 'cont', 'test_simple_6.csv', '{azure_account_name}', '[HIDDEN]', 'CSV', 'none', 'auto')", ], must_not_contain=[password], ) diff --git a/tests/integration/test_s3_cache_locality/__init__.py b/tests/integration/test_s3_cache_locality/__init__.py new file mode 100644 index 000000000000..e69de29bb2d1 diff --git a/tests/integration/test_s3_cache_locality/configs/cluster.xml b/tests/integration/test_s3_cache_locality/configs/cluster.xml new file mode 100644 index 000000000000..db54c35374b9 --- /dev/null +++ b/tests/integration/test_s3_cache_locality/configs/cluster.xml @@ -0,0 +1,126 @@ + + + + + + + + clickhouse1 + 9000 + + + clickhouse2 + 9000 + + + clickhouse3 + 9000 + + + clickhouse4 + 9000 + + + clickhouse5 + 9000 + + + + + + + + clickhouse1 + 9000 + + + clickhouse2 + 9000 + + + clickhouse3 + 9000 + + + clickhouse4 + 9000 + + + + + + + + clickhouse2 + 9000 + + + clickhouse3 + 9000 + + + clickhouse4 + 9000 + + + clickhouse5 + 9000 + + + + + + + + clickhouse3 + 9000 + + + clickhouse4 + 9000 + + + clickhouse5 + 9000 + + + clickhouse1 + 9000 + + + clickhouse2 + 9000 + + + + + + + + clickhouse4 + 9000 + + + clickhouse5 + 9000 + + + clickhouse2 + 9000 + + + clickhouse3 + 9000 + + + + + + + + + /var/lib/clickhouse/raw_s3_cache + 10Gi + + + diff --git a/tests/integration/test_s3_cache_locality/configs/named_collections.xml b/tests/integration/test_s3_cache_locality/configs/named_collections.xml new file mode 100644 index 000000000000..6994aa3f5e77 --- /dev/null +++ b/tests/integration/test_s3_cache_locality/configs/named_collections.xml @@ -0,0 +1,10 @@ + + + + http://minio1:9001/root/data/* + minio + ClickHouse_Minio_P@ssw0rd + CSV> + + + diff --git a/tests/integration/test_s3_cache_locality/configs/users.xml b/tests/integration/test_s3_cache_locality/configs/users.xml new file mode 100644 index 000000000000..4b6ba057ecb1 --- /dev/null +++ b/tests/integration/test_s3_cache_locality/configs/users.xml @@ -0,0 +1,9 @@ + + + + + default + 1 + + + diff --git a/tests/integration/test_s3_cache_locality/test.py b/tests/integration/test_s3_cache_locality/test.py new file mode 100644 index 000000000000..68993d85aeed --- /dev/null +++ b/tests/integration/test_s3_cache_locality/test.py @@ -0,0 +1,262 @@ +import csv +import logging +import os +import shutil +import uuid + +import pytest + +from helpers.cluster import ClickHouseCluster +from helpers.config_cluster import minio_secret_key + + +logging.getLogger().setLevel(logging.INFO) +logging.getLogger().addHandler(logging.StreamHandler()) + +SCRIPT_DIR = os.path.dirname(os.path.realpath(__file__)) + + +def create_buckets_s3(cluster, files=1000): + minio = cluster.minio_client + + s3_data = [] + + for file_number in range(files): + file_name = f"data/generated_{files}/file_{file_number}.csv" + os.makedirs(os.path.join(SCRIPT_DIR, f"data/generated_{files}/"), exist_ok=True) + s3_data.append(file_name) + with open(os.path.join(SCRIPT_DIR, file_name), "w+", encoding="utf-8") as f: + # a String, b UInt64 + data = [] + + # Make all files a bit different + data.append( + ["str_" + str(file_number), file_number] + ) + + writer = csv.writer(f) + writer.writerows(data) + + for file in s3_data: + minio.fput_object( + bucket_name=cluster.minio_bucket, + object_name=file, + file_path=os.path.join(SCRIPT_DIR, file), + ) + + for obj in minio.list_objects(cluster.minio_bucket, recursive=True): + print(obj.object_name) + + +@pytest.fixture(scope="module") +def started_cluster(): + try: + cluster = ClickHouseCluster(__file__) + # clickhouse0 not a member of cluster_XXX + for i in range(6): + cluster.add_instance( + f"clickhouse{i}", + main_configs=["configs/cluster.xml", "configs/named_collections.xml"], + user_configs=["configs/users.xml"], + macros={"replica": f"clickhouse{i}"}, + with_minio=True, + with_zookeeper=True, + stay_alive=True, + ) + + logging.info("Starting cluster...") + cluster.start() + logging.info("Cluster started") + + create_buckets_s3(cluster) + create_buckets_s3(cluster, files=3) + + yield cluster + finally: + shutil.rmtree(os.path.join(SCRIPT_DIR, "data/generated_1000/"), ignore_errors=True) + shutil.rmtree(os.path.join(SCRIPT_DIR, "data/generated_3/"), ignore_errors=True) + cluster.shutdown() + + +def check_s3_gets(cluster, node, expected_result, cluster_first, cluster_second, enable_filesystem_cache, + lock_object_storage_task_distribution_ms, files=1000): + for host in list(cluster.instances.values()): + host.query("SYSTEM DROP FILESYSTEM CACHE 'raw_s3_cache'", ignore_error=True) + + settings = { + "enable_filesystem_cache": enable_filesystem_cache, + "filesystem_cache_name": "'raw_s3_cache'", + } + + settings["lock_object_storage_task_distribution_ms"] = lock_object_storage_task_distribution_ms + + query_id_first = str(uuid.uuid4()) + result_first = node.query( + f""" + SELECT count(*) + FROM s3Cluster('{cluster_first}', 'http://minio1:9001/root/data/generated_{files}/*', 'minio', '{minio_secret_key}', 'CSV', 'a String, b UInt64') + WHERE b=42 + SETTINGS {",".join(f"{k}={v}" for k, v in settings.items())} + """, + query_id=query_id_first, + ) + assert result_first == expected_result + query_id_second = str(uuid.uuid4()) + result_second = node.query( + f""" + SELECT count(*) + FROM s3Cluster('{cluster_second}', 'http://minio1:9001/root/data/generated_{files}/*', 'minio', '{minio_secret_key}', 'CSV', 'a String, b UInt64') + WHERE b=42 + SETTINGS {",".join(f"{k}={v}" for k, v in settings.items())} + """, + query_id=query_id_second, + ) + assert result_second == expected_result + + node.query(f"SYSTEM FLUSH LOGS ON CLUSTER {cluster_first}") + node.query(f"SYSTEM FLUSH LOGS ON CLUSTER {cluster_second}") + + s3_get_first = node.query( + f""" + SELECT sum(ProfileEvents['S3GetObject']) + FROM clusterAllReplicas('{cluster_first}', system.query_log) + WHERE type='QueryFinish' + AND initial_query_id='{query_id_first}' + """, + ) + s3_get_second = node.query( + f""" + SELECT sum(ProfileEvents['S3GetObject']) + FROM clusterAllReplicas('{cluster_second}', system.query_log) + WHERE type='QueryFinish' + AND initial_query_id='{query_id_second}' + """, + ) + + return int(s3_get_first), int(s3_get_second) + + +def check_s3_gets_by_hosts(cluster, node, expected_result, + lock_object_storage_task_distribution_ms, files=1000): + settings = { + "enable_filesystem_cache": False, + } + + settings["lock_object_storage_task_distribution_ms"] = lock_object_storage_task_distribution_ms + query_id = str(uuid.uuid4()) + result = node.query( + f""" + SELECT count(*) + FROM s3Cluster('{cluster}', 'http://minio1:9001/root/data/generated_{files}/*', 'minio', '{minio_secret_key}', 'CSV', 'a String, b UInt64') + WHERE b=42 + SETTINGS {",".join(f"{k}={v}" for k, v in settings.items())} + """, + query_id=query_id, + ) + assert result == expected_result + + node.query(f"SYSTEM FLUSH LOGS ON CLUSTER {cluster}") + + s3_get = node.query( + f""" + SELECT ProfileEvents['S3GetObject'] + FROM clusterAllReplicas('{cluster}', system.query_log) + WHERE type='QueryFinish' + AND initial_query_id='{query_id}' + ORDER BY hostname + """, + ) + + return [int(events) for events in s3_get.strip().split("\n")] + + +def check_s3_gets_repeat(cluster, node, expected_result, cluster_first, cluster_second, enable_filesystem_cache, + lock_object_storage_task_distribution_ms): + # Repeat test several times to get average result + iterations = 1 if lock_object_storage_task_distribution_ms > 0 else 10 + s3_get_first_sum = 0 + s3_get_second_sum = 0 + for _ in range(iterations): + (s3_get_first, s3_get_second) = check_s3_gets(cluster, node, expected_result, cluster_first, cluster_second, enable_filesystem_cache, lock_object_storage_task_distribution_ms) + s3_get_first_sum += s3_get_first + s3_get_second_sum += s3_get_second + return s3_get_first_sum, s3_get_second_sum + + +@pytest.mark.parametrize("lock_object_storage_task_distribution_ms ", [0, 30000]) +def test_cache_locality(started_cluster, lock_object_storage_task_distribution_ms): + node = started_cluster.instances["clickhouse0"] + + expected_result = node.query( + f""" + SELECT count(*) + FROM s3('http://minio1:9001/root/data/generated_1000/*', 'minio', '{minio_secret_key}', 'CSV', 'a String, b UInt64') + WHERE b=42 + """ + ) + + # Algorithm does not give 100% guarantee, so add 10% on dispersion + dispersion = 0.0 if lock_object_storage_task_distribution_ms > 0 else 0.1 + + # No cache + (s3_get_first, s3_get_second) = check_s3_gets_repeat(started_cluster, node, expected_result, 'cluster_12345', 'cluster_12345', 0, lock_object_storage_task_distribution_ms) + assert s3_get_second == s3_get_first + + # With cache + (s3_get_first, s3_get_second) = check_s3_gets_repeat(started_cluster, node, expected_result, 'cluster_12345', 'cluster_12345', 1, lock_object_storage_task_distribution_ms) + assert s3_get_second <= s3_get_first * dispersion + + # Different replicas order + (s3_get_first, s3_get_second) = check_s3_gets_repeat(started_cluster, node, expected_result, 'cluster_12345', 'cluster_34512', 1, lock_object_storage_task_distribution_ms) + assert s3_get_second <= s3_get_first * dispersion + + # No last replica + (s3_get_first, s3_get_second) = check_s3_gets_repeat(started_cluster, node, expected_result, 'cluster_12345', 'cluster_1234', 1, lock_object_storage_task_distribution_ms) + assert s3_get_second <= s3_get_first * (0.179 + dispersion) # actual value - 179 of 1000 files changed replica + + # No first replica + (s3_get_first, s3_get_second) = check_s3_gets_repeat(started_cluster, node, expected_result, 'cluster_12345', 'cluster_2345', 1, lock_object_storage_task_distribution_ms) + assert s3_get_second <= s3_get_first * (0.189 + dispersion) # actual value - 189 of 1000 files changed replica + + # No first replica, different replicas order + (s3_get_first, s3_get_second) = check_s3_gets_repeat(started_cluster, node, expected_result, 'cluster_12345', 'cluster_4523', 1, lock_object_storage_task_distribution_ms) + assert s3_get_second <= s3_get_first * (0.189 + dispersion) + + # Add new replica, different replicas order + (s3_get_first, s3_get_second) = check_s3_gets_repeat(started_cluster, node, expected_result, 'cluster_4523', 'cluster_12345', 1, lock_object_storage_task_distribution_ms) + assert s3_get_second <= s3_get_first * (0.189 + dispersion) + + # New replica and old replica, different replicas order + # All files from removed replica changed replica + # Some files from existed replicas changed replica on the new replica + (s3_get_first, s3_get_second) = check_s3_gets_repeat(started_cluster, node, expected_result, 'cluster_1234', 'cluster_4523', 1, lock_object_storage_task_distribution_ms) + assert s3_get_second <= s3_get_first * (0.368 + dispersion) # actual value - 368 of 1000 changed replica + + if (lock_object_storage_task_distribution_ms > 0): + s3_get = check_s3_gets_by_hosts('cluster_12345', node, expected_result, lock_object_storage_task_distribution_ms, files=1000) + assert s3_get == [189,210,220,202,179] + s3_get = check_s3_gets_by_hosts('cluster_1234', node, expected_result, lock_object_storage_task_distribution_ms, files=1000) + assert s3_get == [247,243,264,246] + s3_get = check_s3_gets_by_hosts('cluster_2345', node, expected_result, lock_object_storage_task_distribution_ms, files=1000) + assert s3_get == [251,280,248,221] + + +def test_cache_locality_few_files(started_cluster): + node = started_cluster.instances["clickhouse0"] + + expected_result = node.query( + f""" + SELECT count(*) + FROM s3('http://minio1:9001/root/data/generated_3/*', 'minio', '{minio_secret_key}', 'CSV', 'a String, b UInt64') + WHERE b=42 + """ + ) + + # Rendezvous hash makes the next distribution: + # file_0 - clickhouse1 + # file_1 - clickhouse4 + # file_2 - clickhouse3 + # The same distribution must be in each query + for _ in range(10): + s3_get = check_s3_gets_by_hosts('cluster_12345', node, expected_result, lock_object_storage_task_distribution_ms=30000, files=3) + assert s3_get == [1,0,1,1,0] diff --git a/tests/integration/test_s3_cluster/configs/cluster.xml b/tests/integration/test_s3_cluster/configs/cluster.xml index 84e6afd12f71..9d98df479576 100644 --- a/tests/integration/test_s3_cluster/configs/cluster.xml +++ b/tests/integration/test_s3_cluster/configs/cluster.xml @@ -20,6 +20,20 @@ + + + + + s0_0_1 + 9000 + + + s0_1_0 + 9000 + + + + @@ -49,6 +63,94 @@ + + + + c2.s0_0_0 + 9000 + + + c2.s0_0_1 + 9000 + + + + + + + + s0_0_1 + 9000 + foo + bar + + + s0_1_0 + 9000 + foo + bar + + + + + + + + c2.s0_0_0 + 9000 + biz + bar + + + c2.s0_0_1 + 9000 + biz + bar + + + + + + baz + + + s0_0_1 + 9000 + foo + + + s0_1_0 + 9000 + foo + + + + + + + + s0_0_0 + 9000 + + + s0_0_1 + 9000 + + + s0_1_0 + 9000 + + + c2.s0_0_0 + 9000 + + + c2.s0_0_1 + 9000 + + + + cluster_simple diff --git a/tests/integration/test_s3_cluster/configs/hidden_clusters.xml b/tests/integration/test_s3_cluster/configs/hidden_clusters.xml new file mode 100644 index 000000000000..8816cca1c79b --- /dev/null +++ b/tests/integration/test_s3_cluster/configs/hidden_clusters.xml @@ -0,0 +1,20 @@ + + + + + + s0_0_1 + 9000 + foo + bar + + + s0_1_0 + 9000 + foo + bar + + + + + diff --git a/tests/integration/test_s3_cluster/configs/users.xml b/tests/integration/test_s3_cluster/configs/users.xml index 2b00ef132a59..a5b4603cc4e2 100644 --- a/tests/integration/test_s3_cluster/configs/users.xml +++ b/tests/integration/test_s3_cluster/configs/users.xml @@ -5,6 +5,9 @@ by default. Allow it for this test. --> 1 + + hidden_cluster_with_username_and_password + @@ -12,5 +15,13 @@ default 1 + + bar + default + + + bar + osc + diff --git a/tests/integration/test_s3_cluster/data/graceful/part0.csv b/tests/integration/test_s3_cluster/data/graceful/part0.csv new file mode 100644 index 000000000000..2a8ceabbea58 --- /dev/null +++ b/tests/integration/test_s3_cluster/data/graceful/part0.csv @@ -0,0 +1 @@ +0,"Foo" \ No newline at end of file diff --git a/tests/integration/test_s3_cluster/data/graceful/part1.csv b/tests/integration/test_s3_cluster/data/graceful/part1.csv new file mode 100644 index 000000000000..1950012fffd2 --- /dev/null +++ b/tests/integration/test_s3_cluster/data/graceful/part1.csv @@ -0,0 +1 @@ +1,"Bar" \ No newline at end of file diff --git a/tests/integration/test_s3_cluster/data/graceful/part2.csv b/tests/integration/test_s3_cluster/data/graceful/part2.csv new file mode 100644 index 000000000000..dc782d5adf9b --- /dev/null +++ b/tests/integration/test_s3_cluster/data/graceful/part2.csv @@ -0,0 +1 @@ +2,"Foo" \ No newline at end of file diff --git a/tests/integration/test_s3_cluster/data/graceful/part3.csv b/tests/integration/test_s3_cluster/data/graceful/part3.csv new file mode 100644 index 000000000000..6e581549d23c --- /dev/null +++ b/tests/integration/test_s3_cluster/data/graceful/part3.csv @@ -0,0 +1 @@ +3,"Bar" \ No newline at end of file diff --git a/tests/integration/test_s3_cluster/data/graceful/part4.csv b/tests/integration/test_s3_cluster/data/graceful/part4.csv new file mode 100644 index 000000000000..bb5a4d956c51 --- /dev/null +++ b/tests/integration/test_s3_cluster/data/graceful/part4.csv @@ -0,0 +1 @@ +4,"Foo" \ No newline at end of file diff --git a/tests/integration/test_s3_cluster/data/graceful/part5.csv b/tests/integration/test_s3_cluster/data/graceful/part5.csv new file mode 100644 index 000000000000..5cb2c6be144b --- /dev/null +++ b/tests/integration/test_s3_cluster/data/graceful/part5.csv @@ -0,0 +1 @@ +5,"Bar" \ No newline at end of file diff --git a/tests/integration/test_s3_cluster/data/graceful/part6.csv b/tests/integration/test_s3_cluster/data/graceful/part6.csv new file mode 100644 index 000000000000..e2e2428d100d --- /dev/null +++ b/tests/integration/test_s3_cluster/data/graceful/part6.csv @@ -0,0 +1 @@ +6,"Foo" \ No newline at end of file diff --git a/tests/integration/test_s3_cluster/data/graceful/part7.csv b/tests/integration/test_s3_cluster/data/graceful/part7.csv new file mode 100644 index 000000000000..3c819a315c20 --- /dev/null +++ b/tests/integration/test_s3_cluster/data/graceful/part7.csv @@ -0,0 +1 @@ +7,"Bar" \ No newline at end of file diff --git a/tests/integration/test_s3_cluster/data/graceful/part8.csv b/tests/integration/test_s3_cluster/data/graceful/part8.csv new file mode 100644 index 000000000000..72f39e512be3 --- /dev/null +++ b/tests/integration/test_s3_cluster/data/graceful/part8.csv @@ -0,0 +1 @@ +8,"Foo" \ No newline at end of file diff --git a/tests/integration/test_s3_cluster/data/graceful/part9.csv b/tests/integration/test_s3_cluster/data/graceful/part9.csv new file mode 100644 index 000000000000..f288cb2051dd --- /dev/null +++ b/tests/integration/test_s3_cluster/data/graceful/part9.csv @@ -0,0 +1 @@ +9,"Bar" \ No newline at end of file diff --git a/tests/integration/test_s3_cluster/data/graceful/partA.csv b/tests/integration/test_s3_cluster/data/graceful/partA.csv new file mode 100644 index 000000000000..da99f68ba784 --- /dev/null +++ b/tests/integration/test_s3_cluster/data/graceful/partA.csv @@ -0,0 +1 @@ +10,"Foo" \ No newline at end of file diff --git a/tests/integration/test_s3_cluster/data/graceful/partB.csv b/tests/integration/test_s3_cluster/data/graceful/partB.csv new file mode 100644 index 000000000000..46591e0be815 --- /dev/null +++ b/tests/integration/test_s3_cluster/data/graceful/partB.csv @@ -0,0 +1 @@ +11,"Bar" \ No newline at end of file diff --git a/tests/integration/test_s3_cluster/data/graceful/partC.csv b/tests/integration/test_s3_cluster/data/graceful/partC.csv new file mode 100644 index 000000000000..24af8010b5c6 --- /dev/null +++ b/tests/integration/test_s3_cluster/data/graceful/partC.csv @@ -0,0 +1 @@ +12,"Foo" \ No newline at end of file diff --git a/tests/integration/test_s3_cluster/data/graceful/partD.csv b/tests/integration/test_s3_cluster/data/graceful/partD.csv new file mode 100644 index 000000000000..0365a5024871 --- /dev/null +++ b/tests/integration/test_s3_cluster/data/graceful/partD.csv @@ -0,0 +1 @@ +13,"Bar" \ No newline at end of file diff --git a/tests/integration/test_s3_cluster/data/graceful/partE.csv b/tests/integration/test_s3_cluster/data/graceful/partE.csv new file mode 100644 index 000000000000..3143c0eed915 --- /dev/null +++ b/tests/integration/test_s3_cluster/data/graceful/partE.csv @@ -0,0 +1 @@ +14,"Foo" \ No newline at end of file diff --git a/tests/integration/test_s3_cluster/data/graceful/partF.csv b/tests/integration/test_s3_cluster/data/graceful/partF.csv new file mode 100644 index 000000000000..d0306b9bb806 --- /dev/null +++ b/tests/integration/test_s3_cluster/data/graceful/partF.csv @@ -0,0 +1 @@ +15,"Bar" \ No newline at end of file diff --git a/tests/integration/test_s3_cluster/test.py b/tests/integration/test_s3_cluster/test.py index 30d5a6190217..00978fb7231c 100644 --- a/tests/integration/test_s3_cluster/test.py +++ b/tests/integration/test_s3_cluster/test.py @@ -2,10 +2,13 @@ import logging import os import shutil +import threading +import time import uuid import pytest +from helpers.client import QueryRuntimeException from helpers.cluster import ClickHouseCluster from helpers.config_cluster import minio_access_key, minio_secret_key from helpers.mock_servers import start_mock_servers @@ -23,6 +26,22 @@ "data/clickhouse/part123.csv", "data/database/part2.csv", "data/database/partition675.csv", + "data/graceful/part0.csv", + "data/graceful/part1.csv", + "data/graceful/part2.csv", + "data/graceful/part3.csv", + "data/graceful/part4.csv", + "data/graceful/part5.csv", + "data/graceful/part6.csv", + "data/graceful/part7.csv", + "data/graceful/part8.csv", + "data/graceful/part9.csv", + "data/graceful/partA.csv", + "data/graceful/partB.csv", + "data/graceful/partC.csv", + "data/graceful/partD.csv", + "data/graceful/partE.csv", + "data/graceful/partF.csv", ] @@ -114,6 +133,7 @@ def started_cluster(): macros={"replica": "node1", "shard": "shard1"}, with_minio=True, with_zookeeper=True, + stay_alive=True, ) cluster.add_instance( "s0_0_1", @@ -121,6 +141,7 @@ def started_cluster(): user_configs=["configs/users.xml"], macros={"replica": "replica2", "shard": "shard1"}, with_zookeeper=True, + stay_alive=True, ) cluster.add_instance( "s0_1_0", @@ -128,6 +149,23 @@ def started_cluster(): user_configs=["configs/users.xml"], macros={"replica": "replica1", "shard": "shard2"}, with_zookeeper=True, + stay_alive=True, + ) + cluster.add_instance( + "c2.s0_0_0", + main_configs=["configs/cluster.xml", "configs/named_collections.xml", "configs/hidden_clusters.xml"], + user_configs=["configs/users.xml"], + macros={"replica": "replica1", "shard": "shard1"}, + with_zookeeper=True, + stay_alive=True, + ) + cluster.add_instance( + "c2.s0_0_1", + main_configs=["configs/cluster.xml", "configs/named_collections.xml", "configs/hidden_clusters.xml"], + user_configs=["configs/users.xml"], + macros={"replica": "replica2", "shard": "shard1"}, + with_zookeeper=True, + stay_alive=True, ) logging.info("Starting cluster...") @@ -274,6 +312,21 @@ def test_wrong_cluster(started_cluster): assert "not found" in error + error = node.query_and_get_error( + f""" + SELECT count(*) from s3( + 'http://minio1:9001/root/data/{{clickhouse,database}}/*', + 'minio', '{minio_secret_key}', 'CSV', 'name String, value UInt32, polygon Array(Array(Tuple(Float64, Float64)))') + UNION ALL + SELECT count(*) from s3( + 'http://minio1:9001/root/data/{{clickhouse,database}}/*', + 'minio', '{minio_secret_key}', 'CSV', 'name String, value UInt32, polygon Array(Array(Tuple(Float64, Float64)))') + SETTINGS object_storage_cluster = 'non_existing_cluster' + """ + ) + + assert "not found" in error + def test_ambiguous_join(started_cluster): node = started_cluster.instances["s0_0_0"] @@ -292,6 +345,20 @@ def test_ambiguous_join(started_cluster): ) assert "AMBIGUOUS_COLUMN_NAME" not in result + result = node.query( + f""" + SELECT l.name, r.value from s3( + 'http://minio1:9001/root/data/{{clickhouse,database}}/*', 'minio', '{minio_secret_key}', 'CSV', + 'name String, value UInt32, polygon Array(Array(Tuple(Float64, Float64)))') as l + JOIN s3( + 'http://minio1:9001/root/data/{{clickhouse,database}}/*', 'minio', '{minio_secret_key}', 'CSV', + 'name String, value UInt32, polygon Array(Array(Tuple(Float64, Float64)))') as r + ON l.name = r.name + SETTINGS object_storage_cluster = 'cluster_simple' + """ + ) + assert "AMBIGUOUS_COLUMN_NAME" not in result + def test_skip_unavailable_shards(started_cluster): node = started_cluster.instances["s0_0_0"] @@ -307,6 +374,17 @@ def test_skip_unavailable_shards(started_cluster): assert result == "10\n" + result = node.query( + f""" + SELECT count(*) from s3( + 'http://minio1:9001/root/data/clickhouse/part1.csv', + 'minio', '{minio_secret_key}', 'CSV', 'name String, value UInt32, polygon Array(Array(Tuple(Float64, Float64)))') + SETTINGS skip_unavailable_shards = 1, object_storage_cluster = 'cluster_non_existent_port' + """ + ) + + assert result == "10\n" + def test_unset_skip_unavailable_shards(started_cluster): # Although skip_unavailable_shards is not set, cluster table functions should always skip unavailable shards. @@ -322,6 +400,17 @@ def test_unset_skip_unavailable_shards(started_cluster): assert result == "10\n" + result = node.query( + f""" + SELECT count(*) from s3( + 'http://minio1:9001/root/data/clickhouse/part1.csv', + 'minio', '{minio_secret_key}', 'CSV', 'name String, value UInt32, polygon Array(Array(Tuple(Float64, Float64)))') + SETTINGS object_storage_cluster = 'cluster_non_existent_port' + """ + ) + + assert result == "10\n" + def test_distributed_insert_select_with_replicated(started_cluster): first_replica_first_shard = started_cluster.instances["s0_0_0"] @@ -502,6 +591,18 @@ def test_cluster_format_detection(started_cluster): assert result == expected_result + result = node.query( + f"SELECT * FROM s3('http://minio1:9001/root/data/generated/*', 'minio', '{minio_secret_key}') order by c1, c2 SETTINGS object_storage_cluster = 'cluster_simple'" + ) + + assert result == expected_result + + result = node.query( + f"SELECT * FROM s3('http://minio1:9001/root/data/generated/*', 'minio', '{minio_secret_key}', auto, 'a String, b UInt64') order by a, b SETTINGS object_storage_cluster = 'cluster_simple'" + ) + + assert result == expected_result + def test_cluster_default_expression(started_cluster): node = started_cluster.instances["s0_0_0"] @@ -550,6 +651,615 @@ def test_cluster_default_expression(started_cluster): assert result == expected_result + result = node.query( + f"SELECT * FROM s3('http://minio1:9001/root/data/data{{1,2,3}}', 'minio', '{minio_secret_key}', 'JSONEachRow', 'id UInt32, date Date DEFAULT 18262') order by id SETTINGS object_storage_cluster = 'cluster_simple'" + ) + + assert result == expected_result + + result = node.query( + f"SELECT * FROM s3('http://minio1:9001/root/data/data{{1,2,3}}', 'minio', '{minio_secret_key}', 'auto', 'id UInt32, date Date DEFAULT 18262') order by id SETTINGS object_storage_cluster = 'cluster_simple'" + ) + + assert result == expected_result + + result = node.query( + f"SELECT * FROM s3('http://minio1:9001/root/data/data{{1,2,3}}', 'minio', '{minio_secret_key}', 'JSONEachRow', 'id UInt32, date Date DEFAULT 18262', 'auto') order by id SETTINGS object_storage_cluster = 'cluster_simple'" + ) + + assert result == expected_result + + result = node.query( + f"SELECT * FROM s3('http://minio1:9001/root/data/data{{1,2,3}}', 'minio', '{minio_secret_key}', 'auto', 'id UInt32, date Date DEFAULT 18262', 'auto') order by id SETTINGS object_storage_cluster = 'cluster_simple'" + ) + + assert result == expected_result + + result = node.query( + "SELECT * FROM s3(test_s3_with_default) order by id SETTINGS object_storage_cluster = 'cluster_simple'" + ) + + assert result == expected_result + + +def test_distributed_s3_table_engine(started_cluster): + node = started_cluster.instances["s0_0_0"] + + resp_def = node.query( + f""" + SELECT * from s3Cluster( + 'cluster_simple', + 'http://minio1:9001/root/data/{{clickhouse,database}}/*', 'minio', '{minio_secret_key}', 'CSV', + 'name String, value UInt32, polygon Array(Array(Tuple(Float64, Float64)))') ORDER BY (name, value, polygon) + """ + ) + + node.query("DROP TABLE IF EXISTS single_node"); + node.query( + f""" + CREATE TABLE single_node + (name String, value UInt32, polygon Array(Array(Tuple(Float64, Float64)))) + ENGINE=S3('http://minio1:9001/root/data/{{clickhouse,database}}/*', 'minio', '{minio_secret_key}', 'CSV') + """ + ) + query_id_engine_single_node = str(uuid.uuid4()) + resp_engine_single_node = node.query( + """ + SELECT * FROM single_node ORDER BY (name, value, polygon) + """, + query_id = query_id_engine_single_node + ) + assert resp_def == resp_engine_single_node + + node.query("DROP TABLE IF EXISTS distributed"); + node.query( + f""" + CREATE TABLE distributed + (name String, value UInt32, polygon Array(Array(Tuple(Float64, Float64)))) + ENGINE=S3('http://minio1:9001/root/data/{{clickhouse,database}}/*', 'minio', '{minio_secret_key}', 'CSV') + SETTINGS object_storage_cluster='cluster_simple' + """ + ) + query_id_engine_distributed = str(uuid.uuid4()) + resp_engine_distributed = node.query( + """ + SELECT * FROM distributed ORDER BY (name, value, polygon) + """, + query_id = query_id_engine_distributed + ) + assert resp_def == resp_engine_distributed + + node.query("SYSTEM FLUSH LOGS ON CLUSTER 'cluster_simple'") + + hosts_engine_single_node = node.query( + f""" + SELECT uniq(hostname) + FROM clusterAllReplicas('cluster_simple', system.query_log) + WHERE type='QueryFinish' AND initial_query_id='{query_id_engine_single_node}' + """ + ) + assert int(hosts_engine_single_node) == 1 + hosts_engine_distributed = node.query( + f""" + SELECT uniq(hostname) + FROM clusterAllReplicas('cluster_simple', system.query_log) + WHERE type='QueryFinish' AND initial_query_id='{query_id_engine_distributed}' + """ + ) + assert int(hosts_engine_distributed) == 3 + + +def test_cluster_hosts_limit(started_cluster): + node = started_cluster.instances["s0_0_0"] + + query_id_def = str(uuid.uuid4()) + resp_def = node.query( + f""" + SELECT * from s3Cluster( + 'cluster_simple', + 'http://minio1:9001/root/data/{{clickhouse,database}}/*', 'minio', '{minio_secret_key}', 'CSV', + 'name String, value UInt32, polygon Array(Array(Tuple(Float64, Float64)))') ORDER BY (name, value, polygon) + """, + query_id = query_id_def + ) + + # object_storage_max_nodes is greater than number of hosts in cluster + query_id_4_hosts = str(uuid.uuid4()) + resp_4_hosts = node.query( + f""" + SELECT * from s3Cluster( + 'cluster_simple', + 'http://minio1:9001/root/data/{{clickhouse,database}}/*', 'minio', '{minio_secret_key}', 'CSV', + 'name String, value UInt32, polygon Array(Array(Tuple(Float64, Float64)))') ORDER BY (name, value, polygon) + SETTINGS object_storage_max_nodes=4 + """, + query_id = query_id_4_hosts + ) + assert resp_def == resp_4_hosts + + # object_storage_max_nodes is equal number of hosts in cluster + query_id_3_hosts = str(uuid.uuid4()) + resp_3_hosts = node.query( + f""" + SELECT * from s3Cluster( + 'cluster_simple', + 'http://minio1:9001/root/data/{{clickhouse,database}}/*', 'minio', '{minio_secret_key}', 'CSV', + 'name String, value UInt32, polygon Array(Array(Tuple(Float64, Float64)))') ORDER BY (name, value, polygon) + SETTINGS object_storage_max_nodes=3 + """, + query_id = query_id_3_hosts + ) + assert resp_def == resp_3_hosts + + # object_storage_max_nodes is less than number of hosts in cluster + query_id_2_hosts = str(uuid.uuid4()) + resp_2_hosts = node.query( + f""" + SELECT * from s3Cluster( + 'cluster_simple', + 'http://minio1:9001/root/data/{{clickhouse,database}}/*', 'minio', '{minio_secret_key}', 'CSV', + 'name String, value UInt32, polygon Array(Array(Tuple(Float64, Float64)))') ORDER BY (name, value, polygon) + SETTINGS object_storage_max_nodes=2 + """, + query_id = query_id_2_hosts + ) + assert resp_def == resp_2_hosts + + node.query("SYSTEM FLUSH LOGS ON CLUSTER 'cluster_simple'") + + hosts_def = node.query( + f""" + SELECT uniq(hostname) + FROM clusterAllReplicas('cluster_simple', system.query_log) + WHERE type='QueryFinish' AND initial_query_id='{query_id_def}' AND query_id!='{query_id_def}' + """ + ) + assert int(hosts_def) == 3 + + hosts_4 = node.query( + f""" + SELECT uniq(hostname) + FROM clusterAllReplicas('cluster_simple', system.query_log) + WHERE type='QueryFinish' AND initial_query_id='{query_id_4_hosts}' AND query_id!='{query_id_4_hosts}' + """ + ) + assert int(hosts_4) == 3 + + hosts_3 = node.query( + f""" + SELECT uniq(hostname) + FROM clusterAllReplicas('cluster_simple', system.query_log) + WHERE type='QueryFinish' AND initial_query_id='{query_id_3_hosts}' AND query_id!='{query_id_3_hosts}' + """ + ) + assert int(hosts_3) == 3 + + hosts_2 = node.query( + f""" + SELECT uniq(hostname) + FROM clusterAllReplicas('cluster_simple', system.query_log) + WHERE type='QueryFinish' AND initial_query_id='{query_id_2_hosts}' AND query_id!='{query_id_2_hosts}' + """ + ) + assert int(hosts_2) == 2 + + +def test_object_storage_remote_initiator(started_cluster): + node = started_cluster.instances["s0_0_0"] + + # Simple cluster + query_id = uuid.uuid4().hex + result = node.query( + f""" + SELECT * from s3Cluster( + 'cluster_remote', + 'http://minio1:9001/root/data/{{clickhouse,database}}/*', 'minio', '{minio_secret_key}', 'CSV', + 'name String, value UInt32, polygon Array(Array(Tuple(Float64, Float64)))') ORDER BY (name, value, polygon) + SETTINGS object_storage_remote_initiator=1 + """, + query_id = query_id, + ) + + assert result is not None + + node.query("SYSTEM FLUSH LOGS ON CLUSTER 'cluster_all'") + queries = node.query( + f""" + SELECT count() + FROM clusterAllReplicas('cluster_all', system.query_log) + WHERE type='QueryFinish' AND initial_query_id='{query_id}' + FORMAT TSV + """ + ).splitlines() + + # initial node + remote initiator + 2 subqueries on replicas + assert queries == ["4"] + + # Cluster with dots in the host names + query_id = uuid.uuid4().hex + result = node.query( + f""" + SELECT * from s3Cluster( + 'cluster_with_dots', + 'http://minio1:9001/root/data/{{clickhouse,database}}/*', 'minio', '{minio_secret_key}', 'CSV', + 'name String, value UInt32, polygon Array(Array(Tuple(Float64, Float64)))') ORDER BY (name, value, polygon) + SETTINGS object_storage_remote_initiator=1 + """, + query_id = query_id, + ) + + assert result is not None + + node.query("SYSTEM FLUSH LOGS ON CLUSTER 'cluster_all'") + queries = node.query( + f""" + SELECT count() + FROM clusterAllReplicas('cluster_all', system.query_log) + WHERE type='QueryFinish' AND initial_query_id='{query_id}' + FORMAT TSV + """ + ).splitlines() + + # initial node + remote initiator + 2 subqueries on replicas + assert queries == ["4"] + + users = node.query( + f""" + SELECT DISTINCT hostname, user + FROM clusterAllReplicas('cluster_all', system.query_log) + WHERE type='QueryFinish' AND initial_query_id='{query_id}' + ORDER BY ALL + FORMAT TSV + """ + ).splitlines() + + assert users == ["c2.s0_0_0\tdefault", + "c2.s0_0_1\tdefault", + "s0_0_0\tdefault"] + + # Cluster with user and password + query_id = uuid.uuid4().hex + result = node.query( + f""" + SELECT * from s3Cluster( + 'cluster_with_username_and_password', + 'http://minio1:9001/root/data/{{clickhouse,database}}/*', 'minio', '{minio_secret_key}', 'CSV', + 'name String, value UInt32, polygon Array(Array(Tuple(Float64, Float64)))') ORDER BY (name, value, polygon) + SETTINGS object_storage_remote_initiator=1 + """, + query_id = query_id, + ) + + assert result is not None + + node.query("SYSTEM FLUSH LOGS ON CLUSTER 'cluster_all'") + queries = node.query( + f""" + SELECT count() + FROM clusterAllReplicas('cluster_all', system.query_log) + WHERE type='QueryFinish' AND initial_query_id='{query_id}' + FORMAT TSV + """ + ).splitlines() + + # initial node + remote initiator + 2 subqueries on replicas + assert queries == ["4"] + + users = node.query( + f""" + SELECT DISTINCT hostname, user + FROM clusterAllReplicas('cluster_all', system.query_log) + WHERE type='QueryFinish' AND initial_query_id='{query_id}' + ORDER BY ALL + FORMAT TSV + """ + ).splitlines() + + assert users == ["s0_0_0\tdefault", + "s0_0_1\tfoo", + "s0_1_0\tfoo"] + + # Cluster with secret + query_id = uuid.uuid4().hex + result = node.query_and_get_error( + f""" + SELECT * from s3Cluster( + 'cluster_with_secret', + 'http://minio1:9001/root/data/{{clickhouse,database}}/*', 'minio', '{minio_secret_key}', 'CSV', + 'name String, value UInt32, polygon Array(Array(Tuple(Float64, Float64)))') ORDER BY (name, value, polygon) + SETTINGS object_storage_remote_initiator=1 + """, + query_id = query_id, + ) + + assert "Can't convert query to remote when cluster uses secret" in result + + # Different cluster for remote initiator and query execution + # with `hidden_cluster_with_username_and_password` existed only in `cluster_with_dots` nodes + query_id = uuid.uuid4().hex + + result = node.query( + f""" + SELECT * from s3( + 'http://minio1:9001/root/data/{{clickhouse,database}}/*', 'minio', '{minio_secret_key}', 'CSV', + 'name String, value UInt32, polygon Array(Array(Tuple(Float64, Float64)))') ORDER BY (name, value, polygon) + SETTINGS + object_storage_remote_initiator=1, + object_storage_cluster='hidden_cluster_with_username_and_password', + object_storage_remote_initiator_cluster='cluster_with_dots' + """, + query_id = query_id, + ) + + assert result is not None + + node.query("SYSTEM FLUSH LOGS ON CLUSTER 'cluster_all'") + queries = node.query( + f""" + SELECT count() + FROM clusterAllReplicas('cluster_all', system.query_log) + WHERE type='QueryFinish' AND initial_query_id='{query_id}' + FORMAT TSV + """ + ).splitlines() + + # initial node + remote initiator + 2 subqueries on replicas + assert queries == ["4"] + + users = node.query( + f""" + SELECT DISTINCT hostname, user + FROM clusterAllReplicas('cluster_all', system.query_log) + WHERE type='QueryFinish' AND initial_query_id='{query_id}' + ORDER BY ALL + FORMAT TSV + """ + ).splitlines() + + # Random host from 'cluster_with_dots' for remote query + assert users[0] in ["c2.s0_0_0\tdefault", "c2.s0_0_1\tdefault"] + assert users[1:] == ["s0_0_0\tdefault", + "s0_0_1\tfoo", + "s0_1_0\tfoo"] + + +def test_remote_hedged(started_cluster): + node = started_cluster.instances["s0_0_0"] + pure_s3 = node.query( + f""" + SELECT * from s3( + 'http://minio1:9001/root/data/{{clickhouse,database}}/*', + 'minio', '{minio_secret_key}', 'CSV', + 'name String, value UInt32, polygon Array(Array(Tuple(Float64, Float64)))') + ORDER BY (name, value, polygon) + LIMIT 1 + """ + ) + s3_distributed = node.query( + f""" + SELECT * from remote('s0_0_1', s3Cluster( + 'cluster_simple', + 'http://minio1:9001/root/data/{{clickhouse,database}}/*', 'minio', '{minio_secret_key}', 'CSV', + 'name String, value UInt32, polygon Array(Array(Tuple(Float64, Float64)))')) + ORDER BY (name, value, polygon) + LIMIT 1 + SETTINGS use_hedged_requests=True + """ + ) + + assert TSV(pure_s3) == TSV(s3_distributed) + + +def test_remote_no_hedged(started_cluster): + node = started_cluster.instances["s0_0_0"] + pure_s3 = node.query( + f""" + SELECT * from s3( + 'http://minio1:9001/root/data/{{clickhouse,database}}/*', + 'minio', '{minio_secret_key}', 'CSV', + 'name String, value UInt32, polygon Array(Array(Tuple(Float64, Float64)))') + ORDER BY (name, value, polygon) + LIMIT 1 + """ + ) + s3_distributed = node.query( + f""" + SELECT * from remote('s0_0_1', s3Cluster( + 'cluster_simple', + 'http://minio1:9001/root/data/{{clickhouse,database}}/*', 'minio', '{minio_secret_key}', 'CSV', + 'name String, value UInt32, polygon Array(Array(Tuple(Float64, Float64)))')) + ORDER BY (name, value, polygon) + LIMIT 1 + SETTINGS use_hedged_requests=False + """ + ) + + assert TSV(pure_s3) == TSV(s3_distributed) + + +@pytest.mark.parametrize("join_mode", ["local", "global"]) +def test_joins(started_cluster, join_mode): + node = started_cluster.instances["s0_0_0"] + + # Table join_table only exists on the node 's0_0_0'. + node.query("DROP TABLE IF EXISTS join_table SYNC") + node.query( + """ + CREATE TABLE IF NOT EXISTS join_table ( + id UInt32, + name String + ) ENGINE=MergeTree() + ORDER BY id; + """ + ) + + node.query( + f""" + INSERT INTO join_table + SELECT value, concat(name, '_jt') FROM s3Cluster('cluster_simple', + 'http://minio1:9001/root/data/{{clickhouse,database}}/*', 'minio', '{minio_secret_key}', 'CSV', + 'name String, value UInt32, polygon Array(Array(Tuple(Float64, Float64)))'); + """ + ) + + result1 = node.query( + f""" + SELECT t1.name, t2.name FROM + s3Cluster('cluster_simple', + 'http://minio1:9001/root/data/{{clickhouse,database}}/*', 'minio', '{minio_secret_key}', 'CSV', + 'name String, value UInt32, polygon Array(Array(Tuple(Float64, Float64)))') AS t1 + JOIN + join_table AS t2 + ON t1.value = t2.id + ORDER BY t1.name + SETTINGS object_storage_cluster_join_mode='{join_mode}'; + """ + ) + + res = list(map(str.split, result1.splitlines())) + assert len(res) == 25 + + for line in res: + if len(line) == 2: + assert line[1] == f"{line[0]}_jt" + else: + assert line == ["_jt"] # for empty name + + result2 = node.query( + f""" + SELECT t1.name, t2.name FROM + join_table AS t2 + JOIN + s3Cluster('cluster_simple', + 'http://minio1:9001/root/data/{{clickhouse,database}}/*', 'minio', '{minio_secret_key}', 'CSV', + 'name String, value UInt32, polygon Array(Array(Tuple(Float64, Float64)))') AS t1 + ON t1.value = t2.id + ORDER BY t1.name + SETTINGS object_storage_cluster_join_mode='{join_mode}'; + """ + ) + + assert result1 == result2 + + # With WHERE clause with remote column only + result3 = node.query( + f""" + SELECT t1.name, t2.name FROM + s3Cluster('cluster_simple', + 'http://minio1:9001/root/data/{{clickhouse,database}}/*', 'minio', '{minio_secret_key}', 'CSV', + 'name String, value UInt32, polygon Array(Array(Tuple(Float64, Float64)))') AS t1 + JOIN + join_table AS t2 + ON t1.value = t2.id + WHERE (t1.value % 2) + ORDER BY t1.name + SETTINGS object_storage_cluster_join_mode='{join_mode}'; + """ + ) + + res = list(map(str.split, result3.splitlines())) + assert len(res) == 8 + + # With WHERE clause with local column only + result4 = node.query( + f""" + SELECT t1.name, t2.name FROM + s3Cluster('cluster_simple', + 'http://minio1:9001/root/data/{{clickhouse,database}}/*', 'minio', '{minio_secret_key}', 'CSV', + 'name String, value UInt32, polygon Array(Array(Tuple(Float64, Float64)))') AS t1 + JOIN + join_table AS t2 + ON t1.value = t2.id + WHERE (t2.id % 2) + ORDER BY t1.name + SETTINGS object_storage_cluster_join_mode='{join_mode}'; + """ + ) + + assert result3 == result4 + + # With WHERE clause with local and remote columns + result5 = node.query( + f""" + SELECT t1.name, t2.name FROM + s3Cluster('cluster_simple', + 'http://minio1:9001/root/data/{{clickhouse,database}}/*', 'minio', '{minio_secret_key}', 'CSV', + 'name String, value UInt32, polygon Array(Array(Tuple(Float64, Float64)))') AS t1 + JOIN + join_table AS t2 + ON t1.value = t2.id + WHERE (t1.value % 2) AND ((t2.id % 3) == 2) + ORDER BY t1.name + SETTINGS object_storage_cluster_join_mode='{join_mode}'; + """ + ) + + res = list(map(str.split, result5.splitlines())) + assert len(res) == 6 + + # With WHERE clause with global subquery + result6 = node.query( + f""" + SELECT name FROM + s3Cluster('cluster_simple', + 'http://minio1:9001/root/data/{{clickhouse,database}}/*', 'minio', '{minio_secret_key}', 'CSV', + 'name String, value UInt32, polygon Array(Array(Tuple(Float64, Float64)))') + WHERE value IN (SELECT id FROM join_table) + ORDER BY name + SETTINGS object_storage_cluster_join_mode='{join_mode}'; + """ + ) + res = list(map(str.split, result6.splitlines())) + assert len(res) == 25 + + # With WHERE clause with global subquery + result6 = node.query( + f""" + SELECT name FROM + s3Cluster('cluster_simple', + 'http://minio1:9001/root/data/{{clickhouse,database}}/*', 'minio', '{minio_secret_key}', 'CSV', + 'name String, value UInt32, polygon Array(Array(Tuple(Float64, Float64)))') + WHERE value GLOBAL IN (SELECT id FROM join_table) + ORDER BY name + SETTINGS object_storage_cluster_join_mode='{join_mode}'; + """ + ) + res = list(map(str.split, result6.splitlines())) + assert len(res) == 25 + + # With WHERE clause without columns in condition + result7 = node.query( + f""" + SELECT count() FROM + s3Cluster('cluster_simple', + 'http://minio1:9001/root/data/{{clickhouse,database}}/*', 'minio', '{minio_secret_key}', 'CSV', + 'name String, value UInt32, polygon Array(Array(Tuple(Float64, Float64)))') AS t1 + JOIN + join_table AS t2 + ON 1 + GROUP BY ALL + SETTINGS object_storage_cluster_join_mode='{join_mode}'; + """ + ) + assert result7.strip() == "625" + + # With WHERE clause without columns in condition and with local column in SELECT + result8 = node.query( + f""" + SELECT count(), t2.id FROM + s3Cluster('cluster_simple', + 'http://minio1:9001/root/data/{{clickhouse,database}}/*', 'minio', '{minio_secret_key}', 'CSV', + 'name String, value UInt32, polygon Array(Array(Tuple(Float64, Float64)))') AS t1 + JOIN + join_table AS t2 + ON 1 + GROUP BY ALL + SETTINGS object_storage_cluster_join_mode='{join_mode}'; + """ + ) + res = list(map(str.split, result8.splitlines())) + assert len(res) == 25 + @pytest.mark.parametrize("allow_experimental_analyzer", [0, 1]) @pytest.mark.parametrize("use_partition_strategy", [False, True]) @@ -764,3 +1474,269 @@ def test_iceberg_s3_cluster_read_task_failpoint(started_cluster): ) node.query(f"DROP TABLE IF EXISTS {dst_table}") node.query(f"DROP TABLE IF EXISTS {iceberg_table}") + + +def test_object_storage_remote_initiator_without_cluster_function(started_cluster): + node = started_cluster.instances["s0_0_0"] + + # Remove initiator without cluster request + # Query executed on random node of object_storage_remote_initiator_cluster + query_id = uuid.uuid4().hex + + result = node.query( + f""" + SELECT * from s3( + 'http://minio1:9001/root/data/{{clickhouse,database}}/*', 'minio', '{minio_secret_key}', 'CSV', + 'name String, value UInt32, polygon Array(Array(Tuple(Float64, Float64)))') ORDER BY (name, value, polygon) + SETTINGS + object_storage_remote_initiator=1, + object_storage_remote_initiator_cluster='cluster_with_dots' + """, + query_id = query_id, + ) + + assert result is not None + + node.query("SYSTEM FLUSH LOGS ON CLUSTER 'cluster_all'") + queries = node.query( + f""" + SELECT count() + FROM clusterAllReplicas('cluster_all', system.query_log) + WHERE type='QueryFinish' AND initial_query_id='{query_id}' + FORMAT TSV + """ + ).splitlines() + + # initial node + remote initiator + assert queries == ["2"] + + users = node.query( + f""" + SELECT DISTINCT hostname, user + FROM clusterAllReplicas('cluster_all', system.query_log) + WHERE type='QueryFinish' AND initial_query_id='{query_id}' + ORDER BY ALL + FORMAT TSV + """ + ).splitlines() + + # Random host from 'cluster_with_dots' for remote query + assert users[0] in ["c2.s0_0_0\tdefault", "c2.s0_0_1\tdefault"] + assert users[1:] == ["s0_0_0\tdefault"] + + # Remove initiator without cluster request + # but with `object_storage_cluster` specified for user on remote cluster + query_id = uuid.uuid4().hex + + result = node.query( + f""" + SELECT * from s3( + 'http://minio1:9001/root/data/{{clickhouse,database}}/*', 'minio', '{minio_secret_key}', 'CSV', + 'name String, value UInt32, polygon Array(Array(Tuple(Float64, Float64)))') ORDER BY (name, value, polygon) + SETTINGS + object_storage_remote_initiator=1, + object_storage_remote_initiator_cluster='cluster_with_dots_and_user' + """, + query_id = query_id, + ) + + assert result is not None + + node.query("SYSTEM FLUSH LOGS ON CLUSTER 'cluster_all'") + queries = node.query( + f""" + SELECT count() + FROM clusterAllReplicas('cluster_all', system.query_log) + WHERE type='QueryFinish' AND initial_query_id='{query_id}' + FORMAT TSV + """ + ).splitlines() + + # initial node + remote initiator + 2 subqueries on replicas + assert queries == ["4"] + + users = node.query( + f""" + SELECT DISTINCT hostname, user + FROM clusterAllReplicas('cluster_all', system.query_log) + WHERE type='QueryFinish' AND initial_query_id='{query_id}' + ORDER BY ALL + FORMAT TSV + """ + ).splitlines() + + # Random host from 'cluster_with_dots' for remote query + assert users[0] in ["c2.s0_0_0\tbiz", "c2.s0_0_1\tbiz"] + assert users[1:] == ["s0_0_0\tdefault", + "s0_0_1\tfoo", + "s0_1_0\tfoo"] + + +def test_object_storage_remote_initiator_aggregation(started_cluster): + node = started_cluster.instances["s0_0_0"] + + # Remove initiator without cluster request + # Check that aggregation works on nodes + query_id = uuid.uuid4().hex + + result = node.query( + f""" + SELECT sum(value) from s3( + 'http://minio1:9001/root/data/{{clickhouse,database}}/*', 'minio', '{minio_secret_key}', 'CSV', + 'name String, value UInt32, polygon Array(Array(Tuple(Float64, Float64)))') + SETTINGS + object_storage_remote_initiator=1, + object_storage_remote_initiator_cluster='cluster_with_dots_and_user' + """, + query_id = query_id, + ) + + assert result == "67802152770\n" + + node.query("SYSTEM FLUSH LOGS ON CLUSTER 'cluster_all'") + result_rows = node.query( + f""" + SELECT sum(result_rows) + FROM clusterAllReplicas('cluster_all', system.query_log) + WHERE type='QueryFinish' AND initial_query_id='{query_id}' + AND is_initial_query = 0 + ORDER BY ALL + FORMAT TSV + """ + ).splitlines() + + # Data processed on cluster 'hidden_cluster_with_username_and_password'. + # Cluster contains two nodes, each returns one row. + assert result_rows == ["2"] + + # Remove initiator without cluster request + # Check that aggregation works on nodes + query_id = uuid.uuid4().hex + + result = node.query( + f""" + SELECT value % 2 as bit, sum(value) from s3( + 'http://minio1:9001/root/data/{{clickhouse,database}}/*', 'minio', '{minio_secret_key}', 'CSV', + 'name String, value UInt32, polygon Array(Array(Tuple(Float64, Float64)))') + GROUP BY bit + ORDER BY bit + SETTINGS + object_storage_remote_initiator=1, + object_storage_remote_initiator_cluster='cluster_with_dots_and_user' + """, + query_id = query_id, + ) + + assert result == "0\t41117771522\n1\t26684381248\n" + + node.query("SYSTEM FLUSH LOGS ON CLUSTER 'cluster_all'") + result_rows = node.query( + f""" + SELECT sum(result_rows) + FROM clusterAllReplicas('cluster_all', system.query_log) + WHERE type='QueryFinish' AND initial_query_id='{query_id}' + AND is_initial_query = 0 + ORDER BY ALL + FORMAT TSV + """ + ).splitlines() + + # Data processed on cluster 'hidden_cluster_with_username_and_password'. + # Cluster contains two nodes, each returns up to two rows, at least two rows totaly. + result_rows = int(result_rows[0]) + assert result_rows >= 2 and result_rows <= 4 + + +def test_hive_partitioning_with_where_condition(started_cluster): + node = started_cluster.instances["s0_0_0"] + test_id = uuid.uuid4().hex[:8] + + for i in range(1, 5): + node.query( + f""" + INSERT INTO FUNCTION s3('http://minio1:9001/root/hive/{test_id}/date=2000-01-0{i}/data.csv', + 'minio','{minio_secret_key}','CSVWithNames','d UInt64') + SELECT number FROM numbers(10) + SETTINGS s3_truncate_on_insert=1 + """) + + # Direct query + result = node.query( + f""" + SELECT count() FROM s3('http://minio1:9001/root/hive/{test_id}/date=*/data.csv', + 'minio','{minio_secret_key}','CSVWithNames','d UInt64') + WHERE date='2000-01-02' + SETTINGS use_hive_partitioning=1 + """ + ) + assert result.strip() == "10" + + result = node.query( + f""" + SELECT count() FROM s3Cluster('cluster_simple', 'http://minio1:9001/root/hive/{test_id}/date=*/data.csv', + 'minio','{minio_secret_key}','CSVWithNames','d UInt64') + WHERE date='2000-01-02' + SETTINGS use_hive_partitioning=1 + """ + ) + assert result.strip() == "10" + + +def test_graceful_shutdown(started_cluster): + node = started_cluster.instances["s0_0_0"] + node_to_shutdown = started_cluster.instances["s0_1_0"] + + expected = TSV("64\tBar\t8\n56\tFoo\t8\n") + + num_lock = threading.Lock() + errors = 0 + + def query_cycle(): + nonlocal errors + try: + i = 0 + while i < 10: + i += 1 + # Query time 3-4 seconds + # Processing single object 1-2 seconds + result = node.query(f""" + SELECT sum(value),name,sum(sleep(1)+1) as sleep FROM s3Cluster( + 'cluster_simple', + 'http://minio1:9001/root/data/graceful/*', 'minio', '{minio_secret_key}', 'CSV', + 'value UInt32, name String') + GROUP BY name + ORDER BY name + SETTINGS max_threads=2 + """) + with num_lock: + if TSV(result) != expected: + errors += 1 + if errors >= 1: + break + except QueryRuntimeException: + with num_lock: + errors += 1 + + threads = [] + + for _ in range(10): + thread = threading.Thread(target=query_cycle) + thread.start() + threads.append(thread) + time.sleep(0.2) + + time.sleep(3) + + node_to_shutdown.query("SYSTEM STOP SWARM MODE") + + # enough time to complete processing of objects, started before "SYSTEM STOP SWARM MODE" + time.sleep(3) + + node_to_shutdown.stop_clickhouse(kill=True) + + for thread in threads: + thread.join() + + node_to_shutdown.start_clickhouse() + + assert errors == 0 diff --git a/tests/integration/test_storage_iceberg_no_spark/configs/config.d/named_collections.xml b/tests/integration/test_storage_iceberg_no_spark/configs/config.d/named_collections.xml index 516e4ba63a3a..7dfec41b2df8 100644 --- a/tests/integration/test_storage_iceberg_no_spark/configs/config.d/named_collections.xml +++ b/tests/integration/test_storage_iceberg_no_spark/configs/config.d/named_collections.xml @@ -11,5 +11,19 @@ + + http://minio1:9001/root/ + minio + ClickHouse_Minio_P@ssw0rd + s3 + + + devstoreaccount1 + Eby8vdM02xNOcqFlqUwJPLlmEtlCDXJ1OUzFT50uSRZ6IFsuFq2UVErCz4I6tq/K1SZFPTOtr/KBHBeksoGMGw== + azure + + + local + diff --git a/tests/integration/test_storage_iceberg_with_spark/configs/config.d/allow_export_partition.xml b/tests/integration/test_storage_iceberg_with_spark/configs/config.d/allow_export_partition.xml new file mode 100644 index 000000000000..514cd710836a --- /dev/null +++ b/tests/integration/test_storage_iceberg_with_spark/configs/config.d/allow_export_partition.xml @@ -0,0 +1,3 @@ + + 1 + diff --git a/tests/integration/test_storage_iceberg_with_spark/configs/config.d/cluster.xml b/tests/integration/test_storage_iceberg_with_spark/configs/config.d/cluster.xml index 54c08b27abe8..5835f155a998 100644 --- a/tests/integration/test_storage_iceberg_with_spark/configs/config.d/cluster.xml +++ b/tests/integration/test_storage_iceberg_with_spark/configs/config.d/cluster.xml @@ -16,5 +16,17 @@ + + + + node2 + 9000 + + + node3 + 9000 + + + diff --git a/tests/integration/test_storage_iceberg_with_spark/configs/config.d/named_collections.xml b/tests/integration/test_storage_iceberg_with_spark/configs/config.d/named_collections.xml index 516e4ba63a3a..7dfec41b2df8 100644 --- a/tests/integration/test_storage_iceberg_with_spark/configs/config.d/named_collections.xml +++ b/tests/integration/test_storage_iceberg_with_spark/configs/config.d/named_collections.xml @@ -11,5 +11,19 @@ + + http://minio1:9001/root/ + minio + ClickHouse_Minio_P@ssw0rd + s3 + + + devstoreaccount1 + Eby8vdM02xNOcqFlqUwJPLlmEtlCDXJ1OUzFT50uSRZ6IFsuFq2UVErCz4I6tq/K1SZFPTOtr/KBHBeksoGMGw== + azure + + + local + diff --git a/tests/integration/test_storage_iceberg_with_spark/configs/users.d/allow_export_partition.xml b/tests/integration/test_storage_iceberg_with_spark/configs/users.d/allow_export_partition.xml new file mode 100644 index 000000000000..db0dd71de565 --- /dev/null +++ b/tests/integration/test_storage_iceberg_with_spark/configs/users.d/allow_export_partition.xml @@ -0,0 +1,6 @@ + + + + + + diff --git a/tests/integration/test_storage_iceberg_with_spark/conftest.py b/tests/integration/test_storage_iceberg_with_spark/conftest.py index ea5282607f7d..e0e76b9e9228 100644 --- a/tests/integration/test_storage_iceberg_with_spark/conftest.py +++ b/tests/integration/test_storage_iceberg_with_spark/conftest.py @@ -82,6 +82,7 @@ def started_cluster_iceberg_with_spark(): with_minio=True, with_azurite=True, stay_alive=True, + with_zookeeper=True, ) cluster.add_instance( "node2", @@ -94,6 +95,7 @@ def started_cluster_iceberg_with_spark(): ], user_configs=["configs/users.d/users.xml"], stay_alive=True, + with_zookeeper=True, ) cluster.add_instance( "node3", @@ -106,6 +108,7 @@ def started_cluster_iceberg_with_spark(): ], user_configs=["configs/users.d/users.xml"], stay_alive=True, + with_zookeeper=True, ) logging.info("Starting cluster...") diff --git a/tests/integration/test_storage_iceberg_with_spark/test_cluster_join_filter_minmax_pruning.py b/tests/integration/test_storage_iceberg_with_spark/test_cluster_join_filter_minmax_pruning.py new file mode 100644 index 000000000000..de594f50a5e6 --- /dev/null +++ b/tests/integration/test_storage_iceberg_with_spark/test_cluster_join_filter_minmax_pruning.py @@ -0,0 +1,152 @@ +import pytest + +from helpers.iceberg_utils import ( + check_validity_and_get_prunned_files_general, + execute_spark_query_general, + get_creation_expression, + get_uuid_str, +) + + +@pytest.mark.parametrize("storage_type", ["s3"]) +def test_cluster_join_filter_minmax_pruning(started_cluster_iceberg_with_spark, storage_type): + """ + icebergCluster lists files on the initiator. A left-only WHERE on + count() of SELECT * … JOIN must still reach that listing so min/max + pruning can skip files (the original icebergCluster JOIN subquery case). + """ + instance = started_cluster_iceberg_with_spark.instances["node1"] + spark = started_cluster_iceberg_with_spark.spark_session + TABLE_NAME = "test_cluster_join_filter_minmax_pruning_" + storage_type + "_" + get_uuid_str() + BAR_NAME = "bar_" + storage_type + "_" + get_uuid_str() + + def execute_spark_query(query: str): + return execute_spark_query_general( + spark, + started_cluster_iceberg_with_spark, + storage_type, + TABLE_NAME, + query, + ) + + execute_spark_query( + f""" + CREATE TABLE {TABLE_NAME} ( + datetime DATE, + symbol VARCHAR(50), + bid INT + ) + USING iceberg + OPTIONS('format-version'='2') + """ + ) + + execute_spark_query(f"INSERT INTO {TABLE_NAME} VALUES (DATE '2024-01-01', 'AAPL', 1)") + execute_spark_query(f"INSERT INTO {TABLE_NAME} VALUES (DATE '2024-01-02', 'AAPL', 2)") + execute_spark_query(f"INSERT INTO {TABLE_NAME} VALUES (DATE '2024-01-03', 'AAPL', 3)") + # Passes `bid >= 3`, fails `datetime >= 2024-01-03`. Distinguishes listing + # that only saw the inner JOIN `WHERE` from listing that also got the outer `WHERE`. + execute_spark_query(f"INSERT INTO {TABLE_NAME} VALUES (DATE '2024-01-01', 'AAPL', 4)") + + iceberg = get_creation_expression( + storage_type, + TABLE_NAME, + started_cluster_iceberg_with_spark, + table_function=True, + run_on_cluster=True, + ) + + instance.query( + f"CREATE TABLE `{BAR_NAME}` (symbol String, comment String) ENGINE = Memory" + ) + instance.query( + f"INSERT INTO `{BAR_NAME}` VALUES ('AAPL', 'comment'), ('AAPL2', 'comment2')" + ) + + common_settings = { + "input_format_parquet_bloom_filter_push_down": 0, + "input_format_parquet_filter_push_down": 0, + "query_plan_filter_push_down": 1, + "enable_analyzer": 1, + "query_plan_join_swap_table": 0, + "enable_join_runtime_filters": 0, + "enable_parallel_replicas": 0, + "join_use_nulls": 1, + } + + def check_validity_and_get_prunned_files(select_expression): + settings1 = {**common_settings, "use_iceberg_partition_pruning": 0} + settings2 = {**common_settings, "use_iceberg_partition_pruning": 1} + return check_validity_and_get_prunned_files_general( + instance, + TABLE_NAME, + settings1, + settings2, + "IcebergMinMaxIndexPrunedFiles", + select_expression, + ) + + # Four data files: bid 1/2/3/4. `bid >= 3` keeps two files (prunes 2). + expected_pruned = 2 + + assert ( + check_validity_and_get_prunned_files( + f"SELECT count() FROM {iceberg} WHERE bid >= 3" + ) + == expected_pruned + ) + + assert ( + check_validity_and_get_prunned_files( + f""" + SELECT count() + FROM {iceberg} AS foo + LEFT JOIN `{BAR_NAME}` AS bar ON foo.symbol = bar.symbol + WHERE foo.bid >= 3 + """ + ) + == expected_pruned + ) + + assert ( + check_validity_and_get_prunned_files( + f"SELECT count() FROM (SELECT * FROM {iceberg} AS foo WHERE foo.bid >= 3)" + ) + == expected_pruned + ) + + assert ( + check_validity_and_get_prunned_files( + f""" + SELECT count() + FROM + ( + SELECT * + FROM {iceberg} AS foo + LEFT JOIN `{BAR_NAME}` AS bar ON foo.symbol = bar.symbol + WHERE foo.bid >= 3 + ) + """ + ) + == expected_pruned + ) + + # Inner `bid >= 3` is copied onto the cluster wrap during planning. The outer + # `datetime` predicate is pushed later; listing must AND it onto the wrap + # DAG or the extra file with bid=4 / datetime=2024-01-01 is not pruned. + assert ( + check_validity_and_get_prunned_files( + f""" + SELECT count() + FROM + ( + SELECT * + FROM {iceberg} AS foo + LEFT JOIN `{BAR_NAME}` AS bar ON foo.symbol = bar.symbol + WHERE foo.bid >= 3 + ) + WHERE datetime >= '2024-01-03' + """ + ) + == 3 + ) diff --git a/tests/integration/test_storage_iceberg_with_spark/test_cluster_joins.py b/tests/integration/test_storage_iceberg_with_spark/test_cluster_joins.py new file mode 100644 index 000000000000..82e9d6c3c572 --- /dev/null +++ b/tests/integration/test_storage_iceberg_with_spark/test_cluster_joins.py @@ -0,0 +1,243 @@ +import pytest + +from helpers.iceberg_utils import ( + get_uuid_str, + get_creation_expression, + execute_spark_query_general, +) + +@pytest.mark.parametrize("join_mode", ["local", "global"]) +@pytest.mark.parametrize("storage_type", ["s3", "azure"]) +def test_cluster_joins(started_cluster_iceberg_with_spark, storage_type, join_mode): + instance = started_cluster_iceberg_with_spark.instances["node1"] + spark = started_cluster_iceberg_with_spark.spark_session + TABLE_NAME = "test_cluster_joins_" + storage_type + "_" + get_uuid_str() + TABLE_NAME_2 = "test_cluster_joins_2_" + storage_type + "_" + get_uuid_str() + TABLE_NAME_LOCAL = "test_cluster_joins_local_" + storage_type + "_" + get_uuid_str() + TABLE_NAME_SOURCE = "test_cluster_joins_source_" + storage_type + "_" + get_uuid_str() + TABLE_NAME_DISTRIBUTED = "test_cluster_joins_distributed_" + storage_type + "_" + get_uuid_str() + + def execute_spark_query(query: str, table_name): + return execute_spark_query_general( + spark, + started_cluster_iceberg_with_spark, + storage_type, + table_name, + query, + ) + + execute_spark_query( + f""" + CREATE TABLE {TABLE_NAME} ( + tag INT, + name VARCHAR(50) + ) + USING iceberg + OPTIONS('format-version'='2') + """, TABLE_NAME + ) + + execute_spark_query( + f""" + INSERT INTO {TABLE_NAME} VALUES + (1, 'john'), + (2, 'jack') + """, TABLE_NAME + ) + + execute_spark_query( + f""" + CREATE TABLE {TABLE_NAME_2} ( + id INT, + second_name VARCHAR(50) + ) + USING iceberg + OPTIONS('format-version'='2') + """, TABLE_NAME_2 + ) + + execute_spark_query( + f""" + INSERT INTO {TABLE_NAME_2} VALUES + (1, 'dow'), + (2, 'sparrow') + """, TABLE_NAME_2 + ) + + creation_expression = get_creation_expression( + storage_type, TABLE_NAME, started_cluster_iceberg_with_spark, table_function=True, run_on_cluster=True + ) + + creation_expression_2 = get_creation_expression( + storage_type, TABLE_NAME_2, started_cluster_iceberg_with_spark, table_function=True, run_on_cluster=True + ) + + instance.query(f"CREATE TABLE `{TABLE_NAME_LOCAL}` (id Int64, second_name String) ENGINE = Memory()") + instance.query(f"INSERT INTO `{TABLE_NAME_LOCAL}` VALUES (1, 'silver'), (2, 'black')") + + instance.query(f"CREATE TABLE `{TABLE_NAME_SOURCE}` ON CLUSTER 'cluster_simple' (id Int64, second_name String) ENGINE = Memory()") + instance.query(f"CREATE TABLE `{TABLE_NAME_DISTRIBUTED}` (id Int64, second_name String) ENGINE = Distributed('cluster_simple', currentDatabase(), '{TABLE_NAME_SOURCE}')") + instance.query(f"INSERT INTO `{TABLE_NAME_DISTRIBUTED}` VALUES (1, 'smith'), (2, 'wesson')") + + res = instance.query( + f""" + SELECT t1.name,t2.second_name + FROM {creation_expression} AS t1 + JOIN {creation_expression_2} AS t2 + ON t1.tag=t2.id + ORDER BY ALL + SETTINGS + object_storage_cluster='cluster_simple', + object_storage_cluster_join_mode='{join_mode}' + """ + ) + + assert res == "jack\tsparrow\njohn\tdow\n" + + res = instance.query( + f""" + SELECT name + FROM {creation_expression} + WHERE tag IN ( + SELECT id + FROM {creation_expression_2} + ) + ORDER BY ALL + SETTINGS + object_storage_cluster='cluster_simple', + object_storage_cluster_join_mode='{join_mode}' + """ + ) + + assert res == "jack\njohn\n" + + res = instance.query( + f""" + SELECT name + FROM {creation_expression} + WHERE tag GLOBAL IN ( + SELECT id + FROM {creation_expression_2} + ) + ORDER BY ALL + SETTINGS + object_storage_cluster='cluster_simple', + object_storage_cluster_join_mode='{join_mode}' + """ + ) + + assert res == "jack\njohn\n" + + res = instance.query( + f""" + SELECT t1.name,t2.second_name + FROM {creation_expression} AS t1 + JOIN `{TABLE_NAME_LOCAL}` AS t2 + ON t1.tag=t2.id + WHERE t1.tag < 10 AND t2.id < 20 + ORDER BY ALL + SETTINGS + object_storage_cluster='cluster_simple', + object_storage_cluster_join_mode='{join_mode}' + """ + ) + + assert res == "jack\tblack\njohn\tsilver\n" + + res = instance.query( + f""" + SELECT name + FROM {creation_expression} + WHERE tag IN ( + SELECT id + FROM `{TABLE_NAME_LOCAL}` + ) + ORDER BY ALL + SETTINGS + object_storage_cluster='cluster_simple', + object_storage_cluster_join_mode='{join_mode}' + """ + ) + + assert res == "jack\njohn\n" + + res = instance.query( + f""" + SELECT name + FROM {creation_expression} + WHERE tag GLOBAL IN ( + SELECT id + FROM `{TABLE_NAME_LOCAL}` + ) + ORDER BY ALL + SETTINGS + object_storage_cluster='cluster_simple', + object_storage_cluster_join_mode='{join_mode}' + """ + ) + + assert res == "jack\njohn\n" + + res = instance.query( + f""" + SELECT t1.name,t2.second_name + FROM {creation_expression} AS t1 + CROSS JOIN `{TABLE_NAME_LOCAL}` AS t2 + WHERE t1.tag < 10 AND t2.id < 20 + ORDER BY ALL + SETTINGS + object_storage_cluster='cluster_simple', + object_storage_cluster_join_mode='{join_mode}' + """ + ) + + assert res == "jack\tblack\njack\tsilver\njohn\tblack\njohn\tsilver\n" + + res = instance.query( + f""" + SELECT t1.name,t2.second_name + FROM {creation_expression} AS t1 + JOIN `{TABLE_NAME_DISTRIBUTED}` AS t2 + ON t1.tag=t2.id + ORDER BY ALL + SETTINGS + object_storage_cluster='cluster_simple', + object_storage_cluster_join_mode='{join_mode}' + """ + ) + + assert res == "jack\twesson\njohn\tsmith\n" + + res = instance.query( + f""" + SELECT name + FROM {creation_expression} + WHERE tag GLOBAL IN ( + SELECT id + FROM `{TABLE_NAME_DISTRIBUTED}` + ) + ORDER BY ALL + SETTINGS + object_storage_cluster='cluster_simple', + object_storage_cluster_join_mode='{join_mode}' + """ + ) + + assert res == "jack\njohn\n" + + res = instance.query( + f""" + SELECT name + FROM {creation_expression} + WHERE tag IN ( + SELECT id + FROM `{TABLE_NAME_DISTRIBUTED}` + ) + ORDER BY ALL + SETTINGS + object_storage_cluster='cluster_simple', + object_storage_cluster_join_mode='{join_mode}' + """ + ) + + assert res == "jack\njohn\n" diff --git a/tests/integration/test_storage_iceberg_with_spark/test_cluster_table_function.py b/tests/integration/test_storage_iceberg_with_spark/test_cluster_table_function.py index 7f1701158bf8..ffbfe3a9c0dd 100644 --- a/tests/integration/test_storage_iceberg_with_spark/test_cluster_table_function.py +++ b/tests/integration/test_storage_iceberg_with_spark/test_cluster_table_function.py @@ -12,13 +12,35 @@ ) import logging +import uuid import pyarrow.parquet as pq from helpers.config_cluster import minio_secret_key +def count_secondary_subqueries(started_cluster, query_id, expected, comment): + for node_name, replica in started_cluster.instances.items(): + cluster_secondary_queries = ( + replica.query( + f""" + SELECT count(*) FROM system.query_log + WHERE + type = 'QueryFinish' + AND NOT is_initial_query + AND initial_query_id='{query_id}' + """ + ) + .strip() + ) + + logging.info( + f"[{node_name}] cluster_secondary_queries {comment}: {cluster_secondary_queries}" + ) + assert int(cluster_secondary_queries) == expected + @pytest.mark.parametrize("format_version", ["1", "2"]) @pytest.mark.parametrize("storage_type", ["s3", "azure", "local"]) -def test_cluster_table_function(started_cluster_iceberg_with_spark, format_version, storage_type): +@pytest.mark.parametrize("cluster_name_as_literal", [True, False]) +def test_cluster_table_function(started_cluster_iceberg_with_spark, format_version, storage_type, cluster_name_as_literal): instance = started_cluster_iceberg_with_spark.instances["node1"] spark = started_cluster_iceberg_with_spark.spark_session @@ -76,59 +98,177 @@ def add_df(mode): # Regular Query only node1 table_function_expr = get_creation_expression( - storage_type, TABLE_NAME, started_cluster_iceberg_with_spark, table_function=True + storage_type, TABLE_NAME, started_cluster_iceberg_with_spark, table_function=True, cluster_name_as_literal=cluster_name_as_literal ) select_regular = ( instance.query(f"SELECT * FROM {table_function_expr}").strip().split() ) + def make_query_from_function( + run_on_cluster=False, + alt_syntax=False, + remote=False, + storage_type_as_arg=False, + storage_type_in_named_collection=False, + ): + expr = get_creation_expression( + storage_type, + TABLE_NAME, + started_cluster_iceberg_with_spark, + table_function=True, + run_on_cluster=run_on_cluster, + storage_type_as_arg=storage_type_as_arg, + storage_type_in_named_collection=storage_type_in_named_collection, + cluster_name_as_literal=cluster_name_as_literal, + ) + query_id = str(uuid.uuid4()) + settings = f"SETTINGS object_storage_cluster='cluster_simple'" if (alt_syntax and not run_on_cluster) else "" + if remote: + query = f"SELECT * FROM remote('node2', {expr}) {settings}" + else: + query = f"SELECT * FROM {expr} {settings}" + responce = instance.query(query, query_id=query_id).strip().split() + return responce, query_id + # Cluster Query with node1 as coordinator - table_function_expr_cluster = get_creation_expression( - storage_type, - TABLE_NAME, - started_cluster_iceberg_with_spark, - table_function=True, + select_cluster, query_id_cluster = make_query_from_function(run_on_cluster=True) + + # Cluster Query with node1 as coordinator with alternative syntax + select_cluster_alt_syntax, query_id_cluster_alt_syntax = make_query_from_function( + run_on_cluster=True, + alt_syntax=True) + + # Cluster Query with node1 as coordinator and storage type as arg + select_cluster_with_type_arg, query_id_cluster_with_type_arg = make_query_from_function( run_on_cluster=True, + storage_type_as_arg=True, ) - select_cluster = ( - instance.query(f"SELECT * FROM {table_function_expr_cluster}").strip().split() + + # Cluster Query with node1 as coordinator and storage type in named collection + select_cluster_with_type_in_nc, query_id_cluster_with_type_in_nc = make_query_from_function( + run_on_cluster=True, + storage_type_in_named_collection=True, + ) + + # Cluster Query with node1 as coordinator and storage type as arg, alternative syntax + select_cluster_with_type_arg_alt_syntax, query_id_cluster_with_type_arg_alt_syntax = make_query_from_function( + storage_type_as_arg=True, + alt_syntax=True, + ) + + # Cluster Query with node1 as coordinator and storage type in named collection, alternative syntax + select_cluster_with_type_in_nc_alt_syntax, query_id_cluster_with_type_in_nc_alt_syntax = make_query_from_function( + storage_type_in_named_collection=True, + alt_syntax=True, ) + #select_remote_cluster, _ = make_query_from_function(run_on_cluster=True, remote=True) + + def make_query_from_table(alt_syntax=False): + query_id = str(uuid.uuid4()) + settings = "SETTINGS object_storage_cluster='cluster_simple'" if alt_syntax else "" + responce = ( + instance.query( + f"SELECT * FROM {TABLE_NAME} {settings}", + query_id=query_id, + ) + .strip() + .split() + ) + return responce, query_id + + create_iceberg_table(storage_type, instance, TABLE_NAME, started_cluster_iceberg_with_spark, object_storage_cluster='cluster_simple') + select_cluster_table_engine, query_id_cluster_table_engine = make_query_from_table() + + #select_remote_cluster = ( + # instance.query(f"SELECT * FROM remote('node2',{table_function_expr_cluster})") + # .strip() + # .split() + #) + + instance.query(f"DROP TABLE IF EXISTS `{TABLE_NAME}` SYNC") + + create_iceberg_table(storage_type, instance, TABLE_NAME, started_cluster_iceberg_with_spark) + select_pure_table_engine, query_id_pure_table_engine = make_query_from_table() + select_pure_table_engine_cluster, query_id_pure_table_engine_cluster = make_query_from_table(alt_syntax=True) + + create_iceberg_table(storage_type, instance, TABLE_NAME, started_cluster_iceberg_with_spark, storage_type_as_arg=True) + select_pure_table_engine_with_type_arg, query_id_pure_table_engine_with_type_arg = make_query_from_table() + select_pure_table_engine_cluster_with_type_arg, query_id_pure_table_engine_cluster_with_type_arg = make_query_from_table(alt_syntax=True) + + create_iceberg_table(storage_type, instance, TABLE_NAME, started_cluster_iceberg_with_spark, storage_type_in_named_collection=True) + select_pure_table_engine_with_type_in_nc, query_id_pure_table_engine_with_type_in_nc = make_query_from_table() + select_pure_table_engine_cluster_with_type_in_nc, query_id_pure_table_engine_cluster_with_type_in_nc = make_query_from_table(alt_syntax=True) + # Simple size check assert len(select_regular) == 600 assert len(select_cluster) == 600 + assert len(select_cluster_alt_syntax) == 600 + assert len(select_cluster_table_engine) == 600 + #assert len(select_remote_cluster) == 600 + assert len(select_cluster_with_type_arg) == 600 + assert len(select_cluster_with_type_in_nc) == 600 + assert len(select_cluster_with_type_arg_alt_syntax) == 600 + assert len(select_cluster_with_type_in_nc_alt_syntax) == 600 + assert len(select_pure_table_engine) == 600 + assert len(select_pure_table_engine_cluster) == 600 + assert len(select_pure_table_engine_with_type_arg) == 600 + assert len(select_pure_table_engine_cluster_with_type_arg) == 600 + assert len(select_pure_table_engine_with_type_in_nc) == 600 + assert len(select_pure_table_engine_cluster_with_type_in_nc) == 600 # Actual check assert select_cluster == select_regular + assert select_cluster_alt_syntax == select_regular + assert select_cluster_table_engine == select_regular + #assert select_remote_cluster == select_regular + assert select_cluster_with_type_arg == select_regular + assert select_cluster_with_type_in_nc == select_regular + assert select_cluster_with_type_arg_alt_syntax == select_regular + assert select_cluster_with_type_in_nc_alt_syntax == select_regular + assert select_pure_table_engine == select_regular + assert select_pure_table_engine_cluster == select_regular + assert select_pure_table_engine_with_type_arg == select_regular + assert select_pure_table_engine_cluster_with_type_arg == select_regular + assert select_pure_table_engine_with_type_in_nc == select_regular + assert select_pure_table_engine_cluster_with_type_in_nc == select_regular # Check query_log for replica in started_cluster_iceberg_with_spark.instances.values(): replica.query("SYSTEM FLUSH LOGS") - for node_name, replica in started_cluster_iceberg_with_spark.instances.items(): - cluster_secondary_queries = ( - replica.query( - f""" - SELECT query, type, is_initial_query, read_rows, read_bytes FROM system.query_log - WHERE - type = 'QueryStart' AND - positionCaseInsensitive(query, '{storage_type}Cluster') != 0 AND - position(query, '{TABLE_NAME}') != 0 AND - position(query, 'system.query_log') = 0 AND - NOT is_initial_query - """ - ) - .strip() - .split("\n") - ) + count_secondary_subqueries(started_cluster_iceberg_with_spark, query_id_cluster, 1, "table function") + count_secondary_subqueries(started_cluster_iceberg_with_spark, query_id_cluster_alt_syntax, 1, "table function alt syntax") + count_secondary_subqueries(started_cluster_iceberg_with_spark, query_id_cluster_table_engine, 1, "cluster table engine") + count_secondary_subqueries(started_cluster_iceberg_with_spark, query_id_cluster_with_type_arg, 1, "table function with storage type in args") + count_secondary_subqueries(started_cluster_iceberg_with_spark, query_id_cluster_with_type_in_nc, 1, "table function with storage type in named collection") + count_secondary_subqueries(started_cluster_iceberg_with_spark, query_id_cluster_with_type_arg_alt_syntax, 1, "table function with storage type in args alt syntax") + count_secondary_subqueries(started_cluster_iceberg_with_spark, query_id_cluster_with_type_in_nc_alt_syntax, 1, "table function with storage type in named collection alt syntax") + count_secondary_subqueries(started_cluster_iceberg_with_spark, query_id_pure_table_engine, 0, "table engine") + count_secondary_subqueries(started_cluster_iceberg_with_spark, query_id_pure_table_engine_cluster, 1, "table engine with cluster setting") + count_secondary_subqueries(started_cluster_iceberg_with_spark, query_id_pure_table_engine_with_type_arg, 0, "table engine with storage type in args") + count_secondary_subqueries(started_cluster_iceberg_with_spark, query_id_pure_table_engine_cluster_with_type_arg, 1, "table engine with cluster setting with storage type in args") + count_secondary_subqueries(started_cluster_iceberg_with_spark, query_id_pure_table_engine_with_type_in_nc, 0, "table engine with storage type in named collection") + count_secondary_subqueries(started_cluster_iceberg_with_spark, query_id_pure_table_engine_cluster_with_type_in_nc, 1, "table engine with cluster setting with storage type in named collection") - logging.info( - f"[{node_name}] cluster_secondary_queries: {cluster_secondary_queries}" - ) - assert len(cluster_secondary_queries) == 1 - # write 3 times - assert int(instance.query(f"SELECT count() FROM {table_function_expr_cluster}")) == 100 * 3 + + # Cluster Query with node1 as coordinator + table_function_expr_cluster = get_creation_expression( + storage_type, + TABLE_NAME, + started_cluster_iceberg_with_spark, + table_function=True, + run_on_cluster=True, + ) + select_remote_cluster = ( + instance.query(f"SELECT * FROM remote('node2',{table_function_expr_cluster})") + .strip() + .split() + ) + assert len(select_remote_cluster) == 600 + assert select_remote_cluster == select_regular + @pytest.mark.parametrize("format_version", ["1", "2"]) @pytest.mark.parametrize("storage_type", ["s3", "azure"]) diff --git a/tests/integration/test_storage_iceberg_with_spark/test_export_partition_iceberg.py b/tests/integration/test_storage_iceberg_with_spark/test_export_partition_iceberg.py new file mode 100644 index 000000000000..59f6ebedd979 --- /dev/null +++ b/tests/integration/test_storage_iceberg_with_spark/test_export_partition_iceberg.py @@ -0,0 +1,829 @@ +""" +Tests for EXPORT PARTITION to an Iceberg table that was created by Apache Spark. + +The destination Iceberg metadata — including field IDs and the partition spec — +is written by Spark, not by ClickHouse, which removes any bias from tests where +both source and destination are ClickHouse-created. + +A separate module-level fixture is used because the package-level +started_cluster_iceberg_with_spark does not include ZooKeeper (which is +required for ReplicatedMergeTree / EXPORT PARTITION). + +Transform coverage (ClickHouse → Iceberg): + identity → identity + toYearNumSinceEpoch → year + toMonthNumSinceEpoch → month + toRelativeDayNum → day + toRelativeHourNum → hour + icebergBucket(N) → bucket(N) + icebergTruncate(N) → truncate(N) + compound → multiple fields +""" + +import logging +import threading +import time +from concurrent.futures import ThreadPoolExecutor + +import pytest +import pyspark + +from helpers.cluster import ClickHouseCluster +from helpers.export_partition_helpers import ( + first_partition_id, + make_iceberg_s3, + make_rmt, + unique_suffix, + wait_for_export_status, +) +from helpers.iceberg_utils import ( + create_iceberg_table, + default_upload_directory, +) +from helpers.s3_tools import S3Uploader, prepare_s3_bucket + + +# --------------------------------------------------------------------------- +# Spark session +# --------------------------------------------------------------------------- + +def get_spark(): + builder = ( + pyspark.sql.SparkSession.builder + .appName("test_export_partition_spark_iceberg") + .config( + "spark.sql.catalog.spark_catalog", + "org.apache.iceberg.spark.SparkSessionCatalog", + ) + .config("spark.sql.catalog.local", "org.apache.iceberg.spark.SparkCatalog") + .config("spark.sql.catalog.spark_catalog.type", "hadoop") + .config( + "spark.sql.catalog.spark_catalog.warehouse", + "/var/lib/clickhouse/user_files/iceberg_data", + ) + .config( + "spark.sql.extensions", + "org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions", + ) + .master("local") + ) + return builder.getOrCreate() + + +# --------------------------------------------------------------------------- +# Cluster fixture +# --------------------------------------------------------------------------- + +@pytest.fixture(scope="module") +def export_cluster(): + try: + cluster = ClickHouseCluster(__file__, with_spark=True) + cluster.add_instance( + "node1", + main_configs=[ + "configs/config.d/named_collections.xml", + "configs/config.d/allow_export_partition.xml", + ], + user_configs=[ + "configs/users.d/allow_export_partition.xml", + ], + with_minio=True, + stay_alive=True, + with_zookeeper=True, + keeper_required_feature_flags=["multi_read"], + ) + for name in ["replica1", "replica2", "replica3"]: + cluster.add_instance( + name, + main_configs=[ + "configs/config.d/named_collections.xml", + "configs/config.d/allow_export_partition.xml", + ], + user_configs=[ + "configs/users.d/allow_export_partition.xml", + ], + stay_alive=True, + with_zookeeper=True, + keeper_required_feature_flags=["multi_read"], + ) + logging.info("Starting export_cluster...") + cluster.start() + prepare_s3_bucket(cluster) + cluster.spark_session = get_spark() + cluster.default_s3_uploader = S3Uploader(cluster.minio_client, cluster.minio_bucket) + yield cluster + finally: + cluster.shutdown() + + +@pytest.fixture(autouse=True) +def drop_tables(export_cluster): + yield + for node_name in ["node1", "replica1", "replica2", "replica3"]: + node = export_cluster.instances[node_name] + try: + tables = node.query( + "SELECT name FROM system.tables WHERE database = 'default' FORMAT TabSeparated" + ).strip() + for table in tables.splitlines(): + table = table.strip() + if table: + node.query(f"DROP TABLE IF EXISTS default.`{table}` SYNC") + except Exception as e: + logging.warning(f"drop_tables cleanup failed on {node_name}: {e}") + + +# --------------------------------------------------------------------------- +# Helpers +# --------------------------------------------------------------------------- + +def spark_iceberg(cluster, spark, iceberg_name: str, ddl: str): + """Execute a Spark DDL and upload the resulting Iceberg files to MinIO.""" + spark.sql(ddl) + default_upload_directory( + cluster, + "s3", + f"/iceberg_data/default/{iceberg_name}/", + f"/iceberg_data/default/{iceberg_name}/", + ) + + +def attach_ch_iceberg(node, iceberg_name: str, schema: str, cluster): + """ + Attach a ClickHouse IcebergS3 table to an existing Spark-written Iceberg path. + No PARTITION BY is specified — the spec is read from Spark's metadata. + """ + create_iceberg_table( + "s3", + node, + iceberg_name, + cluster, + schema=f"({schema})", + if_not_exists=True, + ) + + + +def run_accepted(export_cluster, label, spark_ddl, ch_schema, rmt_columns, rmt_partition_by, insert_values): + """ + Create a Spark-created Iceberg table, attach ClickHouse to it, create the + source RMT, export, wait, and return (node, source, iceberg, partition_id) + so the caller can do additional assertions. + """ + node = export_cluster.instances["node1"] + spark = export_cluster.spark_session + + uid = unique_suffix() + source = f"rmt_{label}_{uid}" + iceberg = f"spark_{label}_{uid}" + + spark_iceberg(export_cluster, spark, iceberg, spark_ddl.format(TABLE=iceberg)) + attach_ch_iceberg(node, iceberg, ch_schema, export_cluster) + make_rmt(node, source, rmt_columns, rmt_partition_by, order_by="id") + node.query(f"INSERT INTO {source} VALUES {insert_values}") + + pid = first_partition_id(node, source) + node.query( + f"ALTER TABLE {source} EXPORT PARTITION ID '{pid}' TO TABLE {iceberg}", + settings={"allow_insert_into_iceberg": 1}, + ) + wait_for_export_status(node, source, iceberg, pid) + + return node, source, iceberg, pid + + +def run_rejected(export_cluster, label, spark_ddl, ch_schema, rmt_columns, rmt_partition_by, insert_values): + """ + Create a mismatched pair and assert that EXPORT PARTITION fails with BAD_ARGUMENTS. + The check fires synchronously before any task is enqueued. + """ + node = export_cluster.instances["node1"] + spark = export_cluster.spark_session + + uid = unique_suffix() + source = f"rmt_{label}_{uid}" + iceberg = f"spark_{label}_{uid}" + + spark_iceberg(export_cluster, spark, iceberg, spark_ddl.format(TABLE=iceberg)) + attach_ch_iceberg(node, iceberg, ch_schema, export_cluster) + make_rmt(node, source, rmt_columns, rmt_partition_by, order_by="id") + node.query(f"INSERT INTO {source} VALUES {insert_values}") + + pid = first_partition_id(node, source) + error = node.query_and_get_error( + f"ALTER TABLE {source} EXPORT PARTITION ID '{pid}' TO TABLE {iceberg}", + settings={"allow_insert_into_iceberg": 1}, + ) + return error + + +# --------------------------------------------------------------------------- +# Replicated helpers +# --------------------------------------------------------------------------- + + +def create_iceberg_s3_table(node, iceberg_table: str, if_not_exists: bool = False): + """Create (or attach to an existing) IcebergS3 table at a per-test MinIO prefix.""" + make_iceberg_s3( + node, iceberg_table, "id Int64, year Int32", + partition_by="year", if_not_exists=if_not_exists, + ) + + +def setup_replicas(cluster, mt_table: str, iceberg_table: str, replica_names: list): + """ + Create RMT on each replica with a per-replica replica_name so all instances share + the same ZooKeeper path. Create IcebergS3 on the primary; attach with IF NOT EXISTS + on the rest. No data is inserted here — callers manage their own test data. + """ + instances = [cluster.instances[n] for n in replica_names] + primary = instances[0] + + for rname, instance in zip(replica_names, instances): + make_rmt(instance, mt_table, "id Int64, year Int32", "year", replica_name=rname) + + create_iceberg_s3_table(primary, iceberg_table) + for instance in instances[1:]: + create_iceberg_s3_table(instance, iceberg_table, if_not_exists=True) + + + +# --------------------------------------------------------------------------- +# Happy-path tests — one per transform +# --------------------------------------------------------------------------- + +def test_identity_transform(export_cluster): + """Spark identity(year) <-> PARTITION BY year.""" + node, _, iceberg, _ = run_accepted( + export_cluster, + "identity", + spark_ddl="CREATE TABLE {TABLE} (id BIGINT, year INT)" + " USING iceberg PARTITIONED BY (identity(year)) OPTIONS('format-version'='2')", + ch_schema="id Int64, year Int32", + rmt_columns="id Int64, year Int32", + rmt_partition_by="year", + insert_values="(1, 2024), (2, 2024), (3, 2024)", + ) + assert int(node.query(f"SELECT count() FROM {iceberg}").strip()) == 3 + + +def test_year_transform(export_cluster): + """Spark years(dt) <-> PARTITION BY toYearNumSinceEpoch(dt).""" + node, _, iceberg, _ = run_accepted( + export_cluster, + "year", + spark_ddl="CREATE TABLE {TABLE} (id BIGINT, dt DATE)" + " USING iceberg PARTITIONED BY (years(dt)) OPTIONS('format-version'='2')", + ch_schema="id Int64, dt Date", + rmt_columns="id Int64, dt Date", + rmt_partition_by="toYearNumSinceEpoch(dt)", + insert_values="(1, '2021-03-01'), (2, '2021-07-15'), (3, '2021-12-31')", + ) + assert int(node.query(f"SELECT count() FROM {iceberg}").strip()) == 3 + + +def test_month_transform(export_cluster): + """Spark months(dt) <-> PARTITION BY toMonthNumSinceEpoch(dt).""" + node, _, iceberg, _ = run_accepted( + export_cluster, + "month", + spark_ddl="CREATE TABLE {TABLE} (id BIGINT, dt DATE)" + " USING iceberg PARTITIONED BY (months(dt)) OPTIONS('format-version'='2')", + ch_schema="id Int64, dt Date", + rmt_columns="id Int64, dt Date", + rmt_partition_by="toMonthNumSinceEpoch(dt)", + insert_values="(1, '2020-06-01'), (2, '2020-06-15'), (3, '2020-06-30')", + ) + assert int(node.query(f"SELECT count() FROM {iceberg}").strip()) == 3 + + +def test_day_transform(export_cluster): + """Spark days(dt) <-> PARTITION BY toRelativeDayNum(dt).""" + node, _, iceberg, _ = run_accepted( + export_cluster, + "day", + spark_ddl="CREATE TABLE {TABLE} (id BIGINT, dt DATE)" + " USING iceberg PARTITIONED BY (days(dt)) OPTIONS('format-version'='2')", + ch_schema="id Int64, dt Date", + rmt_columns="id Int64, dt Date", + rmt_partition_by="toRelativeDayNum(dt)", + insert_values="(1, '2023-03-15'), (2, '2023-03-15'), (3, '2023-03-15')", + ) + assert int(node.query(f"SELECT count() FROM {iceberg}").strip()) == 3 + + +def test_hour_transform(export_cluster): + """Spark hours(ts) <-> PARTITION BY toRelativeHourNum(ts). + + Spark TIMESTAMP maps to Iceberg 'timestamp' which ClickHouse reads as DateTime64(6). + All three rows fall within the same hour so a single partition is exported. + """ + node, _, iceberg, _ = run_accepted( + export_cluster, + "hour", + spark_ddl="CREATE TABLE {TABLE} (id BIGINT, ts TIMESTAMP)" + " USING iceberg PARTITIONED BY (hours(ts)) OPTIONS('format-version'='2')", + ch_schema="id Int64, ts DateTime64(6)", + rmt_columns="id Int64, ts DateTime64(6)", + rmt_partition_by="toRelativeHourNum(ts)", + insert_values=( + "(1, '2023-03-15 10:00:00'), " + "(2, '2023-03-15 10:30:00'), " + "(3, '2023-03-15 10:59:00')" + ), + ) + assert int(node.query(f"SELECT count() FROM {iceberg}").strip()) == 3 + + +def test_bucket_transform(export_cluster): + """Spark bucket(8, user_id) <-> PARTITION BY icebergBucket(8, user_id).""" + node, _, iceberg, _ = run_accepted( + export_cluster, + "bucket", + spark_ddl="CREATE TABLE {TABLE} (id BIGINT, user_id BIGINT)" + " USING iceberg PARTITIONED BY (bucket(8, user_id)) OPTIONS('format-version'='2')", + ch_schema="id Int64, user_id Int64", + rmt_columns="id Int64, user_id Int64", + rmt_partition_by="icebergBucket(8, user_id)", + # All rows share the same user_id → same bucket → single partition. + insert_values="(1, 42), (2, 42), (3, 42)", + ) + assert int(node.query(f"SELECT count() FROM {iceberg}").strip()) == 3 + + +def test_truncate_transform(export_cluster): + """Spark truncate(4, category) <-> PARTITION BY icebergTruncate(4, category).""" + node, _, iceberg, _ = run_accepted( + export_cluster, + "truncate", + spark_ddl="CREATE TABLE {TABLE} (id BIGINT, category STRING)" + " USING iceberg PARTITIONED BY (truncate(4, category)) OPTIONS('format-version'='2')", + ch_schema="id Int64, category String", + rmt_columns="id Int64, category String", + rmt_partition_by="icebergTruncate(4, category)", + # All share the 4-char prefix 'clic' → same truncate bucket. + insert_values="(1, 'clickhouse'), (2, 'click'), (3, 'clickstream')", + ) + assert int(node.query(f"SELECT count() FROM {iceberg}").strip()) == 3 + + +def test_compound_transform(export_cluster): + """Spark (identity(year), identity(region)) <-> PARTITION BY (year, region).""" + node, _, iceberg, _ = run_accepted( + export_cluster, + "compound", + spark_ddl="CREATE TABLE {TABLE} (id BIGINT, year INT, region STRING)" + " USING iceberg PARTITIONED BY (identity(year), identity(region))" + " OPTIONS('format-version'='2')", + ch_schema="id Int64, year Int32, region String", + rmt_columns="id Int64, year Int32, region String", + rmt_partition_by="(year, region)", + insert_values="(1, 2022, 'EU'), (2, 2022, 'EU'), (3, 2022, 'EU')", + ) + assert int(node.query(f"SELECT count() FROM {iceberg}").strip()) == 3 + + +def test_identity_int64(export_cluster): + """Spark identity(user_id) on BIGINT <-> PARTITION BY user_id (Int64). + + Int64 → Avro 'long' is already handled by getAvroType(). This test covers + the identity transform on a 64-bit integer column, which is not covered by + the existing test_identity_transform (which uses Int32). + """ + node, _, iceberg, _ = run_accepted( + export_cluster, + "identity_int64", + spark_ddl="CREATE TABLE {TABLE} (id BIGINT, user_id BIGINT)" + " USING iceberg PARTITIONED BY (identity(user_id))" + " OPTIONS('format-version'='2')", + ch_schema="id Int64, user_id Int64", + rmt_columns="id Int64, user_id Int64", + rmt_partition_by="user_id", + insert_values="(1, 100), (2, 100), (3, 100)", + ) + assert int(node.query(f"SELECT count() FROM {iceberg}").strip()) == 3 + + +def test_identity_date(export_cluster): + """Spark identity(event_date) on DATE <-> PARTITION BY event_date (Date32). + + Date32 → Avro 'int' is already handled by getAvroType(). This test covers + the identity transform directly on a date column. Existing date-related tests + (test_year_transform, test_month_transform, etc.) use time-based transforms + such as years() and months(), not identity(). + """ + node, _, iceberg, _ = run_accepted( + export_cluster, + "identity_date", + spark_ddl="CREATE TABLE {TABLE} (id BIGINT, event_date DATE)" + " USING iceberg PARTITIONED BY (identity(event_date))" + " OPTIONS('format-version'='2')", + ch_schema="id Int64, event_date Date32", + rmt_columns="id Int64, event_date Date32", + rmt_partition_by="event_date", + insert_values="(1, '2024-03-15'), (2, '2024-03-15'), (3, '2024-03-15')", + ) + assert int(node.query(f"SELECT count() FROM {iceberg}").strip()) == 3 + + +def test_identity_string(export_cluster): + """Spark identity(region) on STRING <-> PARTITION BY region (String). + + String → Avro 'string' is already handled by getAvroType(). This test covers + identity on a string column as the sole partition field. The existing + test_compound_transform uses identity(region) only as part of a multi-field spec, + so a standalone string identity partition was not previously exercised end-to-end. + """ + node, _, iceberg, _ = run_accepted( + export_cluster, + "identity_str", + spark_ddl="CREATE TABLE {TABLE} (id BIGINT, region STRING)" + " USING iceberg PARTITIONED BY (identity(region))" + " OPTIONS('format-version'='2')", + ch_schema="id Int64, region String", + rmt_columns="id Int64, region String", + rmt_partition_by="region", + insert_values="(1, 'EU'), (2, 'EU'), (3, 'EU')", + ) + assert int(node.query(f"SELECT count() FROM {iceberg}").strip()) == 3 + + +def test_truncate_int64(export_cluster): + """Spark truncate(10, amount) on BIGINT <-> PARTITION BY icebergTruncate(10, amount) (Int64). + + Int64 truncate produces floor(v / 10) * 10, so all rows with amount=42 land in + partition value 40 (same partition). This is a distinct code path from truncate on + String (which trims a character prefix). The existing test_truncate_transform uses + String only, leaving the integer truncate path untested. + """ + node, _, iceberg, _ = run_accepted( + export_cluster, + "truncate_int64", + spark_ddl="CREATE TABLE {TABLE} (id BIGINT, amount BIGINT)" + " USING iceberg PARTITIONED BY (truncate(10, amount))" + " OPTIONS('format-version'='2')", + ch_schema="id Int64, amount Int64", + rmt_columns="id Int64, amount Int64", + rmt_partition_by="icebergTruncate(10, amount)", + # All rows have amount=42 → truncated partition value is 40. + insert_values="(1, 42), (2, 42), (3, 42)", + ) + assert int(node.query(f"SELECT count() FROM {iceberg}").strip()) == 3 + + +def test_bucket_string(export_cluster): + """Spark bucket(8, name) on STRING <-> PARTITION BY icebergBucket(8, name) (String). + + Bucket on strings uses Murmur3 hash of the UTF-8 bytes, a different hash path than + bucket on integers. All rows share the same name so they land in the same bucket. + The existing test_bucket_transform uses BIGINT only, leaving string bucketing untested. + """ + node, _, iceberg, _ = run_accepted( + export_cluster, + "bucket_str", + spark_ddl="CREATE TABLE {TABLE} (id BIGINT, name STRING)" + " USING iceberg PARTITIONED BY (bucket(8, name))" + " OPTIONS('format-version'='2')", + ch_schema="id Int64, name String", + rmt_columns="id Int64, name String", + rmt_partition_by="icebergBucket(8, name)", + # All rows share the same name → same Murmur3 bucket. + insert_values="(1, 'alice'), (2, 'alice'), (3, 'alice')", + ) + assert int(node.query(f"SELECT count() FROM {iceberg}").strip()) == 3 + + +def test_year_transform_timestamp(export_cluster): + """Spark years(ts) on TIMESTAMP <-> PARTITION BY toYearNumSinceEpoch(ts) (DateTime64(6)). + + DateTime64 → Avro 'long' is already handled by getAvroType(). The year transform on + TIMESTAMP follows a different branch than on DATE (long vs int in Avro). The existing + test_year_transform uses DATE only. All three rows fall within the same year. + """ + node, _, iceberg, _ = run_accepted( + export_cluster, + "year_ts", + spark_ddl="CREATE TABLE {TABLE} (id BIGINT, ts TIMESTAMP)" + " USING iceberg PARTITIONED BY (years(ts))" + " OPTIONS('format-version'='2')", + ch_schema="id Int64, ts DateTime64(6)", + rmt_columns="id Int64, ts DateTime64(6)", + rmt_partition_by="toYearNumSinceEpoch(ts)", + insert_values=( + "(1, '2023-01-15 08:00:00'), " + "(2, '2023-06-01 12:00:00'), " + "(3, '2023-12-31 23:59:59')" + ), + ) + assert int(node.query(f"SELECT count() FROM {iceberg}").strip()) == 3 + + +def test_month_transform_timestamp(export_cluster): + """Spark months(ts) on TIMESTAMP <-> PARTITION BY toMonthNumSinceEpoch(ts) (DateTime64(6)). + + Analogous to test_year_transform_timestamp but for the month transform. + All three rows fall within the same calendar month. + """ + node, _, iceberg, _ = run_accepted( + export_cluster, + "month_ts", + spark_ddl="CREATE TABLE {TABLE} (id BIGINT, ts TIMESTAMP)" + " USING iceberg PARTITIONED BY (months(ts))" + " OPTIONS('format-version'='2')", + ch_schema="id Int64, ts DateTime64(6)", + rmt_columns="id Int64, ts DateTime64(6)", + rmt_partition_by="toMonthNumSinceEpoch(ts)", + insert_values=( + "(1, '2023-06-01 00:00:00'), " + "(2, '2023-06-15 12:00:00'), " + "(3, '2023-06-30 23:59:59')" + ), + ) + assert int(node.query(f"SELECT count() FROM {iceberg}").strip()) == 3 + + +def test_day_transform_timestamp(export_cluster): + """Spark days(ts) on TIMESTAMP <-> PARTITION BY toRelativeDayNum(ts) (DateTime64(6)). + + Analogous to test_year_transform_timestamp but for the day transform. + All three rows fall within the same calendar day. + """ + node, _, iceberg, _ = run_accepted( + export_cluster, + "day_ts", + spark_ddl="CREATE TABLE {TABLE} (id BIGINT, ts TIMESTAMP)" + " USING iceberg PARTITIONED BY (days(ts))" + " OPTIONS('format-version'='2')", + ch_schema="id Int64, ts DateTime64(6)", + rmt_columns="id Int64, ts DateTime64(6)", + rmt_partition_by="toRelativeDayNum(ts)", + insert_values=( + "(1, '2023-06-15 00:00:00'), " + "(2, '2023-06-15 12:00:00'), " + "(3, '2023-06-15 23:59:59')" + ), + ) + assert int(node.query(f"SELECT count() FROM {iceberg}").strip()) == 3 + + +# --------------------------------------------------------------------------- +# Unhappy-path tests — BAD_ARGUMENTS must be raised synchronously +# --------------------------------------------------------------------------- + +def test_rejected_column_mismatch(export_cluster): + """Spark identity(year) — RMT PARTITION BY id: different column.""" + error = run_rejected( + export_cluster, + "rej_col_mismatch", + spark_ddl="CREATE TABLE {TABLE} (id BIGINT, year INT)" + " USING iceberg PARTITIONED BY (identity(year)) OPTIONS('format-version'='2')", + ch_schema="id Int64, year Int32", + rmt_columns="id Int64, year Int32", + rmt_partition_by="id", + insert_values="(1, 2024)", + ) + assert "BAD_ARGUMENTS" in error, f"Expected BAD_ARGUMENTS, got: {error!r}" + + +def test_rejected_transform_mismatch(export_cluster): + """Spark days(dt) destination — RMT PARTITION BY toStartOfMonth(dt): a month partition spans + several days, so it cannot map to a single Iceberg day partition.""" + error = run_rejected( + export_cluster, + "rej_xform_mismatch", + spark_ddl="CREATE TABLE {TABLE} (id BIGINT, dt DATE)" + " USING iceberg PARTITIONED BY (days(dt)) OPTIONS('format-version'='2')", + ch_schema="id Int64, dt Date", + rmt_columns="id Int64, dt Date", + rmt_partition_by="toStartOfMonth(dt)", + insert_values="(1, '2021-06-01'), (2, '2021-06-15')", + ) + assert "BAD_ARGUMENTS" in error, f"Expected BAD_ARGUMENTS, got: {error!r}" + + +def test_rejected_bucket_count_mismatch(export_cluster): + """Spark bucket(8, user_id) — RMT icebergBucket(16, user_id): wrong N.""" + error = run_rejected( + export_cluster, + "rej_bucket_n", + spark_ddl="CREATE TABLE {TABLE} (id BIGINT, user_id BIGINT)" + " USING iceberg PARTITIONED BY (bucket(8, user_id)) OPTIONS('format-version'='2')", + ch_schema="id Int64, user_id Int64", + rmt_columns="id Int64, user_id Int64", + rmt_partition_by="icebergBucket(16, user_id)", + insert_values="(1, 42)", + ) + assert "BAD_ARGUMENTS" in error, f"Expected BAD_ARGUMENTS, got: {error!r}" + + +def test_rejected_truncate_width_mismatch(export_cluster): + """Spark truncate(8, category) destination — RMT icebergTruncate(4, category): the coarser + width-4 source partition splits across several width-8 destination buckets.""" + error = run_rejected( + export_cluster, + "rej_trunc_w", + spark_ddl="CREATE TABLE {TABLE} (id BIGINT, category STRING)" + " USING iceberg PARTITIONED BY (truncate(8, category)) OPTIONS('format-version'='2')", + ch_schema="id Int64, category String", + rmt_columns="id Int64, category String", + rmt_partition_by="icebergTruncate(4, category)", + insert_values="(1, 'clickhouse'), (2, 'clickfast')", + ) + assert "BAD_ARGUMENTS" in error, f"Expected BAD_ARGUMENTS, got: {error!r}" + + +def test_idempotency_after_commit_crash(export_cluster): + """ + Verify that an Iceberg export commit is idempotent when ClickHouse crashes (via + std::terminate() in a failpoint) after the Iceberg metadata is written but before + ZooKeeper is updated to COMPLETED. + Expected behaviour: + - The failpoint fires once: std::terminate() kills the process immediately after the + Iceberg commit; ZK task remains PENDING. + - ClickHouse is restarted. The scheduler picks up the PENDING task and retries the + commit. commitExportPartitionTransaction finds the transaction_id already present in + the Iceberg snapshot summary and skips re-committing. + - The task eventually reaches COMPLETED. + - The row count in the Iceberg table is exactly the number inserted (no duplicates). + """ + node = export_cluster.instances["node1"] + spark = export_cluster.spark_session + uid = unique_suffix() + source = f"rmt_{uid}" + iceberg = f"spark_{uid}" + spark_iceberg( + export_cluster, + spark, + iceberg, + f"CREATE TABLE {iceberg} (id BIGINT, year INT)" + f" USING iceberg PARTITIONED BY (identity(year)) OPTIONS('format-version'='2')", + ) + attach_ch_iceberg(node, iceberg, "id Int64, year Int32", export_cluster) + make_rmt(node, source, "id Int64, year Int32", "year") + node.query(f"INSERT INTO {source} VALUES (1, 2024), (2, 2024), (3, 2024)") + pid = first_partition_id(node, source) + # Enable the ONCE failpoint. When the background scheduler thread reaches the + # injection point (after a successful Iceberg commit), std::terminate() is called + # and the process exits immediately without setting ZK COMPLETED. + node.query("SYSTEM ENABLE FAILPOINT iceberg_export_after_commit_before_zk_completed") + node.query( + f"ALTER TABLE {source} EXPORT PARTITION ID '{pid}' TO TABLE {iceberg}", + settings={"allow_insert_into_iceberg": 1}, + ) + # the fail point will sleep for 10 seconds. Wait for 5 and then re-start clickhouse. + time.sleep(5) + # Restart ClickHouse. The ZK task is still PENDING; the scheduler will pick it up. + node.restart_clickhouse() + time.sleep(5) + # On restart the scheduler retries the commit. commitExportPartitionTransaction + # detects the transaction_id in the existing Iceberg snapshot summary and returns + # without re-writing any data, then sets ZK COMPLETED. + wait_for_export_status(node, source, iceberg, pid, timeout=60) + # Exactly 3 rows — no duplicates from the idempotent re-commit. + count = int(node.query(f"SELECT count() FROM {iceberg}").strip()) + assert count == 3, f"Expected 3 rows (no duplicates), got {count}" + + # The already-committed early-exit in commitExportPartitionTransaction surfaces + # a sentinel note in committed_metadata_file (the original committer's paths + # are not recoverable from inside the call). The sentinel makes the situation + # visible in system.replicated_partition_exports rather than leaving the + # commit_info columns empty. + committed_metadata_file = node.query( + f"SELECT committed_metadata_file FROM system.replicated_partition_exports " + f"WHERE source_table = '{source}' AND partition_id = '{pid}'" + ).strip() + assert committed_metadata_file == "", ( + f"Expected already-committed sentinel after idempotent retry, got: {committed_metadata_file!r}" + ) + + +# --------------------------------------------------------------------------- +# Replicated tests — IcebergS3, no catalog +# --------------------------------------------------------------------------- + + +def test_export_initiated_from_replica2(export_cluster): + """ + Export is initiated from replica2 (not the inserting replica). + Validates that any replica can start the export, not just the writer. + """ + uid = unique_suffix() + mt_table = f"rmt_from_replica2_{uid}" + iceberg_table = f"iceberg_from_replica2_{uid}" + + setup_replicas(export_cluster, mt_table, iceberg_table, ["replica1", "replica2"]) + + r1 = export_cluster.instances["replica1"] + r2 = export_cluster.instances["replica2"] + + r1.query(f"INSERT INTO {mt_table} VALUES (1, 2020), (2, 2020), (3, 2020)") + r2.query(f"SYSTEM SYNC REPLICA {mt_table}") + + r2.query( + f"ALTER TABLE {mt_table} EXPORT PARTITION ID '2020' TO TABLE {iceberg_table}", + settings={"allow_insert_into_iceberg": 1}, + ) + wait_for_export_status(r2, mt_table, iceberg_table, "2020") + + count_r1 = int(r1.query(f"SELECT count() FROM {iceberg_table}").strip()) + assert count_r1 == 3, f"Expected 3 rows from replica1, got {count_r1}" + count_r2 = int(r2.query(f"SELECT count() FROM {iceberg_table}").strip()) + assert count_r2 == 3, f"Expected 3 rows from replica2, got {count_r2}" + + +def test_concurrent_exports_different_partitions_across_replicas(export_cluster): + """ + Three replicas concurrently export distinct partitions (2020, 2021, 2022) to the + same IcebergS3 table. All three commits must succeed and the total row count must + equal the sum of all inserted rows. + """ + uid = unique_suffix() + mt_table = f"rmt_concurrent_diff_parts_{uid}" + iceberg_table = f"iceberg_concurrent_diff_parts_{uid}" + + setup_replicas( + export_cluster, mt_table, iceberg_table, + ["replica1", "replica2", "replica3"], + ) + + r1 = export_cluster.instances["replica1"] + r2 = export_cluster.instances["replica2"] + r3 = export_cluster.instances["replica3"] + + r1.query(f"INSERT INTO {mt_table} VALUES (1, 2020), (2, 2020), (3, 2020)") + r1.query(f"INSERT INTO {mt_table} VALUES (4, 2021), (5, 2021), (6, 2021)") + r1.query(f"INSERT INTO {mt_table} VALUES (7, 2022), (8, 2022), (9, 2022)") + r2.query(f"SYSTEM SYNC REPLICA {mt_table}") + r3.query(f"SYSTEM SYNC REPLICA {mt_table}") + + errors: list = [] + + def export_from(node, pid): + try: + node.query( + f"ALTER TABLE {mt_table} EXPORT PARTITION ID '{pid}' TO TABLE {iceberg_table}", + settings={"allow_insert_into_iceberg": 1}, + ) + wait_for_export_status(node, mt_table, iceberg_table, pid) + except Exception as exc: + errors.append(exc) + + threads = [ + threading.Thread(target=export_from, args=(r1, "2020")), + threading.Thread(target=export_from, args=(r2, "2021")), + threading.Thread(target=export_from, args=(r3, "2022")), + ] + for t in threads: + t.start() + for t in threads: + t.join() + + assert not errors, f"Export threads raised errors: {errors}" + + count = int(r1.query(f"SELECT count() FROM {iceberg_table}").strip()) + assert count == 9, f"Expected 9 rows total (3 per partition), got {count}" + + +def test_three_replica_concurrent_exports(export_cluster): + """ + ThreadPoolExecutor with 3 workers: each replica exports its own distinct partition. + All futures must complete successfully; total row count must be correct. + """ + uid = unique_suffix() + mt_table = f"rmt_three_replicas_concurrent_{uid}" + iceberg_table = f"iceberg_three_replicas_concurrent_{uid}" + + setup_replicas( + export_cluster, mt_table, iceberg_table, + ["replica1", "replica2", "replica3"], + ) + + r1 = export_cluster.instances["replica1"] + r2 = export_cluster.instances["replica2"] + r3 = export_cluster.instances["replica3"] + + r1.query(f"INSERT INTO {mt_table} VALUES (1, 2020), (2, 2020), (3, 2020)") + r1.query(f"INSERT INTO {mt_table} VALUES (4, 2021), (5, 2021), (6, 2021)") + r1.query(f"INSERT INTO {mt_table} VALUES (7, 2022), (8, 2022), (9, 2022)") + r2.query(f"SYSTEM SYNC REPLICA {mt_table}") + r3.query(f"SYSTEM SYNC REPLICA {mt_table}") + + def export_fn(node_pid): + node, pid = node_pid + node.query( + f"ALTER TABLE {mt_table} EXPORT PARTITION ID '{pid}' TO TABLE {iceberg_table}", + settings={"allow_insert_into_iceberg": 1}, + ) + wait_for_export_status(node, mt_table, iceberg_table, pid) + + with ThreadPoolExecutor(max_workers=3) as executor: + futures = [ + executor.submit(export_fn, (r1, "2020")), + executor.submit(export_fn, (r2, "2021")), + executor.submit(export_fn, (r3, "2022")), + ] + for fut in futures: + fut.result() + + count = int(r1.query(f"SELECT count() FROM {iceberg_table}").strip()) + assert count == 9, f"Expected 9 rows total (3 per partition), got {count}" diff --git a/tests/integration/test_storage_iceberg_with_spark/test_export_partition_iceberg_catalog.py b/tests/integration/test_storage_iceberg_with_spark/test_export_partition_iceberg_catalog.py new file mode 100644 index 000000000000..f6f927ec7e30 --- /dev/null +++ b/tests/integration/test_storage_iceberg_with_spark/test_export_partition_iceberg_catalog.py @@ -0,0 +1,552 @@ +""" +Tests for EXPORT PARTITION to a catalog-backed Iceberg table (Glue catalog via Moto). + +These tests verify that the catalog commit path (catalog->updateMetadata) is +exercised correctly for EXPORT PARTITION. A dedicated module-level cluster fixture +combines ZooKeeper (for ReplicatedMergeTree) with the Glue docker-compose stack +(Moto mock + MinIO warehouse bucket). + +Test coverage: + test_catalog_basic_export — single partition exported; catalog shows new snapshot + test_catalog_concurrent_export — two partitions exported in parallel; both commits succeed + test_catalog_idempotent_retry — crash after catalog commit; restart; no data duplication +""" + +import logging +import os +import threading +import time +import uuid + +import pytest +from pyiceberg.catalog import load_catalog +from pyiceberg.partitioning import PartitionField, PartitionSpec +from pyiceberg.schema import Schema +from pyiceberg.transforms import IdentityTransform +from pyiceberg.types import LongType, NestedField, StringType + +from helpers.cluster import ClickHouseCluster +from helpers.config_cluster import minio_access_key, minio_secret_key +from helpers.export_partition_helpers import ( + make_rmt, + wait_for_export_status, +) + + +GLUE_BASE_URL = "http://glue:3000" +CH_CATALOG_DB = "glue_export_catalog" +# The Glue (Moto) container port is mapped to a dynamically allocated host port, +# see `ClickHouseCluster.glue_catalog_port`, so the host-side URL is per-cluster. +GLUE_WAREHOUSE_ENDPOINT = "http://minio1:9001/warehouse-glue" + + +def get_glue_local_url(cluster): + return f"http://localhost:{cluster.glue_catalog_port}" + + +# --------------------------------------------------------------------------- +# Cluster fixture +# --------------------------------------------------------------------------- + + +@pytest.fixture(scope="module") +def catalog_export_cluster(): + """ + Cluster with ZooKeeper (for ReplicatedMergeTree / EXPORT PARTITION) and the + Glue docker-compose stack (Moto mock + MinIO warehouse bucket). + Spark is not needed; pyiceberg handles table creation and catalog inspection. + replica1 and replica2 are additional nodes for replicated-export tests; they + share the same ZooKeeper, Glue, and MinIO containers as node1. + """ + try: + os.environ["AWS_ACCESS_KEY_ID"] = "testing" + os.environ["AWS_SECRET_ACCESS_KEY"] = "testing" + cluster = ClickHouseCluster(__file__) + for name in ["node1", "replica1", "replica2"]: + cluster.add_instance( + name, + main_configs=[ + "configs/config.d/allow_export_partition.xml", + ], + user_configs=[ + "configs/users.d/allow_export_partition.xml", + ], + stay_alive=True, + with_zookeeper=True, + keeper_required_feature_flags=["multi_read"], + with_glue_catalog=True, + ) + cluster.start() + + time.sleep(15) + + yield cluster + finally: + cluster.shutdown() + + +@pytest.fixture(autouse=True) +def cleanup_tables(catalog_export_cluster): + """Drop all default-DB tables on every node after each test.""" + yield + for node_name in ["node1", "replica1", "replica2"]: + node = catalog_export_cluster.instances[node_name] + try: + tables = node.query( + "SELECT name FROM system.tables WHERE database = 'default' FORMAT TabSeparated" + ).strip() + for tbl in tables.splitlines(): + tbl = tbl.strip() + if tbl: + node.query(f"DROP TABLE IF EXISTS default.`{tbl}` SYNC") + except Exception as exc: + logging.warning("cleanup_tables on %s: %s", node_name, exc) + + +# --------------------------------------------------------------------------- +# Helpers +# --------------------------------------------------------------------------- + + +def connect_catalog(cluster): + """ + Connect to the Moto Glue mock from the test host via its mapped host port. + MinIO is accessed via the container IP for S3 operations. Catalogs share the + standard MinIO container (`minio1`), exposed as `cluster.minio_ip`/`minio_port`. + """ + return load_catalog( + "glue_test", + **{ + "type": "glue", + "glue.endpoint": get_glue_local_url(cluster), + "glue.region": "us-east-1", + "s3.endpoint": f"http://{cluster.minio_ip}:{cluster.minio_port}", + "s3.access-key-id": minio_access_key, + "s3.secret-access-key": minio_secret_key, + }, + ) + + +def setup_ch_catalog_db(node, db_name: str = CH_CATALOG_DB) -> None: + """Drop-and-recreate the ClickHouse DataLakeCatalog database pointing at Glue (Moto).""" + node.query(f"DROP DATABASE IF EXISTS {db_name}") + # The Glue catalog API client is subject to the server-managed credential restriction: + # unless the session opts in via `s3_allow_server_credentials_in_user_queries`, the creator + # must pass explicit catalog credentials instead of falling back to the server's AWS + # identity (moto accepts any non-empty values). Same convention as test_database_glue. + node.query( + f""" + SET write_full_path_in_iceberg_metadata = 1; + SET allow_database_glue_catalog = 1; + CREATE DATABASE {db_name} + ENGINE = DataLakeCatalog('{GLUE_BASE_URL}', '{minio_access_key}', '{minio_secret_key}') + SETTINGS catalog_type = 'glue', + warehouse = 'test', + storage_endpoint = '{GLUE_WAREHOUSE_ENDPOINT}', + region = 'us-east-1', + aws_access_key_id = '{minio_access_key}', + aws_secret_access_key = '{minio_secret_key}' + """ + ) + + +def create_catalog_rmt(node, name: str, replica_name: str = "r1") -> None: + """Create an identity(region)-partitioned ReplicatedMergeTree source table.""" + make_rmt(node, name, "id Int64, region String", "region", + replica_name=replica_name, order_by="id") + + +def partition_id_for(node, table: str, region: str) -> str: + return node.query( + f"SELECT DISTINCT partition_id FROM system.parts" + f" WHERE table = '{table}' AND active AND partition = '{region}'" + f" FORMAT TabSeparated" + ).strip() + + +def create_catalog_iceberg_table(catalog, ns: str, tbl: str) -> None: + """ + Create a simple identity(region)-partitioned Iceberg table in the catalog. + Using format-version 2 and uncompressed metadata for test simplicity. + """ + catalog.create_table( + identifier=f"{ns}.{tbl}", + schema=Schema( + NestedField(field_id=1, name="id", field_type=LongType(), required=True), + NestedField(field_id=2, name="region", field_type=StringType(), required=True), + ), + location=f"s3://warehouse-glue/data/{tbl}", + partition_spec=PartitionSpec( + PartitionField( + source_id=2, + field_id=1000, + transform=IdentityTransform(), + name="region", + ) + ), + properties={ + "write.metadata.compression-codec": "none", + "write.format.default": "parquet", + "format-version": "2", + }, + ) + + +# --------------------------------------------------------------------------- +# Replicated catalog helpers +# --------------------------------------------------------------------------- + + +def setup_catalog_replicas(cluster, source_table: str, replica_names: list) -> None: + """ + Create RMT on each named replica (each with its own replica_name so they share + the same ZK path) and set up the DataLakeCatalog database on every node. + No data is inserted here — callers manage their own test data. + """ + for rname in replica_names: + create_catalog_rmt(cluster.instances[rname], source_table, replica_name=rname) + setup_ch_catalog_db(cluster.instances[rname]) + + +# --------------------------------------------------------------------------- +# Tests +# --------------------------------------------------------------------------- + + +def test_catalog_basic_export(catalog_export_cluster): + """ + Create a catalog-registered Iceberg table via pyiceberg, export one partition + from a ReplicatedMergeTree, and verify: + - The catalog (Glue) shows a new snapshot after the export. + - SELECT via the DataLakeCatalog database returns the correct row count. + + This test exercises the catalog commit path: + IcebergMetadata::commitImportPartitionTransactionImpl + → catalog->updateMetadata(namespace, table, new_metadata_file, snapshot) + """ + node = catalog_export_cluster.instances["node1"] + catalog = connect_catalog(catalog_export_cluster) + + ns = f"ns_basic_{uuid.uuid4().hex[:8]}" + tbl = f"tbl_basic_{uuid.uuid4().hex[:8]}" + source = f"rmt_basic_{uuid.uuid4().hex[:8]}" + + catalog.create_namespace((ns,)) + create_catalog_iceberg_table(catalog, ns, tbl) + setup_ch_catalog_db(node) + create_catalog_rmt(node, source) + + node.query(f"INSERT INTO {source} VALUES (1, 'EU'), (2, 'EU'), (3, 'EU')") + + pid = partition_id_for(node, source, "EU") + dest_ch = f"`{CH_CATALOG_DB}`.`{ns}.{tbl}`" + + node.query( + f"ALTER TABLE {source} EXPORT PARTITION ID '{pid}' TO TABLE {dest_ch}", + settings={"write_full_path_in_iceberg_metadata": 1, "allow_insert_into_iceberg": 1}, + ) + wait_for_export_status(node, source, None, pid) + + count = int(node.query(f"SELECT count() FROM {dest_ch}").strip()) + assert count == 3, f"Expected 3 rows, got {count}" + + iceberg_tbl = catalog.load_table(f"{ns}.{tbl}") + assert iceberg_tbl.current_snapshot() is not None, \ + "Expected at least one snapshot in Glue after the export" + + +def test_catalog_concurrent_export(catalog_export_cluster): + """ + Export two partitions concurrently to the same catalog-backed Iceberg table. + + Both commits go through catalog->updateMetadata (Glue). Both commits must + ultimately succeed. + + Verifies: + - Total row count equals total inserted (no rows lost). + - The catalog history contains at least two snapshots (one per partition). + """ + node = catalog_export_cluster.instances["node1"] + catalog = connect_catalog(catalog_export_cluster) + + ns = f"ns_concurrent_{uuid.uuid4().hex[:8]}" + tbl = f"tbl_concurrent_{uuid.uuid4().hex[:8]}" + source = f"rmt_concurrent_{uuid.uuid4().hex[:8]}" + + catalog.create_namespace((ns,)) + create_catalog_iceberg_table(catalog, ns, tbl) + setup_ch_catalog_db(node) + create_catalog_rmt(node, source) + + node.query(f"INSERT INTO {source} VALUES (1, 'EU'), (2, 'EU'), (3, 'EU')") + node.query(f"INSERT INTO {source} VALUES (4, 'US'), (5, 'US'), (6, 'US')") + + pid_eu = partition_id_for(node, source, "EU") + pid_us = partition_id_for(node, source, "US") + dest_ch = f"`{CH_CATALOG_DB}`.`{ns}.{tbl}`" + + errors: list = [] + + def export_partition(pid: str) -> None: + try: + node.query( + f"ALTER TABLE {source} EXPORT PARTITION ID '{pid}' TO TABLE {dest_ch}", + settings={"write_full_path_in_iceberg_metadata": 1, "allow_insert_into_iceberg": 1}, + ) + wait_for_export_status(node, source, None, pid, timeout=120) + except Exception as exc: + errors.append(exc) + + t1 = threading.Thread(target=export_partition, args=(pid_eu,)) + t2 = threading.Thread(target=export_partition, args=(pid_us,)) + t1.start() + t2.start() + t1.join() + t2.join() + + assert not errors, f"Export threads raised errors: {errors}" + + count = int(node.query(f"SELECT count() FROM {dest_ch}").strip()) + assert count == 6, f"Expected 6 rows (3 EU + 3 US), got {count}" + + iceberg_tbl = catalog.load_table(f"{ns}.{tbl}") + history = iceberg_tbl.history() + assert len(history) >= 2, ( + f"Expected ≥2 snapshots (one per concurrent partition commit), got {len(history)}" + ) + + +def test_catalog_idempotent_retry(catalog_export_cluster): + """ + Simulate a crash after the catalog commit but before ZooKeeper is updated to + COMPLETED (via the iceberg_export_after_commit_before_zk_completed failpoint). + + After restart the scheduler retries the PENDING task. + IcebergMetadata::commitExportPartitionTransaction finds the transaction_id already + embedded in a snapshot summary field (clickhouse.export-partition-transaction-id) + and returns without re-committing. + + Verifies: + - Exactly 3 rows in the Iceberg table (no duplicates from the re-commit). + - Exactly 1 snapshot in the Glue catalog (the idempotent retry was a no-op). + """ + node = catalog_export_cluster.instances["node1"] + catalog = connect_catalog(catalog_export_cluster) + + ns = f"ns_idempotent_{uuid.uuid4().hex[:8]}" + tbl = f"tbl_idempotent_{uuid.uuid4().hex[:8]}" + source = f"rmt_idempotent_{uuid.uuid4().hex[:8]}" + + catalog.create_namespace((ns,)) + create_catalog_iceberg_table(catalog, ns, tbl) + setup_ch_catalog_db(node) + create_catalog_rmt(node, source) + + node.query(f"INSERT INTO {source} VALUES (1, 'EU'), (2, 'EU'), (3, 'EU')") + + pid = partition_id_for(node, source, "EU") + dest_ch = f"`{CH_CATALOG_DB}`.`{ns}.{tbl}`" + + # Enable the ONCE failpoint: after a successful catalog commit the process + # calls std::terminate() before writing ZK COMPLETED — simulating a hard crash. + node.query("SYSTEM ENABLE FAILPOINT iceberg_export_after_commit_before_zk_completed") + node.query( + f"ALTER TABLE {source} EXPORT PARTITION ID '{pid}' TO TABLE {dest_ch}", + settings={"write_full_path_in_iceberg_metadata": 1, "allow_insert_into_iceberg": 1}, + ) + + # Give the background scheduler time to export the data files and reach the + # failpoint. The crash is immediate (std::terminate), so 10 s is generous. + time.sleep(10) + node.restart_clickhouse() + + # ClickHouse persists database metadata to disk so the DataLakeCatalog database + # survives the crash. Recreate it anyway to make the test self-contained. + setup_ch_catalog_db(node) + + # The scheduler picks up the PENDING task and retries. commitExportPartitionTransaction + # detects the transaction_id in the existing snapshot summary and skips the + # re-commit, then marks the task COMPLETED in ZooKeeper. + wait_for_export_status(node, source, None, pid, timeout=120) + + count = int(node.query(f"SELECT count() FROM {dest_ch}").strip()) + assert count == 3, f"Expected 3 rows (no duplicates from idempotent retry), got {count}" + + iceberg_tbl = catalog.load_table(f"{ns}.{tbl}") + history = iceberg_tbl.history() + assert len(history) == 1, ( + f"Expected exactly 1 snapshot (idempotent re-commit was a no-op), " + f"got {len(history)}" + ) + + committed_metadata_file = node.query( + f"SELECT committed_metadata_file FROM system.replicated_partition_exports " + f"WHERE source_table = '{source}' AND partition_id = '{pid}'" + ).strip() + assert committed_metadata_file == "", ( + f"Expected already-committed sentinel after idempotent retry, got: {committed_metadata_file!r}" + ) + + +# --------------------------------------------------------------------------- +# Replicated catalog tests +# --------------------------------------------------------------------------- + + +def test_catalog_export_two_replicas_basic(catalog_export_cluster): + """ + End-to-end: export one partition from replica1 in a 2-replica setup. + Export is initiated on replica1; row count is verified from replica2 via + the DataLakeCatalog database to confirm the catalog commit was visible. + """ + catalog = connect_catalog(catalog_export_cluster) + + ns = f"ns_two_replicas_{uuid.uuid4().hex[:8]}" + tbl = f"tbl_two_replicas_{uuid.uuid4().hex[:8]}" + source = f"rmt_two_replicas_{uuid.uuid4().hex[:8]}" + + catalog.create_namespace((ns,)) + create_catalog_iceberg_table(catalog, ns, tbl) + + setup_catalog_replicas(catalog_export_cluster, source, ["replica1", "replica2"]) + + r1 = catalog_export_cluster.instances["replica1"] + r2 = catalog_export_cluster.instances["replica2"] + + r1.query(f"INSERT INTO {source} VALUES (1, 'EU'), (2, 'EU'), (3, 'EU')") + r2.query(f"SYSTEM SYNC REPLICA {source}") + + pid = partition_id_for(r1, source, "EU") + dest_ch = f"`{CH_CATALOG_DB}`.`{ns}.{tbl}`" + + r1.query( + f"ALTER TABLE {source} EXPORT PARTITION ID '{pid}' TO TABLE {dest_ch}", + settings={"write_full_path_in_iceberg_metadata": 1, "allow_insert_into_iceberg": 1}, + ) + wait_for_export_status(r1, source, None, pid) + + iceberg_tbl = catalog.load_table(f"{ns}.{tbl}") + assert iceberg_tbl.current_snapshot() is not None, \ + "Expected at least one snapshot in Glue after export" + + count = int(r2.query(f"SELECT count() FROM {dest_ch}").strip()) + assert count == 3, f"Expected 3 rows from replica2 via catalog, got {count}" + + +def test_catalog_concurrent_export_from_different_replicas(catalog_export_cluster): + """ + Two replicas concurrently export different partitions (EU / US) to the same + catalog-backed Iceberg table. Both catalog commits must succeed; total row count + must equal 6 and Glue history must contain at least 2 snapshots. + """ + catalog = connect_catalog(catalog_export_cluster) + + ns = f"ns_conc_replicas_{uuid.uuid4().hex[:8]}" + tbl = f"tbl_conc_replicas_{uuid.uuid4().hex[:8]}" + source = f"rmt_conc_replicas_{uuid.uuid4().hex[:8]}" + + catalog.create_namespace((ns,)) + create_catalog_iceberg_table(catalog, ns, tbl) + + setup_catalog_replicas(catalog_export_cluster, source, ["replica1", "replica2"]) + + r1 = catalog_export_cluster.instances["replica1"] + r2 = catalog_export_cluster.instances["replica2"] + + r1.query(f"INSERT INTO {source} VALUES (1, 'EU'), (2, 'EU'), (3, 'EU')") + r1.query(f"INSERT INTO {source} VALUES (4, 'US'), (5, 'US'), (6, 'US')") + r2.query(f"SYSTEM SYNC REPLICA {source}") + + pid_eu = partition_id_for(r1, source, "EU") + pid_us = partition_id_for(r1, source, "US") + dest_ch = f"`{CH_CATALOG_DB}`.`{ns}.{tbl}`" + + errors: list = [] + + def export_partition(node, pid): + try: + node.query( + f"ALTER TABLE {source} EXPORT PARTITION ID '{pid}' TO TABLE {dest_ch}", + settings={"write_full_path_in_iceberg_metadata": 1, "allow_insert_into_iceberg": 1}, + ) + wait_for_export_status(node, source, None, pid, timeout=120) + except Exception as exc: + errors.append(exc) + + t1 = threading.Thread(target=export_partition, args=(r1, pid_eu)) + t2 = threading.Thread(target=export_partition, args=(r2, pid_us)) + t1.start() + t2.start() + t1.join() + t2.join() + + assert not errors, f"Export threads raised errors: {errors}" + + count = int(r1.query(f"SELECT count() FROM {dest_ch}").strip()) + assert count == 6, f"Expected 6 rows (3 EU + 3 US), got {count}" + + iceberg_tbl = catalog.load_table(f"{ns}.{tbl}") + history = iceberg_tbl.history() + assert len(history) >= 2, ( + f"Expected ≥2 snapshots (one per concurrent partition commit), got {len(history)}" + ) + + +# TODO arthur fix: TOCTOU in export registration path. +# The exists() pre-check and the tryMulti() commit are not a single atomic ZK +# transaction. Depending on timing, the loser gets either KEEPER_EXCEPTION +# "Node exists" (both replicas race past exists() and collide at tryMulti) or +# BAD_ARGUMENTS "already exported" (the winner commits before the loser's +# exists() check). The test cannot reliably assert either error in isolation. +# def test_catalog_idempotent_same_partition_two_replicas(catalog_export_cluster): +# catalog = connect_catalog(catalog_export_cluster) +# +# ns = f"ns_{uuid.uuid4().hex[:8]}" +# tbl = f"tbl_{uuid.uuid4().hex[:8]}" +# source = f"rmt_{uuid.uuid4().hex[:8]}" +# +# catalog.create_namespace((ns,)) +# create_catalog_iceberg_table(catalog, ns, tbl) +# +# setup_catalog_replicas(catalog_export_cluster, source, ["replica1", "replica2"]) +# +# r1 = catalog_export_cluster.instances["replica1"] +# r2 = catalog_export_cluster.instances["replica2"] +# +# r1.query(f"INSERT INTO {source} VALUES (1, 'EU'), (2, 'EU'), (3, 'EU')") +# r2.query(f"SYSTEM SYNC REPLICA {source}") +# +# pid = partition_id_for(r1, source, "EU") +# dest_ch = f"`{CH_CATALOG_DB}`.`{ns}.{tbl}`" +# +# errors: list = [] +# +# def export_from(node): +# try: +# node.query( +# f"ALTER TABLE {source} EXPORT PARTITION ID '{pid}' TO TABLE {dest_ch}", +# settings={"write_full_path_in_iceberg_metadata": 1, "allow_insert_into_iceberg": 1}, +# ) +# wait_for_export_status(node, source, None, pid, timeout=120) +# except Exception as exc: +# errors.append(exc) +# +# t1 = threading.Thread(target=export_from, args=(r1,)) +# t2 = threading.Thread(target=export_from, args=(r2,)) +# t1.start() +# t2.start() +# t1.join() +# t2.join() +# +# unexpected = [e for e in errors if "already exported" not in str(e)] +# assert not unexpected, f"Unexpected export errors: {unexpected}" +# +# count = int(r1.query(f"SELECT count() FROM {dest_ch}").strip()) +# assert count == 3, f"Expected 3 rows (no duplication), got {count}" +# +# iceberg_tbl = catalog.load_table(f"{ns}.{tbl}") +# history = iceberg_tbl.history() +# assert len(history) == 1, ( +# f"Expected exactly 1 snapshot (one winner, one rejected by export-key guard), " +# f"got {len(history)}" +# ) diff --git a/tests/integration/test_storage_iceberg_with_spark/test_minmax_pruning_with_null.py b/tests/integration/test_storage_iceberg_with_spark/test_minmax_pruning_with_null.py index ceb630acbd73..93ba2f765914 100644 --- a/tests/integration/test_storage_iceberg_with_spark/test_minmax_pruning_with_null.py +++ b/tests/integration/test_storage_iceberg_with_spark/test_minmax_pruning_with_null.py @@ -9,7 +9,10 @@ ) @pytest.mark.parametrize("storage_type", ["s3", "azure", "local"]) -def test_minmax_pruning_with_null(started_cluster_iceberg_with_spark, storage_type): +@pytest.mark.parametrize("run_on_cluster", [False, True]) +def test_minmax_pruning_with_null(started_cluster_iceberg_with_spark, storage_type, run_on_cluster): + if run_on_cluster and storage_type == "local": + pytest.skip("Local storage is not supported on cluster") instance = started_cluster_iceberg_with_spark.instances["node1"] spark = started_cluster_iceberg_with_spark.spark_session TABLE_NAME = "test_minmax_pruning_with_null" + storage_type + "_" + get_uuid_str() @@ -21,6 +24,7 @@ def execute_spark_query(query: str): storage_type, TABLE_NAME, query, + additional_nodes=["node2", "node3"] if storage_type=="local" else [], ) execute_spark_query( @@ -79,7 +83,7 @@ def execute_spark_query(query: str): ) creation_expression = get_creation_expression( - storage_type, TABLE_NAME, started_cluster_iceberg_with_spark, table_function=True + storage_type, TABLE_NAME, started_cluster_iceberg_with_spark, table_function=True, run_on_cluster=run_on_cluster ) def check_validity_and_get_prunned_files(select_expression): diff --git a/tests/integration/test_storage_iceberg_with_spark/test_partition_pruning.py b/tests/integration/test_storage_iceberg_with_spark/test_partition_pruning.py index 6ade42e72537..4c6a6b4c7bd7 100644 --- a/tests/integration/test_storage_iceberg_with_spark/test_partition_pruning.py +++ b/tests/integration/test_storage_iceberg_with_spark/test_partition_pruning.py @@ -9,7 +9,7 @@ @pytest.mark.parametrize( "storage_type, run_on_cluster", - [("s3", False), ("s3", True), ("azure", False), ("local", False), ("local", True)], + [("s3", False), ("s3", True), ("azure", False), ("azure", True), ("local", False), ("local", True)], ) def test_partition_pruning(started_cluster_iceberg_with_spark, storage_type, run_on_cluster): instance = started_cluster_iceberg_with_spark.instances["node1"] diff --git a/tests/integration/test_storage_iceberg_with_spark/test_read_constant_columns_optimization.py b/tests/integration/test_storage_iceberg_with_spark/test_read_constant_columns_optimization.py new file mode 100644 index 000000000000..58ac641f0074 --- /dev/null +++ b/tests/integration/test_storage_iceberg_with_spark/test_read_constant_columns_optimization.py @@ -0,0 +1,289 @@ +import pytest + +from helpers.iceberg_utils import ( + get_uuid_str, + get_creation_expression, + execute_spark_query_general, +) + + +@pytest.mark.parametrize("storage_type", ["s3", "azure"]) +@pytest.mark.parametrize("run_on_cluster", [False, True]) +def test_read_constant_columns_optimization(started_cluster_iceberg_with_spark, storage_type, run_on_cluster): + instance = started_cluster_iceberg_with_spark.instances["node1"] + spark = started_cluster_iceberg_with_spark.spark_session + TABLE_NAME = "test_read_constant_columns_optimization_" + storage_type + "_" + get_uuid_str() + + def execute_spark_query(query: str): + return execute_spark_query_general( + spark, + started_cluster_iceberg_with_spark, + storage_type, + TABLE_NAME, + query, + ) + + execute_spark_query( + f""" + CREATE TABLE {TABLE_NAME} ( + tag INT, + date DATE, + date2 DATE, + name VARCHAR(50), + number BIGINT + ) + USING iceberg + PARTITIONED BY (identity(tag), years(date)) + OPTIONS('format-version'='2') + """ + ) + + execute_spark_query( + f""" + INSERT INTO {TABLE_NAME} VALUES + (1, DATE '2024-01-20', DATE '2024-01-20', 'vasya', 5), + (1, DATE '2024-01-20', DATE '2024-01-20', 'vasilisa', 5), + (1, DATE '2025-01-20', DATE '2025-01-20', 'vasya', 5), + (1, DATE '2025-01-20', DATE '2025-01-20', 'vasya', 5), + (2, DATE '2025-01-20', DATE '2025-01-20', 'vasilisa', 5), + (2, DATE '2025-01-21', DATE '2025-01-20', 'vasilisa', 5) + """ + ) + + execute_spark_query( + f""" + ALTER TABLE {TABLE_NAME} ALTER COLUMN number FIRST; + """ + ) + + execute_spark_query( + f""" + INSERT INTO {TABLE_NAME} VALUES + (5, 3, DATE '2025-01-20', DATE '2024-01-20', 'vasilisa'), + (5, 3, DATE '2025-01-20', DATE '2025-01-20', 'vasilisa') + """ + ) + + execute_spark_query( + f""" + ALTER TABLE {TABLE_NAME} RENAME COLUMN name TO name_old; + """ + ) + + execute_spark_query( + f""" + ALTER TABLE {TABLE_NAME} + ADD COLUMNS ( + name string + ); + """ + ) + + execute_spark_query( + f""" + INSERT INTO {TABLE_NAME} VALUES + (5, 4, DATE '2025-01-20', DATE '2024-01-20', 'vasya', 'iceberg'), + (5, 4, DATE '2025-01-20', DATE '2025-01-20', 'vasilisa', 'iceberg'), + (5, 5, DATE '2025-01-20', DATE '2024-01-20', 'vasya', 'iceberg'), + (5, 5, DATE '2025-01-20', DATE '2024-01-20', 'vasilisa', 'icebreaker'), + (5, 6, DATE '2025-01-20', DATE '2024-01-20', 'vasya', 'iceberg'), + (5, 6, DATE '2025-01-20', DATE '2024-01-20', 'vasya', 'iceberg') + """ + ) + + # Totally must be 7 files + # Partitioned column 'tag' is constant in each file + # Column 'date' is constant in 6 files, has different values in (2-2025) + # Column 'date2' is constant in 4 files (1-2024, 2-2025, 5-2025, 6-2025) + # Column 'name_old' is constant in 3 files (1-2025, 2-2025 as 'name', 6-2025 as 'name_old') + # Column 'number' is globally constant + # New column 'name2' is present only in 3 files (4-2025, 5-2025, 6-2025), constant in two (4-2025, 6-2025) + # Files 1-2025 and 6-2025 have only constant columns + + creation_expression = get_creation_expression( + storage_type, TABLE_NAME, started_cluster_iceberg_with_spark, table_function=True, run_on_cluster=run_on_cluster + ) + + # Warm up metadata cache + for replica in started_cluster_iceberg_with_spark.instances.values(): + replica.query(f"SELECT * FROM {creation_expression} ORDER BY ALL SETTINGS allow_experimental_iceberg_read_optimization=0") + + all_data_expected_query_id = get_uuid_str() + all_data_expected = instance.query( + f"SELECT * FROM {creation_expression} ORDER BY ALL SETTINGS allow_experimental_iceberg_read_optimization=0", + query_id=all_data_expected_query_id, + ) + const_only_expected_query_id = get_uuid_str() + const_only_expected = instance.query( + f"SELECT tag, number FROM {creation_expression} ORDER BY ALL SETTINGS allow_experimental_iceberg_read_optimization=0", + query_id=const_only_expected_query_id, + ) + const_partial_expected_query_id = get_uuid_str() + const_partial_expected = instance.query( + f"SELECT tag, date2, number, name_old FROM {creation_expression} ORDER BY ALL SETTINGS allow_experimental_iceberg_read_optimization=0", + query_id=const_partial_expected_query_id, + ) + const_partial2_expected_query_id = get_uuid_str() + const_partial2_expected = instance.query( + f"SELECT tag, date2, number, name FROM {creation_expression} ORDER BY ALL SETTINGS allow_experimental_iceberg_read_optimization=0", + query_id=const_partial2_expected_query_id, + ) + count_expected_query_id = get_uuid_str() + count_expected = instance.query( + f"SELECT count(),tag FROM {creation_expression} GROUP BY ALL ORDER BY ALL SETTINGS allow_experimental_iceberg_read_optimization=0", + query_id=count_expected_query_id, + ) + + all_data_query_id = get_uuid_str() + all_data_optimized = instance.query( + f"SELECT * FROM {creation_expression} ORDER BY ALL SETTINGS allow_experimental_iceberg_read_optimization=1", + query_id=all_data_query_id, + ) + const_only_query_id = get_uuid_str() + const_only_optimized = instance.query( + f"SELECT tag, number FROM {creation_expression} ORDER BY ALL SETTINGS allow_experimental_iceberg_read_optimization=1", + query_id=const_only_query_id, + ) + const_partial_query_id = get_uuid_str() + const_partial_optimized = instance.query( + f"SELECT tag, date2, number, name_old FROM {creation_expression} ORDER BY ALL SETTINGS allow_experimental_iceberg_read_optimization=1", + query_id=const_partial_query_id, + ) + const_partial2_query_id = get_uuid_str() + const_partial2_optimized = instance.query( + f"SELECT tag, date2, number, name FROM {creation_expression} ORDER BY ALL SETTINGS allow_experimental_iceberg_read_optimization=1", + query_id=const_partial2_query_id, + ) + count_query_id = get_uuid_str() + count_optimized = instance.query( + f"SELECT count(),tag FROM {creation_expression} GROUP BY ALL ORDER BY ALL SETTINGS allow_experimental_iceberg_read_optimization=1", + query_id=count_query_id, + ) + + assert all_data_expected == all_data_optimized + assert const_only_expected == const_only_optimized + assert const_partial_expected == const_partial_optimized + assert const_partial2_expected == const_partial2_optimized + assert count_expected == count_optimized + + for replica in started_cluster_iceberg_with_spark.instances.values(): + replica.query("SYSTEM FLUSH LOGS") + + # Number of object-get requests per data file that are NOT served from caches + # after the warmup query above. The parquet metadata cache (enabled by default) + # caches the parquet footer keyed by the object's etag; the warmup query then + # populates it, so any subsequent read of the same file skips one object-get + # (the footer read). However, AzureObjectStorage::getObjectMetadata does NOT + # populate etag, so the cache guard `!etag.empty()` in + # StorageObjectStorageSource::createReader always fails for Azure, and the + # cache path is never taken there. As a result the multiplier is: + # S3: 2 (footer served from cache, data-only gets remain) + # Azure: 3 (cache never engaged, footer + data gets) + per_file_gets = 2 if storage_type == "s3" else 3 + + def check_events(query_id, event, is_cluster, expected): + res = instance.query( + f""" + SELECT + sum(tupleElement(arrayJoin(ProfileEvents),2)) as value + FROM + clusterAllReplicas('cluster_simple', system.query_log) + WHERE + type='QueryFinish' + AND tupleElement(arrayJoin(ProfileEvents),1)='{event}' + AND initial_query_id='{query_id}' + GROUP BY ALL + FORMAT CSV + """) + assert int(res) == expected + + # Each file contains one row group, so number of reded row groups is equal to readed data files + event = "ParquetReadRowGroups" + + # Without optimization clickhouse reads all 7 files + check_events(all_data_expected_query_id, event, run_on_cluster, 7) + check_events(const_only_expected_query_id, event, run_on_cluster, 7) + check_events(const_partial_expected_query_id, event, run_on_cluster, 7) + check_events(const_partial2_expected_query_id, event, run_on_cluster, 7) + check_events(count_expected_query_id, event, run_on_cluster, 7) + + # If file has only constant columns it is not read + check_events(all_data_query_id, event, run_on_cluster, 5) # 1-2025, 6-2025 must not be read + check_events(const_only_query_id, event, run_on_cluster, 0) # All must not be read + check_events(const_partial_query_id, event, run_on_cluster, 4) # 1-2025, 6-2025 and 2-2025 must not be read + check_events(const_partial2_query_id, event, run_on_cluster, 3) # 6-2025 must not be read, 1-2024, 1-2025, 2-2025 don't have new column 'name' + check_events(count_query_id, event, run_on_cluster, 0) # All must not be read + + def compare_selects(query): + result_expected = instance.query(f"{query} SETTINGS allow_experimental_iceberg_read_optimization=0") + result_optimized = instance.query(f"{query} SETTINGS allow_experimental_iceberg_read_optimization=1") + assert result_expected == result_optimized + + compare_selects(f"SELECT _path,* FROM {creation_expression} ORDER BY ALL") + compare_selects(f"SELECT _path,* FROM {creation_expression} WHERE name_old='vasily' ORDER BY ALL") + compare_selects(f"SELECT _path,* FROM {creation_expression} WHERE ((tag + length(name_old)) % 2 = 1) ORDER BY ALL") + + +@pytest.mark.parametrize("storage_type", ["s3", "azure"]) +@pytest.mark.parametrize("run_on_cluster", [False, True]) +def test_read_constant_columns_optimization_view(started_cluster_iceberg_with_spark, storage_type, run_on_cluster): + instance = started_cluster_iceberg_with_spark.instances["node1"] + spark = started_cluster_iceberg_with_spark.spark_session + TABLE_NAME = "test_read_constant_columns_optimization_view_" + storage_type + "_" + get_uuid_str() + + def execute_spark_query(query: str): + return execute_spark_query_general( + spark, + started_cluster_iceberg_with_spark, + storage_type, + TABLE_NAME, + query, + ) + + execute_spark_query( + f""" + CREATE TABLE {TABLE_NAME} ( + tag INT, + date DATE, + date2 DATE, + name VARCHAR(50), + number BIGINT + ) + USING iceberg + PARTITIONED BY (identity(tag), years(date)) + OPTIONS('format-version'='2') + """ + ) + + execute_spark_query( + f""" + INSERT INTO {TABLE_NAME} VALUES + (1, DATE '2024-01-20', DATE '2024-01-20', 'vasya', 5), + (1, DATE '2024-01-20', DATE '2024-01-20', 'vasilisa', 5), + (1, DATE '2025-01-20', DATE '2025-01-20', 'vasya', 5), + (1, DATE '2025-01-20', DATE '2025-01-20', 'vasya', 5), + (2, DATE '2025-01-20', DATE '2025-01-20', 'vasilisa', 5), + (2, DATE '2025-01-21', DATE '2025-01-20', 'vasilisa', 5) + """ + ) + + creation_expression = get_creation_expression( + storage_type, TABLE_NAME, started_cluster_iceberg_with_spark, table_function=True, run_on_cluster=run_on_cluster + ) + + # Check that view over Iceberg table works + instance.query(f"CREATE VIEW {TABLE_NAME}_view AS SELECT * FROM {creation_expression}") + + expected = instance.query(f"SELECT * FROM {TABLE_NAME}_view ORDER BY ALL SETTINGS allow_experimental_iceberg_read_optimization=0") + # All data + optimized = instance.query(f"SELECT * FROM {TABLE_NAME}_view ORDER BY ALL SETTINGS allow_experimental_iceberg_read_optimization=1") + assert expected == optimized + # Constant column in where + optimized = instance.query(f"SELECT * FROM {TABLE_NAME}_view WHERE number=5 ORDER BY ALL SETTINGS allow_experimental_iceberg_read_optimization=1") + assert expected == optimized + # Partition columns in where + optimized = instance.query(f"SELECT * FROM {TABLE_NAME}_view WHERE tag>0 AND date>'2020-01-01' ORDER BY ALL SETTINGS allow_experimental_iceberg_read_optimization=1") + assert expected == optimized + # Non-constant column in where + optimized = instance.query(f"SELECT * FROM {TABLE_NAME}_view WHERE date2!='2020-01-01' ORDER BY ALL SETTINGS allow_experimental_iceberg_read_optimization=1") + assert expected == optimized diff --git a/tests/integration/test_storage_iceberg_with_spark/test_remote_initiator.py b/tests/integration/test_storage_iceberg_with_spark/test_remote_initiator.py new file mode 100644 index 000000000000..763836d21f60 --- /dev/null +++ b/tests/integration/test_storage_iceberg_with_spark/test_remote_initiator.py @@ -0,0 +1,156 @@ +import pytest +import uuid + +from helpers.iceberg_utils import ( + get_uuid_str, + create_iceberg_table, + execute_spark_query_general, +) + + +@pytest.mark.parametrize("storage_type", ["s3"]) +def test_remote_initiator_after_non_remote(started_cluster_iceberg_with_spark, storage_type): + instance = started_cluster_iceberg_with_spark.instances["node1"] + spark = started_cluster_iceberg_with_spark.spark_session + TABLE_NAME = "test_remote_initiator_after_non_remote_table_" + get_uuid_str() + + def execute_spark_query(query: str): + return execute_spark_query_general( + spark, + started_cluster_iceberg_with_spark, + storage_type, + TABLE_NAME, + query, + ) + + execute_spark_query( + f""" + CREATE TABLE {TABLE_NAME} ( + tag INT, + number INT + ) + USING iceberg + PARTITIONED BY (identity(tag)) + OPTIONS('format-version'='2') + """ + ) + + execute_spark_query( + f""" + INSERT INTO {TABLE_NAME} VALUES + (1, 1) + """ + ) + + create_iceberg_table(storage_type, instance, TABLE_NAME, started_cluster_iceberg_with_spark) + + def flush_logs(): + for node in started_cluster_iceberg_with_spark.instances.values(): + node.query("SYSTEM FLUSH LOGS") + + query_id = uuid.uuid4().hex + res = instance.query(f""" + SELECT * + FROM {TABLE_NAME} + WHERE number=1 + SETTINGS + object_storage_cluster='cluster_23' + """, + query_id = query_id) + assert res == "1\t1\n" + flush_logs() + queries = instance.query(f""" + SELECT count() + FROM clusterAllReplicas('cluster_simple', system.query_log) + WHERE type='QueryFinish' AND initial_query_id='{query_id}' + """) + # initial node + 2 subqueries on replicas + assert queries == "3\n" + + query_id = uuid.uuid4().hex + res = instance.query(f""" + SELECT * + FROM {TABLE_NAME} + WHERE number=1 + SETTINGS + object_storage_remote_initiator=1, + object_storage_cluster='cluster_23' + """, + query_id = query_id) + assert res == "1\t1\n" + flush_logs() + queries = instance.query(f""" + SELECT count() + FROM clusterAllReplicas('cluster_simple', system.query_log) + WHERE type='QueryFinish' AND initial_query_id='{query_id}' + """) + # initial node + remote initiator + 2 subqueries on replicas + assert queries == "4\n" + + query_id = uuid.uuid4().hex + res = instance.query(f""" + SELECT * + FROM {TABLE_NAME} + WHERE number=1 + """, + query_id = query_id) + assert res == "1\t1\n" + flush_logs() + queries = instance.query(f""" + SELECT count() + FROM clusterAllReplicas('cluster_simple', system.query_log) + WHERE type='QueryFinish' AND initial_query_id='{query_id}' + """) + assert queries == "1\n" + + +@pytest.mark.parametrize("storage_type", ["s3"]) +def test_remote_initiator_after_with_join_old_analyzer(started_cluster_iceberg_with_spark, storage_type): + instance = started_cluster_iceberg_with_spark.instances["node1"] + spark = started_cluster_iceberg_with_spark.spark_session + TABLE_NAME = "test_remote_initiator_after_with_join_old_analyzer_table_" + get_uuid_str() + TABLE2_NAME = "test_remote_initiator_after_with_join_old_analyzer_table_2_" + get_uuid_str() + + def execute_spark_query(query: str): + return execute_spark_query_general( + spark, + started_cluster_iceberg_with_spark, + storage_type, + TABLE_NAME, + query, + ) + + execute_spark_query( + f""" + CREATE TABLE {TABLE_NAME} ( + tag INT, + number INT + ) + USING iceberg + PARTITIONED BY (identity(tag)) + OPTIONS('format-version'='2') + """ + ) + + execute_spark_query( + f""" + INSERT INTO {TABLE_NAME} VALUES + (1, 1) + """ + ) + + create_iceberg_table(storage_type, instance, TABLE_NAME, started_cluster_iceberg_with_spark) + + instance.query(f"CREATE TABLE {TABLE2_NAME} (tag INT, number2 INT) ENGINE=Memory") + instance.query(f"INSERT INTO {TABLE2_NAME} VALUES (1, 2)") + + assert "object_storage_cluster_join_mode!='allow' is not supported without allow_experimental_analyzer=true" in instance.query_and_get_error(f""" + SELECT * + FROM {TABLE_NAME} AS t1 + JOIN {TABLE2_NAME} AS t2 USING (tag) + SETTINGS + object_storage_remote_initiator=1, + object_storage_remote_initiator_cluster='cluster_simple', + object_storage_cluster_join_mode='local', + allow_experimental_analyzer=0 + """) diff --git a/tests/integration/test_storage_iceberg_with_spark/test_system_iceberg_metadata.py b/tests/integration/test_storage_iceberg_with_spark/test_system_iceberg_metadata.py index a2d4c75f2933..30773442a355 100644 --- a/tests/integration/test_storage_iceberg_with_spark/test_system_iceberg_metadata.py +++ b/tests/integration/test_storage_iceberg_with_spark/test_system_iceberg_metadata.py @@ -1,6 +1,8 @@ from datetime import datetime, timedelta import pytest import json +import uuid +import time from helpers.iceberg_utils import ( create_iceberg_table, @@ -17,7 +19,7 @@ def __init__(self, not_pruned, partition_pruned, min_max_index_pruned): def __repr__(self): return "PrunedInfo(not_pruned={}, partition_pruned={}, min_max_index_pruned={})".format(self.not_pruned, self.partition_pruned, self.min_max_index_pruned) - + def __eq__(self, other): return (self.not_pruned == other.not_pruned and self.partition_pruned == other.partition_pruned and @@ -230,7 +232,7 @@ def execute_spark_query(query: str): raise date_and_time_columns = get_date_and_time_columns(instance, query_id) - + event_dates = date_and_time_columns['event_date'] event_times = date_and_time_columns['event_time'] @@ -242,4 +244,50 @@ def execute_spark_query(query: str): for time_str in event_times: current_time = datetime.fromisoformat(time_str) assert current_time <= datetime.now(), "Event time is in the future. Event time: {}, current time: {}".format(current_time, datetime.now()) - assert current_time >= (datetime.now() - timedelta(days=1)), "Event time is too old. Event time: {}, current time: {}".format(current_time, datetime.now()) \ No newline at end of file + assert current_time >= (datetime.now() - timedelta(days=1)), "Event time is too old. Event time: {}, current time: {}".format(current_time, datetime.now()) + + +@pytest.mark.parametrize("storage_type", ["s3"]) +def test_system_tables_partition_sorting_keys(started_cluster_iceberg_with_spark, storage_type): + instance = started_cluster_iceberg_with_spark.instances["node1"] + spark = started_cluster_iceberg_with_spark.spark_session + + table_name = f"test_sys_tables_keys_{storage_type}_{uuid.uuid4().hex[:8]}" + fq_table = f"spark_catalog.default.{table_name}" + + spark.sql(f"DROP TABLE IF EXISTS {fq_table}") + spark.sql(f""" + CREATE TABLE {fq_table} ( + id INT, + ts TIMESTAMP, + payload STRING + ) + USING iceberg + PARTITIONED BY (bucket(16, id), day(ts)) + TBLPROPERTIES ('format-version' = '2') + """) + spark.sql(f"ALTER TABLE {fq_table} WRITE ORDERED BY (id DESC NULLS LAST, hour(ts))") + spark.sql(f""" + INSERT INTO {fq_table} VALUES + (1, timestamp'2024-01-01 10:00:00', 'a'), + (2, timestamp'2024-01-02 11:00:00', 'b'), + (NULL, timestamp'2024-01-03 12:00:00', 'c') + """) + + time.sleep(2) + default_upload_directory( + started_cluster_iceberg_with_spark, + storage_type, + f"/iceberg_data/default/{table_name}/", + f"/iceberg_data/default/{table_name}/", + ) + + create_iceberg_table(storage_type, instance, table_name, started_cluster_iceberg_with_spark) + + res = instance.query(f""" + SELECT partition_key, sorting_key + FROM system.tables + WHERE name = '{table_name}' FORMAT csv + """).strip().lower() + + assert res == '"bucket(16, id), day(ts)","id desc, hour(ts) asc"' diff --git a/tests/integration/test_storage_iceberg_with_spark/test_types.py b/tests/integration/test_storage_iceberg_with_spark/test_types.py index 7f63df522db1..1dd605098279 100644 --- a/tests/integration/test_storage_iceberg_with_spark/test_types.py +++ b/tests/integration/test_storage_iceberg_with_spark/test_types.py @@ -86,3 +86,49 @@ def test_types(started_cluster_iceberg_with_spark, format_version, storage_type) ["e", "Nullable(Bool)"], ] ) + + # Test storage type as function argument + table_function_expr = get_creation_expression( + storage_type, + TABLE_NAME, + started_cluster_iceberg_with_spark, + table_function=True, + storage_type_as_arg=True, + ) + assert ( + instance.query(f"SELECT a, b, c, d, e FROM {table_function_expr}").strip() + == "123\tstring\t2000-01-01\t['str1','str2']\ttrue" + ) + + assert instance.query(f"DESCRIBE {table_function_expr} FORMAT TSV") == TSV( + [ + ["a", "Nullable(Int32)"], + ["b", "Nullable(String)"], + ["c", "Nullable(Date32)"], + ["d", "Array(Nullable(String))"], + ["e", "Nullable(Bool)"], + ] + ) + + # Test storage type as field in named collection + table_function_expr = get_creation_expression( + storage_type, + TABLE_NAME, + started_cluster_iceberg_with_spark, + table_function=True, + storage_type_in_named_collection=True, + ) + assert ( + instance.query(f"SELECT a, b, c, d, e FROM {table_function_expr}").strip() + == "123\tstring\t2000-01-01\t['str1','str2']\ttrue" + ) + + assert instance.query(f"DESCRIBE {table_function_expr} FORMAT TSV") == TSV( + [ + ["a", "Nullable(Int32)"], + ["b", "Nullable(String)"], + ["c", "Nullable(Date32)"], + ["d", "Array(Nullable(String))"], + ["e", "Nullable(Bool)"], + ] + ) diff --git a/tests/integration/test_storage_s3/configs/lock_object_storage_task_distribution_ms.xml b/tests/integration/test_storage_s3/configs/lock_object_storage_task_distribution_ms.xml new file mode 100644 index 000000000000..a8239a28293c --- /dev/null +++ b/tests/integration/test_storage_s3/configs/lock_object_storage_task_distribution_ms.xml @@ -0,0 +1,7 @@ + + + + 0 + + + diff --git a/tests/integration/test_storage_s3/test.py b/tests/integration/test_storage_s3/test.py index 96e94b2a89e8..16cc9162234c 100644 --- a/tests/integration/test_storage_s3/test.py +++ b/tests/integration/test_storage_s3/test.py @@ -86,6 +86,7 @@ def started_cluster(): "configs/s3_retry.xml", "configs/sync_insert.xml", "configs/allow_server_credentials.xml", + "configs/lock_object_storage_task_distribution_ms.xml", ], ) cluster.add_instance( @@ -190,6 +191,7 @@ def started_cluster(): "configs/s3_retry.xml", "configs/process_archives_as_whole_with_cluster.xml", "configs/sync_insert.xml", + "configs/lock_object_storage_task_distribution_ms.xml", ], ) cluster.add_instance( diff --git a/tests/integration/test_storage_url/test.py b/tests/integration/test_storage_url/test.py index cd72d82b2fc1..033fba579a4f 100644 --- a/tests/integration/test_storage_url/test.py +++ b/tests/integration/test_storage_url/test.py @@ -77,6 +77,38 @@ def test_partition_by(): assert result.strip() == "1\t2\t3" +def test_hive_partitioning_with_where_condition(): + test_id = uuid.uuid4().hex[:8] + base_url = f"http://nginx:80/hive_url_cluster_{test_id}" + + node1.query( + f""" + INSERT INTO FUNCTION url(url_file, url='{base_url}/date=2000-01-01/data.csv', format='CSVWithNames', structure='d UInt64') + SELECT number FROM numbers(10) + """ + ) + + # 'ur' table function does not work with globs, so we have to test hive partitioning with a single file. + result = node1.query( + f""" + SELECT count() FROM url('{base_url}/date=2000-01-01/data.csv', 'CSVWithNames', 'd UInt64') + WHERE date='2000-01-01' + SETTINGS use_hive_partitioning=1 + """ + ) + assert result.strip() == "10" + + result = node1.query( + f""" + SELECT count() FROM urlCluster( + 'test_cluster_two_shards', '{base_url}/date=2000-01-01/data.csv', 'CSVWithNames', 'd UInt64') + WHERE date='2000-01-01' + SETTINGS use_hive_partitioning=1 + """ + ) + assert result.strip() == "10" + + def test_url_cluster(): result = node1.query( "select * from urlCluster('test_cluster_two_shards', 'http://nginx:80/test_1', 'TSV', 'column1 UInt32, column2 UInt32, column3 UInt32')" diff --git a/tests/queries/0_stateless/01271_show_privileges.reference b/tests/queries/0_stateless/01271_show_privileges.reference index 1d40e020e32a..5f16f7ce7a47 100644 --- a/tests/queries/0_stateless/01271_show_privileges.reference +++ b/tests/queries/0_stateless/01271_show_privileges.reference @@ -46,6 +46,8 @@ ALTER MATERIALIZE TTL ['MATERIALIZE TTL'] TABLE ALTER TABLE ALTER REWRITE PARTS ['REWRITE PARTS'] TABLE ALTER TABLE ALTER SETTINGS ['ALTER SETTING','ALTER MODIFY SETTING','MODIFY SETTING','RESET SETTING'] TABLE ALTER TABLE ALTER MOVE PARTITION ['ALTER MOVE PART','MOVE PARTITION','MOVE PART'] TABLE ALTER TABLE +ALTER EXPORT PART ['ALTER EXPORT PART','EXPORT PART'] TABLE ALTER TABLE +ALTER EXPORT PARTITION ['ALTER EXPORT PARTITION','EXPORT PARTITION'] TABLE ALTER TABLE ALTER FETCH PARTITION ['ALTER FETCH PART','FETCH PARTITION'] TABLE ALTER TABLE ALTER FREEZE PARTITION ['FREEZE PARTITION','UNFREEZE'] TABLE ALTER TABLE ALTER UNLOCK SNAPSHOT ['UNLOCK SNAPSHOT'] TABLE ALTER TABLE @@ -157,6 +159,7 @@ SYSTEM DROP PAGE CACHE ['SYSTEM CLEAR PAGE CACHE','SYSTEM DROP PAGE CACHE','DROP SYSTEM DROP SCHEMA CACHE ['SYSTEM CLEAR SCHEMA CACHE','SYSTEM DROP SCHEMA CACHE','DROP SCHEMA CACHE'] GLOBAL SYSTEM DROP CACHE SYSTEM DROP FORMAT SCHEMA CACHE ['SYSTEM CLEAR FORMAT SCHEMA CACHE','SYSTEM DROP FORMAT SCHEMA CACHE','DROP FORMAT SCHEMA CACHE'] GLOBAL SYSTEM DROP CACHE SYSTEM DROP S3 CLIENT CACHE ['SYSTEM CLEAR S3 CLIENT CACHE','SYSTEM DROP S3 CLIENT','DROP S3 CLIENT CACHE'] GLOBAL SYSTEM DROP CACHE +SYSTEM DROP OBJECT STORAGE LIST OBJECTS CACHE ['SYSTEM DROP OBJECT STORAGE LIST OBJECTS CACHE'] GLOBAL SYSTEM DROP CACHE SYSTEM DROP CACHE ['DROP CACHE'] \N SYSTEM SYSTEM RELOAD CONFIG ['RELOAD CONFIG'] GLOBAL SYSTEM RELOAD SYSTEM RELOAD USERS ['RELOAD USERS'] GLOBAL SYSTEM RELOAD @@ -173,6 +176,7 @@ SYSTEM MERGES ['SYSTEM STOP MERGES','SYSTEM START MERGES','STOP MERGES','START M SYSTEM TTL MERGES ['SYSTEM STOP TTL MERGES','SYSTEM START TTL MERGES','STOP TTL MERGES','START TTL MERGES'] TABLE SYSTEM SYSTEM FETCHES ['SYSTEM STOP FETCHES','SYSTEM START FETCHES','STOP FETCHES','START FETCHES'] TABLE SYSTEM SYSTEM MOVES ['SYSTEM STOP MOVES','SYSTEM START MOVES','STOP MOVES','START MOVES'] TABLE SYSTEM +SYSTEM SWARM ['SYSTEM STOP SWARM MODE','SYSTEM START SWARM MODE','STOP SWARM MODE','START SWARM MODE'] GLOBAL SYSTEM SYSTEM PULLING REPLICATION LOG ['SYSTEM STOP PULLING REPLICATION LOG','SYSTEM START PULLING REPLICATION LOG'] TABLE SYSTEM SYSTEM CLEANUP ['SYSTEM STOP CLEANUP','SYSTEM START CLEANUP'] TABLE SYSTEM SYSTEM VIEWS ['SYSTEM REFRESH VIEW','SYSTEM START VIEWS','SYSTEM STOP VIEWS','SYSTEM START VIEW','SYSTEM STOP VIEW','SYSTEM PAUSE VIEWS','SYSTEM PAUSE VIEW','SYSTEM CANCEL VIEW','REFRESH VIEW','START VIEWS','STOP VIEWS','START VIEW','STOP VIEW','PAUSE VIEWS','PAUSE VIEW','CANCEL VIEW'] VIEW SYSTEM BACKGROUND diff --git a/tests/queries/0_stateless/01625_constraints_index_append.reference b/tests/queries/0_stateless/01625_constraints_index_append.reference index b7ae22a69ead..007816367b7a 100644 --- a/tests/queries/0_stateless/01625_constraints_index_append.reference +++ b/tests/queries/0_stateless/01625_constraints_index_append.reference @@ -1,15 +1,15 @@ - Filter column: and(equals(a, 0), indexHint(greater(plus(i, 40), 0))) (removed) + Filter column: and(indexHint(greater(plus(i, 40), 0)), equals(a, 0)) (removed) Prewhere info Prewhere filter Prewhere filter column: equals(a, 0) Prewhere info Prewhere filter Prewhere filter column: less(a, 0) (removed) - Filter column: and(greaterOrEquals(a, 0), indexHint(greater(plus(i, 40), 0))) (removed) + Filter column: and(indexHint(greater(plus(i, 40), 0)), greaterOrEquals(a, 0)) (removed) Prewhere info Prewhere filter Prewhere filter column: greaterOrEquals(a, 0) - Filter column: and(indexHint(less(i, 100)), less(multiply(2, b), 100)) (removed) + Filter column: and(less(multiply(2, b), 100), indexHint(less(i, 100))) (removed) Prewhere info Prewhere filter Prewhere filter column: less(multiply(2, b), 100) diff --git a/tests/queries/0_stateless/02126_dist_desc.sql.j2 b/tests/queries/0_stateless/02126_dist_desc.sql.j2 index c0e1b5f8abd2..93f23dc9eb14 100644 --- a/tests/queries/0_stateless/02126_dist_desc.sql.j2 +++ b/tests/queries/0_stateless/02126_dist_desc.sql.j2 @@ -10,7 +10,7 @@ select * from remote('{{host}}', {{args}}) format Null; {% endfor -%} system flush logs query_log; -select anyIf(query, is_initial_query), groupArrayIf(query, query_kind = 'Describe' and not is_initial_query) from system.query_log +select anyIf(query, initial_query_id == query_id), groupArrayIf(query, query_kind = 'Describe' and initial_query_id != query_id) from system.query_log where event_date >= yesterday() AND event_time >= now() - 600 AND type = 'QueryFinish' and diff --git a/tests/queries/0_stateless/02221_system_zookeeper_unrestricted.reference b/tests/queries/0_stateless/02221_system_zookeeper_unrestricted.reference index bd379621b73d..2a6bc4c98833 100644 --- a/tests/queries/0_stateless/02221_system_zookeeper_unrestricted.reference +++ b/tests/queries/0_stateless/02221_system_zookeeper_unrestricted.reference @@ -20,6 +20,8 @@ creator_info creator_info deduplication_hashes deduplication_hashes +exports +exports failed_parts failed_parts flags diff --git a/tests/queries/0_stateless/02221_system_zookeeper_unrestricted_like.reference b/tests/queries/0_stateless/02221_system_zookeeper_unrestricted_like.reference index b1c76b04526e..db94343ec003 100644 --- a/tests/queries/0_stateless/02221_system_zookeeper_unrestricted_like.reference +++ b/tests/queries/0_stateless/02221_system_zookeeper_unrestricted_like.reference @@ -9,6 +9,7 @@ columns columns creator_info deduplication_hashes +exports failed_parts flags host @@ -51,6 +52,7 @@ columns columns creator_info deduplication_hashes +exports failed_parts flags host diff --git a/tests/queries/0_stateless/03377_object_storage_list_objects_cache.reference b/tests/queries/0_stateless/03377_object_storage_list_objects_cache.reference new file mode 100644 index 000000000000..76535ad25106 --- /dev/null +++ b/tests/queries/0_stateless/03377_object_storage_list_objects_cache.reference @@ -0,0 +1,103 @@ +-- { echoOn } + +-- The cached key should be `dir_`, and that includes all three files: 1, 2 and 3. Cache should return all three, but ClickHouse should filter out the third. +SELECT _path, * FROM s3(s3_conn, filename='dir_a/dir_b/t_03377_sample_{1..2}.parquet') order by id SETTINGS use_object_storage_list_objects_cache=1; +test/dir_a/dir_b/t_03377_sample_1.parquet 1 +test/dir_a/dir_b/t_03377_sample_2.parquet 2 +-- Make sure the filtering did not interfere with the cached values +SELECT _path, * FROM s3(s3_conn, filename='dir_a/dir_b/t_03377_sample_*.parquet') order by id SETTINGS use_object_storage_list_objects_cache=1; +test/dir_a/dir_b/t_03377_sample_1.parquet 1 +test/dir_a/dir_b/t_03377_sample_2.parquet 2 +test/dir_a/dir_b/t_03377_sample_3.parquet 3 +SYSTEM FLUSH LOGS; +SELECT ProfileEvents['ObjectStorageListObjectsCacheMisses'] > 0 as miss +FROM system.query_log +where log_comment = 'cold_list_cache' +AND type = 'QueryFinish' +ORDER BY event_time desc +LIMIT 1; +1 +SELECT ProfileEvents['ObjectStorageListObjectsCacheHits'] > 0 as hit +FROM system.query_log +where log_comment = 'warm_list_exact_cache' +AND type = 'QueryFinish' +ORDER BY event_time desc +LIMIT 1; +1 +SELECT ProfileEvents['ObjectStorageListObjectsCacheExactMatchHits'] > 0 as hit +FROM system.query_log +where log_comment = 'warm_list_exact_cache' +AND type = 'QueryFinish' +ORDER BY event_time desc +LIMIT 1; +1 +SELECT ProfileEvents['ObjectStorageListObjectsCachePrefixMatchHits'] > 0 as prefix_match_hit +FROM system.query_log +where log_comment = 'warm_list_exact_cache' +AND type = 'QueryFinish' +ORDER BY event_time desc +LIMIT 1; +0 +SELECT ProfileEvents['ObjectStorageListObjectsCacheHits'] > 0 as hit +FROM system.query_log +where log_comment = 'warm_list_prefix_match_cache' +AND type = 'QueryFinish' +ORDER BY event_time desc +LIMIT 1; +1 +SELECT ProfileEvents['ObjectStorageListObjectsCacheExactMatchHits'] > 0 as exact_match_hit +FROM system.query_log +where log_comment = 'warm_list_prefix_match_cache' +AND type = 'QueryFinish' +ORDER BY event_time desc +LIMIT 1; +0 +SELECT ProfileEvents['ObjectStorageListObjectsCachePrefixMatchHits'] > 0 as prefix_match_hit +FROM system.query_log +where log_comment = 'warm_list_prefix_match_cache' +AND type = 'QueryFinish' +ORDER BY event_time desc +LIMIT 1; +1 +SELECT ProfileEvents['ObjectStorageListObjectsCacheHits'] > 0 as hit +FROM system.query_log +where log_comment = 'even_shorter_prefix' +AND type = 'QueryFinish' +ORDER BY event_time desc +LIMIT 1; +0 +SELECT ProfileEvents['ObjectStorageListObjectsCacheMisses'] > 0 as miss +FROM system.query_log +where log_comment = 'even_shorter_prefix' +AND type = 'QueryFinish' +ORDER BY event_time desc +LIMIT 1; +1 +SELECT ProfileEvents['ObjectStorageListObjectsCacheHits'] > 0 as hit +FROM system.query_log +where log_comment = 'still_exact_match_after_shorter_prefix' +AND type = 'QueryFinish' +ORDER BY event_time desc +LIMIT 1; +1 +SELECT ProfileEvents['ObjectStorageListObjectsCacheExactMatchHits'] > 0 as exact_match_hit +FROM system.query_log +where log_comment = 'still_exact_match_after_shorter_prefix' +AND type = 'QueryFinish' +ORDER BY event_time desc +LIMIT 1; +1 +SELECT ProfileEvents['ObjectStorageListObjectsCacheHits'] > 0 as hit +FROM system.query_log +where log_comment = 'after_drop' +AND type = 'QueryFinish' +ORDER BY event_time desc +LIMIT 1; +0 +SELECT ProfileEvents['ObjectStorageListObjectsCacheMisses'] > 0 as miss +FROM system.query_log +where log_comment = 'after_drop' +AND type = 'QueryFinish' +ORDER BY event_time desc +LIMIT 1; +1 diff --git a/tests/queries/0_stateless/03377_object_storage_list_objects_cache.sql b/tests/queries/0_stateless/03377_object_storage_list_objects_cache.sql new file mode 100644 index 000000000000..9638faa88d23 --- /dev/null +++ b/tests/queries/0_stateless/03377_object_storage_list_objects_cache.sql @@ -0,0 +1,115 @@ +-- Tags: no-parallel, no-fasttest + +SYSTEM DROP OBJECT STORAGE LIST OBJECTS CACHE; + +INSERT INTO TABLE FUNCTION s3(s3_conn, filename='dir_a/dir_b/t_03377_sample_{_partition_id}.parquet', format='Parquet', structure='id UInt64') PARTITION BY id SETTINGS s3_truncate_on_insert=1 VALUES (1), (2), (3); + +SELECT * FROM s3(s3_conn, filename='dir_**.parquet') Format Null SETTINGS use_object_storage_list_objects_cache=1, log_comment='cold_list_cache'; +SELECT * FROM s3(s3_conn, filename='dir_**.parquet') Format Null SETTINGS use_object_storage_list_objects_cache=1, log_comment='warm_list_exact_cache'; +SELECT * FROM s3(s3_conn, filename='dir_a/dir_b**.parquet') Format Null SETTINGS use_object_storage_list_objects_cache=1, log_comment='warm_list_prefix_match_cache'; +SELECT * FROM s3(s3_conn, filename='dirr_**.parquet') Format Null SETTINGS use_object_storage_list_objects_cache=1, log_comment='warm_list_cache_miss'; -- { serverError CANNOT_EXTRACT_TABLE_STRUCTURE } +SELECT * FROM s3(s3_conn, filename='d**.parquet') Format Null SETTINGS use_object_storage_list_objects_cache=1, log_comment='even_shorter_prefix'; +SELECT * FROM s3(s3_conn, filename='dir_**.parquet') Format Null SETTINGS use_object_storage_list_objects_cache=1, log_comment='still_exact_match_after_shorter_prefix'; +SYSTEM DROP OBJECT STORAGE LIST OBJECTS CACHE; +SELECT * FROM s3(s3_conn, filename='dir_**.parquet') Format Null SETTINGS use_object_storage_list_objects_cache=1, log_comment='after_drop'; + +-- { echoOn } + +-- The cached key should be `dir_`, and that includes all three files: 1, 2 and 3. Cache should return all three, but ClickHouse should filter out the third. +SELECT _path, * FROM s3(s3_conn, filename='dir_a/dir_b/t_03377_sample_{1..2}.parquet') order by id SETTINGS use_object_storage_list_objects_cache=1; + +-- Make sure the filtering did not interfere with the cached values +SELECT _path, * FROM s3(s3_conn, filename='dir_a/dir_b/t_03377_sample_*.parquet') order by id SETTINGS use_object_storage_list_objects_cache=1; + +SYSTEM FLUSH LOGS; + +SELECT ProfileEvents['ObjectStorageListObjectsCacheMisses'] > 0 as miss +FROM system.query_log +where log_comment = 'cold_list_cache' +AND type = 'QueryFinish' +ORDER BY event_time desc +LIMIT 1; + +SELECT ProfileEvents['ObjectStorageListObjectsCacheHits'] > 0 as hit +FROM system.query_log +where log_comment = 'warm_list_exact_cache' +AND type = 'QueryFinish' +ORDER BY event_time desc +LIMIT 1; + +SELECT ProfileEvents['ObjectStorageListObjectsCacheExactMatchHits'] > 0 as hit +FROM system.query_log +where log_comment = 'warm_list_exact_cache' +AND type = 'QueryFinish' +ORDER BY event_time desc +LIMIT 1; + +SELECT ProfileEvents['ObjectStorageListObjectsCachePrefixMatchHits'] > 0 as prefix_match_hit +FROM system.query_log +where log_comment = 'warm_list_exact_cache' +AND type = 'QueryFinish' +ORDER BY event_time desc +LIMIT 1; + +SELECT ProfileEvents['ObjectStorageListObjectsCacheHits'] > 0 as hit +FROM system.query_log +where log_comment = 'warm_list_prefix_match_cache' +AND type = 'QueryFinish' +ORDER BY event_time desc +LIMIT 1; + +SELECT ProfileEvents['ObjectStorageListObjectsCacheExactMatchHits'] > 0 as exact_match_hit +FROM system.query_log +where log_comment = 'warm_list_prefix_match_cache' +AND type = 'QueryFinish' +ORDER BY event_time desc +LIMIT 1; + +SELECT ProfileEvents['ObjectStorageListObjectsCachePrefixMatchHits'] > 0 as prefix_match_hit +FROM system.query_log +where log_comment = 'warm_list_prefix_match_cache' +AND type = 'QueryFinish' +ORDER BY event_time desc +LIMIT 1; + +SELECT ProfileEvents['ObjectStorageListObjectsCacheHits'] > 0 as hit +FROM system.query_log +where log_comment = 'even_shorter_prefix' +AND type = 'QueryFinish' +ORDER BY event_time desc +LIMIT 1; + +SELECT ProfileEvents['ObjectStorageListObjectsCacheMisses'] > 0 as miss +FROM system.query_log +where log_comment = 'even_shorter_prefix' +AND type = 'QueryFinish' +ORDER BY event_time desc +LIMIT 1; + +SELECT ProfileEvents['ObjectStorageListObjectsCacheHits'] > 0 as hit +FROM system.query_log +where log_comment = 'still_exact_match_after_shorter_prefix' +AND type = 'QueryFinish' +ORDER BY event_time desc +LIMIT 1; + +SELECT ProfileEvents['ObjectStorageListObjectsCacheExactMatchHits'] > 0 as exact_match_hit +FROM system.query_log +where log_comment = 'still_exact_match_after_shorter_prefix' +AND type = 'QueryFinish' +ORDER BY event_time desc +LIMIT 1; + +SELECT ProfileEvents['ObjectStorageListObjectsCacheHits'] > 0 as hit +FROM system.query_log +where log_comment = 'after_drop' +AND type = 'QueryFinish' +ORDER BY event_time desc +LIMIT 1; + +SELECT ProfileEvents['ObjectStorageListObjectsCacheMisses'] > 0 as miss +FROM system.query_log +where log_comment = 'after_drop' +AND type = 'QueryFinish' +ORDER BY event_time desc +LIMIT 1; diff --git a/tests/queries/0_stateless/03413_experimental_settings_cannot_be_enabled_by_default.sql b/tests/queries/0_stateless/03413_experimental_settings_cannot_be_enabled_by_default.sql index 5e9a854e3fa3..65a0e951da9e 100644 --- a/tests/queries/0_stateless/03413_experimental_settings_cannot_be_enabled_by_default.sql +++ b/tests/queries/0_stateless/03413_experimental_settings_cannot_be_enabled_by_default.sql @@ -4,5 +4,11 @@ -- However, some settings in the experimental tier are meant to control another experimental feature, and then they can be enabled as long as the feature itself is disabled. -- These are in the exceptions list inside NOT IN. -SELECT name, value FROM system.settings WHERE tier = 'Experimental' AND type = 'Bool' AND value != '0' AND name NOT IN ('throw_on_unsupported_query_inside_transaction', 'time_series_prefer_recent_samples_table'); +SELECT name, value FROM system.settings WHERE tier = 'Experimental' AND type = 'Bool' AND value != '0' AND name NOT IN ( + 'throw_on_unsupported_query_inside_transaction', + 'time_series_prefer_recent_samples_table', + 'allow_experimental_export_merge_tree_part', +-- turned ON for Altinity Antalya builds specifically + 'allow_experimental_iceberg_read_optimization' +); SELECT name, value FROM system.merge_tree_settings WHERE tier = 'Experimental' AND type = 'Bool' AND value != '0' AND name NOT IN ('remove_rolled_back_parts_immediately'); diff --git a/tests/queries/0_stateless/03550_analyzer_remote_view_columns.sql b/tests/queries/0_stateless/03550_analyzer_remote_view_columns.sql index 833106925266..820a7cd95f2b 100644 --- a/tests/queries/0_stateless/03550_analyzer_remote_view_columns.sql +++ b/tests/queries/0_stateless/03550_analyzer_remote_view_columns.sql @@ -39,4 +39,4 @@ WHERE event_date >= yesterday() AND event_time >= now() - 600 AND AND log_comment = 'THIS IS A COMMENT TO MARK THE INITIAL QUERY' LIMIT 1) AND type = 'QueryFinish' - AND NOT is_initial_query; + AND query_id != initial_query_id; diff --git a/tests/queries/0_stateless/03572_export_merge_tree_part_basic.reference b/tests/queries/0_stateless/03572_export_merge_tree_part_basic.reference new file mode 100644 index 000000000000..8b08677b5d48 --- /dev/null +++ b/tests/queries/0_stateless/03572_export_merge_tree_part_basic.reference @@ -0,0 +1,35 @@ +---- Export 1: Export 2020_1_1_0 and 2021_2_2_0 +---- Export 2: Export 2022_3_3_0 and 2023_4_4_0 to wildcard table +---- Export 3: Export 2020_1_1_0 and 2021_2_2_0 to wildcard table with partition expression with function +---- Export 4: Export the same part again, it should be idempotent +---- Export 5: Export the same part again to wildcard, it should be idempotent +---- Verify Export 1: Both data parts should appear +1 2020 +2 2020 +3 2020 +4 2021 +---- Verify Export 4: Export the same part again, it should be idempotent +1 2020 +2 2020 +3 2020 +4 2021 +---- Verify: Data in roundtrip MergeTree table (should match s3_table) +1 2020 +2 2020 +3 2020 +4 2021 +---- Verify Export 2: Both data parts should appear (2022_3_3_0 and 2023_4_4_0) +5 2022 +6 2022 +7 2023 +8 2023 +---- Verify Export 5: Export the same part again, it should be idempotent +5 2022 +6 2022 +7 2023 +8 2023 +---- Verify Export 3: Both data parts should appear +1 2020 +2 2020 +3 2020 +4 2021 diff --git a/tests/queries/0_stateless/03572_export_merge_tree_part_basic.sh b/tests/queries/0_stateless/03572_export_merge_tree_part_basic.sh new file mode 100755 index 000000000000..fc5df9b541da --- /dev/null +++ b/tests/queries/0_stateless/03572_export_merge_tree_part_basic.sh @@ -0,0 +1,85 @@ +#!/usr/bin/env bash +# Tags: no-fasttest +# Tag no-fasttest: requires s3 storage + +CURDIR=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd) +# shellcheck source=../shell_config.sh +. "$CURDIR"/../shell_config.sh + +mt_table="mt_table_${RANDOM}" +mt_table_partition_expression_with_function="mt_table_partition_expression_with_function_${RANDOM}" +s3_table="s3_table_${RANDOM}" +s3_table_wildcard="s3_table_wildcard_${RANDOM}" +s3_table_wildcard_partition_expression_with_function="s3_table_wildcard_partition_expression_with_function_${RANDOM}" +mt_table_roundtrip="mt_table_roundtrip_${RANDOM}" + +query() { + $CLICKHOUSE_CLIENT --query "$1" +} + +query "DROP TABLE IF EXISTS $mt_table, $s3_table, $mt_table_roundtrip, $s3_table_wildcard, $s3_table_wildcard_partition_expression_with_function, $mt_table_partition_expression_with_function" + +# Create all tables +query "CREATE TABLE $mt_table (id UInt64, year UInt16) ENGINE = MergeTree() PARTITION BY year ORDER BY tuple()" +query "CREATE TABLE $s3_table (id UInt64, year UInt16) ENGINE = S3(s3_conn, filename='$s3_table', format=Parquet, partition_strategy='hive') PARTITION BY year" +query "CREATE TABLE $s3_table_wildcard (id UInt64, year UInt16) ENGINE = S3(s3_conn, filename='$s3_table_wildcard/{_partition_id}/{_file}.parquet', format=Parquet, partition_strategy='wildcard') PARTITION BY year" +query "CREATE TABLE $mt_table_partition_expression_with_function (id UInt64, year UInt16) ENGINE = MergeTree() PARTITION BY toString(year) ORDER BY tuple()" +query "CREATE TABLE $s3_table_wildcard_partition_expression_with_function (id UInt64, year UInt16) ENGINE = S3(s3_conn, filename='$s3_table_wildcard_partition_expression_with_function/{_partition_id}/{_file}.parquet', format=Parquet, partition_strategy='wildcard') PARTITION BY toString(year)" + +# Insert all data +query "INSERT INTO $mt_table VALUES (1, 2020), (2, 2020), (3, 2020), (4, 2021), (5, 2022), (6, 2022), (7, 2023), (8, 2023), (9, 2024), (10, 2024), (11, 2025), (12, 2025)" +query "INSERT INTO $mt_table_partition_expression_with_function VALUES (1, 2020), (2, 2020), (3, 2020), (4, 2021)" + +# ============================================================================ +# ALL EXPORTS HAPPEN HERE +# ============================================================================ + +echo "---- Export 1: Export 2020_1_1_0 and 2021_2_2_0" +query "ALTER TABLE $mt_table EXPORT PART '2020_1_1_0' TO TABLE $s3_table SETTINGS allow_experimental_export_merge_tree_part = 1" +query "ALTER TABLE $mt_table EXPORT PART '2021_2_2_0' TO TABLE $s3_table SETTINGS allow_experimental_export_merge_tree_part = 1" + +echo "---- Export 2: Export 2022_3_3_0 and 2023_4_4_0 to wildcard table" +query "ALTER TABLE $mt_table EXPORT PART '2022_3_3_0' TO TABLE $s3_table_wildcard SETTINGS allow_experimental_export_merge_tree_part = 1" +query "ALTER TABLE $mt_table EXPORT PART '2023_4_4_0' TO TABLE $s3_table_wildcard SETTINGS allow_experimental_export_merge_tree_part = 1" + +echo "---- Export 3: Export 2020_1_1_0 and 2021_2_2_0 to wildcard table with partition expression with function" +query "ALTER TABLE $mt_table_partition_expression_with_function EXPORT PART 'cb217c742dc7d143b61583011996a160_1_1_0' TO TABLE $s3_table_wildcard_partition_expression_with_function SETTINGS allow_experimental_export_merge_tree_part = 1" +query "ALTER TABLE $mt_table_partition_expression_with_function EXPORT PART '3be6d49ecf9749a383964bc6fab22d10_2_2_0' TO TABLE $s3_table_wildcard_partition_expression_with_function SETTINGS allow_experimental_export_merge_tree_part = 1" + +# below exports are using parts that were exported in export 1 and export 2, so we need to wait for them to complete +sleep 5 + +echo "---- Export 4: Export the same part again, it should be idempotent" +query "ALTER TABLE $mt_table EXPORT PART '2020_1_1_0' TO TABLE $s3_table SETTINGS allow_experimental_export_merge_tree_part = 1" + +echo "---- Export 5: Export the same part again to wildcard, it should be idempotent" +query "ALTER TABLE $mt_table EXPORT PART '2022_3_3_0' TO TABLE $s3_table_wildcard SETTINGS allow_experimental_export_merge_tree_part = 1" + +# ONE BIG SLEEP after all exports +sleep 15 + +# ============================================================================ +# ALL SELECTS/VERIFICATIONS HAPPEN HERE +# ============================================================================ + +echo "---- Verify Export 1: Both data parts should appear" +query "SELECT * FROM $s3_table ORDER BY id" + +echo "---- Verify Export 4: Export the same part again, it should be idempotent" +query "SELECT * FROM $s3_table ORDER BY id" + +query "CREATE TABLE $mt_table_roundtrip ENGINE = MergeTree() PARTITION BY year ORDER BY tuple() AS SELECT * FROM $s3_table" + +echo "---- Verify: Data in roundtrip MergeTree table (should match s3_table)" +query "SELECT * FROM $mt_table_roundtrip ORDER BY id" + +echo "---- Verify Export 2: Both data parts should appear (2022_3_3_0 and 2023_4_4_0)" +query "SELECT * FROM s3(s3_conn, filename='$s3_table_wildcard/**.parquet') ORDER BY id" + +echo "---- Verify Export 5: Export the same part again, it should be idempotent" +query "SELECT * FROM s3(s3_conn, filename='$s3_table_wildcard/**.parquet') ORDER BY id" + +echo "---- Verify Export 3: Both data parts should appear" +query "SELECT * FROM s3(s3_conn, filename='$s3_table_wildcard_partition_expression_with_function/**.parquet') ORDER BY id" + +query "DROP TABLE IF EXISTS $mt_table, $s3_table, $mt_table_roundtrip, $s3_table_wildcard, $s3_table_wildcard_partition_expression_with_function, $mt_table_partition_expression_with_function" diff --git a/tests/queries/0_stateless/03572_export_merge_tree_part_limits_and_table_functions.reference b/tests/queries/0_stateless/03572_export_merge_tree_part_limits_and_table_functions.reference new file mode 100644 index 000000000000..b7f1f4411bf6 --- /dev/null +++ b/tests/queries/0_stateless/03572_export_merge_tree_part_limits_and_table_functions.reference @@ -0,0 +1,20 @@ +---- Test max_bytes and max_rows per file +---- Table function with schema inheritance (no schema specified) +---- Table function with explicit compatible schema +Waiting for exports to complete (timeout: 60s)... +All exports completed. +---- Count files in big_destination_max_bytes, should be 5 (4 parquet, 1 commit) +5 +---- Count rows in big_table and big_destination_max_bytes +4194304 +4194304 +---- Count files in big_destination_max_rows, should be 5 (4 parquet, 1 commit) +5 +---- Count rows in big_table and big_destination_max_rows +4194304 +4194304 +---- Data should be exported with inherited schema +100 test1 2022 +101 test2 2022 +---- Data should be exported with explicit schema +102 test3 2023 diff --git a/tests/queries/0_stateless/03572_export_merge_tree_part_limits_and_table_functions.sh b/tests/queries/0_stateless/03572_export_merge_tree_part_limits_and_table_functions.sh new file mode 100755 index 000000000000..dff7332662d0 --- /dev/null +++ b/tests/queries/0_stateless/03572_export_merge_tree_part_limits_and_table_functions.sh @@ -0,0 +1,132 @@ +#!/usr/bin/env bash +# Tags: no-fasttest +# Tag no-fasttest: requires s3 storage + +CURDIR=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd) +# shellcheck source=../shell_config.sh +. "$CURDIR"/../shell_config.sh + +big_table="big_table_${RANDOM}" +big_destination_max_bytes="big_destination_max_bytes_${RANDOM}" +big_destination_max_rows="big_destination_max_rows_${RANDOM}" +tf_schema_inherit="tf_schema_inherit_${RANDOM}" +tf_schema_explicit="tf_schema_explicit_${RANDOM}" +mt_table_tf="mt_table_tf_${RANDOM}" + +query() { + local query_text="$1" + local query_id="$2" + + if [ -n "$query_id" ]; then + $CLICKHOUSE_CLIENT --query_id="$query_id" --query "$query_text" + else + $CLICKHOUSE_CLIENT --query "$query_text" + fi +} + +query "DROP TABLE IF EXISTS $big_table, $big_destination_max_bytes, $big_destination_max_rows, $mt_table_tf" + +echo "---- Test max_bytes and max_rows per file" + +# Create all tables +query "CREATE TABLE $big_table (id UInt64, data String, year UInt16) Engine=MergeTree() order by id partition by year" +query "CREATE TABLE $big_destination_max_bytes(id UInt64, data String, year UInt16) engine=S3(s3_conn, filename='$big_destination_max_bytes', partition_strategy='hive', format=Parquet) partition by year" +query "CREATE TABLE $big_destination_max_rows(id UInt64, data String, year UInt16) engine=S3(s3_conn, filename='$big_destination_max_rows', partition_strategy='hive', format=Parquet) partition by year" +query "CREATE TABLE $mt_table_tf (id UInt64, value String, year UInt16) ENGINE = MergeTree() PARTITION BY year ORDER BY tuple()" + +# Insert all data +# 4194304 is a number that came up during multiple iterations, it does not really mean anything (aside from the fact that the below numbers depend on it) +query "INSERT INTO $big_table SELECT number AS id, repeat('x', 100) AS data, 2025 AS year FROM numbers(4194304)" +query "INSERT INTO $big_table SELECT number AS id, repeat('x', 100) AS data, 2026 AS year FROM numbers(4194304)" +query "INSERT INTO $mt_table_tf VALUES (100, 'test1', 2022), (101, 'test2', 2022), (102, 'test3', 2023)" + +# make sure we have only one part +query "OPTIMIZE TABLE $big_table FINAL" + +# Get part names +big_part_max_bytes=$(query "SELECT name FROM system.parts WHERE database = currentDatabase() AND table = '$big_table' AND partition_id = '2025' AND active = 1 ORDER BY name LIMIT 1" | tr -d '\n') +big_part_max_rows=$(query "SELECT name FROM system.parts WHERE database = currentDatabase() AND table = '$big_table' AND partition_id = '2026' AND active = 1 ORDER BY name LIMIT 1" | tr -d '\n') + +# ============================================================================ +# ALL EXPORTS HAPPEN HERE +# ============================================================================ + +# Generate unique query_ids for each export to track them in part_log +export_query_id_1="export_${RANDOM}_1" +export_query_id_2="export_${RANDOM}_2" +export_query_id_3="export_${RANDOM}_3" +export_query_id_4="export_${RANDOM}_4" + +# this should generate ~4 files +query "ALTER TABLE $big_table EXPORT PART '$big_part_max_bytes' TO TABLE $big_destination_max_bytes SETTINGS allow_experimental_export_merge_tree_part = 1, export_merge_tree_part_max_bytes_per_file=3500000, output_format_parquet_row_group_size_bytes=1000000" "$export_query_id_1" +# export_merge_tree_part_max_rows_per_file = 1048576 (which is 4194304/4) to generate 4 files +query "ALTER TABLE $big_table EXPORT PART '$big_part_max_rows' TO TABLE $big_destination_max_rows SETTINGS allow_experimental_export_merge_tree_part = 1, export_merge_tree_part_max_rows_per_file=1048576" "$export_query_id_2" + +echo "---- Table function with schema inheritance (no schema specified)" +query "ALTER TABLE $mt_table_tf EXPORT PART '2022_1_1_0' TO TABLE FUNCTION s3(s3_conn, filename='$tf_schema_inherit', format='Parquet', partition_strategy='hive') PARTITION BY year SETTINGS allow_experimental_export_merge_tree_part = 1" "$export_query_id_3" + +echo "---- Table function with explicit compatible schema" +query "ALTER TABLE $mt_table_tf EXPORT PART '2023_2_2_0' TO TABLE FUNCTION s3(s3_conn, filename='$tf_schema_explicit', format='Parquet', structure='id UInt64, value String, year UInt16', partition_strategy='hive') PARTITION BY year SETTINGS allow_experimental_export_merge_tree_part = 1" "$export_query_id_4" + +# Wait for all exports to complete +wait_for_exports() { + local timeout=${1:-60} + local poll_interval=${2:-0.5} + local start_time=$(date +%s) + local elapsed=0 + + echo "Waiting for exports to complete (timeout: ${timeout}s)..." + + while [ $elapsed -lt $timeout ]; do + # Flush logs to ensure part_log entries are visible + query "SYSTEM FLUSH LOGS" > /dev/null 2>&1 || true + + # Wait for part_log entries - these are written synchronously when export completes + # Check if all expected exports have corresponding part_log entries by query_id + local completed_count=$(query "SELECT count() FROM system.part_log WHERE event_type = 'ExportPart' AND query_id IN ('$export_query_id_1', '$export_query_id_2', '$export_query_id_3', '$export_query_id_4')" | tr -d '\n') + + if [ "$completed_count" = "4" ]; then + echo "All exports completed." + return 0 + fi + + sleep $poll_interval + elapsed=$(($(date +%s) - start_time)) + done + + echo "Timeout waiting for exports to complete after ${timeout}s" + query "SYSTEM FLUSH LOGS" > /dev/null 2>&1 || true + echo "Completed exports in part_log:" + query "SELECT query_id, table, part_name, event_time FROM system.part_log WHERE event_type = 'ExportPart' AND query_id IN ('$export_query_id_1', '$export_query_id_2', '$export_query_id_3', '$export_query_id_4')" + echo "Remaining exports in system.exports:" + query "SELECT source_table, part_name, elapsed, rows_read, total_rows_to_read FROM system.exports WHERE ((source_table = '$big_table' AND part_name IN ('$big_part_max_bytes', '$big_part_max_rows')) OR (source_table = '$mt_table_tf' AND part_name IN ('2022_1_1_0', '2023_2_2_0')))" + return 1 +} + +wait_for_exports 60 + +# ============================================================================ +# ALL SELECTS/VERIFICATIONS HAPPEN HERE +# ============================================================================ + +echo "---- Count files in big_destination_max_bytes, should be 5 (4 parquet, 1 commit)" +query "SELECT count(_file) FROM s3(s3_conn, filename='$big_destination_max_bytes/**', format='One')" + +echo "---- Count rows in big_table and big_destination_max_bytes" +query "SELECT COUNT() from $big_table WHERE year = 2025" +query "SELECT COUNT() from $big_destination_max_bytes" + +echo "---- Count files in big_destination_max_rows, should be 5 (4 parquet, 1 commit)" +query "SELECT count(_file) FROM s3(s3_conn, filename='$big_destination_max_rows/**', format='One')" + +echo "---- Count rows in big_table and big_destination_max_rows" +query "SELECT COUNT() from $big_table WHERE year = 2026" +query "SELECT COUNT() from $big_destination_max_rows" + +echo "---- Data should be exported with inherited schema" +query "SELECT * FROM s3(s3_conn, filename='$tf_schema_inherit/**.parquet') ORDER BY id" + +echo "---- Data should be exported with explicit schema" +query "SELECT * FROM s3(s3_conn, filename='$tf_schema_explicit/**.parquet') ORDER BY id" + +query "DROP TABLE IF EXISTS $big_table, $big_destination_max_bytes, $big_destination_max_rows, $mt_table_tf" diff --git a/tests/queries/0_stateless/03572_export_merge_tree_part_special_columns.reference b/tests/queries/0_stateless/03572_export_merge_tree_part_special_columns.reference new file mode 100644 index 000000000000..14bcbb452591 --- /dev/null +++ b/tests/queries/0_stateless/03572_export_merge_tree_part_special_columns.reference @@ -0,0 +1,39 @@ +---- Test ALIAS columns export +---- Test MATERIALIZED columns export +---- Test EPHEMERAL column (not stored, ignored during export) +---- Test Mixed ALIAS, MATERIALIZED, and EPHEMERAL in same table +---- Test Complex Expressions in computed columns +---- Test Export to Table Function with mixed columns +---- Verify ALIAS column data in source table (arr_1 computed from arr[1]) +1 [1,2,3] 1 +1 [10,20,30] 10 +---- Verify ALIAS column data exported to S3 (should match source) +1 [1,2,3] 1 +1 [10,20,30] 10 +---- Verify MATERIALIZED column data in source table (arr_1 computed from arr[1]) +1 [1,2,3] 1 +1 [10,20,30] 10 +---- Verify MATERIALIZED column data exported to S3 (should match source) +1 [1,2,3] 1 +1 [10,20,30] 10 +---- Verify data in source +1 ALICE +1 BOB +---- Verify exported data +1 ALICE +1 BOB +---- Verify mixed columns in source table +1 5 10 15 TEST +1 10 20 30 PROD +2 15 30 45 DEV +---- Verify mixed columns exported to S3 +1 5 10 15 TEST +1 10 20 30 PROD +---- Verify mixed columns exported to S3 +2 15 30 45 DEV +---- Verify complex expressions in source table +1 alice ALICE alice-1 +1 bob BOB bob-1 +---- Verify complex expressions exported to S3 (should match source) +1 alice ALICE alice-1 +1 bob BOB bob-1 diff --git a/tests/queries/0_stateless/03572_export_merge_tree_part_special_columns.sh b/tests/queries/0_stateless/03572_export_merge_tree_part_special_columns.sh new file mode 100755 index 000000000000..0164dd70c4e0 --- /dev/null +++ b/tests/queries/0_stateless/03572_export_merge_tree_part_special_columns.sh @@ -0,0 +1,154 @@ +#!/usr/bin/env bash +# Tags: no-fasttest +# Tag no-fasttest: requires s3 storage + +CURDIR=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd) +# shellcheck source=../shell_config.sh +. "$CURDIR"/../shell_config.sh + +mt_alias="mt_alias_${RANDOM}" +mt_materialized="mt_materialized_${RANDOM}" +s3_alias_export="s3_alias_export_${RANDOM}" +s3_materialized_export="s3_materialized_export_${RANDOM}" +mt_mixed="mt_mixed_${RANDOM}" +s3_mixed_export="s3_mixed_export_${RANDOM}" +mt_complex_expr="mt_complex_expr_${RANDOM}" +s3_complex_expr_export="s3_complex_expr_export_${RANDOM}" +mt_ephemeral="mt_ephemeral_${RANDOM}" +s3_ephemeral_export="s3_ephemeral_export_${RANDOM}" +s3_mixed_export_table_function="s3_mixed_export_table_function_${RANDOM}" + +query() { + $CLICKHOUSE_CLIENT --query "$1" +} + +query "DROP TABLE IF EXISTS $mt_alias, $mt_materialized, $s3_alias_export, $s3_materialized_export, $mt_mixed, $s3_mixed_export, $mt_complex_expr, $s3_complex_expr_export, $mt_ephemeral, $s3_ephemeral_export" + +# Create all tables +echo "---- Test ALIAS columns export" +query "CREATE TABLE $mt_alias (a UInt32, arr Array(UInt64), arr_1 UInt64 ALIAS arr[1]) ENGINE = MergeTree() PARTITION BY a ORDER BY (a, arr[1]) SETTINGS index_granularity = 1" +query "CREATE TABLE $s3_alias_export (a UInt32, arr Array(UInt64), arr_1 UInt64) ENGINE = S3(s3_conn, filename='$s3_alias_export', format=Parquet, partition_strategy='hive') PARTITION BY a" + +echo "---- Test MATERIALIZED columns export" +query "CREATE TABLE $mt_materialized (a UInt32, arr Array(UInt64), arr_1 UInt64 MATERIALIZED arr[1]) ENGINE = MergeTree() PARTITION BY a ORDER BY (a, arr_1) SETTINGS index_granularity = 1" +query "CREATE TABLE $s3_materialized_export (a UInt32, arr Array(UInt64), arr_1 UInt64) ENGINE = S3(s3_conn, filename='$s3_materialized_export', format=Parquet, partition_strategy='hive') PARTITION BY a" + +echo "---- Test EPHEMERAL column (not stored, ignored during export)" +query "CREATE TABLE $mt_ephemeral ( + id UInt32, + name_input String EPHEMERAL, + name_upper String DEFAULT upper(name_input) +) ENGINE = MergeTree() PARTITION BY id ORDER BY id SETTINGS index_granularity = 1" + +query "CREATE TABLE $s3_ephemeral_export ( + id UInt32, + name_upper String +) ENGINE = S3(s3_conn, filename='$s3_ephemeral_export', format=Parquet, partition_strategy='hive') PARTITION BY id" + +echo "---- Test Mixed ALIAS, MATERIALIZED, and EPHEMERAL in same table" +query "CREATE TABLE $mt_mixed ( + id UInt32, + value UInt32, + tag_input String EPHEMERAL, + doubled UInt64 ALIAS value * 2, + tripled UInt64 MATERIALIZED value * 3, + tag String DEFAULT upper(tag_input) +) ENGINE = MergeTree() PARTITION BY id ORDER BY id SETTINGS index_granularity = 1" + +query "CREATE TABLE $s3_mixed_export ( + id UInt32, + value UInt32, + doubled UInt64, + tripled UInt64, + tag String +) ENGINE = S3(s3_conn, filename='$s3_mixed_export', format=Parquet, partition_strategy='hive') PARTITION BY id" + +echo "---- Test Complex Expressions in computed columns" +query "CREATE TABLE $mt_complex_expr ( + id UInt32, + name String, + upper_name String ALIAS upper(name), + concat_result String MATERIALIZED concat(name, '-', toString(id)) +) ENGINE = MergeTree() PARTITION BY id ORDER BY id SETTINGS index_granularity = 1" + +query "CREATE TABLE $s3_complex_expr_export ( + id UInt32, + name String, + upper_name String, + concat_result String +) ENGINE = S3(s3_conn, filename='$s3_complex_expr_export', format=Parquet, partition_strategy='hive') PARTITION BY id" + +# Insert all data +query "INSERT INTO $mt_alias VALUES (1, [1, 2, 3]), (1, [10, 20, 30])" +query "INSERT INTO $mt_materialized VALUES (1, [1, 2, 3]), (1, [10, 20, 30])" +query "INSERT INTO $mt_ephemeral (id, name_input) VALUES (1, 'alice'), (1, 'bob')" +query "INSERT INTO $mt_mixed (id, value, tag_input) VALUES (1, 5, 'test'), (1, 10, 'prod')" +query "INSERT INTO $mt_mixed (id, value, tag_input) VALUES (2, 15, 'dev')" +query "INSERT INTO $mt_complex_expr (id, name) VALUES (1, 'alice'), (1, 'bob')" + +# Get all part names +alias_part=$(query "SELECT name FROM system.parts WHERE database = currentDatabase() AND table = '$mt_alias' AND partition_id = '1' AND active = 1 ORDER BY name LIMIT 1" | tr -d '\n') +materialized_part=$(query "SELECT name FROM system.parts WHERE database = currentDatabase() AND table = '$mt_materialized' AND partition_id = '1' AND active = 1 ORDER BY name LIMIT 1" | tr -d '\n') +ephemeral_part=$(query "SELECT name FROM system.parts WHERE database = currentDatabase() AND table = '$mt_ephemeral' AND partition_id = '1' AND active = 1 ORDER BY name LIMIT 1" | tr -d '\n') +mixed_part=$(query "SELECT name FROM system.parts WHERE database = currentDatabase() AND table = '$mt_mixed' AND partition_id = '1' AND active = 1 ORDER BY name LIMIT 1" | tr -d '\n') +mixed_part_2=$(query "SELECT name FROM system.parts WHERE database = currentDatabase() AND table = '$mt_mixed' AND partition_id = '2' AND active = 1 ORDER BY name LIMIT 1" | tr -d '\n') +complex_expr_part=$(query "SELECT name FROM system.parts WHERE database = currentDatabase() AND table = '$mt_complex_expr' AND partition_id = '1' AND active = 1 ORDER BY name LIMIT 1" | tr -d '\n') + +# ============================================================================ +# ALL EXPORTS HAPPEN HERE +# ============================================================================ + +query "ALTER TABLE $mt_alias EXPORT PART '$alias_part' TO TABLE $s3_alias_export SETTINGS allow_experimental_export_merge_tree_part = 1" + +query "ALTER TABLE $mt_materialized EXPORT PART '$materialized_part' TO TABLE $s3_materialized_export SETTINGS allow_experimental_export_merge_tree_part = 1" + +query "ALTER TABLE $mt_ephemeral EXPORT PART '$ephemeral_part' TO TABLE $s3_ephemeral_export SETTINGS allow_experimental_export_merge_tree_part = 1" + +query "ALTER TABLE $mt_mixed EXPORT PART '$mixed_part' TO TABLE $s3_mixed_export SETTINGS allow_experimental_export_merge_tree_part = 1" + +echo "---- Test Export to Table Function with mixed columns" +query "ALTER TABLE $mt_mixed EXPORT PART '$mixed_part_2' TO TABLE FUNCTION s3(s3_conn, filename='$s3_mixed_export_table_function', format=Parquet, partition_strategy='hive') PARTITION BY id SETTINGS allow_experimental_export_merge_tree_part = 1" + +query "ALTER TABLE $mt_complex_expr EXPORT PART '$complex_expr_part' TO TABLE $s3_complex_expr_export SETTINGS allow_experimental_export_merge_tree_part = 1" + +# ONE BIG SLEEP after all exports +sleep 20 + +# ============================================================================ +# ALL SELECTS/VERIFICATIONS HAPPEN HERE +# ============================================================================ + +echo "---- Verify ALIAS column data in source table (arr_1 computed from arr[1])" +query "SELECT a, arr, arr_1 FROM $mt_alias ORDER BY arr" + +echo "---- Verify ALIAS column data exported to S3 (should match source)" +query "SELECT a, arr, arr_1 FROM $s3_alias_export ORDER BY arr" + +echo "---- Verify MATERIALIZED column data in source table (arr_1 computed from arr[1])" +query "SELECT a, arr, arr_1 FROM $mt_materialized ORDER BY arr" + +echo "---- Verify MATERIALIZED column data exported to S3 (should match source)" +query "SELECT a, arr, arr_1 FROM $s3_materialized_export ORDER BY arr" + +echo "---- Verify data in source" +query "SELECT id, name_upper FROM $mt_ephemeral ORDER BY name_upper" + +echo "---- Verify exported data" +query "SELECT id, name_upper FROM $s3_ephemeral_export ORDER BY name_upper" + +echo "---- Verify mixed columns in source table" +query "SELECT id, value, doubled, tripled, tag FROM $mt_mixed ORDER BY value" + +echo "---- Verify mixed columns exported to S3" +query "SELECT id, value, doubled, tripled, tag FROM $s3_mixed_export ORDER BY value" + +echo "---- Verify mixed columns exported to S3" +query "SELECT * FROM s3(s3_conn, filename='$s3_mixed_export_table_function/**.parquet', format=Parquet) ORDER BY value" + +echo "---- Verify complex expressions in source table" +query "SELECT id, name, upper_name, concat_result FROM $mt_complex_expr ORDER BY name" + +echo "---- Verify complex expressions exported to S3 (should match source)" +query "SELECT id, name, upper_name, concat_result FROM $s3_complex_expr_export ORDER BY name" + +query "DROP TABLE IF EXISTS $mt_alias, $mt_materialized, $s3_alias_export, $s3_materialized_export, $mt_ephemeral, $s3_ephemeral_export, $mt_mixed, $s3_mixed_export, $mt_complex_expr, $s3_complex_expr_export" diff --git a/tests/queries/0_stateless/03572_export_merge_tree_part_to_object_storage_simple.reference b/tests/queries/0_stateless/03572_export_merge_tree_part_to_object_storage_simple.reference new file mode 100644 index 000000000000..e69de29bb2d1 diff --git a/tests/queries/0_stateless/03572_export_merge_tree_part_to_object_storage_simple.sql b/tests/queries/0_stateless/03572_export_merge_tree_part_to_object_storage_simple.sql new file mode 100644 index 000000000000..c59cebc45c52 --- /dev/null +++ b/tests/queries/0_stateless/03572_export_merge_tree_part_to_object_storage_simple.sql @@ -0,0 +1,83 @@ +-- Tags: no-parallel, no-fasttest + +DROP TABLE IF EXISTS 03572_mt_table, 03572_invalid_schema_table, 03572_ephemeral_mt_table, 03572_matching_ephemeral_s3_table, 03572_partition_type_mismatch_mt, 03572_partition_type_mismatch_s3, 03572_lossy_mt, 03572_lossy_s3, 03572_lossless_mt, 03572_lossless_s3, 03572_coarser_source_mt, 03572_finer_dest_s3; + +SET allow_experimental_export_merge_tree_part=1; + +CREATE TABLE 03572_mt_table (id UInt64, year UInt16) ENGINE = MergeTree() PARTITION BY year ORDER BY tuple(); + +INSERT INTO 03572_mt_table VALUES (1, 2020); + +-- Create a table partitioned by a column that is not part of the source partition key. The unified +-- plain-storage partition gate rejects it because the destination partition column is not covered by +-- the source partition key (schema compat follows INSERT SELECT positional semantics, so the column +-- shape matches and the partition-compatibility check is what fires). +CREATE TABLE 03572_invalid_schema_table (id UInt64, x UInt16) ENGINE = S3(s3_conn, filename='03572_invalid_schema_table', format='Parquet', partition_strategy='hive') PARTITION BY x; + +ALTER TABLE 03572_mt_table EXPORT PART '2020_1_1_0' TO TABLE 03572_invalid_schema_table +SETTINGS allow_experimental_export_merge_tree_part = 1; -- {serverError BAD_ARGUMENTS} + +DROP TABLE 03572_invalid_schema_table; + +-- The only partition strategy that supports exports is hive. Wildcard should throw +CREATE TABLE 03572_invalid_schema_table (id UInt64, year UInt16) ENGINE = S3(s3_conn, filename='03572_invalid_schema_table/{_partition_id}', format='Parquet', partition_strategy='wildcard') PARTITION BY (id, year); + +ALTER TABLE 03572_mt_table EXPORT PART '2020_1_1_0' TO TABLE 03572_invalid_schema_table; -- {serverError NOT_IMPLEMENTED} + +-- Not a table function, should throw +ALTER TABLE 03572_mt_table EXPORT PART '2020_1_1_0' TO TABLE FUNCTION extractKeyValuePairs('name:ronaldo'); -- {serverError UNKNOWN_FUNCTION} + +-- It is a table function, but the engine does not support exports/imports, should throw +ALTER TABLE 03572_mt_table EXPORT PART '2020_1_1_0' TO TABLE FUNCTION url('a.parquet'); -- {serverError NOT_IMPLEMENTED} + +-- Source-side ephemeral columns are not readable, so the destination must not declare a matching +-- ordinary column or the column count will not align under positional matching. +CREATE TABLE 03572_ephemeral_mt_table (id UInt64, year UInt16, name String EPHEMERAL) ENGINE = MergeTree() PARTITION BY year ORDER BY tuple(); + +CREATE TABLE 03572_matching_ephemeral_s3_table (id UInt64, year UInt16, name String) ENGINE = S3(s3_conn, filename='03572_matching_ephemeral_s3_table', format='Parquet', partition_strategy='hive') PARTITION BY year; + +INSERT INTO 03572_ephemeral_mt_table (id, year, name) VALUES (1, 2020, 'alice'); + +ALTER TABLE 03572_ephemeral_mt_table EXPORT PART '2020_1_1_0' TO TABLE 03572_matching_ephemeral_s3_table; -- {serverError NUMBER_OF_COLUMNS_DOESNT_MATCH} + +-- Partition columns follow the same lossy-cast gate as any other column (no special +-- exact-type guard). String -> UInt16 is a lossy cast, so with the default +-- export_merge_tree_part_allow_lossy_cast = 0 it is rejected synchronously. +CREATE TABLE 03572_partition_type_mismatch_mt (id UInt64, year String) ENGINE = MergeTree() PARTITION BY year ORDER BY tuple(); +CREATE TABLE 03572_partition_type_mismatch_s3 (id UInt64, year UInt16) ENGINE = S3(s3_conn, filename='03572_partition_type_mismatch_s3', format='Parquet', partition_strategy='hive') PARTITION BY year; + +ALTER TABLE 03572_partition_type_mismatch_mt EXPORT PART '2020_1_1_0' TO TABLE 03572_partition_type_mismatch_s3 +SETTINGS allow_experimental_export_merge_tree_part = 1; -- {serverError INCOMPATIBLE_COLUMNS} + +CREATE TABLE 03572_lossy_mt (id Int64, year UInt16) ENGINE = MergeTree() PARTITION BY year ORDER BY tuple(); +CREATE TABLE 03572_lossy_s3 (id Int32, year UInt16) ENGINE = S3(s3_conn, filename='03572_lossy_s3', format='Parquet', partition_strategy='hive') PARTITION BY year; + +ALTER TABLE 03572_lossy_mt EXPORT PART '2020_1_1_0' TO TABLE 03572_lossy_s3 +SETTINGS allow_experimental_export_merge_tree_part = 1; -- {serverError INCOMPATIBLE_COLUMNS} + +-- With the acknowledgment setting enabled, the lossy cast passes validation and reaches the +-- part lookup, which fails because the part does not exist. +ALTER TABLE 03572_lossy_mt EXPORT PART '2020_1_1_0' TO TABLE 03572_lossy_s3 +SETTINGS allow_experimental_export_merge_tree_part = 1, export_merge_tree_part_allow_lossy_cast = 1; -- {serverError NO_SUCH_DATA_PART} + +-- A lossless widening cast (Int32 -> Int64) passes validation without the setting and reaches +-- the part lookup, which fails because the part does not exist. +CREATE TABLE 03572_lossless_mt (id Int32, year UInt16) ENGINE = MergeTree() PARTITION BY year ORDER BY tuple(); +CREATE TABLE 03572_lossless_s3 (id Int64, year UInt16) ENGINE = S3(s3_conn, filename='03572_lossless_s3', format='Parquet', partition_strategy='hive') PARTITION BY year; + +ALTER TABLE 03572_lossless_mt EXPORT PART '2020_1_1_0' TO TABLE 03572_lossless_s3 +SETTINGS allow_experimental_export_merge_tree_part = 1; -- {serverError NO_SUCH_DATA_PART} + +-- Unified plain-storage partition gate: the destination partitioning must be single-valued within +-- each exported source part. The source is partitioned monthly (toYYYYMM(dt)) while the destination +-- is partitioned by the raw date, so a single source part holding two different days would map to two +-- destination partitions. The gate rejects it (the part exists, so the data-dependent check runs). +CREATE TABLE 03572_coarser_source_mt (id UInt64, dt Date) ENGINE = MergeTree() PARTITION BY toYYYYMM(dt) ORDER BY tuple(); +CREATE TABLE 03572_finer_dest_s3 (id UInt64, dt Date) ENGINE = S3(s3_conn, filename='03572_finer_dest_s3', format='Parquet', partition_strategy='hive') PARTITION BY dt; + +INSERT INTO 03572_coarser_source_mt VALUES (1, '2024-03-05'), (2, '2024-03-20'); + +ALTER TABLE 03572_coarser_source_mt EXPORT PART '202403_1_1_0' TO TABLE 03572_finer_dest_s3 +SETTINGS allow_experimental_export_merge_tree_part = 1; -- {serverError BAD_ARGUMENTS} + +DROP TABLE IF EXISTS 03572_mt_table, 03572_invalid_schema_table, 03572_ephemeral_mt_table, 03572_matching_ephemeral_s3_table, 03572_partition_type_mismatch_mt, 03572_partition_type_mismatch_s3, 03572_lossy_mt, 03572_lossy_s3, 03572_lossless_mt, 03572_lossless_s3, 03572_coarser_source_mt, 03572_finer_dest_s3; diff --git a/tests/queries/0_stateless/03572_export_part_hive_partition_subset.reference b/tests/queries/0_stateless/03572_export_part_hive_partition_subset.reference new file mode 100644 index 000000000000..9e404f989566 --- /dev/null +++ b/tests/queries/0_stateless/03572_export_part_hive_partition_subset.reference @@ -0,0 +1,9 @@ +---- Export each source part into the coarser destination +---- Destination should hold all rows +1 2020 US +2 2020 FR +3 2021 US +---- Round-trip back into a MergeTree table (should match the source) +1 2020 US +2 2020 FR +3 2021 US diff --git a/tests/queries/0_stateless/03572_export_part_hive_partition_subset.sh b/tests/queries/0_stateless/03572_export_part_hive_partition_subset.sh new file mode 100755 index 000000000000..cf9f43684001 --- /dev/null +++ b/tests/queries/0_stateless/03572_export_part_hive_partition_subset.sh @@ -0,0 +1,41 @@ +#!/usr/bin/env bash +# Tags: replica, no-parallel, no-replicated-database, no-fasttest + +CURDIR=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd) +# shellcheck source=../shell_config.sh +. "$CURDIR"/../shell_config.sh + +rmt_table="rmt_table_${RANDOM}" +s3_table="s3_table_${RANDOM}" +rmt_table_roundtrip="rmt_table_roundtrip_${RANDOM}" + +query() { + $CLICKHOUSE_CLIENT --query "$1" +} + +query "DROP TABLE IF EXISTS $rmt_table, $s3_table, $rmt_table_roundtrip" + +# The source partitions by (year, country); the destination partitions by year only - a coarser key +# that is covered by the source partition key. Every source part has a single year, so it maps to +# exactly one destination partition and the unified plain-storage gate accepts the export even though +# the partition keys are not identical (this was rejected before the unification). +query "CREATE TABLE $rmt_table (id UInt64, year UInt16, country String) ENGINE = ReplicatedMergeTree('/clickhouse/tables/{database}/$rmt_table', 'replica1') PARTITION BY (year, country) ORDER BY tuple()" +query "CREATE TABLE $s3_table (id UInt64, year UInt16, country String) ENGINE = S3(s3_conn, filename='$s3_table', format=Parquet, partition_strategy='hive') PARTITION BY year" + +query "INSERT INTO $rmt_table VALUES (1, 2020, 'US'), (2, 2020, 'FR'), (3, 2021, 'US')" + +echo "---- Export each source part into the coarser destination" +part_names=$(query "SELECT name FROM system.parts WHERE database = currentDatabase() AND table = '$rmt_table' AND active ORDER BY name") +for part in $part_names; do + query "ALTER TABLE $rmt_table EXPORT PART '$part' TO TABLE $s3_table SETTINGS allow_experimental_export_merge_tree_part = 1" +done + +echo "---- Destination should hold all rows" +query "SELECT * FROM $s3_table ORDER BY id" + +echo "---- Round-trip back into a MergeTree table (should match the source)" +query "CREATE TABLE $rmt_table_roundtrip (id UInt64, year UInt16, country String) ENGINE = ReplicatedMergeTree('/clickhouse/tables/{database}/$rmt_table_roundtrip', 'replica1') PARTITION BY (year, country) ORDER BY tuple()" +query "INSERT INTO $rmt_table_roundtrip SELECT * FROM $s3_table" +query "SELECT * FROM $rmt_table_roundtrip ORDER BY id" + +query "DROP TABLE IF EXISTS $rmt_table, $s3_table, $rmt_table_roundtrip" diff --git a/tests/queries/0_stateless/03572_export_replicated_merge_tree_part_to_object_storage.reference b/tests/queries/0_stateless/03572_export_replicated_merge_tree_part_to_object_storage.reference new file mode 100644 index 000000000000..07f1ec6376a6 --- /dev/null +++ b/tests/queries/0_stateless/03572_export_replicated_merge_tree_part_to_object_storage.reference @@ -0,0 +1,16 @@ +---- Get actual part names and export them +---- Both data parts should appear +1 2020 +2 2020 +3 2020 +4 2021 +---- Export the same part again, it should be idempotent +1 2020 +2 2020 +3 2020 +4 2021 +---- Data in roundtrip ReplicatedMergeTree table (should match s3_table) +1 2020 +2 2020 +3 2020 +4 2021 diff --git a/tests/queries/0_stateless/03572_export_replicated_merge_tree_part_to_object_storage.sh b/tests/queries/0_stateless/03572_export_replicated_merge_tree_part_to_object_storage.sh new file mode 100755 index 000000000000..a691d4bdf37a --- /dev/null +++ b/tests/queries/0_stateless/03572_export_replicated_merge_tree_part_to_object_storage.sh @@ -0,0 +1,44 @@ +#!/usr/bin/env bash +# Tags: replica, no-parallel, no-replicated-database, no-fasttest + +CURDIR=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd) +# shellcheck source=../shell_config.sh +. "$CURDIR"/../shell_config.sh + +rmt_table="rmt_table_${RANDOM}" +s3_table="s3_table_${RANDOM}" +rmt_table_roundtrip="rmt_table_roundtrip_${RANDOM}" + +query() { + $CLICKHOUSE_CLIENT --query "$1" +} + +query "DROP TABLE IF EXISTS $rmt_table, $s3_table, $rmt_table_roundtrip" + +query "CREATE TABLE $rmt_table (id UInt64, year UInt16) ENGINE = ReplicatedMergeTree('/clickhouse/tables/{database}/$rmt_table', 'replica1') PARTITION BY year ORDER BY tuple()" +query "CREATE TABLE $s3_table (id UInt64, year UInt16) ENGINE = S3(s3_conn, filename='$s3_table', format=Parquet, partition_strategy='hive') PARTITION BY year" + +query "INSERT INTO $rmt_table VALUES (1, 2020), (2, 2020), (3, 2020), (4, 2021)" + +echo "---- Get actual part names and export them" +part_2020=$(query "SELECT name FROM system.parts WHERE database = currentDatabase() AND table = '$rmt_table' AND partition = '2020' ORDER BY name LIMIT 1" | tr -d '\n') +part_2021=$(query "SELECT name FROM system.parts WHERE database = currentDatabase() AND table = '$rmt_table' AND partition = '2021' ORDER BY name LIMIT 1" | tr -d '\n') + +query "ALTER TABLE $rmt_table EXPORT PART '$part_2020' TO TABLE $s3_table SETTINGS allow_experimental_export_merge_tree_part = 1" +query "ALTER TABLE $rmt_table EXPORT PART '$part_2021' TO TABLE $s3_table SETTINGS allow_experimental_export_merge_tree_part = 1" + +echo "---- Both data parts should appear" +query "SELECT * FROM $s3_table ORDER BY id" + +echo "---- Export the same part again, it should be idempotent" +query "ALTER TABLE $rmt_table EXPORT PART '$part_2020' TO TABLE $s3_table SETTINGS allow_experimental_export_merge_tree_part = 1" + +query "SELECT * FROM $s3_table ORDER BY id" + +query "CREATE TABLE $rmt_table_roundtrip (id UInt64, year UInt16) ENGINE = ReplicatedMergeTree('/clickhouse/tables/{database}/$rmt_table_roundtrip', 'replica1') PARTITION BY year ORDER BY tuple()" +query "INSERT INTO $rmt_table_roundtrip SELECT * FROM $s3_table" + +echo "---- Data in roundtrip ReplicatedMergeTree table (should match s3_table)" +query "SELECT * FROM $rmt_table_roundtrip ORDER BY id" + +query "DROP TABLE IF EXISTS $rmt_table, $s3_table, $rmt_table_roundtrip" diff --git a/tests/queries/0_stateless/03572_export_replicated_merge_tree_part_to_object_storage_simple.reference b/tests/queries/0_stateless/03572_export_replicated_merge_tree_part_to_object_storage_simple.reference new file mode 100644 index 000000000000..e69de29bb2d1 diff --git a/tests/queries/0_stateless/03572_export_replicated_merge_tree_part_to_object_storage_simple.sql b/tests/queries/0_stateless/03572_export_replicated_merge_tree_part_to_object_storage_simple.sql new file mode 100644 index 000000000000..628d1bd4cf46 --- /dev/null +++ b/tests/queries/0_stateless/03572_export_replicated_merge_tree_part_to_object_storage_simple.sql @@ -0,0 +1,53 @@ +-- Tags: no-parallel, no-fasttest + +DROP TABLE IF EXISTS 03572_rmt_table, 03572_invalid_schema_table, 03572_rmt_partition_type_mismatch_mt, 03572_rmt_partition_type_mismatch_s3, 03572_rmt_lossy_mt, 03572_rmt_lossy_s3, 03572_rmt_lossless_mt, 03572_rmt_lossless_s3; + +CREATE TABLE 03572_rmt_table (id UInt64, year UInt16) ENGINE = ReplicatedMergeTree('/clickhouse/{database}/test_03572_rmt/03572_rmt_table', 'replica1') PARTITION BY year ORDER BY tuple(); + +INSERT INTO 03572_rmt_table VALUES (1, 2020); + +-- Create a table with a different partition key and export a partition to it. It should throw +-- on the partition-key AST mismatch (schema compat now follows INSERT SELECT positional semantics, +-- so the column shape matches and the partition-key check is what fires). +CREATE TABLE 03572_invalid_schema_table (id UInt64, x UInt16) ENGINE = S3(s3_conn, filename='03572_invalid_schema_table', format='Parquet', partition_strategy='hive') PARTITION BY x; + +ALTER TABLE 03572_rmt_table EXPORT PART '2020_0_0_0' TO TABLE 03572_invalid_schema_table +SETTINGS allow_experimental_export_merge_tree_part = 1; -- {serverError BAD_ARGUMENTS} + +DROP TABLE 03572_invalid_schema_table; + +-- The only partition strategy that supports exports is hive. Wildcard should throw +CREATE TABLE 03572_invalid_schema_table (id UInt64, year UInt16) ENGINE = S3(s3_conn, filename='03572_invalid_schema_table/{_partition_id}', format='Parquet', partition_strategy='wildcard') PARTITION BY (id, year); + +ALTER TABLE 03572_rmt_table EXPORT PART '2020_0_0_0' TO TABLE 03572_invalid_schema_table SETTINGS allow_experimental_export_merge_tree_part = 1; -- {serverError NOT_IMPLEMENTED} + +-- Partition columns follow the same lossy-cast gate as any other column (no special +-- exact-type guard). String -> UInt16 is a lossy cast, so with the default +-- export_merge_tree_part_allow_lossy_cast = 0 it is rejected synchronously. +CREATE TABLE 03572_rmt_partition_type_mismatch_mt (id UInt64, year String) ENGINE = ReplicatedMergeTree('/clickhouse/{database}/test_03572_rmt_pcol_type/03572_rmt_partition_type_mismatch_mt', 'replica1') PARTITION BY year ORDER BY tuple(); +CREATE TABLE 03572_rmt_partition_type_mismatch_s3 (id UInt64, year UInt16) ENGINE = S3(s3_conn, filename='03572_rmt_partition_type_mismatch_s3', format='Parquet', partition_strategy='hive') PARTITION BY year; + +ALTER TABLE 03572_rmt_partition_type_mismatch_mt EXPORT PART '2020_0_0_0' TO TABLE 03572_rmt_partition_type_mismatch_s3 +SETTINGS allow_experimental_export_merge_tree_part = 1; -- {serverError INCOMPATIBLE_COLUMNS} + +-- A lossy cast on a non-partition column (Int64 -> Int32) is rejected synchronously by default. +CREATE TABLE 03572_rmt_lossy_mt (id Int64, year UInt16) ENGINE = ReplicatedMergeTree('/clickhouse/{database}/test_03572_rmt_lossy/03572_rmt_lossy_mt', 'replica1') PARTITION BY year ORDER BY tuple(); +CREATE TABLE 03572_rmt_lossy_s3 (id Int32, year UInt16) ENGINE = S3(s3_conn, filename='03572_rmt_lossy_s3', format='Parquet', partition_strategy='hive') PARTITION BY year; + +ALTER TABLE 03572_rmt_lossy_mt EXPORT PART '2020_0_0_0' TO TABLE 03572_rmt_lossy_s3 +SETTINGS allow_experimental_export_merge_tree_part = 1; -- {serverError INCOMPATIBLE_COLUMNS} + +-- With the acknowledgment setting enabled, the lossy cast passes validation and reaches the +-- part lookup, which fails because the part does not exist. +ALTER TABLE 03572_rmt_lossy_mt EXPORT PART '2020_0_0_0' TO TABLE 03572_rmt_lossy_s3 +SETTINGS allow_experimental_export_merge_tree_part = 1, export_merge_tree_part_allow_lossy_cast = 1; -- {serverError NO_SUCH_DATA_PART} + +-- A lossless widening cast (Int32 -> Int64) passes validation without the setting and reaches +-- the part lookup, which fails because the part does not exist. +CREATE TABLE 03572_rmt_lossless_mt (id Int32, year UInt16) ENGINE = ReplicatedMergeTree('/clickhouse/{database}/test_03572_rmt_lossless/03572_rmt_lossless_mt', 'replica1') PARTITION BY year ORDER BY tuple(); +CREATE TABLE 03572_rmt_lossless_s3 (id Int64, year UInt16) ENGINE = S3(s3_conn, filename='03572_rmt_lossless_s3', format='Parquet', partition_strategy='hive') PARTITION BY year; + +ALTER TABLE 03572_rmt_lossless_mt EXPORT PART '2020_0_0_0' TO TABLE 03572_rmt_lossless_s3 +SETTINGS allow_experimental_export_merge_tree_part = 1; -- {serverError NO_SUCH_DATA_PART} + +DROP TABLE IF EXISTS 03572_rmt_table, 03572_invalid_schema_table, 03572_rmt_partition_type_mismatch_mt, 03572_rmt_partition_type_mismatch_s3, 03572_rmt_lossy_mt, 03572_rmt_lossy_s3, 03572_rmt_lossless_mt, 03572_rmt_lossless_s3; diff --git a/tests/queries/0_stateless/03604_export_merge_tree_partition.reference b/tests/queries/0_stateless/03604_export_merge_tree_partition.reference new file mode 100644 index 000000000000..d48023362b99 --- /dev/null +++ b/tests/queries/0_stateless/03604_export_merge_tree_partition.reference @@ -0,0 +1,31 @@ +Select from source table +1 2020 +2 2020 +3 2020 +4 2021 +5 2021 +6 2022 +7 2022 +Select from destination table +1 2020 +2 2020 +3 2020 +4 2021 +5 2021 +Export partition 2022 +Select from destination table again +1 2020 +2 2020 +3 2020 +4 2021 +5 2021 +6 2022 +7 2022 +---- Data in roundtrip ReplicatedMergeTree table (should match s3_table) +1 2020 +2 2020 +3 2020 +4 2021 +5 2021 +6 2022 +7 2022 diff --git a/tests/queries/0_stateless/03604_export_merge_tree_partition.sh b/tests/queries/0_stateless/03604_export_merge_tree_partition.sh new file mode 100755 index 000000000000..87503112aadb --- /dev/null +++ b/tests/queries/0_stateless/03604_export_merge_tree_partition.sh @@ -0,0 +1,57 @@ +#!/usr/bin/env bash +# Tags: no-fasttest, replica, no-parallel, no-replicated-database + +CURDIR=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd) +# shellcheck source=../shell_config.sh +. "$CURDIR"/../shell_config.sh + +rmt_table="rmt_table_${RANDOM}" +s3_table="s3_table_${RANDOM}" +rmt_table_roundtrip="rmt_table_roundtrip_${RANDOM}" + +query() { + $CLICKHOUSE_CLIENT --query "$1" +} + +query "DROP TABLE IF EXISTS $rmt_table, $s3_table, $rmt_table_roundtrip" + +query "CREATE TABLE $rmt_table (id UInt64, year UInt16) ENGINE = ReplicatedMergeTree('/clickhouse/tables/{database}/$rmt_table', 'replica1') PARTITION BY year ORDER BY tuple()" +query "CREATE TABLE $s3_table (id UInt64, year UInt16) ENGINE = S3(s3_conn, filename='$s3_table', format=Parquet, partition_strategy='hive') PARTITION BY year" + +query "INSERT INTO $rmt_table VALUES (1, 2020), (2, 2020), (4, 2021)" + +query "INSERT INTO $rmt_table VALUES (3, 2020), (5, 2021)" + +query "INSERT INTO $rmt_table VALUES (6, 2022), (7, 2022)" + +# sync replicas +query "SYSTEM SYNC REPLICA $rmt_table" + +query "ALTER TABLE $rmt_table EXPORT PARTITION ID '2020' TO TABLE $s3_table SETTINGS allow_experimental_export_merge_tree_part = 1" + +query "ALTER TABLE $rmt_table EXPORT PARTITION ID '2021' TO TABLE $s3_table SETTINGS allow_experimental_export_merge_tree_part = 1" + +# todo poll some kind of status +sleep 15 + +echo "Select from source table" +query "SELECT * FROM $rmt_table ORDER BY id" + +echo "Select from destination table" +query "SELECT * FROM $s3_table ORDER BY id" + +echo "Export partition 2022" +query "ALTER TABLE $rmt_table EXPORT PARTITION ID '2022' TO TABLE $s3_table SETTINGS allow_experimental_export_merge_tree_part = 1" + +# todo poll some kind of status +sleep 5 + +echo "Select from destination table again" +query "SELECT * FROM $s3_table ORDER BY id" + +query "CREATE TABLE $rmt_table_roundtrip ENGINE = ReplicatedMergeTree('/clickhouse/tables/{database}/$rmt_table_roundtrip', 'replica1') PARTITION BY year ORDER BY tuple() AS SELECT * FROM $s3_table" + +echo "---- Data in roundtrip ReplicatedMergeTree table (should match s3_table)" +query "SELECT * FROM $rmt_table_roundtrip ORDER BY id" + +query "DROP TABLE IF EXISTS $rmt_table, $s3_table, $rmt_table_roundtrip" \ No newline at end of file diff --git a/tests/queries/0_stateless/03608_export_merge_tree_part_filename_pattern.reference b/tests/queries/0_stateless/03608_export_merge_tree_part_filename_pattern.reference new file mode 100644 index 000000000000..8016f5aa113e --- /dev/null +++ b/tests/queries/0_stateless/03608_export_merge_tree_part_filename_pattern.reference @@ -0,0 +1,16 @@ +---- Test: Default pattern {part_name}_{checksum} +1 2020 +2 2020 +3 2020 +---- Verify filename matches 2020_1_1_0_*.1.parquet +1 +---- Test: Custom prefix pattern +4 2021 +---- Verify filename matches myprefix_2021_2_2_0.1.parquet +1 +---- Test: Pattern with macros +1 2020 +2 2020 +3 2020 +---- Verify macros expanded (no literal braces in parquet filenames, that's the best we can do for stateless tests) +1 diff --git a/tests/queries/0_stateless/03608_export_merge_tree_part_filename_pattern.sh b/tests/queries/0_stateless/03608_export_merge_tree_part_filename_pattern.sh new file mode 100755 index 000000000000..12b47f4f2664 --- /dev/null +++ b/tests/queries/0_stateless/03608_export_merge_tree_part_filename_pattern.sh @@ -0,0 +1,49 @@ +#!/usr/bin/env bash +# Tags: no-fasttest +# Tag no-fasttest: requires s3 storage + +CURDIR=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd) +# shellcheck source=../shell_config.sh +. "$CURDIR"/../shell_config.sh + +R=$RANDOM +mt="mt_${R}" +dest1="fp_dest1_${R}" +dest2="fp_dest2_${R}" +dest3="fp_dest3_${R}" + +query() { + $CLICKHOUSE_CLIENT --query "$1" +} + +query "DROP TABLE IF EXISTS $mt, $dest1, $dest2, $dest3" + +query "CREATE TABLE $mt (id UInt64, year UInt16) ENGINE = MergeTree() PARTITION BY year ORDER BY tuple()" +query "INSERT INTO $mt VALUES (1, 2020), (2, 2020), (3, 2020), (4, 2021)" + +query "CREATE TABLE $dest1 (id UInt64, year UInt16) ENGINE = S3(s3_conn, filename='$dest1', format=Parquet, partition_strategy='hive') PARTITION BY year" +query "CREATE TABLE $dest2 (id UInt64, year UInt16) ENGINE = S3(s3_conn, filename='$dest2', format=Parquet, partition_strategy='hive') PARTITION BY year" +query "CREATE TABLE $dest3 (id UInt64, year UInt16) ENGINE = S3(s3_conn, filename='$dest3', format=Parquet, partition_strategy='hive') PARTITION BY year" + +echo "---- Test: Default pattern {part_name}_{checksum}" +query "ALTER TABLE $mt EXPORT PART '2020_1_1_0' TO TABLE $dest1 SETTINGS allow_experimental_export_merge_tree_part = 1, export_merge_tree_part_filename_pattern = '{part_name}_{checksum}'" +sleep 3 +query "SELECT * FROM $dest1 ORDER BY id" +echo "---- Verify filename matches 2020_1_1_0_*.1.parquet" +query "SELECT count() FROM s3(s3_conn, filename='$dest1/**/2020_1_1_0_*.1.parquet', format='One')" + +echo "---- Test: Custom prefix pattern" +query "ALTER TABLE $mt EXPORT PART '2021_2_2_0' TO TABLE $dest2 SETTINGS allow_experimental_export_merge_tree_part = 1, export_merge_tree_part_filename_pattern = 'myprefix_{part_name}'" +sleep 3 +query "SELECT * FROM $dest2 ORDER BY id" +echo "---- Verify filename matches myprefix_2021_2_2_0.1.parquet" +query "SELECT count() FROM s3(s3_conn, filename='$dest2/**/myprefix_2021_2_2_0.1.parquet', format='One')" + +echo "---- Test: Pattern with macros" +query "ALTER TABLE $mt EXPORT PART '2020_1_1_0' TO TABLE $dest3 SETTINGS allow_experimental_export_merge_tree_part = 1, export_merge_tree_part_filename_pattern = '{database}_{table}_{part_name}'" +sleep 3 +query "SELECT * FROM $dest3 ORDER BY id" +echo "---- Verify macros expanded (no literal braces in parquet filenames, that's the best we can do for stateless tests)" +query "SELECT count() = 0 FROM s3(s3_conn, filename='$dest3/**/*.1.parquet', format='One') WHERE _file LIKE '%{%'" + +query "DROP TABLE IF EXISTS $mt, $dest1, $dest2, $dest3" diff --git a/tests/queries/0_stateless/03620_analyzer_distributed_global_in.reference b/tests/queries/0_stateless/03620_analyzer_distributed_global_in.reference index 689e784cc037..dc1a034d49e0 100644 --- a/tests/queries/0_stateless/03620_analyzer_distributed_global_in.reference +++ b/tests/queries/0_stateless/03620_analyzer_distributed_global_in.reference @@ -60,7 +60,7 @@ CreatingSets (Create sets before main query execution) ReadFromSystemNumbers system flush logs query_log; -- SKIP: current_database = currentDatabase() -select normalizeQuery(replace(query, currentDatabase(), 'default')) from system.query_log where event_date >= yesterday() AND event_time >= now() - 600 and log_comment like '%' || currentDatabase() || '%' and not is_initial_query and type != 'QueryStart' and query_kind = 'Select' order by event_time_microseconds; +select normalizeQuery(replace(query, currentDatabase(), 'default')) from system.query_log where event_date >= yesterday() AND event_time >= now() - 600 and log_comment like '%' || currentDatabase() || '%' and initial_query_id != query_id and type != 'QueryStart' and query_kind = 'Select' order by event_time_microseconds; SELECT `__table1`.`x` AS `x`, `__table1`.`y` AS `y` FROM `default`.`tab0` AS `__table1` HAVING in(`x`, (SELECT `__table1`.`number` + ? AS `?` FROM numbers(?) AS `__table1`)) SELECT `__table1`.`x` AS `x`, `__table1`.`y` AS `y` FROM `default`.`tab0` AS `__table1` HAVING globalIn(`x`, `?`) SELECT `__table1`.`x` AS `x`, `__table1`.`y` AS `y` FROM `default`.`tab0` AS `__table1` HAVING in(`x`, (SELECT `__table1`.`number` + ? AS `?` FROM numbers(?) AS `__table1`)) diff --git a/tests/queries/0_stateless/03620_analyzer_distributed_global_in.sql b/tests/queries/0_stateless/03620_analyzer_distributed_global_in.sql index fe44464ca303..6279a2823ceb 100644 --- a/tests/queries/0_stateless/03620_analyzer_distributed_global_in.sql +++ b/tests/queries/0_stateless/03620_analyzer_distributed_global_in.sql @@ -31,4 +31,4 @@ select * from (explain indexes=1, distributed=1 ); system flush logs query_log; -- SKIP: current_database = currentDatabase() -select normalizeQuery(replace(query, currentDatabase(), 'default')) from system.query_log where event_date >= yesterday() AND event_time >= now() - 600 and log_comment like '%' || currentDatabase() || '%' and not is_initial_query and type != 'QueryStart' and query_kind = 'Select' order by event_time_microseconds; +select normalizeQuery(replace(query, currentDatabase(), 'default')) from system.query_log where event_date >= yesterday() AND event_time >= now() - 600 and log_comment like '%' || currentDatabase() || '%' and initial_query_id != query_id and type != 'QueryStart' and query_kind = 'Select' order by event_time_microseconds; diff --git a/tests/queries/0_stateless/03745_system_background_schedule_pool.reference b/tests/queries/0_stateless/03745_system_background_schedule_pool.reference index 51fa0fc836d3..a08c46054c42 100644 --- a/tests/queries/0_stateless/03745_system_background_schedule_pool.reference +++ b/tests/queries/0_stateless/03745_system_background_schedule_pool.reference @@ -1,6 +1,7 @@ 1 buffer_flush default test_buffer_03745 1 StorageBuffer (default.test_buffer_03745)/Bg schedule default test_merge_tree_03745 1 BackgroundJobsAssignee:DataProcessing +schedule default test_merge_tree_03745 1 BackgroundJobsAssignee:Moving schedule default test_merge_tree_03745 1 default.test_merge_tree_03745 (CleanupThread) streaming default test_merge_tree_03745 1 BackgroundJobsAssignee:Streaming distributed default test_distributed_03745 1 default.test_distributed_03745.DistributedInsertQueue.default/Bg diff --git a/tests/queries/0_stateless/04302_iceberg_read_optimization_no_column_stats.reference b/tests/queries/0_stateless/04302_iceberg_read_optimization_no_column_stats.reference new file mode 100644 index 000000000000..6092186edaaf --- /dev/null +++ b/tests/queries/0_stateless/04302_iceberg_read_optimization_no_column_stats.reference @@ -0,0 +1,3 @@ +1 alice 1.5 +2 bob 2.5 +3 carol 3.5 diff --git a/tests/queries/0_stateless/04302_iceberg_read_optimization_no_column_stats.sql b/tests/queries/0_stateless/04302_iceberg_read_optimization_no_column_stats.sql new file mode 100644 index 000000000000..447788ca8e6d --- /dev/null +++ b/tests/queries/0_stateless/04302_iceberg_read_optimization_no_column_stats.sql @@ -0,0 +1,7 @@ +-- Tags: no-fasttest +-- Tag no-fasttest: Depends on S3/MinIO + +-- A stats-less Iceberg manifest (no per-column statistics) must return real +-- values, not NULLs: empty stats were misread as "all columns absent" (#1545). +SELECT * FROM icebergS3(s3_conn, filename = 'iceberg_no_column_stats') ORDER BY ALL +SETTINGS allow_experimental_iceberg_read_optimization = 1; diff --git a/tests/queries/0_stateless/04306_create_handler_http.sh b/tests/queries/0_stateless/04306_create_handler_http.sh index 949fc703e885..2052aef18f05 100755 --- a/tests/queries/0_stateless/04306_create_handler_http.sh +++ b/tests/queries/0_stateless/04306_create_handler_http.sh @@ -54,12 +54,13 @@ CREATE DATABASE db2_${DB}; CREATE TABLE ${DB}.t (x UInt32) ENGINE = Memory; CREATE TABLE ${DB}.secret (x UInt32) ENGINE = Memory AS SELECT 111; CREATE VIEW ${DB}.sv DEFINER=default SQL SECURITY DEFINER AS SELECT x FROM ${DB}.secret; +CREATE VIEW ${DB}.dv AS SELECT currentHandler() = '${HDIST}' AS h_ok, currentRequestURL() = '${P}/dist' AS u_ok; GRANT SELECT ON ${DB}.sv TO \`$RUSER\`; CREATE HANDLER \`$HA\` URL '${P}/exact' AS SELECT 1 AS a, 'hello' AS b FORMAT TSV; CREATE HANDLER \`$HP\` URL PREFIX '${P}/prefix/' AS SELECT 'prefixed' AS r FORMAT TSV; CREATE HANDLER \`$HB\` URL '${P}/introspect' AS SELECT currentHandler() = '${HB}' AS h_ok, currentRequestURL() = '${P}/introspect?max_block_size=100' AS u_ok FORMAT TSV; CREATE HANDLER \`$HBRANCH\` URL '${P}/branch' AS SELECT if(currentHandler() = '${HBRANCH}', 'matched', 'no') AS r FORMAT TSV; -CREATE HANDLER \`$HDIST\` URL '${P}/dist' AS SELECT * FROM remote('127.0.0.2', view(SELECT currentHandler() = '${HDIST}' AS h_ok, currentRequestURL() = '${P}/dist' AS u_ok)) FORMAT TSV; +CREATE HANDLER \`$HDIST\` URL '${P}/dist' AS SELECT * FROM remote('127.0.0.2', ${DB}, dv) FORMAT TSV; CREATE HANDLER \`$HC\` URL REGEXP '${P}/item/(?P[0-9]+)' AS SELECT {id:UInt32} AS id FORMAT TSV; CREATE HANDLER \`$HPOST\` URL '${P}/param' METHODS (GET, POST) AS SELECT {n:UInt32} * 2 AS doubled FORMAT TSV; CREATE HANDLER \`$HHDR\` URL '${P}/rheaders' AS SELECT 1 SETTINGS http_response_headers = {'X-Custom':'yes'}; @@ -95,8 +96,8 @@ ${CLICKHOUSE_CURL} -sS "${BASE}${P}/branch" echo "=== currentHandler() and currentRequestURL() are visible on remote shards of a distributed query ===" # The handler name and request URL live in ClientInfo, so they are serialized on distributed fan-out. -# The handler evaluates them on a remote shard via remote(view(...)) and compares them with the values -# seen locally. +# The handler reads a view on a remote shard through remote(), and the view compares the values the +# shard sees with the ones the handler was invoked with. ${CLICKHOUSE_CURL} -sS "${BASE}${P}/dist" echo "=== parameterized query via regexp URL capture ===" diff --git a/tests/queries/0_stateless/04337_iceberg_v3_row_lineage_reserved_field_id.sh b/tests/queries/0_stateless/04337_iceberg_v3_row_lineage_reserved_field_id.sh index 65710df27614..13f56710a84f 100755 --- a/tests/queries/0_stateless/04337_iceberg_v3_row_lineage_reserved_field_id.sh +++ b/tests/queries/0_stateless/04337_iceberg_v3_row_lineage_reserved_field_id.sh @@ -47,7 +47,11 @@ pq.write_table(table, path) PY # A spec-compliant reader ignores the reserved field id and returns the projected column. -${CLICKHOUSE_CLIENT} --query "SELECT x FROM icebergLocal('${ICEBERG_TABLE_PATH}') ORDER BY x;" +# `allow_experimental_iceberg_read_optimization` is disabled here (and below) on purpose: it can +# answer a query from the manifest statistics alone (e.g. when a projected column is constant in the +# file), in which case the data file is never opened and the parquet schema is never converted, so +# this test would not exercise the reserved-field-id handling it is about. +${CLICKHOUSE_CLIENT} --query "SELECT x FROM icebergLocal('${ICEBERG_TABLE_PATH}') ORDER BY x SETTINGS allow_experimental_iceberg_read_optimization = 0;" # Conversely, 2147483447 (Integer.MAX_VALUE - 200) is the highest field id a table may use, i.e. # NOT reserved. An unmapped column with that id is a genuine schema mismatch and must still be @@ -83,7 +87,7 @@ table = pa.table( pq.write_table(table, path) PY -${CLICKHOUSE_CLIENT} --query "SELECT x FROM icebergLocal('${ICEBERG_TABLE_PATH_UNMAPPED}') ORDER BY x;" 2>&1 | grep -oF "ICEBERG_SPECIFICATION_VIOLATION" | head -1 +${CLICKHOUSE_CLIENT} --query "SELECT x FROM icebergLocal('${ICEBERG_TABLE_PATH_UNMAPPED}') ORDER BY x SETTINGS allow_experimental_iceberg_read_optimization = 0;" 2>&1 | grep -oF "ICEBERG_SPECIFICATION_VIOLATION" | head -1 # Cleanup ${CLICKHOUSE_CLIENT} --query "DROP TABLE IF EXISTS t_v3_row_lineage;" diff --git a/tests/queries/0_stateless/04673_join_filter_pushdown_count_subquery.reference b/tests/queries/0_stateless/04673_join_filter_pushdown_count_subquery.reference new file mode 100644 index 000000000000..9b231627ac1d --- /dev/null +++ b/tests/queries/0_stateless/04673_join_filter_pushdown_count_subquery.reference @@ -0,0 +1,2 @@ +40 +40 diff --git a/tests/queries/0_stateless/04673_join_filter_pushdown_count_subquery.sql b/tests/queries/0_stateless/04673_join_filter_pushdown_count_subquery.sql new file mode 100644 index 000000000000..8699c29c74df --- /dev/null +++ b/tests/queries/0_stateless/04673_join_filter_pushdown_count_subquery.sql @@ -0,0 +1,91 @@ +-- Tags: no-parallel-replicas, no-random-settings +-- no-parallel-replicas: EXPLAIN Prewhere differs with parallel replicas. +-- no-random-settings: `optimize_move_to_prewhere` / `query_plan_optimize_prewhere` are randomized off. + +-- Left-only WHERE on `count()` of `SELECT * … JOIN` must still be pushed through +-- the JOIN (and composed through identifier-rename expressions) so the left +-- read can apply PREWHERE / index analysis. + +DROP TABLE IF EXISTS t_left; +DROP TABLE IF EXISTS t_right; + +CREATE TABLE t_left +( + a Int32, + b Int32 +) +ENGINE = MergeTree +ORDER BY a +SETTINGS index_granularity = 1024, index_granularity_bytes = '10Mi'; + +CREATE TABLE t_right +( + a Int32, + b Int32 +) +ENGINE = Memory; + +INSERT INTO t_left SELECT number, number FROM numbers(100); +INSERT INTO t_right SELECT number, number FROM numbers(100); + +SET enable_parallel_replicas = 0; +SET query_plan_join_swap_table = 0; +SET enable_analyzer = 1; +SET query_plan_filter_push_down = 1; +SET enable_join_runtime_filters = 0; +SET join_use_nulls = 1; +SET optimize_move_to_prewhere = 1; +SET query_plan_optimize_prewhere = 1; + +SELECT count() +FROM +( + SELECT * + FROM t_left AS foo + LEFT JOIN t_right AS bar ON foo.b = bar.b + WHERE foo.a < 40 +); + +SELECT throwIf(count() = 0) +FROM +( + EXPLAIN actions = 1 + SELECT count() + FROM + ( + SELECT * + FROM t_left AS foo + LEFT JOIN t_right AS bar ON foo.b = bar.b + WHERE foo.a < 40 + ) +) +WHERE explain ILIKE '%Prewhere%' +FORMAT Null; + +SELECT count() +FROM +( + SELECT * + FROM (SELECT * FROM t_left) AS foo + LEFT JOIN t_right AS bar ON foo.b = bar.b + WHERE foo.a < 40 +); + +SELECT throwIf(count() = 0) +FROM +( + EXPLAIN actions = 1 + SELECT count() + FROM + ( + SELECT * + FROM (SELECT * FROM t_left) AS foo + LEFT JOIN t_right AS bar ON foo.b = bar.b + WHERE foo.a < 40 + ) +) +WHERE explain ILIKE '%Prewhere%' +FORMAT Null; + +DROP TABLE t_left; +DROP TABLE t_right; diff --git a/tests/queries/0_stateless/04674_cluster_wrap_asof_paste_filter.reference b/tests/queries/0_stateless/04674_cluster_wrap_asof_paste_filter.reference new file mode 100644 index 000000000000..3c83ca38c2a7 --- /dev/null +++ b/tests/queries/0_stateless/04674_cluster_wrap_asof_paste_filter.reference @@ -0,0 +1,2 @@ +0 +1 b diff --git a/tests/queries/0_stateless/04674_cluster_wrap_asof_paste_filter.sql b/tests/queries/0_stateless/04674_cluster_wrap_asof_paste_filter.sql new file mode 100644 index 000000000000..d434660e4e46 --- /dev/null +++ b/tests/queries/0_stateless/04674_cluster_wrap_asof_paste_filter.sql @@ -0,0 +1,70 @@ +-- Tags: no-fasttest +-- no-fasttest: `fileCluster` is not in the fast test build. +-- Copying a wrap `WHERE` onto `IStorageCluster` must not prefilter the right +-- side of an `ASOF JOIN` or either side of a `PASTE JOIN`. + +SET enable_analyzer = 1; +SET query_plan_filter_push_down = 1; +SET query_plan_join_swap_table = 0; +SET enable_join_runtime_filters = 0; +SET enable_parallel_replicas = 0; + +DROP TABLE IF EXISTS t_asof_left; +CREATE TABLE t_asof_left +( + id Int32, + t Int32 +) +ENGINE = Memory; +INSERT INTO t_asof_left VALUES (1, 10); + +INSERT INTO FUNCTION file(currentDatabase() || '_04674_asof_right.tsv', 'TSV', 'id Int32, t Int32, flag Int32') +SELECT * +FROM +( + SELECT 1 AS id, 9 AS t, 0 AS flag + UNION ALL + SELECT 1, 8, 1 +) +SETTINGS engine_file_truncate_on_insert = 1; + +-- Nearest right row is `(t = 9, flag = 0)`. `WHERE flag = 1` must run after +-- `ASOF`, so the result is empty. Prefiltering the wrapped `fileCluster` would +-- keep only `t = 8` and incorrectly match it. +SELECT count() +FROM t_asof_left AS l +ASOF JOIN fileCluster( + 'test_cluster_one_shard_two_replicas', + currentDatabase() || '_04674_asof_right.tsv', + 'TSV', + 'id Int32, t Int32, flag Int32') AS r ON l.id = r.id AND l.t >= r.t +WHERE r.flag = 1; + +DROP TABLE t_asof_left; + +INSERT INTO FUNCTION file(currentDatabase() || '_04674_paste_left.tsv', 'TSV', 'n Int32, flag Int32') +SELECT number, if(number = 1, 1, 0) +FROM numbers(3) +SETTINGS engine_file_truncate_on_insert = 1; + +DROP TABLE IF EXISTS t_paste_right; +CREATE TABLE t_paste_right +( + s String +) +ENGINE = Memory; +INSERT INTO t_paste_right VALUES ('a'), ('b'), ('c'); + +-- `PASTE` pairs by position, then `WHERE flag = 1` keeps the middle pair +-- `(1, b)`. Prefiltering the wrapped left table would pair `1` with `a`. +SELECT l.n, r.s +FROM fileCluster( + 'test_cluster_one_shard_two_replicas', + currentDatabase() || '_04674_paste_left.tsv', + 'TSV', + 'n Int32, flag Int32') AS l +PASTE JOIN t_paste_right AS r +WHERE l.flag = 1 +SETTINGS max_threads = 1; + +DROP TABLE t_paste_right; diff --git a/tests/queries/0_stateless/04675_cluster_wrap_stateful_filter.reference b/tests/queries/0_stateless/04675_cluster_wrap_stateful_filter.reference new file mode 100644 index 000000000000..0cfbf08886fc --- /dev/null +++ b/tests/queries/0_stateless/04675_cluster_wrap_stateful_filter.reference @@ -0,0 +1 @@ +2 diff --git a/tests/queries/0_stateless/04675_cluster_wrap_stateful_filter.sql b/tests/queries/0_stateless/04675_cluster_wrap_stateful_filter.sql new file mode 100644 index 000000000000..dc0709966b03 --- /dev/null +++ b/tests/queries/0_stateless/04675_cluster_wrap_stateful_filter.sql @@ -0,0 +1,35 @@ +-- Tags: no-fasttest +-- no-fasttest: `fileCluster` is not in the fast test build. +-- `timeSeriesStoreTags` is stateful and still `isDeterministicInScopeOfQuery`. +-- Copying it into the `IStorageCluster` wrap would store tags twice (the same +-- class of bug as wrapping `aiEmbed`). The `n < 2` conjunct must still be copied. + +SET enable_analyzer = 1; +SET query_plan_filter_push_down = 1; +SET query_plan_join_swap_table = 0; +SET enable_join_runtime_filters = 0; +SET enable_parallel_replicas = 0; + +INSERT INTO FUNCTION file(currentDatabase() || '_04675_wrap_left.tsv', 'TSV', 'n UInt64') +SELECT number +FROM numbers(3) +SETTINGS engine_file_truncate_on_insert = 1; + +DROP TABLE IF EXISTS t_wrap_right; +CREATE TABLE t_wrap_right +( + n UInt64 +) +ENGINE = Memory; +INSERT INTO t_wrap_right VALUES (0), (1), (2); + +SELECT count() +FROM fileCluster( + 'test_cluster_one_shard_two_replicas', + currentDatabase() || '_04675_wrap_left.tsv', + 'TSV', + 'n UInt64') AS l +LEFT JOIN t_wrap_right AS r ON l.n = r.n +WHERE l.n < 2 AND timeSeriesStoreTags(l.n, []) = l.n; + +DROP TABLE t_wrap_right; diff --git a/tests/queries/0_stateless/04676_cluster_wrap_bare_other_side_where.reference b/tests/queries/0_stateless/04676_cluster_wrap_bare_other_side_where.reference new file mode 100644 index 000000000000..d00491fd7e5b --- /dev/null +++ b/tests/queries/0_stateless/04676_cluster_wrap_bare_other_side_where.reference @@ -0,0 +1 @@ +1 diff --git a/tests/queries/0_stateless/04676_cluster_wrap_bare_other_side_where.sql b/tests/queries/0_stateless/04676_cluster_wrap_bare_other_side_where.sql new file mode 100644 index 000000000000..17a77adf2ca2 --- /dev/null +++ b/tests/queries/0_stateless/04676_cluster_wrap_bare_other_side_where.sql @@ -0,0 +1,37 @@ +-- Tags: no-fasttest +-- no-fasttest: `fileCluster` is not in the fast test build. +-- A wrap `WHERE` that is a bare column from the other JOIN side must be dropped +-- rather than copied onto `SELECT cols FROM fileCluster`. Otherwise planning +-- fails with an unknown identifier. + +SET enable_analyzer = 1; +SET query_plan_filter_push_down = 1; +SET query_plan_join_swap_table = 0; +SET enable_join_runtime_filters = 0; +SET enable_parallel_replicas = 0; + +INSERT INTO FUNCTION file(currentDatabase() || '_04676_wrap_left.tsv', 'TSV', 'n Int32') +SELECT number + 1 +FROM numbers(2) +SETTINGS engine_file_truncate_on_insert = 1; + +DROP TABLE IF EXISTS t_04676_right; +CREATE TABLE t_04676_right +( + id Int32, + flag UInt8 +) +ENGINE = Memory; +INSERT INTO t_04676_right VALUES (1, 1), (2, 0); + +SELECT l.n +FROM fileCluster( + 'test_cluster_one_shard_two_replicas', + currentDatabase() || '_04676_wrap_left.tsv', + 'TSV', + 'n Int32') AS l +INNER JOIN t_04676_right AS r ON l.n = r.id +WHERE r.flag +ORDER BY l.n; + +DROP TABLE t_04676_right; diff --git a/tests/queries/0_stateless/04677_cluster_wrap_server_constant_filter.reference b/tests/queries/0_stateless/04677_cluster_wrap_server_constant_filter.reference new file mode 100644 index 000000000000..0cfbf08886fc --- /dev/null +++ b/tests/queries/0_stateless/04677_cluster_wrap_server_constant_filter.reference @@ -0,0 +1 @@ +2 diff --git a/tests/queries/0_stateless/04677_cluster_wrap_server_constant_filter.sql b/tests/queries/0_stateless/04677_cluster_wrap_server_constant_filter.sql new file mode 100644 index 000000000000..9f5f996dcdef --- /dev/null +++ b/tests/queries/0_stateless/04677_cluster_wrap_server_constant_filter.sql @@ -0,0 +1,71 @@ +-- Tags: no-fasttest, no-parallel-replicas, no-random-settings +-- no-fasttest: `fileCluster` is not in the fast test build. +-- no-parallel-replicas: EXPLAIN of the cluster wrap differs with parallel replicas. +-- no-random-settings: join / filter EXPLAIN text is randomized otherwise. +-- +-- `hostName` is `isServerConstant` and must not be copied into the wrap query +-- sent to remotes (`ReadFromCluster` "Query:" line). `count()` cannot see that: +-- `hostName() = hostName()` is true on every node. + +SET enable_analyzer = 1; +SET query_plan_filter_push_down = 1; +SET query_plan_join_swap_table = 0; +SET enable_join_runtime_filters = 0; +SET enable_parallel_replicas = 0; + +INSERT INTO FUNCTION file(currentDatabase() || '_04677_wrap_left.tsv', 'TSV', 'n UInt64') +SELECT number +FROM numbers(3) +SETTINGS engine_file_truncate_on_insert = 1; + +DROP TABLE IF EXISTS t_04677_right; +CREATE TABLE t_04677_right +( + n UInt64 +) +ENGINE = Memory; +INSERT INTO t_04677_right VALUES (0), (1), (2); + +SELECT count() +FROM fileCluster( + 'test_cluster_one_shard_two_replicas', + currentDatabase() || '_04677_wrap_left.tsv', + 'TSV', + 'n UInt64') AS l +LEFT JOIN t_04677_right AS r ON l.n = r.n +WHERE l.n < 2 AND hostName() = hostName(); + +SELECT throwIf(count() = 0) +FROM +( + EXPLAIN actions = 1 + SELECT count() + FROM fileCluster( + 'test_cluster_one_shard_two_replicas', + currentDatabase() || '_04677_wrap_left.tsv', + 'TSV', + 'n UInt64') AS l + LEFT JOIN t_04677_right AS r ON l.n = r.n + WHERE l.n < 2 AND hostName() = hostName() +) +WHERE explain LIKE '%Query:%' + AND (explain LIKE '%n < 2%' OR explain LIKE '%less(%2%') +FORMAT Null; + +SELECT throwIf(count() != 0) +FROM +( + EXPLAIN actions = 1 + SELECT count() + FROM fileCluster( + 'test_cluster_one_shard_two_replicas', + currentDatabase() || '_04677_wrap_left.tsv', + 'TSV', + 'n UInt64') AS l + LEFT JOIN t_04677_right AS r ON l.n = r.n + WHERE l.n < 2 AND hostName() = hostName() +) +WHERE explain LIKE '%Query:%' AND explain ILIKE '%hostName%' +FORMAT Null; + +DROP TABLE t_04677_right; diff --git a/tests/queries/0_stateless/04678_cluster_wrap_dictget_joinget_filter.reference b/tests/queries/0_stateless/04678_cluster_wrap_dictget_joinget_filter.reference new file mode 100644 index 000000000000..083edaac2489 --- /dev/null +++ b/tests/queries/0_stateless/04678_cluster_wrap_dictget_joinget_filter.reference @@ -0,0 +1,3 @@ +2 +2 +2 diff --git a/tests/queries/0_stateless/04678_cluster_wrap_dictget_joinget_filter.sql b/tests/queries/0_stateless/04678_cluster_wrap_dictget_joinget_filter.sql new file mode 100644 index 000000000000..103dfba0299c --- /dev/null +++ b/tests/queries/0_stateless/04678_cluster_wrap_dictget_joinget_filter.sql @@ -0,0 +1,76 @@ +-- Tags: no-fasttest, no-parallel-replicas +-- no-fasttest: `fileCluster` is not in the fast test build. +-- no-parallel-replicas: dictionaries created here are not on parallel replica workers. +-- `dictGet` / `joinGet` / `FQDN` are not safe to copy onto the `IStorageCluster` wrap +-- `WHERE` (node-local dictionary, Join table, or hostname). The `n < 2` conjunct +-- must still be copied. + +SET enable_analyzer = 1; +SET query_plan_filter_push_down = 1; +SET query_plan_join_swap_table = 0; +SET enable_join_runtime_filters = 0; +SET enable_parallel_replicas = 0; + +INSERT INTO FUNCTION file(currentDatabase() || '_04678_wrap_left.tsv', 'TSV', 'n UInt64') +SELECT number +FROM numbers(3) +SETTINGS engine_file_truncate_on_insert = 1; + +DROP TABLE IF EXISTS t_04678_right; +CREATE TABLE t_04678_right +( + n UInt64 +) +ENGINE = Memory; +INSERT INTO t_04678_right VALUES (0), (1), (2); + +DROP DICTIONARY IF EXISTS dict_04678; +CREATE DICTIONARY dict_04678 +( + id UInt64, + flag UInt8 +) +PRIMARY KEY id +SOURCE(CLICKHOUSE(QUERY $$SELECT c1 AS id, c2 AS flag FROM VALUES((0, 1), (1, 1), (2, 0))$$)) +LAYOUT(FLAT()) +LIFETIME(0); + +SELECT count() +FROM fileCluster( + 'test_cluster_one_shard_two_replicas', + currentDatabase() || '_04678_wrap_left.tsv', + 'TSV', + 'n UInt64') AS l +LEFT JOIN t_04678_right AS r ON l.n = r.n +WHERE l.n < 2 AND dictGet(currentDatabase() || '.dict_04678', 'flag', l.n) = 1; + +DROP TABLE IF EXISTS j_04678; +CREATE TABLE j_04678 +( + id UInt64, + flag UInt8 +) +ENGINE = Join(ANY, LEFT, id); +INSERT INTO j_04678 VALUES (0, 1), (1, 1), (2, 0); + +SELECT count() +FROM fileCluster( + 'test_cluster_one_shard_two_replicas', + currentDatabase() || '_04678_wrap_left.tsv', + 'TSV', + 'n UInt64') AS l +LEFT JOIN t_04678_right AS r ON l.n = r.n +WHERE l.n < 2 AND joinGet(currentDatabase() || '.j_04678', 'flag', l.n) = 1; + +SELECT count() +FROM fileCluster( + 'test_cluster_one_shard_two_replicas', + currentDatabase() || '_04678_wrap_left.tsv', + 'TSV', + 'n UInt64') AS l +LEFT JOIN t_04678_right AS r ON l.n = r.n +WHERE l.n < 2 AND FQDN() = FQDN(); + +DROP TABLE j_04678; +DROP DICTIONARY dict_04678; +DROP TABLE t_04678_right; diff --git a/tests/queries/0_stateless/04679_join_disjunction_pushdown_count_subquery.reference b/tests/queries/0_stateless/04679_join_disjunction_pushdown_count_subquery.reference new file mode 100644 index 000000000000..0cfbf08886fc --- /dev/null +++ b/tests/queries/0_stateless/04679_join_disjunction_pushdown_count_subquery.reference @@ -0,0 +1 @@ +2 diff --git a/tests/queries/0_stateless/04679_join_disjunction_pushdown_count_subquery.sql b/tests/queries/0_stateless/04679_join_disjunction_pushdown_count_subquery.sql new file mode 100644 index 000000000000..674605dc70a0 --- /dev/null +++ b/tests/queries/0_stateless/04679_join_disjunction_pushdown_count_subquery.sql @@ -0,0 +1,49 @@ +-- Tags: no-parallel-replicas, no-random-settings +-- no-parallel-replicas: EXPLAIN / Prewhere differ with parallel replicas. +-- no-random-settings: `use_join_disjunctions_push_down` is randomized off. + +-- Partial (disjunction) JOIN filter pushdown must remap `JoinStepLogical` +-- identifier aliases (`__table1.a`) to the child's physical column (`a`). +-- `count()` of `SELECT * … JOIN` drops unused JOIN-output names, so without +-- the remap `addFilterOnTop` threw `NOT_FOUND_COLUMN_IN_BLOCK`. + +DROP TABLE IF EXISTS t_left; +DROP TABLE IF EXISTS t_right; + +CREATE TABLE t_left +( + a Int32, + b Int32 +) +ENGINE = MergeTree +ORDER BY a; + +CREATE TABLE t_right +( + a Int32, + b Int32 +) +ENGINE = Memory; + +INSERT INTO t_left VALUES (10, 1), (90, 2), (30, 3); +INSERT INTO t_right VALUES (60, 1), (5, 2), (30, 3); + +SET enable_analyzer = 1; +SET query_plan_filter_push_down = 1; +SET use_join_disjunctions_push_down = 1; +SET enable_join_runtime_filters = 0; +SET enable_parallel_replicas = 0; +SET query_plan_join_swap_table = 0; +SET join_use_nulls = 1; + +SELECT count() +FROM +( + SELECT * + FROM t_left AS foo + LEFT JOIN t_right AS bar ON foo.b = bar.b + WHERE (foo.a < 40 AND bar.a > 50) OR (foo.a > 80 AND bar.a < 10) +); + +DROP TABLE t_left; +DROP TABLE t_right; diff --git a/tests/queries/0_stateless/data_minio/iceberg_no_column_stats/README.md b/tests/queries/0_stateless/data_minio/iceberg_no_column_stats/README.md new file mode 100644 index 000000000000..d3914258df0d --- /dev/null +++ b/tests/queries/0_stateless/data_minio/iceberg_no_column_stats/README.md @@ -0,0 +1,29 @@ +## How this data is generated? + +A tiny Iceberg v2 table with three nullable columns and three rows: + +| id | name | value | +|----|-------|-------| +| 1 | alice | 1.5 | +| 2 | bob | 2.5 | +| 3 | carol | 3.5 | + +The point of this fixture is that its manifest is **stats-less**: the `data_file` +entry carries no per-column statistics at all (`column_sizes`, `value_counts`, +`null_value_counts`, `lower_bounds`, `upper_bounds` are all empty). Such manifests +are produced by writers that do not collect metrics. They left ClickHouse's +`DataFileMetaInfo::columns_info` empty, which the Iceberg read optimization used +to misread as "every column is absent", returning `NULL` for all of them. + +It is generated by `generate.py` (needs `pyiceberg`, `pyarrow`, `fastavro`): + +```bash +python3 generate.py +``` + +The script creates the table via `pyiceberg` with +`write.metadata.metrics.default = none`, then post-processes the manifest avro to +drop every per-column statistic and rewrites all internal paths to the stable +`s3a://test/iceberg_no_column_stats` prefix used by the test bucket. + +Used by `tests/queries/0_stateless/04302_iceberg_read_optimization_no_column_stats.sql`. diff --git a/tests/queries/0_stateless/data_minio/iceberg_no_column_stats/data/00000-0-39d4e713-69c8-49f9-ab8e-f887af4bcecb.parquet b/tests/queries/0_stateless/data_minio/iceberg_no_column_stats/data/00000-0-39d4e713-69c8-49f9-ab8e-f887af4bcecb.parquet new file mode 100644 index 000000000000..fcf707c09c17 Binary files /dev/null and b/tests/queries/0_stateless/data_minio/iceberg_no_column_stats/data/00000-0-39d4e713-69c8-49f9-ab8e-f887af4bcecb.parquet differ diff --git a/tests/queries/0_stateless/data_minio/iceberg_no_column_stats/generate.py b/tests/queries/0_stateless/data_minio/iceberg_no_column_stats/generate.py new file mode 100644 index 000000000000..13262e92530e --- /dev/null +++ b/tests/queries/0_stateless/data_minio/iceberg_no_column_stats/generate.py @@ -0,0 +1,145 @@ +#!/usr/bin/env python3 +"""Generate the stats-less Iceberg fixture for 04302_iceberg_read_optimization_no_column_stats. + +Creates a 3-row table, strips every per-column statistic from the manifest so +ClickHouse's `DataFileMetaInfo::columns_info` is empty, and rewrites internal +paths to a stable `s3a://test/` prefix. See README.md. Usage: generate.py . +""" +import json +import shutil +import sys +import tempfile +from pathlib import Path + +import fastavro +import pyarrow as pa +from pyiceberg.catalog.sql import SqlCatalog +from pyiceberg.schema import Schema +from pyiceberg.types import NestedField, LongType, StringType, DoubleType + +AVRO_RESERVED = {"avro.schema", "avro.codec"} + +# Per-column statistics on a manifest data_file entry (all optional); clearing +# them all is what makes the manifest stats-less. +STAT_FIELDS = ( + "column_sizes", + "value_counts", + "null_value_counts", + "nan_value_counts", + "lower_bounds", + "upper_bounds", +) + + +def deep_replace(obj, old, new): + if isinstance(obj, str): + return obj.replace(old, new) + if isinstance(obj, dict): + return {k: deep_replace(v, old, new) for k, v in obj.items()} + if isinstance(obj, list): + return [deep_replace(v, old, new) for v in obj] + return obj + + +def clear_stats(record): + df = record.get("data_file") + if isinstance(df, dict): + for field in STAT_FIELDS: + if field in df and df[field]: + df[field] = [] + return record + + +def rewrite_avro(src: Path, dst: Path, old: str, new: str, strip_stats: bool): + with open(src, "rb") as f: + reader = fastavro.reader(f) + schema = reader.writer_schema + meta = {k: v for k, v in reader.metadata.items() if k not in AVRO_RESERVED} + records = [deep_replace(r, old, new) for r in reader] + if strip_stats: + records = [clear_stats(r) for r in records] + with open(dst, "wb") as f: + fastavro.writer(f, schema, records, metadata=meta) + + +def main(out_dir: str): + work = Path(tempfile.mkdtemp(prefix="iceberg_gen_")) + warehouse = work / "warehouse" + warehouse.mkdir(parents=True) + + catalog = SqlCatalog( + "gen", + uri=f"sqlite:///{work}/catalog.db", + warehouse=f"file://{warehouse}", + ) + catalog.create_namespace("ns") + + schema = Schema( + NestedField(1, "id", LongType(), required=False), + NestedField(2, "name", StringType(), required=False), + NestedField(3, "value", DoubleType(), required=False), + ) + + # Reduces metrics, but pyiceberg still writes column_sizes (stripped below). + table = catalog.create_table( + "ns.no_stats", + schema=schema, + properties={"write.metadata.metrics.default": "none"}, + ) + + data = pa.table( + { + "id": pa.array([1, 2, 3], type=pa.int64()), + "name": pa.array(["alice", "bob", "carol"], type=pa.string()), + "value": pa.array([1.5, 2.5, 3.5], type=pa.float64()), + } + ) + table.append(data) + + table_location = Path(table.location().replace("file://", "")) + old_prefix = table.location() # file:///tmp/.../ns.db/no_stats + out = Path(out_dir) + new_prefix = f"s3a://test/{out.name}" # s3a://test/iceberg_no_column_stats + + if out.exists(): + shutil.rmtree(out) + (out / "metadata").mkdir(parents=True) + (out / "data").mkdir(parents=True) + + for f in (table_location / "data").rglob("*"): + if f.is_file(): + rel = f.relative_to(table_location / "data") + target = out / "data" / rel + target.parent.mkdir(parents=True, exist_ok=True) + shutil.copy2(f, target) + + meta_dir = table_location / "metadata" + + # Keep only the latest metadata.json (the post-append snapshot). The empty + # create-time version and the history logs that reference it aren't read. + latest_json = max( + (f for f in meta_dir.iterdir() if f.name.endswith(".metadata.json")), + key=lambda f: int(f.name.split("-", 1)[0]), + ) + meta = json.loads(latest_json.read_text().replace(old_prefix, new_prefix)) + meta["metadata-log"] = [] + meta["snapshot-log"] = [] + (out / "metadata" / latest_json.name).write_text(json.dumps(meta, separators=(",", ":"))) + + for f in meta_dir.iterdir(): + if f.name.endswith(".avro"): + # Only manifests carry data_file stats; the manifest list (snap-*) does not. + strip = not f.name.startswith("snap-") + rewrite_avro(f, out / "metadata" / f.name, old_prefix, new_prefix, strip) + + print(f"old prefix: {old_prefix}") + print(f"new prefix: {new_prefix}") + print(f"table uuid: {table.metadata.table_uuid}") + print(f"copied to: {out}") + shutil.rmtree(work) + + +if __name__ == "__main__": + if len(sys.argv) != 2: + sys.exit(f"usage: {sys.argv[0]} ") + main(sys.argv[1]) diff --git a/tests/queries/0_stateless/data_minio/iceberg_no_column_stats/metadata/00001-1347065c-1de2-40e5-8774-255dfdff698c.metadata.json b/tests/queries/0_stateless/data_minio/iceberg_no_column_stats/metadata/00001-1347065c-1de2-40e5-8774-255dfdff698c.metadata.json new file mode 100644 index 000000000000..892735e3b6b7 --- /dev/null +++ b/tests/queries/0_stateless/data_minio/iceberg_no_column_stats/metadata/00001-1347065c-1de2-40e5-8774-255dfdff698c.metadata.json @@ -0,0 +1 @@ +{"location":"s3a://test/iceberg_no_column_stats","table-uuid":"d0e63068-996d-41ee-9b7d-31f1fb17f1b1","last-updated-ms":1782851862557,"last-column-id":3,"schemas":[{"type":"struct","fields":[{"id":1,"name":"id","type":"long","required":false},{"id":2,"name":"name","type":"string","required":false},{"id":3,"name":"value","type":"double","required":false}],"schema-id":0,"identifier-field-ids":[]}],"current-schema-id":0,"partition-specs":[{"spec-id":0,"fields":[]}],"default-spec-id":0,"last-partition-id":999,"properties":{"write.metadata.metrics.default":"none"},"current-snapshot-id":7564025723254944482,"snapshots":[{"snapshot-id":7564025723254944482,"sequence-number":1,"timestamp-ms":1782851862557,"manifest-list":"s3a://test/iceberg_no_column_stats/metadata/snap-7564025723254944482-0-39d4e713-69c8-49f9-ab8e-f887af4bcecb.avro","summary":{"operation":"append","added-files-size":"1346","added-data-files":"1","added-records":"3","total-data-files":"1","total-delete-files":"0","total-records":"3","total-files-size":"1346","total-position-deletes":"0","total-equality-deletes":"0"},"schema-id":0}],"snapshot-log":[],"metadata-log":[],"sort-orders":[{"order-id":0,"fields":[]}],"default-sort-order-id":0,"refs":{"main":{"snapshot-id":7564025723254944482,"type":"branch"}},"statistics":[],"partition-statistics":[],"format-version":2,"last-sequence-number":1} \ No newline at end of file diff --git a/tests/queries/0_stateless/data_minio/iceberg_no_column_stats/metadata/39d4e713-69c8-49f9-ab8e-f887af4bcecb-m0.avro b/tests/queries/0_stateless/data_minio/iceberg_no_column_stats/metadata/39d4e713-69c8-49f9-ab8e-f887af4bcecb-m0.avro new file mode 100644 index 000000000000..4b0b3acfe267 Binary files /dev/null and b/tests/queries/0_stateless/data_minio/iceberg_no_column_stats/metadata/39d4e713-69c8-49f9-ab8e-f887af4bcecb-m0.avro differ diff --git a/tests/queries/0_stateless/data_minio/iceberg_no_column_stats/metadata/snap-7564025723254944482-0-39d4e713-69c8-49f9-ab8e-f887af4bcecb.avro b/tests/queries/0_stateless/data_minio/iceberg_no_column_stats/metadata/snap-7564025723254944482-0-39d4e713-69c8-49f9-ab8e-f887af4bcecb.avro new file mode 100644 index 000000000000..1ebf121994f5 Binary files /dev/null and b/tests/queries/0_stateless/data_minio/iceberg_no_column_stats/metadata/snap-7564025723254944482-0-39d4e713-69c8-49f9-ab8e-f887af4bcecb.avro differ