Antalya 26.8: auto-grp-pr-2141: combined port of 13 PRs - #2399
Merged
Merged
Conversation
…next commit) --- Original cherry-pick message follows: Merge pull request #2141 from Altinity/feature/antalya-26.6/auto-grp-pr-1662 Antalya 26.6: Expose IcebergS3 partition_key / sorting_key in system.tables # Conflicts: # src/Storages/ObjectStorage/DataLakes/Iceberg/IcebergMetadata.h
…next commit) --- Original cherry-pick message follows: Merge pull request #2145 from Altinity/feature/antalya-26.6/auto-grp-pr-1687 Antalya 26.6: Cluster request improvements + Iceberg/Parquet read fixes # Conflicts: # programs/server/Server.cpp # src/Common/ProfileEvents.cpp # src/Core/Settings.cpp # src/Disks/DiskObjectStorage/ObjectStorages/IObjectStorage.h # src/Interpreters/ClusterFunctionReadTask.cpp # src/QueryPipeline/RemoteQueryExecutor.cpp # src/QueryPipeline/RemoteQueryExecutorReadContext.cpp # src/Storages/ObjectStorage/DataLakes/Iceberg/ManifestFileIterator.cpp # src/Storages/ObjectStorage/StorageObjectStorageSource.cpp # src/Storages/ObjectStorage/StorageObjectStorageStableTaskDistributor.cpp # tests/integration/test_storage_s3/test.py # tests/queries/0_stateless/03413_experimental_settings_cannot_be_enabled_by_default.sql
Kept both sides where antalya-26.8 and the port add independent lines (ProfileEvents rows, Settings declarations, obsolete settings, test configs / experimental-settings exception list, `got_data_from_replica` plus `replica_has_processed_data`, `read_source_index` plus `file_meta_info`). Dropped the cherry-pick's stale duplicates of code that antalya-26.8 already has elsewhere: the second copy of the shutdown `SCOPE_EXIT_SAFE` block in Server.cpp (only `global_context->stopSwarmMode()` was added into the existing block) and the duplicated `namespace ErrorCodes` block in StorageObjectStorageSource.cpp. Adapted: moved `deserializeFieldFromBinaryRepr` out of ManifestFileIterator.cpp into Iceberg::IcebergFieldParseHelpers as the port does, but carrying antalya-26.8's body (Decimal128/256 support added by acb870c) instead of the port's older implementation. Adapted: used antalya-26.8's `getSchedulingIdentifier()` helper (added by cbd81e0) for the `unprocessed_files` keys and log message in StorageObjectStorageStableTaskDistributor instead of the port's raw path expressions. Adapted: kept the `AsyncTaskExecutor` name argument required on antalya-26.8 in RemoteQueryExecutorReadContext's constructor. Source-PR: #2145 (#2145)
…next commit) --- Original cherry-pick message follows: Merge pull request #2146 from Altinity/feature/antalya-26.6/auto-grp-pr-1718 Antalya 26.6: Partition export + cluster functions # Conflicts: # ci/jobs/scripts/integration_tests_configs.py # docs/reference/engines/database-engines/datalake.mdx # src/CMakeLists.txt # src/Common/ErrorCodes.cpp # src/Common/ProfileEvents.cpp # src/Common/setThreadName.h # src/Core/ServerSettings.cpp # src/Core/Settings.cpp # src/Core/Settings.h # src/Databases/DataLake/DatabaseDataLake.cpp # src/Databases/DataLake/DatabaseDataLake.h # src/Databases/DataLake/DatabaseDataLakeSettings.cpp # src/Databases/DataLake/GlueCatalog.cpp # src/Databases/DataLake/ICatalog.h # src/Databases/DataLake/RestCatalog.cpp # src/Databases/DataLake/RestCatalog.h # src/Databases/DataLake/S3TablesCatalog.cpp # src/Databases/DataLake/UnityCatalog.h # src/Disks/DiskObjectStorage/ObjectStorages/AzureBlobStorage/AzureObjectStorage.cpp # src/Interpreters/IcebergMetadataLog.h # src/Interpreters/InterpreterSystemQuery.cpp # src/Interpreters/PreparedSets.cpp # src/Interpreters/PreparedSets.h # src/Parsers/ASTSetQuery.cpp # src/Parsers/ASTSystemQuery.h # src/Parsers/FunctionSecretArgumentsFinder.cpp # src/Planner/Planner.cpp # src/Processors/QueryPlan/ReadFromRemote.cpp # src/Storages/IStorage.h # src/Storages/IStorageCluster.cpp # src/Storages/MergeTree/MergeTreeData.cpp # src/Storages/MergeTree/MergeTreeData.h # src/Storages/ObjectStorage/Azure/Configuration.cpp # src/Storages/ObjectStorage/DataLakes/DataLakeConfiguration.h # src/Storages/ObjectStorage/DataLakes/Iceberg/IcebergMetadata.cpp # src/Storages/ObjectStorage/DataLakes/Iceberg/IcebergMetadata.h # src/Storages/ObjectStorage/DataLakes/Iceberg/IcebergWrites.cpp # src/Storages/ObjectStorage/DataLakes/Iceberg/IcebergWrites.h # src/Storages/ObjectStorage/DataLakes/Iceberg/ManifestFileIterator.cpp # src/Storages/ObjectStorage/DataLakes/Iceberg/MultipleFileWriter.cpp # src/Storages/ObjectStorage/DataLakes/Iceberg/Mutations.cpp # src/Storages/ObjectStorage/DataLakes/Iceberg/PersistentTableComponents.h # src/Storages/ObjectStorage/DataLakes/Iceberg/SchemaProcessor.cpp # src/Storages/ObjectStorage/DataLakes/Iceberg/SchemaProcessor.h # src/Storages/ObjectStorage/DataLakes/Iceberg/StatelessMetadataFileGetter.cpp # src/Storages/ObjectStorage/DataLakes/Iceberg/Utils.cpp # src/Storages/ObjectStorage/DataLakes/Iceberg/tests/gtest_iceberg_schema_processor.cpp # src/Storages/ObjectStorage/ReadBufferIterator.cpp # src/Storages/ObjectStorage/StorageObjectStorage.cpp # src/Storages/ObjectStorage/StorageObjectStorageConfiguration.cpp # src/Storages/ObjectStorage/StorageObjectStorageConfiguration.h # src/Storages/ObjectStorage/StorageObjectStorageSource.cpp # src/Storages/ObjectStorage/registerStorageObjectStorage.cpp # src/Storages/ObjectStorageQueue/StorageObjectStorageQueue.cpp # src/Storages/ObjectStorageQueue/registerQueueStorage.cpp # src/Storages/StorageDistributed.cpp # src/Storages/StorageMergeTree.cpp # src/Storages/System/StorageSystemIcebergHistory.cpp # src/Storages/System/attachSystemTables.cpp # src/Storages/buildQueryTreeForShard.cpp # src/Storages/buildQueryTreeForShard.h # src/TableFunctions/TableFunctionObjectStorage.cpp # src/TableFunctions/TableFunctionObjectStorageCluster.cpp # src/TableFunctions/TableFunctionRemote.h # tests/integration/test_database_iceberg/test.py # tests/queries/0_stateless/01625_constraints_index_append.reference # tests/queries/0_stateless/03413_experimental_settings_cannot_be_enabled_by_default.sql # tests/queries/0_stateless/03745_system_background_schedule_pool.reference
Kept antalya-26.8 behaviour everywhere the cherry-pick only carried pre-existing antalya-26.6 code, and translated the PR's real changes to the current shape of antalya-26.8: - Append-only registries (ErrorCodes, ProfileEvents, setThreadName, Settings/ServerSettings, obsolete settings): kept only the rows the PR adds, renumbering the new error codes after the ones antalya-26.8 already uses. - `IcebergSchemaProcessor`: took the PR's context-carrying signatures (`addIcebergTableSchema(ptr, context)`, `getSchemaTransformationDagByIds(context, ...)`, `getSimpleType(name, context, ...)`) and kept antalya-26.8's `getClickHouseTableSchemaById` spelling, adapting the call sites accordingly. - `generateManifestFile`: appended the PR's `per_file_stats` after the extra defaulted parameters antalya-26.8 added, and named the intermediate defaults at the PR's call site. - `StorageObjectStorageConfiguration::initialize` became an instance method, so the remaining static call sites (StorageURL, TableFunctionURL, registerQueueStorage) were converted. - `RestCatalog`/`DeltaSharingCatalog`/`HorizonCatalog` (antalya-26.8-only subclasses) gained the `namespaces` constructor argument the PR adds to the base catalog. - `MergeTreeData::changeSettings`: dropped the now-unused `has_storage_policy_changed` flag, since the PR replaced the conditional `startBackgroundMovesIfNeeded()` with an unconditional `startBackgroundMoves()` (exports run on the moves assignee). - `registerTableFunctionObjectStorage` / `registerTableFunctionIceberg` / `registerTableFunctionDeltaLake` / `registerTableFunctionHudi`: dropped the registrations the PR moved into the new `TableFunctionObjectStorageClusterFallback`, keeping the antalya-26.8 documentation for the functions that stay. - Dropped the parts of "theirs" that antalya-26.8 already implements in another shape (lazy `insertRowToLogTable`, catalog-aware `getLatestMetadataFileAndVersionWithCatalog`, the parallel-replicas branch in `DatabaseDataLake::tryGetTableImpl`, the `RewriteInToGlobalInVisitor` copy in StorageDistributed) and the antalya-26.6-only test bodies that do not apply here (`test_partitioning_by_time`, the second result block of 01625_constraints_index_append). Source-PR: #2146 (#2146)
…next commit) --- Original cherry-pick message follows: Merge pull request #2201 from Altinity/feature/antalya-26.6/pr-1748 Antalya 26.6: Fix file identifier in rescheduleTasksFromReplica # Conflicts: # src/Storages/ObjectStorage/StorageObjectStorageStableTaskDistributor.cpp
antalya-26.8 already carries the file-identifier unification the source PR introduces as StorageObjectStorageStableTaskDistributor::getFileIdentifier: the free function getSchedulingIdentifier(object_info, send_over_whole_archive) is already used at all call sites, including rescheduleTasksFromReplica, and uses the newer archive identifier scheme from "Preserve URL shard identity in object tasks" (cbd81e0). Kept "ours" at every conflict and routed the source PR's new code through the existing helper. Adapted: source PR's getFileIdentifier(file) call in getNextTask routed to the equivalent getSchedulingIdentifier(file, send_over_whole_archive) already on antalya-26.8; the duplicate helper and its header declaration were removed. Source-PR: #2201 (#2201)
…next commit) --- Original cherry-pick message follows: Merge pull request #2249 from Altinity/fix/join-filter-pushdown-through-rename Fix join filter pushdown # Conflicts: # src/Analyzer/Utils.cpp # src/Processors/QueryPlan/Optimizations/filterPushDown.cpp
Kept antalya-26.8's ANY INNER JOIN pushdown guard (used later in the function) after the new canPrefilterJoinSide assignments; dropped the logical_join Left/Right lines the PR replaces. Used std::deque<QueryTreeNodePtr> instead of the QueryTreeNodesDeque alias, which does not exist on antalya-26.8, and dropped the Core/Streaming/CursorTree_fwd.h include (context from the source branch, not part of the PR diff; the header does not exist here). Source-PR: #2249 (#2249)
zvonand
added
releasy
# Conflicts: # src/Databases/DataLake/DatabaseDataLake.cpp # src/Interpreters/InterpreterOptimizeQuery.cpp # src/Storages/ObjectStorage/DataLakes/DataLakeConfiguration.h # src/Storages/ObjectStorage/DataLakes/Iceberg/IcebergWrites.cpp # src/Storages/ObjectStorage/DataLakes/Iceberg/IcebergWrites.h # src/Storages/ObjectStorage/DataLakes/Iceberg/ManifestFileIterator.cpp # src/Storages/ObjectStorage/StorageObjectStorage.cpp # src/Storages/ObjectStorage/StorageObjectStorage.h # src/Storages/ObjectStorage/StorageObjectStorageCluster.cpp # src/Storages/ObjectStorage/StorageObjectStorageCluster.h # src/Storages/StorageFileCluster.cpp # src/Storages/StorageFileCluster.h # src/Storages/System/StorageSystemIcebergFiles.cpp # src/Storages/buildQueryTreeForShard.cpp # tests/integration/test_database_iceberg/test.py # tests/queries/0_stateless/03413_experimental_settings_cannot_be_enabled_by_default.sql
# Conflicts: # src/Common/ProfileEvents.cpp # src/Databases/DataLake/UnityCatalog.cpp # src/Parsers/ASTSystemQuery.cpp # src/Storages/ObjectStorage/DataLakes/Common/AvroForIcebergDeserializer.cpp # src/Storages/ObjectStorage/DataLakes/Iceberg/Snapshot.h # src/Storages/ObjectStorage/StorageObjectStorageSource.cpp
Restore the setting name in the `object_storage_cluster` entry of the
`25.8.16.20001.altinityantalya` block, which was blanked to `{"", "", "",
"Antalya: New setting"}`. An empty name is reported by
`03999_stateless_settings_history` both as a dangling history entry and as a
missing entry for `object_storage_cluster`.
Drop the duplicate `export_merge_tree_partition_max_retries` entry from the
`26.6` block and keep the oldest one, in `25.8.16.20001.altinityantalya`,
where the setting was first introduced.
`05057_file_rename_after_processing_write_grant`: the port of #2146 rewrote `IStorageCluster` and dropped the `updateBeforeRead` hook, which carried the initiator-side `WRITE ON FILE` check of `StorageFileCluster` for `rename_files_after_processing`. Only the check in `getTaskIteratorExtension` was left, and `EXPLAIN PLAN optimize = 0` builds no task iterator, so the rule was not enforced there. Restore the hook and call it from `IStorageCluster::read`. `01625_constraints_index_append`: #2146 hashes `settings_changes` in `FunctionNode::updateTreeHashImpl`, which changes the tree hash of every function node. CNF orders atoms by that hash, so the arguments of `and` swap. Update the reference, as was done for the same change in antalya-26.6. `04306_create_handler_http`: `remote(host, <table function>)` on a single shard is sent as an initial query with `interface` forced to `TCP`, because `validateClientInfo` requires the interface of a non-secondary query to match the session. `ClientInfo::write` serializes `http_handler_name` and `http_request_url` only for the `HTTP` interface, so `currentHandler` and `currentRequestURL` are empty on the shard. Read a view through `remote('127.0.0.2', db, table)` instead, which keeps the remote hop and the assertion while staying a secondary query. CI report: https://altinity-build-artifacts.s3.amazonaws.com/json.html?PR=2399&sha=d61911997bd6a66fd1d758c117f15ab225d0bc5d&name_0=PR&name_1=Fast+test&name_2=Tests Related: #2399
The cherry-pick of Altinity PR #2146 added `findIcebergStorageType` to `src/Parsers/FunctionSecretArgumentsFinder.cpp`, and it consults `NamedCollectionFactory`. `src/Parsers/*.cpp` is globbed wholesale into the standalone WebAssembly parser, which links no named collection machinery, so `wasm-ld` failed with undefined `DB::NamedCollectionFactory::instance`, `DB::NamedCollectionFactory::tryGet`, `DB::NamedCollection::has` and `DB::NamedCollection::get<std::string>`. Guard the lookup with `CLICKHOUSE_PARSER_MINIMAL_BUILD`, the macro that `utils/wasm-parser` already defines and that `src/Common/ErrorCodes.cpp`, `src/Common/StackTrace.h` and `base/base/demangle.cpp` use for the same purpose. The WebAssembly build has no named collection registry, so `tryGet` could only ever return null there and the storage type stays at its `s3` default; the server build is unaffected, because the macro is undefined for it. Both configurations that the job builds were configured, built and run through `ctest` locally in `altinityinfra/wasm-builder`, and both stay under their size ceilings. CI report: https://altinity-build-artifacts.s3.amazonaws.com/json.html?PR=2399&sha=f2807ce47f87e01e7a6746b16c587c734d9e6f88&name_0=PR&name_1=Build%20%28wasm_parser%29 Related: #2399
…ned 26.8 port `04146_iceberg_orc_row_policy_prewhere`, `04147_iceberg_orc_schema_evolution_row_policy`: the port of #2146 makes `createStorageObjectStorage` return a `StorageObjectStorageCluster` for every object storage engine, and `IStorageCluster::isRemote` is `final` and returns true. The planner gated the row policy push-down on `!isRemote`, so the policy was no longer lowered into `query_info.row_level_filter` and was appended as a `FilterStep` above the read instead. When `PREWHERE` removes the column the policy reads, that step cannot evaluate it and the query fails with `NOT_FOUND_COLUMN_IN_BLOCK`. Add `IStorage::appliesRowLevelFilterInRead`, defaulting to `!isRemote` so every other storage is unchanged, and override it in `StorageObjectStorageCluster` for the queries it serves from `pure_storage`, whose read does lower the filter. `04627_object_storage_lazy_hive_partitioning`, `04909_s3_globbed_path_implicit_partition_strategy`, `03363_hive_style_partition`: the `StorageObjectStorageCluster` constructor resolves the hive partitioning sample path eagerly and without a handler, so `CREATE` and `ATTACH` list the object storage. They hang on an unreachable endpoint, and an `AccessDenied` or an `INCORRECT_DATA` from a schema of partition columns only propagates out of `CREATE`. Carry the deferral of `StorageObjectStorage`: defer for a non-table-function whose columns and format are known, guard the eager call the same way, and delegate `updateExternalDynamicMetadataIfExists` to `pure_storage` so the resolution runs on the first use of the table. A clustered read builds its file iterator from `hive_partition_columns_to_read_from_file_path`, so refresh it once the deferred resolution has filled it in. `04909_s3_globbed_path_implicit_partition_strategy` also lost the `none` partition strategy across `DETACH`/`ATTACH`, because the `addInferredEngineArgsToCreateQuery` of the cluster storage does not persist an implicit `partition_strategy = 'none'` the way its non-cluster counterpart does, and `initPartitionStrategy` then resolves a non-globbed path to `hive` on reload. Delegate to `pure_storage`, which shares the configuration. `04512_every_database_engine_must_have_documentation`: the `Iceberg` database engine is registered without documentation. Register it with a description of what it is, the legacy name of `DataLakeCatalog`. `Parser memory check`: the `EXPORT PART` and `EXPORT PARTITION` syntax adds `to_table_function` and `partition_by_expr` to `ASTAlterCommand`, growing it from 504 to 520 bytes and moving its allocation from the 512-byte jemalloc size class into the 640-byte one, which reports a flat +128 bytes on every `ALTER`. Move `replace` next to the other flags, into the padding that follows them, so that `sizeof(DB::ASTAlterCommand)` is 512 again, the size class of `master`. All six tests pass locally. Of the 158 tests matching `hive`, `row_policy`, `cluster_wrap`, `replace_partition`, `attach_partition`, `partition_strategy` and `object_storage`, 139 pass and the rest need a second server, the S3 and Azure storage policies, or binaries that the local run does not have. CI report: https://altinity-build-artifacts.s3.amazonaws.com/json.html?PR=2399&sha=601848e8ac02890a4ea59f6bf42c09be79883dba&name_0=PR Related: #2399
Member
Author
This comment was marked as outdated.
This comment was marked as outdated.
…ables Restore `StorageObjectStorageCluster::write` (from the upstream backport of ClickHouse#119214) that was lost while resolving the port: direct inserts such as `INSERT ... VALUES` and async inserts are written from the initiator via `pure_storage`, instead of reaching `IStorageCluster::write`, which throws `Method write is not supported` when a cluster name is set. Fixes `test_database_iceberg/test.py::test_cluster_insert` and `test_storage_iceberg_no_spark/test_writes_parallel_replicas_no_catalog.py`. #2399 (comment) #2399
Member
Author
Collaborator
CI triage — 3 red checks, 0 caused by this PRSummary: all three failures are pre-existing/infra, none are related to the code in this port. The actual test matrix is green.
1. SQLLogic test — timeout, not PR-causedThe praktika report records the failure as a plain wall-clock timeout, with no failed statements: All three phases (self-test, statements-test, complete-test) were launched; the job was killed at the 3h limit before finishing. There are no recorded query/assertion failures. Evidence it's not this PR: the merged sibling combined-port #2379 (same base branch, same CI profile) passed SQLLogic in 2h26m. The suite simply runs close to the 3h cap on the Next step: re-run the SQLLogic job — it should pass. If it times out again, the fix is a CI-infra change, not a code change: bump the SQLLogic job timeout or shard the 2. Grype scans — pre-existing base-image CVEsBoth scans report "Completed with 1 high/critical vulnerabilities" — one for Evidence it's not this PR: the merged sibling #2379 had the exact same four Grype failures ( Next step: none for this PR. Handle via the usual base-image bump / CVE-waiver flow. PR health checkThe functional matrix is fully green on Net: this port is CI-clean — the only red is a timeout flake on a suite that passes on sibling ports, plus the standing image-CVE scans that don't block sibling combined-port merges. Recommend re-running SQLLogic; no code fix needed. 🤖 Triaged from the praktika S3 report ( |
zvonand
added
verified
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Changelog category (leave one):
Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):
Add support for exporting MergeTree parts and partitions to object storage, including experimental Apache Iceberg exports, with a command to schedule every partition at once, automatic type casting as in INSERT ... SELECT, retries with backoff, flexible column and partition-expression matching between source and destination, and a persistent export history in system.replicated_partition_exports. Also add alternative cluster table function syntax with distributed JOIN execution on swarm nodes, expose Iceberg partition and sorting keys in system.tables, allow cluster discovery settings to be reloaded from remote_servers without a restart, and fix Iceberg and Parquet read problems, JOIN filter pushdown, and incomplete data when a multi-file partition export resumes after a restart.
Add support for exporting MergeTree parts and partitions to object storage, including experimental Apache Iceberg exports, with a command to schedule every partition at once, automatic type casting as in INSERT ... SELECT, retries with backoff, flexible column and partition-expression matching between source and destination, and a persistent export history in system.replicated_partition_exports. Also add alternative cluster table function syntax with distributed JOIN execution on swarm nodes, expose Iceberg partition and sorting keys in system.tables, allow cluster discovery settings to be reloaded from remote_servers without a restart, and fix Iceberg and Parquet read problems, JOIN filter pushdown, and incomplete data when a multi-file partition export resumes after a restart (#2141 by @zvonand, #2145 by @zvonand, #2146 by @zvonand, #2200 by @zvonand, #2201 by @zvonand, #2209 by @zvonand, #2210 by @zvonand, #2218 by @k-morozov, #2229 by @k-morozov, #2253 by @arthurpassos, #2249 by @ianton-ru, #2197 by @ianton-ru, #2294 by @arthurpassos).
CI/CD Options
Exclude tests:
Regression jobs to run:
Combined port of 13 PR(s) (group
auto-grp-pr-2141). Cherry-picked from #2141, #2145, #2146, #2200, #2201, #2209, #2210, #2218, #2229, #2253, #2249, #2197, #2294.