`CREATE TABLE` in a DataLakeCatalog over a REST catalog takes ~33 s whenever the namespace already exists

[Selfeer]

Creating a table in a DataLakeCatalog database backed by an Iceberg REST catalog is fast the first time a namespace is used and takes about 33 seconds every time after that. The statement succeeds in the end, and the server log shows the whole delay is spent re-sending one request, POST /v1/namespaces, which the catalog keeps answering with 409 Conflict because the namespace is already there. The same delay occurs on both creation forms the PR adds, the engine-less CREATE TABLE db.\ns.t` (...)andCREATE TABLE ... ENGINE = IcebergS3(...)`.

We found it while running the explicit engine create scenario of the suite: the first table in the namespace was created in under a second, the second one in 32.9 s.

Why we consider this a bug

• Existing namespaces are the normal case. Every table but the first in a namespace hits this path. A user creating ten tables in one namespace waits about five minutes for statements that should each take well under a second.
• A 409 is a definite answer, not a transient failure. The catalog says the namespace exists, which is exactly what createNamespaceIfNotExists wants. Retrying it ten times with growing backoff cannot change the outcome and only delays the statement.
• It logs a false error. A normal CREATE TABLE leaves an <Error> RestCatalog ... HTTP status code: 409 'Conflict' line in the server log, which will alarm anyone monitoring logs and hides real catalog errors among noise.
• The PR already solves this for Glue. GlueCatalog::createNamespaceIfNotExists in the same PR calls GetDatabase first and skips the create when the namespace exists, with the comment "must not be called when there is nothing to create". The REST implementation does not do the equivalent, so the two catalogs behave differently for the same statement.
• It is introduced by this PR. Before PR Native CREATE TABLE / DROP TABLE for DataLakeCatalog; optionally prune on DROP #2305 no CREATE TABLE path called createNamespaceIfNotExists on every statement.

Steps to reproduce

Environment: 26.6.2.20001.altinityantalya (PR #2305 build), release; ice-rest-catalog 0.16.0 on http://ice-rest-catalog:5000 (any Iceberg REST catalog that answers 409 for an existing namespace behaves the same); MinIO on http://minio:9000, bucket warehouse.

1. Create the database.

SET allow_experimental_database_iceberg = 1;
CREATE DATABASE lake
ENGINE = DataLakeCatalog('http://ice-rest-catalog:5000', 'admin', 'password')
SETTINGS catalog_type = 'rest',
         warehouse = 's3://bucket1/',
         storage_endpoint = 'http://minio:9000/warehouse',
         auth_header = 'Authorization: Bearer foo',
         default_base_location = 's3://warehouse/data';

2. Create the first table in a fresh namespace and note the time.

CREATE TABLE lake.`ns1.t1` (id Int64);
-- Ok. 0 rows in set. Elapsed: 0.4 sec.

3. Create a second table in the same namespace and note the time.

CREATE TABLE lake.`ns1.t2` (id Int64);
-- Ok. 0 rows in set. Elapsed: 32.9 sec.

4. Look at the server log for the second statement.

grep 'v1/namespaces' /var/log/clickhouse-server/clickhouse-server.log | tail -12

Expected: step 3 takes about as long as step 2, and the log shows no error.

Actual: step 3 takes ~33 s. The log shows ten attempts of the same request with backoff 0.1, 0.2, 0.4, 0.8, 1.6, 3.2, 6.4, 10, 10 s, then an error line, then the table creation continues:

14:31:52.130 <Trace> ReadWriteBufferFromHTTP: Failed to make request to 'http://ice-rest-catalog:5000/v1/namespaces'. Error: DB::HTTPException: Received error from remote server ... HTTP status code: 409 'Conflict'
14:31:52.233 <Trace> ReadWriteBufferFromHTTP: Failed to make request to 'http://ice-rest-catalog:5000/v1/namespaces'. ... 409 'Conflict'
14:31:52.436 ... 14:31:52.839 ... 14:31:53.643 ... 14:31:55.248 ... 14:31:58.454 ... 14:32:04.858 ... 14:32:14.861 (same line)
14:32:24.866 <Debug> ReadWriteBufferFromHTTP: Failed to make request to 'http://ice-rest-catalog:5000/v1/namespaces'. Error: 'DB::HTTPException: ... HTTP status code: 409 'Conflict', body: '{"error":{"message":"Namespace already exists: ns1 ...
14:32:24.867 <Error> RestCatalog(s3://bucket1/): Code: 86. DB::HTTPException: Received error from remote server http://ice-rest-catalog:5000/v1/namespaces. HTTP status code: 409 'Conflict' ...
14:32:24.867 <Trace> WriteBufferFromS3: Create WriteBufferFromS3 ... key data/ns1/t2/metadata/v1-....metadata.json

Related to: #2305
Build used: https://altinity-build-artifacts.s3.amazonaws.com/PRs/2305/6ec36d21729a28815041df233317bd454908c5a0/build_amd_release/clickhouse-common-static_26.6.2.20001.altinityantalya_amd64.deb

[subkanthi]

This is related to #1907