CAS: parallel delete blobs

[k-morozov]

Bench results:

   cas_gc_io_concurrency    pending_deletes       Throughput    Active GC time    Full drain time
  ━━━━━━━━━━━━━━━━━━━━━━━  ━━━━━━━━━━━━━━━━━  ━━━━━━━━━━━━━━━  ━━━━━━━━━━━━━━━━  ━━━━━━━━━━━━━━━━━
                       1            33.99 s      247 blobs/s           34.18 s             88.8 s
  ───────────────────────  ─────────────────  ───────────────  ────────────────  ─────────────────
                      16             2.90 s    2,900 blobs/s            3.10 s             53.3 s

Changelog category (leave one):

• Performance Improvement

Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):

Added cas_gc_io_concurrency to run blob deletes in the CAS GC pending_deletes phase in parallel.

Documentation entry for user-facing changes

...

CI/CD Options

Exclude tests:

• Fast test
• Integration Tests
• Stateless tests
• Stateful tests
• Unit tests
• Performance tests
• Aarch64 tests
• All with ASAN
• All with TSAN
• All with MSAN
• All with UBSAN
• All with Coverage
• All Regression
• Disable CI Cache

Regression jobs to run:

• Fast suites (mostly <1h)
• Aggregate Functions (2h)
• Alter (1.5h)
• Benchmark (30m)
• CAS (content-addressed storage; Antalya only)
• ClickHouse Keeper (1h)
• Iceberg (2h)
• LDAP (1h)
• OAuth (5m)
• Parquet (1.5h)
• RBAC (1.5h)
• SSL Server (1h)
• S3 (2h)
• S3 Export (2h)
• Swarms (30m)
• Tiered Storage (2h)

[github-actions]

Workflow [PR], commit [8661257]

[k-morozov]

@codex review

[chatgpt-codex-connector]

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-09-11T11:40:35.803992Z	8e0ec8d	Manual request

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 8e0ec8de87

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

Batch re-deletes across shard boundaries

When cas_gc_shards > 1, this invocation completes and waits for one shard before the loop reaches the next shard, so re-deletes from different shards never overlap. If each shard has fewer than cas_gc_redelete_min_batch_size pending entries, the entire phase remains sequential despite cas_gc_redelete_concurrency > 1; even larger per-shard batches cannot use more concurrency than one shard contains. Schedule the phase's eligible entries before a single wait so the configured fan-out applies across shards.

Useful? React with 👍 / 👎.

[k-morozov]

Default value gc_shards = 1 . Yes, in some cases we could obtain seq execution. But in these cases we don;t have a lot blobs. I suggest skip this impovment now.

[chatgpt-codex-connector]

Account for worker I/O in pending-delete metrics

When parallel re-delete is enabled, performRedeleteIo executes the blob HEAD and conditional DELETE on these pool threads, while GcPhaseTimer snapshots only the round thread's ProfileEvents. As a result, the pending_deletes row in system.cas_gc_log omits the underlying request and retry counters precisely when cas_gc_redelete_concurrency > 1, breaking per-phase operation accounting; collect the worker deltas or expose equivalent explicit phase metrics.

Useful? React with 👍 / 👎.

[k-morozov]

This is a regression only when the feature is enabled. It is a known limitation that read-ahead also has, and it will be fixed in a separate PR for both places at once.

[filimonov]

doc combining my + AI review https://gist.github.com/filimonov/a5377e47bf29f92fe12b345d10fe29ad

spec + plan to rework (you can pass it to AI agent) https://gist.github.com/filimonov/eb2ef24fbc8a49e2d3cbb43fe43795d8

[alsugiliazova]

PR #2351 Distributed-Systems Audit

AI audit note: This review was generated by AI (Claude Opus 5.5).

Field	Value
PR	Altinity/ClickHouse#2351 — CAS: parallel delete blobs
Base	antalya-26.6
Head	3a13b956ae19153434eea3e31e375639719204be
Scope	pending_deletes blob re-delete fan-out through the shared GC I/O pool; rename of cas_gc_read_concurrency to cas_gc_io_concurrency
Method	distributed-systems-audit, static review only
Related	PR-2351-CI-TRIAGE.md — CI verification of the same PR

Summary

2 confirmed defects, both Medium. No safety defects found. Fencing, the etag-conditional delete, and crash recovery behave as they did before the PR.

#	Severity	Fault class	Defect
1	Medium	Backpressure / partial failure	Re-delete no longer fails fast
2	Medium	Configuration & version skew	Released setting removed without an alias

Model assumptions

Actors	Per-server GC round thread (the GC-lease holder), GC heartbeat thread, up to cas_gc_io_concurrency re-delete workers, GcMetaWriter pool, the object store (S3, MinIO, RustFS), and writers on other nodes that may re-upload a condemned blob
Channels	HTTP to shared object storage (HEAD, then a DELETE conditional on the etag); the gc/state lease and its CAS commit; per-node mount-lease renewal on the same store
Failure model assumed by the code	Crash-recover. Replay is idempotent because every destructive step is justified by already-published delete_pending state. Object-store faults are transient, and each request is bounded by Retry::standard() (90 s).

Confirmed defects

1. Re-delete no longer fails fast

Severity: Medium · Fault class: Backpressure & resource limits / Partial & gray failure

Impact. While the object store is failing, one GC round works through its whole re-delete batch instead of stopping at the first error. The batch holds up to 5000 entries (cas_gc_round_redelete_budget), and each entry gets its own 90-second retry window. A single round can therefore run about 5000 / 16 × 90 s ≈ 7.8 h at the default concurrency, or about 125 h at cas_gc_io_concurrency = 1. For that whole time:

• GC stalls for the whole pool. The heartbeat keeps this node's GC lease, so no other node can take over.
• CasGcScheduler::stop() blocks in thread.join(), and so do server shutdown and SYSTEM ... GC STOP.
• The store, already degraded, receives up to about 16 × 5000 failing requests and one WARNING log line per entry. This competes with the node's mount-lease renewals.
• After all that, the round rethrows the first error and commits nothing.

Anchor.

File	Code path	What it does
Gc/CasGc.cpp	Gc::redeleteBlobs, sequential branch (~502–511)	Catches each exception and continues
Gc/CasGc.cpp	Gc::redeleteBlobs, parallel branch (~523–547)	Enqueues every entry and never checks earlier failures
Gc/CasGc.cpp	Gc::redeleteBlobs (~550–591)	Waits for all tasks, then rethrows the first error
Gc/CasGc.cpp	Gc::performRedeleteIo	Each HEAD and DELETE uses Retry::standard()
Backend/CasRetry.h	Retry::standard	within(90'000)
Gc/CasGcScheduler.cpp	CasGcScheduler::stop	Joins the round thread; nothing cancels a round in flight

Before this PR, the first op.head / op.remove exception left runRegularRound after a single retry window.

Trigger. The object store is unavailable or throttling (503 or connection refused) for longer than the retry window, while a shard has 17 or more delete_pending entries.

Why it's a defect. The worst-case round length, the outage load, and the time stop() and shutdown can block all grew from one retry window to N / concurrency windows. It also makes the PoolConfig comment false on the failure path: it says 1 is "the sequential round, request for request".

Fix direction. Stop at the first failure: set a shared flag that each task checks before it starts, and skip entries that haven't started. Alternatively, freeze one round-level deadline with CasOperation::freeze and share it across all re-delete tasks. Also check for a scheduler stop between entries.

Regression test direction. Use a backend that answers every HEAD with 503, and give the round 100 entries. Assert that the round takes about one policy window and sends no more than concurrency × attempts requests, at both cas_gc_io_concurrency = 1 and 16. Also assert that stop() returns within that bound.

2. Released setting removed without an alias

Severity: Medium · Fault class: Configuration & version skew

Impact. cas_gc_read_concurrency shipped in v26.6.4.20001.altinityantalya. During a rolling upgrade, what happens to a node depends on how its config spells the setting:

Spelling in config	After upgrade
cas_gc_read_concurrency	The CAS disk fails to load with UNKNOWN_SETTING, and the node doesn't come back.
gc_read_concurrency (unprefixed; still supported for other settings)	The setting is ignored with no error. A node pinned to 1 for safety silently switches to 16-way GC I/O, including the new parallel deletes.

The PR's docs say the old name is "rejected", which is only true for the prefixed spelling.

Anchor.

File	Code path	What it does
ContentAddressedSettings.cpp	loadFromConfig (~163–166)	An unknown cas_* key reaches throwSettingNotFound
ContentAddressedSettings.cpp	loadFromConfig (~140–148)	An unknown unprefixed key is treated as "belongs to another consumer" and ignored
ContentAddressedSettings.cpp	DECLARE list	gc_read_concurrency renamed to gc_io_concurrency

Trigger. Upgrade a node from v26.6.4.20001.altinityantalya while either spelling of the old setting is in storage_configuration.

Why it's a defect. A released setting now either stops the node at startup or is silently ignored, depending on spelling. The silent case contradicts the documented behaviour.

Fix direction. Accept cas_gc_read_concurrency and gc_read_concurrency as a deprecated alias that logs a warning for one release cycle, consistent with the existing unprefixed-spelling policy. At minimum, reject both spellings the same way.

Regression test direction. In gtest_cas_settings, load a config with each old spelling. Assert that it maps to gc_io_concurrency with a warning, or that both throw the same error.

Coverage summary

Fault class	Status	Notes
Network / transport	Reviewed	Per-request retry and error classification unchanged. The fail-fast loss is defect 1.
Partition / availability	Reviewed	GC-lease ownership, heartbeat and the open GC fence unchanged. Workers call resume(admitted_generation) with no liveness predicate, exactly as the round's own admit() at CasGc.cpp:642 does, so fencing is not weakened.
Timing / ordering	Reviewed	Outcomes, audit rows and meta deletes are applied on the round thread in entry order after the I/O completes.
Partial / gray failure	Reviewed	Defect 1
Idempotency & duplicates	Reviewed	HEAD followed by an etag-conditional DELETE per entry is unchanged. A replay after a crash, a lost lease or an ambiguous timeout resolves to Gone or Absent.
Backpressure & resource limits	Reviewed	Defect 1
Recovery & rollback	Reviewed	A crash with up to 16 deletes in flight leaves the entries delete_pending, and the next round records them as Absent. gc/state is not committed on failure.
Configuration & version skew	Reviewed	Defect 2
Security / trust	N/A	No authentication or authorization paths touched
Observability	Reviewed	The new redelete_failed counter and the jobs_scheduled / jobs_failed phase metrics are good. The per-entry WARNING flood is covered under defect 1.

Deferred. Can one shard's merge.redelete list contain the same blob key twice? If it can, two concurrent conditional DELETEs would classify the loser as Replaced rather than Gone. That would skip the meta delete and mislabel the audit row. Settling it needs the fold's deduplication logic, which was not read.

Limits. Static review of the PR head 3a13b956 covering Gc/CasGc.cpp, Gc/CasGc.h, Gc/CasGcScheduler.*, Pool/CasPool.h, Backend/CasRequests.*, Backend/CasRetry.h and ContentAddressedSettings.cpp. No runtime reproduction, and the new gtest suite was not re-run. The broader audit-review skill that distributed-systems-audit refers to doesn't exist in this repo, so local C++ concurrency bugs were only considered where they cause cross-node harm.

[alsugiliazova]

PR #2351 CI Verification Report

Verification (2026-09-24)

Field	Value
PR	Altinity/ClickHouse#2351 — CAS: parallel delete blobs
Base	antalya-26.6
Head	3a13b956ae19153434eea3e31e375639719204be
Run	35341493723 (4 attempts, 2026-09-18 → 2026-09-21)
CI report	ci_run_report.html
Method	pr-ci-failure-triage + DB rates (gh-data.checks, gh-data.clickhouse_regression_results) + artifact and server-log reads

Verdict

No failure is caused by this PR. CI can be approved. Rebase onto antalya-26.6 to pick up #2406, which fixes the two integration shards.

9 checks fail, plus the aggregate PR status:

Category	Checks	Blocks #2351?
regression	0	—
unknown	0	—
infrastructure	4 — Grype keeper, Grype server-alpine, Integration amd_tsan 5/6, Integration amd_asan_ubsan, db disk, old analyzer, 1/8	No
pre-existing-flaky	5 — Stateless amd_tsan, cas s3 storage, parallel, 2/2, cas_alter_attach_3, cas_s3_cache_alter_attach_3, cas_selects, cas_lightweight_delete_4	No
cascade	Parent features and modules in the regression reports fail only because a scenario under them failed	—

All failing jobs failed the same way on every attempt. Rerunning them won't turn them green.

Still red on this run — categorized

Cluster	Category	Blocks #2351?	Notes
Grype clickhouse-keeper, clickhouse-server-alpine	infrastructure	No	CVE-2026-85091 (High) in the Alpine package zlib 1.3.2-r0 in the base image. The same two scans fail on #2427, #2420, #2419 and #2414. Only #2398 passes, and it was scanned before the CVE was published.
Integration amd_tsan 5/6, amd_asan_ubsan, db disk, old analyzer, 1/8	infrastructure	No — rebase	Failed to pre-pull Docker images, caused by pull access denied for minio/minio, minio/mc: repository does not exist. Failed identically on attempts 1–3. #2406 (merged 2026-09-21) moves the images to clickhouse/minio-* mirrors. This branch forked on 2026-09-18, before that merge.
Stateless amd_tsan, cas s3 storage, parallel, 2/2	pre-existing-flaky	No	See Stateless tsan CAS shard.
cas_alter_attach_3, cas_s3_cache_alter_attach_3	pre-existing-flaky	No	See Alter attach part 3.
cas_selects	pre-existing-flaky	No	See cas_selects.
cas_lightweight_delete_4	pre-existing-flaky	No — watch after rebase	See cas_lightweight_delete_4. Weakest verdict in this report.

Pre-existing flaky — evidence

Rates are runs failed / runs over the last 45 days on 26.6, x86_64. "Branch" means antalya-26.6 MasterCI runs, "other PRs" means every other PR run, and "this PR" covers all four commits of #2351.

Stateless tsan CAS shard

Field	Finding
Failing tests	03156_tuple_map_low_cardinality, 02833_concurrent_sessions
Cause	The test logic passed. Both failed only because the server printed ConnectionGroup: Too many active sessions in group Disk, count 8200, warning limit 8000 into the test's stderr.
Warning rate (tsan CAS stateless runs)	Branch 3/40, other PRs 2/82, this PR 2/18
Link to the PR	The PR adds at most 16 concurrent GC requests, against an 8000-session warning threshold.

Alter attach part 3

Field	Finding
Errors	CAS write could not be committed ... persistent CAS contention, the checkpoint contribution was not published and ... gave up at the policy deadline (NETWORK_ERROR)
Mechanism	Every failing scenario falls inside a two-minute burst of 503 Service Unavailable from RustFS (16:29–16:30 server time, 14:27–14:31 UTC in the test log). None of the three nodes logged a failed GC delete (pending delete of blob ... failed).
History	Both signatures appear on the branch and on 21 other PR runs, and predate this PR (first seen 2026-09-05).
Feature-level rates	Branch about 20–30%, other PRs about 40%, this PR 20–60% (5 runs)

Suite	Branch	Other PRs	This PR
cas_alter_attach_3	4/6	16/27	1/2
cas_s3_cache_alter_attach_3	1/5	12/26	2/2

cas_selects

Field	Finding
Error	content-addressed disk 'cas_disk' -- mount lease not held ... TRANSIENT unavailability (NETWORK_ERROR)
Mechanism	About 76 concurrent JOIN queries used up the server's local ports to MinIO (Cannot assign requested address, errno 99, starting 11:09:36). Mount-lease renewal then failed 65 times within 4 seconds, and the disk stopped accepting operations.
Link to the PR	GC's pending-delete path wasn't active at the time. Only orphan-manifest sweeps ran, and they used the same thread pool before this PR.
Suite rate	Branch 4/5, other PRs 28/32, this PR 3/3
History	Signature seen since 2026-08-20, on 9 branch runs and 42 other-PR runs. The same class is tracked for #2300 as #2332.

cas_lightweight_delete_4

Field	Finding
Failing scenario	/lightweight delete/concurrent delete/MergeTree/random delete half of the table without overlap
Assertion	SELECT count() returned 512 where 500 was expected, so 12 rows were not deleted.
Scenario rate	Branch 3/15 (510, 503 and 501 rows left), other PRs 1/32, this PR 3/4
Family rate (all "without overlap" scenarios)	Branch about 12%, this PR about 19% (small sample)
Link to the PR	Leftover rows mean a delete didn't take effect. GC only removes blobs nothing references, so it can't bring deleted rows back.
Caveat	Weakest verdict here. If it keeps failing more often than on the branch after a rebase, investigate it on its own.

Code-review notes (not CI)

These don't affect the CI verdict.

• The changelog entry names cas_gc_redelete_concurrency, but the code adds cas_gc_io_concurrency.
• cas_gc_read_concurrency shipped in v26.6.4.20001.altinityantalya and is removed with no alias.
  • A config that still sets it fails with UNKNOWN_SETTING when the disk loads.
  • The unprefixed spelling gc_read_concurrency is ignored with no error.
• Pending deletes now run 16 at a time by default, and a failed delete no longer stops the batch.

Recommendations

1. Rebase onto antalya-26.6 to pick up #2406, then rerun the integration shards.
2. Track the Grype CVE on the base image separately. It affects every 26.6 PR.
3. After the rebase, check that cas_lightweight_delete_4 no longer fails more often here than on the branch.