S3_ERROR is not actionable, and CAS compare-and-swap contention has no first-class metric

[BorisTyshkevich]

S3_ERROR is not actionable, and CAS contention has no first-class metric

Build: 26.6.4.20001.altinityantalya
Environment: 2-replica ReplicatedMergeTree, CAS as the default storage policy, AWS S3 us-east-1
Observed: 2026-09-17, cluster healthy throughout (0 anomalies, 0 dangling_access, replication clean)

What happened

A routine review flagged 31,489 S3_ERROR (code 499) and attributed them to "412
PreconditionFailed responses from the CAS state plane — lost compare-and-swap races on
ref-log checkpoint objects."

That attribution is wrong, and the way the counters are exposed makes it a very easy mistake
to make. The real lost-CAS-race count on that replica is 1,390, not 31,489 — off by 23x.

Measurements

Cumulative since start (~24.6 h uptime), both replicas:

counter	replica 0-0 (GC leader)	replica 0-1
system.errors S3_ERROR (code 499)	118,692	31,494
ReadBufferFromS3RequestsErrors	118,389	28,610
S3ReadRequestsErrors	298,032	181,390
S3SingleAttemptRetryConsultations	300,117	187,339
S3WriteRequestsErrors	747	1,266
S3ReadRequestsThrottling	284	1,637
CASConditionalWriteAttempts	1,920,981	1,523,703
CASConditionalWriteCommitted	1,920,197	1,522,313
lost CAS races (attempts − committed)	784 (0.041%)	1,390 (0.091%)

Three things follow directly:

1. S3_ERROR tracks ReadBufferFromS3RequestsErrors, not the write path. It exceeds
   total write errors by 160x on 0-0, so it cannot be dominated by conditional-write 412s.
2. Retry consultations ≈ read errors on both replicas, i.e. essentially every one of
   these went through the retry policy and recovered. They are transient, not failures.
3. CAS compare-and-swap contention is healthy — 0.04–0.09% lost races. That is the
   number someone reviewing "is the state plane contended?" actually wants, and nothing
   surfaces it.

Why the wrong conclusion was easy to reach

• system.errors keeps only last_error_message. On replica 0-1 that message is
  Poco::Exception … Timeout; on 0-0 it is PreconditionFailed. The same counter, same
  code, two different labels — whichever error fired last. Reading the counter's meaning
  off its last message is the natural move and gives the wrong answer.
• text_log cannot corroborate it either. Replica 0-1 has zero PreconditionFailed
  lines in its entire retained log against a counter of 31,494 — these are retried below the
  logging threshold. So log-based attribution is impossible in either direction.
• The name S3_ERROR invites alerting. A counter in the hundreds of thousands, named
  "error", with no adjacent "…and here is how many actually failed", will be escalated.

Requests

Core ClickHouse (not CAS-specific)

1. Separate retried-and-recovered from terminal failures. The plumbing exists —
S3SingleAttemptRetryConsultations already shows these were retried. What an operator needs
is "requests that ultimately failed after retries", which today is not distinguishable from
"attempts that hiccuped and succeeded". This is the root cause of the false alarm.

2. Document or split system.errors semantics. A counter that aggregates several causes
while exposing only the most recent message is not diagnosable. Either state plainly that it
counts occurrences rather than failures, or split by cause.

CAS

3. Expose compare-and-swap contention as a first-class metric.
CASConditionalWriteAttempts − CASConditionalWriteCommitted is the meaningful signal for
"the state plane is contended". Today it must be derived by hand from two counters, which is
exactly why a reviewer reached for S3_ERROR instead. A ratio or a dedicated
CASConditionalWriteRejected counter would make the healthy case obviously healthy and the
unhealthy case obviously unhealthy.

Explicitly NOT requested: lowering the log level

The first instinct was to ask for quieter logging. That would make this worse. These
conditions are already barely logged — one replica has none at all — and the alarm came from
a counter, not from logs. Reducing verbosity removes the diagnostic trail while leaving the
alarming counter untouched.

Side observation worth keeping

S3ReadRequestsThrottling reads 1,637 on replica 0-1 (284 on 0-0) — real AWS-side
throttling on a cluster where none was assumed. It is small, but it is a genuine signal
currently buried in the same undifferentiated population. It is an argument for better
separation of error classes, not for suppressing them.

What was verified vs inferred

Verified: all counter values above, the correlation between S3_ERROR and
ReadBufferFromS3RequestsErrors, the absence of PreconditionFailed in replica 0-1's
text_log, and the CAS attempt/commit delta.

Not determined: the precise breakdown of the ~298k/181k read errors by HTTP status. An
earlier hypothesis that they were 404 existence probes was not supported once
ReadBufferFromS3RequestsErrors was measured — that counter covers object data reads. The
class composition would need either richer per-status counters or packet-level capture, which
is itself part of request 1.

[BorisTyshkevich]

@filimonov