CAS: the mount-lease renewal can be killed by the memory tracker, which loses the mount

[filimonov]

Problem

The CAS mount lease is renewed by a dedicated background thread. If that thread stops renewing, the
mount is lost: writes stop being admitted once the fence deadline passes, and the pool has to remount.
Today an ordinary server memory-pressure event is enough to cause this. The renewal thread has no
thread group, so its allocations are charged straight to the global memory tracker, and any allocation
it makes can throw MEMORY_LIMIT_EXCEEDED when the server is above max_server_memory_usage. One
failed renewal is already terminal: the mount is marked lost and a remount is requested. Worse, an
allocation failure in the part of the renewal that is outside the inner try ends the renewal thread
permanently.

This is a liveness problem: a transient, recoverable condition (the server is briefly over its memory
limit) takes down something that must never stop.

Evidence

• The renewal runs on a plain ThreadFromGlobalPool: Pool/CasMountRuntime.cpp:606 (creation),
  :578 (makeWorker), :643 (renewalLoop).
• That thread gets a fresh ThreadStatus with no thread group: src/Common/ThreadPool.cpp:1103.
• Its MemoryTracker's parent is the global tracker: src/Common/ThreadStatus.h:195 plus
  src/Common/MemoryTracker.cpp:155. So MEMORY_LIMIT_EXCEEDED is reachable
  (src/Common/MemoryTracker.cpp:344, :419).
• About 1 MiB is allocated per renewal attempt just for the PUT buffer:
  Backend/CasObjectStorageBackend.cpp:312-313 with src/IO/WriteBufferFromS3.cpp:112 and
  DBMS_DEFAULT_BUFFER_SIZE at src/Core/Defines.h:22. A settling read adds about another 1 MiB:
  Backend/CasObjectStorageBackend.cpp:635 and S3ObjectStorage.cpp:383.
• A MEMORY_LIMIT_EXCEEDED inside the write attempt is classified as an ambiguous transport fault and
  reissued until the lease bound: Backend/CasRequests.cpp:945 and :226-230. Under sustained
  pressure every reissue fails identically and the budget is spent.
• One terminal renewal trips the mount: Pool/CasMountRuntime.cpp:302-320.
• An exception raised outside the inner try — encodeBody at Pool/CasServerRoot.cpp:1594,
  plane.admit at :1611, or consumeRenewResult — reaches the loop's catch-all at
  Pool/CasMountRuntime.cpp:717-739, which trips the fence and returns, ending the renewal thread.
• The cause is invisible afterwards: the give-up result keeps no transport error text
  (Backend/CasRequests.cpp:796-800, Backend/CasRequests.h:329-343).
• The PUT currently runs on a different thread than the renewal: WriteSettings is default
  constructed at Backend/CasObjectStorageBackend.cpp:807, s3_allow_parallel_part_upload defaults to
  true at src/IO/WriteSettings.h:37, and the upload is scheduled onto the remote-FS write pool at
  S3ObjectStorage.cpp:446-447 and src/IO/WriteBufferFromS3.cpp:841.

Proposed change

1. Hold LockMemoryExceptionInThread lock(VariableContext::Global) across a whole renewal iteration in
   CasMountRuntime::renewalLoop, so every allocation of the renewal — ours and the SDK's — is still
   accounted but the tracker never throws on this thread. This is the same mechanism Keeper
   (src/Coordination/KeeperServer.cpp:397) and the transaction log
   (src/Interpreters/TransactionLog.cpp:491) already use for work that must not be failed by a memory
   limit.
2. Set s3_allow_parallel_part_upload = false in ObjectStorageBackend::conditionalWriteSettings, so
   the control-plane PUT runs inline on the renewal thread and is actually covered by the guard. CAS
   control-plane writes are single small objects, so nothing is lost.
3. Size the control-plane buffers to the payload instead of using the 1 MiB default: pass the body size
   as buf_size in nativeConditionalPut, and pass the known size to casSizedReadSettings on the
   control-plane read path.
4. Carry the last transport exception's text into the GaveUp result so a memory-caused failure is
   identifiable in the log.

Alternatives considered

• A persistent preallocated buffer per renewer. Rejected as the primary fix. The object-storage API
  in this codebase has no read-into-caller-buffer entry point; readObject returns a buffer that owns
  its memory, and the only knob is ReadSettings::adjustBufferSize. The PUT buffer is allocated by
  WriteBufferFromS3's own constructor. And the AWS SDK allocates per request regardless, so a
  renewal can never be made allocation-free. Item 3 above captures the useful part of this idea.
• MemoryTrackerBlockerInThread. It hides the renewal's memory from accounting entirely; its own
  header recommends LockMemoryExceptionInThread instead.
• Treating MEMORY_LIMIT_EXCEEDED as a deterministic local failure in the retry classifier. This
  makes the renewal fail faster rather than survive, which is the wrong direction.
• Giving the renewal a longer budget or more attempts. It does not help: under sustained pressure
  every attempt fails the same way, and stretching the budget eats into the lease.

Risks

• With the guard in place, the renewal can push the global tracker slightly past its limit instead of
  failing. This is the intended trade and matches existing precedent. The amount is small, and item 3
  makes it smaller.
• Turning off parallel part upload for control-plane writes must not be reverted accidentally; it needs
  a comment explaining that it keeps the PUT on the guarded thread.
• A genuine allocator OOM still ends the renewal. This change does not claim to fix that.
• The renewal thread still exits on any exception escaping its own state machine
  (Pool/CasMountRuntime.cpp:717). The change removes the memory-tracker source of such exceptions but
  not the policy.

Acceptance criteria

• With the server memory limit set so that the global tracker is above its limit, the mount-lease
  renewal continues to commit and the mount is not lost, whereas without the change it reports a
  terminal renewal.
• A fault-injected memory-limit exception on the renewal path does not end the renewal thread and does
  not trip the mount fence.
• The control-plane PUT is issued on the CAS_LEASE_RENEWER thread, not on a remote-FS write-pool
  thread.
• The renewal's memory is still visible in the server's memory accounting (the guard suppresses the
  throw, not the accounting).
• When a renewal does fail for a transport reason, the log names the underlying exception.