Overview
The moment a tablet transitions RESTORE→REPLICA it is included in the tablet throttler's shard-scope lag aggregation with whatever lag its backup age implies. In the incident that surfaced this (#20922), a replica restored from a ~26h-old backup made the shard lag metric (vttablet_throttler_aggregated_shard_lag) jump from 0.2s to 92,886s — exactly the backup's age — within one probe interval. Every lag-gated app on the shard (vreplication, Online DDL, external apps at the default 5s threshold) was then denied for the replica's entire multi-hour catch-up, triggered by a routine node replacement.
The fix for #20922 (#20925) makes vreplication workflows survive such a window (the vplayer stall detector no longer fires while throttled), but they still make zero progress for hours, shard-wide, because of one rejoining tablet.
There is a genuine tension here, which is why this needs throttler-maintainer input rather than a quick patch: throttling on a lagged replica is the throttler doing its job, and reducing primary write inflow does help a catching-up replica converge. The counter-argument: a replica hours behind is not lagging because of write pressure, is not serving reads (the healthcheck marks it unhealthy at that lag), and backpressure cannot bring it into a useful range on any timescale that justifies stalling the whole shard's maintenance work.
Suggested approach
Options, roughly in order of preference:
- Hysteresis: a replica only participates in shard-lag aggregation once its lag has been under the threshold at least once since (re)joining — "you must become healthy before you can make the shard unhealthy." This naturally covers restores, brand-new replicas, and long-partitioned rejoins without RESTORE-specific plumbing, and roughly aligns throttler participation with serving eligibility (it starts counting at about the point the tablet could start serving reads that need protecting).
- Transition-keyed grace window: exclude a tablet from aggregation for a period after RESTORE→REPLICA. Simpler, but introduces a tunable that will be wrong somewhere, and does not cover the non-restore variants of the same problem.
- Clamp/exclude outliers: ignore lag values beyond some multiple of the threshold as "not backpressure-addressable." Crude, and changes aggregation semantics for everyone.
A no-core-change position worth acknowledging: orchestration can keep restored tablets DRAINED until caught up and only then flip them to REPLICA. But then every operator/orchestrator reimplements it, and the default behavior remains a footgun.
References
Overview
The moment a tablet transitions RESTORE→REPLICA it is included in the tablet throttler's shard-scope lag aggregation with whatever lag its backup age implies. In the incident that surfaced this (#20922), a replica restored from a ~26h-old backup made the shard lag metric (
vttablet_throttler_aggregated_shard_lag) jump from 0.2s to 92,886s — exactly the backup's age — within one probe interval. Every lag-gated app on the shard (vreplication, Online DDL, external apps at the default 5s threshold) was then denied for the replica's entire multi-hour catch-up, triggered by a routine node replacement.The fix for #20922 (#20925) makes vreplication workflows survive such a window (the vplayer stall detector no longer fires while throttled), but they still make zero progress for hours, shard-wide, because of one rejoining tablet.
There is a genuine tension here, which is why this needs throttler-maintainer input rather than a quick patch: throttling on a lagged replica is the throttler doing its job, and reducing primary write inflow does help a catching-up replica converge. The counter-argument: a replica hours behind is not lagging because of write pressure, is not serving reads (the healthcheck marks it unhealthy at that lag), and backpressure cannot bring it into a useful range on any timescale that justifies stalling the whole shard's maintenance work.
Suggested approach
Options, roughly in order of preference:
A no-core-change position worth acknowledging: orchestration can keep restored tablets DRAINED until caught up and only then flip them to REPLICA. But then every operator/orchestrator reimplements it, and the default behavior remains a footgun.
References