Nothing re-arms a backing-off control connection: recovery latency grows from `idle_heartbeat_interval` to `max_reconnection_delay`

[dkropachev]

Problem

Once ControlConnection's reconnection handler is parked on its backoff schedule, the only clock that can bring the control connection back is reconnection_policy's own schedule. No event re-arms it early -- not a host coming back up, not the idle heartbeat. With the defaults (ExponentialReconnectionPolicy(1.0, 600.0), cassandra/cluster.py:774) the retry interval climbs 1s, 2s, 4s ... 512s and then plateaus at 600s, so roughly 17 minutes into an outage the driver is only probing once every 10 minutes. A cluster that becomes reachable one second after a failed attempt keeps its control connection down for up to max_reconnection_delay.

Concretely:

• ControlConnection.on_up() is pass (cassandra/cluster.py:4700-4701). A host coming back up never touches the parked handler.
• ControlConnection.return_connection() (cassandra/cluster.py:4731-4733) is driven by the idle heartbeat and calls reconnect() once per idle_heartbeat_interval (default 30s, cassandra/cluster.py:935) for as long as the connection stays defunct. reconnect() now returns early when self._reconnection_handler is not None (cassandra/cluster.py:4058-4061), so this path no longer advances anything.
• on_add() only refreshes metadata; on_down()/on_remove() call the same reconnect() and hit the same guard.

Relationship to #1024

This is a deliberate consequence of #1024, not an accident, and it is the correct trade for the bug that PR fixes -- but the second-order cost was never decided on.

Before #1024, each heartbeat-driven reconnect() cancelled the in-flight handler and rebuilt its schedule from base_delay, which meant the control connection was effectively retried every ~30s forever. That was also the bug: a burst of errors, or a merely idle defunct connection, reset the backoff on every pass so it never grew, which is the reconnect storm #295 reports. #1024 closed that by making reconnect() respect a handler that is already retrying. The backoff now genuinely grows -- and with nothing else able to re-arm it, growth is unbounded up to max_delay.

So the driver traded a storm for a stall. Worst-case control-connection recovery latency went from ~idle_heartbeat_interval to ~max_reconnection_delay: 30s to 10 minutes at defaults. The control connection does eventually recover (the default policy has max_attempts=None, so the handler retries forever); this is a latency regression, not a permanent loss.

Why it was not fixed in #1024

Every candidate fix picks a different point on that same trade-off, and picking wrongly reopens #295:

1. Make ControlConnection.on_up() re-arm. Cancel the parked handler and call reconnect() when self._connection is None or defunct. Most targeted -- the node being back is exactly the signal worth reacting to -- but it fires once per host coming back, so a whole-cluster restart produces the burst control-connection: reconnect when down handling is skipped #1024 set out to collapse unless the re-arm is itself deduplicated.
2. Clamp the control connection's own schedule below reconnection_policy.max_delay. Keeps the fix off the event paths entirely, but invents a second implicit config surface: the control connection would stop honouring the reconnection_policy the user configured, with no knob saying so.
3. Keep a heartbeat-driven floor. Let return_connection() advance a handler that has been parked longer than some bound. Cheapest to reason about, but partially undoes control-connection: reconnect when down handling is skipped #1024's stated acceptance criterion that the schedule no longer resets once per idle_heartbeat_interval.

This needs an author/maintainer decision about which cadence the control connection is supposed to promise, which is why #1024 left it out rather than guessing.

Minimum acceptable outcome

Even if the behaviour is judged acceptable as-is, the CHANGELOG should state it: after #1024 the control connection's retry cadence follows reconnection_policy up to max_reconnection_delay instead of being re-driven every idle_heartbeat_interval. Operators who relied on the old ~30s floor need to know to lower max_reconnection_delay.

Related

• control-connection: reconnect when down handling is skipped #1024 -- the PR that introduced this trade-off
• After a node restarts, driver reconnects multiple times, causing queries to fail #295 -- the reconnect storm any fix here must not reintroduce
• Session might not reconnect if all nodes are restarted at once #230 -- session not reconnecting after all nodes restart at once; different diagnosis (refresh failure), same symptom class