CASSJAVA-124: Add GRACEFUL_DISCONNECT support (CEP-59) - #2091
Open
Shanzita wants to merge 22 commits into
Open
Conversation
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Implements client-side support for CEP-59 (Graceful Disconnect – In-Band
Connection Draining) in the Java driver.
When a Cassandra node shuts down, it emits a GRACEFUL_DISCONNECT event to
subscribed connections. The driver:
Design
register for GRACEFUL_DISCONNECT sends OPTIONS and filters its REGISTER
against its own SUPPORTED response. There is no driver-global capability
flag, so mixed-version clusters (rolling upgrades) work correctly.
GracefulDisconnectEventis node-scoped: it means "this nodeis shutting down", regardless of whether it arrived on a pool connection or
the control connection, and the pool drains accordingly.
retries once without the event type instead of failing channel init
(mixed-version / evolving-server tolerance).
advanced.connection.graceful-disconnect-enabledconfig option (default true).graceful-disconnects(session) andpool.graceful-disconnects(node)counters, initialized in the Dropwizard, Micrometer and MicroProfile
backends and incremented where events are received.
Testing
filtering and REGISTER-rejection degradation, node-scoped ChannelPool drain,
metric increments, duplicate/late event tolerance)
GracefulDisconnectIT: bounded 2-node ccm test that drains a node understeady query load and asserts the metric observed the event, queries fail
over, and zero exceptions reach the application. Gated on Cassandra 7.0+, so
it is skipped until a server with CEP-59 is available in CI.
(CASSANDRA-21191: [CEP-59] Implementation of In-Band Connection Draining (Graceful Disconnect) cassandra#4953) on 2-node and 3-node ccm clusters:
nodetool drainmid-load → 0 disruptive exceptions, thousands of successfulqueries after the event, drained node goes DOWN cleanly, failover invisible
to the application. The 3-node run exercised the event arriving on the
control connection and a pool connection simultaneously.
Depends on: native-protocol 1.5.3-SNAPSHOT (Shanzita/native-protocol:cep-59,
the branch behind datastax/native-protocol#61; installed by install-snapshots.sh
until 1.5.3 is released)
JIRA: CASSJAVA-124