feat: add support for dynamic KRaft quorum scaling by razvan · Pull Request #1010 · stackabletech/kafka-operator

razvan · 2026-08-18T12:51:47Z

Description

See CHANGELOG for a high level view of everything that changed.

✅ : OKD integration tests work. I ran them many times but there was always one or two tests that failed due to flakiness. I added some cleanup steps so that kuttl doesn't timeout during namespace cleanups.

Definition of Done Checklist

Not all of these items are applicable to all PRs, the author should update this template to only leave the boxes in that are relevant
Please make sure all these things are done and tick the boxes

Author

Changes are OpenShift compatible
CRD changes approved
CRD documentation for all fields, following the style guide.
Helm chart can be installed and deployed operator works
Integration tests passed (for non trivial changes)
Changes need to be "offline" compatible
Links to generated (nightly) docs added
Release note snippet added

Reviewer

Code contains useful comments
Code contains useful logging statements
(Integration-)Test cases added
Documentation added or updated. Follows the style guide.
Changelog updated
Cargo.toml only contains references to git tags (not specific commits or branches)

Acceptance

Feature Tracker has been updated
Proper release label has been added
Links to generated (nightly) docs added
Release note snippet added
Add type/deprecation label & add to the deprecation schedule
Add type/experimental label & add to the experimental features tracker

Controllers now run a quorum-manager sidecar that admits itself into the KRaft voter set on startup (add-controller) and removes itself before termination (remove-controller via preStop), so controller role groups can be scaled up/down on a running cluster without a full rolling restart. controller.quorum.bootstrap.servers now points at each controller role group's headless Service DNS name instead of individual pod addresses, keeping container commands stable across replica changes. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

Controllers get a startupProbe (plain TCP, generous failure threshold for slow metadata-log replay on boot), a plain-TCP livenessProbe, and a readinessProbe that checks the node's Raft state via its metrics endpoint instead of a bare TCP check, so a controller stuck rejoining the quorum is correctly reported as not ready. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

Controller pods now start/scale sequentially (OrderedReady) instead of in parallel, since the quorum-manager sidecar's admission flow assumes one voter joins at a time. Brokers are unaffected and keep Parallel. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

Scaling controllers to 0 replicas while brokers keep running is now rejected at validation time with an actionable error, instead of failing much later and confusingly while building the broker's ConfigMap. Scaling controllers and brokers to 0 together (a coordinated whole-cluster stop) is still allowed and now actually builds, since downstream resource builders no longer assume a non-empty controller quorum whenever KRaft mode is active. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

…ns tests Enable the previously version-gated scale-up/down steps (Kafka 3.7 no longer needs special-casing), assert quorum voter counts via kafka-metadata-quorum.sh after each scale, and add a final step scaling both controllers and brokers to 0 to exercise the whole-cluster-stop path before namespace teardown. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

Update the KRaft controller usage guide for scale-up/down support, record the design spec and implementation plan, add the CHANGELOG entries for this branch's changes, and ignore .worktrees/ for local worktree checkouts. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

razvan · 2026-08-18T15:53:34Z

The failed test was due to namespace deletion timeout

--- FAIL: kuttl (2409.35s)
    --- FAIL: kuttl/harness (0.00s)
        --- PASS: kuttl/harness/logging_kafka-3.9.2_zookeeper-latest-3.9.5_openshift-false (134.90s)
        --- FAIL: kuttl/harness/smoke-kraft_kafka-kraft-4.2.1_openshift-false (634.74s)
        --- PASS: kuttl/harness/smoke_kafka-3.9.2_zookeeper-3.9.5_use-client-tls-false_openshift-false (81.56s)
        --- PASS: kuttl/harness/upgrade_upgrade_old-3.9.2_upgrade_new-4.2.1_use-client-tls-false_use-client-auth-tls-false_openshift-false (120.71s)
        --- PASS: kuttl/harness/kerberos_kafka-3.9.2_zookeeper-latest-3.9.5_openshift-false_krb5-1.21.1_kerberos-realm-PROD.MYCORP_kerberos-backend-mit_broker-listener-class-cluster-internal_bootstrap-listener-class-external-unstable (113.64s)
        --- PASS: kuttl/harness/tls_kafka-3.9.2_zookeeper-latest-3.9.5_use-client-tls-false_use-client-auth-tls-false_openshift-false (33.64s)
        --- PASS: kuttl/harness/cluster-operation_kafka-latest-3.9.2_zookeeper-latest-3.9.5_openshift-false (41.49s)
        --- PASS: kuttl/harness/configuration_kafka-latest-3.9.2_openshift-false (11.45s)
        --- PASS: kuttl/harness/operations-kraft_kafka-kraft-4.2.1_openshift-false (1079.48s)
        --- PASS: kuttl/harness/opa_kafka-latest-3.9.2_zookeeper-latest-3.9.5_opa-latest-1.16.2_use-opa-tls-false_openshift-false_krb5-1.21.1 (128.33s)
        --- PASS: kuttl/harness/delete-rolegroup_kafka-3.9.2_zookeeper-latest-3.9.5_openshift-false (29.40s)
FAIL
ERROR:root:kuttl failed

…ter-membership

maltesander

This is not quite stable yet. Also, alot of AI slur in the code comments etc. that can be skimmed down.

maltesander · 2026-09-07T06:42:39Z

+          describe=$(timeout --kill-after={cli_kill_after} {cli_timeout} {binary} --bootstrap-controller "$BOOTSTRAP_SERVERS" --command-config {config} describe --replication 2>/dev/null)
+          if [ -n "$describe" ]; then
+            voters=$(echo "$describe" | tail -n +2 | awk '$NF == "Leader" || $NF == "Follower"')
+            total_voters=$(echo "$voters" | grep -c .)


This works for scaling operations, not for namespace deletion or node drainages? Meaning when all prestop hooks run at "the same time" they all read the same $voters and all issue remove-controller? The voter set drains to zero and we reformat at restart? That is also the reason for 90-stop-kafka.yaml.j2 ? Meaning scaledown / stop cluster before namespace deletion.

This works for scaling operations, not for namespace deletion or node drainages?

It works for all graceful operations except hard node failures.

Meaning when all prestop hooks run at "the same time" they all read the same $voters and all issue remove-controller?

The maxUnavailable: 1 of the controller PDB and the OrderedReady scaling strategy ensure scaling up/down is done one controller at a time.

The voter set drains to zero and we reformat at restart?

I'm preparing documentation for handling the case there the controllers lose the PVC. In that case, manual intervention is necessary to avoid a split-brain or just faulty quorum.

I will post a separate comment.

That is also the reason for 90-stop-kafka.yaml.j2 ? Meaning scaledown / stop cluster before namespace deletion.

No. The reason for 90-stop-kafka.yaml.j2 is to remove the cluster deletion time quota from the namespace termination budget. This is the strategy we wanted for all tests in all ops but that is postponed due to better testing tools comming.

This adds troubleshooting docs for the case where a controller loses it's PVC.

In addition, it extracts the quorum command into a separate script. A new controller is now added to the quorum only if the quorum is healthy and the new controller is not crash-lopping.

Cant we fix the script rather than add the troubleshoot guide?

The troubleshooting guide is for incidents that can happen and that cannot be handled by the operator. Not sure what should be fixed here.

Co-authored-by: maltesander <malte.sander.it@gmail.com>

…ommand into separate script

# Conflicts: # rust/operator-binary/src/controller/build/kerberos.rs # rust/operator-binary/src/controller/build/resource/statefulset.rs # rust/operator-binary/src/controller/build/security.rs

maltesander

This needs a full resweep for -- in any docs and code or script comments / headers (not the CLI flag, the em/en dash usage).

maltesander · 2026-09-10T14:13:59Z

@@ -0,0 +1,95 @@
+#!/usr/bin/env bash


I think this should go into a dedicated script folder? (same for the other script)

maltesander · 2026-09-11T05:56:03Z

@@ -0,0 +1,95 @@
+#!/usr/bin/env bash
+# The `preStop` hook of a KRaft controller pod's `kafka` container: remove this pod from the
+# KRaft voter set before it terminates -- unless that would remove the *last* voter.


The -- is a new kind of dash and appears alot, could be rewritten properly? (same for the other script)

razvan and others added 6 commits August 18, 2026 14:24

razvan self-assigned this Aug 18, 2026

Cleanup inline comments and documentation

fc21e2e

razvan marked this pull request as draft August 18, 2026 15:57

razvan added 3 commits August 19, 2026 14:03

Remove the custom bash trap functions

d02b920

Comments abd documentation cleanups

62adbff

Merge remote-tracking branch 'origin/main' into feat/kraft-dynamic-vo…

95839be

…ter-membership

razvan mentioned this pull request Aug 20, 2026

fix: kraft kerberos support #999

Draft

20 tasks

razvan added 2 commits August 20, 2026 10:04

Cleanup CHANGELOG and KRaft docs.

c4cb3a5

Remove the kcat-prober container and consolidate container probes

4b42def

razvan marked this pull request as ready for review August 20, 2026 11:10

razvan added 2 commits August 20, 2026 14:30

Cleanups

0596a0a

Stop Kafka to prevent kuttl namespace deletion timeouts

ed3180b

razvan added this to Stackable Engineering Aug 21, 2026

razvan moved this to Development: Waiting for Review in Stackable Engineering Aug 21, 2026

maltesander self-requested a review September 1, 2026 11:40

maltesander moved this from Development: Waiting for Review to Development: In Review in Stackable Engineering Sep 1, 2026

maltesander requested changes Sep 1, 2026

View reviewed changes

Merge branch 'main' into feat/kraft-dynamic-voter-membership

6276e1f

maltesander requested changes Sep 7, 2026

View reviewed changes

razvan and others added 4 commits September 9, 2026 15:15

Apply batched suggestions from code review

cfc9745

Co-authored-by: maltesander <malte.sander.it@gmail.com>

Apply batched suggestions from code review

d9406f8

Co-authored-by: maltesander <malte.sander.it@gmail.com>

fix pre commit hooks

12c090f

implement review feedback: more idiomatic rust

8b7391d

razvan added 3 commits September 9, 2026 18:40

review feedback: extract pre-stop command into separate script

c551a4d

fix the nightly test suite

f52775e

review feedback: in kraft mode, controllers are mandatory

9f19d62

razvan mentioned this pull request Sep 10, 2026

Kafka: Add support for managing Kafka topics in a declarative way stackabletech/issues#888

Open

razvan added 2 commits September 10, 2026 17:31

review feedback: add troublshooting docs and extract quorum manager c…

a0b4985

…ommand into separate script

Merge branch 'main' into feat/kraft-dynamic-voter-membership

ee8304b

# Conflicts: # rust/operator-binary/src/controller/build/kerberos.rs # rust/operator-binary/src/controller/build/resource/statefulset.rs # rust/operator-binary/src/controller/build/security.rs

maltesander requested changes Sep 11, 2026

View reviewed changes

review feedback: move controller scripts to scripts/ folder

581e81b

Uh oh!

Uh oh!

Conversation

razvan commented Aug 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Definition of Done Checklist

Author

Reviewer

Acceptance

Uh oh!

razvan commented Aug 18, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

maltesander left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

maltesander Sep 7, 2026

Choose a reason for hiding this comment

Uh oh!

razvan Sep 10, 2026

Choose a reason for hiding this comment

Uh oh!

razvan Sep 10, 2026

Choose a reason for hiding this comment

Uh oh!

maltesander Sep 11, 2026

Choose a reason for hiding this comment

Uh oh!

razvan Sep 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

maltesander left a comment

Choose a reason for hiding this comment

Uh oh!

maltesander Sep 10, 2026

Choose a reason for hiding this comment

Uh oh!

maltesander Sep 11, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

razvan commented Aug 18, 2026 •

edited

Loading

razvan Sep 11, 2026 •

edited

Loading