Skip to content

[CELEBORN-2463] Add ManuallyExcludedWorkerCount metric to distinguish manually excluded workers - #3848

Open
yew1eb wants to merge 1 commit into
apache:mainfrom
yew1eb:CELEBORN-2463
Open

yew1eb wants to merge 1 commit into
apache:mainfrom
yew1eb:CELEBORN-2463

Conversation

@yew1eb

@yew1eb yew1eb commented Sep 15, 2026

Copy link
Copy Markdown
Contributor

What changes were proposed in this pull request?

Add a new master gauge metric ManuallyExcludedWorkerCount that reports the count of workers in the manually excluded list (statusSystem.manuallyExcludedWorkers.size).

The existing ExcludedWorkerCount metric is kept unchanged (still the sum of both excluded and manually excluded workers) for backward compatibility.

Why are the changes needed?

ExcludedWorkerCount counts both workers excluded due to failure reports / unhealthy heartbeats, and workers excluded manually via the /exclude API during maintenance. As a result, alerts based on ExcludedWorkerCount fire during routine maintenance even though the cluster is healthy.

The two sets are already tracked separately internally (excludedWorkers vs manuallyExcludedWorkers) and exposed separately in the /workers API.

With this PR, users can alert on ExcludedWorkerCount - ManuallyExcludedWorkerCount to monitor only failure-driven exclusions.

Does this PR resolve a correctness bug?

  • Yes

Does this PR introduce any user-facing change?

  • Yes

Add a new master metric ManuallyExcludedWorkerCount.

How was this patch tested?

Compiled the master module locally. The new gauge follows the exact same pattern as the existing ExcludedWorkerCount gauge.

@github-actions

github-actions Bot commented Oct 7, 2026

Copy link
Copy Markdown

This PR is stale because it has been open 20 days with no activity. Remove stale label or comment or this will be closed in 10 days.

@github-actions github-actions Bot added the stale label Oct 7, 2026
@yew1eb yew1eb closed this Oct 8, 2026
@yew1eb yew1eb reopened this Oct 8, 2026
@yew1eb yew1eb closed this Oct 8, 2026
@yew1eb yew1eb reopened this Oct 8, 2026
@github-actions github-actions Bot removed the stale label Oct 8, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants