Skip to content

fix(dashboard): scale worker table warning colors with worker-ttl - #9373

Open
nil68657 wants to merge 1 commit into
dask:mainfrom
nil68657:fix-worker-table-ttl
Open

nil68657 wants to merge 1 commit into
dask:mainfrom
nil68657:fix-worker-table-ttl

Conversation

@nil68657

Copy link
Copy Markdown

The worker table marked a worker red once its last heartbeat was more
than 60s old, regardless of distributed.scheduler.worker-ttl. With the
default 5 minute TTL that flags healthy-but-busy workers (e.g. ones
holding the GIL) long before the scheduler would act, and users had no
way to tune it.

Color rows orange after 25% and red after 50% of the effective TTL,
computed the same way as Scheduler.check_worker_ttl. When worker-ttl is
disabled, keep the old 60s red threshold.

This PR is a joint contribution from Nilanjan Chatterjee and Claude (Claude Opus 5.5, via Claude Code). Nilanjan chose the issue and directed the work; Claude reproduced the problem, wrote the change, its tests and this description.

Authored by Nilanjan Chatterjee (@nil68657).

The worker table marked a worker red once its last heartbeat was more
than 60s old, regardless of distributed.scheduler.worker-ttl. With the
default 5 minute TTL that flags healthy-but-busy workers (e.g. ones
holding the GIL) long before the scheduler would act, and users had no
way to tune it.

Color rows orange after 25% and red after 50% of the effective TTL,
computed the same way as Scheduler.check_worker_ttl. When worker-ttl is
disabled, keep the old 60s red threshold.

Closes dask#8749

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@github-actions

Copy link
Copy Markdown
Contributor

Unit Test Results

See test report for an extended history of previous test failures. This is useful for diagnosing flaky tests.

    40 files  ± 0      40 suites  ±0   14h 6m 15s ⏱️ - 1m 27s
 4 162 tests + 2   3 981 ✅ + 2    178 💤 ±0  3 ❌ ±0 
80 999 runs  +40  76 758 ✅ +41  4 238 💤  - 1  3 ❌ ±0 

For more details on these failures, see this check.

Results for commit 2469b39. ± Comparison against base commit dc182bd.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Adjust threshold for bad worker on worker table based on worker-ttl

1 participant