Skip to content

Monitor: show image version per worker; exclude running tasks from backlog - #199

Merged
maximusunc merged 2 commits into
develfrom
monitor-worker-versions
Nov 1, 2026
Merged

maximusunc merged 2 commits into
develfrom
monitor-worker-versions

Conversation

@maximusunc

Copy link
Copy Markdown
Collaborator

Summary

Two monitor dashboard improvements.

1. Image version on worker cards

  • The release workflow passes the release tag into every image as a SHEPHERD_VERSION build arg. All worker Dockerfiles and shepherd_server/Dockerfile declare it, and local builds default to dev. A deployment can also override it through the environment.
  • The worker heartbeat now includes version. The monitor carries it per replica and lists the distinct versions for each worker type.
  • Card: shows the version under the worker name. When replicas run different versions (e.g. mid-rollout), the line turns amber and reads Mixed: v2.7.0 ×1, v2.8.1 ×2.
  • Replica detail popup: gains a Version column.
  • Images built before this change report no version and show as unknown until the next release.

2. Backlog no longer counts running tasks

Workers XDEL a stream entry only after finishing it, so XLEN includes tasks already being worked on. The card's Backlog was raw XLEN, so a single running query showed as Running 1 / Backlog 1. Backlog is now XLEN − pending, clamped at 0.

The utilization bar and the recorded backlog:* history series still use XLEN (total load). This keeps the History page's "Queue backlog (XLEN)" chart and its utilization heatmap unchanged. The popup's "Queue length" label is renamed "Backlog" to match the card.

Testing

  • New unit tests cover the heartbeat version field, the version rollup, the backlog calculation (including the clamp) and utilization. The monitor tests and tests/unit/test_worker_lifecycle.py pass (104 tests).
  • The dashboard was rendered with mock snapshot data in headless Chromium to check how the cards and popup look.

Generated by Claude Code

Bake the release tag into every image as SHEPHERD_VERSION (build arg set
by the release workflow, "dev" for local builds), report it in the worker
heartbeat, and roll the distinct versions up per worker type. Cards show
the version under the worker name and flag a mixed fleet (e.g.
mid-rollout) with per-version replica counts; the replica modal gains a
Version column. Older images without the field show as "unknown".

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01S2yuRD9F9c4T7rkGwPJxRm
Workers XDEL a stream entry only after finishing it, so XLEN includes
tasks a consumer has already read and is working on. The card's Backlog
was raw XLEN, so a single running task showed as Running 1 / Backlog 1.
Backlog is now XLEN minus the group's pending entries (clamped at 0).

Utilization and the recorded history series still use XLEN (total load),
so the utilization bar, History chart ("Queue backlog (XLEN)") and
heatmap keep their meaning. The detail popup's "Queue length" label is
renamed "Backlog" to match the card.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01S2yuRD9F9c4T7rkGwPJxRm
@codecov

codecov Bot commented Nov 1, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 88.88889% with 1 line in your changes missing coverage. Please review.
✅ Project coverage is 54.14%. Comparing base (0e19968) to head (d1c6221).
⚠️ Report is 1 commits behind head on devel.

Files with missing lines Patch % Lines
workers/monitor/poller.py 80.00% 1 Missing ⚠️
Files with missing lines Coverage Δ
shepherd_utils/heartbeat.py 55.23% <100.00%> (+0.86%) ⬆️
workers/monitor/poller.py 53.67% <80.00%> (+2.99%) ⬆️

... and 2 files with indirect coverage changes


Continue to review full report in Codecov by Harness.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 5786ce5...d1c6221. Read the comment docs.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@maximusunc
maximusunc merged commit ca0ed67 into devel Nov 1, 2026
2 checks passed
@maximusunc
maximusunc deleted the monitor-worker-versions branch November 1, 2026 12:39
maximusunc pushed a commit that referenced this pull request Nov 1, 2026
Worker image versions on the monitor dashboard (SHEPHERD_VERSION build arg,
kept alongside this branch's latest-trapi-1.6 tags) and workers naming the
HTTP code and description their query fails with. The ARAX route keeps
attaching the logs to the response, as 1.6 does, and passes the failure
description through; the lost-response test drops its 2.0 Response check.
Version stays 1.3.4 (untagged).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TtsBikU9y9AFQ5uu9tcioz
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants