[dan-petty]

The Service counters gave a false picture. A label set had no series until its first count, so increase() lost that count on each new pod, and the failure counter mixed all repositories. The Service now creates each bounded label set of its counters at zero when it starts. The failure counter has the label repo, and the job counter has the label row and counts each row. RoadmapServiceJobsFailing groups and joins on the row, and a row that the stop of the Service ends counts under stopped. The webhook alerts lose their #2505 workaround terms. The documents state the scrape behavior and the labels, and the action of RoadmapServiceGitHubRequestsHigh no longer says to lower the budget. A round that fails before its first row stays out of this change: #3666 holds it.

Closes #3495

Change: openspec/changes/archive/2026-11-05-3495-service-counters-zero-series-repo-label/

[dan-petty]

Results

Fail-to-pass (local). uv run devops ci fail-to-pass --base origin/main: 26 tests failed on the base (26 failed and 229 passed in tests/prometheus/test_k8s_monitoring_alerting.py, tests/roadmap/test_roadmap_implement_submit.py, tests/roadmap/test_roadmap_run.py, tests/server/test_server_service.py, tests/telemetry/test_service_counter_series.py; 0 errored).

Full CI (local). uv run devops ci ran on the commit before the archive, on top of a base one commit older than this head. The test row had 4 failures and 1 error, all in the known host failures that also fail on origin/main: tests/crypto/test_ssh.py, tests/telemetry/test_config_env_overrides.py, tests/commands/test_devcontainer_install_tools_dry_run.py, tests/ai/test_tool_executor_contract.py (#3585). 18894 tests passed. The test row took 7m 27s, over the 5m budget (#3593). The coverage row failed only because the test run failed: total coverage is 94.53%, above the 85.0% floor. All other rows passed.

After the archive and the last rebase. uv run devops openspec check passed, and tests/telemetry/test_service_counter_series.py with tests/prometheus/test_k8s_monitoring_alerting.py passed (58). The full CI did not run again on this head locally. The rebase brought in #3669, which touches none of the files of this change except text in docs/ROADMAP_WORKFLOW.md and openspec/specs/roadmap-implement-job/spec.md.

Review findings

All 6 findings are correct, and the branch fixes them.

1. k8s/README.md, step 12: len(FailureClass) is 18, not 19. The sentence now gives no count and links to the table of classes, so it cannot go stale. The records of the change now say 18.
2. docs/ROADMAP_WORKFLOW.md: the sentence now gives the labels class and repo. After the rebase it is at line 143.
3. tasks.md: task 3.2 now names create_webhook_router in src/devops_cli/server/routes/webhooks.py, and task 4.1 gives the real name of the promtool case.
4. design.md: the decision now says that run_scoped_update counts each scoped update, and that the worker counts no job.
5. roadmap-service-counters spec: the requirement now uses [1h], the window of the promtool case, and the scenario gives the 5 counts plus the extrapolation to the window edges, about 5.03. The delta spec and the main spec agree.
6. src/devops_cli/server/counter_series.py: the three tables are now _SERVICE_JOB_RESULTS, _SERVICE_TRIGGER_SOURCES and _SERVICE_CHAT_RESULTS.

Check after the deploy

Not run. The series change only after the deploy, so no live Prometheus query proves this change before the merge. Run these queries after the deploy:

# 18 for each repository of service.repos, in each pod.
count by (pod, repo) (devops_cli_service_implement_failures_total{job="roadmap-service"})

# Only values of service.repos.
count by (repo) (devops_cli_service_implement_failures_total{job="roadmap-service"})

# Each row of each lane, with the results success, error and stopped.
count by (lane, row) (devops_cli_service_jobs_total{job="roadmap-service"})

# The split of the failures by repository. Compare it with the stop comments over the same window.
sum by (repo, class) (increase(devops_cli_service_implement_failures_total{job="roadmap-service"}[24h]))

devops_cli_service_identity_refused_total has no series until the first refusal. This is its design, and it is not a failure of this check.

[dan-petty]

Fail-to-pass check run of the head 6fb8ac7b: https://github.com/dan-petty/devops-cli/actions/runs/42086262362/job/127452066715. On this head, 21 check runs passed and 2 were skipped, so the full suite passed on the GitHub runner.