From 6209fb53569dea9854bf434de51af363fab635dc Mon Sep 17 00:00:00 2001 From: Daniel Petty Date: Thu, 5 Nov 2026 03:23:20 +0000 Subject: [PATCH 01/17] docs(openspec): plan #3495 service counters zero series Plan the zero-start label sets, the repo label of the failure counter, the row label of the job counter and the promtool cases as one change. Change: openspec/changes/3495-service-counters-zero-series-repo-label --- .../.openspec.yaml | 2 + .../design.md | 18 ++++++ .../proposal.md | 47 +++++++++++++++ .../specs/roadmap-implement-job/spec.md | 16 +++++ .../specs/roadmap-service-counters/spec.md | 41 +++++++++++++ .../tasks.md | 58 +++++++++++++++++++ 6 files changed, 182 insertions(+) create mode 100644 openspec/changes/3495-service-counters-zero-series-repo-label/.openspec.yaml create mode 100644 openspec/changes/3495-service-counters-zero-series-repo-label/design.md create mode 100644 openspec/changes/3495-service-counters-zero-series-repo-label/proposal.md create mode 100644 openspec/changes/3495-service-counters-zero-series-repo-label/specs/roadmap-implement-job/spec.md create mode 100644 openspec/changes/3495-service-counters-zero-series-repo-label/specs/roadmap-service-counters/spec.md create mode 100644 openspec/changes/3495-service-counters-zero-series-repo-label/tasks.md diff --git a/openspec/changes/3495-service-counters-zero-series-repo-label/.openspec.yaml b/openspec/changes/3495-service-counters-zero-series-repo-label/.openspec.yaml new file mode 100644 index 000000000..1491b921e --- /dev/null +++ b/openspec/changes/3495-service-counters-zero-series-repo-label/.openspec.yaml @@ -0,0 +1,2 @@ +schema: spec-driven +created: 2026-11-05 diff --git a/openspec/changes/3495-service-counters-zero-series-repo-label/design.md b/openspec/changes/3495-service-counters-zero-series-repo-label/design.md new file mode 100644 index 000000000..96349f907 --- /dev/null +++ b/openspec/changes/3495-service-counters-zero-series-repo-label/design.md @@ -0,0 +1,18 @@ +# Design + +## Decisions + +- **The Service creates the bounded label sets of its counters at zero when it starts.** A labelled counter has no series until `.labels()` makes a child (see the Feasibility of the proposal). One function calls `.labels()` for each bounded label set, so the first scrape of a pod reads a zero series for each label set. +- **The label sets are bounded.** The failure classes are the 19 members of `FailureClass`. The repositories are `service.repos`. The rows are the rows of the three lane tables of the roadmap run. The events of the webhook counter are the declared events of each provider, plus `other`. No dynamic label value is created at start. +- **The exposition format stays text 0.0.4.** The Service serves `/metrics` in this format (`src/devops_cli/server/service.py`). The zero children appear in this format, so the change does not switch the format. +- **The jobs counter counts each row.** The roadmap run counts each row of a lane with its result, with the row name of the row table. The worker counts the scoped updates with the name of their job. A row counted in the run is not counted again in the worker, so no job is counted twice. +- **The OTLP workaround terms stay.** The GitHub request and AI request series reach Prometheus over OTLP, not the `/metrics` scrape. The comment at `k8s/monitoring/prometheus-values.yaml` says that an OTLP series gets a sample only when the Service sends a new observation. No live Prometheus query ran in this change, so the OTLP terms stay until a live check shows that they are not needed. +- **The gap of a new pod stays.** The zero children are created at start. A count made before the first scrape of a new pod is still lost, because `increase()` does not count the first sample of a series. The documents state this gap. +- **No recording rule.** The promtool case of a pod change decides whether a recording rule is needed. The case runs with the zero series of the new pod. + +## Rejected alternatives + +- **Switch `/metrics` to OpenMetrics or PrometheusProto for start timestamps.** Rejected: the change of format reaches every consumer of `/metrics`, and the start-timestamp injection is not verified for this path. The zero children solve the missing first sample without a format change. +- **Keep the per-batch count and add a second count per row.** Rejected: one job would be counted twice, once per lane batch and once per row. +- **Prime the identity label of `devops_cli_service_identity_refused_total`.** Rejected for this change: the identity values come from the configured GitHub App, the alert `RoadmapServiceIdentityRefused` reads `> 0` with no workaround term, and the label set is not in the acceptance criteria of the issue. It stays an open item. +- **Add a recording rule for the true total without a test.** Rejected: a recording rule adds a second series name for one counter. It needs a promtool case that shows the need first. diff --git a/openspec/changes/3495-service-counters-zero-series-repo-label/proposal.md b/openspec/changes/3495-service-counters-zero-series-repo-label/proposal.md new file mode 100644 index 000000000..d3dc99bd7 --- /dev/null +++ b/openspec/changes/3495-service-counters-zero-series-repo-label/proposal.md @@ -0,0 +1,47 @@ +# Proposal + +## Why + +Issue #3495 reports that the Service counters give a false picture. The counters come from the `/metrics` scrape, and each release starts a new pod. A label set has no series until its first increment, so `increase()` loses the first count of each new label set. The failure counter `devops_cli_service_implement_failures_total` also has no `repo` label, so one series mixes all repositories. This change fits one pull request. + +## What Changes + +- When the Service starts, it creates each bounded label set of its counters at zero: each failure class for each repository of `service.repos`, each repository, lane, row and result of `devops_cli_service_jobs_total`, each repository and lane of `devops_cli_service_job_seconds_total`, and each provider, declared event and outcome of `devops_cli_service_webhook_deliveries_total`. +- `devops_cli_service_implement_failures_total` gets the label `repo`. +- `devops_cli_service_jobs_total` gets the label `row`. The roadmap run counts each row of a lane with its result. The worker no longer counts a whole lane batch. +- The alert expressions of `RoadmapServiceWebhooksRejected` and `RoadmapServiceSelfDeliveriesHigh` lose their #2505 workaround terms. The OTLP alert expressions keep theirs. +- Promtool cases prove the true total over a pod change, and prove that the webhook alerts hold without the workaround. +- `docs/TELEMETRY.md` and `k8s/README.md` state the scrape behavior and the labels. The action text of `RoadmapServiceGitHubRequestsHigh` no longer says to lower the budget. The class count of the Service changes from 17 to 19. + +## Capabilities + +### New Capabilities + +- `roadmap-service-counters`: The label sets of the Service counters, which the Service creates at zero when it starts, and the labels that the counters of the Service carry. + +### Modified Capabilities + +- `roadmap-implement-job`: The failure counter adds the label `repo` to the label `class`. + +## Impact + +- Code: `src/devops_cli/telemetry/metrics.py`, `src/devops_cli/server/service.py`, `src/devops_cli/roadmap/implement_submit.py`, `src/devops_cli/roadmap/run.py`, `src/devops_cli/server/routes/webhooks.py`. +- Monitoring: `k8s/monitoring/prometheus-values.yaml`. +- Tests: `tests/prometheus/test_implement_failure_labels.py`, `tests/telemetry/test_service_counter_series.py`, `tests/prometheus/test_k8s_monitoring_alerting.py`, `tests/fixtures/prometheus_rules/roadmap_service_tests.yaml`, `tests/fixtures/prometheus_rules/roadmap_service_webhooks_tests.yaml`. +- Documents: `docs/TELEMETRY.md`, `k8s/README.md`. +- The change adds no dependency. + +## Feasibility + +- The failure counter increments with the label `class` only. This is confirmed by reading `src/devops_cli/roadmap/implement_submit.py` at origin/main, where `GLOBAL_METRICS.increment_counter(CONST_SERVICE_METRIC_IMPLEMENT_FAILURES, labels={"class": failure.value})` runs. +- The jobs counter has the labels `repo`, `lane` and `result`, and no `row`. This is confirmed by reading `src/devops_cli/server/service.py` at origin/main, where the worker increments it with `{**self._labels, "result": result}`. +- A labelled counter emits no sample until a child exists. This is confirmed by a probe with prometheus_client 0.26.0 in the devcontainer: the counter with the labels `class` and `repo` emitted no line, and after `.labels("oom", "example/app")` it emitted `probe_total{class="oom",repo="example/app"} 0.0`. +- The enum `FailureClass` in `src/devops_cli/roadmap/failure.py` has 19 members. `k8s/README.md:630` says 17. This is confirmed by reading both files at origin/main. +- The action text of `RoadmapServiceGitHubRequestsHigh` in `k8s/README.md:705` says "Lower the budget or remove a repository". This is confirmed by reading the file at origin/main. +- `k8s/monitoring/prometheus-values.yaml:23` enables `created-timestamp-zero-ingestion`. The stale comment at lines 670-673 says that neither path has a zero sample. This is confirmed by reading the file at origin/main. +- The upstream statement that start-timestamp injection covers only PrometheusProto and OpenMetrics 1.0.0 scrapes is recorded from the issue research and is not checked again here. The design does not depend on it. +- The live evidence table of the issue is not checked. No live Prometheus query ran in this change. + +## Existing Tool Considered + +The open tool `prometheus_client` covers the zero series: calling `.labels(...)` at start creates the child at 0, and the text exposition shows it (see Feasibility). The open tool `promtool` checks the rules and runs their unit tests. Both are present, so the change adds no code for the exposition format. The change deletes these internal parts: the per-batch count of `devops_cli_service_jobs_total` in the worker (`src/devops_cli/server/service.py`), the two #2505 workaround terms of the webhook alerts in `k8s/monitoring/prometheus-values.yaml`, and the stale comment at lines 670-673 of the same file. The code of the zero-start function is about 80 lines. It holds the label sets of the counters, and it adds no custom metric type. diff --git a/openspec/changes/3495-service-counters-zero-series-repo-label/specs/roadmap-implement-job/spec.md b/openspec/changes/3495-service-counters-zero-series-repo-label/specs/roadmap-implement-job/spec.md new file mode 100644 index 000000000..55ae120f0 --- /dev/null +++ b/openspec/changes/3495-service-counters-zero-series-repo-label/specs/roadmap-implement-job/spec.md @@ -0,0 +1,16 @@ +## MODIFIED Requirements + +### Requirement: The Service counts the failures by class +When a round reports a Job that left a stop, the Service SHALL add 1 to `devops_cli_service_implement_failures_total` with the labels `class` and `repo`, where `repo` is the repository of the round. A Job that left the item without a result SHALL get the class of the tail of its log, recorded in `implement.failure` and named, with the action left, in the report. A Job that ended with a result SHALL add nothing. + +#### Scenario: A stop of the runner +- **WHEN** a round of `example/app` reports a Job whose item holds the stop with the class `parser` +- **THEN** the counter for `parser` and `example/app` grows by 1, the counter for `oom` does not change, and a second round adds nothing + +#### Scenario: A Job with no result +- **WHEN** a round of `example/app` reports a Job that failed and left the item In Progress, and its log holds "out of memory" +- **THEN** the item is Blocked with `implement.failure` `oom`, the report names the class and the action left `raise_limit`, and the counter for `oom` and `example/app` grows by 1 + +#### Scenario: A Job with a result +- **WHEN** a round reports a Job that succeeded and left the item In Review +- **THEN** no counter changes diff --git a/openspec/changes/3495-service-counters-zero-series-repo-label/specs/roadmap-service-counters/spec.md b/openspec/changes/3495-service-counters-zero-series-repo-label/specs/roadmap-service-counters/spec.md new file mode 100644 index 000000000..9ef4aeafb --- /dev/null +++ b/openspec/changes/3495-service-counters-zero-series-repo-label/specs/roadmap-service-counters/spec.md @@ -0,0 +1,41 @@ +## ADDED Requirements + +### Requirement: The Service starts each label set of its counters at zero +The Service SHALL create each bounded label set of its counters at zero when it starts. The bounded label sets are: each failure class for each repository of `service.repos` on `devops_cli_service_implement_failures_total`; each repository, lane, row and result on `devops_cli_service_jobs_total`; each repository and lane on `devops_cli_service_job_seconds_total`; and each provider, declared event or `other`, and outcome on `devops_cli_service_webhook_deliveries_total`. + +#### Scenario: The failure counter exists at zero after start +- **WHEN** the Service starts with the repositories `example/app` and `example/tools` +- **THEN** the failure counter exists at 0 for each of the 19 failure classes and each of the two repositories + +#### Scenario: The label repo holds only configured repositories +- **WHEN** the Service starts with the repositories `example/app` and `example/tools` +- **THEN** every value of the label `repo` of the failure counter is `example/app` or `example/tools` + +#### Scenario: The webhook counter exists at zero for each declared event +- **WHEN** the Service starts with a provider that declares the events `issues` and `push` +- **THEN** the webhook counter exists at 0 for each of the outcomes with the events `issues`, `push` and `other` + +### Requirement: The job counter carries the row of the job +The Service SHALL add 1 to `devops_cli_service_jobs_total` with the labels `repo`, `lane`, `row` and `result` for each row of a lane that ends in the roadmap run. A scoped update SHALL add 1 with the name of its job as the label `row`. The Service SHALL NOT count a lane batch as one job. + +#### Scenario: A row that succeeds +- **WHEN** the status lane runs the row `metrics` for `example/app` and the row succeeds +- **THEN** the job counter for `repo` `example/app`, `lane` `status`, `row` `metrics` and `result` `success` grows by 1 + +#### Scenario: Two rows of one batch +- **WHEN** one round of the status lane runs the rows `metrics` and `implement`, and both fail +- **THEN** the job counter for each row with `result` `error` grows by 1, and no series counts the batch + +### Requirement: Alerts read a counter that starts at zero with increase +The alerts `RoadmapServiceWebhooksRejected` and `RoadmapServiceSelfDeliveriesHigh` SHALL read their counter with `increase()` and no `offset` term, because the Service starts each label set at zero. + +#### Scenario: A rejected burst on a new pod +- **WHEN** `promtool test rules` evaluates the rejected series of a pod that starts at 0 and reaches 25 within 15 minutes +- **THEN** `RoadmapServiceWebhooksRejected` fires + +### Requirement: A query gives the true total over a pod change +A query of `sum(increase(devops_cli_service_implement_failures_total[24h]))` SHALL give the true total of the counts of all pods in the window, including the counts of a pod that started during the window. + +#### Scenario: A new pod adds its counts +- **WHEN** `promtool test rules` evaluates the failure counter of a pod that counted 3 and of a new pod that counted 2 after a zero sample at its start +- **THEN** the sum of `increase()` over the window is 5 diff --git a/openspec/changes/3495-service-counters-zero-series-repo-label/tasks.md b/openspec/changes/3495-service-counters-zero-series-repo-label/tasks.md new file mode 100644 index 000000000..463b19d5a --- /dev/null +++ b/openspec/changes/3495-service-counters-zero-series-repo-label/tasks.md @@ -0,0 +1,58 @@ +# Tasks + +## 1. Zero series and the repo label of the failure counter + +Reads: `src/devops_cli/roadmap/implement_submit.py`, `src/devops_cli/telemetry/metrics.py`, `src/devops_cli/server/service.py`, `tests/prometheus/test_implement_failure_labels.py`. Changes: `src/devops_cli/telemetry/metrics.py`, `src/devops_cli/roadmap/implement_submit.py`, `src/devops_cli/roadmap/run.py`, `src/devops_cli/server/service.py`, `src/devops_cli/server/counter_series.py`, `tests/prometheus/test_implement_failure_labels.py`. + +- [ ] 1.1 Add a test to `tests/prometheus/test_implement_failure_labels.py` that starts the Service with two repositories and asserts that the failure counter exists at 0 for each of the 19 classes and each repository + - Verification: `uv run devops ci test tests/prometheus/test_implement_failure_labels.py` fails on the base +- [ ] 1.2 Add a test to `tests/prometheus/test_implement_failure_labels.py` that asserts that every value of the label `repo` of the failure counter is a value of `service.repos` + - Verification: `uv run devops ci test tests/prometheus/test_implement_failure_labels.py` fails on the base +- [ ] 1.3 Add the function that creates the bounded label sets of the Service counters at zero in `src/devops_cli/server/counter_series.py`, call it at the start of the Service, and pass `repo` from `run_implement_round` to the failure count of `src/devops_cli/roadmap/implement_submit.py` + - Verification: `uv run devops ci test tests/prometheus/test_implement_failure_labels.py` passes + +## 2. The job counter carries the row + +Reads: `src/devops_cli/roadmap/run.py`, `src/devops_cli/server/service.py`, `tests/roadmap/test_roadmap_run.py`. Changes: `tests/telemetry/test_service_counter_series.py`, `src/devops_cli/roadmap/run.py`, `src/devops_cli/server/service.py`. + +- [ ] 2.1 Add `tests/telemetry/test_service_counter_series.py` with a test that runs one row of a lane and asserts that the job counter holds the labels `repo`, `lane`, `row` and `result`, and that each row of a lane has its own count + - Verification: `uv run devops ci test tests/telemetry/test_service_counter_series.py` fails on the base +- [ ] 2.2 Count each row of a lane in the roadmap run with its result, and remove the per-batch count of the worker in `src/devops_cli/server/service.py`. Count each scoped update with the name of its job + - Verification: `uv run devops ci test tests/telemetry/test_service_counter_series.py tests/roadmap/test_roadmap_run.py` passes +- [ ] 2.3 Prime the job counter for each repository, lane, row and result, and the job seconds counter for each repository and lane, in `src/devops_cli/server/counter_series.py` + - Verification: `uv run devops ci test tests/telemetry/test_service_counter_series.py` passes + +## 3. The webhook counter starts at zero + +Reads: `src/devops_cli/server/routes/webhooks.py`, `src/devops_cli/server/counter_series.py`. Changes: `src/devops_cli/server/counter_series.py`, `tests/telemetry/test_service_counter_series.py`. + +- [ ] 3.1 Add a test to `tests/telemetry/test_service_counter_series.py` that asserts that the webhook counter exists at 0 for each provider, each declared event with `other`, and each outcome + - Verification: `uv run devops ci test tests/telemetry/test_service_counter_series.py` fails on the base +- [ ] 3.2 Prime the webhook counter in `src/devops_cli/server/counter_series.py` from the declared providers and their events + - Verification: `uv run devops ci test tests/telemetry/test_service_counter_series.py` passes + +## 4. The promtool cases hold the rules and the zero series + +Reads: `tests/fixtures/prometheus_rules/roadmap_service_tests.yaml`, `tests/fixtures/prometheus_rules/roadmap_service_webhooks_tests.yaml`, `tests/prometheus/test_k8s_monitoring_alerting.py`, `k8s/monitoring/prometheus-values.yaml`. Changes: `tests/fixtures/prometheus_rules/roadmap_service_tests.yaml`, `tests/fixtures/prometheus_rules/roadmap_service_webhooks_tests.yaml`, `k8s/monitoring/prometheus-values.yaml`. + +- [ ] 4.1 Add a promtool case `pod_change_gives_true_total` to `tests/fixtures/prometheus_rules/roadmap_service_tests.yaml`. Its series of the failure counter start at 0 in each pod, and the case asserts the sum of `increase()` over a pod change + - Verification: `promtool test rules tests/fixtures/prometheus_rules/roadmap_service_tests.yaml` passes +- [ ] 4.2 Add a promtool case to `tests/fixtures/prometheus_rules/roadmap_service_webhooks_tests.yaml` that fires `RoadmapServiceWebhooksRejected` on a rejected series that starts at 0 + - Verification: `promtool test rules tests/fixtures/prometheus_rules/roadmap_service_webhooks_tests.yaml` passes +- [ ] 4.3 Remove the #2505 workaround terms of `RoadmapServiceWebhooksRejected` and `RoadmapServiceSelfDeliveriesHigh` from `k8s/monitoring/prometheus-values.yaml`, and replace the stale comment at lines 670-673 with the scrape behavior. Keep the OTLP terms + - Verification: `uv run devops ci test tests/prometheus/test_k8s_monitoring_alerting.py` passes and `promtool test rules tests/fixtures/prometheus_rules/roadmap_service_webhooks_tests.yaml` passes + +## 5. The documents state the scrape behavior and the labels + +Reads: `docs/TELEMETRY.md`, `k8s/README.md`. Changes: `docs/TELEMETRY.md`, `k8s/README.md`, `tests/prometheus/test_k8s_monitoring_alerting.py`. + +- [ ] 5.1 Add a test to `tests/prometheus/test_k8s_monitoring_alerting.py` that asserts that the action text of `RoadmapServiceGitHubRequestsHigh` in `k8s/README.md` does not say to lower the budget + - Verification: `uv run devops ci test tests/prometheus/test_k8s_monitoring_alerting.py` fails on the base +- [ ] 5.2 Rewrite the action text of `RoadmapServiceGitHubRequestsHigh` in `k8s/README.md` without the word to lower the budget, rewrite `docs/TELEMETRY.md` lines 19-32 to state the scrape behavior and the labels, state the class count of 19 in `k8s/README.md`, and run `uv run devops docs generate --sync-readme` + - Verification: `uv run devops ci test tests/prometheus/test_k8s_monitoring_alerting.py` passes and `uv run devops ci docs` passes + +## Workflow follow-up + +- Open item: agree the label `repo` of `devops_cli_service_implement_failures_total` with #3069 before this change lands. The issue says that #3069 reuses the label `class`. +- Open item: check the live series with a Prometheus query after the deploy. This change runs no live query. +- Pending a person: review the diff in tuicr (tuicr -r ..HEAD). From 6613ed1ec7a826dbacc1c507da4dd49bb21046f3 Mon Sep 17 00:00:00 2001 From: Daniel Petty Date: Thu, 5 Nov 2026 03:28:06 +0000 Subject: [PATCH 02/17] fix(telemetry): failure counter starts at zero for each repository The Service creates each failure class for each repository at zero when it starts, so increase() reads the first count of each pod. The failure counter adds the label repo, the round passes its repository down to the report. Change: openspec/changes/3495-service-counters-zero-series-repo-label --- .../tasks.md | 18 ++-- src/devops_cli/roadmap/implement_submit.py | 18 ++-- src/devops_cli/server/counter_series.py | 24 ++++++ src/devops_cli/server/service.py | 2 + src/devops_cli/telemetry/metrics.py | 11 +++ .../roadmap/test_roadmap_implement_submit.py | 6 +- .../telemetry/test_service_counter_series.py | 85 +++++++++++++++++++ 7 files changed, 146 insertions(+), 18 deletions(-) create mode 100644 src/devops_cli/server/counter_series.py create mode 100644 tests/telemetry/test_service_counter_series.py diff --git a/openspec/changes/3495-service-counters-zero-series-repo-label/tasks.md b/openspec/changes/3495-service-counters-zero-series-repo-label/tasks.md index 463b19d5a..4c95294fe 100644 --- a/openspec/changes/3495-service-counters-zero-series-repo-label/tasks.md +++ b/openspec/changes/3495-service-counters-zero-series-repo-label/tasks.md @@ -2,20 +2,20 @@ ## 1. Zero series and the repo label of the failure counter -Reads: `src/devops_cli/roadmap/implement_submit.py`, `src/devops_cli/telemetry/metrics.py`, `src/devops_cli/server/service.py`, `tests/prometheus/test_implement_failure_labels.py`. Changes: `src/devops_cli/telemetry/metrics.py`, `src/devops_cli/roadmap/implement_submit.py`, `src/devops_cli/roadmap/run.py`, `src/devops_cli/server/service.py`, `src/devops_cli/server/counter_series.py`, `tests/prometheus/test_implement_failure_labels.py`. +Reads: `src/devops_cli/roadmap/implement_submit.py`, `src/devops_cli/telemetry/metrics.py`, `src/devops_cli/server/service.py`, `tests/roadmap/test_roadmap_implement_submit.py`. Changes: `src/devops_cli/telemetry/metrics.py`, `src/devops_cli/roadmap/implement_submit.py`, `src/devops_cli/server/service.py`, `src/devops_cli/server/counter_series.py`, `tests/telemetry/test_service_counter_series.py`, `tests/roadmap/test_roadmap_implement_submit.py`. -- [ ] 1.1 Add a test to `tests/prometheus/test_implement_failure_labels.py` that starts the Service with two repositories and asserts that the failure counter exists at 0 for each of the 19 classes and each repository - - Verification: `uv run devops ci test tests/prometheus/test_implement_failure_labels.py` fails on the base -- [ ] 1.2 Add a test to `tests/prometheus/test_implement_failure_labels.py` that asserts that every value of the label `repo` of the failure counter is a value of `service.repos` - - Verification: `uv run devops ci test tests/prometheus/test_implement_failure_labels.py` fails on the base -- [ ] 1.3 Add the function that creates the bounded label sets of the Service counters at zero in `src/devops_cli/server/counter_series.py`, call it at the start of the Service, and pass `repo` from `run_implement_round` to the failure count of `src/devops_cli/roadmap/implement_submit.py` - - Verification: `uv run devops ci test tests/prometheus/test_implement_failure_labels.py` passes +- [x] 1.1 Add a test to `tests/telemetry/test_service_counter_series.py` that builds the Service manager with two repositories and asserts that the failure counter exists at 0 for each of the 19 classes and each repository + - Verification: `uv run devops ci test tests/telemetry/test_service_counter_series.py` fails on the base +- [x] 1.2 Add a test to `tests/telemetry/test_service_counter_series.py` that asserts that every value of the label `repo` of the failure counter is a value of `service.repos`, and change the counter assertions of `tests/roadmap/test_roadmap_implement_submit.py` to read the label `repo` + - Verification: `uv run devops ci test tests/telemetry/test_service_counter_series.py tests/roadmap/test_roadmap_implement_submit.py` fails on the base +- [x] 1.3 Add the function that creates the bounded label sets of the Service counters at zero in `src/devops_cli/server/counter_series.py`, call it when the Service manager is built, and pass `repo` from `run_implement_round` to the failure count of `src/devops_cli/roadmap/implement_submit.py` + - Verification: `uv run devops ci test tests/telemetry/test_service_counter_series.py tests/roadmap/test_roadmap_implement_submit.py` passes ## 2. The job counter carries the row -Reads: `src/devops_cli/roadmap/run.py`, `src/devops_cli/server/service.py`, `tests/roadmap/test_roadmap_run.py`. Changes: `tests/telemetry/test_service_counter_series.py`, `src/devops_cli/roadmap/run.py`, `src/devops_cli/server/service.py`. +Reads: `src/devops_cli/roadmap/run.py`, `src/devops_cli/server/service.py`, `tests/roadmap/test_roadmap_run.py`. Changes: `tests/telemetry/test_service_counter_series.py`, `src/devops_cli/roadmap/run.py`, `src/devops_cli/server/service.py`, `src/devops_cli/server/counter_series.py`. -- [ ] 2.1 Add `tests/telemetry/test_service_counter_series.py` with a test that runs one row of a lane and asserts that the job counter holds the labels `repo`, `lane`, `row` and `result`, and that each row of a lane has its own count +- [ ] 2.1 Add a test to `tests/telemetry/test_service_counter_series.py` that asserts that the job counter holds the labels `repo`, `lane`, `row` and `result`, and that the row `metrics` of the status lane exists at 0 for `result` `success` - Verification: `uv run devops ci test tests/telemetry/test_service_counter_series.py` fails on the base - [ ] 2.2 Count each row of a lane in the roadmap run with its result, and remove the per-batch count of the worker in `src/devops_cli/server/service.py`. Count each scoped update with the name of its job - Verification: `uv run devops ci test tests/telemetry/test_service_counter_series.py tests/roadmap/test_roadmap_run.py` passes diff --git a/src/devops_cli/roadmap/implement_submit.py b/src/devops_cli/roadmap/implement_submit.py index 011064e57..734c5569b 100644 --- a/src/devops_cli/roadmap/implement_submit.py +++ b/src/devops_cli/roadmap/implement_submit.py @@ -35,7 +35,7 @@ time of the create, recorded with the Job. **Failure class** (#3065). Each Job that left a stop is counted once, in the counter -`devops_cli_service_implement_failures_total` by `class`, when the round reports it. The runner +`devops_cli_service_implement_failures_total` by `class` and `repo`, when the round reports it. The runner records the class of its own stop in `implement.failure`. A Job that left the item without a result has only its log, so the round classifies that log (`failure.py`), records the class with the stop, and names the class and the action left in its comment. The runner pod is not scraped, so the Service @@ -227,7 +227,12 @@ def _log_of(jobs: ClusterJobs, name: str) -> str: def _report( - store: RoadmapStore, jobs: ClusterJobs, item: Item, name: str, ended: ClusterJob | None + store: RoadmapStore, + jobs: ClusterJobs, + repo: str, + item: Item, + name: str, + ended: ClusterJob | None, ) -> None: """Comment on the item what became of its Job, record the state, and stop the item when the Job left it eligible. The comment comes first, so a write that fails is reported again. The @@ -258,19 +263,20 @@ def _report( ) if failure is not None: GLOBAL_METRICS.increment_counter( - CONST_SERVICE_METRIC_IMPLEMENT_FAILURES, labels={"class": failure.value} + CONST_SERVICE_METRIC_IMPLEMENT_FAILURES, + labels={"class": failure.value, "repo": repo}, ) def _report_ended( - store: RoadmapStore, jobs: ClusterJobs, held: Mapping[str, ClusterJob] + store: RoadmapStore, jobs: ClusterJobs, held: Mapping[str, ClusterJob], repo: str ) -> tuple[str, ...]: reported: list[str] = [] for item, name in _unreported(store): job = held.get(name) if job is not None and job.state is JobState.RUNNING: continue - _report(store, jobs, item, name, job) + _report(store, jobs, repo, item, name, job) reported.append(name) return tuple(reported) @@ -333,7 +339,7 @@ def run_implement_round( api = jobs or ClusterJobs() label = repository_label(repo) held = {job.name: job for job in api.list_jobs({CONST_ROADMAP_IMPLEMENT_JOB_LABEL: label})} - reported = _report_ended(store, api, held) + reported = _report_ended(store, api, held, repo) manage_deadlines(api, held.values()) planned = read_planned_changes(Path(source), operation=GIT_OPERATION) swept = sync_statuses(store, repo=repo, planned=planned).failed diff --git a/src/devops_cli/server/counter_series.py b/src/devops_cli/server/counter_series.py new file mode 100644 index 000000000..8ee963723 --- /dev/null +++ b/src/devops_cli/server/counter_series.py @@ -0,0 +1,24 @@ +"""The label sets of the Service counters that exist at zero from the start (#3495). + +A labelled counter has no series until its first count, and `increase()` loses that first count. +So the Service creates each bounded label set at zero when it starts, before the first scrape of +its `/metrics`. A label set that a new pod counts later still starts at zero in that pod. +""" + +from __future__ import annotations + +from collections.abc import Iterable + +from devops_cli.config.constants import CONST_SERVICE_METRIC_IMPLEMENT_FAILURES +from devops_cli.roadmap.failure import FailureClass +from devops_cli.telemetry.metrics import GLOBAL_METRICS + + +def prime_service_counters(repos: Iterable[str]) -> None: + """Create the bounded label sets of the Service counters at zero for the repositories + `repos`: each failure class for each repository on the failure counter.""" + repo_names = list(repos) + GLOBAL_METRICS.prime_counter( + CONST_SERVICE_METRIC_IMPLEMENT_FAILURES, + ({"class": klass.value, "repo": repo} for klass in FailureClass for repo in repo_names), + ) diff --git a/src/devops_cli/server/service.py b/src/devops_cli/server/service.py index c5bef90b7..f6dfb93ed 100644 --- a/src/devops_cli/server/service.py +++ b/src/devops_cli/server/service.py @@ -56,6 +56,7 @@ from devops_cli.github.rate_limiter import set_request_gate from devops_cli.github.request_scope import request_scope from devops_cli.lang import MESSAGES +from devops_cli.server.counter_series import prime_service_counters from devops_cli.server.routes.webhooks import ( DeliveryLRUCache, WebhookConsumer, @@ -757,6 +758,7 @@ def __init__( for repo in config.repos for lane, job in jobs.items() } + prime_service_counters(config.repos) self._stop_event = threading.Event() self._ready = False self._draining = False diff --git a/src/devops_cli/telemetry/metrics.py b/src/devops_cli/telemetry/metrics.py index 01e3516a3..6df252aa8 100644 --- a/src/devops_cli/telemetry/metrics.py +++ b/src/devops_cli/telemetry/metrics.py @@ -5,6 +5,7 @@ import contextlib import threading import time +from collections.abc import Iterable from typing import Any from prometheus_client import CollectorRegistry, Counter, Gauge, Histogram, generate_latest @@ -99,6 +100,16 @@ def increment_counter( else: collector.inc(value) + def prime_counter(self, name: str, label_sets: Iterable[dict[str, str]]) -> None: + """Create the child of each label set of the counter `name` at zero, so that its series + exists before its first count. An `increase()` over a window loses that first count + (#3495).""" + with self._lock: + for labels in label_sets: + collector, label_keys = self._get_or_create_collector(name, Counter, labels) + if label_keys: + collector.labels(*(str(labels[k]) for k in label_keys)) + def set_gauge( self, name: str, diff --git a/tests/roadmap/test_roadmap_implement_submit.py b/tests/roadmap/test_roadmap_implement_submit.py index 77af3714f..4ee8af1e3 100644 --- a/tests/roadmap/test_roadmap_implement_submit.py +++ b/tests/roadmap/test_roadmap_implement_submit.py @@ -398,10 +398,10 @@ def test_a_job_that_failed_stops_an_item_that_is_in_progress_and_no_job_is_made_ ) == ("Blocked", "failed", "job", True, True, 1) -def _failures(failure_class: str) -> float: - """The count of the Service's counter of failures for `failure_class`.""" +def _failures(failure_class: str, repo: str = STAGE_REPO) -> float: + """The count of the Service's counter of failures for `failure_class` in `repo` (#3495).""" return GLOBAL_METRICS.get_counter( - CONST_SERVICE_METRIC_IMPLEMENT_FAILURES, labels={"class": failure_class} + CONST_SERVICE_METRIC_IMPLEMENT_FAILURES, labels={"class": failure_class, "repo": repo} ) diff --git a/tests/telemetry/test_service_counter_series.py b/tests/telemetry/test_service_counter_series.py new file mode 100644 index 000000000..8bc71909f --- /dev/null +++ b/tests/telemetry/test_service_counter_series.py @@ -0,0 +1,85 @@ +"""The label sets of the Service counters exist at zero from the start (#3495). + +A labelled counter has no series until its first count, and `increase()` loses that first count, +so each pod of the Service starts each bounded label set at zero. The tests read the registry of +the process, as the `/metrics` scrape does, and start no thread. +""" + +from __future__ import annotations + +from datetime import UTC, datetime + +from devops_cli.config.constants import ( + CONST_SERVICE_METRIC_IMPLEMENT_FAILURES, + CONST_SERVICE_METRIC_JOBS, +) +from devops_cli.config.settings import GitHubAppConfig, ServiceConfig +from devops_cli.roadmap.failure import FailureClass +from devops_cli.server.service import ServiceManager +from devops_cli.telemetry.metrics import GLOBAL_METRICS + +REPOS = ["example/app", "example/tools"] + + +def _manager(repos: list[str]) -> ServiceManager: + """A Service manager with two lanes of dummy jobs, which starts no thread. The registry of + the process starts empty, so a series that another test counted is not read here.""" + GLOBAL_METRICS.reset() + config = ServiceConfig(repos=repos, github_app=GitHubAppConfig(bot_login="test-bot[bot]")) + now = datetime(2026, 11, 5, 12, 0, 0, tzinfo=UTC) + return ServiceManager( + config=config, + secrets={}, + jobs={"status": lambda _batch: None, "model": lambda _batch: None}, + clock=lambda: now, + sleep_func=lambda _seconds: _noop(), + ) + + +async def _noop() -> None: + """A sleep that returns at once: the manager's tick loop never runs in these tests.""" + + +def _series(name: str) -> dict[frozenset[tuple[str, str]], float]: + """Each sample of the counter `name` in the registry, keyed by its label set.""" + return { + frozenset(sample.labels.items()): sample.value + for family in GLOBAL_METRICS.registry.collect() + for sample in family.samples + if sample.name == name + } + + +def _label_values(name: str, label: str) -> set[str]: + """Every value of the label `label` of the counter `name` in the registry.""" + return {value for labels in _series(name) for key, value in labels if key == label} + + +def test_each_failure_class_exists_at_zero_for_each_repository() -> None: + _manager(REPOS) + + series = _series(CONST_SERVICE_METRIC_IMPLEMENT_FAILURES) + expected = { + frozenset({("class", klass.value), ("repo", repo)}): 0.0 + for klass in FailureClass + for repo in REPOS + } + + assert {key: series.get(key) for key in expected} == expected + + +def test_repo_label_holds_only_service_repos() -> None: + _manager(REPOS) + + assert _label_values(CONST_SERVICE_METRIC_IMPLEMENT_FAILURES, "repo") == set(REPOS) + + +def test_jobs_total_carries_row_label() -> None: + _manager(REPOS) + + label_sets = [dict(key) for key in _series(CONST_SERVICE_METRIC_JOBS)] + + assert all(set(labels) == {"repo", "lane", "row", "result"} for labels in label_sets) + assert {"repo": "example/app", "lane": "status", "row": "metrics", "result": "success"} in ( + label_sets + ) From 0008fdd3712d2940aa146ea860724d6f77fd48d3 Mon Sep 17 00:00:00 2001 From: Daniel Petty Date: Thu, 5 Nov 2026 03:39:37 +0000 Subject: [PATCH 03/17] fix(telemetry): job counter carries the row and starts at zero The roadmap run counts each row of a lane with its repository, lane, row and result, and a scoped update counts its job. The worker no longer counts a whole lane batch. The Service primes the job and job seconds counters at zero for each repository and lane. Change: openspec/changes/3495-service-counters-zero-series-repo-label --- .../tasks.md | 6 +-- src/devops_cli/roadmap/run.py | 42 +++++++++++++++---- src/devops_cli/server/counter_series.py | 38 ++++++++++++++++- src/devops_cli/server/service.py | 7 ---- tests/roadmap/test_roadmap_run.py | 37 ++++++++++++++-- 5 files changed, 108 insertions(+), 22 deletions(-) diff --git a/openspec/changes/3495-service-counters-zero-series-repo-label/tasks.md b/openspec/changes/3495-service-counters-zero-series-repo-label/tasks.md index 4c95294fe..ff675424d 100644 --- a/openspec/changes/3495-service-counters-zero-series-repo-label/tasks.md +++ b/openspec/changes/3495-service-counters-zero-series-repo-label/tasks.md @@ -15,11 +15,11 @@ Reads: `src/devops_cli/roadmap/implement_submit.py`, `src/devops_cli/telemetry/m Reads: `src/devops_cli/roadmap/run.py`, `src/devops_cli/server/service.py`, `tests/roadmap/test_roadmap_run.py`. Changes: `tests/telemetry/test_service_counter_series.py`, `src/devops_cli/roadmap/run.py`, `src/devops_cli/server/service.py`, `src/devops_cli/server/counter_series.py`. -- [ ] 2.1 Add a test to `tests/telemetry/test_service_counter_series.py` that asserts that the job counter holds the labels `repo`, `lane`, `row` and `result`, and that the row `metrics` of the status lane exists at 0 for `result` `success` +- [x] 2.1 Add a test to `tests/telemetry/test_service_counter_series.py` that asserts that the job counter holds the labels `repo`, `lane`, `row` and `result`, and that the row `metrics` of the status lane exists at 0 for `result` `success`. Add a test to `tests/roadmap/test_roadmap_run.py` that runs a lane with one row that fails, and asserts that each row counts its own job under its result - Verification: `uv run devops ci test tests/telemetry/test_service_counter_series.py` fails on the base -- [ ] 2.2 Count each row of a lane in the roadmap run with its result, and remove the per-batch count of the worker in `src/devops_cli/server/service.py`. Count each scoped update with the name of its job +- [x] 2.2 Count each row of a lane in the roadmap run with its result, and remove the per-batch count of the worker in `src/devops_cli/server/service.py`. Count each scoped update with the name of its job - Verification: `uv run devops ci test tests/telemetry/test_service_counter_series.py tests/roadmap/test_roadmap_run.py` passes -- [ ] 2.3 Prime the job counter for each repository, lane, row and result, and the job seconds counter for each repository and lane, in `src/devops_cli/server/counter_series.py` +- [x] 2.3 Prime the job counter for each repository, lane, row and result, and the job seconds counter for each repository and lane, in `src/devops_cli/server/counter_series.py` - Verification: `uv run devops ci test tests/telemetry/test_service_counter_series.py` passes ## 3. The webhook counter starts at zero diff --git a/src/devops_cli/roadmap/run.py b/src/devops_cli/roadmap/run.py index 6a40d7f06..d93e73e9f 100644 --- a/src/devops_cli/roadmap/run.py +++ b/src/devops_cli/roadmap/run.py @@ -63,6 +63,7 @@ CONST_SERVICE_LANE_STATUS, CONST_SERVICE_LANE_TRIAGE, CONST_SERVICE_METRIC_JOB_SUSPENDED, + CONST_SERVICE_METRIC_JOBS, CONST_SERVICE_SOURCE_POLL, ) from devops_cli.config.defaults import ( @@ -830,13 +831,19 @@ def run_scoped_update( CONST_ROADMAP_RUN_STATE_FILENAME, error_cls=GitOperationError, ) - return update.runner( - store, - delivery, - repo_dir=schedule_path.parent, - now=now or datetime.now(UTC), - model=model, - ) + try: + outcome = update.runner( + store, + delivery, + repo_dir=schedule_path.parent, + now=now or datetime.now(UTC), + model=model, + ) + except Exception: + _count_job(delivery.repo, update.lane, update.job, "error") + raise + _count_job(delivery.repo, update.lane, update.job, "success") + return outcome def build_stub_table( @@ -1146,6 +1153,16 @@ def _format_error_line(exc: BaseException) -> str: return mask_secrets(first)[:256] +def _count_job(repo: str, lane: str | None, row: str, result: str) -> None: + """Count one job that ended under its repository, lane, row and result (#3495). A run with + no lane, such as one of the command line, counts no job.""" + if lane is not None: + GLOBAL_METRICS.increment_counter( + CONST_SERVICE_METRIC_JOBS, + labels={"repo": repo, "lane": lane, "row": row, "result": result}, + ) + + def _execute_due_rows( rows: Sequence[JobRow], initially_due: Mapping[str, bool], @@ -1159,6 +1176,7 @@ def _execute_due_rows( current_time: datetime, clone_dirname: str, failures: FailureLedger, + lane: str | None, ) -> tuple[str, ...]: """Execute all due jobs in order with failure tracking, the clone rows in `clone_dirname`. Each runner also gets the repository's data directory and the round's time. With the @@ -1185,6 +1203,7 @@ def _execute_due_rows( if row.needs_clone and clone_error is not None: failed_jobs.append(row.name) clone_jobs.append(row.name) + _count_job(repo, lane, row.name, "error") continue try: @@ -1205,8 +1224,10 @@ def _execute_due_rows( succeeded_jobs.append(row.name) _record_success(schedule_path, row.name, current_time) failures.succeeded(repo, row.name) + _count_job(repo, lane, row.name, "success") except Exception as exc: failed_jobs.append(row.name) + _count_job(repo, lane, row.name, "error") outcomes[row.name] = _outcome_of_failure(exc) _count_failure(failures, repo, row.name, current, exc) if row.needs_clone and clone_path is None: @@ -1244,6 +1265,7 @@ def run_due_jobs( now: datetime | None = None, clone_dirname: str = CONST_ROADMAP_RUN_CLONE_DIRNAME, failures: FailureLedger | None = None, + lane: str | None = None, ) -> tuple[str, ...]: """Evaluate and execute due roadmap jobs, the clone rows in `clone_dirname`. @@ -1288,6 +1310,7 @@ def run_due_jobs( current_time, clone_dirname, failures or FailureLedger(), + lane, ) @@ -1296,6 +1319,7 @@ def service_job( *, table: Sequence[JobRow], clone_dirname: str, + lane: str | None = None, failures: FailureLedger | None = None, ) -> None: """Service mode adapter that runs one lane's due roadmap jobs, `table`, in its clone @@ -1312,6 +1336,7 @@ def service_job( table=table, clone_dirname=clone_dirname, failures=failures, + lane=lane, ) @@ -1325,18 +1350,21 @@ def service_jobs(state_dir: Path) -> Mapping[str, LaneJob]: service_job, table=STATUS_LANE_ROWS, clone_dirname=CONST_ROADMAP_RUN_CLONE_DIRNAME, + lane=CONST_SERVICE_LANE_STATUS, failures=failures, ), CONST_SERVICE_LANE_TRIAGE: partial( service_job, table=TRIAGE_LANE_ROWS, clone_dirname=CONST_ROADMAP_RUN_TRIAGE_CLONE_DIRNAME, + lane=CONST_SERVICE_LANE_TRIAGE, failures=failures, ), CONST_SERVICE_LANE_MODEL: partial( service_job, table=MODEL_LANE_ROWS, clone_dirname=CONST_ROADMAP_RUN_MODEL_CLONE_DIRNAME, + lane=CONST_SERVICE_LANE_MODEL, failures=failures, ), } diff --git a/src/devops_cli/server/counter_series.py b/src/devops_cli/server/counter_series.py index 8ee963723..50ba40ce4 100644 --- a/src/devops_cli/server/counter_series.py +++ b/src/devops_cli/server/counter_series.py @@ -9,16 +9,50 @@ from collections.abc import Iterable -from devops_cli.config.constants import CONST_SERVICE_METRIC_IMPLEMENT_FAILURES +from devops_cli.config.constants import ( + CONST_SERVICE_LANE_MODEL, + CONST_SERVICE_LANE_STATUS, + CONST_SERVICE_LANE_TRIAGE, + CONST_SERVICE_METRIC_IMPLEMENT_FAILURES, + CONST_SERVICE_METRIC_JOB_SECONDS, + CONST_SERVICE_METRIC_JOBS, +) from devops_cli.roadmap.failure import FailureClass from devops_cli.telemetry.metrics import GLOBAL_METRICS +SERVICE_JOB_RESULTS = ("success", "error") +"""The value of the label `result` of a job that ended, as the roadmap run counts it.""" + def prime_service_counters(repos: Iterable[str]) -> None: """Create the bounded label sets of the Service counters at zero for the repositories - `repos`: each failure class for each repository on the failure counter.""" + `repos`: each failure class for each repository on the failure counter, each repository, + lane, row and result on the job counter, and each repository and lane on the job seconds.""" + # The roadmap run imports the Service, so the lane tables are read when the counters are + # primed, not when this module is imported. + from devops_cli.roadmap.run import MODEL_LANE_ROWS, STATUS_LANE_ROWS, TRIAGE_LANE_ROWS + + lane_rows = ( + (CONST_SERVICE_LANE_STATUS, STATUS_LANE_ROWS), + (CONST_SERVICE_LANE_TRIAGE, TRIAGE_LANE_ROWS), + (CONST_SERVICE_LANE_MODEL, MODEL_LANE_ROWS), + ) repo_names = list(repos) GLOBAL_METRICS.prime_counter( CONST_SERVICE_METRIC_IMPLEMENT_FAILURES, ({"class": klass.value, "repo": repo} for klass in FailureClass for repo in repo_names), ) + GLOBAL_METRICS.prime_counter( + CONST_SERVICE_METRIC_JOBS, + ( + {"repo": repo, "lane": lane, "row": row.name, "result": result} + for lane, rows in lane_rows + for row in rows + for repo in repo_names + for result in SERVICE_JOB_RESULTS + ), + ) + GLOBAL_METRICS.prime_counter( + CONST_SERVICE_METRIC_JOB_SECONDS, + ({"repo": repo, "lane": lane} for lane, _rows in lane_rows for repo in repo_names), + ) diff --git a/src/devops_cli/server/service.py b/src/devops_cli/server/service.py index f6dfb93ed..90cc428bf 100644 --- a/src/devops_cli/server/service.py +++ b/src/devops_cli/server/service.py @@ -34,7 +34,6 @@ CONST_SERVICE_METRIC_IDENTITY_REFUSED, CONST_SERVICE_METRIC_JOB_SECONDS, CONST_SERVICE_METRIC_JOB_START_TIMESTAMP, - CONST_SERVICE_METRIC_JOBS, CONST_SERVICE_METRIC_QUEUE_DEPTH, CONST_SERVICE_METRIC_TRIGGERS, CONST_SERVICE_PROBE_PATHS, @@ -600,9 +599,6 @@ def _execute_scoped(self, scoped_func: Callable[[], None]) -> None: finally: span.set_attribute("service.result", result) duration = time.perf_counter() - start_time - GLOBAL_METRICS.increment_counter( - CONST_SERVICE_METRIC_JOBS, labels={**self._labels, "result": result} - ) GLOBAL_METRICS.increment_counter( CONST_SERVICE_METRIC_JOB_SECONDS, value=duration, labels=self._labels ) @@ -661,9 +657,6 @@ def _execute_batch(self, batch: TriggerBatch) -> None: finally: span.set_attribute("service.result", result) duration = time.perf_counter() - start_time - GLOBAL_METRICS.increment_counter( - CONST_SERVICE_METRIC_JOBS, labels={**self._labels, "result": result} - ) GLOBAL_METRICS.increment_counter( CONST_SERVICE_METRIC_JOB_SECONDS, value=duration, labels=self._labels ) diff --git a/tests/roadmap/test_roadmap_run.py b/tests/roadmap/test_roadmap_run.py index e4004e1ae..3001ba5e9 100644 --- a/tests/roadmap/test_roadmap_run.py +++ b/tests/roadmap/test_roadmap_run.py @@ -49,6 +49,7 @@ CONST_SERVICE_LANE_STATUS, CONST_SERVICE_LANE_TRIAGE, CONST_SERVICE_METRIC_JOB_SUSPENDED, + CONST_SERVICE_METRIC_JOBS, CONST_SERVICE_METRIC_QUEUE_DEPTH, CONST_URL_CLOUDFLARE_STATUS_SUMMARY, CONST_URL_GITHUB_STATUS_SUMMARY, @@ -1972,11 +1973,15 @@ def _service_github(events: list[dict[str, Any]]) -> GitHubFake: return github -def _serve_over(tmp_path: Path, monkeypatch: pytest.MonkeyPatch, gh: Callable[..., Any]) -> None: +def _serve_over( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch, gh: Callable[..., Any], *, stamped: bool = True +) -> None: """`service_job` opens its stores over `gh`, logged in as `roadmap-bot`, with every job of - the default table run a minute ago, so a poll runs only what changes since then make due.""" + the default table run a minute ago, so a poll runs only what changes since then make due. + Unstamped, no job has run yet, so each row with a first run due is due on the first poll.""" minute_ago = datetime.now(UTC) - timedelta(minutes=1) - _stamp(tmp_path, **dict.fromkeys(DEFAULT_JOB_NAMES, minute_ago)) + if stamped: + _stamp(tmp_path, **dict.fromkeys(DEFAULT_JOB_NAMES, minute_ago)) monkeypatch.setenv("DEVOPS_CLI_DATA_DIR", str(tmp_path)) monkeypatch.setattr("devops_cli.roadmap.run._get_session_login", lambda: "roadmap-bot") monkeypatch.setattr( @@ -2921,3 +2926,29 @@ def test_the_release_comments_adapter_comments_a_release_and_keeps_its_ledger( assert github.comments[101][0].startswith("") assert (tmp_path / "release_comments.json").is_file() + + +def test_each_row_of_a_lane_counts_its_own_job( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch +) -> None: + """A row that ends counts one job of the job counter under its own row and result, so a lane + with one row that fails counts that row as an error and the other row as a success (#3495).""" + GLOBAL_METRICS.reset() + github = _service_github(_labeled(2, "alice")) + _serve_over(tmp_path, monkeypatch, github, stamped=False) + + def fails(**_: object) -> JobOutcome: + raise RuntimeError("the implement job failed") + + table = build_stub_table({"metrics": lambda **_: JobOutcome(), "implement": fails}) + with pytest.raises(RoadmapRunError) as raised: + service_jobs(tmp_path)[CONST_SERVICE_LANE_STATUS](POLL, table=table) + assert list(raised.value.failed_jobs) == ["implement"] + + def count(row: str, result: str) -> float: + return GLOBAL_METRICS.get_counter( + CONST_SERVICE_METRIC_JOBS, + labels={"repo": REPO, "lane": CONST_SERVICE_LANE_STATUS, "row": row, "result": result}, + ) + + assert (count("metrics", "success"), count("implement", "error")) == (1, 1) From 2c31833ba416c6f845a136f9871cd084cc6299e2 Mon Sep 17 00:00:00 2001 From: Daniel Petty Date: Thu, 5 Nov 2026 03:49:40 +0000 Subject: [PATCH 04/17] fix(telemetry): webhook counter starts at zero for each event The Service creates each declared event of each provider, with other, and each outcome of the webhook deliveries counter at zero when its routes are built. The server test reads the series that count, not the series at zero. Change: openspec/changes/3495-service-counters-zero-series-repo-label --- .../tasks.md | 6 +-- src/devops_cli/server/routes/webhooks.py | 39 ++++++++++++++++++- tests/server/test_server_service.py | 15 ++++--- .../telemetry/test_service_counter_series.py | 29 ++++++++++++++ 4 files changed, 78 insertions(+), 11 deletions(-) diff --git a/openspec/changes/3495-service-counters-zero-series-repo-label/tasks.md b/openspec/changes/3495-service-counters-zero-series-repo-label/tasks.md index ff675424d..d2f90cfe1 100644 --- a/openspec/changes/3495-service-counters-zero-series-repo-label/tasks.md +++ b/openspec/changes/3495-service-counters-zero-series-repo-label/tasks.md @@ -24,11 +24,11 @@ Reads: `src/devops_cli/roadmap/run.py`, `src/devops_cli/server/service.py`, `tes ## 3. The webhook counter starts at zero -Reads: `src/devops_cli/server/routes/webhooks.py`, `src/devops_cli/server/counter_series.py`. Changes: `src/devops_cli/server/counter_series.py`, `tests/telemetry/test_service_counter_series.py`. +Reads: `src/devops_cli/server/routes/webhooks.py`, `src/devops_cli/server/counter_series.py`. Changes: `src/devops_cli/server/routes/webhooks.py`, `tests/telemetry/test_service_counter_series.py`, `tests/server/test_server_service.py`. -- [ ] 3.1 Add a test to `tests/telemetry/test_service_counter_series.py` that asserts that the webhook counter exists at 0 for each provider, each declared event with `other`, and each outcome +- [x] 3.1 Add a test to `tests/telemetry/test_service_counter_series.py` that asserts that the webhook counter exists at 0 for each provider, each declared event with `other`, and each outcome - Verification: `uv run devops ci test tests/telemetry/test_service_counter_series.py` fails on the base -- [ ] 3.2 Prime the webhook counter in `src/devops_cli/server/counter_series.py` from the declared providers and their events +- [x] 3.2 Prime the webhook counter in `src/devops_cli/server/counter_series.py` from the declared providers and their events - Verification: `uv run devops ci test tests/telemetry/test_service_counter_series.py` passes ## 4. The promtool cases hold the rules and the zero series diff --git a/src/devops_cli/server/routes/webhooks.py b/src/devops_cli/server/routes/webhooks.py index 366e3c5dc..c005d8d25 100644 --- a/src/devops_cli/server/routes/webhooks.py +++ b/src/devops_cli/server/routes/webhooks.py @@ -571,6 +571,39 @@ async def _read_and_route( return _route(delivery, provider, payload, service_manager, consumers) +def _declared_events( + provider: WebhookProvider, consumers: Sequence[WebhookConsumer] +) -> frozenset[str]: + """The events that `provider` declares: its ping event and the events of the consumers that + take its events.""" + taking = [consumer for consumer in consumers if consumer.provider is provider.name] + ping_events = {provider.ping_event} if provider.ping_event is not None else set() + return frozenset(ping_events).union(*(c.events for c in taking)) + + +_DELIVERY_OUTCOMES: Final[tuple[str, ...]] = ( + CONST_SERVICE_OUTCOME_ACCEPTED, + CONST_SERVICE_OUTCOME_IGNORED, + CONST_SERVICE_OUTCOME_REJECTED, + CONST_SERVICE_OUTCOME_NO_SUBSCRIBER, + CONST_SERVICE_OUTCOME_SELF, +) +"""The outcomes that a delivery of a provider counts, each under its declared event (#3495).""" + + +def _delivery_label_sets( + provider: WebhookProvider, consumers: Sequence[WebhookConsumer] +) -> list[dict[str, str]]: + """Each label set that a delivery of `provider` counts: each declared event and `other`, with + each outcome. The Service creates each of them at zero when its routes are built (#3495).""" + events = _declared_events(provider, consumers) | {CONST_SERVICE_EVENT_OTHER} + return [ + {"provider": str(provider.name), "event": event, "outcome": outcome} + for event in sorted(events) + for outcome in _DELIVERY_OUTCOMES + ] + + def _receiver( provider: WebhookProvider, service_manager: ServiceManager, @@ -578,8 +611,7 @@ def _receiver( ) -> Callable[[Request], Awaitable[Response]]: """The route handler of one provider, over the consumers that take its events.""" taking = [consumer for consumer in consumers if consumer.provider is provider.name] - ping_events = {provider.ping_event} if provider.ping_event is not None else set() - declared_events = frozenset(ping_events).union(*(c.events for c in taking)) + declared_events = _declared_events(provider, consumers) async def receive(request: Request) -> Response: event = request.headers.get(provider.event_header, "") if provider.event_header else "" @@ -622,6 +654,9 @@ def create_webhook_router( active = (roadmap_consumer(service_manager),) if consumers is None else tuple(consumers) router = APIRouter(tags=["webhooks"]) for provider in WEBHOOK_PROVIDERS.values(): + GLOBAL_METRICS.prime_counter( + CONST_SERVICE_METRIC_WEBHOOK_DELIVERIES, _delivery_label_sets(provider, active) + ) router.add_api_route( provider.path, _receiver(provider, service_manager, active), diff --git a/tests/server/test_server_service.py b/tests/server/test_server_service.py index c2959e84b..e56cff833 100644 --- a/tests/server/test_server_service.py +++ b/tests/server/test_server_service.py @@ -617,7 +617,8 @@ def flake_job(batch: TriggerBatch) -> None: CONST_SERVICE_METRIC_JOBS, labels={"repo": "example-org/repo1", "lane": LANE, "result": "success"}, ) - assert (err_count, succ_count >= 1.0) == (1.0, True) + # A lane job with no row counts no job: jobs_total counts the rows of a roadmap lane (#3495). + assert (err_count, succ_count) == (0.0, 0.0) def test_readyz_and_healthz_lifecycle() -> None: @@ -1144,8 +1145,9 @@ def job(batch: TriggerBatch) -> None: success_count_after = GLOBAL_METRICS.get_counter_value( CONST_SERVICE_METRIC_JOBS, labels={**counter_labels, "result": "success"} ) - assert err_count_after - err_count_before == 5.0 - assert success_count_after - success_count_before == 1.0 + # The worker counts no job: jobs_total counts the rows of a roadmap lane (#3495). + assert err_count_after - err_count_before == 0.0 + assert success_count_after - success_count_before == 0.0 # JsonLogFormatter formatting formatter = JsonLogFormatter() @@ -1196,8 +1198,9 @@ def counted(result: str) -> float: last = worker.last_round assert last is not None + # The worker counts no job of a round: jobs_total counts the rows of a roadmap lane (#3495). assert (counted("stopped") - before[0], counted("error") - before[1], last.result) == ( - 1.0, + 0.0, 0.0, "stopped", ) @@ -1655,7 +1658,7 @@ def exported_as(metric: str, labels: str) -> bool: exported_as(CONST_SERVICE_METRIC_JOB_START_TIMESTAMP, f'lane="model",{repo}'), exported_as(CONST_SERVICE_METRIC_JOBS, f'lane="model",{repo},result="success"'), lanes.model_released, - ) == (True, True, True, True, True, False, True) + ) == (True, True, False, True, True, False, True) def test_a_budget_refusal_in_the_model_lane_holds_the_status_lane_too() -> None: @@ -1789,7 +1792,7 @@ def _delivery_series(client: TestClient) -> dict[tuple[str, str, str], float]: (sample.labels["provider"], sample.labels["event"], sample.labels["outcome"]): sample.value for family in text_string_to_metric_families(client.get("/metrics").text) for sample in family.samples - if sample.name == CONST_SERVICE_METRIC_WEBHOOK_DELIVERIES + if sample.name == CONST_SERVICE_METRIC_WEBHOOK_DELIVERIES and sample.value > 0 } diff --git a/tests/telemetry/test_service_counter_series.py b/tests/telemetry/test_service_counter_series.py index 8bc71909f..f6137de25 100644 --- a/tests/telemetry/test_service_counter_series.py +++ b/tests/telemetry/test_service_counter_series.py @@ -10,11 +10,20 @@ from datetime import UTC, datetime from devops_cli.config.constants import ( + CONST_ROADMAP_WEBHOOK_EVENTS, + CONST_SERVICE_EVENT_OTHER, CONST_SERVICE_METRIC_IMPLEMENT_FAILURES, CONST_SERVICE_METRIC_JOBS, + CONST_SERVICE_METRIC_WEBHOOK_DELIVERIES, + CONST_SERVICE_OUTCOME_ACCEPTED, + CONST_SERVICE_OUTCOME_IGNORED, + CONST_SERVICE_OUTCOME_NO_SUBSCRIBER, + CONST_SERVICE_OUTCOME_REJECTED, + CONST_SERVICE_OUTCOME_SELF, ) from devops_cli.config.settings import GitHubAppConfig, ServiceConfig from devops_cli.roadmap.failure import FailureClass +from devops_cli.server.routes.webhooks import create_webhook_router from devops_cli.server.service import ServiceManager from devops_cli.telemetry.metrics import GLOBAL_METRICS @@ -83,3 +92,23 @@ def test_jobs_total_carries_row_label() -> None: assert {"repo": "example/app", "lane": "status", "row": "metrics", "result": "success"} in ( label_sets ) + + +def test_webhook_counter_exists_at_zero_for_each_declared_event_and_outcome() -> None: + create_webhook_router(_manager(REPOS)) + + series = _series(CONST_SERVICE_METRIC_WEBHOOK_DELIVERIES) + outcomes = ( + CONST_SERVICE_OUTCOME_ACCEPTED, + CONST_SERVICE_OUTCOME_IGNORED, + CONST_SERVICE_OUTCOME_REJECTED, + CONST_SERVICE_OUTCOME_NO_SUBSCRIBER, + CONST_SERVICE_OUTCOME_SELF, + ) + expected = { + frozenset({("provider", "github"), ("event", event), ("outcome", outcome)}) + for event in (*CONST_ROADMAP_WEBHOOK_EVENTS, CONST_SERVICE_EVENT_OTHER) + for outcome in outcomes + } + + assert {key: series.get(key) for key in expected} == dict.fromkeys(expected, 0.0) From 03e743137670a85597db87d2b8349340475112c4 Mon Sep 17 00:00:00 2001 From: Daniel Petty Date: Thu, 5 Nov 2026 03:59:11 +0000 Subject: [PATCH 05/17] fix(k8s): webhook alerts read the zero series without the workaround The Service starts the webhook counter at zero, so the rejected and self delivery rules sum increase() without the #2505 offset terms. The promtool cases start the rejected series at zero and prove the true total of the failure counter over a pod change. The OTLP terms stay. Change: openspec/changes/3495-service-counters-zero-series-repo-label --- k8s/monitoring/prometheus-values.yaml | 39 +++++++------------ .../tasks.md | 6 +-- .../roadmap_service_tests.yaml | 16 ++++++++ .../roadmap_service_webhooks_tests.yaml | 8 ++-- .../test_k8s_monitoring_alerting.py | 15 ++++--- 5 files changed, 43 insertions(+), 41 deletions(-) diff --git a/k8s/monitoring/prometheus-values.yaml b/k8s/monitoring/prometheus-values.yaml index b8a6a9e57..72a7a474e 100644 --- a/k8s/monitoring/prometheus-values.yaml +++ b/k8s/monitoring/prometheus-values.yaml @@ -667,14 +667,14 @@ serverFiles: # names the job after OTEL_SERVICE_NAME. Each selector names the job, so the same series of # a workstation (job="devops-cli"), or of the server's kubernetes-pods job, which scraped the # pod before #1130, never counts. - # kube-state-metrics gives the restarts and the replicas. On both paths a series starts at - # its first count, with no zero sample before it, and increase() does not count that first - # value. The OTLP series also get a sample only when the Service sends a new observation, and - # the collector starts a stream again from zero after an hour with none (max_stale). - # So each count of the last hour reads a series with no sample in the hour before as new, - # and gives its last value, which is all of its count. Every other series gives its - # increase(). Neither part reads one sample near the start of the hour, so a sparse series - # from before the hour adds none of its earlier count (#2505). + # kube-state-metrics gives the restarts and the replicas. The Service starts its /metrics + # counters at zero (#3495), so their series need no zero sample here. The OTLP series get a + # sample only when the Service sends a new observation, and the collector starts a stream + # again from zero after an hour with none (max_stale). So each count of the last hour reads + # a series with no sample in the hour before as new, and gives its last value, which is all + # of its count. Each other series gives its increase(). Neither part reads one sample near + # the start of the hour, so a sparse series from before the hour adds none of its earlier + # count (#2505). - name: roadmap-service rules: # A refusal stops every round, also after a restart, until a person removes the halt @@ -778,21 +778,13 @@ serverFiles: # The route counts each delivery that it refuses under outcome="rejected": unsigned, signed # with no secret of a managed repository, too large, not JSON, or without its event name or # delivery ID. A forged delivery counts under the event that it names, or under "other", so a - # flood can start a new label set. A new label set has no zero sample, so the count of the 15 - # minutes reads a series with no sample in the 15 minutes before as new, and gives its last - # value (#2505). Each other series gives its increase(). The Service serves this counter only - # on its /metrics, not through OTLP, and the scrape reads it each minute. So a label set that - # counted before has a sample in each minute while its pod runs, and only a new label set, or - # the label set of a new pod, has no sample in the 15 minutes before. + # flood can start a new label set. The Service starts each label set at zero (#3495), so + # increase() counts each delivery of a label set from its start, and the rule needs no + # workaround term. The Service serves this counter only on its /metrics, not through OTLP. - name: roadmap-service-webhooks rules: - alert: RoadmapServiceWebhooksRejected - expr: >- - sum( - (last_over_time(devops_cli_service_webhook_deliveries_total{job="roadmap-service", outcome="rejected"}[15m]) - unless last_over_time(devops_cli_service_webhook_deliveries_total{job="roadmap-service", outcome="rejected"}[15m] offset 15m)) - or increase(devops_cli_service_webhook_deliveries_total{job="roadmap-service", outcome="rejected"}[15m]) - ) > 20 + expr: sum(increase(devops_cli_service_webhook_deliveries_total{job="roadmap-service", outcome="rejected"}[15m])) > 20 for: 0m labels: severity: warning @@ -827,12 +819,7 @@ serverFiles: # can come back as one delivery, so many of them can be another bot or workflow that # answers each write of the Service. - alert: RoadmapServiceSelfDeliveriesHigh - expr: >- - sum( - (last_over_time(devops_cli_service_webhook_deliveries_total{job="roadmap-service", outcome="self"}[1h]) - unless last_over_time(devops_cli_service_webhook_deliveries_total{job="roadmap-service", outcome="self"}[1h] offset 1h)) - or increase(devops_cli_service_webhook_deliveries_total{job="roadmap-service", outcome="self"}[1h]) - ) > 150 + expr: sum(increase(devops_cli_service_webhook_deliveries_total{job="roadmap-service", outcome="self"}[1h])) > 150 for: 0m labels: severity: warning diff --git a/openspec/changes/3495-service-counters-zero-series-repo-label/tasks.md b/openspec/changes/3495-service-counters-zero-series-repo-label/tasks.md index d2f90cfe1..45b693d37 100644 --- a/openspec/changes/3495-service-counters-zero-series-repo-label/tasks.md +++ b/openspec/changes/3495-service-counters-zero-series-repo-label/tasks.md @@ -35,11 +35,11 @@ Reads: `src/devops_cli/server/routes/webhooks.py`, `src/devops_cli/server/counte Reads: `tests/fixtures/prometheus_rules/roadmap_service_tests.yaml`, `tests/fixtures/prometheus_rules/roadmap_service_webhooks_tests.yaml`, `tests/prometheus/test_k8s_monitoring_alerting.py`, `k8s/monitoring/prometheus-values.yaml`. Changes: `tests/fixtures/prometheus_rules/roadmap_service_tests.yaml`, `tests/fixtures/prometheus_rules/roadmap_service_webhooks_tests.yaml`, `k8s/monitoring/prometheus-values.yaml`. -- [ ] 4.1 Add a promtool case `pod_change_gives_true_total` to `tests/fixtures/prometheus_rules/roadmap_service_tests.yaml`. Its series of the failure counter start at 0 in each pod, and the case asserts the sum of `increase()` over a pod change +- [x] 4.1 Add a promtool case `pod_change_gives_true_total` to `tests/fixtures/prometheus_rules/roadmap_service_tests.yaml`. Its series of the failure counter start at 0 in each pod, and the case asserts the sum of `increase()` over a pod change - Verification: `promtool test rules tests/fixtures/prometheus_rules/roadmap_service_tests.yaml` passes -- [ ] 4.2 Add a promtool case to `tests/fixtures/prometheus_rules/roadmap_service_webhooks_tests.yaml` that fires `RoadmapServiceWebhooksRejected` on a rejected series that starts at 0 +- [x] 4.2 Add a promtool case to `tests/fixtures/prometheus_rules/roadmap_service_webhooks_tests.yaml` that fires `RoadmapServiceWebhooksRejected` on a rejected series that starts at 0 - Verification: `promtool test rules tests/fixtures/prometheus_rules/roadmap_service_webhooks_tests.yaml` passes -- [ ] 4.3 Remove the #2505 workaround terms of `RoadmapServiceWebhooksRejected` and `RoadmapServiceSelfDeliveriesHigh` from `k8s/monitoring/prometheus-values.yaml`, and replace the stale comment at lines 670-673 with the scrape behavior. Keep the OTLP terms +- [x] 4.3 Remove the #2505 workaround terms of `RoadmapServiceWebhooksRejected` and `RoadmapServiceSelfDeliveriesHigh` from `k8s/monitoring/prometheus-values.yaml`, and replace the stale comment at lines 670-673 with the scrape behavior. Keep the OTLP terms - Verification: `uv run devops ci test tests/prometheus/test_k8s_monitoring_alerting.py` passes and `promtool test rules tests/fixtures/prometheus_rules/roadmap_service_webhooks_tests.yaml` passes ## 5. The documents state the scrape behavior and the labels diff --git a/tests/fixtures/prometheus_rules/roadmap_service_tests.yaml b/tests/fixtures/prometheus_rules/roadmap_service_tests.yaml index ed7f47167..10d1f4c35 100644 --- a/tests/fixtures/prometheus_rules/roadmap_service_tests.yaml +++ b/tests/fixtures/prometheus_rules/roadmap_service_tests.yaml @@ -444,3 +444,19 @@ tests: - eval_time: 90m alertname: RoadmapServiceModelRequestsHigh exp_alerts: [] + - name: A pod change keeps the true total of the failure counter, because each pod starts at zero + interval: 1m + input_series: + # Pod a counts 3 failures from minute 0 to minute 3, then stays at 3. Pod b starts at zero at + # minute 10 (#3495) and counts 2 failures by minute 12. + - series: 'devops_cli_service_implement_failures_total{job="roadmap-service", pod="a", repo="example/app", class="ci_infra"}' + values: '0+1x3 3x40' + - series: 'devops_cli_service_implement_failures_total{job="roadmap-service", pod="b", repo="example/app", class="ci_infra"}' + values: '_x10 0+1x2 2x40' + promql_expr_test: + # increase() extrapolates to the window edges, so the sum reads 5.03 where the counts are 5. + - expr: sum(increase(devops_cli_service_implement_failures_total{job="roadmap-service", class="ci_infra"}[1h])) + eval_time: 50m + exp_samples: + - labels: '{}' + value: 5.034090909090909 diff --git a/tests/fixtures/prometheus_rules/roadmap_service_webhooks_tests.yaml b/tests/fixtures/prometheus_rules/roadmap_service_webhooks_tests.yaml index dc59c1c09..04c9d7d2b 100644 --- a/tests/fixtures/prometheus_rules/roadmap_service_webhooks_tests.yaml +++ b/tests/fixtures/prometheus_rules/roadmap_service_webhooks_tests.yaml @@ -5,9 +5,9 @@ # # tests/test_k8s_monitoring_alerting.py writes the group to the file that `rule_files` names, # beside a copy of this file, and runs `promtool test rules` on the copy. promtool starts each -# series at time 0, with one sample each `interval`. A label set of rejected deliveries with no -# sample in the 15 minutes before the range started in the range, and the rule gives its last -# value (#2505). Each other series gives its increase(), which promtool extrapolates to the range. +# series at time 0, with one sample each `interval`. The Service starts each label set of rejected +# deliveries at zero (#3495), so the rule reads each series with its increase(), which promtool +# extrapolates to the range, and needs no workaround term (#2505 before #3495). rule_files: - roadmap-service-webhooks.rules.yaml @@ -19,7 +19,7 @@ tests: input_series: # Forged deliveries from minute 30, 5 each minute, under an event that no consumer takes. - series: 'devops_cli_service_webhook_deliveries_total{job="roadmap-service", provider="github", event="other", outcome="rejected"}' - values: '_x30 5+5x10' + values: '0x29 5+5x10' - series: 'devops_cli_service_webhook_deliveries_total{job="roadmap-service", provider="github", event="issues", outcome="accepted"}' values: '0+10x45' - series: 'devops_cli_service_webhook_deliveries_total{job="devops-cli", provider="github", event="issues", outcome="rejected"}' diff --git a/tests/prometheus/test_k8s_monitoring_alerting.py b/tests/prometheus/test_k8s_monitoring_alerting.py index a3a368a97..48c6b5cac 100644 --- a/tests/prometheus/test_k8s_monitoring_alerting.py +++ b/tests/prometheus/test_k8s_monitoring_alerting.py @@ -927,12 +927,7 @@ def _window_count(series: str, window: str) -> str: ), ( "RoadmapServiceSelfDeliveriesHigh", - "sum( " - + _window_count( - 'devops_cli_service_webhook_deliveries_total{job="roadmap-service", outcome="self"}', - "1h", - ) - + " ) > 150", + 'sum(increase(devops_cli_service_webhook_deliveries_total{job="roadmap-service", outcome="self"}[1h])) > 150', "0m", "warning", ), @@ -1007,7 +1002,7 @@ def test_the_webhook_rule_warns_on_more_than_20_rejected_deliveries_in_15_minute ] == [ ( "RoadmapServiceWebhooksRejected", - "sum( " + _window_count(rejected, "15m") + " ) > 20", + f"sum(increase({rejected}[15m])) > 20", "0m", "warning", ) @@ -1261,7 +1256,11 @@ def test_each_unit_tested_rule_fires_on_a_sample_and_stays_quiet_on_a_normal_ser yaml.safe_dump({"groups": [g for g in groups if g["name"] == group]}), encoding="utf-8", ) - tested = {case["alertname"] for test in unit_tests["tests"] for case in test["alert_rule_test"]} + tested = { + case["alertname"] + for test in unit_tests["tests"] + for case in test.get("alert_rule_test", []) + } copy = tmp_path / rule_tests.name copy.write_text(rule_tests.read_text(encoding="utf-8"), encoding="utf-8") result = _promtool("test", "rules", copy) From fcd338bc15a7674f82a803b9ccc445c5490c0dd8 Mon Sep 17 00:00:00 2001 From: Daniel Petty Date: Thu, 5 Nov 2026 04:04:20 +0000 Subject: [PATCH 06/17] docs(telemetry): the scrape starts at zero and the job counter has a row The telemetry document and the Service document state the zero series, the labels of the failure counter and the job counter, and the gap of a count before the first scrape. The action of the GitHub requests alert no longer says to lower the budget. Change: openspec/changes/3495-service-counters-zero-series-repo-label --- docs/TELEMETRY.md | 16 +++++++++++----- k8s/README.md | 6 +++--- .../tasks.md | 4 ++-- src/devops_cli/docs/generator.py | 16 +++++++++++----- tests/prometheus/test_k8s_monitoring_alerting.py | 13 +++++++++++++ 5 files changed, 40 insertions(+), 15 deletions(-) diff --git a/docs/TELEMETRY.md b/docs/TELEMETRY.md index ceae921fe..504a43500 100644 --- a/docs/TELEMETRY.md +++ b/docs/TELEMETRY.md @@ -19,17 +19,23 @@ resource, scope, metric and attributes) into a running total before Prometheus s it. Prometheus names a series by `service.name` (`job`), `service.instance.id` (`instance`) and the metric's attributes. The `service.instance.id` is the host name, which every command on the host shares. The standard `OTEL_RESOURCE_ATTRIBUTES` sets -another one: roadmap-service sets `service.instance.id=roadmap-service`, so a new pod -writes the series of the pod before it. A running total that starts again from zero, -for a new pod or for a stream that gets no delta for `max_stale` (`1h` in +another one: roadmap-service sets `service.instance.id=roadmap-service`, so on the OTLP +path a new pod continues the series of the pod before it. The `/metrics` scrape adds the +label `pod`, so each pod has its own series there. A running total that starts again from +zero, for a new pod or for a stream that gets no delta for `max_stale` (`1h` in `k8s/otel/values.yaml`), is a counter reset to `rate()` and `increase()`. Cluster job pods have unique instance names and export metrics only once when they exit. PromQL functions like `increase()` and `rate()` drop series that contain only one sample. To calculate these counters, the collector transmits created timestamps via remote write 2.0 (`remote_write_version: 2` in `k8s/otel/values.yaml`). Prometheus sets `enable-feature: created-timestamp-zero-ingestion` in `k8s/monitoring/prometheus-values.yaml`. -Prometheus then writes a zero-value sample at the start timestamp of each series, so -`increase()` calculates the full counter value. +On this path, Prometheus writes a zero-value sample at the start timestamp of each series, +so `increase()` calculates the full counter value. The `/metrics` scrape gets no such sample. +The Service starts each label set of its counters at zero (#3495), so the first count of a +label set is in its series. A count that a pod makes before its first scrape is still lost, +because `increase()` does not count the first sample of a series. The labels of the Service +counters are: `devops_cli_service_implement_failures_total` with `class` and `repo`, and +`devops_cli_service_jobs_total` with `repo`, `lane`, `row` and `result`. A gauge holds a snapshot, such as a project metric that the roadmap Service sends every 15 minutes. Prometheus finds a plain selector only within five minutes of a sample, so a dashboard reads a gauge with `last_over_time([$__range])`, and diff --git a/k8s/README.md b/k8s/README.md index bd28ad73e..cebc4924f 100644 --- a/k8s/README.md +++ b/k8s/README.md @@ -627,12 +627,12 @@ kubectl -n devops logs -f job/ 9. To rotate a credential, update it in the keyring: `uv run devops config set service.webhook_secrets`, or, for the App, generate a new private key and run `uv run devops config set service.github_app.private_key --from-file `. Then run `uv run devops k8s push-secrets --only devops/devops-cli --rotate`. It restarts `roadmap-service`. Delete the old private key on the App's page after the restart. 10. Run one service per set of repos. While it runs, use `devops roadmap run --dry-run` (#981). 11. The Service keeps three files in `/home/devops/.data/roadmap/` on volume `roadmap-service-home`, beside the `schedule.json` of each repository: `service-halt.json` (GitHub refused the identity), `service-budget.json` (the GitHub requests of the last hour) and `service-failures.json` (the jobs that failed rounds in a row). A restart reads them back, so a restart does not clear a halt, the budget or a suspended job. A round that ends because the Service is stopping, for example in a rollout, does not count as a failed round: the count of its job stays as it was. The log line that suspends a job names the class of the error of each of its failed rounds in a row; the file keeps only the count, so a round that an earlier process counted reads as `unknown`. A rate limit that one request gets stops the requests of all workers until its wait ends. When GitHub refuses the identity, for example because the App's installation is suspended, the Service halts and logs `GitHub refused the identity`. To clear the halt after you fix the cause, remove the file and restart the Service: `kubectl -n devops exec deploy/roadmap-service -- rm /home/devops/.data/roadmap/service-halt.json`, then `kubectl -n devops delete pod -l app.kubernetes.io/name=roadmap-service`. -12. The Service's `/metrics` shows the budget and the suspended jobs (#2475): `devops_cli_service_budget_limit_requests` (the hourly budget), `devops_cli_service_budget_spent_requests` (the GitHub requests of the last hour), `devops_cli_service_budget_holds_total` (one count each time a hold begins), `devops_cli_service_job_suspended` (1 while a job of a repository, by `repo` and `row`, waits for new input after it failed 3 rounds in a row) and `devops_cli_service_implement_failures_total` (one count for each runner Job that left a stop, by the failure `class`, when a round reports the Job; #3065; the 17 values of `class` are the classes in [docs/ROADMAP_WORKFLOW.md](../docs/ROADMAP_WORKFLOW.md), The failure classes, and a panel or rule matches no other value, #3476). Each GitHub request counts in `devops_cli_github_requests_total` under its `method`, the `repo` of the worker that sent it and the job `row` (`metrics`, `intake`, `refine`, `stage`, `implement`, `stage_ready` or `alternatives`); a request outside a worker or a row counts as `other`. To find what spent the budget, run `sum by (repo, row, method) (increase(devops_cli_github_requests_total{job="roadmap-service"}[1h]))` in Grafana Explore. A delivery of the App bot's own write counts as outcome `self` in `devops_cli_service_webhook_deliveries_total`. +12. The Service's `/metrics` shows the budget and the suspended jobs (#2475): `devops_cli_service_budget_limit_requests` (the hourly budget), `devops_cli_service_budget_spent_requests` (the GitHub requests of the last hour), `devops_cli_service_budget_holds_total` (one count each time a hold begins), `devops_cli_service_job_suspended` (1 while a job of a repository, by `repo` and `row`, waits for new input after it failed 3 rounds in a row) and `devops_cli_service_implement_failures_total` (one count for each runner Job that left a stop, by the failure `class` and `repo`, when a round reports the Job; #3065; the 19 values of `class` are the classes in [docs/ROADMAP_WORKFLOW.md](../docs/ROADMAP_WORKFLOW.md), The failure classes, and a panel or rule matches no other value, #3476). Each counter of the Service starts its label sets at zero (#3495, [docs/TELEMETRY.md](../docs/TELEMETRY.md)). Each GitHub request counts in `devops_cli_github_requests_total` under its `method`, the `repo` of the worker that sent it and the job `row` (`metrics`, `intake`, `refine`, `stage`, `implement`, `stage_ready` or `alternatives`); a request outside a worker or a row counts as `other`. To find what spent the budget, run `sum by (repo, row, method) (increase(devops_cli_github_requests_total{job="roadmap-service"}[1h]))` in Grafana Explore. A delivery of the App bot's own write counts as outcome `self` in `devops_cli_service_webhook_deliveries_total`. 13. A refine round writes at most 2 comments on one item in 24 hours. A round that finds 2 refine comments of the last 24 hours on an item leaves the item alone, with no model call and no comment. The item waits on a person and takes no place under the round's limit, so the round refines the next item. For each newest comment on the item that the cap leaves with no answer, the round records the time of that comment in the item's job record (`refine.capped_after`), logs `Refine left # alone and wrote no comment`, and adds 1 to `devops_cli_roadmap_refine_comments_capped_total`. A later round that finds no newer comment does not log or count the item again. A person refines the item at the cap with `devops roadmap refine --item --confirm`. Each Firecrawl search counts in `devops_cli_firecrawl_searches_total` by `result`: `ok`, the HTTP status of a refused search, or `error`. A failed search logs `A Firecrawl web search failed` with the status or the error class, and never the key or the query. 14. The row `alternatives` of the model lane runs after `refine`: on a first run, and 719 minutes after its last success. It runs `devops roadmap alternatives --confirm` for the repository. A round checks at most `roadmap.alternatives.first_pass_limit` items (20 by default) while an item has no check, and at most `roadmap.alternatives.limit` items (5 by default) after. The row runs about twice a day, so a first pass of 141 items takes about 4 days. A round takes first the items with no check, then the items whose check is older than `roadmap.alternatives.recheck_days` (30 by default) or whose text changed. A webhook makes the row due at no other time. The row changes no label and closes no issue. It does nothing, and logs `The alternatives check of is off`, for a private repository and when no Firecrawl key is set. If the check of an item fails, the row records the failure in the item's job record (`alternatives.failed_at` and `alternatives.failed_error`, the class of the error), writes the other checks and succeeds. The row fails, and counts toward the suspension after 3 failed rounds, only when it tried items and checked none. A round takes a failed item again after `roadmap.alternatives.retry_days` (3 by default). Each written check adds 1 to `devops_cli_roadmap_alternatives_total` by `verdict`: `covers`, `partly`, `none` or `not-applicable`. An item of the type `test`, `docs`, `refactor`, `chore`, `ci` or `perf`, and a `fix` with a Python traceback in its body, gets `not-applicable` with no model call; every other item must run a search before the check accepts a verdict, and gets one more turn to search. A `covers` verdict makes refine ask a person to adopt the tool or to keep the item with a reason. The item waits on a person until a reply answers the question, and Ready items with a new covering tool go back to New. Refine records the tool it asked about in the job record (`refine.alternatives_asked`). `devops roadmap run --dry-run` lists the row when it is due. 15. The row `stage` follows `refine` on the model lane (#2659). It runs `devops roadmap stage --confirm` in the model clone: it is due every 37 minutes, on a first run, and in a round where `refine` set an item Ready. It stages the Ready items that `roadmap.stage.filter` names, at most `roadmap.stage.limit` in a round, plans no new item while the count of open Planned items is at `roadmap.stage.planned_limit` (an item with the priority `P0-Critical` passes it, #3487), and splits a task group over `roadmap.stage.step_tokens`. Each task group has a kind (#3005): a `behavior` group has a failing test, and an `other` group has no test but a command or the steps of a manual check that prove it. The job sends a reply that breaks this rule back to the model once. It leaves out an item whose change folder is on the default branch or in its archive (#2714): a round reads that tree once with `git ls-tree`, so a merged staging pull request never makes a second one. The pull request it opens is a draft with the title `docs(openspec): plan # `. The job does not mark it ready: the row `stage_ready` does (step 18). Before it opens one, it checks each path of the plan (`reads`, `changes`, the code spans of the tasks and `verify`) against the tracked files of the commit it plans, read with one `git ls-files` for the round (#3010). A path that is no file there, and that no group up to its own creates, goes back to the model once with the files of the same name. When a later group creates the path, the problem names that group and tells the model to move it before the group that names the path (#3017). A second reply with a missing path stops the item, so no draft opens and the implement Job never meets a wrong path. `k8s/devops/configmap.example.yaml` holds the five settings, and `devops k8s deploy-stack --stack devops` applies them; each also has a variable, `DEVOPS_CLI_ROADMAP_STAGE_FILTER`, `DEVOPS_CLI_ROADMAP_STAGE_LIMIT`, `DEVOPS_CLI_ROADMAP_STAGE_PLANNED_LIMIT`, `DEVOPS_CLI_ROADMAP_STAGE_STEP_TOKENS` and `DEVOPS_CLI_ROADMAP_STAGE_TOOLSET`. The setting `roadmap.stage.toolset` is `extended` by default (#3238): a plan run then has three tools more than the nine that read the staged commit. `references` counts a text in every file of the commit. `read_issue` and `find_issues` read issues and pull requests of the repository on GitHub, as GET requests with the token of the Service and no other permission. `base` leaves them out. Each item of a round counts once in `devops_cli_roadmap_stage_total` by its `result`: `staged`, `skipped`, `needs_split` or `failed`. A failed item is recorded in its job record with the class of its error and the time (#3493), and the job leaves it out for `roadmap.stage.retry_hours` hours (6 by default) while it takes the other items. The row fails a round only when it tried items and staged, stopped or returned none, or when a failure belongs to no item; three such rounds on the same input suspend the row. The ledger file keeps the class of each error, and a round that the stop of the Service ends counts as `stopped`, not as an error. The image carries the pinned OpenSpec and a Node runtime, so the job validates each change before it pushes: `kubectl -n devops exec deploy/roadmap-service -- openspec --version` prints the locked version. 16. The row `implement` follows `metrics` on the status lane (#2667, #2757). It calls no model, so a model-lane round that has not ended never delays it. It is due every 7 minutes and on a first run, in the clone of the status lane. The Service runs no step: a round only talks to the Kubernetes API, in three moves. It reports each Job that ended, with a comment on the item that holds the state and the last 40 lines of the log of the Job's pod, masked of secrets, and records `implement.job_state` as `succeeded`, `failed` or `gone`. It moves the deadline of each Job that has not ended (#3063, below). It waits while a Job of the repository has not ended. Else it makes one Job from CronJob `devops-cli-runner` for the first Staged or Planned item that the default filter names (a Planned item has its change folder on the default branch and no branch: a person merged the staging pull request, and the job makes the branch and the draft itself), with the args `roadmap implement --repo --item --confirm` and the item's number as the only item text, and records the name of the Job in `implement.job`, and the digest of the runner image that the CronJob template names in `implement.job_digest`. The same write sets the label `owner/service` on the item (#2910), and the report of the Job removes it. The comment that reports the Job names the digest. The Service makes no Job for an item that an agent or a person claimed, and a claim of an agent or a person ends a running Job at its next group boundary, with no push and no pull request. A round also sets the Status label of the items from their record (`status_sync.py`): it reads the plans on the default branch once for the sweep and for the plan, writes `status/planned` once a plan is merged, and writes `status/in-progress` as soon as the record names a Job, and again after it makes one. It writes only an open item that holds Ready or Planned with the label that a job last recorded, so a label that a person set stays. The rows `stage` and `stage_ready` use the same function. The Jobs of a repository carry the label `implement-repo`, whose value is a digest of the repository name, so the API is the lock and a restart of the Service loses nothing. A Job that leaves its item In Progress, or Ready or Planned with no stop, is a stop that the runner could not write, such as a pod that the kernel killed for its memory, unless an agent or a person claimed the item. The round then sets the item Blocked (`status/blocked`) and records `implement.stopped` as `job`, so that no Job is made for it again. A stop that the runner writes sets Blocked in the same way, with the step in `implement.stopped`. After the report, the round clears the stop of each item that it set Blocked, by one of two rules, and a comment names the rule that ran. Rule `retry`: a comment `implement: retry`, made after the report by the owner, a member or a collaborator, clears the stop at once, whatever the image. Rule `digest`: the digest in the template of CronJob `devops-cli-runner` is not `implement.job_digest`, so a new runner image has rolled out since the stop, and the class in `implement.failure` has the action `retry` in `ACTION_BY_CLASS` (#3481). A new image cannot fix a plan, a premise, a refused tool or a failed test, so a stop of another class, and a stop with no class on record, wait for a retry. Each clear by digest adds the class, the new digest and the time to `implement.digest_clears`, and the list stays after the clear. The settings `roadmap.implement.digest_clears_per_class` (default 1) and `roadmap.implement.digest_clears_per_item` (default 3) bound the entries; a stop at a limit stays Blocked until a retry, also for `ci_infra`. The comment of a clear by digest shows at most one time in 24 hours for an item (`implement.digest_clear_commented_at`), and the clear is recorded either way. The rule reads no image content: the Service holds digests and no map from a digest to commits. A clear sets Ready or Planned and removes `implement.stopped`, and the same round can make a Job for the item. The new Job overwrites `implement.job_digest`, so a second stop on the same image waits for a retry, and the round makes no Job in a loop. A stop with no digest on record is cleared by a retry only. Inside one priority, a round takes an item with no digest clear before an item with one, and a retried item of a higher priority still goes before a new item of a lower priority. A Blocked stop that no rule clears costs one request in each round, the listing of its comments (6 an hour of `service.github_requests_per_hour`); the round reads the issue again only when a rule holds, and reads the template at most one time. The round clears no item that a Job holds, that an agent or a person claimed, or that a person labeled Blocked by hand. The failures ledger is unchanged: three failed rounds on the same input still suspend the row. If CronJob `devops-cli-runner` does not exist, the round fails with a message that names it: run `devops k8s deploy-stack --stack devops`, or wait for Argo CD. A Job that cannot pull its image (the package `runner` is not public, or no release published it) has an empty log, so it ends by its stall after 40 minutes, and the next round records the stop. Read the Jobs with `kubectl -n devops get jobs -l implement-repo`. The Job runs the checks that a person runs at a commit and at a push (#3043). Before the first step it runs `pre-commit install-hooks`, with `PRE_COMMIT_HOME` in the cache directory of the data directory, on the home volume. The two remote hook repositories come over public HTTPS (9 s and 51 MB, measured). A failed install stops the item at the step `hooks`, before any model request. Each step then runs the `pre-commit` stage on its staged changes as its last exit check. The child is `uv`, which holds no token. The Job installs no git hook, because git runs a hook with the token of the session, and the agent can edit the hook configuration. For the same reason every hook run gets `--config` with a copy of the configuration of the default branch, so an edit of the agent skips no hook. A hook run that runs out of time stops the step. A failed hook goes to the agent as a repair turn with its output. The setting `roadmap.implement.repair_rounds` (variable `DEVOPS_CLI_ROADMAP_IMPLEMENT_REPAIR_ROUNDS`, default 3) bounds the extra turns of a step for `ruff`, `mypy` and the hooks. Only a used budget stops the step. After the last group, the docs, the archive and the final rebase, and before the push, the Job runs the scoped checks: the `pre-push` stage of the hooks, `devops ci --check --only constructs,docs,openspec`, and `devops ci test` with the test modules that cover the changed Python files of `src/` and `tests/`. Above 30 test modules, which a change to a shared module gives, the tests check runs only the changed test modules, and it does not run when no test covers the change. A failed check starts a fix step, up to `repair_rounds` times for all scoped checks. The Job generates the docs again after each fix. It pushes only a head that passed all of them. A used budget stops the item at the step `scoped checks` and pushes nothing. An error or a command that runs out of time in this phase stops the item and pushes nothing too. The comment on the pull request shows the repair rounds of each step, and each run of a scoped check with its seconds. The extra run time of an item is about 10 s of hooks for each step, 9 s for the install, 35 s for the first two scoped checks, and at most about 3.5 minutes for the tests (12 test modules of the roadmap ran in 88 s on 2 CPUs). So it is 2 to 5 minutes in a Job that runs about an hour. The workflow checks `actionlint` and `zizmor` run in the `pre-push` stage only when the push changes a workflow, because the runner image holds no `actionlint`. A failed check of the pull request after the push gets a fix step, a push and a new watch, up to `repair_rounds` times, with a comment on the pull request for each round. -17. Each repository has three lanes, and each lane runs one round at a time in its own clone (#1532, #1534). The status lane runs the rows `metrics`, `implement`, `stage_ready` and `release_comments` in the clone `clone`. The triage lane runs the row `intake` on the `devops-triage` model in the clone `triage-clone`. The model lane runs the rows `refine`, `stage` and `alternatives` on `devops-background` in the clone `model-clone`. A poll and a webhook go to all three lanes. The lanes share the file `schedule.json` of the repository, the ledger `service-failures.json` and the lock that each write of an item's job record takes (#2757). When intake places a critical fix or P0 item, it writes a `refine_due` timestamp to `schedule.json` so refinement becomes due in the next model-lane round. The label `lane` of `devops_cli_service_jobs_total`, `devops_cli_service_job_seconds_total` and `devops_cli_service_queue_depth` is `status`, `triage` or `model`. The label `row` of `devops_cli_github_requests_total` and `devops_cli_service_job_suspended` is the name of the row. A model-lane round that has not ended does not delay a status-lane or triage-lane round. +17. Each repository has three lanes, and each lane runs one round at a time in its own clone (#1532, #1534). The status lane runs the rows `metrics`, `implement`, `stage_ready` and `release_comments` in the clone `clone`. The triage lane runs the row `intake` on the `devops-triage` model in the clone `triage-clone`. The model lane runs the rows `refine`, `stage` and `alternatives` on `devops-background` in the clone `model-clone`. A poll and a webhook go to all three lanes. The lanes share the file `schedule.json` of the repository, the ledger `service-failures.json` and the lock that each write of an item's job record takes (#2757). When intake places a critical fix or P0 item, it writes a `refine_due` timestamp to `schedule.json` so refinement becomes due in the next model-lane round. The label `lane` of `devops_cli_service_jobs_total`, `devops_cli_service_job_seconds_total` and `devops_cli_service_queue_depth` is `status`, `triage` or `model`. The label `row` of `devops_cli_service_jobs_total` is the name of the row that ended, and the label `result` is `success` or `error`. The label `row` of `devops_cli_github_requests_total` and `devops_cli_service_job_suspended` is the name of the row. A model-lane round that has not ended does not delay a status-lane or triage-lane round. 18. The row `stage_ready` follows `implement` on the status lane (#2865). It calls no model and needs no clone. It is due every 3 minutes and on a first run. A round reads the open items whose record holds `stage.pr` and no `stage.ready` of the same number and no `implement.job`; with no such item it sends no request of its own, since the listing of the issues is the one that `metrics` and `implement` read in the round. For each such item it reads the pull request (`GET pulls/`). A pull request that is not an open draft is only recorded in `stage.ready`. An open draft goes to the function `draft_readiness` with its check runs (`GET commits//check-runs`): when every check ended and passed, the row marks the pull request ready with `gh pr ready` (the same function as the implement job) and records `stage.ready`. A draft with a failed, cancelled or running check, or with no check, stays a draft, and the row does not retry a check. The row reads only the pull request that the stage job recorded, and never the draft of a person. It reads the check runs and not the commit statuses, because the App token holds `checks: read` and no `statuses` permission. A draft that waits for its checks costs 2 requests a round (12 an hour) of `service.github_requests_per_hour`; a pull request that the row marks costs 2 more for `gh pr ready` and 2 to record it, once. A refusal of `gh pr ready` fails the round, names the item, and the ledger of failed rounds suspends the row after three rounds in a row: check that the App holds `pull_requests: write` and `contents: write` (ADR 0015). A pull request that the row marked ready is no longer a draft, so the implement job leaves its item until a person merges the plan and the item is Planned. The row counts under `row="stage_ready"` in `devops_cli_github_requests_total`. 19. The model of each stage comes from the settings `ai.tasks..model` of the ConfigMap `devops-cli-config` (#3155). The slot `analysis` serves the model lane: the stage job (planning), refine and the alternatives check. The slot `verification` names the model that reviews a plan; no code reads it before the verification pass of #3153 is built. The slot `implement` serves every step of the implement job, and `roadmap.implement.ladder` names the same group as the one tier of each step class (#3573). The ConfigMap is set to `deepseek-v4.1-flash-1017` for `analysis` and `implement`, and to `claude-sonnet-5-5` for `verification`. The window of `implement` is 256,000 tokens: the smallest that an fp8 provider of the group offers, where the group allows 1,048,576. A step has one tier, so a failure that selects `change_tier` ends the step; the item stops and waits for a retry. The prompts of a step hold the code of the repository, so they leave the cluster to OpenRouter (SECURITY.md rule 13). `claude-sonnet-5-5` serves no step. A live run of `devops roadmap eval implement --model deepseek-v4.1-flash-1017 -F 2152-numeric-host-doubles` on 2026-11-05 passed every property in 281 s, with 42 model requests, 2,126,241 input tokens (the sum over the 42 requests) and 11,590 output tokens. The evaluation of 2026-11-02 chose them: on 16 golden cases the local `qwen3.8:27b` passed the stage gates in 3, `claude-sonnet-5-5` in 7 and `deepseek-v4.1-flash-1017` in 11, at about 0.002 USD a call for DeepSeek; in the review of 12 golden plans with 82 known defects, Sonnet 5.5 found 56% (68% of the major ones), DeepSeek 38% and the local models 22 to 23%. The plans of every model still need fixes, because the planner cannot read code (#3037). A failed Sonnet call falls back to DeepSeek at the gateway (#3152). The ConfigMap is not under Argo CD: change it with `kubectl -n devops patch configmap devops-cli-config`, then `kubectl -n devops rollout restart deployment/roadmap-service`; the CronJobs read it at each run. Measure a change of model with `devops roadmap eval stage --model `. @@ -702,7 +702,7 @@ The rule group `roadmap-service` in `monitoring/prometheus-values.yaml` watches | `RoadmapServiceLaneStalled` | No job of a repository and lane started in 2 hours while the deployment has an available replica. In a new pod, the 2 hours start when the worker of the lane starts. | 10m | warning | Find in the Service log a pause of the rounds, a spent request budget, or a job that does not end. If a job does not end, restart the deployment. | | `RoadmapServiceRestarted` | The `service` container restarted in the last hour | — | warning | Read the log before the restart: `kubectl -n devops logs deployment/roadmap-service --previous`. | | `RoadmapServiceUnavailable` | The deployment has an unavailable replica. A rollout stops the old pod before it starts the new one, so the rule waits 10 minutes. | 10m | warning | Read the events: `kubectl -n devops describe deployment/roadmap-service` and `kubectl -n devops get events --sort-by=.lastTimestamp`. | -| `RoadmapServiceGitHubRequestsHigh` | The Service sent more GitHub requests in 1 hour than 90% of the budget it reports in `devops_cli_service_budget_limit_requests` (`service.github_requests_per_hour`, 500 by default) | — | warning | Read the GitHub Requests by Resource panel of the `devops-cli Telemetry` dashboard, and find the repository in the Service log. Lower the budget or remove a repository from `service.repos`, then run `devops k8s deploy-stack --stack devops`. | +| `RoadmapServiceGitHubRequestsHigh` | The Service sent more GitHub requests in 1 hour than 90% of the budget it reports in `devops_cli_service_budget_limit_requests` (`service.github_requests_per_hour`, 500 by default) | — | warning | Read the GitHub Requests by Resource panel of the `devops-cli Telemetry` dashboard, and find the repository in the Service log. If the requests are expected, raise the budget with `devops config set service.github_requests_per_hour `. Otherwise, remove the repository from `service.repos`. Then run `devops k8s deploy-stack --stack devops`. | | `RoadmapServiceModelRequestsHigh` | The Service sent more than 352 model requests in 1 hour. A request that the cache answered does not count. | — | warning | Find the model on the `AI Spend & LLM Usage Telemetry` dashboard, and the repository in the Service log. Remove the repository from `service.repos` until the cause is known, then run `devops k8s deploy-stack --stack devops`. | | `RoadmapServiceWebhooksRejected` | The webhook route rejected more than 20 deliveries in 15 minutes (Webhook route). A label set that starts in the 15 minutes counts from zero. | — | warning | Check the last response of each hook (step 8). A hook whose last `code` is 401 has a secret that is not the one in `service.webhook_secrets`: set the same secret on both sides (step 9). If each hook gets 200 or 202, the deliveries are forged: read them in the Service log (`kubectl -n devops logs deployment/roadmap-service \| grep rejected`), and the 429 answers of Traefik (Webhook route). | diff --git a/openspec/changes/3495-service-counters-zero-series-repo-label/tasks.md b/openspec/changes/3495-service-counters-zero-series-repo-label/tasks.md index 45b693d37..c5d64404e 100644 --- a/openspec/changes/3495-service-counters-zero-series-repo-label/tasks.md +++ b/openspec/changes/3495-service-counters-zero-series-repo-label/tasks.md @@ -46,9 +46,9 @@ Reads: `tests/fixtures/prometheus_rules/roadmap_service_tests.yaml`, `tests/fixt Reads: `docs/TELEMETRY.md`, `k8s/README.md`. Changes: `docs/TELEMETRY.md`, `k8s/README.md`, `tests/prometheus/test_k8s_monitoring_alerting.py`. -- [ ] 5.1 Add a test to `tests/prometheus/test_k8s_monitoring_alerting.py` that asserts that the action text of `RoadmapServiceGitHubRequestsHigh` in `k8s/README.md` does not say to lower the budget +- [x] 5.1 Add a test to `tests/prometheus/test_k8s_monitoring_alerting.py` that asserts that the action text of `RoadmapServiceGitHubRequestsHigh` in `k8s/README.md` does not say to lower the budget - Verification: `uv run devops ci test tests/prometheus/test_k8s_monitoring_alerting.py` fails on the base -- [ ] 5.2 Rewrite the action text of `RoadmapServiceGitHubRequestsHigh` in `k8s/README.md` without the word to lower the budget, rewrite `docs/TELEMETRY.md` lines 19-32 to state the scrape behavior and the labels, state the class count of 19 in `k8s/README.md`, and run `uv run devops docs generate --sync-readme` +- [x] 5.2 Rewrite the action text of `RoadmapServiceGitHubRequestsHigh` in `k8s/README.md` without the word to lower the budget, rewrite `docs/TELEMETRY.md` lines 19-32 to state the scrape behavior and the labels, state the class count of 19 in `k8s/README.md`, and run `uv run devops docs generate --sync-readme` - Verification: `uv run devops ci test tests/prometheus/test_k8s_monitoring_alerting.py` passes and `uv run devops ci docs` passes ## Workflow follow-up diff --git a/src/devops_cli/docs/generator.py b/src/devops_cli/docs/generator.py index 43d3a5385..1bef3eb20 100644 --- a/src/devops_cli/docs/generator.py +++ b/src/devops_cli/docs/generator.py @@ -1096,17 +1096,23 @@ def generate_telemetry_docs(self) -> str: "it. Prometheus names a series by `service.name` (`job`), `service.instance.id`", "(`instance`) and the metric's attributes. The `service.instance.id` is the host name,", "which every command on the host shares. The standard `OTEL_RESOURCE_ATTRIBUTES` sets", - "another one: roadmap-service sets `service.instance.id=roadmap-service`, so a new pod", - "writes the series of the pod before it. A running total that starts again from zero,", - "for a new pod or for a stream that gets no delta for `max_stale` (`1h` in", + "another one: roadmap-service sets `service.instance.id=roadmap-service`, so on the OTLP", + "path a new pod continues the series of the pod before it. The `/metrics` scrape adds the", + "label `pod`, so each pod has its own series there. A running total that starts again from", + "zero, for a new pod or for a stream that gets no delta for `max_stale` (`1h` in", "`k8s/otel/values.yaml`), is a counter reset to `rate()` and `increase()`.", "Cluster job pods have unique instance names and export metrics only once when they exit.", "PromQL functions like `increase()` and `rate()` drop series that contain only one sample.", "To calculate these counters, the collector transmits created timestamps via remote write 2.0", "(`remote_write_version: 2` in `k8s/otel/values.yaml`). Prometheus sets", "`enable-feature: created-timestamp-zero-ingestion` in `k8s/monitoring/prometheus-values.yaml`.", - "Prometheus then writes a zero-value sample at the start timestamp of each series, so", - "`increase()` calculates the full counter value.", + "On this path, Prometheus writes a zero-value sample at the start timestamp of each series,", + "so `increase()` calculates the full counter value. The `/metrics` scrape gets no such sample.", + "The Service starts each label set of its counters at zero (#3495), so the first count of a", + "label set is in its series. A count that a pod makes before its first scrape is still lost,", + "because `increase()` does not count the first sample of a series. The labels of the Service", + "counters are: `devops_cli_service_implement_failures_total` with `class` and `repo`, and", + "`devops_cli_service_jobs_total` with `repo`, `lane`, `row` and `result`.", "A gauge holds a snapshot, such as a project metric that the roadmap Service sends", "every 15 minutes. Prometheus finds a plain selector only within five minutes of a", "sample, so a dashboard reads a gauge with `last_over_time([$__range])`, and", diff --git a/tests/prometheus/test_k8s_monitoring_alerting.py b/tests/prometheus/test_k8s_monitoring_alerting.py index 48c6b5cac..a4fbdea0a 100644 --- a/tests/prometheus/test_k8s_monitoring_alerting.py +++ b/tests/prometheus/test_k8s_monitoring_alerting.py @@ -1268,3 +1268,16 @@ def test_each_unit_tested_rule_fires_on_a_sample_and_stays_quiet_on_a_normal_ser sorted(rule["alert"] for rule in _group(group)), 0, ), result.stdout + result.stderr + + +def test_github_requests_high_action_does_not_lower_budget() -> None: + """The action of `RoadmapServiceGitHubRequestsHigh` names no lower budget. The alert follows + the budget, so a lower budget makes the alert fire sooner (#3495).""" + readme = (K8S / "README.md").read_text(encoding="utf-8") + row = next( + line + for line in readme.splitlines() + if line.startswith("| `RoadmapServiceGitHubRequestsHigh` |") + ) + + assert "lower the budget" not in row.lower() From d4ba04cf30a4a0360d0cb74ec20c60e71f6c564d Mon Sep 17 00:00:00 2001 From: Daniel Petty Date: Thu, 5 Nov 2026 04:40:22 +0000 Subject: [PATCH 07/17] docs(openspec): archive #3495 service counter series Archive the change folder and write the capability roadmap-service-counters into the main specs. Change: openspec/changes/archive/2026-11-05-3495-service-counters-zero-series-repo-label --- .../.openspec.yaml | 0 .../design.md | 0 .../proposal.md | 0 .../specs/roadmap-implement-job/spec.md | 0 .../specs/roadmap-service-counters/spec.md | 0 .../tasks.md | 0 openspec/specs/roadmap-implement-job/spec.md | 10 ++-- .../specs/roadmap-service-counters/spec.md | 46 +++++++++++++++++++ .../roadmap_service_tests.yaml | 8 ++-- tests/roadmap/test_roadmap_run.py | 37 +++++++++++++++ 10 files changed, 92 insertions(+), 9 deletions(-) rename openspec/changes/{3495-service-counters-zero-series-repo-label => archive/2026-11-05-3495-service-counters-zero-series-repo-label}/.openspec.yaml (100%) rename openspec/changes/{3495-service-counters-zero-series-repo-label => archive/2026-11-05-3495-service-counters-zero-series-repo-label}/design.md (100%) rename openspec/changes/{3495-service-counters-zero-series-repo-label => archive/2026-11-05-3495-service-counters-zero-series-repo-label}/proposal.md (100%) rename openspec/changes/{3495-service-counters-zero-series-repo-label => archive/2026-11-05-3495-service-counters-zero-series-repo-label}/specs/roadmap-implement-job/spec.md (100%) rename openspec/changes/{3495-service-counters-zero-series-repo-label => archive/2026-11-05-3495-service-counters-zero-series-repo-label}/specs/roadmap-service-counters/spec.md (100%) rename openspec/changes/{3495-service-counters-zero-series-repo-label => archive/2026-11-05-3495-service-counters-zero-series-repo-label}/tasks.md (100%) create mode 100644 openspec/specs/roadmap-service-counters/spec.md diff --git a/openspec/changes/3495-service-counters-zero-series-repo-label/.openspec.yaml b/openspec/changes/archive/2026-11-05-3495-service-counters-zero-series-repo-label/.openspec.yaml similarity index 100% rename from openspec/changes/3495-service-counters-zero-series-repo-label/.openspec.yaml rename to openspec/changes/archive/2026-11-05-3495-service-counters-zero-series-repo-label/.openspec.yaml diff --git a/openspec/changes/3495-service-counters-zero-series-repo-label/design.md b/openspec/changes/archive/2026-11-05-3495-service-counters-zero-series-repo-label/design.md similarity index 100% rename from openspec/changes/3495-service-counters-zero-series-repo-label/design.md rename to openspec/changes/archive/2026-11-05-3495-service-counters-zero-series-repo-label/design.md diff --git a/openspec/changes/3495-service-counters-zero-series-repo-label/proposal.md b/openspec/changes/archive/2026-11-05-3495-service-counters-zero-series-repo-label/proposal.md similarity index 100% rename from openspec/changes/3495-service-counters-zero-series-repo-label/proposal.md rename to openspec/changes/archive/2026-11-05-3495-service-counters-zero-series-repo-label/proposal.md diff --git a/openspec/changes/3495-service-counters-zero-series-repo-label/specs/roadmap-implement-job/spec.md b/openspec/changes/archive/2026-11-05-3495-service-counters-zero-series-repo-label/specs/roadmap-implement-job/spec.md similarity index 100% rename from openspec/changes/3495-service-counters-zero-series-repo-label/specs/roadmap-implement-job/spec.md rename to openspec/changes/archive/2026-11-05-3495-service-counters-zero-series-repo-label/specs/roadmap-implement-job/spec.md diff --git a/openspec/changes/3495-service-counters-zero-series-repo-label/specs/roadmap-service-counters/spec.md b/openspec/changes/archive/2026-11-05-3495-service-counters-zero-series-repo-label/specs/roadmap-service-counters/spec.md similarity index 100% rename from openspec/changes/3495-service-counters-zero-series-repo-label/specs/roadmap-service-counters/spec.md rename to openspec/changes/archive/2026-11-05-3495-service-counters-zero-series-repo-label/specs/roadmap-service-counters/spec.md diff --git a/openspec/changes/3495-service-counters-zero-series-repo-label/tasks.md b/openspec/changes/archive/2026-11-05-3495-service-counters-zero-series-repo-label/tasks.md similarity index 100% rename from openspec/changes/3495-service-counters-zero-series-repo-label/tasks.md rename to openspec/changes/archive/2026-11-05-3495-service-counters-zero-series-repo-label/tasks.md diff --git a/openspec/specs/roadmap-implement-job/spec.md b/openspec/specs/roadmap-implement-job/spec.md index 9e5b3b0de..51d4476b3 100644 --- a/openspec/specs/roadmap-implement-job/spec.md +++ b/openspec/specs/roadmap-implement-job/spec.md @@ -771,15 +771,15 @@ The stop comment of the job SHALL name the failure class, the action taken and t - **THEN** `implement.failure` is removed with `implement.stopped` ### Requirement: The Service counts the failures by class -When a round reports a Job that left a stop, the Service SHALL add 1 to `devops_cli_service_implement_failures_total` with the label `class`. A Job that left the item without a result SHALL get the class of the tail of its log, recorded in `implement.failure` and named, with the action left, in the report. A Job that ended with a result SHALL add nothing. +When a round reports a Job that left a stop, the Service SHALL add 1 to `devops_cli_service_implement_failures_total` with the labels `class` and `repo`, where `repo` is the repository of the round. A Job that left the item without a result SHALL get the class of the tail of its log, recorded in `implement.failure` and named, with the action left, in the report. A Job that ended with a result SHALL add nothing. #### Scenario: A stop of the runner -- **WHEN** a round reports a Job whose item holds the stop with the class `parser` -- **THEN** the counter for `parser` grows by 1, the counter for `oom` does not change, and a second round adds nothing +- **WHEN** a round of `example/app` reports a Job whose item holds the stop with the class `parser` +- **THEN** the counter for `parser` and `example/app` grows by 1, the counter for `oom` does not change, and a second round adds nothing #### Scenario: A Job with no result -- **WHEN** a round reports a Job that failed and left the item In Progress, and its log holds "out of memory" -- **THEN** the item is Blocked with `implement.failure` `oom`, the report names the class and the action left `raise_limit`, and the counter for `oom` grows by 1 +- **WHEN** a round of `example/app` reports a Job that failed and left the item In Progress, and its log holds "out of memory" +- **THEN** the item is Blocked with `implement.failure` `oom`, the report names the class and the action left `raise_limit`, and the counter for `oom` and `example/app` grows by 1 #### Scenario: A Job with a result - **WHEN** a round reports a Job that succeeded and left the item In Review diff --git a/openspec/specs/roadmap-service-counters/spec.md b/openspec/specs/roadmap-service-counters/spec.md new file mode 100644 index 000000000..5cd1d0c44 --- /dev/null +++ b/openspec/specs/roadmap-service-counters/spec.md @@ -0,0 +1,46 @@ +# roadmap-service-counters Specification + +## Purpose +The label sets of the Service counters that the Service creates at zero when it starts, the labels that the job counter and the failure counter carry, and the promtool cases that prove the alerts and the true total over a pod change. A count of a label set that starts at zero is not lost to increase(). + +## Requirements + +### Requirement: The Service starts each label set of its counters at zero +The Service SHALL create each bounded label set of its counters at zero when it starts. The bounded label sets are: each failure class for each repository of `service.repos` on `devops_cli_service_implement_failures_total`; each repository, lane, row and result on `devops_cli_service_jobs_total`; each repository and lane on `devops_cli_service_job_seconds_total`; and each provider, declared event or `other`, and outcome on `devops_cli_service_webhook_deliveries_total`. + +#### Scenario: The failure counter exists at zero after start +- **WHEN** the Service starts with the repositories `example/app` and `example/tools` +- **THEN** the failure counter exists at 0 for each of the 19 failure classes and each of the two repositories + +#### Scenario: The label repo holds only configured repositories +- **WHEN** the Service starts with the repositories `example/app` and `example/tools` +- **THEN** every value of the label `repo` of the failure counter is `example/app` or `example/tools` + +#### Scenario: The webhook counter exists at zero for each declared event +- **WHEN** the Service starts with a provider that declares the events `issues` and `push` +- **THEN** the webhook counter exists at 0 for each of the outcomes with the events `issues`, `push` and `other` + +### Requirement: The job counter carries the row of the job +The Service SHALL add 1 to `devops_cli_service_jobs_total` with the labels `repo`, `lane`, `row` and `result` for each row of a lane that ends in the roadmap run. A scoped update SHALL add 1 with the name of its job as the label `row`. The Service SHALL NOT count a lane batch as one job. + +#### Scenario: A row that succeeds +- **WHEN** the status lane runs the row `metrics` for `example/app` and the row succeeds +- **THEN** the job counter for `repo` `example/app`, `lane` `status`, `row` `metrics` and `result` `success` grows by 1 + +#### Scenario: Two rows of one batch +- **WHEN** one round of the status lane runs the rows `metrics` and `implement`, and both fail +- **THEN** the job counter for each row with `result` `error` grows by 1, and no series counts the batch + +### Requirement: Alerts read a counter that starts at zero with increase +The alerts `RoadmapServiceWebhooksRejected` and `RoadmapServiceSelfDeliveriesHigh` SHALL read their counter with `increase()` and no `offset` term, because the Service starts each label set at zero. + +#### Scenario: A rejected burst on a new pod +- **WHEN** `promtool test rules` evaluates the rejected series of a pod that starts at 0 and reaches 25 within 15 minutes +- **THEN** `RoadmapServiceWebhooksRejected` fires + +### Requirement: A query gives the true total over a pod change +A query of `sum(increase(devops_cli_service_implement_failures_total[24h]))` SHALL give the true total of the counts of all pods in the window, including the counts of a pod that started during the window. + +#### Scenario: A new pod adds its counts +- **WHEN** `promtool test rules` evaluates the failure counter of a pod that counted 3 and of a new pod that counted 2 after a zero sample at its start +- **THEN** the sum of `increase()` over the window is 5 diff --git a/tests/fixtures/prometheus_rules/roadmap_service_tests.yaml b/tests/fixtures/prometheus_rules/roadmap_service_tests.yaml index 10d1f4c35..2f92590a2 100644 --- a/tests/fixtures/prometheus_rules/roadmap_service_tests.yaml +++ b/tests/fixtures/prometheus_rules/roadmap_service_tests.yaml @@ -448,15 +448,15 @@ tests: interval: 1m input_series: # Pod a counts 3 failures from minute 0 to minute 3, then stays at 3. Pod b starts at zero at - # minute 10 (#3495) and counts 2 failures by minute 12. + # minute 10 (#3495) and counts 2 failures by minute 12. Both series reach the eval time, so + # increase() extrapolates to no window edge, and the sum is exactly the 5 counts. - series: 'devops_cli_service_implement_failures_total{job="roadmap-service", pod="a", repo="example/app", class="ci_infra"}' - values: '0+1x3 3x40' + values: '0+1x3 3x46' - series: 'devops_cli_service_implement_failures_total{job="roadmap-service", pod="b", repo="example/app", class="ci_infra"}' values: '_x10 0+1x2 2x40' promql_expr_test: - # increase() extrapolates to the window edges, so the sum reads 5.03 where the counts are 5. - expr: sum(increase(devops_cli_service_implement_failures_total{job="roadmap-service", class="ci_infra"}[1h])) eval_time: 50m exp_samples: - labels: '{}' - value: 5.034090909090909 + value: 5 diff --git a/tests/roadmap/test_roadmap_run.py b/tests/roadmap/test_roadmap_run.py index 3001ba5e9..a14da2891 100644 --- a/tests/roadmap/test_roadmap_run.py +++ b/tests/roadmap/test_roadmap_run.py @@ -2666,6 +2666,43 @@ def test_the_issues_actions_run_a_scoped_intake_on_the_triage_lane() -> None: } +def test_a_scoped_update_counts_its_job_under_its_result( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch +) -> None: + """A scoped update counts one job of the job counter under its repository, its lane, its job + as the row and its result: `success` when its runner returns, and `error` when it raises, and + the error still reaches the caller (#3495).""" + GLOBAL_METRICS.reset() + store, _, _ = _candidates(3) + _stamp(tmp_path, intake=TWENTY_MINUTES_AGO) + model = _IntakeModel() + + run_scoped_update(_delivery("opened", 2), store, data_dir=tmp_path, now=NOW, model=model) + + def boom(*_args: object, **_kwargs: object) -> JobOutcome: + raise RuntimeError("the intake model failed") + + monkeypatch.setattr( + "devops_cli.roadmap.run.SCOPED_UPDATES", + {key: replace(update, runner=boom) for key, update in SCOPED_UPDATES.items()}, + ) + with pytest.raises(RuntimeError, match="the intake model failed"): + run_scoped_update(_delivery("opened", 2), store, data_dir=tmp_path, now=NOW, model=model) + + def count(result: str) -> float: + return GLOBAL_METRICS.get_counter( + CONST_SERVICE_METRIC_JOBS, + labels={ + "repo": REPO, + "lane": CONST_SERVICE_LANE_TRIAGE, + "row": "intake", + "result": result, + }, + ) + + assert (count("success"), count("error")) == (1, 1) + + @pytest.mark.parametrize("action", ["opened", "edited", "reopened"]) def test_an_issues_delivery_intakes_only_its_issue_and_writes_no_last_success( action: str, tmp_path: Path From 59e1660975e011471e3cbc59d76e3e0587011589 Mon Sep 17 00:00:00 2001 From: Daniel Petty Date: Thu, 5 Nov 2026 07:45:36 +0000 Subject: [PATCH 08/17] Revert "docs(openspec): archive #3495 service counter series" This reverts commit 923a0753b8cbce6ff4db154c29088470f4fa1d0b. --- .../.openspec.yaml | 0 .../design.md | 0 .../proposal.md | 0 .../specs/roadmap-implement-job/spec.md | 0 .../specs/roadmap-service-counters/spec.md | 0 .../tasks.md | 0 openspec/specs/roadmap-implement-job/spec.md | 10 ++-- .../specs/roadmap-service-counters/spec.md | 46 ------------------- .../roadmap_service_tests.yaml | 8 ++-- tests/roadmap/test_roadmap_run.py | 37 --------------- 10 files changed, 9 insertions(+), 92 deletions(-) rename openspec/changes/{archive/2026-11-05-3495-service-counters-zero-series-repo-label => 3495-service-counters-zero-series-repo-label}/.openspec.yaml (100%) rename openspec/changes/{archive/2026-11-05-3495-service-counters-zero-series-repo-label => 3495-service-counters-zero-series-repo-label}/design.md (100%) rename openspec/changes/{archive/2026-11-05-3495-service-counters-zero-series-repo-label => 3495-service-counters-zero-series-repo-label}/proposal.md (100%) rename openspec/changes/{archive/2026-11-05-3495-service-counters-zero-series-repo-label => 3495-service-counters-zero-series-repo-label}/specs/roadmap-implement-job/spec.md (100%) rename openspec/changes/{archive/2026-11-05-3495-service-counters-zero-series-repo-label => 3495-service-counters-zero-series-repo-label}/specs/roadmap-service-counters/spec.md (100%) rename openspec/changes/{archive/2026-11-05-3495-service-counters-zero-series-repo-label => 3495-service-counters-zero-series-repo-label}/tasks.md (100%) delete mode 100644 openspec/specs/roadmap-service-counters/spec.md diff --git a/openspec/changes/archive/2026-11-05-3495-service-counters-zero-series-repo-label/.openspec.yaml b/openspec/changes/3495-service-counters-zero-series-repo-label/.openspec.yaml similarity index 100% rename from openspec/changes/archive/2026-11-05-3495-service-counters-zero-series-repo-label/.openspec.yaml rename to openspec/changes/3495-service-counters-zero-series-repo-label/.openspec.yaml diff --git a/openspec/changes/archive/2026-11-05-3495-service-counters-zero-series-repo-label/design.md b/openspec/changes/3495-service-counters-zero-series-repo-label/design.md similarity index 100% rename from openspec/changes/archive/2026-11-05-3495-service-counters-zero-series-repo-label/design.md rename to openspec/changes/3495-service-counters-zero-series-repo-label/design.md diff --git a/openspec/changes/archive/2026-11-05-3495-service-counters-zero-series-repo-label/proposal.md b/openspec/changes/3495-service-counters-zero-series-repo-label/proposal.md similarity index 100% rename from openspec/changes/archive/2026-11-05-3495-service-counters-zero-series-repo-label/proposal.md rename to openspec/changes/3495-service-counters-zero-series-repo-label/proposal.md diff --git a/openspec/changes/archive/2026-11-05-3495-service-counters-zero-series-repo-label/specs/roadmap-implement-job/spec.md b/openspec/changes/3495-service-counters-zero-series-repo-label/specs/roadmap-implement-job/spec.md similarity index 100% rename from openspec/changes/archive/2026-11-05-3495-service-counters-zero-series-repo-label/specs/roadmap-implement-job/spec.md rename to openspec/changes/3495-service-counters-zero-series-repo-label/specs/roadmap-implement-job/spec.md diff --git a/openspec/changes/archive/2026-11-05-3495-service-counters-zero-series-repo-label/specs/roadmap-service-counters/spec.md b/openspec/changes/3495-service-counters-zero-series-repo-label/specs/roadmap-service-counters/spec.md similarity index 100% rename from openspec/changes/archive/2026-11-05-3495-service-counters-zero-series-repo-label/specs/roadmap-service-counters/spec.md rename to openspec/changes/3495-service-counters-zero-series-repo-label/specs/roadmap-service-counters/spec.md diff --git a/openspec/changes/archive/2026-11-05-3495-service-counters-zero-series-repo-label/tasks.md b/openspec/changes/3495-service-counters-zero-series-repo-label/tasks.md similarity index 100% rename from openspec/changes/archive/2026-11-05-3495-service-counters-zero-series-repo-label/tasks.md rename to openspec/changes/3495-service-counters-zero-series-repo-label/tasks.md diff --git a/openspec/specs/roadmap-implement-job/spec.md b/openspec/specs/roadmap-implement-job/spec.md index 51d4476b3..9e5b3b0de 100644 --- a/openspec/specs/roadmap-implement-job/spec.md +++ b/openspec/specs/roadmap-implement-job/spec.md @@ -771,15 +771,15 @@ The stop comment of the job SHALL name the failure class, the action taken and t - **THEN** `implement.failure` is removed with `implement.stopped` ### Requirement: The Service counts the failures by class -When a round reports a Job that left a stop, the Service SHALL add 1 to `devops_cli_service_implement_failures_total` with the labels `class` and `repo`, where `repo` is the repository of the round. A Job that left the item without a result SHALL get the class of the tail of its log, recorded in `implement.failure` and named, with the action left, in the report. A Job that ended with a result SHALL add nothing. +When a round reports a Job that left a stop, the Service SHALL add 1 to `devops_cli_service_implement_failures_total` with the label `class`. A Job that left the item without a result SHALL get the class of the tail of its log, recorded in `implement.failure` and named, with the action left, in the report. A Job that ended with a result SHALL add nothing. #### Scenario: A stop of the runner -- **WHEN** a round of `example/app` reports a Job whose item holds the stop with the class `parser` -- **THEN** the counter for `parser` and `example/app` grows by 1, the counter for `oom` does not change, and a second round adds nothing +- **WHEN** a round reports a Job whose item holds the stop with the class `parser` +- **THEN** the counter for `parser` grows by 1, the counter for `oom` does not change, and a second round adds nothing #### Scenario: A Job with no result -- **WHEN** a round of `example/app` reports a Job that failed and left the item In Progress, and its log holds "out of memory" -- **THEN** the item is Blocked with `implement.failure` `oom`, the report names the class and the action left `raise_limit`, and the counter for `oom` and `example/app` grows by 1 +- **WHEN** a round reports a Job that failed and left the item In Progress, and its log holds "out of memory" +- **THEN** the item is Blocked with `implement.failure` `oom`, the report names the class and the action left `raise_limit`, and the counter for `oom` grows by 1 #### Scenario: A Job with a result - **WHEN** a round reports a Job that succeeded and left the item In Review diff --git a/openspec/specs/roadmap-service-counters/spec.md b/openspec/specs/roadmap-service-counters/spec.md deleted file mode 100644 index 5cd1d0c44..000000000 --- a/openspec/specs/roadmap-service-counters/spec.md +++ /dev/null @@ -1,46 +0,0 @@ -# roadmap-service-counters Specification - -## Purpose -The label sets of the Service counters that the Service creates at zero when it starts, the labels that the job counter and the failure counter carry, and the promtool cases that prove the alerts and the true total over a pod change. A count of a label set that starts at zero is not lost to increase(). - -## Requirements - -### Requirement: The Service starts each label set of its counters at zero -The Service SHALL create each bounded label set of its counters at zero when it starts. The bounded label sets are: each failure class for each repository of `service.repos` on `devops_cli_service_implement_failures_total`; each repository, lane, row and result on `devops_cli_service_jobs_total`; each repository and lane on `devops_cli_service_job_seconds_total`; and each provider, declared event or `other`, and outcome on `devops_cli_service_webhook_deliveries_total`. - -#### Scenario: The failure counter exists at zero after start -- **WHEN** the Service starts with the repositories `example/app` and `example/tools` -- **THEN** the failure counter exists at 0 for each of the 19 failure classes and each of the two repositories - -#### Scenario: The label repo holds only configured repositories -- **WHEN** the Service starts with the repositories `example/app` and `example/tools` -- **THEN** every value of the label `repo` of the failure counter is `example/app` or `example/tools` - -#### Scenario: The webhook counter exists at zero for each declared event -- **WHEN** the Service starts with a provider that declares the events `issues` and `push` -- **THEN** the webhook counter exists at 0 for each of the outcomes with the events `issues`, `push` and `other` - -### Requirement: The job counter carries the row of the job -The Service SHALL add 1 to `devops_cli_service_jobs_total` with the labels `repo`, `lane`, `row` and `result` for each row of a lane that ends in the roadmap run. A scoped update SHALL add 1 with the name of its job as the label `row`. The Service SHALL NOT count a lane batch as one job. - -#### Scenario: A row that succeeds -- **WHEN** the status lane runs the row `metrics` for `example/app` and the row succeeds -- **THEN** the job counter for `repo` `example/app`, `lane` `status`, `row` `metrics` and `result` `success` grows by 1 - -#### Scenario: Two rows of one batch -- **WHEN** one round of the status lane runs the rows `metrics` and `implement`, and both fail -- **THEN** the job counter for each row with `result` `error` grows by 1, and no series counts the batch - -### Requirement: Alerts read a counter that starts at zero with increase -The alerts `RoadmapServiceWebhooksRejected` and `RoadmapServiceSelfDeliveriesHigh` SHALL read their counter with `increase()` and no `offset` term, because the Service starts each label set at zero. - -#### Scenario: A rejected burst on a new pod -- **WHEN** `promtool test rules` evaluates the rejected series of a pod that starts at 0 and reaches 25 within 15 minutes -- **THEN** `RoadmapServiceWebhooksRejected` fires - -### Requirement: A query gives the true total over a pod change -A query of `sum(increase(devops_cli_service_implement_failures_total[24h]))` SHALL give the true total of the counts of all pods in the window, including the counts of a pod that started during the window. - -#### Scenario: A new pod adds its counts -- **WHEN** `promtool test rules` evaluates the failure counter of a pod that counted 3 and of a new pod that counted 2 after a zero sample at its start -- **THEN** the sum of `increase()` over the window is 5 diff --git a/tests/fixtures/prometheus_rules/roadmap_service_tests.yaml b/tests/fixtures/prometheus_rules/roadmap_service_tests.yaml index 2f92590a2..10d1f4c35 100644 --- a/tests/fixtures/prometheus_rules/roadmap_service_tests.yaml +++ b/tests/fixtures/prometheus_rules/roadmap_service_tests.yaml @@ -448,15 +448,15 @@ tests: interval: 1m input_series: # Pod a counts 3 failures from minute 0 to minute 3, then stays at 3. Pod b starts at zero at - # minute 10 (#3495) and counts 2 failures by minute 12. Both series reach the eval time, so - # increase() extrapolates to no window edge, and the sum is exactly the 5 counts. + # minute 10 (#3495) and counts 2 failures by minute 12. - series: 'devops_cli_service_implement_failures_total{job="roadmap-service", pod="a", repo="example/app", class="ci_infra"}' - values: '0+1x3 3x46' + values: '0+1x3 3x40' - series: 'devops_cli_service_implement_failures_total{job="roadmap-service", pod="b", repo="example/app", class="ci_infra"}' values: '_x10 0+1x2 2x40' promql_expr_test: + # increase() extrapolates to the window edges, so the sum reads 5.03 where the counts are 5. - expr: sum(increase(devops_cli_service_implement_failures_total{job="roadmap-service", class="ci_infra"}[1h])) eval_time: 50m exp_samples: - labels: '{}' - value: 5 + value: 5.034090909090909 diff --git a/tests/roadmap/test_roadmap_run.py b/tests/roadmap/test_roadmap_run.py index a14da2891..3001ba5e9 100644 --- a/tests/roadmap/test_roadmap_run.py +++ b/tests/roadmap/test_roadmap_run.py @@ -2666,43 +2666,6 @@ def test_the_issues_actions_run_a_scoped_intake_on_the_triage_lane() -> None: } -def test_a_scoped_update_counts_its_job_under_its_result( - tmp_path: Path, monkeypatch: pytest.MonkeyPatch -) -> None: - """A scoped update counts one job of the job counter under its repository, its lane, its job - as the row and its result: `success` when its runner returns, and `error` when it raises, and - the error still reaches the caller (#3495).""" - GLOBAL_METRICS.reset() - store, _, _ = _candidates(3) - _stamp(tmp_path, intake=TWENTY_MINUTES_AGO) - model = _IntakeModel() - - run_scoped_update(_delivery("opened", 2), store, data_dir=tmp_path, now=NOW, model=model) - - def boom(*_args: object, **_kwargs: object) -> JobOutcome: - raise RuntimeError("the intake model failed") - - monkeypatch.setattr( - "devops_cli.roadmap.run.SCOPED_UPDATES", - {key: replace(update, runner=boom) for key, update in SCOPED_UPDATES.items()}, - ) - with pytest.raises(RuntimeError, match="the intake model failed"): - run_scoped_update(_delivery("opened", 2), store, data_dir=tmp_path, now=NOW, model=model) - - def count(result: str) -> float: - return GLOBAL_METRICS.get_counter( - CONST_SERVICE_METRIC_JOBS, - labels={ - "repo": REPO, - "lane": CONST_SERVICE_LANE_TRIAGE, - "row": "intake", - "result": result, - }, - ) - - assert (count("success"), count("error")) == (1, 1) - - @pytest.mark.parametrize("action", ["opened", "edited", "reopened"]) def test_an_issues_delivery_intakes_only_its_issue_and_writes_no_last_success( action: str, tmp_path: Path From 127a6a6d7a8062b39ef461a7ce7ac85ba8cc9c65 Mon Sep 17 00:00:00 2001 From: Daniel Petty Date: Thu, 5 Nov 2026 07:53:28 +0000 Subject: [PATCH 09/17] fix(telemetry): trigger and chat counters start at zero The trigger counter and the chat command counter had no series until their first count. The Service now creates each label set of both counters at zero when it starts. The result values of the chat command are named constants. The chat route and the priming use the same constants. The spec lists these label sets in their own requirement. The identity counter stays out of the list, as design.md records. --- .../design.md | 2 +- .../proposal.md | 6 +-- .../specs/roadmap-service-counters/spec.md | 9 ++++- .../tasks.md | 15 ++++++- src/devops_cli/config/constants.py | 7 ++++ src/devops_cli/server/counter_series.py | 39 +++++++++++++++++- src/devops_cli/server/routes/webhooks.py | 40 +++++++++---------- .../telemetry/test_service_counter_series.py | 29 ++++++++++++++ 8 files changed, 119 insertions(+), 28 deletions(-) diff --git a/openspec/changes/3495-service-counters-zero-series-repo-label/design.md b/openspec/changes/3495-service-counters-zero-series-repo-label/design.md index 96349f907..98d625919 100644 --- a/openspec/changes/3495-service-counters-zero-series-repo-label/design.md +++ b/openspec/changes/3495-service-counters-zero-series-repo-label/design.md @@ -3,7 +3,7 @@ ## Decisions - **The Service creates the bounded label sets of its counters at zero when it starts.** A labelled counter has no series until `.labels()` makes a child (see the Feasibility of the proposal). One function calls `.labels()` for each bounded label set, so the first scrape of a pod reads a zero series for each label set. -- **The label sets are bounded.** The failure classes are the 19 members of `FailureClass`. The repositories are `service.repos`. The rows are the rows of the three lane tables of the roadmap run. The events of the webhook counter are the declared events of each provider, plus `other`. No dynamic label value is created at start. +- **The label sets are bounded.** The failure classes are the 19 members of `FailureClass`. The repositories are `service.repos`. The rows are the rows of the three lane tables of the roadmap run. The events of the webhook counter are the declared events of each provider, plus `other`. The sources of the trigger counter are `webhook` and `poll`. The results of the chat command are `usage`, `status`, `unknown_repo` and `error`. No dynamic label value is created at start. - **The exposition format stays text 0.0.4.** The Service serves `/metrics` in this format (`src/devops_cli/server/service.py`). The zero children appear in this format, so the change does not switch the format. - **The jobs counter counts each row.** The roadmap run counts each row of a lane with its result, with the row name of the row table. The worker counts the scoped updates with the name of their job. A row counted in the run is not counted again in the worker, so no job is counted twice. - **The OTLP workaround terms stay.** The GitHub request and AI request series reach Prometheus over OTLP, not the `/metrics` scrape. The comment at `k8s/monitoring/prometheus-values.yaml` says that an OTLP series gets a sample only when the Service sends a new observation. No live Prometheus query ran in this change, so the OTLP terms stay until a live check shows that they are not needed. diff --git a/openspec/changes/3495-service-counters-zero-series-repo-label/proposal.md b/openspec/changes/3495-service-counters-zero-series-repo-label/proposal.md index d3dc99bd7..aba124636 100644 --- a/openspec/changes/3495-service-counters-zero-series-repo-label/proposal.md +++ b/openspec/changes/3495-service-counters-zero-series-repo-label/proposal.md @@ -6,7 +6,7 @@ Issue #3495 reports that the Service counters give a false picture. The counters ## What Changes -- When the Service starts, it creates each bounded label set of its counters at zero: each failure class for each repository of `service.repos`, each repository, lane, row and result of `devops_cli_service_jobs_total`, each repository and lane of `devops_cli_service_job_seconds_total`, and each provider, declared event and outcome of `devops_cli_service_webhook_deliveries_total`. +- When the Service starts, it creates each bounded label set of its counters at zero: each failure class for each repository of `service.repos`, each repository, lane, row and result of `devops_cli_service_jobs_total`, each repository and lane of `devops_cli_service_job_seconds_total`, each provider, declared event and outcome of `devops_cli_service_webhook_deliveries_total`, each repository and source of `devops_cli_service_triggers_total`, and each result of the `/roadmap` chat command on `devops_cli_service_chat_commands_total`. The identity counter `devops_cli_service_identity_refused_total` is left out (see design.md). - `devops_cli_service_implement_failures_total` gets the label `repo`. - `devops_cli_service_jobs_total` gets the label `row`. The roadmap run counts each row of a lane with its result. The worker no longer counts a whole lane batch. - The alert expressions of `RoadmapServiceWebhooksRejected` and `RoadmapServiceSelfDeliveriesHigh` lose their #2505 workaround terms. The OTLP alert expressions keep theirs. @@ -25,7 +25,7 @@ Issue #3495 reports that the Service counters give a false picture. The counters ## Impact -- Code: `src/devops_cli/telemetry/metrics.py`, `src/devops_cli/server/service.py`, `src/devops_cli/roadmap/implement_submit.py`, `src/devops_cli/roadmap/run.py`, `src/devops_cli/server/routes/webhooks.py`. +- Code: `src/devops_cli/telemetry/metrics.py`, `src/devops_cli/server/service.py`, `src/devops_cli/server/counter_series.py`, `src/devops_cli/roadmap/implement_submit.py`, `src/devops_cli/roadmap/run.py`, `src/devops_cli/server/routes/webhooks.py`, `src/devops_cli/config/constants.py`, `src/devops_cli/docs/generator.py`. - Monitoring: `k8s/monitoring/prometheus-values.yaml`. - Tests: `tests/prometheus/test_implement_failure_labels.py`, `tests/telemetry/test_service_counter_series.py`, `tests/prometheus/test_k8s_monitoring_alerting.py`, `tests/fixtures/prometheus_rules/roadmap_service_tests.yaml`, `tests/fixtures/prometheus_rules/roadmap_service_webhooks_tests.yaml`. - Documents: `docs/TELEMETRY.md`, `k8s/README.md`. @@ -44,4 +44,4 @@ Issue #3495 reports that the Service counters give a false picture. The counters ## Existing Tool Considered -The open tool `prometheus_client` covers the zero series: calling `.labels(...)` at start creates the child at 0, and the text exposition shows it (see Feasibility). The open tool `promtool` checks the rules and runs their unit tests. Both are present, so the change adds no code for the exposition format. The change deletes these internal parts: the per-batch count of `devops_cli_service_jobs_total` in the worker (`src/devops_cli/server/service.py`), the two #2505 workaround terms of the webhook alerts in `k8s/monitoring/prometheus-values.yaml`, and the stale comment at lines 670-673 of the same file. The code of the zero-start function is about 80 lines. It holds the label sets of the counters, and it adds no custom metric type. +The open tool `prometheus_client` covers the zero series: calling `.labels(...)` at start creates the child at 0, and the text exposition shows it (see Feasibility). The open tool `promtool` checks the rules and runs their unit tests. Both are present, so the change adds no code for the exposition format. The change deletes these internal parts: the per-batch count of `devops_cli_service_jobs_total` in the worker (`src/devops_cli/server/service.py`), the two #2505 workaround terms of the webhook alerts in `k8s/monitoring/prometheus-values.yaml`, and the stale comment at lines 670-673 of the same file. The module `src/devops_cli/server/counter_series.py` has 95 lines, and its function `prime_service_counters` has 48 lines. The function holds the label sets of the counters, and the change adds no custom metric type. diff --git a/openspec/changes/3495-service-counters-zero-series-repo-label/specs/roadmap-service-counters/spec.md b/openspec/changes/3495-service-counters-zero-series-repo-label/specs/roadmap-service-counters/spec.md index 9ef4aeafb..7e8b94890 100644 --- a/openspec/changes/3495-service-counters-zero-series-repo-label/specs/roadmap-service-counters/spec.md +++ b/openspec/changes/3495-service-counters-zero-series-repo-label/specs/roadmap-service-counters/spec.md @@ -1,7 +1,7 @@ ## ADDED Requirements ### Requirement: The Service starts each label set of its counters at zero -The Service SHALL create each bounded label set of its counters at zero when it starts. The bounded label sets are: each failure class for each repository of `service.repos` on `devops_cli_service_implement_failures_total`; each repository, lane, row and result on `devops_cli_service_jobs_total`; each repository and lane on `devops_cli_service_job_seconds_total`; and each provider, declared event or `other`, and outcome on `devops_cli_service_webhook_deliveries_total`. +The Service SHALL create each bounded label set of its counters at zero when it starts. The bounded label sets are: each failure class for each repository of `service.repos` on `devops_cli_service_implement_failures_total`; each repository, lane, row and result on `devops_cli_service_jobs_total`; each repository and lane on `devops_cli_service_job_seconds_total`; each provider, declared event or `other`, and outcome on `devops_cli_service_webhook_deliveries_total`. #### Scenario: The failure counter exists at zero after start - **WHEN** the Service starts with the repositories `example/app` and `example/tools` @@ -15,6 +15,13 @@ The Service SHALL create each bounded label set of its counters at zero when it - **WHEN** the Service starts with a provider that declares the events `issues` and `push` - **THEN** the webhook counter exists at 0 for each of the outcomes with the events `issues`, `push` and `other` +### Requirement: The Service starts the trigger and chat counters at zero +The Service SHALL create each bounded label set of `devops_cli_service_triggers_total` and `devops_cli_service_chat_commands_total` at zero when it starts. The bounded label sets are: each repository of `service.repos` with each source `webhook` or `poll`; and each result `usage`, `status`, `unknown_repo` or `error` of the `/roadmap` chat command. + +#### Scenario: The trigger and chat counters exist at zero after start +- **WHEN** the Service starts with the repositories `example/app` and `example/tools` +- **THEN** the trigger counter exists at 0 for each repository with the sources `webhook` and `poll`, and the chat counter exists at 0 for each result `usage`, `status`, `unknown_repo` and `error` + ### Requirement: The job counter carries the row of the job The Service SHALL add 1 to `devops_cli_service_jobs_total` with the labels `repo`, `lane`, `row` and `result` for each row of a lane that ends in the roadmap run. A scoped update SHALL add 1 with the name of its job as the label `row`. The Service SHALL NOT count a lane batch as one job. diff --git a/openspec/changes/3495-service-counters-zero-series-repo-label/tasks.md b/openspec/changes/3495-service-counters-zero-series-repo-label/tasks.md index c5d64404e..324cf7ade 100644 --- a/openspec/changes/3495-service-counters-zero-series-repo-label/tasks.md +++ b/openspec/changes/3495-service-counters-zero-series-repo-label/tasks.md @@ -37,7 +37,7 @@ Reads: `tests/fixtures/prometheus_rules/roadmap_service_tests.yaml`, `tests/fixt - [x] 4.1 Add a promtool case `pod_change_gives_true_total` to `tests/fixtures/prometheus_rules/roadmap_service_tests.yaml`. Its series of the failure counter start at 0 in each pod, and the case asserts the sum of `increase()` over a pod change - Verification: `promtool test rules tests/fixtures/prometheus_rules/roadmap_service_tests.yaml` passes -- [x] 4.2 Add a promtool case to `tests/fixtures/prometheus_rules/roadmap_service_webhooks_tests.yaml` that fires `RoadmapServiceWebhooksRejected` on a rejected series that starts at 0 +- [x] 4.2 Edit the first case of `tests/fixtures/prometheus_rules/roadmap_service_webhooks_tests.yaml`, so that its rejected series with the event `other` starts at 0 and the case fires `RoadmapServiceWebhooksRejected` at 34 minutes. The case replaces the #2505 gap with the zero samples and adds no new case - Verification: `promtool test rules tests/fixtures/prometheus_rules/roadmap_service_webhooks_tests.yaml` passes - [x] 4.3 Remove the #2505 workaround terms of `RoadmapServiceWebhooksRejected` and `RoadmapServiceSelfDeliveriesHigh` from `k8s/monitoring/prometheus-values.yaml`, and replace the stale comment at lines 670-673 with the scrape behavior. Keep the OTLP terms - Verification: `uv run devops ci test tests/prometheus/test_k8s_monitoring_alerting.py` passes and `promtool test rules tests/fixtures/prometheus_rules/roadmap_service_webhooks_tests.yaml` passes @@ -51,8 +51,21 @@ Reads: `docs/TELEMETRY.md`, `k8s/README.md`. Changes: `docs/TELEMETRY.md`, `k8s/ - [x] 5.2 Rewrite the action text of `RoadmapServiceGitHubRequestsHigh` in `k8s/README.md` without the word to lower the budget, rewrite `docs/TELEMETRY.md` lines 19-32 to state the scrape behavior and the labels, state the class count of 19 in `k8s/README.md`, and run `uv run devops docs generate --sync-readme` - Verification: `uv run devops ci test tests/prometheus/test_k8s_monitoring_alerting.py` passes and `uv run devops ci docs` passes +## 6. The trigger and chat counters start at zero + +Reads: `src/devops_cli/server/counter_series.py`, `src/devops_cli/server/routes/webhooks.py`, `src/devops_cli/server/service.py`. Changes: `src/devops_cli/config/constants.py`, `src/devops_cli/server/counter_series.py`, `src/devops_cli/server/routes/webhooks.py`, `src/devops_cli/docs/generator.py`, `docs/TELEMETRY.md`, `k8s/README.md`, `tests/telemetry/test_service_counter_series.py`. + +- [x] 6.1 Add tests to `tests/telemetry/test_service_counter_series.py` that assert that the trigger counter exists at 0 for each repository with the sources `webhook` and `poll`, and that the chat counter exists at 0 for each result `usage`, `status`, `unknown_repo` and `error` + - Verification: `uv run devops ci test tests/telemetry/test_service_counter_series.py` fails on the base +- [x] 6.2 Prime the trigger counter and the chat counter in `src/devops_cli/server/counter_series.py`, define the chat command constants in `src/devops_cli/config/constants.py`, and use them in the chat route of `src/devops_cli/server/routes/webhooks.py` + - Verification: `uv run devops ci test tests/telemetry/test_service_counter_series.py tests/server/test_mattermost_slash_command.py` passes +- [ ] 6.3 State in `src/devops_cli/docs/generator.py`, the source of `docs/TELEMETRY.md`, and in `k8s/README.md` that the identity counter is the one counter whose label set does not start at zero, and regenerate `docs/TELEMETRY.md` with `uv run devops docs generate --no-sync-readme` + - Verification: `uv run devops ci docs` passes + ## Workflow follow-up - Open item: agree the label `repo` of `devops_cli_service_implement_failures_total` with #3069 before this change lands. The issue says that #3069 reuses the label `class`. - Open item: check the live series with a Prometheus query after the deploy. This change runs no live query. +- Open item: a person decides whether `devops_cli_service_identity_refused_total` gets a zero label set. The design defers it, and no issue records the deferral yet. +- Open item: a round that fails before its first row counts no job, so `RoadmapServiceJobsFailing` may miss a failure at store level. A person decides whether to count it. - Pending a person: review the diff in tuicr (tuicr -r ..HEAD). diff --git a/src/devops_cli/config/constants.py b/src/devops_cli/config/constants.py index 11c5d7956..bd12f84a3 100644 --- a/src/devops_cli/config/constants.py +++ b/src/devops_cli/config/constants.py @@ -1573,6 +1573,13 @@ class WebhookExposure(StrEnum): CONST_SERVICE_METRIC_IMPLEMENT_FAILURES: Final[str] = "devops_cli_service_implement_failures_total" # Counts chat commands handled through the webhook router by `command` and `result` (#1395). CONST_SERVICE_METRIC_CHAT_COMMANDS: Final[str] = "devops_cli_service_chat_commands_total" +# The `command` label of the chat command that the webhook router counts, and its `result` values: +# each one exists at zero from the start, so the constants serve the route and its priming (#3495). +CONST_SERVICE_CHAT_COMMAND_ROADMAP: Final[str] = "/roadmap" +CONST_SERVICE_CHAT_RESULT_USAGE: Final[str] = "usage" +CONST_SERVICE_CHAT_RESULT_STATUS: Final[str] = "status" +CONST_SERVICE_CHAT_RESULT_UNKNOWN_REPO: Final[str] = "unknown_repo" +CONST_SERVICE_CHAT_RESULT_ERROR: Final[str] = "error" # The Service's window for its GitHub request budget (#2161). CONST_SERVICE_BUDGET_WINDOW_SECONDS: Final[int] = 3600 # What the Service keeps on its volume so that a restart does not clear it (#2474), each a file in diff --git a/src/devops_cli/server/counter_series.py b/src/devops_cli/server/counter_series.py index 50ba40ce4..419175050 100644 --- a/src/devops_cli/server/counter_series.py +++ b/src/devops_cli/server/counter_series.py @@ -10,12 +10,21 @@ from collections.abc import Iterable from devops_cli.config.constants import ( + CONST_SERVICE_CHAT_COMMAND_ROADMAP, + CONST_SERVICE_CHAT_RESULT_ERROR, + CONST_SERVICE_CHAT_RESULT_STATUS, + CONST_SERVICE_CHAT_RESULT_UNKNOWN_REPO, + CONST_SERVICE_CHAT_RESULT_USAGE, CONST_SERVICE_LANE_MODEL, CONST_SERVICE_LANE_STATUS, CONST_SERVICE_LANE_TRIAGE, + CONST_SERVICE_METRIC_CHAT_COMMANDS, CONST_SERVICE_METRIC_IMPLEMENT_FAILURES, CONST_SERVICE_METRIC_JOB_SECONDS, CONST_SERVICE_METRIC_JOBS, + CONST_SERVICE_METRIC_TRIGGERS, + CONST_SERVICE_SOURCE_POLL, + CONST_SERVICE_SOURCE_WEBHOOK, ) from devops_cli.roadmap.failure import FailureClass from devops_cli.telemetry.metrics import GLOBAL_METRICS @@ -23,11 +32,24 @@ SERVICE_JOB_RESULTS = ("success", "error") """The value of the label `result` of a job that ended, as the roadmap run counts it.""" +SERVICE_TRIGGER_SOURCES = (CONST_SERVICE_SOURCE_WEBHOOK, CONST_SERVICE_SOURCE_POLL) +"""The values of the label `source` of a trigger, as the Service enqueues it.""" + +SERVICE_CHAT_RESULTS = ( + CONST_SERVICE_CHAT_RESULT_USAGE, + CONST_SERVICE_CHAT_RESULT_STATUS, + CONST_SERVICE_CHAT_RESULT_UNKNOWN_REPO, + CONST_SERVICE_CHAT_RESULT_ERROR, +) +"""The values of the label `result` of a `/roadmap` chat command, as the webhook router counts +them.""" + def prime_service_counters(repos: Iterable[str]) -> None: """Create the bounded label sets of the Service counters at zero for the repositories `repos`: each failure class for each repository on the failure counter, each repository, - lane, row and result on the job counter, and each repository and lane on the job seconds.""" + lane, row and result on the job counter, each repository and lane on the job seconds, each + repository and source on the trigger counter, and each result of the chat command.""" # The roadmap run imports the Service, so the lane tables are read when the counters are # primed, not when this module is imported. from devops_cli.roadmap.run import MODEL_LANE_ROWS, STATUS_LANE_ROWS, TRIAGE_LANE_ROWS @@ -56,3 +78,18 @@ def prime_service_counters(repos: Iterable[str]) -> None: CONST_SERVICE_METRIC_JOB_SECONDS, ({"repo": repo, "lane": lane} for lane, _rows in lane_rows for repo in repo_names), ) + GLOBAL_METRICS.prime_counter( + CONST_SERVICE_METRIC_TRIGGERS, + ( + {"repo": repo, "source": source} + for repo in repo_names + for source in SERVICE_TRIGGER_SOURCES + ), + ) + GLOBAL_METRICS.prime_counter( + CONST_SERVICE_METRIC_CHAT_COMMANDS, + ( + {"command": CONST_SERVICE_CHAT_COMMAND_ROADMAP, "result": result} + for result in SERVICE_CHAT_RESULTS + ), + ) diff --git a/src/devops_cli/server/routes/webhooks.py b/src/devops_cli/server/routes/webhooks.py index c005d8d25..c84106f90 100644 --- a/src/devops_cli/server/routes/webhooks.py +++ b/src/devops_cli/server/routes/webhooks.py @@ -28,6 +28,11 @@ CONST_GH_WEBHOOK_SIGNATURE_PREFIX, CONST_MAX_ERROR_DETAIL_LENGTH, CONST_ROADMAP_WEBHOOK_EVENTS, + CONST_SERVICE_CHAT_COMMAND_ROADMAP, + CONST_SERVICE_CHAT_RESULT_ERROR, + CONST_SERVICE_CHAT_RESULT_STATUS, + CONST_SERVICE_CHAT_RESULT_UNKNOWN_REPO, + CONST_SERVICE_CHAT_RESULT_USAGE, CONST_SERVICE_DELIVERY_LRU_CAPACITY, CONST_SERVICE_EVENT_OTHER, CONST_SERVICE_MAX_BODY_BYTES, @@ -170,6 +175,14 @@ def _format_lane_status(status: Any) -> str: ) +def _count_chat_command(result: str) -> None: + """Count one `/roadmap` chat command under its `result` (#1395, #3495).""" + GLOBAL_METRICS.increment_counter( + CONST_SERVICE_METRIC_CHAT_COMMANDS, + labels={"command": CONST_SERVICE_CHAT_COMMAND_ROADMAP, "result": result}, + ) + + async def mattermost_roadmap_responder( request: Request, service_manager: ServiceManager, @@ -183,9 +196,7 @@ async def mattermost_roadmap_responder( try: args = shlex.split(raw_text) except ValueError: - GLOBAL_METRICS.increment_counter( - CONST_SERVICE_METRIC_CHAT_COMMANDS, labels={"command": "/roadmap", "result": "usage"} - ) + _count_chat_command(CONST_SERVICE_CHAT_RESULT_USAGE) return { "response_type": "ephemeral", "text": MESSAGES.serve.chat_roadmap_usage, @@ -195,20 +206,14 @@ async def mattermost_roadmap_responder( if not args or args == ["status"]: statuses = service_manager.lane_statuses() lines = [_format_lane_status(s) for s in statuses] - GLOBAL_METRICS.increment_counter( - CONST_SERVICE_METRIC_CHAT_COMMANDS, - labels={"command": "/roadmap", "result": "status"}, - ) + _count_chat_command(CONST_SERVICE_CHAT_RESULT_STATUS) return {"response_type": "ephemeral", "text": "\n".join(lines)} if len(args) == 2 and args[0] == "status": target_repo = args[1] if target_repo not in service_manager.managed_repos: known = ", ".join(service_manager.config.repos) - GLOBAL_METRICS.increment_counter( - CONST_SERVICE_METRIC_CHAT_COMMANDS, - labels={"command": "/roadmap", "result": "unknown_repo"}, - ) + _count_chat_command(CONST_SERVICE_CHAT_RESULT_UNKNOWN_REPO) return { "response_type": "ephemeral", "text": MESSAGES.serve.chat_roadmap_unknown_repo.format( @@ -218,24 +223,17 @@ async def mattermost_roadmap_responder( statuses = service_manager.lane_statuses(target_repo) lines = [_format_lane_status(s) for s in statuses] - GLOBAL_METRICS.increment_counter( - CONST_SERVICE_METRIC_CHAT_COMMANDS, - labels={"command": "/roadmap", "result": "status"}, - ) + _count_chat_command(CONST_SERVICE_CHAT_RESULT_STATUS) return {"response_type": "ephemeral", "text": "\n".join(lines)} - GLOBAL_METRICS.increment_counter( - CONST_SERVICE_METRIC_CHAT_COMMANDS, labels={"command": "/roadmap", "result": "usage"} - ) + _count_chat_command(CONST_SERVICE_CHAT_RESULT_USAGE) return { "response_type": "ephemeral", "text": MESSAGES.serve.chat_roadmap_usage, } except Exception as exc: logger.error("Error executing /roadmap chat command: %s", exc, exc_info=True) - GLOBAL_METRICS.increment_counter( - CONST_SERVICE_METRIC_CHAT_COMMANDS, labels={"command": "/roadmap", "result": "error"} - ) + _count_chat_command(CONST_SERVICE_CHAT_RESULT_ERROR) return { "response_type": "ephemeral", "text": f"Error executing roadmap command: {exc}", diff --git a/tests/telemetry/test_service_counter_series.py b/tests/telemetry/test_service_counter_series.py index f6137de25..dc476c1f6 100644 --- a/tests/telemetry/test_service_counter_series.py +++ b/tests/telemetry/test_service_counter_series.py @@ -12,14 +12,18 @@ from devops_cli.config.constants import ( CONST_ROADMAP_WEBHOOK_EVENTS, CONST_SERVICE_EVENT_OTHER, + CONST_SERVICE_METRIC_CHAT_COMMANDS, CONST_SERVICE_METRIC_IMPLEMENT_FAILURES, CONST_SERVICE_METRIC_JOBS, + CONST_SERVICE_METRIC_TRIGGERS, CONST_SERVICE_METRIC_WEBHOOK_DELIVERIES, CONST_SERVICE_OUTCOME_ACCEPTED, CONST_SERVICE_OUTCOME_IGNORED, CONST_SERVICE_OUTCOME_NO_SUBSCRIBER, CONST_SERVICE_OUTCOME_REJECTED, CONST_SERVICE_OUTCOME_SELF, + CONST_SERVICE_SOURCE_POLL, + CONST_SERVICE_SOURCE_WEBHOOK, ) from devops_cli.config.settings import GitHubAppConfig, ServiceConfig from devops_cli.roadmap.failure import FailureClass @@ -112,3 +116,28 @@ def test_webhook_counter_exists_at_zero_for_each_declared_event_and_outcome() -> } assert {key: series.get(key) for key in expected} == dict.fromkeys(expected, 0.0) + + +def test_trigger_counter_exists_at_zero_for_each_repository_and_source() -> None: + _manager(REPOS) + + series = _series(CONST_SERVICE_METRIC_TRIGGERS) + expected = { + frozenset({("repo", repo), ("source", source)}) + for repo in REPOS + for source in (CONST_SERVICE_SOURCE_WEBHOOK, CONST_SERVICE_SOURCE_POLL) + } + + assert {key: series.get(key) for key in expected} == dict.fromkeys(expected, 0.0) + + +def test_chat_command_counter_exists_at_zero_for_each_result() -> None: + _manager(REPOS) + + series = _series(CONST_SERVICE_METRIC_CHAT_COMMANDS) + expected = { + frozenset({("command", "/roadmap"), ("result", result)}) + for result in ("usage", "status", "unknown_repo", "error") + } + + assert {key: series.get(key) for key in expected} == dict.fromkeys(expected, 0.0) From ebfd5ee52afc3d9313002d811a16a2bd74de7b04 Mon Sep 17 00:00:00 2001 From: Daniel Petty Date: Thu, 5 Nov 2026 07:54:06 +0000 Subject: [PATCH 10/17] docs(telemetry): the identity counter is the one that does not start at zero The generator text of docs/TELEMETRY.md and the k8s README said that each counter of the Service starts its label sets at zero. The identity counter does not, so both texts now name it as the exception. The telemetry page is regenerated from the generator. --- docs/TELEMETRY.md | 14 +++++++++----- k8s/README.md | 2 +- .../tasks.md | 2 +- src/devops_cli/docs/generator.py | 14 +++++++++----- 4 files changed, 20 insertions(+), 12 deletions(-) diff --git a/docs/TELEMETRY.md b/docs/TELEMETRY.md index 504a43500..611136de5 100644 --- a/docs/TELEMETRY.md +++ b/docs/TELEMETRY.md @@ -31,11 +31,15 @@ To calculate these counters, the collector transmits created timestamps via remo `enable-feature: created-timestamp-zero-ingestion` in `k8s/monitoring/prometheus-values.yaml`. On this path, Prometheus writes a zero-value sample at the start timestamp of each series, so `increase()` calculates the full counter value. The `/metrics` scrape gets no such sample. -The Service starts each label set of its counters at zero (#3495), so the first count of a -label set is in its series. A count that a pod makes before its first scrape is still lost, -because `increase()` does not count the first sample of a series. The labels of the Service -counters are: `devops_cli_service_implement_failures_total` with `class` and `repo`, and -`devops_cli_service_jobs_total` with `repo`, `lane`, `row` and `result`. + +The Service starts each label set of its counters at zero (#3495). So the first count +of a label set is in its series. The exception is +`devops_cli_service_identity_refused_total`: its label set starts with the +first refusal. A pod loses a count that it makes before its first scrape, because +`increase()` does not count the first sample of a series. The labels of +`devops_cli_service_implement_failures_total` are `class` and `repo`. The labels of +`devops_cli_service_jobs_total` are `repo`, `lane`, `row` and `result`. + A gauge holds a snapshot, such as a project metric that the roadmap Service sends every 15 minutes. Prometheus finds a plain selector only within five minutes of a sample, so a dashboard reads a gauge with `last_over_time([$__range])`, and diff --git a/k8s/README.md b/k8s/README.md index cebc4924f..1d8a72246 100644 --- a/k8s/README.md +++ b/k8s/README.md @@ -627,7 +627,7 @@ kubectl -n devops logs -f job/ 9. To rotate a credential, update it in the keyring: `uv run devops config set service.webhook_secrets`, or, for the App, generate a new private key and run `uv run devops config set service.github_app.private_key --from-file `. Then run `uv run devops k8s push-secrets --only devops/devops-cli --rotate`. It restarts `roadmap-service`. Delete the old private key on the App's page after the restart. 10. Run one service per set of repos. While it runs, use `devops roadmap run --dry-run` (#981). 11. The Service keeps three files in `/home/devops/.data/roadmap/` on volume `roadmap-service-home`, beside the `schedule.json` of each repository: `service-halt.json` (GitHub refused the identity), `service-budget.json` (the GitHub requests of the last hour) and `service-failures.json` (the jobs that failed rounds in a row). A restart reads them back, so a restart does not clear a halt, the budget or a suspended job. A round that ends because the Service is stopping, for example in a rollout, does not count as a failed round: the count of its job stays as it was. The log line that suspends a job names the class of the error of each of its failed rounds in a row; the file keeps only the count, so a round that an earlier process counted reads as `unknown`. A rate limit that one request gets stops the requests of all workers until its wait ends. When GitHub refuses the identity, for example because the App's installation is suspended, the Service halts and logs `GitHub refused the identity`. To clear the halt after you fix the cause, remove the file and restart the Service: `kubectl -n devops exec deploy/roadmap-service -- rm /home/devops/.data/roadmap/service-halt.json`, then `kubectl -n devops delete pod -l app.kubernetes.io/name=roadmap-service`. -12. The Service's `/metrics` shows the budget and the suspended jobs (#2475): `devops_cli_service_budget_limit_requests` (the hourly budget), `devops_cli_service_budget_spent_requests` (the GitHub requests of the last hour), `devops_cli_service_budget_holds_total` (one count each time a hold begins), `devops_cli_service_job_suspended` (1 while a job of a repository, by `repo` and `row`, waits for new input after it failed 3 rounds in a row) and `devops_cli_service_implement_failures_total` (one count for each runner Job that left a stop, by the failure `class` and `repo`, when a round reports the Job; #3065; the 19 values of `class` are the classes in [docs/ROADMAP_WORKFLOW.md](../docs/ROADMAP_WORKFLOW.md), The failure classes, and a panel or rule matches no other value, #3476). Each counter of the Service starts its label sets at zero (#3495, [docs/TELEMETRY.md](../docs/TELEMETRY.md)). Each GitHub request counts in `devops_cli_github_requests_total` under its `method`, the `repo` of the worker that sent it and the job `row` (`metrics`, `intake`, `refine`, `stage`, `implement`, `stage_ready` or `alternatives`); a request outside a worker or a row counts as `other`. To find what spent the budget, run `sum by (repo, row, method) (increase(devops_cli_github_requests_total{job="roadmap-service"}[1h]))` in Grafana Explore. A delivery of the App bot's own write counts as outcome `self` in `devops_cli_service_webhook_deliveries_total`. +12. The Service's `/metrics` shows the budget and the suspended jobs (#2475): `devops_cli_service_budget_limit_requests` (the hourly budget), `devops_cli_service_budget_spent_requests` (the GitHub requests of the last hour), `devops_cli_service_budget_holds_total` (one count each time a hold begins), `devops_cli_service_job_suspended` (1 while a job of a repository, by `repo` and `row`, waits for new input after it failed 3 rounds in a row) and `devops_cli_service_implement_failures_total` (one count for each runner Job that left a stop, by the failure `class` and `repo`, when a round reports the Job; #3065; the 19 values of `class` are the classes in [docs/ROADMAP_WORKFLOW.md](../docs/ROADMAP_WORKFLOW.md), The failure classes, and a panel or rule matches no other value, #3476). Each counter of the Service starts its label sets at zero (#3495, [docs/TELEMETRY.md](../docs/TELEMETRY.md)), except `devops_cli_service_identity_refused_total`, whose label set starts with the first refusal. Each GitHub request counts in `devops_cli_github_requests_total` under its `method`, the `repo` of the worker that sent it and the job `row` (`metrics`, `intake`, `refine`, `stage`, `implement`, `stage_ready` or `alternatives`); a request outside a worker or a row counts as `other`. To find what spent the budget, run `sum by (repo, row, method) (increase(devops_cli_github_requests_total{job="roadmap-service"}[1h]))` in Grafana Explore. A delivery of the App bot's own write counts as outcome `self` in `devops_cli_service_webhook_deliveries_total`. 13. A refine round writes at most 2 comments on one item in 24 hours. A round that finds 2 refine comments of the last 24 hours on an item leaves the item alone, with no model call and no comment. The item waits on a person and takes no place under the round's limit, so the round refines the next item. For each newest comment on the item that the cap leaves with no answer, the round records the time of that comment in the item's job record (`refine.capped_after`), logs `Refine left # alone and wrote no comment`, and adds 1 to `devops_cli_roadmap_refine_comments_capped_total`. A later round that finds no newer comment does not log or count the item again. A person refines the item at the cap with `devops roadmap refine --item --confirm`. Each Firecrawl search counts in `devops_cli_firecrawl_searches_total` by `result`: `ok`, the HTTP status of a refused search, or `error`. A failed search logs `A Firecrawl web search failed` with the status or the error class, and never the key or the query. 14. The row `alternatives` of the model lane runs after `refine`: on a first run, and 719 minutes after its last success. It runs `devops roadmap alternatives --confirm` for the repository. A round checks at most `roadmap.alternatives.first_pass_limit` items (20 by default) while an item has no check, and at most `roadmap.alternatives.limit` items (5 by default) after. The row runs about twice a day, so a first pass of 141 items takes about 4 days. A round takes first the items with no check, then the items whose check is older than `roadmap.alternatives.recheck_days` (30 by default) or whose text changed. A webhook makes the row due at no other time. The row changes no label and closes no issue. It does nothing, and logs `The alternatives check of is off`, for a private repository and when no Firecrawl key is set. If the check of an item fails, the row records the failure in the item's job record (`alternatives.failed_at` and `alternatives.failed_error`, the class of the error), writes the other checks and succeeds. The row fails, and counts toward the suspension after 3 failed rounds, only when it tried items and checked none. A round takes a failed item again after `roadmap.alternatives.retry_days` (3 by default). Each written check adds 1 to `devops_cli_roadmap_alternatives_total` by `verdict`: `covers`, `partly`, `none` or `not-applicable`. An item of the type `test`, `docs`, `refactor`, `chore`, `ci` or `perf`, and a `fix` with a Python traceback in its body, gets `not-applicable` with no model call; every other item must run a search before the check accepts a verdict, and gets one more turn to search. A `covers` verdict makes refine ask a person to adopt the tool or to keep the item with a reason. The item waits on a person until a reply answers the question, and Ready items with a new covering tool go back to New. Refine records the tool it asked about in the job record (`refine.alternatives_asked`). `devops roadmap run --dry-run` lists the row when it is due. 15. The row `stage` follows `refine` on the model lane (#2659). It runs `devops roadmap stage --confirm` in the model clone: it is due every 37 minutes, on a first run, and in a round where `refine` set an item Ready. It stages the Ready items that `roadmap.stage.filter` names, at most `roadmap.stage.limit` in a round, plans no new item while the count of open Planned items is at `roadmap.stage.planned_limit` (an item with the priority `P0-Critical` passes it, #3487), and splits a task group over `roadmap.stage.step_tokens`. Each task group has a kind (#3005): a `behavior` group has a failing test, and an `other` group has no test but a command or the steps of a manual check that prove it. The job sends a reply that breaks this rule back to the model once. It leaves out an item whose change folder is on the default branch or in its archive (#2714): a round reads that tree once with `git ls-tree`, so a merged staging pull request never makes a second one. The pull request it opens is a draft with the title `docs(openspec): plan # `. The job does not mark it ready: the row `stage_ready` does (step 18). Before it opens one, it checks each path of the plan (`reads`, `changes`, the code spans of the tasks and `verify`) against the tracked files of the commit it plans, read with one `git ls-files` for the round (#3010). A path that is no file there, and that no group up to its own creates, goes back to the model once with the files of the same name. When a later group creates the path, the problem names that group and tells the model to move it before the group that names the path (#3017). A second reply with a missing path stops the item, so no draft opens and the implement Job never meets a wrong path. `k8s/devops/configmap.example.yaml` holds the five settings, and `devops k8s deploy-stack --stack devops` applies them; each also has a variable, `DEVOPS_CLI_ROADMAP_STAGE_FILTER`, `DEVOPS_CLI_ROADMAP_STAGE_LIMIT`, `DEVOPS_CLI_ROADMAP_STAGE_PLANNED_LIMIT`, `DEVOPS_CLI_ROADMAP_STAGE_STEP_TOKENS` and `DEVOPS_CLI_ROADMAP_STAGE_TOOLSET`. The setting `roadmap.stage.toolset` is `extended` by default (#3238): a plan run then has three tools more than the nine that read the staged commit. `references` counts a text in every file of the commit. `read_issue` and `find_issues` read issues and pull requests of the repository on GitHub, as GET requests with the token of the Service and no other permission. `base` leaves them out. Each item of a round counts once in `devops_cli_roadmap_stage_total` by its `result`: `staged`, `skipped`, `needs_split` or `failed`. A failed item is recorded in its job record with the class of its error and the time (#3493), and the job leaves it out for `roadmap.stage.retry_hours` hours (6 by default) while it takes the other items. The row fails a round only when it tried items and staged, stopped or returned none, or when a failure belongs to no item; three such rounds on the same input suspend the row. The ledger file keeps the class of each error, and a round that the stop of the Service ends counts as `stopped`, not as an error. The image carries the pinned OpenSpec and a Node runtime, so the job validates each change before it pushes: `kubectl -n devops exec deploy/roadmap-service -- openspec --version` prints the locked version. diff --git a/openspec/changes/3495-service-counters-zero-series-repo-label/tasks.md b/openspec/changes/3495-service-counters-zero-series-repo-label/tasks.md index 324cf7ade..a380f779c 100644 --- a/openspec/changes/3495-service-counters-zero-series-repo-label/tasks.md +++ b/openspec/changes/3495-service-counters-zero-series-repo-label/tasks.md @@ -59,7 +59,7 @@ Reads: `src/devops_cli/server/counter_series.py`, `src/devops_cli/server/routes/ - Verification: `uv run devops ci test tests/telemetry/test_service_counter_series.py` fails on the base - [x] 6.2 Prime the trigger counter and the chat counter in `src/devops_cli/server/counter_series.py`, define the chat command constants in `src/devops_cli/config/constants.py`, and use them in the chat route of `src/devops_cli/server/routes/webhooks.py` - Verification: `uv run devops ci test tests/telemetry/test_service_counter_series.py tests/server/test_mattermost_slash_command.py` passes -- [ ] 6.3 State in `src/devops_cli/docs/generator.py`, the source of `docs/TELEMETRY.md`, and in `k8s/README.md` that the identity counter is the one counter whose label set does not start at zero, and regenerate `docs/TELEMETRY.md` with `uv run devops docs generate --no-sync-readme` +- [x] 6.3 State in `src/devops_cli/docs/generator.py`, the source of `docs/TELEMETRY.md`, and in `k8s/README.md` that the identity counter is the one counter whose label set does not start at zero, and regenerate `docs/TELEMETRY.md` with `uv run devops docs generate --no-sync-readme` - Verification: `uv run devops ci docs` passes ## Workflow follow-up diff --git a/src/devops_cli/docs/generator.py b/src/devops_cli/docs/generator.py index 1bef3eb20..3828ac1d4 100644 --- a/src/devops_cli/docs/generator.py +++ b/src/devops_cli/docs/generator.py @@ -1108,11 +1108,15 @@ def generate_telemetry_docs(self) -> str: "`enable-feature: created-timestamp-zero-ingestion` in `k8s/monitoring/prometheus-values.yaml`.", "On this path, Prometheus writes a zero-value sample at the start timestamp of each series,", "so `increase()` calculates the full counter value. The `/metrics` scrape gets no such sample.", - "The Service starts each label set of its counters at zero (#3495), so the first count of a", - "label set is in its series. A count that a pod makes before its first scrape is still lost,", - "because `increase()` does not count the first sample of a series. The labels of the Service", - "counters are: `devops_cli_service_implement_failures_total` with `class` and `repo`, and", - "`devops_cli_service_jobs_total` with `repo`, `lane`, `row` and `result`.", + "", + "The Service starts each label set of its counters at zero (#3495). So the first count", + "of a label set is in its series. The exception is", + "`devops_cli_service_identity_refused_total`: its label set starts with the", + "first refusal. A pod loses a count that it makes before its first scrape, because", + "`increase()` does not count the first sample of a series. The labels of", + "`devops_cli_service_implement_failures_total` are `class` and `repo`. The labels of", + "`devops_cli_service_jobs_total` are `repo`, `lane`, `row` and `result`.", + "", "A gauge holds a snapshot, such as a project metric that the roadmap Service sends", "every 15 minutes. Prometheus finds a plain selector only within five minutes of a", "sample, so a dashboard reads a gauge with `last_over_time([$__range])`, and", From a47ac211f206e906d510080f61c26f99f5723cae Mon Sep 17 00:00:00 2001 From: Daniel Petty Date: Thu, 5 Nov 2026 07:59:44 +0000 Subject: [PATCH 11/17] docs(openspec): archive #3495 service counter series The change folder moves to the archive, and its requirements merge into the main spec of roadmap-service-counters. The main spec has a Purpose that states the scrape behavior. --- .../.openspec.yaml | 0 .../design.md | 0 .../proposal.md | 0 .../specs/roadmap-implement-job/spec.md | 0 .../specs/roadmap-service-counters/spec.md | 0 .../tasks.md | 0 openspec/specs/roadmap-implement-job/spec.md | 10 ++-- .../specs/roadmap-service-counters/spec.md | 53 +++++++++++++++++++ 8 files changed, 58 insertions(+), 5 deletions(-) rename openspec/changes/{3495-service-counters-zero-series-repo-label => archive/2026-11-05-3495-service-counters-zero-series-repo-label}/.openspec.yaml (100%) rename openspec/changes/{3495-service-counters-zero-series-repo-label => archive/2026-11-05-3495-service-counters-zero-series-repo-label}/design.md (100%) rename openspec/changes/{3495-service-counters-zero-series-repo-label => archive/2026-11-05-3495-service-counters-zero-series-repo-label}/proposal.md (100%) rename openspec/changes/{3495-service-counters-zero-series-repo-label => archive/2026-11-05-3495-service-counters-zero-series-repo-label}/specs/roadmap-implement-job/spec.md (100%) rename openspec/changes/{3495-service-counters-zero-series-repo-label => archive/2026-11-05-3495-service-counters-zero-series-repo-label}/specs/roadmap-service-counters/spec.md (100%) rename openspec/changes/{3495-service-counters-zero-series-repo-label => archive/2026-11-05-3495-service-counters-zero-series-repo-label}/tasks.md (100%) create mode 100644 openspec/specs/roadmap-service-counters/spec.md diff --git a/openspec/changes/3495-service-counters-zero-series-repo-label/.openspec.yaml b/openspec/changes/archive/2026-11-05-3495-service-counters-zero-series-repo-label/.openspec.yaml similarity index 100% rename from openspec/changes/3495-service-counters-zero-series-repo-label/.openspec.yaml rename to openspec/changes/archive/2026-11-05-3495-service-counters-zero-series-repo-label/.openspec.yaml diff --git a/openspec/changes/3495-service-counters-zero-series-repo-label/design.md b/openspec/changes/archive/2026-11-05-3495-service-counters-zero-series-repo-label/design.md similarity index 100% rename from openspec/changes/3495-service-counters-zero-series-repo-label/design.md rename to openspec/changes/archive/2026-11-05-3495-service-counters-zero-series-repo-label/design.md diff --git a/openspec/changes/3495-service-counters-zero-series-repo-label/proposal.md b/openspec/changes/archive/2026-11-05-3495-service-counters-zero-series-repo-label/proposal.md similarity index 100% rename from openspec/changes/3495-service-counters-zero-series-repo-label/proposal.md rename to openspec/changes/archive/2026-11-05-3495-service-counters-zero-series-repo-label/proposal.md diff --git a/openspec/changes/3495-service-counters-zero-series-repo-label/specs/roadmap-implement-job/spec.md b/openspec/changes/archive/2026-11-05-3495-service-counters-zero-series-repo-label/specs/roadmap-implement-job/spec.md similarity index 100% rename from openspec/changes/3495-service-counters-zero-series-repo-label/specs/roadmap-implement-job/spec.md rename to openspec/changes/archive/2026-11-05-3495-service-counters-zero-series-repo-label/specs/roadmap-implement-job/spec.md diff --git a/openspec/changes/3495-service-counters-zero-series-repo-label/specs/roadmap-service-counters/spec.md b/openspec/changes/archive/2026-11-05-3495-service-counters-zero-series-repo-label/specs/roadmap-service-counters/spec.md similarity index 100% rename from openspec/changes/3495-service-counters-zero-series-repo-label/specs/roadmap-service-counters/spec.md rename to openspec/changes/archive/2026-11-05-3495-service-counters-zero-series-repo-label/specs/roadmap-service-counters/spec.md diff --git a/openspec/changes/3495-service-counters-zero-series-repo-label/tasks.md b/openspec/changes/archive/2026-11-05-3495-service-counters-zero-series-repo-label/tasks.md similarity index 100% rename from openspec/changes/3495-service-counters-zero-series-repo-label/tasks.md rename to openspec/changes/archive/2026-11-05-3495-service-counters-zero-series-repo-label/tasks.md diff --git a/openspec/specs/roadmap-implement-job/spec.md b/openspec/specs/roadmap-implement-job/spec.md index 9e5b3b0de..51d4476b3 100644 --- a/openspec/specs/roadmap-implement-job/spec.md +++ b/openspec/specs/roadmap-implement-job/spec.md @@ -771,15 +771,15 @@ The stop comment of the job SHALL name the failure class, the action taken and t - **THEN** `implement.failure` is removed with `implement.stopped` ### Requirement: The Service counts the failures by class -When a round reports a Job that left a stop, the Service SHALL add 1 to `devops_cli_service_implement_failures_total` with the label `class`. A Job that left the item without a result SHALL get the class of the tail of its log, recorded in `implement.failure` and named, with the action left, in the report. A Job that ended with a result SHALL add nothing. +When a round reports a Job that left a stop, the Service SHALL add 1 to `devops_cli_service_implement_failures_total` with the labels `class` and `repo`, where `repo` is the repository of the round. A Job that left the item without a result SHALL get the class of the tail of its log, recorded in `implement.failure` and named, with the action left, in the report. A Job that ended with a result SHALL add nothing. #### Scenario: A stop of the runner -- **WHEN** a round reports a Job whose item holds the stop with the class `parser` -- **THEN** the counter for `parser` grows by 1, the counter for `oom` does not change, and a second round adds nothing +- **WHEN** a round of `example/app` reports a Job whose item holds the stop with the class `parser` +- **THEN** the counter for `parser` and `example/app` grows by 1, the counter for `oom` does not change, and a second round adds nothing #### Scenario: A Job with no result -- **WHEN** a round reports a Job that failed and left the item In Progress, and its log holds "out of memory" -- **THEN** the item is Blocked with `implement.failure` `oom`, the report names the class and the action left `raise_limit`, and the counter for `oom` grows by 1 +- **WHEN** a round of `example/app` reports a Job that failed and left the item In Progress, and its log holds "out of memory" +- **THEN** the item is Blocked with `implement.failure` `oom`, the report names the class and the action left `raise_limit`, and the counter for `oom` and `example/app` grows by 1 #### Scenario: A Job with a result - **WHEN** a round reports a Job that succeeded and left the item In Review diff --git a/openspec/specs/roadmap-service-counters/spec.md b/openspec/specs/roadmap-service-counters/spec.md new file mode 100644 index 000000000..fb11437bb --- /dev/null +++ b/openspec/specs/roadmap-service-counters/spec.md @@ -0,0 +1,53 @@ +# roadmap-service-counters Specification + +## Purpose +The label sets of the Service counters that the Service creates at zero when it starts, the labels of the job counter and the failure counter, and the promtool cases that prove the alerts and the true total over a pod change. A count that a pod makes before its first scrape is still lost to increase(). + +## Requirements + +### Requirement: The Service starts each label set of its counters at zero +The Service SHALL create each bounded label set of its counters at zero when it starts. The bounded label sets are: each failure class for each repository of `service.repos` on `devops_cli_service_implement_failures_total`; each repository, lane, row and result on `devops_cli_service_jobs_total`; each repository and lane on `devops_cli_service_job_seconds_total`; each provider, declared event or `other`, and outcome on `devops_cli_service_webhook_deliveries_total`. + +#### Scenario: The failure counter exists at zero after start +- **WHEN** the Service starts with the repositories `example/app` and `example/tools` +- **THEN** the failure counter exists at 0 for each of the 19 failure classes and each of the two repositories + +#### Scenario: The label repo holds only configured repositories +- **WHEN** the Service starts with the repositories `example/app` and `example/tools` +- **THEN** every value of the label `repo` of the failure counter is `example/app` or `example/tools` + +#### Scenario: The webhook counter exists at zero for each declared event +- **WHEN** the Service starts with a provider that declares the events `issues` and `push` +- **THEN** the webhook counter exists at 0 for each of the outcomes with the events `issues`, `push` and `other` + +### Requirement: The Service starts the trigger and chat counters at zero +The Service SHALL create each bounded label set of `devops_cli_service_triggers_total` and `devops_cli_service_chat_commands_total` at zero when it starts. The bounded label sets are: each repository of `service.repos` with each source `webhook` or `poll`; and each result `usage`, `status`, `unknown_repo` or `error` of the `/roadmap` chat command. + +#### Scenario: The trigger and chat counters exist at zero after start +- **WHEN** the Service starts with the repositories `example/app` and `example/tools` +- **THEN** the trigger counter exists at 0 for each repository with the sources `webhook` and `poll`, and the chat counter exists at 0 for each result `usage`, `status`, `unknown_repo` and `error` + +### Requirement: The job counter carries the row of the job +The Service SHALL add 1 to `devops_cli_service_jobs_total` with the labels `repo`, `lane`, `row` and `result` for each row of a lane that ends in the roadmap run. A scoped update SHALL add 1 with the name of its job as the label `row`. The Service SHALL NOT count a lane batch as one job. + +#### Scenario: A row that succeeds +- **WHEN** the status lane runs the row `metrics` for `example/app` and the row succeeds +- **THEN** the job counter for `repo` `example/app`, `lane` `status`, `row` `metrics` and `result` `success` grows by 1 + +#### Scenario: Two rows of one batch +- **WHEN** one round of the status lane runs the rows `metrics` and `implement`, and both fail +- **THEN** the job counter for each row with `result` `error` grows by 1, and no series counts the batch + +### Requirement: Alerts read a counter that starts at zero with increase +The alerts `RoadmapServiceWebhooksRejected` and `RoadmapServiceSelfDeliveriesHigh` SHALL read their counter with `increase()` and no `offset` term, because the Service starts each label set at zero. + +#### Scenario: A rejected burst on a new pod +- **WHEN** `promtool test rules` evaluates the rejected series of a pod that starts at 0 and reaches 25 within 15 minutes +- **THEN** `RoadmapServiceWebhooksRejected` fires + +### Requirement: A query gives the true total over a pod change +A query of `sum(increase(devops_cli_service_implement_failures_total[24h]))` SHALL give the true total of the counts of all pods in the window, including the counts of a pod that started during the window. + +#### Scenario: A new pod adds its counts +- **WHEN** `promtool test rules` evaluates the failure counter of a pod that counted 3 and of a new pod that counted 2 after a zero sample at its start +- **THEN** the sum of `increase()` over the window is 5 From 64d84676751ace846be6bc9ff04b8f582fd65d78 Mon Sep 17 00:00:00 2001 From: Daniel Petty Date: Thu, 5 Nov 2026 16:32:31 +0000 Subject: [PATCH 12/17] Revert "docs(openspec): archive #3495 service counter series" This reverts commit 58c06b28c22a41a0203f4c2c8a1089b8711bd814. --- .../.openspec.yaml | 0 .../design.md | 0 .../proposal.md | 0 .../specs/roadmap-implement-job/spec.md | 0 .../specs/roadmap-service-counters/spec.md | 0 .../tasks.md | 0 openspec/specs/roadmap-implement-job/spec.md | 10 ++-- .../specs/roadmap-service-counters/spec.md | 53 ------------------- 8 files changed, 5 insertions(+), 58 deletions(-) rename openspec/changes/{archive/2026-11-05-3495-service-counters-zero-series-repo-label => 3495-service-counters-zero-series-repo-label}/.openspec.yaml (100%) rename openspec/changes/{archive/2026-11-05-3495-service-counters-zero-series-repo-label => 3495-service-counters-zero-series-repo-label}/design.md (100%) rename openspec/changes/{archive/2026-11-05-3495-service-counters-zero-series-repo-label => 3495-service-counters-zero-series-repo-label}/proposal.md (100%) rename openspec/changes/{archive/2026-11-05-3495-service-counters-zero-series-repo-label => 3495-service-counters-zero-series-repo-label}/specs/roadmap-implement-job/spec.md (100%) rename openspec/changes/{archive/2026-11-05-3495-service-counters-zero-series-repo-label => 3495-service-counters-zero-series-repo-label}/specs/roadmap-service-counters/spec.md (100%) rename openspec/changes/{archive/2026-11-05-3495-service-counters-zero-series-repo-label => 3495-service-counters-zero-series-repo-label}/tasks.md (100%) delete mode 100644 openspec/specs/roadmap-service-counters/spec.md diff --git a/openspec/changes/archive/2026-11-05-3495-service-counters-zero-series-repo-label/.openspec.yaml b/openspec/changes/3495-service-counters-zero-series-repo-label/.openspec.yaml similarity index 100% rename from openspec/changes/archive/2026-11-05-3495-service-counters-zero-series-repo-label/.openspec.yaml rename to openspec/changes/3495-service-counters-zero-series-repo-label/.openspec.yaml diff --git a/openspec/changes/archive/2026-11-05-3495-service-counters-zero-series-repo-label/design.md b/openspec/changes/3495-service-counters-zero-series-repo-label/design.md similarity index 100% rename from openspec/changes/archive/2026-11-05-3495-service-counters-zero-series-repo-label/design.md rename to openspec/changes/3495-service-counters-zero-series-repo-label/design.md diff --git a/openspec/changes/archive/2026-11-05-3495-service-counters-zero-series-repo-label/proposal.md b/openspec/changes/3495-service-counters-zero-series-repo-label/proposal.md similarity index 100% rename from openspec/changes/archive/2026-11-05-3495-service-counters-zero-series-repo-label/proposal.md rename to openspec/changes/3495-service-counters-zero-series-repo-label/proposal.md diff --git a/openspec/changes/archive/2026-11-05-3495-service-counters-zero-series-repo-label/specs/roadmap-implement-job/spec.md b/openspec/changes/3495-service-counters-zero-series-repo-label/specs/roadmap-implement-job/spec.md similarity index 100% rename from openspec/changes/archive/2026-11-05-3495-service-counters-zero-series-repo-label/specs/roadmap-implement-job/spec.md rename to openspec/changes/3495-service-counters-zero-series-repo-label/specs/roadmap-implement-job/spec.md diff --git a/openspec/changes/archive/2026-11-05-3495-service-counters-zero-series-repo-label/specs/roadmap-service-counters/spec.md b/openspec/changes/3495-service-counters-zero-series-repo-label/specs/roadmap-service-counters/spec.md similarity index 100% rename from openspec/changes/archive/2026-11-05-3495-service-counters-zero-series-repo-label/specs/roadmap-service-counters/spec.md rename to openspec/changes/3495-service-counters-zero-series-repo-label/specs/roadmap-service-counters/spec.md diff --git a/openspec/changes/archive/2026-11-05-3495-service-counters-zero-series-repo-label/tasks.md b/openspec/changes/3495-service-counters-zero-series-repo-label/tasks.md similarity index 100% rename from openspec/changes/archive/2026-11-05-3495-service-counters-zero-series-repo-label/tasks.md rename to openspec/changes/3495-service-counters-zero-series-repo-label/tasks.md diff --git a/openspec/specs/roadmap-implement-job/spec.md b/openspec/specs/roadmap-implement-job/spec.md index 51d4476b3..9e5b3b0de 100644 --- a/openspec/specs/roadmap-implement-job/spec.md +++ b/openspec/specs/roadmap-implement-job/spec.md @@ -771,15 +771,15 @@ The stop comment of the job SHALL name the failure class, the action taken and t - **THEN** `implement.failure` is removed with `implement.stopped` ### Requirement: The Service counts the failures by class -When a round reports a Job that left a stop, the Service SHALL add 1 to `devops_cli_service_implement_failures_total` with the labels `class` and `repo`, where `repo` is the repository of the round. A Job that left the item without a result SHALL get the class of the tail of its log, recorded in `implement.failure` and named, with the action left, in the report. A Job that ended with a result SHALL add nothing. +When a round reports a Job that left a stop, the Service SHALL add 1 to `devops_cli_service_implement_failures_total` with the label `class`. A Job that left the item without a result SHALL get the class of the tail of its log, recorded in `implement.failure` and named, with the action left, in the report. A Job that ended with a result SHALL add nothing. #### Scenario: A stop of the runner -- **WHEN** a round of `example/app` reports a Job whose item holds the stop with the class `parser` -- **THEN** the counter for `parser` and `example/app` grows by 1, the counter for `oom` does not change, and a second round adds nothing +- **WHEN** a round reports a Job whose item holds the stop with the class `parser` +- **THEN** the counter for `parser` grows by 1, the counter for `oom` does not change, and a second round adds nothing #### Scenario: A Job with no result -- **WHEN** a round of `example/app` reports a Job that failed and left the item In Progress, and its log holds "out of memory" -- **THEN** the item is Blocked with `implement.failure` `oom`, the report names the class and the action left `raise_limit`, and the counter for `oom` and `example/app` grows by 1 +- **WHEN** a round reports a Job that failed and left the item In Progress, and its log holds "out of memory" +- **THEN** the item is Blocked with `implement.failure` `oom`, the report names the class and the action left `raise_limit`, and the counter for `oom` grows by 1 #### Scenario: A Job with a result - **WHEN** a round reports a Job that succeeded and left the item In Review diff --git a/openspec/specs/roadmap-service-counters/spec.md b/openspec/specs/roadmap-service-counters/spec.md deleted file mode 100644 index fb11437bb..000000000 --- a/openspec/specs/roadmap-service-counters/spec.md +++ /dev/null @@ -1,53 +0,0 @@ -# roadmap-service-counters Specification - -## Purpose -The label sets of the Service counters that the Service creates at zero when it starts, the labels of the job counter and the failure counter, and the promtool cases that prove the alerts and the true total over a pod change. A count that a pod makes before its first scrape is still lost to increase(). - -## Requirements - -### Requirement: The Service starts each label set of its counters at zero -The Service SHALL create each bounded label set of its counters at zero when it starts. The bounded label sets are: each failure class for each repository of `service.repos` on `devops_cli_service_implement_failures_total`; each repository, lane, row and result on `devops_cli_service_jobs_total`; each repository and lane on `devops_cli_service_job_seconds_total`; each provider, declared event or `other`, and outcome on `devops_cli_service_webhook_deliveries_total`. - -#### Scenario: The failure counter exists at zero after start -- **WHEN** the Service starts with the repositories `example/app` and `example/tools` -- **THEN** the failure counter exists at 0 for each of the 19 failure classes and each of the two repositories - -#### Scenario: The label repo holds only configured repositories -- **WHEN** the Service starts with the repositories `example/app` and `example/tools` -- **THEN** every value of the label `repo` of the failure counter is `example/app` or `example/tools` - -#### Scenario: The webhook counter exists at zero for each declared event -- **WHEN** the Service starts with a provider that declares the events `issues` and `push` -- **THEN** the webhook counter exists at 0 for each of the outcomes with the events `issues`, `push` and `other` - -### Requirement: The Service starts the trigger and chat counters at zero -The Service SHALL create each bounded label set of `devops_cli_service_triggers_total` and `devops_cli_service_chat_commands_total` at zero when it starts. The bounded label sets are: each repository of `service.repos` with each source `webhook` or `poll`; and each result `usage`, `status`, `unknown_repo` or `error` of the `/roadmap` chat command. - -#### Scenario: The trigger and chat counters exist at zero after start -- **WHEN** the Service starts with the repositories `example/app` and `example/tools` -- **THEN** the trigger counter exists at 0 for each repository with the sources `webhook` and `poll`, and the chat counter exists at 0 for each result `usage`, `status`, `unknown_repo` and `error` - -### Requirement: The job counter carries the row of the job -The Service SHALL add 1 to `devops_cli_service_jobs_total` with the labels `repo`, `lane`, `row` and `result` for each row of a lane that ends in the roadmap run. A scoped update SHALL add 1 with the name of its job as the label `row`. The Service SHALL NOT count a lane batch as one job. - -#### Scenario: A row that succeeds -- **WHEN** the status lane runs the row `metrics` for `example/app` and the row succeeds -- **THEN** the job counter for `repo` `example/app`, `lane` `status`, `row` `metrics` and `result` `success` grows by 1 - -#### Scenario: Two rows of one batch -- **WHEN** one round of the status lane runs the rows `metrics` and `implement`, and both fail -- **THEN** the job counter for each row with `result` `error` grows by 1, and no series counts the batch - -### Requirement: Alerts read a counter that starts at zero with increase -The alerts `RoadmapServiceWebhooksRejected` and `RoadmapServiceSelfDeliveriesHigh` SHALL read their counter with `increase()` and no `offset` term, because the Service starts each label set at zero. - -#### Scenario: A rejected burst on a new pod -- **WHEN** `promtool test rules` evaluates the rejected series of a pod that starts at 0 and reaches 25 within 15 minutes -- **THEN** `RoadmapServiceWebhooksRejected` fires - -### Requirement: A query gives the true total over a pod change -A query of `sum(increase(devops_cli_service_implement_failures_total[24h]))` SHALL give the true total of the counts of all pods in the window, including the counts of a pod that started during the window. - -#### Scenario: A new pod adds its counts -- **WHEN** `promtool test rules` evaluates the failure counter of a pod that counted 3 and of a new pod that counted 2 after a zero sample at its start -- **THEN** the sum of `increase()` over the window is 5 From 632fdfabbdc3cb82379c401094d4efd0f7a5ebfc Mon Sep 17 00:00:00 2001 From: Daniel Petty Date: Thu, 5 Nov 2026 17:39:40 +0000 Subject: [PATCH 13/17] fix(telemetry): the job failure alert reads each row and a stop is no failed row A round now counts each of its rows. So a join of RoadmapServiceJobsFailing on the lane hides a row that fails while another row of its lane succeeds: the alert now groups and joins on the row. A row or a scoped update that the stop of the Service ends counts under result stopped, as a round did before, so a rollout does not fire the alert. The budget hold counter also starts at zero. --- k8s/README.md | 6 +- k8s/monitoring/prometheus-values.yaml | 12 ++-- .../design.md | 11 +++- .../proposal.md | 12 ++-- .../specs/roadmap-service-counters/spec.md | 20 +++++- .../tasks.md | 36 ++++++++-- src/devops_cli/config/constants.py | 4 ++ src/devops_cli/roadmap/run.py | 22 +++++-- src/devops_cli/server/counter_series.py | 14 +++- .../roadmap_service_tests.yaml | 45 ++++++++++--- .../test_k8s_monitoring_alerting.py | 19 ++++-- tests/roadmap/test_roadmap_run.py | 65 +++++++++++++++++-- .../telemetry/test_service_counter_series.py | 23 +++++++ 13 files changed, 240 insertions(+), 49 deletions(-) diff --git a/k8s/README.md b/k8s/README.md index 1d8a72246..5c4b85679 100644 --- a/k8s/README.md +++ b/k8s/README.md @@ -475,7 +475,7 @@ Alertmanager runs from the prometheus chart (`alertmanager` in `monitoring/prome | `ValkeyCacheErrors` | A Valkey exporter reports `redis_up == 0` or rejected client connections | 5m | warning | | `ValkeyMemoryNearBound` | A Valkey with the policy `noeviction` and a bound keeps its data above 80% of its `maxmemory`; at the bound it refuses each write | 10m | warning | | `RoadmapServiceIdentityRefused`, `RoadmapServiceGitHubThrottled` | GitHub refused the identity of the roadmap Service, or throttled its requests; see Roadmap service, Alerts | — | critical | -| `RoadmapServiceJobsFailing` | All jobs of one repository and lane failed in 1 hour | 30m | warning | +| `RoadmapServiceJobsFailing` | All jobs of one row of a repository failed in 1 hour | 30m | warning | | `RoadmapServiceLaneStalled`, `RoadmapServiceRestarted`, `RoadmapServiceUnavailable`, `RoadmapServiceGitHubRequestsHigh`, `RoadmapServiceModelRequestsHigh` | The roadmap Service stalls, its pod restarts or is unavailable, or its GitHub or model requests in an hour are high; see Roadmap service, Alerts | 0–10m | warning | | `RoadmapServiceWebhooksRejected` | The webhook route of the roadmap Service rejected more than 20 deliveries in 15 minutes; see Roadmap service, Alerts | — | warning | | `ClusterMetricsMissing` | `kube_node_status_condition` is absent, so the node rules cannot fire | 10m | critical | @@ -632,7 +632,7 @@ kubectl -n devops logs -f job/ 14. The row `alternatives` of the model lane runs after `refine`: on a first run, and 719 minutes after its last success. It runs `devops roadmap alternatives --confirm` for the repository. A round checks at most `roadmap.alternatives.first_pass_limit` items (20 by default) while an item has no check, and at most `roadmap.alternatives.limit` items (5 by default) after. The row runs about twice a day, so a first pass of 141 items takes about 4 days. A round takes first the items with no check, then the items whose check is older than `roadmap.alternatives.recheck_days` (30 by default) or whose text changed. A webhook makes the row due at no other time. The row changes no label and closes no issue. It does nothing, and logs `The alternatives check of is off`, for a private repository and when no Firecrawl key is set. If the check of an item fails, the row records the failure in the item's job record (`alternatives.failed_at` and `alternatives.failed_error`, the class of the error), writes the other checks and succeeds. The row fails, and counts toward the suspension after 3 failed rounds, only when it tried items and checked none. A round takes a failed item again after `roadmap.alternatives.retry_days` (3 by default). Each written check adds 1 to `devops_cli_roadmap_alternatives_total` by `verdict`: `covers`, `partly`, `none` or `not-applicable`. An item of the type `test`, `docs`, `refactor`, `chore`, `ci` or `perf`, and a `fix` with a Python traceback in its body, gets `not-applicable` with no model call; every other item must run a search before the check accepts a verdict, and gets one more turn to search. A `covers` verdict makes refine ask a person to adopt the tool or to keep the item with a reason. The item waits on a person until a reply answers the question, and Ready items with a new covering tool go back to New. Refine records the tool it asked about in the job record (`refine.alternatives_asked`). `devops roadmap run --dry-run` lists the row when it is due. 15. The row `stage` follows `refine` on the model lane (#2659). It runs `devops roadmap stage --confirm` in the model clone: it is due every 37 minutes, on a first run, and in a round where `refine` set an item Ready. It stages the Ready items that `roadmap.stage.filter` names, at most `roadmap.stage.limit` in a round, plans no new item while the count of open Planned items is at `roadmap.stage.planned_limit` (an item with the priority `P0-Critical` passes it, #3487), and splits a task group over `roadmap.stage.step_tokens`. Each task group has a kind (#3005): a `behavior` group has a failing test, and an `other` group has no test but a command or the steps of a manual check that prove it. The job sends a reply that breaks this rule back to the model once. It leaves out an item whose change folder is on the default branch or in its archive (#2714): a round reads that tree once with `git ls-tree`, so a merged staging pull request never makes a second one. The pull request it opens is a draft with the title `docs(openspec): plan # `. The job does not mark it ready: the row `stage_ready` does (step 18). Before it opens one, it checks each path of the plan (`reads`, `changes`, the code spans of the tasks and `verify`) against the tracked files of the commit it plans, read with one `git ls-files` for the round (#3010). A path that is no file there, and that no group up to its own creates, goes back to the model once with the files of the same name. When a later group creates the path, the problem names that group and tells the model to move it before the group that names the path (#3017). A second reply with a missing path stops the item, so no draft opens and the implement Job never meets a wrong path. `k8s/devops/configmap.example.yaml` holds the five settings, and `devops k8s deploy-stack --stack devops` applies them; each also has a variable, `DEVOPS_CLI_ROADMAP_STAGE_FILTER`, `DEVOPS_CLI_ROADMAP_STAGE_LIMIT`, `DEVOPS_CLI_ROADMAP_STAGE_PLANNED_LIMIT`, `DEVOPS_CLI_ROADMAP_STAGE_STEP_TOKENS` and `DEVOPS_CLI_ROADMAP_STAGE_TOOLSET`. The setting `roadmap.stage.toolset` is `extended` by default (#3238): a plan run then has three tools more than the nine that read the staged commit. `references` counts a text in every file of the commit. `read_issue` and `find_issues` read issues and pull requests of the repository on GitHub, as GET requests with the token of the Service and no other permission. `base` leaves them out. Each item of a round counts once in `devops_cli_roadmap_stage_total` by its `result`: `staged`, `skipped`, `needs_split` or `failed`. A failed item is recorded in its job record with the class of its error and the time (#3493), and the job leaves it out for `roadmap.stage.retry_hours` hours (6 by default) while it takes the other items. The row fails a round only when it tried items and staged, stopped or returned none, or when a failure belongs to no item; three such rounds on the same input suspend the row. The ledger file keeps the class of each error, and a round that the stop of the Service ends counts as `stopped`, not as an error. The image carries the pinned OpenSpec and a Node runtime, so the job validates each change before it pushes: `kubectl -n devops exec deploy/roadmap-service -- openspec --version` prints the locked version. 16. The row `implement` follows `metrics` on the status lane (#2667, #2757). It calls no model, so a model-lane round that has not ended never delays it. It is due every 7 minutes and on a first run, in the clone of the status lane. The Service runs no step: a round only talks to the Kubernetes API, in three moves. It reports each Job that ended, with a comment on the item that holds the state and the last 40 lines of the log of the Job's pod, masked of secrets, and records `implement.job_state` as `succeeded`, `failed` or `gone`. It moves the deadline of each Job that has not ended (#3063, below). It waits while a Job of the repository has not ended. Else it makes one Job from CronJob `devops-cli-runner` for the first Staged or Planned item that the default filter names (a Planned item has its change folder on the default branch and no branch: a person merged the staging pull request, and the job makes the branch and the draft itself), with the args `roadmap implement --repo --item --confirm` and the item's number as the only item text, and records the name of the Job in `implement.job`, and the digest of the runner image that the CronJob template names in `implement.job_digest`. The same write sets the label `owner/service` on the item (#2910), and the report of the Job removes it. The comment that reports the Job names the digest. The Service makes no Job for an item that an agent or a person claimed, and a claim of an agent or a person ends a running Job at its next group boundary, with no push and no pull request. A round also sets the Status label of the items from their record (`status_sync.py`): it reads the plans on the default branch once for the sweep and for the plan, writes `status/planned` once a plan is merged, and writes `status/in-progress` as soon as the record names a Job, and again after it makes one. It writes only an open item that holds Ready or Planned with the label that a job last recorded, so a label that a person set stays. The rows `stage` and `stage_ready` use the same function. The Jobs of a repository carry the label `implement-repo`, whose value is a digest of the repository name, so the API is the lock and a restart of the Service loses nothing. A Job that leaves its item In Progress, or Ready or Planned with no stop, is a stop that the runner could not write, such as a pod that the kernel killed for its memory, unless an agent or a person claimed the item. The round then sets the item Blocked (`status/blocked`) and records `implement.stopped` as `job`, so that no Job is made for it again. A stop that the runner writes sets Blocked in the same way, with the step in `implement.stopped`. After the report, the round clears the stop of each item that it set Blocked, by one of two rules, and a comment names the rule that ran. Rule `retry`: a comment `implement: retry`, made after the report by the owner, a member or a collaborator, clears the stop at once, whatever the image. Rule `digest`: the digest in the template of CronJob `devops-cli-runner` is not `implement.job_digest`, so a new runner image has rolled out since the stop, and the class in `implement.failure` has the action `retry` in `ACTION_BY_CLASS` (#3481). A new image cannot fix a plan, a premise, a refused tool or a failed test, so a stop of another class, and a stop with no class on record, wait for a retry. Each clear by digest adds the class, the new digest and the time to `implement.digest_clears`, and the list stays after the clear. The settings `roadmap.implement.digest_clears_per_class` (default 1) and `roadmap.implement.digest_clears_per_item` (default 3) bound the entries; a stop at a limit stays Blocked until a retry, also for `ci_infra`. The comment of a clear by digest shows at most one time in 24 hours for an item (`implement.digest_clear_commented_at`), and the clear is recorded either way. The rule reads no image content: the Service holds digests and no map from a digest to commits. A clear sets Ready or Planned and removes `implement.stopped`, and the same round can make a Job for the item. The new Job overwrites `implement.job_digest`, so a second stop on the same image waits for a retry, and the round makes no Job in a loop. A stop with no digest on record is cleared by a retry only. Inside one priority, a round takes an item with no digest clear before an item with one, and a retried item of a higher priority still goes before a new item of a lower priority. A Blocked stop that no rule clears costs one request in each round, the listing of its comments (6 an hour of `service.github_requests_per_hour`); the round reads the issue again only when a rule holds, and reads the template at most one time. The round clears no item that a Job holds, that an agent or a person claimed, or that a person labeled Blocked by hand. The failures ledger is unchanged: three failed rounds on the same input still suspend the row. If CronJob `devops-cli-runner` does not exist, the round fails with a message that names it: run `devops k8s deploy-stack --stack devops`, or wait for Argo CD. A Job that cannot pull its image (the package `runner` is not public, or no release published it) has an empty log, so it ends by its stall after 40 minutes, and the next round records the stop. Read the Jobs with `kubectl -n devops get jobs -l implement-repo`. The Job runs the checks that a person runs at a commit and at a push (#3043). Before the first step it runs `pre-commit install-hooks`, with `PRE_COMMIT_HOME` in the cache directory of the data directory, on the home volume. The two remote hook repositories come over public HTTPS (9 s and 51 MB, measured). A failed install stops the item at the step `hooks`, before any model request. Each step then runs the `pre-commit` stage on its staged changes as its last exit check. The child is `uv`, which holds no token. The Job installs no git hook, because git runs a hook with the token of the session, and the agent can edit the hook configuration. For the same reason every hook run gets `--config` with a copy of the configuration of the default branch, so an edit of the agent skips no hook. A hook run that runs out of time stops the step. A failed hook goes to the agent as a repair turn with its output. The setting `roadmap.implement.repair_rounds` (variable `DEVOPS_CLI_ROADMAP_IMPLEMENT_REPAIR_ROUNDS`, default 3) bounds the extra turns of a step for `ruff`, `mypy` and the hooks. Only a used budget stops the step. After the last group, the docs, the archive and the final rebase, and before the push, the Job runs the scoped checks: the `pre-push` stage of the hooks, `devops ci --check --only constructs,docs,openspec`, and `devops ci test` with the test modules that cover the changed Python files of `src/` and `tests/`. Above 30 test modules, which a change to a shared module gives, the tests check runs only the changed test modules, and it does not run when no test covers the change. A failed check starts a fix step, up to `repair_rounds` times for all scoped checks. The Job generates the docs again after each fix. It pushes only a head that passed all of them. A used budget stops the item at the step `scoped checks` and pushes nothing. An error or a command that runs out of time in this phase stops the item and pushes nothing too. The comment on the pull request shows the repair rounds of each step, and each run of a scoped check with its seconds. The extra run time of an item is about 10 s of hooks for each step, 9 s for the install, 35 s for the first two scoped checks, and at most about 3.5 minutes for the tests (12 test modules of the roadmap ran in 88 s on 2 CPUs). So it is 2 to 5 minutes in a Job that runs about an hour. The workflow checks `actionlint` and `zizmor` run in the `pre-push` stage only when the push changes a workflow, because the runner image holds no `actionlint`. A failed check of the pull request after the push gets a fix step, a push and a new watch, up to `repair_rounds` times, with a comment on the pull request for each round. -17. Each repository has three lanes, and each lane runs one round at a time in its own clone (#1532, #1534). The status lane runs the rows `metrics`, `implement`, `stage_ready` and `release_comments` in the clone `clone`. The triage lane runs the row `intake` on the `devops-triage` model in the clone `triage-clone`. The model lane runs the rows `refine`, `stage` and `alternatives` on `devops-background` in the clone `model-clone`. A poll and a webhook go to all three lanes. The lanes share the file `schedule.json` of the repository, the ledger `service-failures.json` and the lock that each write of an item's job record takes (#2757). When intake places a critical fix or P0 item, it writes a `refine_due` timestamp to `schedule.json` so refinement becomes due in the next model-lane round. The label `lane` of `devops_cli_service_jobs_total`, `devops_cli_service_job_seconds_total` and `devops_cli_service_queue_depth` is `status`, `triage` or `model`. The label `row` of `devops_cli_service_jobs_total` is the name of the row that ended, and the label `result` is `success` or `error`. The label `row` of `devops_cli_github_requests_total` and `devops_cli_service_job_suspended` is the name of the row. A model-lane round that has not ended does not delay a status-lane or triage-lane round. +17. Each repository has three lanes, and each lane runs one round at a time in its own clone (#1532, #1534). The status lane runs the rows `metrics`, `implement`, `stage_ready` and `release_comments` in the clone `clone`. The triage lane runs the row `intake` on the `devops-triage` model in the clone `triage-clone`. The model lane runs the rows `refine`, `stage` and `alternatives` on `devops-background` in the clone `model-clone`. A poll and a webhook go to all three lanes. The lanes share the file `schedule.json` of the repository, the ledger `service-failures.json` and the lock that each write of an item's job record takes (#2757). When intake places a critical fix or P0 item, it writes a `refine_due` timestamp to `schedule.json` so refinement becomes due in the next model-lane round. The label `lane` of `devops_cli_service_jobs_total`, `devops_cli_service_job_seconds_total` and `devops_cli_service_queue_depth` is `status`, `triage` or `model`. The label `row` of `devops_cli_service_jobs_total` is the name of the row that ended, and the label `result` is `success`, `error` or `stopped` (the stop of the Service ended the row). The label `row` of `devops_cli_github_requests_total` and `devops_cli_service_job_suspended` is the name of the row. A model-lane round that has not ended does not delay a status-lane or triage-lane round. 18. The row `stage_ready` follows `implement` on the status lane (#2865). It calls no model and needs no clone. It is due every 3 minutes and on a first run. A round reads the open items whose record holds `stage.pr` and no `stage.ready` of the same number and no `implement.job`; with no such item it sends no request of its own, since the listing of the issues is the one that `metrics` and `implement` read in the round. For each such item it reads the pull request (`GET pulls/`). A pull request that is not an open draft is only recorded in `stage.ready`. An open draft goes to the function `draft_readiness` with its check runs (`GET commits//check-runs`): when every check ended and passed, the row marks the pull request ready with `gh pr ready` (the same function as the implement job) and records `stage.ready`. A draft with a failed, cancelled or running check, or with no check, stays a draft, and the row does not retry a check. The row reads only the pull request that the stage job recorded, and never the draft of a person. It reads the check runs and not the commit statuses, because the App token holds `checks: read` and no `statuses` permission. A draft that waits for its checks costs 2 requests a round (12 an hour) of `service.github_requests_per_hour`; a pull request that the row marks costs 2 more for `gh pr ready` and 2 to record it, once. A refusal of `gh pr ready` fails the round, names the item, and the ledger of failed rounds suspends the row after three rounds in a row: check that the App holds `pull_requests: write` and `contents: write` (ADR 0015). A pull request that the row marked ready is no longer a draft, so the implement job leaves its item until a person merges the plan and the item is Planned. The row counts under `row="stage_ready"` in `devops_cli_github_requests_total`. 19. The model of each stage comes from the settings `ai.tasks..model` of the ConfigMap `devops-cli-config` (#3155). The slot `analysis` serves the model lane: the stage job (planning), refine and the alternatives check. The slot `verification` names the model that reviews a plan; no code reads it before the verification pass of #3153 is built. The slot `implement` serves every step of the implement job, and `roadmap.implement.ladder` names the same group as the one tier of each step class (#3573). The ConfigMap is set to `deepseek-v4.1-flash-1017` for `analysis` and `implement`, and to `claude-sonnet-5-5` for `verification`. The window of `implement` is 256,000 tokens: the smallest that an fp8 provider of the group offers, where the group allows 1,048,576. A step has one tier, so a failure that selects `change_tier` ends the step; the item stops and waits for a retry. The prompts of a step hold the code of the repository, so they leave the cluster to OpenRouter (SECURITY.md rule 13). `claude-sonnet-5-5` serves no step. A live run of `devops roadmap eval implement --model deepseek-v4.1-flash-1017 -F 2152-numeric-host-doubles` on 2026-11-05 passed every property in 281 s, with 42 model requests, 2,126,241 input tokens (the sum over the 42 requests) and 11,590 output tokens. The evaluation of 2026-11-02 chose them: on 16 golden cases the local `qwen3.8:27b` passed the stage gates in 3, `claude-sonnet-5-5` in 7 and `deepseek-v4.1-flash-1017` in 11, at about 0.002 USD a call for DeepSeek; in the review of 12 golden plans with 82 known defects, Sonnet 5.5 found 56% (68% of the major ones), DeepSeek 38% and the local models 22 to 23%. The plans of every model still need fixes, because the planner cannot read code (#3037). A failed Sonnet call falls back to DeepSeek at the gateway (#3152). The ConfigMap is not under Argo CD: change it with `kubectl -n devops patch configmap devops-cli-config`, then `kubectl -n devops rollout restart deployment/roadmap-service`; the CronJobs read it at each run. Measure a change of model with `devops roadmap eval stage --model `. @@ -698,7 +698,7 @@ The rule group `roadmap-service` in `monitoring/prometheus-values.yaml` watches | :--- | :--- | :--- | :--- | :--- | | `RoadmapServiceIdentityRefused` | GitHub refused the identity of the Service. The Service stopped all rounds and starts no round, also after a restart, until a person removes its halt record, and `/healthz` stays healthy. A start that finds the halt record counts the refusal again, so the alert holds through a restart. | — | critical | Read the refusal in the Service log. Repair the App or its installation (step 1), or rotate its private key (step 9). Then remove the halt record and restart the Service (step 11). | | `RoadmapServiceGitHubThrottled` | GitHub refused a request of the Service with a rate limit, or gave a 5xx for a read, in the last 15 minutes | — | critical | Read the GitHub API panels of the `devops-cli Telemetry` dashboard. Lower `service.github_requests_per_hour` or remove a repository from `service.repos`, then run `devops k8s deploy-stack --stack devops`. | -| `RoadmapServiceJobsFailing` | All jobs of one repository and lane failed in 1 hour (increase in errors is greater than zero, and increase in successes is zero). | 30m | warning | Read the errors of that repository in the Service log: `kubectl -n devops logs deployment/roadmap-service`. Repair the cause, for example the repository's `.github/roadmap.toml`. | +| `RoadmapServiceJobsFailing` | All jobs of one row of a repository failed in 1 hour (increase in errors is greater than zero, and increase in successes is zero). A success of another row of the lane does not clear the alert. | 30m | warning | Read the errors of that repository in the Service log: `kubectl -n devops logs deployment/roadmap-service`. The log line of a failed round names its rows. Repair the cause, for example the repository's `.github/roadmap.toml`. | | `RoadmapServiceLaneStalled` | No job of a repository and lane started in 2 hours while the deployment has an available replica. In a new pod, the 2 hours start when the worker of the lane starts. | 10m | warning | Find in the Service log a pause of the rounds, a spent request budget, or a job that does not end. If a job does not end, restart the deployment. | | `RoadmapServiceRestarted` | The `service` container restarted in the last hour | — | warning | Read the log before the restart: `kubectl -n devops logs deployment/roadmap-service --previous`. | | `RoadmapServiceUnavailable` | The deployment has an unavailable replica. A rollout stops the old pod before it starts the new one, so the rule waits 10 minutes. | 10m | warning | Read the events: `kubectl -n devops describe deployment/roadmap-service` and `kubectl -n devops get events --sort-by=.lastTimestamp`. | diff --git a/k8s/monitoring/prometheus-values.yaml b/k8s/monitoring/prometheus-values.yaml index 72a7a474e..f240d5adf 100644 --- a/k8s/monitoring/prometheus-values.yaml +++ b/k8s/monitoring/prometheus-values.yaml @@ -698,17 +698,19 @@ serverFiles: annotations: summary: "GitHub throttled {{ $labels.resource }} requests of the roadmap Service" description: "In the last 15 minutes GitHub refused a {{ $labels.resource }} request of the roadmap Service with a rate limit, or gave a 5xx for a read. The Service waited and sent the request again (k8s/README.md, Roadmap service)." - # When all rounds of a repository and lane fail in one hour, alert after 30 minutes. + # When all jobs of one row of a repository fail in one hour, alert after 30 minutes. A + # round counts each of its rows, so a join on the lane hides a row that fails while + # another row of its lane succeeds (#3495). - alert: RoadmapServiceJobsFailing expr: >- - sum by (repo, lane) (increase(devops_cli_service_jobs_total{job="roadmap-service", result="error"}[1h])) > 0 - unless on (repo, lane) - sum by (repo, lane) (increase(devops_cli_service_jobs_total{job="roadmap-service", result="success"}[1h])) > 0 + sum by (repo, lane, row) (increase(devops_cli_service_jobs_total{job="roadmap-service", result="error"}[1h])) > 0 + unless on (repo, lane, row) + sum by (repo, lane, row) (increase(devops_cli_service_jobs_total{job="roadmap-service", result="success"}[1h])) > 0 for: 30m labels: severity: warning annotations: - summary: "All {{ $labels.lane }} jobs of {{ $labels.repo }} failed in the last hour" + summary: "All {{ $labels.row }} jobs of {{ $labels.repo }} failed in the last hour, on the {{ $labels.lane }} lane" description: '{namespace="devops"} | json | repo="{{ $labels.repo }}" | lane="{{ $labels.lane }}" (k8s/README.md, Roadmap service)' # Each poll starts a round on every lane, every 5 minutes by default. A worker sets the # job start of its lane when it starts, so a lane of a new pod that starts no job also diff --git a/openspec/changes/3495-service-counters-zero-series-repo-label/design.md b/openspec/changes/3495-service-counters-zero-series-repo-label/design.md index 98d625919..9ea97da37 100644 --- a/openspec/changes/3495-service-counters-zero-series-repo-label/design.md +++ b/openspec/changes/3495-service-counters-zero-series-repo-label/design.md @@ -3,9 +3,13 @@ ## Decisions - **The Service creates the bounded label sets of its counters at zero when it starts.** A labelled counter has no series until `.labels()` makes a child (see the Feasibility of the proposal). One function calls `.labels()` for each bounded label set, so the first scrape of a pod reads a zero series for each label set. -- **The label sets are bounded.** The failure classes are the 19 members of `FailureClass`. The repositories are `service.repos`. The rows are the rows of the three lane tables of the roadmap run. The events of the webhook counter are the declared events of each provider, plus `other`. The sources of the trigger counter are `webhook` and `poll`. The results of the chat command are `usage`, `status`, `unknown_repo` and `error`. No dynamic label value is created at start. +- **The label sets are bounded.** The failure classes are the 19 members of `FailureClass`. The repositories are `service.repos`. The rows are the rows of the three lane tables of the roadmap run. The events of the webhook counter are the declared events of each provider, plus `other`. The results of the job counter are `success`, `error` and `stopped`. The sources of the trigger counter are `webhook` and `poll`. The results of the chat command are `usage`, `status`, `unknown_repo` and `error`. No dynamic label value is created at start. - **The exposition format stays text 0.0.4.** The Service serves `/metrics` in this format (`src/devops_cli/server/service.py`). The zero children appear in this format, so the change does not switch the format. - **The jobs counter counts each row.** The roadmap run counts each row of a lane with its result, with the row name of the row table. The worker counts the scoped updates with the name of their job. A row counted in the run is not counted again in the worker, so no job is counted twice. +- **A stop is no failed row.** A row or a scoped update that the stop of the Service ends counts under `result="stopped"`. `RoadmapServiceJobsFailing` reads only `error` and `success`, so a rollout does not fire it for a row that runs twice a day. +- **`RoadmapServiceJobsFailing` reads the row.** The rule groups and joins on `repo`, `lane` and `row`. Before this change, a failed row made its whole round an error. With a count for each row, the row `metrics` succeeds in each status-lane round, so a join on the lane never fires for a failed row of that lane. A promtool case holds this. +- **The Loki query of `RoadmapServiceJobsFailing` has no field `row`.** The log line of a failed round has no such field, and its text names the rows. The summary of the alert names the row. +- **The budget hold counter is primed too.** It has no label, but the registry creates a counter at its first count, so it has no series before the first hold. - **The OTLP workaround terms stay.** The GitHub request and AI request series reach Prometheus over OTLP, not the `/metrics` scrape. The comment at `k8s/monitoring/prometheus-values.yaml` says that an OTLP series gets a sample only when the Service sends a new observation. No live Prometheus query ran in this change, so the OTLP terms stay until a live check shows that they are not needed. - **The gap of a new pod stays.** The zero children are created at start. A count made before the first scrape of a new pod is still lost, because `increase()` does not count the first sample of a series. The documents state this gap. - **No recording rule.** The promtool case of a pod change decides whether a recording rule is needed. The case runs with the zero series of the new pod. @@ -14,5 +18,8 @@ - **Switch `/metrics` to OpenMetrics or PrometheusProto for start timestamps.** Rejected: the change of format reaches every consumer of `/metrics`, and the start-timestamp injection is not verified for this path. The zero children solve the missing first sample without a format change. - **Keep the per-batch count and add a second count per row.** Rejected: one job would be counted twice, once per lane batch and once per row. -- **Prime the identity label of `devops_cli_service_identity_refused_total`.** Rejected for this change: the identity values come from the configured GitHub App, the alert `RoadmapServiceIdentityRefused` reads `> 0` with no workaround term, and the label set is not in the acceptance criteria of the issue. It stays an open item. +- **Prime the identity label of `devops_cli_service_identity_refused_total`.** Rejected: its label `identity` holds the identity that GitHub refuses, so its value is not known before a refusal. The alert `RoadmapServiceIdentityRefused` reads `> 0` with no workaround term, so the alert does not need a zero series. +- **Add the field `row` to the Loki query of `RoadmapServiceJobsFailing`.** Rejected: the log line has no field `row`, so the query would give no line. +- **Keep the join of `RoadmapServiceJobsFailing` on the lane.** Rejected: the promtool case of a row that fails each round, while the row `metrics` of its lane succeeds, gives no alert with this join. +- **Count a round that fails before its first row in this change.** Rejected: the failure of a round has no row, so it needs its own label value or counter and its own alert. #3666 holds this work. - **Add a recording rule for the true total without a test.** Rejected: a recording rule adds a second series name for one counter. It needs a promtool case that shows the need first. diff --git a/openspec/changes/3495-service-counters-zero-series-repo-label/proposal.md b/openspec/changes/3495-service-counters-zero-series-repo-label/proposal.md index aba124636..354246b24 100644 --- a/openspec/changes/3495-service-counters-zero-series-repo-label/proposal.md +++ b/openspec/changes/3495-service-counters-zero-series-repo-label/proposal.md @@ -6,9 +6,11 @@ Issue #3495 reports that the Service counters give a false picture. The counters ## What Changes -- When the Service starts, it creates each bounded label set of its counters at zero: each failure class for each repository of `service.repos`, each repository, lane, row and result of `devops_cli_service_jobs_total`, each repository and lane of `devops_cli_service_job_seconds_total`, each provider, declared event and outcome of `devops_cli_service_webhook_deliveries_total`, each repository and source of `devops_cli_service_triggers_total`, and each result of the `/roadmap` chat command on `devops_cli_service_chat_commands_total`. The identity counter `devops_cli_service_identity_refused_total` is left out (see design.md). +- When the Service starts, it creates each bounded label set of its counters at zero: each failure class for each repository of `service.repos`, each repository, lane, row and result of `devops_cli_service_jobs_total`, each repository and lane of `devops_cli_service_job_seconds_total`, each provider, declared event and outcome of `devops_cli_service_webhook_deliveries_total`, each repository and source of `devops_cli_service_triggers_total`, each result of the `/roadmap` chat command on `devops_cli_service_chat_commands_total`, and the one series of `devops_cli_service_budget_holds_total`. The identity counter `devops_cli_service_identity_refused_total` is left out (see design.md). - `devops_cli_service_implement_failures_total` gets the label `repo`. - `devops_cli_service_jobs_total` gets the label `row`. The roadmap run counts each row of a lane with its result. The worker no longer counts a whole lane batch. +- A row that the stop of the Service ends counts under `result="stopped"`, as a round did before, and not as an error. +- `RoadmapServiceJobsFailing` groups and joins on `repo`, `lane` and `row`. With a count for each row, a join on the lane hides a row that fails while another row of its lane succeeds. - The alert expressions of `RoadmapServiceWebhooksRejected` and `RoadmapServiceSelfDeliveriesHigh` lose their #2505 workaround terms. The OTLP alert expressions keep theirs. - Promtool cases prove the true total over a pod change, and prove that the webhook alerts hold without the workaround. - `docs/TELEMETRY.md` and `k8s/README.md` state the scrape behavior and the labels. The action text of `RoadmapServiceGitHubRequestsHigh` no longer says to lower the budget. The class count of the Service changes from 17 to 19. @@ -27,7 +29,7 @@ Issue #3495 reports that the Service counters give a false picture. The counters - Code: `src/devops_cli/telemetry/metrics.py`, `src/devops_cli/server/service.py`, `src/devops_cli/server/counter_series.py`, `src/devops_cli/roadmap/implement_submit.py`, `src/devops_cli/roadmap/run.py`, `src/devops_cli/server/routes/webhooks.py`, `src/devops_cli/config/constants.py`, `src/devops_cli/docs/generator.py`. - Monitoring: `k8s/monitoring/prometheus-values.yaml`. -- Tests: `tests/prometheus/test_implement_failure_labels.py`, `tests/telemetry/test_service_counter_series.py`, `tests/prometheus/test_k8s_monitoring_alerting.py`, `tests/fixtures/prometheus_rules/roadmap_service_tests.yaml`, `tests/fixtures/prometheus_rules/roadmap_service_webhooks_tests.yaml`. +- Tests: `tests/telemetry/test_service_counter_series.py`, `tests/roadmap/test_roadmap_run.py`, `tests/roadmap/test_roadmap_implement_submit.py`, `tests/server/test_server_service.py`, `tests/prometheus/test_k8s_monitoring_alerting.py`, `tests/fixtures/prometheus_rules/roadmap_service_tests.yaml`, `tests/fixtures/prometheus_rules/roadmap_service_webhooks_tests.yaml`. - Documents: `docs/TELEMETRY.md`, `k8s/README.md`. - The change adds no dependency. @@ -40,8 +42,10 @@ Issue #3495 reports that the Service counters give a false picture. The counters - The action text of `RoadmapServiceGitHubRequestsHigh` in `k8s/README.md:705` says "Lower the budget or remove a repository". This is confirmed by reading the file at origin/main. - `k8s/monitoring/prometheus-values.yaml:23` enables `created-timestamp-zero-ingestion`. The stale comment at lines 670-673 says that neither path has a zero sample. This is confirmed by reading the file at origin/main. - The upstream statement that start-timestamp injection covers only PrometheusProto and OpenMetrics 1.0.0 scrapes is recorded from the issue research and is not checked again here. The design does not depend on it. -- The live evidence table of the issue is not checked. No live Prometheus query ran in this change. +- The worker on origin/main counted the job counter with the result of the round, so a failed row made the round an error, and a stop counted as `stopped`. This is confirmed by reading `src/devops_cli/server/service.py` and its test `test_a_round_that_the_stop_of_the_service_ends_counts_as_stopped_and_logs_no_error` at origin/main. +- The log line of a failed round has the fields `repo`, `lane` and `result`, and no field `row`. Its text names the rows. This is confirmed by reading `_execute_batch` in `src/devops_cli/server/service.py` and `_execute_due_rows` in `src/devops_cli/roadmap/run.py`. So the Loki query of `RoadmapServiceJobsFailing` keeps its fields. +- The live evidence table of the issue is not checked. No live Prometheus query ran in this change, because the series change only after the deploy. The pull request comment holds the queries to run after the deploy. ## Existing Tool Considered -The open tool `prometheus_client` covers the zero series: calling `.labels(...)` at start creates the child at 0, and the text exposition shows it (see Feasibility). The open tool `promtool` checks the rules and runs their unit tests. Both are present, so the change adds no code for the exposition format. The change deletes these internal parts: the per-batch count of `devops_cli_service_jobs_total` in the worker (`src/devops_cli/server/service.py`), the two #2505 workaround terms of the webhook alerts in `k8s/monitoring/prometheus-values.yaml`, and the stale comment at lines 670-673 of the same file. The module `src/devops_cli/server/counter_series.py` has 95 lines, and its function `prime_service_counters` has 48 lines. The function holds the label sets of the counters, and the change adds no custom metric type. +The open tool `prometheus_client` covers the zero series: calling `.labels(...)` at start creates the child at 0, and the text exposition shows it (see Feasibility). The open tool `promtool` checks the rules and runs their unit tests. Both are present, so the change adds no code for the exposition format. The change deletes these internal parts: the per-batch count of `devops_cli_service_jobs_total` in the worker (`src/devops_cli/server/service.py`), the two #2505 workaround terms of the webhook alerts in `k8s/monitoring/prometheus-values.yaml`, and the stale comment at lines 670-673 of the same file. The module `src/devops_cli/server/counter_series.py` has about 100 lines, and its function `prime_service_counters` has about 50 lines. The function holds the label sets of the counters, and the change adds no custom metric type. diff --git a/openspec/changes/3495-service-counters-zero-series-repo-label/specs/roadmap-service-counters/spec.md b/openspec/changes/3495-service-counters-zero-series-repo-label/specs/roadmap-service-counters/spec.md index 7e8b94890..40db7223a 100644 --- a/openspec/changes/3495-service-counters-zero-series-repo-label/specs/roadmap-service-counters/spec.md +++ b/openspec/changes/3495-service-counters-zero-series-repo-label/specs/roadmap-service-counters/spec.md @@ -22,8 +22,15 @@ The Service SHALL create each bounded label set of `devops_cli_service_triggers_ - **WHEN** the Service starts with the repositories `example/app` and `example/tools` - **THEN** the trigger counter exists at 0 for each repository with the sources `webhook` and `poll`, and the chat counter exists at 0 for each result `usage`, `status`, `unknown_repo` and `error` +### Requirement: The Service starts the budget hold counter at zero +The Service SHALL create the one series of `devops_cli_service_budget_holds_total` at zero when it starts. + +#### Scenario: The budget hold counter exists at zero after start +- **WHEN** the Service starts +- **THEN** the budget hold counter exists at 0 with no label + ### Requirement: The job counter carries the row of the job -The Service SHALL add 1 to `devops_cli_service_jobs_total` with the labels `repo`, `lane`, `row` and `result` for each row of a lane that ends in the roadmap run. A scoped update SHALL add 1 with the name of its job as the label `row`. The Service SHALL NOT count a lane batch as one job. +The Service SHALL add 1 to `devops_cli_service_jobs_total` with the labels `repo`, `lane`, `row` and `result` for each row of a lane that ends in the roadmap run. A scoped update SHALL add 1 with the name of its job as the label `row`. The Service SHALL NOT count a lane batch as one job. The label `result` SHALL be `success`, `error` or `stopped`. A row or a scoped update that the stop of the Service ends SHALL count under `stopped`, and not under `error`. #### Scenario: A row that succeeds - **WHEN** the status lane runs the row `metrics` for `example/app` and the row succeeds @@ -33,6 +40,17 @@ The Service SHALL add 1 to `devops_cli_service_jobs_total` with the labels `repo - **WHEN** one round of the status lane runs the rows `metrics` and `implement`, and both fail - **THEN** the job counter for each row with `result` `error` grows by 1, and no series counts the batch +#### Scenario: A row that the stop of the Service ends +- **WHEN** the stop of the Service ends the row `implement` of the status lane +- **THEN** the job counter of that row with `result` `stopped` grows by 1, and the counter with `result` `error` does not change + +### Requirement: The job failure alert reads each row +`RoadmapServiceJobsFailing` SHALL group and join on `repo`, `lane` and `row`, so that a success of another row of the lane does not hide a row whose jobs all failed. + +#### Scenario: A row fails while another row of its lane succeeds +- **WHEN** `promtool test rules` evaluates a status lane whose row `implement` fails each round and whose row `metrics` succeeds +- **THEN** `RoadmapServiceJobsFailing` fires with the labels `repo`, `lane` `status` and `row` `implement` + ### Requirement: Alerts read a counter that starts at zero with increase The alerts `RoadmapServiceWebhooksRejected` and `RoadmapServiceSelfDeliveriesHigh` SHALL read their counter with `increase()` and no `offset` term, because the Service starts each label set at zero. diff --git a/openspec/changes/3495-service-counters-zero-series-repo-label/tasks.md b/openspec/changes/3495-service-counters-zero-series-repo-label/tasks.md index a380f779c..59ac297e7 100644 --- a/openspec/changes/3495-service-counters-zero-series-repo-label/tasks.md +++ b/openspec/changes/3495-service-counters-zero-series-repo-label/tasks.md @@ -62,10 +62,38 @@ Reads: `src/devops_cli/server/counter_series.py`, `src/devops_cli/server/routes/ - [x] 6.3 State in `src/devops_cli/docs/generator.py`, the source of `docs/TELEMETRY.md`, and in `k8s/README.md` that the identity counter is the one counter whose label set does not start at zero, and regenerate `docs/TELEMETRY.md` with `uv run devops docs generate --no-sync-readme` - Verification: `uv run devops ci docs` passes +## 7. The job failure alert reads each row + +Reads: `k8s/monitoring/prometheus-values.yaml`, `tests/prometheus/test_k8s_monitoring_alerting.py`, `src/devops_cli/server/service.py`. Changes: `tests/fixtures/prometheus_rules/roadmap_service_tests.yaml`, `tests/prometheus/test_k8s_monitoring_alerting.py`, `k8s/monitoring/prometheus-values.yaml`, `k8s/README.md`. + +- [x] 7.1 Add a promtool case to `tests/fixtures/prometheus_rules/roadmap_service_tests.yaml` in which the row `implement` fails each round while the row `metrics` of its lane succeeds, label the job series of the other cases with a row, and pin the rule in `tests/prometheus/test_k8s_monitoring_alerting.py` + - Verification: `uv run devops ci test tests/prometheus/test_k8s_monitoring_alerting.py` fails on the rule that joins on the lane +- [x] 7.2 Group and join `RoadmapServiceJobsFailing` on `repo`, `lane` and `row`, and name the row in its summary, in `k8s/monitoring/prometheus-values.yaml`. Keep the fields of its Loki query + - Verification: `uv run devops ci test tests/prometheus/test_k8s_monitoring_alerting.py` passes +- [x] 7.3 State in the two alert tables of `k8s/README.md` that the alert reads each row + - Verification: `uv run devops ci docs` passes + +## 8. A stop is no failed row + +Reads: `src/devops_cli/exceptions/roadmap.py`, `src/devops_cli/roadmap/run.py`. Changes: `src/devops_cli/config/constants.py`, `src/devops_cli/roadmap/run.py`, `src/devops_cli/server/counter_series.py`, `tests/roadmap/test_roadmap_run.py`, `tests/telemetry/test_service_counter_series.py`, `k8s/README.md`. + +- [x] 8.1 Add tests to `tests/roadmap/test_roadmap_run.py` that assert that a row and a scoped update that the stop of the Service ends count under `stopped` and a scoped update that fails counts under `error`, and a test to `tests/telemetry/test_service_counter_series.py` that asserts that a row exists at 0 for each result + - Verification: `uv run devops ci test tests/roadmap/test_roadmap_run.py tests/telemetry/test_service_counter_series.py` fails on the count that reads only `error` +- [x] 8.2 Count a row and a scoped update under the result of their error in `src/devops_cli/roadmap/run.py`, and prime the result `stopped` in `src/devops_cli/server/counter_series.py` + - Verification: `uv run devops ci test tests/roadmap/test_roadmap_run.py tests/telemetry/test_service_counter_series.py` passes +- [x] 8.3 State the three results of the job counter in `k8s/README.md` + - Verification: `uv run devops ci docs` passes + +## 9. The budget hold counter starts at zero + +Reads: `src/devops_cli/telemetry/metrics.py`. Changes: `tests/telemetry/test_service_counter_series.py`, `src/devops_cli/server/counter_series.py`. + +- [x] 9.1 Add a test to `tests/telemetry/test_service_counter_series.py` that asserts that the budget hold counter exists at 0 with no label + - Verification: `uv run devops ci test tests/telemetry/test_service_counter_series.py` fails on the base +- [x] 9.2 Prime the budget hold counter in `src/devops_cli/server/counter_series.py` + - Verification: `uv run devops ci test tests/telemetry/test_service_counter_series.py` passes + ## Workflow follow-up -- Open item: agree the label `repo` of `devops_cli_service_implement_failures_total` with #3069 before this change lands. The issue says that #3069 reuses the label `class`. -- Open item: check the live series with a Prometheus query after the deploy. This change runs no live query. -- Open item: a person decides whether `devops_cli_service_identity_refused_total` gets a zero label set. The design defers it, and no issue records the deferral yet. -- Open item: a round that fails before its first row counts no job, so `RoadmapServiceJobsFailing` may miss a failure at store level. A person decides whether to count it. +- A round that fails before its first row counts no job, so `RoadmapServiceJobsFailing` does not read it: #3666. - Pending a person: review the diff in tuicr (tuicr -r ..HEAD). diff --git a/src/devops_cli/config/constants.py b/src/devops_cli/config/constants.py index bd12f84a3..cef20c829 100644 --- a/src/devops_cli/config/constants.py +++ b/src/devops_cli/config/constants.py @@ -1550,6 +1550,10 @@ class WebhookExposure(StrEnum): CONST_SERVICE_METRIC_TRIGGERS: Final[str] = "devops_cli_service_triggers_total" CONST_SERVICE_METRIC_JOBS: Final[str] = "devops_cli_service_jobs_total" CONST_SERVICE_METRIC_JOB_SECONDS: Final[str] = "devops_cli_service_job_seconds_total" +# The `result` values of the jobs counter: a stop of the Service is no failure of its row (#2753). +CONST_SERVICE_JOB_RESULT_SUCCESS: Final[str] = "success" +CONST_SERVICE_JOB_RESULT_ERROR: Final[str] = "error" +CONST_SERVICE_JOB_RESULT_STOPPED: Final[str] = "stopped" # The time the last job of a lane started, or the time its worker started before its first job # (#2505). CONST_SERVICE_METRIC_JOB_START_TIMESTAMP: Final[str] = ( diff --git a/src/devops_cli/roadmap/run.py b/src/devops_cli/roadmap/run.py index d93e73e9f..2f27a349f 100644 --- a/src/devops_cli/roadmap/run.py +++ b/src/devops_cli/roadmap/run.py @@ -59,6 +59,9 @@ CONST_ROADMAP_RUN_TRIAGE_CLONE_DIRNAME, CONST_SERVICE_CAUSE_UNKNOWN, CONST_SERVICE_FAILURES_FILENAME, + CONST_SERVICE_JOB_RESULT_ERROR, + CONST_SERVICE_JOB_RESULT_STOPPED, + CONST_SERVICE_JOB_RESULT_SUCCESS, CONST_SERVICE_LANE_MODEL, CONST_SERVICE_LANE_STATUS, CONST_SERVICE_LANE_TRIAGE, @@ -839,10 +842,10 @@ def run_scoped_update( now=now or datetime.now(UTC), model=model, ) - except Exception: - _count_job(delivery.repo, update.lane, update.job, "error") + except Exception as exc: + _count_job(delivery.repo, update.lane, update.job, _failed_result(exc)) raise - _count_job(delivery.repo, update.lane, update.job, "success") + _count_job(delivery.repo, update.lane, update.job, CONST_SERVICE_JOB_RESULT_SUCCESS) return outcome @@ -1153,6 +1156,13 @@ def _format_error_line(exc: BaseException) -> str: return mask_secrets(first)[:256] +def _failed_result(exc: BaseException) -> str: + """The `result` of a job that ended with `exc`: a stop of the Service is no failure (#2753).""" + return ( + CONST_SERVICE_JOB_RESULT_STOPPED if ended_by_stop(exc) else CONST_SERVICE_JOB_RESULT_ERROR + ) + + def _count_job(repo: str, lane: str | None, row: str, result: str) -> None: """Count one job that ended under its repository, lane, row and result (#3495). A run with no lane, such as one of the command line, counts no job.""" @@ -1203,7 +1213,7 @@ def _execute_due_rows( if row.needs_clone and clone_error is not None: failed_jobs.append(row.name) clone_jobs.append(row.name) - _count_job(repo, lane, row.name, "error") + _count_job(repo, lane, row.name, _failed_result(clone_error)) continue try: @@ -1224,10 +1234,10 @@ def _execute_due_rows( succeeded_jobs.append(row.name) _record_success(schedule_path, row.name, current_time) failures.succeeded(repo, row.name) - _count_job(repo, lane, row.name, "success") + _count_job(repo, lane, row.name, CONST_SERVICE_JOB_RESULT_SUCCESS) except Exception as exc: failed_jobs.append(row.name) - _count_job(repo, lane, row.name, "error") + _count_job(repo, lane, row.name, _failed_result(exc)) outcomes[row.name] = _outcome_of_failure(exc) _count_failure(failures, repo, row.name, current, exc) if row.needs_clone and clone_path is None: diff --git a/src/devops_cli/server/counter_series.py b/src/devops_cli/server/counter_series.py index 419175050..194f058bc 100644 --- a/src/devops_cli/server/counter_series.py +++ b/src/devops_cli/server/counter_series.py @@ -15,9 +15,13 @@ CONST_SERVICE_CHAT_RESULT_STATUS, CONST_SERVICE_CHAT_RESULT_UNKNOWN_REPO, CONST_SERVICE_CHAT_RESULT_USAGE, + CONST_SERVICE_JOB_RESULT_ERROR, + CONST_SERVICE_JOB_RESULT_STOPPED, + CONST_SERVICE_JOB_RESULT_SUCCESS, CONST_SERVICE_LANE_MODEL, CONST_SERVICE_LANE_STATUS, CONST_SERVICE_LANE_TRIAGE, + CONST_SERVICE_METRIC_BUDGET_HOLDS, CONST_SERVICE_METRIC_CHAT_COMMANDS, CONST_SERVICE_METRIC_IMPLEMENT_FAILURES, CONST_SERVICE_METRIC_JOB_SECONDS, @@ -29,7 +33,11 @@ from devops_cli.roadmap.failure import FailureClass from devops_cli.telemetry.metrics import GLOBAL_METRICS -SERVICE_JOB_RESULTS = ("success", "error") +SERVICE_JOB_RESULTS = ( + CONST_SERVICE_JOB_RESULT_SUCCESS, + CONST_SERVICE_JOB_RESULT_ERROR, + CONST_SERVICE_JOB_RESULT_STOPPED, +) """The value of the label `result` of a job that ended, as the roadmap run counts it.""" SERVICE_TRIGGER_SOURCES = (CONST_SERVICE_SOURCE_WEBHOOK, CONST_SERVICE_SOURCE_POLL) @@ -49,7 +57,8 @@ def prime_service_counters(repos: Iterable[str]) -> None: """Create the bounded label sets of the Service counters at zero for the repositories `repos`: each failure class for each repository on the failure counter, each repository, lane, row and result on the job counter, each repository and lane on the job seconds, each - repository and source on the trigger counter, and each result of the chat command.""" + repository and source on the trigger counter, each result of the chat command, and the one + series of the budget holds, which has no label.""" # The roadmap run imports the Service, so the lane tables are read when the counters are # primed, not when this module is imported. from devops_cli.roadmap.run import MODEL_LANE_ROWS, STATUS_LANE_ROWS, TRIAGE_LANE_ROWS @@ -93,3 +102,4 @@ def prime_service_counters(repos: Iterable[str]) -> None: for result in SERVICE_CHAT_RESULTS ), ) + GLOBAL_METRICS.prime_counter(CONST_SERVICE_METRIC_BUDGET_HOLDS, [{}]) diff --git a/tests/fixtures/prometheus_rules/roadmap_service_tests.yaml b/tests/fixtures/prometheus_rules/roadmap_service_tests.yaml index 10d1f4c35..e54a6bdbd 100644 --- a/tests/fixtures/prometheus_rules/roadmap_service_tests.yaml +++ b/tests/fixtures/prometheus_rules/roadmap_service_tests.yaml @@ -60,18 +60,18 @@ tests: alertname: RoadmapServiceGitHubThrottled exp_alerts: [] - - name: A lane whose new error series reaches 3 fires; one more error of an older series does not + - name: A row whose new error series reaches 3 fires; one more error of an older series does not interval: 1m input_series: # A new series: 3 failed rounds in a row, then the job is suspended. - - series: 'devops_cli_service_jobs_total{job="roadmap-service", repo="example/app", lane="model", result="error"}' + - series: 'devops_cli_service_jobs_total{job="roadmap-service", repo="example/app", lane="model", row="refine", result="error"}' values: '_x60 1x4 2x4 3x40' # A series that was there an hour ago and gets one more error. - - series: 'devops_cli_service_jobs_total{job="roadmap-service", repo="example/app", lane="status", result="error"}' + - series: 'devops_cli_service_jobs_total{job="roadmap-service", repo="example/app", lane="status", row="metrics", result="error"}' values: '5x70 6x40' - - series: 'devops_cli_service_jobs_total{job="roadmap-service", repo="example/app", lane="status", result="success"}' + - series: 'devops_cli_service_jobs_total{job="roadmap-service", repo="example/app", lane="status", row="metrics", result="success"}' values: '0+1x110' - - series: 'devops_cli_service_jobs_total{job="kubernetes-pods", repo="example/app", lane="model", result="error"}' + - series: 'devops_cli_service_jobs_total{job="kubernetes-pods", repo="example/app", lane="model", row="refine", result="error"}' values: '0+1x110' alert_rule_test: - eval_time: 62m @@ -87,8 +87,9 @@ tests: severity: warning repo: example/app lane: model + row: refine exp_annotations: - summary: "All model jobs of example/app failed in the last hour" + summary: "All refine jobs of example/app failed in the last hour, on the model lane" description: '{namespace="devops"} | json | repo="example/app" | lane="model" (k8s/README.md, Roadmap service)' - name: An error series with no sample near the start of the hour counts only the errors of the hour @@ -96,7 +97,7 @@ tests: input_series: # Samples at 50m, 70m and 110m: two jobs failed in the hour before 120m. The series has no # sample in the 5 minutes before 60m, and it is not new. - - series: 'devops_cli_service_jobs_total{job="roadmap-service", repo="example/app", lane="model", result="error"}' + - series: 'devops_cli_service_jobs_total{job="roadmap-service", repo="example/app", lane="model", row="refine", result="error"}' values: '_x5 5 _ 6 _ _ _ 7' alert_rule_test: - eval_time: 120m @@ -106,15 +107,16 @@ tests: severity: warning repo: example/app lane: model + row: refine exp_annotations: - summary: "All model jobs of example/app failed in the last hour" + summary: "All refine jobs of example/app failed in the last hour, on the model lane" description: '{namespace="devops"} | json | repo="example/app" | lane="model" (k8s/README.md, Roadmap service)' - name: A new error series with one sample of 3 fires interval: 10m input_series: # One sample at 60m: 3 jobs failed before the Service sent the series. - - series: 'devops_cli_service_jobs_total{job="roadmap-service", repo="example/app", lane="model", result="error"}' + - series: 'devops_cli_service_jobs_total{job="roadmap-service", repo="example/app", lane="model", row="refine", result="error"}' values: '_x5 0 3' alert_rule_test: - eval_time: 65m @@ -127,10 +129,33 @@ tests: severity: warning repo: example/app lane: model + row: refine exp_annotations: - summary: "All model jobs of example/app failed in the last hour" + summary: "All refine jobs of example/app failed in the last hour, on the model lane" description: '{namespace="devops"} | json | repo="example/app" | lane="model" (k8s/README.md, Roadmap service)' + - name: A row that fails each round fires while another row of its lane succeeds + interval: 1m + input_series: + - series: 'devops_cli_service_jobs_total{job="roadmap-service", repo="example/app", lane="status", row="implement", result="error"}' + values: '0+1x110' + - series: 'devops_cli_service_jobs_total{job="roadmap-service", repo="example/app", lane="status", row="implement", result="success"}' + values: '0x110' + - series: 'devops_cli_service_jobs_total{job="roadmap-service", repo="example/app", lane="status", row="metrics", result="success"}' + values: '0+1x110' + alert_rule_test: + - eval_time: 45m + alertname: RoadmapServiceJobsFailing + exp_alerts: + - exp_labels: + severity: warning + repo: example/app + lane: status + row: implement + exp_annotations: + summary: "All implement jobs of example/app failed in the last hour, on the status lane" + description: '{namespace="devops"} | json | repo="example/app" | lane="status" (k8s/README.md, Roadmap service)' + - name: A lane with no job start for over 2 hours fires after 10 minutes while a replica is available interval: 1m input_series: diff --git a/tests/prometheus/test_k8s_monitoring_alerting.py b/tests/prometheus/test_k8s_monitoring_alerting.py index a4fbdea0a..5ad028e10 100644 --- a/tests/prometheus/test_k8s_monitoring_alerting.py +++ b/tests/prometheus/test_k8s_monitoring_alerting.py @@ -864,9 +864,9 @@ def _window_count(series: str, window: str) -> str: ), ( "RoadmapServiceJobsFailing", - 'sum by (repo, lane) (increase(devops_cli_service_jobs_total{job="roadmap-service", result="error"}[1h])) > 0' - " unless on (repo, lane)" - ' sum by (repo, lane) (increase(devops_cli_service_jobs_total{job="roadmap-service", result="success"}[1h])) > 0', + 'sum by (repo, lane, row) (increase(devops_cli_service_jobs_total{job="roadmap-service", result="error"}[1h])) > 0' + " unless on (repo, lane, row)" + ' sum by (repo, lane, row) (increase(devops_cli_service_jobs_total{job="roadmap-service", result="success"}[1h])) > 0', "30m", "warning", ), @@ -967,25 +967,30 @@ def test_the_roadmap_service_rules_watch_identity_jobs_lanes_pods_and_request_vo ] == ROADMAP_SERVICE_RULES -def test_roadmap_service_jobs_failing_rule_groups_and_joins_on_repo_and_lane() -> None: - """The rule groups by and joins on exactly (repo, lane), and describes a Loki query.""" +def test_roadmap_service_jobs_failing_rule_groups_and_joins_on_repo_lane_and_row() -> None: + """The rule groups by and joins on exactly (repo, lane, row), and describes a Loki query. A + round counts each of its rows (#3495), so a join on the lane alone hides a row that fails + while another row of its lane succeeds. The log line of a failed round has the fields repo + and lane and names its rows in its text, so the query has no field row.""" rule = next(r for r in _rules() if r.get("alert") == "RoadmapServiceJobsFailing") assert ( rule["for"], rule["labels"]["severity"], "{{ $labels.repo }}" in rule["annotations"]["summary"], "{{ $labels.lane }}" in rule["annotations"]["summary"], + "{{ $labels.row }}" in rule["annotations"]["summary"], '{namespace="devops"} | json | repo="{{ $labels.repo }}" | lane="{{ $labels.lane }}" (k8s/README.md, Roadmap service)', ) == ( "30m", "warning", True, True, + True, rule["annotations"]["description"], ) expr = rule["expr"] - assert expr.count("sum by (repo, lane)") == 2 - assert "unless on (repo, lane)" in expr + assert expr.count("sum by (repo, lane, row)") == 2 + assert "unless on (repo, lane, row)" in expr def test_the_webhook_rule_warns_on_more_than_20_rejected_deliveries_in_15_minutes() -> None: diff --git a/tests/roadmap/test_roadmap_run.py b/tests/roadmap/test_roadmap_run.py index 3001ba5e9..a37264f73 100644 --- a/tests/roadmap/test_roadmap_run.py +++ b/tests/roadmap/test_roadmap_run.py @@ -61,6 +61,7 @@ GitOperationError, RoadmapRefineError, RoadmapRunError, + ServiceStoppingError, ) from devops_cli.exceptions.git import GitHubOperationError, GitHubUnauthenticatedError from devops_cli.github.rate_limiter import get_github_rate_limiter @@ -2928,6 +2929,14 @@ def test_the_release_comments_adapter_comments_a_release_and_keeps_its_ledger( assert (tmp_path / "release_comments.json").is_file() +def _job_count(lane: str, row: str, result: str) -> float: + """The value of the job counter of the row `row` of `lane` under `result`.""" + return GLOBAL_METRICS.get_counter( + CONST_SERVICE_METRIC_JOBS, + labels={"repo": REPO, "lane": lane, "row": row, "result": result}, + ) + + def test_each_row_of_a_lane_counts_its_own_job( tmp_path: Path, monkeypatch: pytest.MonkeyPatch ) -> None: @@ -2945,10 +2954,56 @@ def fails(**_: object) -> JobOutcome: service_jobs(tmp_path)[CONST_SERVICE_LANE_STATUS](POLL, table=table) assert list(raised.value.failed_jobs) == ["implement"] - def count(row: str, result: str) -> float: - return GLOBAL_METRICS.get_counter( - CONST_SERVICE_METRIC_JOBS, - labels={"repo": REPO, "lane": CONST_SERVICE_LANE_STATUS, "row": row, "result": result}, + assert ( + _job_count(CONST_SERVICE_LANE_STATUS, "metrics", "success"), + _job_count(CONST_SERVICE_LANE_STATUS, "implement", "error"), + ) == (1, 1) + + +def test_a_row_that_the_stop_of_the_service_ends_counts_as_stopped( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch +) -> None: + """A stop is no failure of the row (#2753), so the row counts under `stopped` and the job + failure alert does not read it (#3495).""" + GLOBAL_METRICS.reset() + github = _service_github(_labeled(2, "alice")) + _serve_over(tmp_path, monkeypatch, github, stamped=False) + + def stops(**_: object) -> JobOutcome: + raise ServiceStoppingError("the Service is stopping") + + table = build_stub_table({"metrics": lambda **_: JobOutcome(), "implement": stops}) + with pytest.raises(RoadmapRunError) as ended: + service_jobs(tmp_path)[CONST_SERVICE_LANE_STATUS](POLL, table=table) + + assert ( + list(ended.value.failed_jobs), + _job_count(CONST_SERVICE_LANE_STATUS, "implement", "stopped"), + _job_count(CONST_SERVICE_LANE_STATUS, "implement", "error"), + ) == (["implement"], 1, 0) + + +@pytest.mark.parametrize( + ("raised", "result"), + [ + (RuntimeError("the store failed"), "error"), + (ServiceStoppingError("the Service is stopping"), "stopped"), + ], +) +def test_a_scoped_update_that_raises_counts_under_the_result_of_its_error( + raised: Exception, result: str, tmp_path: Path +) -> None: + """A scoped update counts as a row does: a stop under `stopped`, a failure under `error`.""" + GLOBAL_METRICS.reset() + + class RaisingModel(_IntakeModel): + def embed(self, texts: Sequence[str]) -> list[list[float]]: + raise raised + + store, _, _ = _candidates(1) + with pytest.raises(type(raised)) as ended: + run_scoped_update( + _delivery("opened", 1), store, data_dir=tmp_path, now=NOW, model=RaisingModel() ) - assert (count("metrics", "success"), count("implement", "error")) == (1, 1) + assert (ended.value, _job_count(CONST_SERVICE_LANE_TRIAGE, "intake", result)) == (raised, 1) diff --git a/tests/telemetry/test_service_counter_series.py b/tests/telemetry/test_service_counter_series.py index dc476c1f6..e61f7554d 100644 --- a/tests/telemetry/test_service_counter_series.py +++ b/tests/telemetry/test_service_counter_series.py @@ -12,6 +12,7 @@ from devops_cli.config.constants import ( CONST_ROADMAP_WEBHOOK_EVENTS, CONST_SERVICE_EVENT_OTHER, + CONST_SERVICE_METRIC_BUDGET_HOLDS, CONST_SERVICE_METRIC_CHAT_COMMANDS, CONST_SERVICE_METRIC_IMPLEMENT_FAILURES, CONST_SERVICE_METRIC_JOBS, @@ -98,6 +99,20 @@ def test_jobs_total_carries_row_label() -> None: ) +def test_jobs_total_exists_at_zero_for_each_result_of_a_row() -> None: + _manager(REPOS) + + series = _series(CONST_SERVICE_METRIC_JOBS) + expected = { + frozenset( + {("repo", "example/app"), ("lane", "status"), ("row", "metrics"), ("result", result)} + ) + for result in ("success", "error", "stopped") + } + + assert {key: series.get(key) for key in expected} == dict.fromkeys(expected, 0.0) + + def test_webhook_counter_exists_at_zero_for_each_declared_event_and_outcome() -> None: create_webhook_router(_manager(REPOS)) @@ -141,3 +156,11 @@ def test_chat_command_counter_exists_at_zero_for_each_result() -> None: } assert {key: series.get(key) for key in expected} == dict.fromkeys(expected, 0.0) + + +def test_budget_hold_counter_exists_at_zero() -> None: + """The hold counter has no label, and it has no series until its first count without a + prime, so its first hold does not show in `increase()`.""" + _manager(REPOS) + + assert _series(CONST_SERVICE_METRIC_BUDGET_HOLDS) == {frozenset(): 0.0} From e354bfd825a956d2c37231aa4a0880e1e3dc8055 Mon Sep 17 00:00:00 2001 From: Daniel Petty Date: Thu, 5 Nov 2026 17:56:43 +0000 Subject: [PATCH 14/17] docs(openspec): archive #3495 service counter series --- .../.openspec.yaml | 0 .../design.md | 0 .../proposal.md | 0 .../specs/roadmap-implement-job/spec.md | 0 .../specs/roadmap-service-counters/spec.md | 4 ++ .../tasks.md | 0 openspec/specs/roadmap-implement-job/spec.md | 10 +-- .../specs/roadmap-service-counters/spec.md | 71 +++++++++++++++++++ 8 files changed, 80 insertions(+), 5 deletions(-) rename openspec/changes/{3495-service-counters-zero-series-repo-label => archive/2026-11-05-3495-service-counters-zero-series-repo-label}/.openspec.yaml (100%) rename openspec/changes/{3495-service-counters-zero-series-repo-label => archive/2026-11-05-3495-service-counters-zero-series-repo-label}/design.md (100%) rename openspec/changes/{3495-service-counters-zero-series-repo-label => archive/2026-11-05-3495-service-counters-zero-series-repo-label}/proposal.md (100%) rename openspec/changes/{3495-service-counters-zero-series-repo-label => archive/2026-11-05-3495-service-counters-zero-series-repo-label}/specs/roadmap-implement-job/spec.md (100%) rename openspec/changes/{3495-service-counters-zero-series-repo-label => archive/2026-11-05-3495-service-counters-zero-series-repo-label}/specs/roadmap-service-counters/spec.md (96%) rename openspec/changes/{3495-service-counters-zero-series-repo-label => archive/2026-11-05-3495-service-counters-zero-series-repo-label}/tasks.md (100%) create mode 100644 openspec/specs/roadmap-service-counters/spec.md diff --git a/openspec/changes/3495-service-counters-zero-series-repo-label/.openspec.yaml b/openspec/changes/archive/2026-11-05-3495-service-counters-zero-series-repo-label/.openspec.yaml similarity index 100% rename from openspec/changes/3495-service-counters-zero-series-repo-label/.openspec.yaml rename to openspec/changes/archive/2026-11-05-3495-service-counters-zero-series-repo-label/.openspec.yaml diff --git a/openspec/changes/3495-service-counters-zero-series-repo-label/design.md b/openspec/changes/archive/2026-11-05-3495-service-counters-zero-series-repo-label/design.md similarity index 100% rename from openspec/changes/3495-service-counters-zero-series-repo-label/design.md rename to openspec/changes/archive/2026-11-05-3495-service-counters-zero-series-repo-label/design.md diff --git a/openspec/changes/3495-service-counters-zero-series-repo-label/proposal.md b/openspec/changes/archive/2026-11-05-3495-service-counters-zero-series-repo-label/proposal.md similarity index 100% rename from openspec/changes/3495-service-counters-zero-series-repo-label/proposal.md rename to openspec/changes/archive/2026-11-05-3495-service-counters-zero-series-repo-label/proposal.md diff --git a/openspec/changes/3495-service-counters-zero-series-repo-label/specs/roadmap-implement-job/spec.md b/openspec/changes/archive/2026-11-05-3495-service-counters-zero-series-repo-label/specs/roadmap-implement-job/spec.md similarity index 100% rename from openspec/changes/3495-service-counters-zero-series-repo-label/specs/roadmap-implement-job/spec.md rename to openspec/changes/archive/2026-11-05-3495-service-counters-zero-series-repo-label/specs/roadmap-implement-job/spec.md diff --git a/openspec/changes/3495-service-counters-zero-series-repo-label/specs/roadmap-service-counters/spec.md b/openspec/changes/archive/2026-11-05-3495-service-counters-zero-series-repo-label/specs/roadmap-service-counters/spec.md similarity index 96% rename from openspec/changes/3495-service-counters-zero-series-repo-label/specs/roadmap-service-counters/spec.md rename to openspec/changes/archive/2026-11-05-3495-service-counters-zero-series-repo-label/specs/roadmap-service-counters/spec.md index 40db7223a..3cc47aab3 100644 --- a/openspec/changes/3495-service-counters-zero-series-repo-label/specs/roadmap-service-counters/spec.md +++ b/openspec/changes/archive/2026-11-05-3495-service-counters-zero-series-repo-label/specs/roadmap-service-counters/spec.md @@ -1,3 +1,7 @@ +## Purpose + +The label sets of the counters of the roadmap Service, which the Service creates at zero when it starts, the labels that these counters carry, and the alerts that read them. + ## ADDED Requirements ### Requirement: The Service starts each label set of its counters at zero diff --git a/openspec/changes/3495-service-counters-zero-series-repo-label/tasks.md b/openspec/changes/archive/2026-11-05-3495-service-counters-zero-series-repo-label/tasks.md similarity index 100% rename from openspec/changes/3495-service-counters-zero-series-repo-label/tasks.md rename to openspec/changes/archive/2026-11-05-3495-service-counters-zero-series-repo-label/tasks.md diff --git a/openspec/specs/roadmap-implement-job/spec.md b/openspec/specs/roadmap-implement-job/spec.md index 9e5b3b0de..51d4476b3 100644 --- a/openspec/specs/roadmap-implement-job/spec.md +++ b/openspec/specs/roadmap-implement-job/spec.md @@ -771,15 +771,15 @@ The stop comment of the job SHALL name the failure class, the action taken and t - **THEN** `implement.failure` is removed with `implement.stopped` ### Requirement: The Service counts the failures by class -When a round reports a Job that left a stop, the Service SHALL add 1 to `devops_cli_service_implement_failures_total` with the label `class`. A Job that left the item without a result SHALL get the class of the tail of its log, recorded in `implement.failure` and named, with the action left, in the report. A Job that ended with a result SHALL add nothing. +When a round reports a Job that left a stop, the Service SHALL add 1 to `devops_cli_service_implement_failures_total` with the labels `class` and `repo`, where `repo` is the repository of the round. A Job that left the item without a result SHALL get the class of the tail of its log, recorded in `implement.failure` and named, with the action left, in the report. A Job that ended with a result SHALL add nothing. #### Scenario: A stop of the runner -- **WHEN** a round reports a Job whose item holds the stop with the class `parser` -- **THEN** the counter for `parser` grows by 1, the counter for `oom` does not change, and a second round adds nothing +- **WHEN** a round of `example/app` reports a Job whose item holds the stop with the class `parser` +- **THEN** the counter for `parser` and `example/app` grows by 1, the counter for `oom` does not change, and a second round adds nothing #### Scenario: A Job with no result -- **WHEN** a round reports a Job that failed and left the item In Progress, and its log holds "out of memory" -- **THEN** the item is Blocked with `implement.failure` `oom`, the report names the class and the action left `raise_limit`, and the counter for `oom` grows by 1 +- **WHEN** a round of `example/app` reports a Job that failed and left the item In Progress, and its log holds "out of memory" +- **THEN** the item is Blocked with `implement.failure` `oom`, the report names the class and the action left `raise_limit`, and the counter for `oom` and `example/app` grows by 1 #### Scenario: A Job with a result - **WHEN** a round reports a Job that succeeded and left the item In Review diff --git a/openspec/specs/roadmap-service-counters/spec.md b/openspec/specs/roadmap-service-counters/spec.md new file mode 100644 index 000000000..6045728df --- /dev/null +++ b/openspec/specs/roadmap-service-counters/spec.md @@ -0,0 +1,71 @@ +# roadmap-service-counters Specification + +## Purpose +The label sets of the counters of the roadmap Service, which the Service creates at zero when it starts, the labels that these counters carry, and the alerts that read them. + +## Requirements + +### Requirement: The Service starts each label set of its counters at zero +The Service SHALL create each bounded label set of its counters at zero when it starts. The bounded label sets are: each failure class for each repository of `service.repos` on `devops_cli_service_implement_failures_total`; each repository, lane, row and result on `devops_cli_service_jobs_total`; each repository and lane on `devops_cli_service_job_seconds_total`; each provider, declared event or `other`, and outcome on `devops_cli_service_webhook_deliveries_total`. + +#### Scenario: The failure counter exists at zero after start +- **WHEN** the Service starts with the repositories `example/app` and `example/tools` +- **THEN** the failure counter exists at 0 for each of the 19 failure classes and each of the two repositories + +#### Scenario: The label repo holds only configured repositories +- **WHEN** the Service starts with the repositories `example/app` and `example/tools` +- **THEN** every value of the label `repo` of the failure counter is `example/app` or `example/tools` + +#### Scenario: The webhook counter exists at zero for each declared event +- **WHEN** the Service starts with a provider that declares the events `issues` and `push` +- **THEN** the webhook counter exists at 0 for each of the outcomes with the events `issues`, `push` and `other` + +### Requirement: The Service starts the trigger and chat counters at zero +The Service SHALL create each bounded label set of `devops_cli_service_triggers_total` and `devops_cli_service_chat_commands_total` at zero when it starts. The bounded label sets are: each repository of `service.repos` with each source `webhook` or `poll`; and each result `usage`, `status`, `unknown_repo` or `error` of the `/roadmap` chat command. + +#### Scenario: The trigger and chat counters exist at zero after start +- **WHEN** the Service starts with the repositories `example/app` and `example/tools` +- **THEN** the trigger counter exists at 0 for each repository with the sources `webhook` and `poll`, and the chat counter exists at 0 for each result `usage`, `status`, `unknown_repo` and `error` + +### Requirement: The Service starts the budget hold counter at zero +The Service SHALL create the one series of `devops_cli_service_budget_holds_total` at zero when it starts. + +#### Scenario: The budget hold counter exists at zero after start +- **WHEN** the Service starts +- **THEN** the budget hold counter exists at 0 with no label + +### Requirement: The job counter carries the row of the job +The Service SHALL add 1 to `devops_cli_service_jobs_total` with the labels `repo`, `lane`, `row` and `result` for each row of a lane that ends in the roadmap run. A scoped update SHALL add 1 with the name of its job as the label `row`. The Service SHALL NOT count a lane batch as one job. The label `result` SHALL be `success`, `error` or `stopped`. A row or a scoped update that the stop of the Service ends SHALL count under `stopped`, and not under `error`. + +#### Scenario: A row that succeeds +- **WHEN** the status lane runs the row `metrics` for `example/app` and the row succeeds +- **THEN** the job counter for `repo` `example/app`, `lane` `status`, `row` `metrics` and `result` `success` grows by 1 + +#### Scenario: Two rows of one batch +- **WHEN** one round of the status lane runs the rows `metrics` and `implement`, and both fail +- **THEN** the job counter for each row with `result` `error` grows by 1, and no series counts the batch + +#### Scenario: A row that the stop of the Service ends +- **WHEN** the stop of the Service ends the row `implement` of the status lane +- **THEN** the job counter of that row with `result` `stopped` grows by 1, and the counter with `result` `error` does not change + +### Requirement: The job failure alert reads each row +`RoadmapServiceJobsFailing` SHALL group and join on `repo`, `lane` and `row`, so that a success of another row of the lane does not hide a row whose jobs all failed. + +#### Scenario: A row fails while another row of its lane succeeds +- **WHEN** `promtool test rules` evaluates a status lane whose row `implement` fails each round and whose row `metrics` succeeds +- **THEN** `RoadmapServiceJobsFailing` fires with the labels `repo`, `lane` `status` and `row` `implement` + +### Requirement: Alerts read a counter that starts at zero with increase +The alerts `RoadmapServiceWebhooksRejected` and `RoadmapServiceSelfDeliveriesHigh` SHALL read their counter with `increase()` and no `offset` term, because the Service starts each label set at zero. + +#### Scenario: A rejected burst on a new pod +- **WHEN** `promtool test rules` evaluates the rejected series of a pod that starts at 0 and reaches 25 within 15 minutes +- **THEN** `RoadmapServiceWebhooksRejected` fires + +### Requirement: A query gives the true total over a pod change +A query of `sum(increase(devops_cli_service_implement_failures_total[24h]))` SHALL give the true total of the counts of all pods in the window, including the counts of a pod that started during the window. + +#### Scenario: A new pod adds its counts +- **WHEN** `promtool test rules` evaluates the failure counter of a pod that counted 3 and of a new pod that counted 2 after a zero sample at its start +- **THEN** the sum of `increase()` over the window is 5 From a545aaaf9e817b07458f5a85ade53e091defac9b Mon Sep 17 00:00:00 2001 From: Daniel Petty Date: Thu, 5 Nov 2026 18:10:16 +0000 Subject: [PATCH 15/17] Revert "docs(openspec): archive #3495 service counter series" This reverts commit 5a26c38a746a40122171d4b27d3bb5dd2ff05aa3. --- .../.openspec.yaml | 0 .../design.md | 0 .../proposal.md | 0 .../specs/roadmap-implement-job/spec.md | 0 .../specs/roadmap-service-counters/spec.md | 4 -- .../tasks.md | 0 openspec/specs/roadmap-implement-job/spec.md | 10 +-- .../specs/roadmap-service-counters/spec.md | 71 ------------------- 8 files changed, 5 insertions(+), 80 deletions(-) rename openspec/changes/{archive/2026-11-05-3495-service-counters-zero-series-repo-label => 3495-service-counters-zero-series-repo-label}/.openspec.yaml (100%) rename openspec/changes/{archive/2026-11-05-3495-service-counters-zero-series-repo-label => 3495-service-counters-zero-series-repo-label}/design.md (100%) rename openspec/changes/{archive/2026-11-05-3495-service-counters-zero-series-repo-label => 3495-service-counters-zero-series-repo-label}/proposal.md (100%) rename openspec/changes/{archive/2026-11-05-3495-service-counters-zero-series-repo-label => 3495-service-counters-zero-series-repo-label}/specs/roadmap-implement-job/spec.md (100%) rename openspec/changes/{archive/2026-11-05-3495-service-counters-zero-series-repo-label => 3495-service-counters-zero-series-repo-label}/specs/roadmap-service-counters/spec.md (96%) rename openspec/changes/{archive/2026-11-05-3495-service-counters-zero-series-repo-label => 3495-service-counters-zero-series-repo-label}/tasks.md (100%) delete mode 100644 openspec/specs/roadmap-service-counters/spec.md diff --git a/openspec/changes/archive/2026-11-05-3495-service-counters-zero-series-repo-label/.openspec.yaml b/openspec/changes/3495-service-counters-zero-series-repo-label/.openspec.yaml similarity index 100% rename from openspec/changes/archive/2026-11-05-3495-service-counters-zero-series-repo-label/.openspec.yaml rename to openspec/changes/3495-service-counters-zero-series-repo-label/.openspec.yaml diff --git a/openspec/changes/archive/2026-11-05-3495-service-counters-zero-series-repo-label/design.md b/openspec/changes/3495-service-counters-zero-series-repo-label/design.md similarity index 100% rename from openspec/changes/archive/2026-11-05-3495-service-counters-zero-series-repo-label/design.md rename to openspec/changes/3495-service-counters-zero-series-repo-label/design.md diff --git a/openspec/changes/archive/2026-11-05-3495-service-counters-zero-series-repo-label/proposal.md b/openspec/changes/3495-service-counters-zero-series-repo-label/proposal.md similarity index 100% rename from openspec/changes/archive/2026-11-05-3495-service-counters-zero-series-repo-label/proposal.md rename to openspec/changes/3495-service-counters-zero-series-repo-label/proposal.md diff --git a/openspec/changes/archive/2026-11-05-3495-service-counters-zero-series-repo-label/specs/roadmap-implement-job/spec.md b/openspec/changes/3495-service-counters-zero-series-repo-label/specs/roadmap-implement-job/spec.md similarity index 100% rename from openspec/changes/archive/2026-11-05-3495-service-counters-zero-series-repo-label/specs/roadmap-implement-job/spec.md rename to openspec/changes/3495-service-counters-zero-series-repo-label/specs/roadmap-implement-job/spec.md diff --git a/openspec/changes/archive/2026-11-05-3495-service-counters-zero-series-repo-label/specs/roadmap-service-counters/spec.md b/openspec/changes/3495-service-counters-zero-series-repo-label/specs/roadmap-service-counters/spec.md similarity index 96% rename from openspec/changes/archive/2026-11-05-3495-service-counters-zero-series-repo-label/specs/roadmap-service-counters/spec.md rename to openspec/changes/3495-service-counters-zero-series-repo-label/specs/roadmap-service-counters/spec.md index 3cc47aab3..40db7223a 100644 --- a/openspec/changes/archive/2026-11-05-3495-service-counters-zero-series-repo-label/specs/roadmap-service-counters/spec.md +++ b/openspec/changes/3495-service-counters-zero-series-repo-label/specs/roadmap-service-counters/spec.md @@ -1,7 +1,3 @@ -## Purpose - -The label sets of the counters of the roadmap Service, which the Service creates at zero when it starts, the labels that these counters carry, and the alerts that read them. - ## ADDED Requirements ### Requirement: The Service starts each label set of its counters at zero diff --git a/openspec/changes/archive/2026-11-05-3495-service-counters-zero-series-repo-label/tasks.md b/openspec/changes/3495-service-counters-zero-series-repo-label/tasks.md similarity index 100% rename from openspec/changes/archive/2026-11-05-3495-service-counters-zero-series-repo-label/tasks.md rename to openspec/changes/3495-service-counters-zero-series-repo-label/tasks.md diff --git a/openspec/specs/roadmap-implement-job/spec.md b/openspec/specs/roadmap-implement-job/spec.md index 51d4476b3..9e5b3b0de 100644 --- a/openspec/specs/roadmap-implement-job/spec.md +++ b/openspec/specs/roadmap-implement-job/spec.md @@ -771,15 +771,15 @@ The stop comment of the job SHALL name the failure class, the action taken and t - **THEN** `implement.failure` is removed with `implement.stopped` ### Requirement: The Service counts the failures by class -When a round reports a Job that left a stop, the Service SHALL add 1 to `devops_cli_service_implement_failures_total` with the labels `class` and `repo`, where `repo` is the repository of the round. A Job that left the item without a result SHALL get the class of the tail of its log, recorded in `implement.failure` and named, with the action left, in the report. A Job that ended with a result SHALL add nothing. +When a round reports a Job that left a stop, the Service SHALL add 1 to `devops_cli_service_implement_failures_total` with the label `class`. A Job that left the item without a result SHALL get the class of the tail of its log, recorded in `implement.failure` and named, with the action left, in the report. A Job that ended with a result SHALL add nothing. #### Scenario: A stop of the runner -- **WHEN** a round of `example/app` reports a Job whose item holds the stop with the class `parser` -- **THEN** the counter for `parser` and `example/app` grows by 1, the counter for `oom` does not change, and a second round adds nothing +- **WHEN** a round reports a Job whose item holds the stop with the class `parser` +- **THEN** the counter for `parser` grows by 1, the counter for `oom` does not change, and a second round adds nothing #### Scenario: A Job with no result -- **WHEN** a round of `example/app` reports a Job that failed and left the item In Progress, and its log holds "out of memory" -- **THEN** the item is Blocked with `implement.failure` `oom`, the report names the class and the action left `raise_limit`, and the counter for `oom` and `example/app` grows by 1 +- **WHEN** a round reports a Job that failed and left the item In Progress, and its log holds "out of memory" +- **THEN** the item is Blocked with `implement.failure` `oom`, the report names the class and the action left `raise_limit`, and the counter for `oom` grows by 1 #### Scenario: A Job with a result - **WHEN** a round reports a Job that succeeded and left the item In Review diff --git a/openspec/specs/roadmap-service-counters/spec.md b/openspec/specs/roadmap-service-counters/spec.md deleted file mode 100644 index 6045728df..000000000 --- a/openspec/specs/roadmap-service-counters/spec.md +++ /dev/null @@ -1,71 +0,0 @@ -# roadmap-service-counters Specification - -## Purpose -The label sets of the counters of the roadmap Service, which the Service creates at zero when it starts, the labels that these counters carry, and the alerts that read them. - -## Requirements - -### Requirement: The Service starts each label set of its counters at zero -The Service SHALL create each bounded label set of its counters at zero when it starts. The bounded label sets are: each failure class for each repository of `service.repos` on `devops_cli_service_implement_failures_total`; each repository, lane, row and result on `devops_cli_service_jobs_total`; each repository and lane on `devops_cli_service_job_seconds_total`; each provider, declared event or `other`, and outcome on `devops_cli_service_webhook_deliveries_total`. - -#### Scenario: The failure counter exists at zero after start -- **WHEN** the Service starts with the repositories `example/app` and `example/tools` -- **THEN** the failure counter exists at 0 for each of the 19 failure classes and each of the two repositories - -#### Scenario: The label repo holds only configured repositories -- **WHEN** the Service starts with the repositories `example/app` and `example/tools` -- **THEN** every value of the label `repo` of the failure counter is `example/app` or `example/tools` - -#### Scenario: The webhook counter exists at zero for each declared event -- **WHEN** the Service starts with a provider that declares the events `issues` and `push` -- **THEN** the webhook counter exists at 0 for each of the outcomes with the events `issues`, `push` and `other` - -### Requirement: The Service starts the trigger and chat counters at zero -The Service SHALL create each bounded label set of `devops_cli_service_triggers_total` and `devops_cli_service_chat_commands_total` at zero when it starts. The bounded label sets are: each repository of `service.repos` with each source `webhook` or `poll`; and each result `usage`, `status`, `unknown_repo` or `error` of the `/roadmap` chat command. - -#### Scenario: The trigger and chat counters exist at zero after start -- **WHEN** the Service starts with the repositories `example/app` and `example/tools` -- **THEN** the trigger counter exists at 0 for each repository with the sources `webhook` and `poll`, and the chat counter exists at 0 for each result `usage`, `status`, `unknown_repo` and `error` - -### Requirement: The Service starts the budget hold counter at zero -The Service SHALL create the one series of `devops_cli_service_budget_holds_total` at zero when it starts. - -#### Scenario: The budget hold counter exists at zero after start -- **WHEN** the Service starts -- **THEN** the budget hold counter exists at 0 with no label - -### Requirement: The job counter carries the row of the job -The Service SHALL add 1 to `devops_cli_service_jobs_total` with the labels `repo`, `lane`, `row` and `result` for each row of a lane that ends in the roadmap run. A scoped update SHALL add 1 with the name of its job as the label `row`. The Service SHALL NOT count a lane batch as one job. The label `result` SHALL be `success`, `error` or `stopped`. A row or a scoped update that the stop of the Service ends SHALL count under `stopped`, and not under `error`. - -#### Scenario: A row that succeeds -- **WHEN** the status lane runs the row `metrics` for `example/app` and the row succeeds -- **THEN** the job counter for `repo` `example/app`, `lane` `status`, `row` `metrics` and `result` `success` grows by 1 - -#### Scenario: Two rows of one batch -- **WHEN** one round of the status lane runs the rows `metrics` and `implement`, and both fail -- **THEN** the job counter for each row with `result` `error` grows by 1, and no series counts the batch - -#### Scenario: A row that the stop of the Service ends -- **WHEN** the stop of the Service ends the row `implement` of the status lane -- **THEN** the job counter of that row with `result` `stopped` grows by 1, and the counter with `result` `error` does not change - -### Requirement: The job failure alert reads each row -`RoadmapServiceJobsFailing` SHALL group and join on `repo`, `lane` and `row`, so that a success of another row of the lane does not hide a row whose jobs all failed. - -#### Scenario: A row fails while another row of its lane succeeds -- **WHEN** `promtool test rules` evaluates a status lane whose row `implement` fails each round and whose row `metrics` succeeds -- **THEN** `RoadmapServiceJobsFailing` fires with the labels `repo`, `lane` `status` and `row` `implement` - -### Requirement: Alerts read a counter that starts at zero with increase -The alerts `RoadmapServiceWebhooksRejected` and `RoadmapServiceSelfDeliveriesHigh` SHALL read their counter with `increase()` and no `offset` term, because the Service starts each label set at zero. - -#### Scenario: A rejected burst on a new pod -- **WHEN** `promtool test rules` evaluates the rejected series of a pod that starts at 0 and reaches 25 within 15 minutes -- **THEN** `RoadmapServiceWebhooksRejected` fires - -### Requirement: A query gives the true total over a pod change -A query of `sum(increase(devops_cli_service_implement_failures_total[24h]))` SHALL give the true total of the counts of all pods in the window, including the counts of a pod that started during the window. - -#### Scenario: A new pod adds its counts -- **WHEN** `promtool test rules` evaluates the failure counter of a pod that counted 3 and of a new pod that counted 2 after a zero sample at its start -- **THEN** the sum of `increase()` over the window is 5 From 5a88cb0e2a2ab061622bc24bc8876b7c13039dc3 Mon Sep 17 00:00:00 2001 From: Daniel Petty Date: Thu, 5 Nov 2026 18:11:28 +0000 Subject: [PATCH 16/17] docs(telemetry): state the failure classes and labels as the code holds them k8s/README.md gave a false count of classes, and docs/ROADMAP_WORKFLOW.md did not name the repo label. The records of the change now name the true owner of the scoped count, the true priming site, the real promtool case and the window of the query. The tables that only counter_series.py reads get private names. --- docs/ROADMAP_WORKFLOW.md | 2 +- k8s/README.md | 2 +- .../design.md | 4 ++-- .../proposal.md | 6 +++--- .../specs/roadmap-service-counters/spec.md | 6 +++--- .../tasks.md | 10 +++++----- src/devops_cli/server/counter_series.py | 12 ++++++------ 7 files changed, 21 insertions(+), 21 deletions(-) diff --git a/docs/ROADMAP_WORKFLOW.md b/docs/ROADMAP_WORKFLOW.md index 56f6250c4..6ede5563d 100644 --- a/docs/ROADMAP_WORKFLOW.md +++ b/docs/ROADMAP_WORKFLOW.md @@ -140,7 +140,7 @@ Each clear by digest adds an entry (the class, the new digest and the time) to ` ## The failure classes -A stop has a class (`roadmap/failure.py`, #3065). A stop site that knows its cause passes the class itself (#3476): the classes `no_change_in_fix`, `tests_lost`, `head_moved`, `output_cap` and `limit_reached` come from the step, and no rule makes them. For any other stop, the classifier is a pure function of the error text, the exit status, the refusal of a tool, the name of the check and the log of the check. It never reads the last message of the agent, which is a claim that no check verified. A failure that no rule matches is `ci_infra`. The stop comment starts with the failed check and the end of its log, then names the class, the action taken and the action left. It shows the last message of the agent as a claim that no check verified. The Job log holds one line with the class and the action left. The Service adds 1 to `devops_cli_service_implement_failures_total` with the label `class` when a round reports the Job. A Job that left the Item with no result gets the class of the tail of its log. +A stop has a class (`roadmap/failure.py`, #3065). A stop site that knows its cause passes the class itself (#3476): the classes `no_change_in_fix`, `tests_lost`, `head_moved`, `output_cap` and `limit_reached` come from the step, and no rule makes them. For any other stop, the classifier is a pure function of the error text, the exit status, the refusal of a tool, the name of the check and the log of the check. It never reads the last message of the agent, which is a claim that no check verified. A failure that no rule matches is `ci_infra`. The stop comment starts with the failed check and the end of its log, then names the class, the action taken and the action left. It shows the last message of the agent as a claim that no check verified. The Job log holds one line with the class and the action left. The Service adds 1 to `devops_cli_service_implement_failures_total` with the labels `class` and `repo` when a round reports the Job. A Job that left the Item with no result gets the class of the tail of its log. | Class | Default action | Meaning | | :--- | :--- | :--- | diff --git a/k8s/README.md b/k8s/README.md index 5c4b85679..4951312fd 100644 --- a/k8s/README.md +++ b/k8s/README.md @@ -627,7 +627,7 @@ kubectl -n devops logs -f job/ 9. To rotate a credential, update it in the keyring: `uv run devops config set service.webhook_secrets`, or, for the App, generate a new private key and run `uv run devops config set service.github_app.private_key --from-file `. Then run `uv run devops k8s push-secrets --only devops/devops-cli --rotate`. It restarts `roadmap-service`. Delete the old private key on the App's page after the restart. 10. Run one service per set of repos. While it runs, use `devops roadmap run --dry-run` (#981). 11. The Service keeps three files in `/home/devops/.data/roadmap/` on volume `roadmap-service-home`, beside the `schedule.json` of each repository: `service-halt.json` (GitHub refused the identity), `service-budget.json` (the GitHub requests of the last hour) and `service-failures.json` (the jobs that failed rounds in a row). A restart reads them back, so a restart does not clear a halt, the budget or a suspended job. A round that ends because the Service is stopping, for example in a rollout, does not count as a failed round: the count of its job stays as it was. The log line that suspends a job names the class of the error of each of its failed rounds in a row; the file keeps only the count, so a round that an earlier process counted reads as `unknown`. A rate limit that one request gets stops the requests of all workers until its wait ends. When GitHub refuses the identity, for example because the App's installation is suspended, the Service halts and logs `GitHub refused the identity`. To clear the halt after you fix the cause, remove the file and restart the Service: `kubectl -n devops exec deploy/roadmap-service -- rm /home/devops/.data/roadmap/service-halt.json`, then `kubectl -n devops delete pod -l app.kubernetes.io/name=roadmap-service`. -12. The Service's `/metrics` shows the budget and the suspended jobs (#2475): `devops_cli_service_budget_limit_requests` (the hourly budget), `devops_cli_service_budget_spent_requests` (the GitHub requests of the last hour), `devops_cli_service_budget_holds_total` (one count each time a hold begins), `devops_cli_service_job_suspended` (1 while a job of a repository, by `repo` and `row`, waits for new input after it failed 3 rounds in a row) and `devops_cli_service_implement_failures_total` (one count for each runner Job that left a stop, by the failure `class` and `repo`, when a round reports the Job; #3065; the 19 values of `class` are the classes in [docs/ROADMAP_WORKFLOW.md](../docs/ROADMAP_WORKFLOW.md), The failure classes, and a panel or rule matches no other value, #3476). Each counter of the Service starts its label sets at zero (#3495, [docs/TELEMETRY.md](../docs/TELEMETRY.md)), except `devops_cli_service_identity_refused_total`, whose label set starts with the first refusal. Each GitHub request counts in `devops_cli_github_requests_total` under its `method`, the `repo` of the worker that sent it and the job `row` (`metrics`, `intake`, `refine`, `stage`, `implement`, `stage_ready` or `alternatives`); a request outside a worker or a row counts as `other`. To find what spent the budget, run `sum by (repo, row, method) (increase(devops_cli_github_requests_total{job="roadmap-service"}[1h]))` in Grafana Explore. A delivery of the App bot's own write counts as outcome `self` in `devops_cli_service_webhook_deliveries_total`. +12. The Service's `/metrics` shows the budget and the suspended jobs (#2475): `devops_cli_service_budget_limit_requests` (the hourly budget), `devops_cli_service_budget_spent_requests` (the GitHub requests of the last hour), `devops_cli_service_budget_holds_total` (one count each time a hold begins), `devops_cli_service_job_suspended` (1 while a job of a repository, by `repo` and `row`, waits for new input after it failed 3 rounds in a row) and `devops_cli_service_implement_failures_total` (one count for each runner Job that left a stop, by the failure `class` and `repo`, when a round reports the Job; #3065; the values of `class` are the classes in [docs/ROADMAP_WORKFLOW.md](../docs/ROADMAP_WORKFLOW.md), The failure classes, and a panel or rule matches no other value, #3476). Each counter of the Service starts its label sets at zero (#3495, [docs/TELEMETRY.md](../docs/TELEMETRY.md)), except `devops_cli_service_identity_refused_total`, whose label set starts with the first refusal. Each GitHub request counts in `devops_cli_github_requests_total` under its `method`, the `repo` of the worker that sent it and the job `row` (`metrics`, `intake`, `refine`, `stage`, `implement`, `stage_ready` or `alternatives`); a request outside a worker or a row counts as `other`. To find what spent the budget, run `sum by (repo, row, method) (increase(devops_cli_github_requests_total{job="roadmap-service"}[1h]))` in Grafana Explore. A delivery of the App bot's own write counts as outcome `self` in `devops_cli_service_webhook_deliveries_total`. 13. A refine round writes at most 2 comments on one item in 24 hours. A round that finds 2 refine comments of the last 24 hours on an item leaves the item alone, with no model call and no comment. The item waits on a person and takes no place under the round's limit, so the round refines the next item. For each newest comment on the item that the cap leaves with no answer, the round records the time of that comment in the item's job record (`refine.capped_after`), logs `Refine left # alone and wrote no comment`, and adds 1 to `devops_cli_roadmap_refine_comments_capped_total`. A later round that finds no newer comment does not log or count the item again. A person refines the item at the cap with `devops roadmap refine --item --confirm`. Each Firecrawl search counts in `devops_cli_firecrawl_searches_total` by `result`: `ok`, the HTTP status of a refused search, or `error`. A failed search logs `A Firecrawl web search failed` with the status or the error class, and never the key or the query. 14. The row `alternatives` of the model lane runs after `refine`: on a first run, and 719 minutes after its last success. It runs `devops roadmap alternatives --confirm` for the repository. A round checks at most `roadmap.alternatives.first_pass_limit` items (20 by default) while an item has no check, and at most `roadmap.alternatives.limit` items (5 by default) after. The row runs about twice a day, so a first pass of 141 items takes about 4 days. A round takes first the items with no check, then the items whose check is older than `roadmap.alternatives.recheck_days` (30 by default) or whose text changed. A webhook makes the row due at no other time. The row changes no label and closes no issue. It does nothing, and logs `The alternatives check of is off`, for a private repository and when no Firecrawl key is set. If the check of an item fails, the row records the failure in the item's job record (`alternatives.failed_at` and `alternatives.failed_error`, the class of the error), writes the other checks and succeeds. The row fails, and counts toward the suspension after 3 failed rounds, only when it tried items and checked none. A round takes a failed item again after `roadmap.alternatives.retry_days` (3 by default). Each written check adds 1 to `devops_cli_roadmap_alternatives_total` by `verdict`: `covers`, `partly`, `none` or `not-applicable`. An item of the type `test`, `docs`, `refactor`, `chore`, `ci` or `perf`, and a `fix` with a Python traceback in its body, gets `not-applicable` with no model call; every other item must run a search before the check accepts a verdict, and gets one more turn to search. A `covers` verdict makes refine ask a person to adopt the tool or to keep the item with a reason. The item waits on a person until a reply answers the question, and Ready items with a new covering tool go back to New. Refine records the tool it asked about in the job record (`refine.alternatives_asked`). `devops roadmap run --dry-run` lists the row when it is due. 15. The row `stage` follows `refine` on the model lane (#2659). It runs `devops roadmap stage --confirm` in the model clone: it is due every 37 minutes, on a first run, and in a round where `refine` set an item Ready. It stages the Ready items that `roadmap.stage.filter` names, at most `roadmap.stage.limit` in a round, plans no new item while the count of open Planned items is at `roadmap.stage.planned_limit` (an item with the priority `P0-Critical` passes it, #3487), and splits a task group over `roadmap.stage.step_tokens`. Each task group has a kind (#3005): a `behavior` group has a failing test, and an `other` group has no test but a command or the steps of a manual check that prove it. The job sends a reply that breaks this rule back to the model once. It leaves out an item whose change folder is on the default branch or in its archive (#2714): a round reads that tree once with `git ls-tree`, so a merged staging pull request never makes a second one. The pull request it opens is a draft with the title `docs(openspec): plan # `. The job does not mark it ready: the row `stage_ready` does (step 18). Before it opens one, it checks each path of the plan (`reads`, `changes`, the code spans of the tasks and `verify`) against the tracked files of the commit it plans, read with one `git ls-files` for the round (#3010). A path that is no file there, and that no group up to its own creates, goes back to the model once with the files of the same name. When a later group creates the path, the problem names that group and tells the model to move it before the group that names the path (#3017). A second reply with a missing path stops the item, so no draft opens and the implement Job never meets a wrong path. `k8s/devops/configmap.example.yaml` holds the five settings, and `devops k8s deploy-stack --stack devops` applies them; each also has a variable, `DEVOPS_CLI_ROADMAP_STAGE_FILTER`, `DEVOPS_CLI_ROADMAP_STAGE_LIMIT`, `DEVOPS_CLI_ROADMAP_STAGE_PLANNED_LIMIT`, `DEVOPS_CLI_ROADMAP_STAGE_STEP_TOKENS` and `DEVOPS_CLI_ROADMAP_STAGE_TOOLSET`. The setting `roadmap.stage.toolset` is `extended` by default (#3238): a plan run then has three tools more than the nine that read the staged commit. `references` counts a text in every file of the commit. `read_issue` and `find_issues` read issues and pull requests of the repository on GitHub, as GET requests with the token of the Service and no other permission. `base` leaves them out. Each item of a round counts once in `devops_cli_roadmap_stage_total` by its `result`: `staged`, `skipped`, `needs_split` or `failed`. A failed item is recorded in its job record with the class of its error and the time (#3493), and the job leaves it out for `roadmap.stage.retry_hours` hours (6 by default) while it takes the other items. The row fails a round only when it tried items and staged, stopped or returned none, or when a failure belongs to no item; three such rounds on the same input suspend the row. The ledger file keeps the class of each error, and a round that the stop of the Service ends counts as `stopped`, not as an error. The image carries the pinned OpenSpec and a Node runtime, so the job validates each change before it pushes: `kubectl -n devops exec deploy/roadmap-service -- openspec --version` prints the locked version. diff --git a/openspec/changes/3495-service-counters-zero-series-repo-label/design.md b/openspec/changes/3495-service-counters-zero-series-repo-label/design.md index 9ea97da37..0793d0e71 100644 --- a/openspec/changes/3495-service-counters-zero-series-repo-label/design.md +++ b/openspec/changes/3495-service-counters-zero-series-repo-label/design.md @@ -3,9 +3,9 @@ ## Decisions - **The Service creates the bounded label sets of its counters at zero when it starts.** A labelled counter has no series until `.labels()` makes a child (see the Feasibility of the proposal). One function calls `.labels()` for each bounded label set, so the first scrape of a pod reads a zero series for each label set. -- **The label sets are bounded.** The failure classes are the 19 members of `FailureClass`. The repositories are `service.repos`. The rows are the rows of the three lane tables of the roadmap run. The events of the webhook counter are the declared events of each provider, plus `other`. The results of the job counter are `success`, `error` and `stopped`. The sources of the trigger counter are `webhook` and `poll`. The results of the chat command are `usage`, `status`, `unknown_repo` and `error`. No dynamic label value is created at start. +- **The label sets are bounded.** The failure classes are the 18 members of `FailureClass`. The repositories are `service.repos`. The rows are the rows of the three lane tables of the roadmap run. The events of the webhook counter are the declared events of each provider, plus `other`. The results of the job counter are `success`, `error` and `stopped`. The sources of the trigger counter are `webhook` and `poll`. The results of the chat command are `usage`, `status`, `unknown_repo` and `error`. No dynamic label value is created at start. - **The exposition format stays text 0.0.4.** The Service serves `/metrics` in this format (`src/devops_cli/server/service.py`). The zero children appear in this format, so the change does not switch the format. -- **The jobs counter counts each row.** The roadmap run counts each row of a lane with its result, with the row name of the row table. The worker counts the scoped updates with the name of their job. A row counted in the run is not counted again in the worker, so no job is counted twice. +- **The jobs counter counts each row.** The roadmap run counts each row of a lane with its result, with the row name of the row table. `run_scoped_update` of the roadmap run counts each scoped update with the name of its job. The worker counts no job, so no job is counted twice. - **A stop is no failed row.** A row or a scoped update that the stop of the Service ends counts under `result="stopped"`. `RoadmapServiceJobsFailing` reads only `error` and `success`, so a rollout does not fire it for a row that runs twice a day. - **`RoadmapServiceJobsFailing` reads the row.** The rule groups and joins on `repo`, `lane` and `row`. Before this change, a failed row made its whole round an error. With a count for each row, the row `metrics` succeeds in each status-lane round, so a join on the lane never fires for a failed row of that lane. A promtool case holds this. - **The Loki query of `RoadmapServiceJobsFailing` has no field `row`.** The log line of a failed round has no such field, and its text names the rows. The summary of the alert names the row. diff --git a/openspec/changes/3495-service-counters-zero-series-repo-label/proposal.md b/openspec/changes/3495-service-counters-zero-series-repo-label/proposal.md index 354246b24..b949886df 100644 --- a/openspec/changes/3495-service-counters-zero-series-repo-label/proposal.md +++ b/openspec/changes/3495-service-counters-zero-series-repo-label/proposal.md @@ -13,7 +13,7 @@ Issue #3495 reports that the Service counters give a false picture. The counters - `RoadmapServiceJobsFailing` groups and joins on `repo`, `lane` and `row`. With a count for each row, a join on the lane hides a row that fails while another row of its lane succeeds. - The alert expressions of `RoadmapServiceWebhooksRejected` and `RoadmapServiceSelfDeliveriesHigh` lose their #2505 workaround terms. The OTLP alert expressions keep theirs. - Promtool cases prove the true total over a pod change, and prove that the webhook alerts hold without the workaround. -- `docs/TELEMETRY.md` and `k8s/README.md` state the scrape behavior and the labels. The action text of `RoadmapServiceGitHubRequestsHigh` no longer says to lower the budget. The class count of the Service changes from 17 to 19. +- `docs/TELEMETRY.md` and `k8s/README.md` state the scrape behavior and the labels. The action text of `RoadmapServiceGitHubRequestsHigh` no longer says to lower the budget. `k8s/README.md` gives the values of `class` by a link to the table of classes and no longer states a false count. `docs/ROADMAP_WORKFLOW.md` states the labels of the failure counter. ## Capabilities @@ -30,7 +30,7 @@ Issue #3495 reports that the Service counters give a false picture. The counters - Code: `src/devops_cli/telemetry/metrics.py`, `src/devops_cli/server/service.py`, `src/devops_cli/server/counter_series.py`, `src/devops_cli/roadmap/implement_submit.py`, `src/devops_cli/roadmap/run.py`, `src/devops_cli/server/routes/webhooks.py`, `src/devops_cli/config/constants.py`, `src/devops_cli/docs/generator.py`. - Monitoring: `k8s/monitoring/prometheus-values.yaml`. - Tests: `tests/telemetry/test_service_counter_series.py`, `tests/roadmap/test_roadmap_run.py`, `tests/roadmap/test_roadmap_implement_submit.py`, `tests/server/test_server_service.py`, `tests/prometheus/test_k8s_monitoring_alerting.py`, `tests/fixtures/prometheus_rules/roadmap_service_tests.yaml`, `tests/fixtures/prometheus_rules/roadmap_service_webhooks_tests.yaml`. -- Documents: `docs/TELEMETRY.md`, `k8s/README.md`. +- Documents: `docs/TELEMETRY.md`, `docs/ROADMAP_WORKFLOW.md`, `k8s/README.md`. - The change adds no dependency. ## Feasibility @@ -38,7 +38,7 @@ Issue #3495 reports that the Service counters give a false picture. The counters - The failure counter increments with the label `class` only. This is confirmed by reading `src/devops_cli/roadmap/implement_submit.py` at origin/main, where `GLOBAL_METRICS.increment_counter(CONST_SERVICE_METRIC_IMPLEMENT_FAILURES, labels={"class": failure.value})` runs. - The jobs counter has the labels `repo`, `lane` and `result`, and no `row`. This is confirmed by reading `src/devops_cli/server/service.py` at origin/main, where the worker increments it with `{**self._labels, "result": result}`. - A labelled counter emits no sample until a child exists. This is confirmed by a probe with prometheus_client 0.26.0 in the devcontainer: the counter with the labels `class` and `repo` emitted no line, and after `.labels("oom", "example/app")` it emitted `probe_total{class="oom",repo="example/app"} 0.0`. -- The enum `FailureClass` in `src/devops_cli/roadmap/failure.py` has 19 members. `k8s/README.md:630` says 17. This is confirmed by reading both files at origin/main. +- The enum `FailureClass` in `src/devops_cli/roadmap/failure.py` has 18 members. `k8s/README.md:630` says 17. This is confirmed by reading both files at origin/main. - The action text of `RoadmapServiceGitHubRequestsHigh` in `k8s/README.md:705` says "Lower the budget or remove a repository". This is confirmed by reading the file at origin/main. - `k8s/monitoring/prometheus-values.yaml:23` enables `created-timestamp-zero-ingestion`. The stale comment at lines 670-673 says that neither path has a zero sample. This is confirmed by reading the file at origin/main. - The upstream statement that start-timestamp injection covers only PrometheusProto and OpenMetrics 1.0.0 scrapes is recorded from the issue research and is not checked again here. The design does not depend on it. diff --git a/openspec/changes/3495-service-counters-zero-series-repo-label/specs/roadmap-service-counters/spec.md b/openspec/changes/3495-service-counters-zero-series-repo-label/specs/roadmap-service-counters/spec.md index 40db7223a..246d06542 100644 --- a/openspec/changes/3495-service-counters-zero-series-repo-label/specs/roadmap-service-counters/spec.md +++ b/openspec/changes/3495-service-counters-zero-series-repo-label/specs/roadmap-service-counters/spec.md @@ -5,7 +5,7 @@ The Service SHALL create each bounded label set of its counters at zero when it #### Scenario: The failure counter exists at zero after start - **WHEN** the Service starts with the repositories `example/app` and `example/tools` -- **THEN** the failure counter exists at 0 for each of the 19 failure classes and each of the two repositories +- **THEN** the failure counter exists at 0 for each of the 18 failure classes and each of the two repositories #### Scenario: The label repo holds only configured repositories - **WHEN** the Service starts with the repositories `example/app` and `example/tools` @@ -59,8 +59,8 @@ The alerts `RoadmapServiceWebhooksRejected` and `RoadmapServiceSelfDeliveriesHig - **THEN** `RoadmapServiceWebhooksRejected` fires ### Requirement: A query gives the true total over a pod change -A query of `sum(increase(devops_cli_service_implement_failures_total[24h]))` SHALL give the true total of the counts of all pods in the window, including the counts of a pod that started during the window. +A query of `sum(increase(devops_cli_service_implement_failures_total[1h]))` SHALL give the counts of all pods in the window, including the counts of a pod that started during the window. The only difference from the counts SHALL be the extrapolation of `increase()` to the window edges. #### Scenario: A new pod adds its counts - **WHEN** `promtool test rules` evaluates the failure counter of a pod that counted 3 and of a new pod that counted 2 after a zero sample at its start -- **THEN** the sum of `increase()` over the window is 5 +- **THEN** the sum of `increase()` over the window is the 5 counts plus the extrapolation to the window edges, about 5.03 diff --git a/openspec/changes/3495-service-counters-zero-series-repo-label/tasks.md b/openspec/changes/3495-service-counters-zero-series-repo-label/tasks.md index 59ac297e7..905f93a67 100644 --- a/openspec/changes/3495-service-counters-zero-series-repo-label/tasks.md +++ b/openspec/changes/3495-service-counters-zero-series-repo-label/tasks.md @@ -4,7 +4,7 @@ Reads: `src/devops_cli/roadmap/implement_submit.py`, `src/devops_cli/telemetry/metrics.py`, `src/devops_cli/server/service.py`, `tests/roadmap/test_roadmap_implement_submit.py`. Changes: `src/devops_cli/telemetry/metrics.py`, `src/devops_cli/roadmap/implement_submit.py`, `src/devops_cli/server/service.py`, `src/devops_cli/server/counter_series.py`, `tests/telemetry/test_service_counter_series.py`, `tests/roadmap/test_roadmap_implement_submit.py`. -- [x] 1.1 Add a test to `tests/telemetry/test_service_counter_series.py` that builds the Service manager with two repositories and asserts that the failure counter exists at 0 for each of the 19 classes and each repository +- [x] 1.1 Add a test to `tests/telemetry/test_service_counter_series.py` that builds the Service manager with two repositories and asserts that the failure counter exists at 0 for each of the 18 classes and each repository - Verification: `uv run devops ci test tests/telemetry/test_service_counter_series.py` fails on the base - [x] 1.2 Add a test to `tests/telemetry/test_service_counter_series.py` that asserts that every value of the label `repo` of the failure counter is a value of `service.repos`, and change the counter assertions of `tests/roadmap/test_roadmap_implement_submit.py` to read the label `repo` - Verification: `uv run devops ci test tests/telemetry/test_service_counter_series.py tests/roadmap/test_roadmap_implement_submit.py` fails on the base @@ -28,14 +28,14 @@ Reads: `src/devops_cli/server/routes/webhooks.py`, `src/devops_cli/server/counte - [x] 3.1 Add a test to `tests/telemetry/test_service_counter_series.py` that asserts that the webhook counter exists at 0 for each provider, each declared event with `other`, and each outcome - Verification: `uv run devops ci test tests/telemetry/test_service_counter_series.py` fails on the base -- [x] 3.2 Prime the webhook counter in `src/devops_cli/server/counter_series.py` from the declared providers and their events +- [x] 3.2 Prime the webhook counter in `create_webhook_router` of `src/devops_cli/server/routes/webhooks.py` from the declared providers and their events - Verification: `uv run devops ci test tests/telemetry/test_service_counter_series.py` passes ## 4. The promtool cases hold the rules and the zero series Reads: `tests/fixtures/prometheus_rules/roadmap_service_tests.yaml`, `tests/fixtures/prometheus_rules/roadmap_service_webhooks_tests.yaml`, `tests/prometheus/test_k8s_monitoring_alerting.py`, `k8s/monitoring/prometheus-values.yaml`. Changes: `tests/fixtures/prometheus_rules/roadmap_service_tests.yaml`, `tests/fixtures/prometheus_rules/roadmap_service_webhooks_tests.yaml`, `k8s/monitoring/prometheus-values.yaml`. -- [x] 4.1 Add a promtool case `pod_change_gives_true_total` to `tests/fixtures/prometheus_rules/roadmap_service_tests.yaml`. Its series of the failure counter start at 0 in each pod, and the case asserts the sum of `increase()` over a pod change +- [x] 4.1 Add the promtool case `A pod change keeps the true total of the failure counter, because each pod starts at zero` to `tests/fixtures/prometheus_rules/roadmap_service_tests.yaml`. Its series of the failure counter start at 0 in each pod, and the case asserts the sum of `increase()` over a pod change - Verification: `promtool test rules tests/fixtures/prometheus_rules/roadmap_service_tests.yaml` passes - [x] 4.2 Edit the first case of `tests/fixtures/prometheus_rules/roadmap_service_webhooks_tests.yaml`, so that its rejected series with the event `other` starts at 0 and the case fires `RoadmapServiceWebhooksRejected` at 34 minutes. The case replaces the #2505 gap with the zero samples and adds no new case - Verification: `promtool test rules tests/fixtures/prometheus_rules/roadmap_service_webhooks_tests.yaml` passes @@ -44,11 +44,11 @@ Reads: `tests/fixtures/prometheus_rules/roadmap_service_tests.yaml`, `tests/fixt ## 5. The documents state the scrape behavior and the labels -Reads: `docs/TELEMETRY.md`, `k8s/README.md`. Changes: `docs/TELEMETRY.md`, `k8s/README.md`, `tests/prometheus/test_k8s_monitoring_alerting.py`. +Reads: `docs/TELEMETRY.md`, `docs/ROADMAP_WORKFLOW.md`, `k8s/README.md`. Changes: `docs/TELEMETRY.md`, `docs/ROADMAP_WORKFLOW.md`, `k8s/README.md`, `tests/prometheus/test_k8s_monitoring_alerting.py`. - [x] 5.1 Add a test to `tests/prometheus/test_k8s_monitoring_alerting.py` that asserts that the action text of `RoadmapServiceGitHubRequestsHigh` in `k8s/README.md` does not say to lower the budget - Verification: `uv run devops ci test tests/prometheus/test_k8s_monitoring_alerting.py` fails on the base -- [x] 5.2 Rewrite the action text of `RoadmapServiceGitHubRequestsHigh` in `k8s/README.md` without the word to lower the budget, rewrite `docs/TELEMETRY.md` lines 19-32 to state the scrape behavior and the labels, state the class count of 19 in `k8s/README.md`, and run `uv run devops docs generate --sync-readme` +- [x] 5.2 Rewrite the action text of `RoadmapServiceGitHubRequestsHigh` in `k8s/README.md` without the word to lower the budget, rewrite `docs/TELEMETRY.md` lines 19-32 to state the scrape behavior and the labels, give the values of `class` in `k8s/README.md` by a link to the table of classes, state the labels `class` and `repo` of the failure counter in `docs/ROADMAP_WORKFLOW.md`, and run `uv run devops docs generate --sync-readme` - Verification: `uv run devops ci test tests/prometheus/test_k8s_monitoring_alerting.py` passes and `uv run devops ci docs` passes ## 6. The trigger and chat counters start at zero diff --git a/src/devops_cli/server/counter_series.py b/src/devops_cli/server/counter_series.py index 194f058bc..b0eff11dd 100644 --- a/src/devops_cli/server/counter_series.py +++ b/src/devops_cli/server/counter_series.py @@ -33,17 +33,17 @@ from devops_cli.roadmap.failure import FailureClass from devops_cli.telemetry.metrics import GLOBAL_METRICS -SERVICE_JOB_RESULTS = ( +_SERVICE_JOB_RESULTS = ( CONST_SERVICE_JOB_RESULT_SUCCESS, CONST_SERVICE_JOB_RESULT_ERROR, CONST_SERVICE_JOB_RESULT_STOPPED, ) """The value of the label `result` of a job that ended, as the roadmap run counts it.""" -SERVICE_TRIGGER_SOURCES = (CONST_SERVICE_SOURCE_WEBHOOK, CONST_SERVICE_SOURCE_POLL) +_SERVICE_TRIGGER_SOURCES = (CONST_SERVICE_SOURCE_WEBHOOK, CONST_SERVICE_SOURCE_POLL) """The values of the label `source` of a trigger, as the Service enqueues it.""" -SERVICE_CHAT_RESULTS = ( +_SERVICE_CHAT_RESULTS = ( CONST_SERVICE_CHAT_RESULT_USAGE, CONST_SERVICE_CHAT_RESULT_STATUS, CONST_SERVICE_CHAT_RESULT_UNKNOWN_REPO, @@ -80,7 +80,7 @@ def prime_service_counters(repos: Iterable[str]) -> None: for lane, rows in lane_rows for row in rows for repo in repo_names - for result in SERVICE_JOB_RESULTS + for result in _SERVICE_JOB_RESULTS ), ) GLOBAL_METRICS.prime_counter( @@ -92,14 +92,14 @@ def prime_service_counters(repos: Iterable[str]) -> None: ( {"repo": repo, "source": source} for repo in repo_names - for source in SERVICE_TRIGGER_SOURCES + for source in _SERVICE_TRIGGER_SOURCES ), ) GLOBAL_METRICS.prime_counter( CONST_SERVICE_METRIC_CHAT_COMMANDS, ( {"command": CONST_SERVICE_CHAT_COMMAND_ROADMAP, "result": result} - for result in SERVICE_CHAT_RESULTS + for result in _SERVICE_CHAT_RESULTS ), ) GLOBAL_METRICS.prime_counter(CONST_SERVICE_METRIC_BUDGET_HOLDS, [{}]) From 6fb8ac7be6fa2761d8d6483911e4126b39046783 Mon Sep 17 00:00:00 2001 From: Daniel Petty Date: Thu, 5 Nov 2026 18:34:51 +0000 Subject: [PATCH 17/17] docs(openspec): archive #3495 service counter series --- .../.openspec.yaml | 0 .../design.md | 0 .../proposal.md | 0 .../specs/roadmap-implement-job/spec.md | 0 .../specs/roadmap-service-counters/spec.md | 0 .../tasks.md | 0 openspec/specs/roadmap-implement-job/spec.md | 10 +-- .../specs/roadmap-service-counters/spec.md | 71 +++++++++++++++++++ 8 files changed, 76 insertions(+), 5 deletions(-) rename openspec/changes/{3495-service-counters-zero-series-repo-label => archive/2026-11-05-3495-service-counters-zero-series-repo-label}/.openspec.yaml (100%) rename openspec/changes/{3495-service-counters-zero-series-repo-label => archive/2026-11-05-3495-service-counters-zero-series-repo-label}/design.md (100%) rename openspec/changes/{3495-service-counters-zero-series-repo-label => archive/2026-11-05-3495-service-counters-zero-series-repo-label}/proposal.md (100%) rename openspec/changes/{3495-service-counters-zero-series-repo-label => archive/2026-11-05-3495-service-counters-zero-series-repo-label}/specs/roadmap-implement-job/spec.md (100%) rename openspec/changes/{3495-service-counters-zero-series-repo-label => archive/2026-11-05-3495-service-counters-zero-series-repo-label}/specs/roadmap-service-counters/spec.md (100%) rename openspec/changes/{3495-service-counters-zero-series-repo-label => archive/2026-11-05-3495-service-counters-zero-series-repo-label}/tasks.md (100%) create mode 100644 openspec/specs/roadmap-service-counters/spec.md diff --git a/openspec/changes/3495-service-counters-zero-series-repo-label/.openspec.yaml b/openspec/changes/archive/2026-11-05-3495-service-counters-zero-series-repo-label/.openspec.yaml similarity index 100% rename from openspec/changes/3495-service-counters-zero-series-repo-label/.openspec.yaml rename to openspec/changes/archive/2026-11-05-3495-service-counters-zero-series-repo-label/.openspec.yaml diff --git a/openspec/changes/3495-service-counters-zero-series-repo-label/design.md b/openspec/changes/archive/2026-11-05-3495-service-counters-zero-series-repo-label/design.md similarity index 100% rename from openspec/changes/3495-service-counters-zero-series-repo-label/design.md rename to openspec/changes/archive/2026-11-05-3495-service-counters-zero-series-repo-label/design.md diff --git a/openspec/changes/3495-service-counters-zero-series-repo-label/proposal.md b/openspec/changes/archive/2026-11-05-3495-service-counters-zero-series-repo-label/proposal.md similarity index 100% rename from openspec/changes/3495-service-counters-zero-series-repo-label/proposal.md rename to openspec/changes/archive/2026-11-05-3495-service-counters-zero-series-repo-label/proposal.md diff --git a/openspec/changes/3495-service-counters-zero-series-repo-label/specs/roadmap-implement-job/spec.md b/openspec/changes/archive/2026-11-05-3495-service-counters-zero-series-repo-label/specs/roadmap-implement-job/spec.md similarity index 100% rename from openspec/changes/3495-service-counters-zero-series-repo-label/specs/roadmap-implement-job/spec.md rename to openspec/changes/archive/2026-11-05-3495-service-counters-zero-series-repo-label/specs/roadmap-implement-job/spec.md diff --git a/openspec/changes/3495-service-counters-zero-series-repo-label/specs/roadmap-service-counters/spec.md b/openspec/changes/archive/2026-11-05-3495-service-counters-zero-series-repo-label/specs/roadmap-service-counters/spec.md similarity index 100% rename from openspec/changes/3495-service-counters-zero-series-repo-label/specs/roadmap-service-counters/spec.md rename to openspec/changes/archive/2026-11-05-3495-service-counters-zero-series-repo-label/specs/roadmap-service-counters/spec.md diff --git a/openspec/changes/3495-service-counters-zero-series-repo-label/tasks.md b/openspec/changes/archive/2026-11-05-3495-service-counters-zero-series-repo-label/tasks.md similarity index 100% rename from openspec/changes/3495-service-counters-zero-series-repo-label/tasks.md rename to openspec/changes/archive/2026-11-05-3495-service-counters-zero-series-repo-label/tasks.md diff --git a/openspec/specs/roadmap-implement-job/spec.md b/openspec/specs/roadmap-implement-job/spec.md index 9e5b3b0de..51d4476b3 100644 --- a/openspec/specs/roadmap-implement-job/spec.md +++ b/openspec/specs/roadmap-implement-job/spec.md @@ -771,15 +771,15 @@ The stop comment of the job SHALL name the failure class, the action taken and t - **THEN** `implement.failure` is removed with `implement.stopped` ### Requirement: The Service counts the failures by class -When a round reports a Job that left a stop, the Service SHALL add 1 to `devops_cli_service_implement_failures_total` with the label `class`. A Job that left the item without a result SHALL get the class of the tail of its log, recorded in `implement.failure` and named, with the action left, in the report. A Job that ended with a result SHALL add nothing. +When a round reports a Job that left a stop, the Service SHALL add 1 to `devops_cli_service_implement_failures_total` with the labels `class` and `repo`, where `repo` is the repository of the round. A Job that left the item without a result SHALL get the class of the tail of its log, recorded in `implement.failure` and named, with the action left, in the report. A Job that ended with a result SHALL add nothing. #### Scenario: A stop of the runner -- **WHEN** a round reports a Job whose item holds the stop with the class `parser` -- **THEN** the counter for `parser` grows by 1, the counter for `oom` does not change, and a second round adds nothing +- **WHEN** a round of `example/app` reports a Job whose item holds the stop with the class `parser` +- **THEN** the counter for `parser` and `example/app` grows by 1, the counter for `oom` does not change, and a second round adds nothing #### Scenario: A Job with no result -- **WHEN** a round reports a Job that failed and left the item In Progress, and its log holds "out of memory" -- **THEN** the item is Blocked with `implement.failure` `oom`, the report names the class and the action left `raise_limit`, and the counter for `oom` grows by 1 +- **WHEN** a round of `example/app` reports a Job that failed and left the item In Progress, and its log holds "out of memory" +- **THEN** the item is Blocked with `implement.failure` `oom`, the report names the class and the action left `raise_limit`, and the counter for `oom` and `example/app` grows by 1 #### Scenario: A Job with a result - **WHEN** a round reports a Job that succeeded and left the item In Review diff --git a/openspec/specs/roadmap-service-counters/spec.md b/openspec/specs/roadmap-service-counters/spec.md new file mode 100644 index 000000000..36d42092d --- /dev/null +++ b/openspec/specs/roadmap-service-counters/spec.md @@ -0,0 +1,71 @@ +# roadmap-service-counters Specification + +## Purpose +The label sets of the counters of the roadmap Service, which the Service creates at zero when it starts, the labels that these counters carry, and the alerts that read them. + +## Requirements + +### Requirement: The Service starts each label set of its counters at zero +The Service SHALL create each bounded label set of its counters at zero when it starts. The bounded label sets are: each failure class for each repository of `service.repos` on `devops_cli_service_implement_failures_total`; each repository, lane, row and result on `devops_cli_service_jobs_total`; each repository and lane on `devops_cli_service_job_seconds_total`; each provider, declared event or `other`, and outcome on `devops_cli_service_webhook_deliveries_total`. + +#### Scenario: The failure counter exists at zero after start +- **WHEN** the Service starts with the repositories `example/app` and `example/tools` +- **THEN** the failure counter exists at 0 for each of the 18 failure classes and each of the two repositories + +#### Scenario: The label repo holds only configured repositories +- **WHEN** the Service starts with the repositories `example/app` and `example/tools` +- **THEN** every value of the label `repo` of the failure counter is `example/app` or `example/tools` + +#### Scenario: The webhook counter exists at zero for each declared event +- **WHEN** the Service starts with a provider that declares the events `issues` and `push` +- **THEN** the webhook counter exists at 0 for each of the outcomes with the events `issues`, `push` and `other` + +### Requirement: The Service starts the trigger and chat counters at zero +The Service SHALL create each bounded label set of `devops_cli_service_triggers_total` and `devops_cli_service_chat_commands_total` at zero when it starts. The bounded label sets are: each repository of `service.repos` with each source `webhook` or `poll`; and each result `usage`, `status`, `unknown_repo` or `error` of the `/roadmap` chat command. + +#### Scenario: The trigger and chat counters exist at zero after start +- **WHEN** the Service starts with the repositories `example/app` and `example/tools` +- **THEN** the trigger counter exists at 0 for each repository with the sources `webhook` and `poll`, and the chat counter exists at 0 for each result `usage`, `status`, `unknown_repo` and `error` + +### Requirement: The Service starts the budget hold counter at zero +The Service SHALL create the one series of `devops_cli_service_budget_holds_total` at zero when it starts. + +#### Scenario: The budget hold counter exists at zero after start +- **WHEN** the Service starts +- **THEN** the budget hold counter exists at 0 with no label + +### Requirement: The job counter carries the row of the job +The Service SHALL add 1 to `devops_cli_service_jobs_total` with the labels `repo`, `lane`, `row` and `result` for each row of a lane that ends in the roadmap run. A scoped update SHALL add 1 with the name of its job as the label `row`. The Service SHALL NOT count a lane batch as one job. The label `result` SHALL be `success`, `error` or `stopped`. A row or a scoped update that the stop of the Service ends SHALL count under `stopped`, and not under `error`. + +#### Scenario: A row that succeeds +- **WHEN** the status lane runs the row `metrics` for `example/app` and the row succeeds +- **THEN** the job counter for `repo` `example/app`, `lane` `status`, `row` `metrics` and `result` `success` grows by 1 + +#### Scenario: Two rows of one batch +- **WHEN** one round of the status lane runs the rows `metrics` and `implement`, and both fail +- **THEN** the job counter for each row with `result` `error` grows by 1, and no series counts the batch + +#### Scenario: A row that the stop of the Service ends +- **WHEN** the stop of the Service ends the row `implement` of the status lane +- **THEN** the job counter of that row with `result` `stopped` grows by 1, and the counter with `result` `error` does not change + +### Requirement: The job failure alert reads each row +`RoadmapServiceJobsFailing` SHALL group and join on `repo`, `lane` and `row`, so that a success of another row of the lane does not hide a row whose jobs all failed. + +#### Scenario: A row fails while another row of its lane succeeds +- **WHEN** `promtool test rules` evaluates a status lane whose row `implement` fails each round and whose row `metrics` succeeds +- **THEN** `RoadmapServiceJobsFailing` fires with the labels `repo`, `lane` `status` and `row` `implement` + +### Requirement: Alerts read a counter that starts at zero with increase +The alerts `RoadmapServiceWebhooksRejected` and `RoadmapServiceSelfDeliveriesHigh` SHALL read their counter with `increase()` and no `offset` term, because the Service starts each label set at zero. + +#### Scenario: A rejected burst on a new pod +- **WHEN** `promtool test rules` evaluates the rejected series of a pod that starts at 0 and reaches 25 within 15 minutes +- **THEN** `RoadmapServiceWebhooksRejected` fires + +### Requirement: A query gives the true total over a pod change +A query of `sum(increase(devops_cli_service_implement_failures_total[1h]))` SHALL give the counts of all pods in the window, including the counts of a pod that started during the window. The only difference from the counts SHALL be the extrapolation of `increase()` to the window edges. + +#### Scenario: A new pod adds its counts +- **WHEN** `promtool test rules` evaluates the failure counter of a pod that counted 3 and of a new pod that counted 2 after a zero sample at its start +- **THEN** the sum of `increase()` over the window is the 5 counts plus the extrapolation to the window edges, about 5.03