fix(telemetry): Service counters read true across pods and repos

docs/ROADMAP_WORKFLOW.md

@@ -140,7 +140,7 @@ Each clear by digest adds an entry (the class, the new digest and the time) to `
 
 ## The failure classes
 
-A stop has a class (`roadmap/failure.py`, #3065). A stop site that knows its cause passes the class itself (#3476): the classes `no_change_in_fix`, `tests_lost`, `head_moved`, `output_cap` and `limit_reached` come from the step, and no rule makes them. For any other stop, the classifier is a pure function of the error text, the exit status, the refusal of a tool, the name of the check and the log of the check. It never reads the last message of the agent, which is a claim that no check verified. A failure that no rule matches is `ci_infra`. The stop comment starts with the failed check and the end of its log, then names the class, the action taken and the action left. It shows the last message of the agent as a claim that no check verified. The Job log holds one line with the class and the action left. The Service adds 1 to `devops_cli_service_implement_failures_total` with the label `class` when a round reports the Job. A Job that left the Item with no result gets the class of the tail of its log.
+A stop has a class (`roadmap/failure.py`, #3065). A stop site that knows its cause passes the class itself (#3476): the classes `no_change_in_fix`, `tests_lost`, `head_moved`, `output_cap` and `limit_reached` come from the step, and no rule makes them. For any other stop, the classifier is a pure function of the error text, the exit status, the refusal of a tool, the name of the check and the log of the check. It never reads the last message of the agent, which is a claim that no check verified. A failure that no rule matches is `ci_infra`. The stop comment starts with the failed check and the end of its log, then names the class, the action taken and the action left. It shows the last message of the agent as a claim that no check verified. The Job log holds one line with the class and the action left. The Service adds 1 to `devops_cli_service_implement_failures_total` with the labels `class` and `repo` when a round reports the Job. A Job that left the Item with no result gets the class of the tail of its log.
 
 | Class | Default action | Meaning |
 | :--- | :--- | :--- |

docs/TELEMETRY.md

@@ -19,17 +19,27 @@ resource, scope, metric and attributes) into a running total before Prometheus s
 it. Prometheus names a series by `service.name` (`job`), `service.instance.id`
 (`instance`) and the metric's attributes. The `service.instance.id` is the host name,
 which every command on the host shares. The standard `OTEL_RESOURCE_ATTRIBUTES` sets
-another one: roadmap-service sets `service.instance.id=roadmap-service`, so a new pod
-writes the series of the pod before it. A running total that starts again from zero,
-for a new pod or for a stream that gets no delta for `max_stale` (`1h` in
+another one: roadmap-service sets `service.instance.id=roadmap-service`, so on the OTLP
+path a new pod continues the series of the pod before it. The `/metrics` scrape adds the
+label `pod`, so each pod has its own series there. A running total that starts again from
+zero, for a new pod or for a stream that gets no delta for `max_stale` (`1h` in
 `k8s/otel/values.yaml`), is a counter reset to `rate()` and `increase()`.
 Cluster job pods have unique instance names and export metrics only once when they exit.
 PromQL functions like `increase()` and `rate()` drop series that contain only one sample.
 To calculate these counters, the collector transmits created timestamps via remote write 2.0
 (`remote_write_version: 2` in `k8s/otel/values.yaml`). Prometheus sets
 `enable-feature: created-timestamp-zero-ingestion` in `k8s/monitoring/prometheus-values.yaml`.
-Prometheus then writes a zero-value sample at the start timestamp of each series, so
-`increase()` calculates the full counter value.
+On this path, Prometheus writes a zero-value sample at the start timestamp of each series,
+so `increase()` calculates the full counter value. The `/metrics` scrape gets no such sample.
+
+The Service starts each label set of its counters at zero (#3495). So the first count
+of a label set is in its series. The exception is
+`devops_cli_service_identity_refused_total`: its label set starts with the
+first refusal. A pod loses a count that it makes before its first scrape, because
+`increase()` does not count the first sample of a series. The labels of
+`devops_cli_service_implement_failures_total` are `class` and `repo`. The labels of
+`devops_cli_service_jobs_total` are `repo`, `lane`, `row` and `result`.
+
 A gauge holds a snapshot, such as a project metric that the roadmap Service sends
 every 15 minutes. Prometheus finds a plain selector only within five minutes of a
 sample, so a dashboard reads a gauge with `last_over_time(<metric>[$__range])`, and

k8s/monitoring/prometheus-values.yaml

@@ -667,14 +667,14 @@ serverFiles:
       # names the job after OTEL_SERVICE_NAME. Each selector names the job, so the same series of
       # a workstation (job="devops-cli"), or of the server's kubernetes-pods job, which scraped the
       # pod before #1130, never counts.
-      # kube-state-metrics gives the restarts and the replicas. On both paths a series starts at
-      # its first count, with no zero sample before it, and increase() does not count that first
-      # value. The OTLP series also get a sample only when the Service sends a new observation, and
-      # the collector starts a stream again from zero after an hour with none (max_stale).
-      # So each count of the last hour reads a series with no sample in the hour before as new,
-      # and gives its last value, which is all of its count. Every other series gives its
-      # increase(). Neither part reads one sample near the start of the hour, so a sparse series
-      # from before the hour adds none of its earlier count (#2505).
+      # kube-state-metrics gives the restarts and the replicas. The Service starts its /metrics
+      # counters at zero (#3495), so their series need no zero sample here. The OTLP series get a
+      # sample only when the Service sends a new observation, and the collector starts a stream
+      # again from zero after an hour with none (max_stale). So each count of the last hour reads
+      # a series with no sample in the hour before as new, and gives its last value, which is all
+      # of its count. Each other series gives its increase(). Neither part reads one sample near
+      # the start of the hour, so a sparse series from before the hour adds none of its earlier
+      # count (#2505).
       - name: roadmap-service
         rules:
           # A refusal stops every round, also after a restart, until a person removes the halt
@@ -698,17 +698,19 @@ serverFiles:
             annotations:
               summary: "GitHub throttled {{ $labels.resource }} requests of the roadmap Service"
               description: "In the last 15 minutes GitHub refused a {{ $labels.resource }} request of the roadmap Service with a rate limit, or gave a 5xx for a read. The Service waited and sent the request again (k8s/README.md, Roadmap service)."
-          # When all rounds of a repository and lane fail in one hour, alert after 30 minutes.
+          # When all jobs of one row of a repository fail in one hour, alert after 30 minutes. A
+          # round counts each of its rows, so a join on the lane hides a row that fails while
+          # another row of its lane succeeds (#3495).
           - alert: RoadmapServiceJobsFailing
             expr: >-
-              sum by (repo, lane) (increase(devops_cli_service_jobs_total{job="roadmap-service", result="error"}[1h])) > 0
-              unless on (repo, lane)
-              sum by (repo, lane) (increase(devops_cli_service_jobs_total{job="roadmap-service", result="success"}[1h])) > 0
+              sum by (repo, lane, row) (increase(devops_cli_service_jobs_total{job="roadmap-service", result="error"}[1h])) > 0
+              unless on (repo, lane, row)
+              sum by (repo, lane, row) (increase(devops_cli_service_jobs_total{job="roadmap-service", result="success"}[1h])) > 0
             for: 30m
             labels:
               severity: warning
             annotations:
-              summary: "All {{ $labels.lane }} jobs of {{ $labels.repo }} failed in the last hour"
+              summary: "All {{ $labels.row }} jobs of {{ $labels.repo }} failed in the last hour, on the {{ $labels.lane }} lane"
               description: '{namespace="devops"} | json | repo="{{ $labels.repo }}" | lane="{{ $labels.lane }}" (k8s/README.md, Roadmap service)'
           # Each poll starts a round on every lane, every 5 minutes by default. A worker sets the
           # job start of its lane when it starts, so a lane of a new pod that starts no job also
@@ -778,21 +780,13 @@ serverFiles:
       # The route counts each delivery that it refuses under outcome="rejected": unsigned, signed
       # with no secret of a managed repository, too large, not JSON, or without its event name or
       # delivery ID. A forged delivery counts under the event that it names, or under "other", so a
-      # flood can start a new label set. A new label set has no zero sample, so the count of the 15
-      # minutes reads a series with no sample in the 15 minutes before as new, and gives its last
-      # value (#2505). Each other series gives its increase(). The Service serves this counter only
-      # on its /metrics, not through OTLP, and the scrape reads it each minute. So a label set that
-      # counted before has a sample in each minute while its pod runs, and only a new label set, or
-      # the label set of a new pod, has no sample in the 15 minutes before.
+      # flood can start a new label set. The Service starts each label set at zero (#3495), so
+      # increase() counts each delivery of a label set from its start, and the rule needs no
+      # workaround term. The Service serves this counter only on its /metrics, not through OTLP.
       - name: roadmap-service-webhooks
         rules:
           - alert: RoadmapServiceWebhooksRejected
-            expr: >-
-              sum(
-              (last_over_time(devops_cli_service_webhook_deliveries_total{job="roadmap-service", outcome="rejected"}[15m])
-              unless last_over_time(devops_cli_service_webhook_deliveries_total{job="roadmap-service", outcome="rejected"}[15m] offset 15m))
-              or increase(devops_cli_service_webhook_deliveries_total{job="roadmap-service", outcome="rejected"}[15m])
-              ) > 20
+            expr: sum(increase(devops_cli_service_webhook_deliveries_total{job="roadmap-service", outcome="rejected"}[15m])) > 20
             for: 0m
             labels:
               severity: warning
@@ -827,12 +821,7 @@ serverFiles:
           # can come back as one delivery, so many of them can be another bot or workflow that
           # answers each write of the Service.
           - alert: RoadmapServiceSelfDeliveriesHigh
-            expr: >-
-              sum(
-              (last_over_time(devops_cli_service_webhook_deliveries_total{job="roadmap-service", outcome="self"}[1h])
-              unless last_over_time(devops_cli_service_webhook_deliveries_total{job="roadmap-service", outcome="self"}[1h] offset 1h))
-              or increase(devops_cli_service_webhook_deliveries_total{job="roadmap-service", outcome="self"}[1h])
-              ) > 150
+            expr: sum(increase(devops_cli_service_webhook_deliveries_total{job="roadmap-service", outcome="self"}[1h])) > 150
             for: 0m
             labels:
               severity: warning

openspec/changes/archive/2026-11-05-3495-service-counters-zero-series-repo-label/.openspec.yaml

@@ -0,0 +1,2 @@
+schema: spec-driven
+created: 2026-11-05

openspec/changes/archive/2026-11-05-3495-service-counters-zero-series-repo-label/design.md

@@ -0,0 +1,25 @@
+# Design
+
+## Decisions
+
+- **The Service creates the bounded label sets of its counters at zero when it starts.** A labelled counter has no series until `.labels()` makes a child (see the Feasibility of the proposal). One function calls `.labels()` for each bounded label set, so the first scrape of a pod reads a zero series for each label set.
+- **The label sets are bounded.** The failure classes are the 18 members of `FailureClass`. The repositories are `service.repos`. The rows are the rows of the three lane tables of the roadmap run. The events of the webhook counter are the declared events of each provider, plus `other`. The results of the job counter are `success`, `error` and `stopped`. The sources of the trigger counter are `webhook` and `poll`. The results of the chat command are `usage`, `status`, `unknown_repo` and `error`. No dynamic label value is created at start.
+- **The exposition format stays text 0.0.4.** The Service serves `/metrics` in this format (`src/devops_cli/server/service.py`). The zero children appear in this format, so the change does not switch the format.
+- **The jobs counter counts each row.** The roadmap run counts each row of a lane with its result, with the row name of the row table. `run_scoped_update` of the roadmap run counts each scoped update with the name of its job. The worker counts no job, so no job is counted twice.
+- **A stop is no failed row.** A row or a scoped update that the stop of the Service ends counts under `result="stopped"`. `RoadmapServiceJobsFailing` reads only `error` and `success`, so a rollout does not fire it for a row that runs twice a day.
+- **`RoadmapServiceJobsFailing` reads the row.** The rule groups and joins on `repo`, `lane` and `row`. Before this change, a failed row made its whole round an error. With a count for each row, the row `metrics` succeeds in each status-lane round, so a join on the lane never fires for a failed row of that lane. A promtool case holds this.
+- **The Loki query of `RoadmapServiceJobsFailing` has no field `row`.** The log line of a failed round has no such field, and its text names the rows. The summary of the alert names the row.
+- **The budget hold counter is primed too.** It has no label, but the registry creates a counter at its first count, so it has no series before the first hold.
+- **The OTLP workaround terms stay.** The GitHub request and AI request series reach Prometheus over OTLP, not the `/metrics` scrape. The comment at `k8s/monitoring/prometheus-values.yaml` says that an OTLP series gets a sample only when the Service sends a new observation. No live Prometheus query ran in this change, so the OTLP terms stay until a live check shows that they are not needed.
+- **The gap of a new pod stays.** The zero children are created at start. A count made before the first scrape of a new pod is still lost, because `increase()` does not count the first sample of a series. The documents state this gap.
+- **No recording rule.** The promtool case of a pod change decides whether a recording rule is needed. The case runs with the zero series of the new pod.
+
+## Rejected alternatives
+
+- **Switch `/metrics` to OpenMetrics or PrometheusProto for start timestamps.** Rejected: the change of format reaches every consumer of `/metrics`, and the start-timestamp injection is not verified for this path. The zero children solve the missing first sample without a format change.
+- **Keep the per-batch count and add a second count per row.** Rejected: one job would be counted twice, once per lane batch and once per row.
+- **Prime the identity label of `devops_cli_service_identity_refused_total`.** Rejected: its label `identity` holds the identity that GitHub refuses, so its value is not known before a refusal. The alert `RoadmapServiceIdentityRefused` reads `> 0` with no workaround term, so the alert does not need a zero series.
+- **Add the field `row` to the Loki query of `RoadmapServiceJobsFailing`.** Rejected: the log line has no field `row`, so the query would give no line.
+- **Keep the join of `RoadmapServiceJobsFailing` on the lane.** Rejected: the promtool case of a row that fails each round, while the row `metrics` of its lane succeeds, gives no alert with this join.
+- **Count a round that fails before its first row in this change.** Rejected: the failure of a round has no row, so it needs its own label value or counter and its own alert. #3666 holds this work.
+- **Add a recording rule for the true total without a test.** Rejected: a recording rule adds a second series name for one counter. It needs a promtool case that shows the need first.