feat(telemetry): count the tool calls of the stage planner by outcome

docs/TELEMETRY.md

@@ -17,7 +17,9 @@ were collected.
 The collector's `deltatocumulative` processor adds the deltas of each stream (its
 resource, scope, metric and attributes) into a running total before Prometheus stores
 it. Prometheus names a series by `service.name` (`job`), `service.instance.id`
-(`instance`) and the metric's attributes. The `service.instance.id` is the host name,
+(`instance`) and the metric's attributes. The collector sets `job` and `instance` from
+the resource after it copies the attributes. So an attribute with either name is lost.
+A job row is the `row` attribute. The `service.instance.id` is the host name,
 which every command on the host shares. The standard `OTEL_RESOURCE_ATTRIBUTES` sets
 another one: roadmap-service sets `service.instance.id=roadmap-service`, so on the OTLP
 path a new pod continues the series of the pod before it. The `/metrics` scrape adds the
@@ -92,6 +94,10 @@ writes by name.
 | `devops_cli_roadmap_refine_comments_capped_total` | Counter | `1` | Refine comments that the cap on refine's comments on one item in 24 hours stopped: one count for each newest comment on an item that the cap left with no answer. The item waits on a person and takes no place under the round's limit. A later round that finds no newer comment on the item does not count it again. |
 | `devops_cli_roadmap_alternatives_total` | Counter | `1` | Checks of open items for an open tool that covers them, by verdict: covers, partly, none or not-applicable. One count for each check that the alternatives job writes to an item. A check that fails, or that the job skips because a person changed the item, counts nothing. |
 | `devops_cli_roadmap_stage_total` | Counter | `1` | Items that a stage round handled, one count for each item, by result: staged for an item written as a branch and a draft pull request, needs_split for an item stopped on a task group over the budget, failed for an item that failed in the plan or in the writes, and skipped for an item the job left out with a reason. A plan with no write counts nothing. |
+| `devops_cli_roadmap_tool_calls_total` | Counter | `1` | Tool calls of the roadmap planner, by tool, row, toolset and outcome. The outcome ok is a call that gave its result. The outcome empty is a call that ran and found nothing, such as a search with no match. The outcome error is a call that could not run or got no answer. The row is the job row that ran the planner, such as stage. Each plan run adds 0 for every tool of its toolset before its first model request. So a tool that no run calls keeps a series. |
+| `devops_cli_roadmap_tool_answer_chars` | Histogram | `{char}` | Characters in the answer of each tool call of the roadmap planner, after the cut of the tool, by tool, row and toolset. The text of a refusal counts. A call that got no answer adds no observation. |
+| `devops_cli_roadmap_tool_calls_per_run` | Histogram | `{call}` | Tool calls of each plan run of the roadmap planner, by row and toolset. A run that raised because a model call failed adds no observation. |
+| `devops_cli_roadmap_tool_limit_hits_total` | Counter | `1` | Plan runs of the roadmap planner whose tools ended on a limit, by row, toolset and limit. The limit calls means fewer tool calls were left than a turn needs. The limit requests means one model request was left. The limit window means the messages filled the context window. One count for each run. |
 | `devops_cli_roadmap_item_age_seconds` | Gauge | `s` | Age of the oldest open roadmap item in each status, by repo, state and plan. The state is the status. Plan is true when the item holds a stage pull request. That pull request waits for a person. The age is the time since the status label was set. Zero when no item is in the pair. |
 | `devops_cli_roadmap_oldest_item_number` | Gauge | `1` | Issue number of the oldest open roadmap item in each pair of repo, state and plan. The pair is the one that devops_cli_roadmap_item_age_seconds reports. Zero when no item is in the pair. |
 | `devops_cli_roadmap_claim_age_seconds` | Gauge | `s` | Age of the oldest claim on an open roadmap item, by repo. The age is the time since the owner label was set. Zero when no open item holds a claim. |

k8s/README.md

@@ -720,9 +720,15 @@ The rule group `roadmap-service-items` watches how long an open roadmap item sta
 | :--- | :--- | :--- | :--- | :--- |
 | `RoadmapServiceItemStuck` | An item has stayed in `blocked`, or in `ready` with a plan, for more than 24 hours. The summary names the item number. | — | warning | Run `devops roadmap inbox` to see the items that wait for a person. Move the item on. Its age starts again when its status changes. |
 
+The rule group `roadmap-service-tools` watches the tools of the stage planner (#3240). Each plan run adds 0 to `devops_cli_roadmap_tool_calls_total` for every tool of its toolset. Each tool call adds 1 with its `outcome`: `ok`, `empty` or `error`. The tool sets the outcome. An `empty` call found nothing, and an `error` call could not run. The row "Planner Tools" of the `devops-cli Telemetry` dashboard shows the calls and the mean size of the answers. The `RunTrace` comment of a staged item lists the calls and the empty answers of each tool.
+
+| Alert | Fires when | For | Severity | Do this |
+| :--- | :--- | :--- | :--- | :--- |
+| `RoadmapServiceToolUnused` | A tool of a toolset had no call in 7 days. Its series is older than 7 days. The Service made plan runs of the same row and toolset in the 7 days. A series that is younger than 7 days stays quiet, so a new tool has 7 days. A tool that leaves the toolset keeps its alert until its series leaves the 7 days. | — | warning | Read the row "Planner Tools" of the `devops-cli Telemetry` dashboard. Change the description of the tool in the task prompt. Then the planner knows when to call it. Or remove the tool from the toolset. |
+
 No alert leaves the cluster until the Secret `monitoring/alertmanager-mattermost` exists (Alerting, Receiver). Creating it is a step for the owner, who chose Mattermost as the route for the alerts (#1391, #1392). Until then, read the alerts in Grafana (Alerting, with the `Alertmanager` datasource) or on the Alerts page of Prometheus.
 
-`tests/test_k8s_monitoring_alerting.py` pins each rule and runs the pinned promtool (`devops install-tools --tool promtool`) on it. `promtool test rules` runs the unit tests of each group. The tests are in `tests/fixtures/prometheus_rules/roadmap_service_tests.yaml`, `roadmap_service_webhooks_tests.yaml`, `roadmap_service_activity_tests.yaml` and `roadmap_service_items_tests.yaml`. Each alert fires on a sample series with the labels and annotations that the tests pin. It stays quiet on a normal series and on the same series of another job. To run them by hand, write the group to the file that the tests' `rule_files` names. Then run `promtool test rules <tests-file>` in that directory.
+`tests/test_k8s_monitoring_alerting.py` pins each rule and runs the pinned promtool (`devops install-tools --tool promtool`) on it. `promtool test rules` runs the unit tests of each group. The tests are in `tests/fixtures/prometheus_rules/roadmap_service_tests.yaml`, `roadmap_service_webhooks_tests.yaml`, `roadmap_service_activity_tests.yaml`, `roadmap_service_items_tests.yaml` and `roadmap_service_tools_tests.yaml`. Each alert fires on a sample series with the labels and annotations that the tests pin. It stays quiet on a normal series and on the same series of another job. To run them by hand, write the group to the file that the tests' `rule_files` names. Then run `promtool test rules <tests-file>` in that directory.
 
 #### Go-live check
 

k8s/monitoring/dashboards/devops-cli.json

@@ -1211,6 +1211,86 @@
           "refId": "A"
         }
       ]
+    },
+    {
+      "collapsed": false,
+      "gridPos": {
+        "h": 1,
+        "w": 24,
+        "x": 0,
+        "y": 94
+      },
+      "id": 33,
+      "title": "Planner Tools",
+      "type": "row"
+    },
+    {
+      "datasource": {
+        "type": "prometheus",
+        "uid": "prometheus"
+      },
+      "description": "Tool calls of the stage planner in the last hour, by tool and outcome (ok, empty or error), on the left axis, and the mean size of the answers of each tool in characters, after the cut of the tool, on the right axis. Approximate, and low for sparse series: increase() needs two samples of a series inside its window, so a series' first sample, and any series seen only once in the window, add nothing. Delta points from one process can arrive out of order, and totals then come out low (#792). A tool that no run calls shows a line at 0.",
+      "fieldConfig": {
+        "defaults": {
+          "color": {
+            "mode": "palette-classic"
+          },
+          "custom": {
+            "axisCenteredZero": false,
+            "drawStyle": "line"
+          },
+          "unit": "short"
+        },
+        "overrides": [
+          {
+            "matcher": {
+              "id": "byFrameRefID",
+              "options": "B"
+            },
+            "properties": [
+              {
+                "id": "custom.axisPlacement",
+                "value": "right"
+              }
+            ]
+          }
+        ]
+      },
+      "gridPos": {
+        "h": 8,
+        "w": 24,
+        "x": 0,
+        "y": 95
+      },
+      "id": 34,
+      "options": {
+        "legend": {
+          "displayMode": "list",
+          "placement": "bottom"
+        }
+      },
+      "title": "Planner Tool Calls and Answer Size",
+      "type": "timeseries",
+      "targets": [
+        {
+          "datasource": {
+            "type": "prometheus",
+            "uid": "prometheus"
+          },
+          "expr": "sum by (tool, outcome) (increase(devops_cli_roadmap_tool_calls_total{job=\"roadmap-service\", row=\"stage\"}[1h]))",
+          "legendFormat": "{{tool}} {{outcome}}",
+          "refId": "A"
+        },
+        {
+          "datasource": {
+            "type": "prometheus",
+            "uid": "prometheus"
+          },
+          "expr": "histogram_avg(sum by (tool) (increase(devops_cli_roadmap_tool_answer_chars{job=\"roadmap-service\", row=\"stage\"}[1h])))",
+          "legendFormat": "{{tool}} mean chars",
+          "refId": "B"
+        }
+      ]
     }
   ],
   "refresh": "10s",

k8s/monitoring/prometheus-values.yaml

@@ -864,6 +864,28 @@ serverFiles:
             annotations:
               summary: 'Item {{ with printf "max by (repo, state, plan) (last_over_time(devops_cli_roadmap_oldest_item_number{job=\"roadmap-service\", repo=%q, state=%q, plan=%q}[30m]))" $labels.repo $labels.state $labels.plan | query }}#{{ . | first | value | printf "%.0f" }} {{ end }}of {{ $labels.repo }} has been in state {{ $labels.state }} for {{ $value | humanizeDuration }}'
               description: "An item of the roadmap stayed in one state for more than 24 hours. Run devops roadmap inbox to see the item (k8s/README.md, Roadmap service)."
+      # The tools of the stage planner that no run called (#3240). Each plan run adds 0 to the call
+      # counter for every tool of its toolset, so a tool has a series before a run calls it, and
+      # the rule needs no list of tool names. The call count of 7 days sums the outcomes of a tool.
+      # A series that is younger than 7 days has no sample before the window, so it stays quiet: a
+      # new tool gets 7 days. The plan runs are the observations of the native histogram of the
+      # calls per run, of the same row and toolset. A tool that leaves the toolset keeps its alert
+      # until its series leaves the 7 days.
+      - name: roadmap-service-tools
+        rules:
+          - alert: RoadmapServiceToolUnused
+            expr: >-
+              sum by (row, tool, toolset) (increase(devops_cli_roadmap_tool_calls_total{job="roadmap-service"}[7d])) == 0
+              and on (row, tool, toolset)
+              count by (row, tool, toolset) (count_over_time(devops_cli_roadmap_tool_calls_total{job="roadmap-service"}[7d] offset 7d)) > 0
+              and on (row, toolset)
+              sum by (row, toolset) (histogram_count(increase(devops_cli_roadmap_tool_calls_per_run{job="roadmap-service"}[7d]))) > 0
+            for: 0m
+            labels:
+              severity: warning
+            annotations:
+              summary: "Tool {{ $labels.tool }} of the toolset {{ $labels.toolset }} had no call in 7 days"
+              description: "No plan run of the row {{ $labels.row }} called the tool {{ $labels.tool }} of the toolset {{ $labels.toolset }} in 7 days, and the Service made plan runs in that time. Read the Planner Tools row of the devops-cli Telemetry dashboard, and decide to change the tool description or to remove the tool (k8s/README.md, Roadmap service)."
       - name: mattermost
         rules:
           - alert: MattermostUnavailable

openspec/changes/archive/2026-11-05-3240-count-tool-calls-of-the-stage/.openspec.yaml

@@ -0,0 +1,2 @@
+schema: spec-driven
+created: 2026-11-05