[dan-petty]

The stage planner calls read-only tools, and no metric shows which tools it calls or how each call ends. This change adds four instruments: the calls by tool and outcome, the answer size, the calls per run and the limit hits. Each tool sets its own outcome: empty for a call that found nothing, and error for a call that could not run. The change adds one line of tool counts to the trace comment, a panel to the devops-cli dashboard, and the alert RoadmapServiceToolUnused for a tool that no run called in 7 days. A promtool test covers the alert.

Closes #3240

Change: openspec/changes/archive/2026-11-05-3240-count-tool-calls-of-the-stage/

[dan-petty]

Test results and review findings for #3240

Review findings

I checked each finding of the first review against the code. All seven were right. Each one is fixed in this pull request.

1. empty could not occur, and a failed call counted as ok (two findings, one cause). I ran each tool of StageTools on edge inputs with the first rule (blank text is empty). 11 of 12 cases gave ok, for example No match., a path outside the repository, and GitHub is not available to this run. The one empty was git_show that returned '' when git ran out of time. Now each tool sets its own outcome with a pydantic-ai ToolReturn (ok_answer, empty_answer, error_answer), and RunTrace reads ToolReturnPart.metadata. A search with no match is empty. A refused path, a missing file or a failed GitHub read is error. A git command that runs out of time is now an error in every tool that runs git. The model reads the same text as before. I used a bare ToolReturn as the return annotation: str | ToolReturn made tools() 20 times slower (265 ms against 13 ms), because pydantic-ai builds the return schema for each tool.
2. Plan text and code disagreed on a hard stop by UsageLimitExceeded. The code is right: the hook flags requests and window before they can stop a run, so a hard stop with no flag is calls. The proposal, the design and the tasks now say so. A test runs the case.
3. seven cases of the promtool fixture. The fixture has eight. The text says eight, and the spec has the missing scenario "No run series exists".
4. Patches of devops-cli code in the new tests. All five are gone. The limit tests call investigate with their own UsageLimits and window. The metric tests use the real tools and the real limits. A test of a git timeout uses process_fake.
5. The job and instance check saw only an inline dict. It now reads a dict display, a name that holds a dict, and a dict that ** unpacks. I checked that it names a job label if I put one in tool_metrics.py.
6. Sentences over 25 words. I split the text of the two counters, the alert row in k8s/README.md, and the sentence in the generator. docs/TELEMETRY.md is regenerated.

Fail-to-pass

uv run devops ci fail-to-pass --base origin/main exits 0: the replay of the new tests on the base tree gives 6 failed and 5 errored. The errors are imports of names that the base does not have, such as RunLimit. This pull request is a feature, so the replay is evidence and not a gate.

Checks on this host

• uv run devops ci passes 23 rows. The row test fails with 4 tests that read the host and not the code: test_get_ssh_key_prefix_resolution, test_find_project_config_path_skips_directory (a file /tmp/config.yaml on the host), test_mark_only_commands_under_global_dry_run (a devcontainer tool in ~/.local/bin) and test_the_direct_executor_refuses_a_network_profile_before_it_starts_the_tool[direct]. The same 4 fail in the main checkout, which has none of these changes. Open issues track them: fix(ci): host state fails four tests and one case #3585, fix(tests): four tests fail on the devcontainer host #3590, fix(ci): the pytest step of devops ci exceeds the 5-minute budget #3593, fix(tests): four host-dependent tests fail on main #3594 and fix(ci): four tests read host state and fail in a devcontainer #3597. The row coverage stops with the row test.
• The row test took 8 min 59 s on this host, which other agents share. None of the 10 slowest tests is new in this change.
• 18,813 tests passed. After the rebase onto origin/main, 2,319 selected tests of tests/roadmap, tests/telemetry and tests/lang passed, apart from the one host test above.
• The promtool cases of RoadmapServiceToolUnused pass (8 cases).