Surface MCP tool-discovery failures instead of running tool-less and silent

[TonsOfFun]

Closes #425.

MCPToolDispatcher#tool_definitions rescued a failed tools/list to [] and logged a warning, so a server that 401s and a server that legitimately serves no tools were indistinguishable. The agent then ran tool-less, the model fabricated an answer, and the run reported a plausible low score.

#425 records that costing several hours to diagnose, because the report offered "improve the instructions" recommendations for what was a transport failure. The only evidence was two logger.warn lines.

What changed

Contributing nothing still keeps the run alive — that part was deliberate and is unchanged. What was missing is the reason:

• discovery_errors — why each declared server contributed no tools, naming the server, its URL and the underlying error:

  Cannot load tools from MCP server 'records' (https://host.example/mcp/records): 401 Unauthorized. Check its URL, transport and credentials.

• all_servers_failed? — true when every declared HTTP server failed discovery. That is the case where a tool-backed scenario cannot produce a meaningful score, so a caller can fail loudly instead of grading an invented answer.

Errors reset per tool_definitions call, so a transient failure does not leak into a later run. An agent declaring no servers is not "failed" — its tool-less execution is the configured behaviour.

Relationship to #421

This takes the approach from the unmerged #421 (MCPToolBinding's @discovery_errors), narrowed to the dispatcher on current main. #421 is CONFLICTING/DIRTY and 54 commits behind, and #420 already solved execution a different way — so rebasing it wholesale was not worth it, while this idea was.

Tests

5 new tests, all verified to fail without the change (NoMethodError: undefined method 'discovery_errors').

• actionagent/test: 345 runs, 0 failures (was 340)
• test/evals: 115 runs, 0 failures
• RuboCop clean

Follow-up

Surfacing these in the evaluation report is what turns "improve the instructions" back into "records returned 401". The scoring-side counterpart — that a fabricated answer raises no fault at all, so the judge is never asked for a tool — is #433.

🤖 Generated with Claude Code

https://claude.ai/code/session_01He34kWksjpqPPDpCCqJq2C

---

Update — second commit

093f8b4b makes something actually read the recorded errors, and fixes a related understatement:

The run now says when discovery failed. A run whose declared MCP servers could not be reached scored an agent with no tools to call — the model answered from its own weights, so the scores are real but meaningless. error_message now carries that, naming each server and its error, and distinguishes every server failing from some. The column is already rendered by the evaluations API (:362), so it reaches the UI without new plumbing.

The diagnosis roster was understated. tool_roster listed only AgentToolbox definitions, so an agent whose tools come from MCP servers was diagnosed against a roster those tools were missing from — Diagnosis would report "none of the available tools covers this task" while naming an incomplete list. It now includes the dispatcher's discovered tools, matching what AgentExecutionService#tool_schemas offers the provider.

Tests: actionagent/test 347 runs, 0 failures (main: 340). All new tests verified to fail without their implementation. RuboCop clean.