Execute host MCP tools in dashboard runs and evaluations - #421
Conversation
|
Heads-up: #420 has merged (cd65327), which lands host MCP tool execution on That was driven by release timing rather than a judgement about this PR — a host app needed the execution path in a tagged gem to get off Worth rebasing onto the new One known gap this PR may already handle better: #425 — (Generated using Claude Code) |
A failed tools/list was rescued to [] and logged as a warning, so a server that 401s and a server that legitimately serves no tools were indistinguishable. The agent then ran tool-less, the model fabricated an answer, and the run reported a plausible low score — #425 records that costing hours to diagnose, because the report recommended prompt changes for what was a transport failure. Contributing nothing still keeps the run alive; that part was deliberate and is unchanged. What was missing is the reason, so callers can tell the two cases apart: - discovery_errors returns why each declared server contributed no tools, naming the server, its url and the underlying error. - all_servers_failed? reports the case where nothing could be discovered at all, which is when a tool-backed scenario cannot produce a meaningful score and a caller should fail loudly rather than grade an invented answer. Errors reset per tool_definitions call, so a transient failure does not leak into a later run. This takes the approach from the unmerged #421 (MCPToolBinding's @discovery_errors), narrowed to the dispatcher on current main rather than rebasing that branch across 54 commits of divergence. Surfacing these in the evaluation report is the follow-up, and is what turns "improve the instructions" back into "sparkle-diagnostic returned 401". Filed as the scoring-side counterpart in #433. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01He34kWksjpqPPDpCCqJq2C
1 participant
Closes #419.
Dashboard runs previously dropped host MCP tools from the model’s tool roster, allowing a healthcheck prompt to succeed without touching the host. This change resolves every declared tool before generation, offers live schemas from enabled MCP servers, and dispatches calls through their MCP session. Unbound tools fail with an actionable error.
Validation: 318 engine and SolidAgent integration tests, 1,441 assertions, zero failures/errors/skips (Ruby 4.0.2, Rails 8.1.3.1). Includes native Anthropic tool-loop replay against simulated MCP HTTP responses. Repository-wide RuboCop: 516 files, no offenses.
git diff --checkpasses.Built on current
main. #414 remains a compatible host-adapter path; #405 overlapsAgentExecutionServiceand will need to preserve MCP precedence and callback dispatch when rebased. Supports Streamable HTTP endpoints; does not launch stdio processes or implement OAuth/custom authentication headers.Issue, branch, milestones, and PR review notes:
docs-tracking/issues/419-host-mcp-execution/. Setup example:actionagent/README.md.