Repository navigation
feat(server): tool-use evals, and thread_start says the access a thread got - #958
Merged
Merged
Conversation
…ad got Nothing checked how real models use the tools Threadlines gives agents, so a tool description or instruction change that made them misuse a tool went unnoticed. apps/server/evals/toolUse runs real Claude Code and Codex through our real adapters, with our real prompt, instructions and tool list. The room and browser MCP endpoints are production's (McpRoomServer and McpHttpServer now take their handlers through makeLayer) with canned answers that log every call. Twelve cases grade the calls, the agent's own tools and its reply. Runs are local (scripts/eval-tool-use.ts), from eval-only sign-in folders with no plugins, connectors or personal instructions, and each session is checked for a subscription sign-in and only Threadlines' tool servers before its first model request. First finding, fixed: asked for a thread with more access than its own, an agent told the user the thread had it, because thread_start's result didn't say it was capped. Each started thread now reports its access, and the detail says which ones got less than asked. Second finding, left as is: Codex uses its own subagents when asked to start threads.
|
The latest updates on your projects. Learn more about Vercel for GitHub. 1 Skipped Deployment
|
badcuban
added a commit
that referenced
this pull request
Nov 16, 2026
The evals added in #958 list a canned handler for every room tool; thread_move was missing, so the server typecheck failed once main and this branch met.
This branch was previously deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Nothing checked how real models use the tools Threadlines gives agents. A change to a tool's description or to our instructions that made agents misuse a tool would go unnoticed.
Tool-use evals (
apps/server/evals/toolUse, run withnode scripts/eval-tool-use.ts --provider claude|codex):McpRoomServerandMcpHttpServernow take their handlers throughmakeLayer; production passes its real ones. Nothing real starts or changes.First finding, fixed: asked for a thread with more access than its own, Opus told the user "I started a thread with full access". Threadlines caps it at the caller's access, but
thread_start's result didn't say so. Each started thread now reports itsaccess, and the detail names the threads that got less than asked. After the fix Opus says the thread "didn't get full access" (3/3).Second finding, left as is: asked to start threads, Codex uses its own built-in subagents and calls them threads. Two rewordings didn't change it; turning off Codex's own subagents did, but we're keeping Codex as is for now. The eval keeps flagging it.
First results: Claude Opus 5.5 23/24 before the fix, GPT-6.1 Sol 22/24.
Checked: full server suite, typecheck, lint, the new stand-in, grader and guard tests, and the existing MCP server tests (101).