Skip to content

feat(server): tool-use evals, and thread_start says the access a thread got - #958

Merged
badcuban merged 1 commit into
mainfrom
tool-use-evals
Nov 16, 2026
Merged

badcuban merged 1 commit into
mainfrom
tool-use-evals

Conversation

@badcuban

Copy link
Copy Markdown
Collaborator

Nothing checked how real models use the tools Threadlines gives agents. A change to a tool's description or to our instructions that made agents misuse a tool would go unnoticed.

Tool-use evals (apps/server/evals/toolUse, run with node scripts/eval-tool-use.ts --provider claude|codex):

  • Real Claude Code and Codex run through our real adapters, so the model gets exactly a user's prompt, instructions and tool list.
  • The room and browser MCP endpoints are production's code with canned answers that log every call. McpRoomServer and McpHttpServer now take their handlers through makeLayer; production passes its real ones. Nothing real starts or changes.
  • Twelve cases grade the tool calls, the agent's own tools (e.g. CronCreate, its own subagents) and what it tells the user.
  • Local only, not CI. Runs come from eval-only sign-in folders with no plugins, connectors or personal instructions. Each session is checked for a subscription sign-in and only Threadlines' tool servers before its first model request; a test proves an API-key Codex session is stopped before its first turn.

First finding, fixed: asked for a thread with more access than its own, Opus told the user "I started a thread with full access". Threadlines caps it at the caller's access, but thread_start's result didn't say so. Each started thread now reports its access, and the detail names the threads that got less than asked. After the fix Opus says the thread "didn't get full access" (3/3).

Second finding, left as is: asked to start threads, Codex uses its own built-in subagents and calls them threads. Two rewordings didn't change it; turning off Codex's own subagents did, but we're keeping Codex as is for now. The eval keeps flagging it.

First results: Claude Opus 5.5 23/24 before the fix, GPT-6.1 Sol 22/24.

Checked: full server suite, typecheck, lint, the new stand-in, grader and guard tests, and the existing MCP server tests (101).

…ad got

Nothing checked how real models use the tools Threadlines gives agents, so
a tool description or instruction change that made them misuse a tool went
unnoticed.

apps/server/evals/toolUse runs real Claude Code and Codex through our real
adapters, with our real prompt, instructions and tool list. The room and
browser MCP endpoints are production's (McpRoomServer and McpHttpServer now
take their handlers through makeLayer) with canned answers that log every
call. Twelve cases grade the calls, the agent's own tools and its reply.
Runs are local (scripts/eval-tool-use.ts), from eval-only sign-in folders
with no plugins, connectors or personal instructions, and each session is
checked for a subscription sign-in and only Threadlines' tool servers before
its first model request.

First finding, fixed: asked for a thread with more access than its own, an
agent told the user the thread had it, because thread_start's result didn't
say it was capped. Each started thread now reports its access, and the
detail says which ones got less than asked. Second finding, left as is:
Codex uses its own subagents when asked to start threads.
@vercel

vercel Bot commented Nov 16, 2026

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

1 Skipped Deployment
Project Deployment Actions Updated
threadlines Skipped Skipped Nov 16, 2026 11:25am UTC

Request Review

@badcuban
badcuban added this pull request to the merge queue Nov 16, 2026
Merged via the queue into main with commit 8a40ea7 Nov 16, 2026
24 checks passed
@badcuban
badcuban deleted the tool-use-evals branch November 16, 2026 11:37
badcuban added a commit that referenced this pull request Nov 16, 2026
The evals added in #958 list a canned handler for every room tool; thread_move was missing, so the server typecheck failed once main and this branch met.

This branch was previously deployed

1 inactive deployment
Preview — 873d773f Deployed Nov 16, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant