Skip to content

fix(agent): match tool_search queries against tool parameters - #1573

Open
bwachman wants to merge 4 commits into
mainfrom
cl-11366-tool-search-misses-tools-whose-matching-capability-is-a
Open

bwachman wants to merge 4 commits into
mainfrom
cl-11366-tool-search-misses-tools-whose-matching-capability-is-a

Conversation

@bwachman

@bwachman bwachman commented Oct 26, 2026 •

Copy link
Copy Markdown
Contributor

Summary

  • tool_search also scores each tool's parameter names and descriptions, so a tool whose matching capability is a filter, such as Linear's list_issues and its release parameter, reaches the shortlist. camelCase keys split into words first, so an undescribed includeArchived flag matches "archived". A parameter match weighs less than a name substring and only boosts a tool that already matches by name or description.
  • The shared lexical ranker counts each query word once, so a repeated word ("release ID or release slug") does not multiply the bonus for every tool named after it.

Verification

  • bun run check passes (9520 tests).
  • Unit tests with Linear-shaped definitions: "Linear list issues filtered by release ID or release slug" shortlists list_issues, and a tool named for the query outranks a parameter-only match.
  • A unit test with two undescribed boolean flags (includeClosed listed before includeArchived) ranks the archived tool first for "list records archived".

Fixes CL-11366

@linear-code

linear-code Bot commented Oct 26, 2026

Copy link
Copy Markdown

CL-11366

@bwachman

bwachman commented Oct 26, 2026 •

Copy link
Copy Markdown
Contributor Author

Known limitations, deliberately out of scope:

  • Singular/plural: "issue list" does not match list_issues. Tracked in CL-11435.
  • A query that matches only parameter text returns nothing. Parameters boost but never qualify, so a miss still waits for MCP servers that are connecting.
  • Filler parameter descriptions (for example the creation/update-time text on orderBy) give small boosts on common words. They can reorder the shortlist but have not pushed a correct tool off it.

@TheGreatAxios TheGreatAxios left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review · Request changes

Ranks tool_search hits by name, description, and parameter text, counting each query word once. A parameter hit only raises a tool that already matches on name or description.

Findings

  • src/agent/lexical-rank.ts:40 — parameter names go through tokenizeLexical, which only splits on non-alphanumerics (lexical-rank.ts:11). includeArchived becomes the one token includearchived, so the query word archived adds nothing. include_archived tokenizes to archived and scores 0.5. paramTokenScore (lexical-rank.ts:59) then requires that exact token. Tool names still match inside a larger token via name.includes (lexical-rank.ts:53); parameter names do not, and nothing splits a camelCase boundary. Against two tools both described as "List records", with includeClosed listed before includeArchived and neither description containing "archived", search("list records archived") returns the includeClosed tool first. The same catalog with include_closed / include_archived returns the archived tool first (scores 6 and 6.5). A bare releaseId likewise adds nothing for the query word release; the parameter named release does (6.5 against 6). The new fixtures use teamId and includeReleaseNotes, but those tests pass because the descriptions repeat the word ("Owning team", "release notes"). A boolean flag with no description, a normal JSON Schema shape, stays tied with catalog order. Splitting camelCase boundaries on parameter names before tokenizing (includeArchived → include / archived, releaseId → release / id) would close it. Substring-matching the raw lowercased name would not: the query token id would then hit valid and width.

Checked with bun -e in the PR worktree on those two catalogs. bun test of the lexical-rank, tool-search, tools-mcp-disconnect, skill-search, agent-search, and search-scorer-parity suites was 141 pass, 0 fail. None of those tests use a camelCase parameter name whose description does not already contain the query word.

@bwachman
bwachman force-pushed the cl-11366-tool-search-misses-tools-whose-matching-capability-is-a branch from 1e1ff30 to eef2bd2 Compare October 28, 2026 17:57
@bwachman

Copy link
Copy Markdown
Contributor Author

Fixed in eef2bd2. Parameter keys now split on camelCase boundaries before tokenizing, so includeArchived becomes include / archived and releaseId becomes release / id; acronym runs split too (releaseURL, HTTPMethod). This applies only to parameter names, so the tokenizer shared with skill and agent search is unchanged, and there is no substring matching, so id still does not hit valid or width.

Your scenario is a regression test in tool-search.test.ts: two undescribed boolean flags with includeClosed listed first, and "list records archived" must rank the includeArchived tool first. It failed before the change. bun run check passes (9520 tests).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants