Skip to content

fix(agent): match tool names across a trailing plural s - #1574

Merged
bwachman merged 3 commits into
mainfrom
cl-11435-tool-search-misses-tools-when-the-query-and-tool-name-differ
Oct 29, 2026
Merged

bwachman merged 3 commits into
mainfrom
cl-11435-tool-search-misses-tools-when-the-query-and-tool-name-differ

Conversation

@bwachman

@bwachman bwachman commented Oct 26, 2026 •

Copy link
Copy Markdown
Contributor

Summary

  • Tool, skill, and agent search match a query word against name tokens across a trailing plural "s", so "issue list" finds mcp__linear__list_issues and "issues" finds get_issue and save_issue. An exact-spelling name match outranks a folded one, so "list issues" and "linear issues" rank list_issues first in any catalog order. Description and parameter tokens keep exact matching.
  • A query word given in both spellings ("issue ... issues") counts once, at whichever spelling scores higher.
  • The whole-query bonus (the tool name contains the raw query) no longer applies when a one-word query already matches a name token, exact or folded. That bonus counted the same word twice, so "issues" lifted list_issues to 4, set the cutoff at 3, and cut every singular-named tool at 2.5. A one-word query found only inside a longer name token ("granola" in mygranolahoard) keeps the bonus.
  • A replay test runs the queries models actually sent while looking for a release's issues against 26 live Linear tool definitions, each at its logged limit.
  • Known limits, all accepted:
    • "Linear read pipeline release issues and issue details" (limit 5) no longer shortlists list_issues (it ranks 10th). It passed on the base branch only because "issue" and "issues" counted as two words. The query names both releases and issues, and the release tools match more of its words. The replay test keeps it as test.failing, so a later scorer change that recovers it shows up.
    • The spelling merge applies on every tier, so an unrelated pair like "new" and "news" in one query counts once against a description.
    • Folding inside names also merges real word pairs such as "news" and "new", or "canvas" and "canva".
  • Stacked on fix(agent): match tool_search queries against tool parameters #1573 (CL-11366). This repo keeps merged branches, so after fix(agent): match tool_search queries against tool parameters #1573 merges, retarget this PR's base to main by hand.

Verification

  • bun run check (lint, typecheck, dead exports, build, test) exits 0 on this branch: 9535 pass, 0 fail.
  • Replay on 26 live Linear definitions: four of the five logged queries shortlist list_issues, against three on the base branch. "Linear retrieve pipeline release issue list and issue details" and "issue list" now pass; "Linear read pipeline release issues and issue details" is the known miss above.
  • Existing tool, skill, and agent search tests, including search-scorer-parity.test.ts, pass with no change to their expected order; the diff adds tests and changes no existing assertion.
  • Against the base branch's scorer, 8 of the lexical-rank, tool-search, and replay tests fail (98 pass): the new singular/plural, one-word plural, and replay tests, plus the test.failing replay case, which passes there.

Fixes CL-11435

@linear-code

linear-code Bot commented Oct 26, 2026

Copy link
Copy Markdown

CL-11435

@bwachman

Copy link
Copy Markdown
Contributor Author

Self-Review (code-review skill)

The branch carries the following intentional exceptions, each authorized by Greybeard:

  • src/agent/lexical-rank.ts:93-96: query words group by folded form before scoring every tier, so unrelated pairs like "new"/"news" count once against a description. Authorized by Greybeard on grounds that the effect only lowers a score (never a false hit or an empty result) and a second scoring path in the shared scorer costs more than the bug.
  • src/agent/lexical-rank.ts:16-21: name folding merges real word pairs inside tool names (news/new, canvas/canva), which can turn a tool_search miss into a hit. Authorized by Greybeard as an accepted cost of folding tool names.

@TheGreatAxios TheGreatAxios left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review · Request changes

Folds one trailing plural "s" on name tokens so a singular query can match a plural tool name and the reverse, with an exact spelling still scoring higher.

Findings

  • src/agent/lexical-rank.ts:24 — FOLDED_NAME_TOKEN_SCORE is 2.5 so a folded name hit stays under the exact token score of 3. That clears TOOL_SEARCH_SCORE_RATIO (0.75) for a two-word query whose top is 6, which is what the comment next to the constant describes. It does not clear the same cut for a one-word plural query. name.includes(rawQuery) adds 1 on top of the exact token score, so list_issues scores 4 for "issues" and the floor is 3. get_issue and save_issue only fold, so they score 2.5 and createToolIndex(...).search("issues", 20) returns just ["mcp__linear__list_issues"]. The other direction still works: search("issue", 20) keeps list_issues at 3.5. No weight strictly under 3 can clear a floor of 3, and 3 would tie search("issue") (both names also take the substring point), which drops "exact spelling outranks folded." One way out is to stop the one-token raw-substring bonus from setting that floor — a one-word query is already paid for by the token score — so the top stays 3, 2.5 still clears 0.75 * 3, and the exact token still ranks first. The new tests never call search("issues").

Checked with bun -e against a Linear-shaped catalog (list_issues, get_issue, save_issue) in the PR worktree: scores 4 / 2.5 / 2.5 for "issues", and only list_issues returned. bun test src/agent/lexical-rank.test.ts src/agent/tool-search.test.ts was 99 pass, 0 fail; those tests do not cover this query.

@bwachman
bwachman force-pushed the cl-11366-tool-search-misses-tools-whose-matching-capability-is-a branch from 1e1ff30 to eef2bd2 Compare October 28, 2026 17:57
@bwachman
bwachman force-pushed the cl-11435-tool-search-misses-tools-when-the-query-and-tool-name-differ branch from f96a9c2 to 1ed4a78 Compare October 28, 2026 21:47
@bwachman

Copy link
Copy Markdown
Contributor Author

Thanks, good catch. Reproduced: "issues" alone returned only list_issues, because the whole-query bonus counted the word a second time on top of its name-token hit, which set the cutoff above every folded match.

Fixed in 1ed4a78 (fix(agent): stop the whole-query bonus recounting a one-word name match), with a test in tool-search.test.ts that "issues" returns list_issues, get_issue, and save_issue on name-only fixtures.

I didn't drop the bonus for every one-word query as suggested: search-scorer-parity.test.ts needs it for a word found only inside a longer name token ("granola" in mygranolahoard), to keep that above a description-only hit. So the bonus now skips only the case where it double-counts. The Summary bullet that claimed "issues" found get_issue was wrong before this; it's accurate now and updated.

@TheGreatAxios TheGreatAxios left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review · Approve

Matches name tokens across a trailing plural s without double-counting spelling variants or a one-word whole-query bonus.

No findings.

Reviewed 1ed4a78af29424e93ec6f1ba9193c3c7858f47aa against this PR's declared #1573 base. The prior whole-query-bonus finding is closed by the regression at src/agent/tool-search.test.ts:531–541. The documented trailing-s collisions and five-result replay miss remain accepted limitations, not new findings.

Verification: focused lexical-rank, tool-search, replay, and scorer-parity suites: 107 passed; three temporary probes passed. bun run check rerun exited 0; the full-suite review run reported 9,535 passed, 0 failed. Temporary tests were removed.

@corbits-triage

corbits-triage Bot commented Oct 29, 2026 •

Copy link
Copy Markdown
Contributor

Triage: In progress

Triage is waiting for CI to finish before it decides. This comment shows the verdict once it has.

@bwachman
bwachman changed the base branch from cl-11366-tool-search-misses-tools-whose-matching-capability-is-a to main October 29, 2026 18:02
@bwachman
bwachman force-pushed the cl-11435-tool-search-misses-tools-when-the-query-and-tool-name-differ branch from 1ed4a78 to 5fd61fe Compare October 29, 2026 18:04
@bwachman
bwachman merged commit 8b006eb into main Oct 29, 2026
14 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants