Repository navigation
Tool calling: return types? #138
Description
Our current proposal for tool-calling is that the execute() function always returns a string. (Any other returned value will be converted to a string, e.g. an object will become "[object Object]".)
This is a good simple starting point, and upgrading it is probably backward-compatible in the future (since it is hard to rely on "[object Object]" being the correct result).
But here is some research on alternatives:
- OpenAI function calling behaves like ours, in that it must always be a string.
- Vercel AI SDK doesn't seem to have clear docs, but shows examples of returning arbitrary objects. Probably they are stringified?
- MCP has a lot more structure to result types. It seems the basic idea is to allow inserting messages into the stream, so translated to our spec, you return an array of
LanguageModelResultContents. They also have a separate output type calledstructuredContentfor objects that conform to theoutputSchema.
As I said above, I think we can expand our current output format, to support both simple string returns and objects in the future. We should determine whether doing that would be valuable. From a developer ergonomics perspective, maybe it's nice to just not have to call JSON.stringify(). But maybe there's more benefits possible.
Developer experience on more complicated return value formats would be welcome.
Activity
domenic commentedon Aug 28, 2025
#149 takes a first step at expanding beyond strings, by allowing return types in the form
Array<{ type, value }>(mostly used for multimodal return types).I also want to note that we can always dodge back-compat issues here by minting new constructors. E.g., we could define a new
LanguageModelVerySpecialToolReturnTypeclass and then force developers to doreturn new LanguageModelVerySpecialToolReturnType({ ... interesting stuff ... }). It's slightly inelegant, compared to letting web developers just doreturn { ... interesting stuff ... }but it's a useful possibility if we need it.jimmywarting commentedon Jan 4, 2026
Uhm, wouldn't other thing other than string be useful... what if you had a tools that could generate image, audio, pdf, convert tar to zip and other binary stuff. returning at least typed arrays and arrayBuffer would be a +
I mean: you could as well return a string or a json as bytes instead also if you wanted to do that:
this are just ways you "could" return the same data but with different formats, making it still possible to be kind of MCP friendly towards others service it would kind of be the same way you could choose if you want to send blob, string, arraybuffer on websocket... al goes out to the other end the same way or the other. it could be the same as the way Request/Response body serialize the content of anything into bytes. and then transfers it.
made me think: why could we just return a
ReadableStreamthen?{ name: "getWeather", description: "Get the weather in a location.", inputSchema: { type: "object", properties: { location: { type: "string", description: "The city to check for the weather condition.", }, }, required: ["location"], }, async execute({ location }) { const res = await fetch("https://weatherapi.example/?location=" + location); - // Returns the result as a JSON string. - return JSON.stringify(await res.json()); // btw, this can be just: return await res.text() + return res.body }, }I know this is a bit different of what you are asking... about having to in return get a json response back instead of having to do
JSON.parse(result)all the time. but maybe the result shouldn't always be JSON all the time...? hence why i mention binary tools.I'm trying to think of a way servers could could respond back using the same structure as the above syntax flavor. they can't adopt the same logic of transfering other object literals.
michaelwasserman commentedon Jan 13, 2026
@FrankLi-MSFT is leading Chromium's initial prototyping for tool use, which we're hoping can accept a variety of IDL types for successful tool responses; see LanguageModelToolResultType, LanguageModelToolResultContent, and (in flux) LanguageModelToolSuccess.
We'll aim to update the explainer soon; hopefully this addresses your request.
nico-martin commentedon Jun 23, 2026
What is the current status of this? As it stands,
executestill expects a string as the return value. In my opinion, however, a tool should have exactly the same return type as the content of a user request.So basically this: https://github.com/DefinitelyTyped/DefinitelyTyped/blob/master/types/dom-chromium-ai/index.d.ts#L206-L209
Background: I am working on an Agent SDK for Transformers.js and I would like to align it closely with other on-device AI/Agent-implementation. And multimodal tool responses is one thing I needed for my implementation.
tomayac commentedon Jun 23, 2026
@nico-martin, see the in-flight #162 for the latest thinking around a closed and open loop model.
tomayac commentedon Sep 15, 2026
We should align the Prompt API's
outputSchemawith WebMCP's: webmachinelearning/webmcp#9.