tool use without auto execution by jingyun19 · Pull Request #162 · webmachinelearning/prompt-api

jingyun19 · 2025-11-19T18:02:24Z

Update explainer and spec to support tool use functionalities without automatic execution.

Explainer: added an example and explained how to make tool calls
Spec: reflect IDL changes in https://chromium-review.googlesource.com/c/chromium/src/+/7092943

Preview | Diff

Added detailed explanations for tool use modes, including examples for open loop and closed loop execution.

Updated README to clarify tool-call and tool-result usage.

Added new types and enums for tool calls and responses.

jingyun19 · 2025-11-19T18:06:50Z

cc @FrankLi-MSFT @sushraja-msft @reillyeon @michaelwasserman

tomayac

Tried to make the code samples more readable and correct. Maybe consider running them all through a tool like prettier, which catches typos like missing commas or parentheses.

As general feedback, could the explainer outline why developers would choose closed vs. open?

Co-authored-by: Thomas Steiner <tomac@google.com>

Added explanation about automatic execution and constraints in planner loop.

jingyun19 · 2025-12-01T17:30:30Z

I added a new section to describe use cases where open loop is preferred. cc @tomayac

reillyeon · 2025-12-11T01:33:53Z

I hadn't previously considered the context compression use case. That's interesting and motivating to enable developers to manipulate the conversation at this low level.

nico-martin

I think this implementation has a few weaknesses when it comes to distinguishing between a Message and a Message.Content element:
Message: Can have a specific role (whether it comes from the user, the assistant, or a tool call); it essentially describes the sender.
Message.Content: There can be multiple instances per Message; it describes the type of content.
I tried to make this concrete with a couple of comments.

nico-martin · 2026-06-23T07:13:45Z

+
+#### Do I need auto execution?
+
+In general, automatic execution is suitable for use cases where the model quality is good enough via prompt tuning. That can either mean you are tolerable for certain mistakes that the model makes when making tool calls, or the task is simple enough for the model to handle (e.g, just a few distinct tools, short and clean tool output, short context window, etc)


Not sure if I get this right. For the user, both, the closed and the open loop, are executed automatically. The only difference is that in an open loop, the developer has to execute the tools and start the next generation, while in the closed loop the loop will run without any extra steps.
Also if I dont want to have "automatic execution" as a developer, I could always intercept in the execute function. I would even argue for the wohle LLM conversation it is better to intercept a tool execution inside the execute function. Because then it allows you to return a reason why the tool was not executed intead of letting the model generate the tool call and then it does not know why it was not executed.

From the API client's perspective, there's no automatic execution in open loop. The API client need to read the tool name and arguments and invoke the functino themselves.
I also updated those sections in the explainer, PTAL and let me know if they make more sense now!

anssiko · 2026-08-14T12:12:46Z

@jingyun19 does the current PR reflect the Chromium implementation? The initial CL was merged. Remaining work?

jingyun19 · 2026-08-14T16:48:23Z

@jingyun19 does the current PR reflect the Chromium implementation? The initial CL was merged. Remaining work?

Yes it reflects the Chromium implementation. I believe the only remaining work is to support ToolCall type in input.

However, for us to do a dev trial, we also have the remaining work to fully support parsing and formatting tool types in the inference engine infrastructure.

Co-authored-by: Thomas Steiner <tomac@google.com>

jingyun19 · 2026-09-29T19:53:44Z

The updated spec is ready for review.

@reillyeon @michaelwasserman

reillyeon · 2026-09-29T23:35:30Z

+        1. If |content|["{{LanguageModelMessageContent/value}}"] is not a {{LanguageModelToolResponse}}, then throw a {{TypeError}}.
+
+        1. If |content|["{{LanguageModelMessageContent/value}}"] is a {{LanguageModelToolSuccess}}, then [=list/for each=] |resultItem| of |content|["{{LanguageModelMessageContent/value}}"]'s [=LanguageModelToolSuccess/result=]:
+          1. If |resultItem|["{{LanguageModelToolResultContent/type}}"] is "{{LanguageModelToolResultType/image}}" or "{{LanguageModelToolResultType/audio}}" and the user agent does not support multimodal tool result content, then throw a "{{NotSupportedError}}" {{DOMException}}.


I feel like this deserves a note that implementations should only expect to receive tool call response objects that they themselves could generate, so this case should never be hit in practice..

Because LanguageModelToolSuccess is constructed by API client, a developer may pass { type: "image", value: ... } or { type: "audio", value: ... }. If the expectedInputs only supports "text" and "object" tool results (or when the session wasn't configured with "image" / "audio" in expectedInputs), this NotSupportedError check can be hit in practice

michaelwasserman

Thanks for picking up this complex and important spec work again!

michaelwasserman · 2026-10-01T09:51:34Z

+console.log(result);
+```
+
+Note that a `"tool-response"` message should immediately follow the `"tool-call"` generated by the model.


Should the spec yield an error if that isn't the case?

Because "measureContextUsage(input)" runs validate and canonicalize a prompt on input on just the input, and clients may want to measure the token cost of a tool-response message before or independent of the current session state. If validate and canonicalize a prompt required a preceding tool-call, measureContextUsage() on a tool-response would throw.

Also for the model, the sequence will actually get prefilled to the model, so nothing throws an error in the stack, it just produces a non-standard token sequence in the model context, which is why I put it as a note (suggestion) rather than a normative error step.

michaelwasserman · 2026-10-01T10:10:29Z

FYI @FrankLi-MSFT and @sushraja-msft

Clarified expectedInputs and LanguageModelToolSuccess.result structure in README.

Clarify behavior of session.prompt() with expectedOutputs.

jingyun19 added 3 commits November 18, 2025 14:16

Enhance README with tool use modes and examples

14d59fc

Added detailed explanations for tool use modes, including examples for open loop and closed loop execution.

Clarify tool-call and tool-result in README

5379da1

Updated README to clarify tool-call and tool-result usage.

Enhance LanguageModel with tool call definitions

8e076f7

Added new types and enums for tool calls and responses.

tomayac reviewed Nov 26, 2025

View reviewed changes

jingyun19 and others added 5 commits November 26, 2025 09:08

Update README.md

ed78c12

Co-authored-by: Thomas Steiner <tomac@google.com>

Update README.md

5b11a1b

Co-authored-by: Thomas Steiner <tomac@google.com>

Merge branch 'webmachinelearning:main' into patch-1

cb8d807

add open vs. loop use case diff

5a0cbef

Enhance README with planner loop constraints details

81e75af

Added explanation about automatic execution and constraints in planner loop.

franzellekirk01-del approved these changes Jun 2, 2026

View reviewed changes

tomayac mentioned this pull request Jun 23, 2026

Tool calling: return types? #138

Open

nico-martin reviewed Jun 23, 2026

View reviewed changes

jingyun19 and others added 10 commits September 28, 2026 14:43

Merge branch 'main' into patch-1

a1e51c6

Apply suggestion from @tomayac

7afff12

Co-authored-by: Thomas Steiner <tomac@google.com>

Apply suggestion from @tomayac

fd46bb6

Co-authored-by: Thomas Steiner <tomac@google.com>

Apply suggestion from @tomayac

8ddbb23

Co-authored-by: Thomas Steiner <tomac@google.com>

Apply suggestion from @tomayac

b9bcf7b

Co-authored-by: Thomas Steiner <tomac@google.com>

Apply suggestion from @tomayac

b04dec1

Co-authored-by: Thomas Steiner <tomac@google.com>

Apply suggestion from @tomayac

0e6f916

Co-authored-by: Thomas Steiner <tomac@google.com>

Update tool use documentation in explainer to match implementation

3e35c89

Update tool use spec based on IDL and implementation

a5b01a3

update algorithms for tool call in spec

ef39f5d

reillyeon reviewed Sep 29, 2026

View reviewed changes

Comment thread index.bs

Comment thread index.bs

add note to spec

3d7b68b

reillyeon reviewed Sep 29, 2026

View reviewed changes

Comment thread index.bs Outdated

Comment thread index.bs Outdated

Comment thread index.bs Outdated

Comment thread index.bs Outdated

Comment thread index.bs Outdated

Comment thread index.bs Outdated

reillyeon reviewed Sep 29, 2026

View reviewed changes

tomayac reviewed Sep 30, 2026

View reviewed changes

Comment thread README.md Outdated

Comment thread README.md Outdated

rename callID and add defaults in spec

4a192b1

michaelwasserman reviewed Sep 30, 2026

View reviewed changes

Comment thread index.bs

jingyun19 added 2 commits September 30, 2026 16:55

update explainer to address comments

9730635

Rename callID to callId in explainer to match spec

1477f48

reillyeon approved these changes Oct 1, 2026

View reviewed changes

michaelwasserman reviewed Oct 1, 2026

View reviewed changes

update spec to address misc formatting issues

268e3d1

sushraja-msft reviewed Oct 1, 2026

View reviewed changes

Comment thread README.md Outdated

sushraja-msft reviewed Oct 1, 2026

View reviewed changes

Comment thread README.md Outdated

jingyun19 added 9 commits October 1, 2026 13:14

Refactor type and properties retrieval in index.bs

81ff758

Update explainer for tool use

04f5d0a

Clarified expectedInputs and LanguageModelToolSuccess.result structure in README.

Enhance README with details on session.prompt()

9547fe5

Clarify behavior of session.prompt() with expectedOutputs.

Refactor algorithm for handling expected outputs

200a2da

Clarify tool use modes and examples in README

55b4c65

add back concurrent tool use

7ce4f5b

fix PR bot error

d5eee6c

fix rendering

2b64d82

Fix formatting issues in spec

2fdf71a


		#### Do I need auto execution?

		In general, automatic execution is suitable for use cases where the model quality is good enough via prompt tuning. That can either mean you are tolerable for certain mistakes that the model makes when making tool calls, or the task is simple enough for the model to handle (e.g, just a few distinct tools, short and clean tool output, short context window, etc)

Conversation

jingyun19 commented Nov 19, 2025 • edited by pr-preview Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jingyun19 commented Nov 19, 2025

Uh oh!

tomayac left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jingyun19 commented Dec 1, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

reillyeon commented Dec 11, 2025

Uh oh!

nico-martin left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

nico-martin Jun 23, 2026

Choose a reason for hiding this comment

Uh oh!

jingyun19 Oct 1, 2026

Choose a reason for hiding this comment

Uh oh!

anssiko commented Aug 14, 2026

Uh oh!

jingyun19 commented Aug 14, 2026

Uh oh!

jingyun19 commented Sep 29, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

reillyeon Sep 29, 2026

Choose a reason for hiding this comment

Uh oh!

jingyun19 Oct 1, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

michaelwasserman left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

michaelwasserman Oct 1, 2026

Choose a reason for hiding this comment

Uh oh!

jingyun19 Oct 1, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jingyun19 commented Nov 19, 2025 •

edited by pr-preview Bot

Loading

jingyun19 commented Dec 1, 2025 •

edited

Loading