Skip to content

feat(exec): evaluate ToolExecution-annotated calc definitions through the registered tool - #581

Merged
HuiJun merged 13 commits into
developfrom
feature/tool-execution-calc
Sep 26, 2026
Merged

HuiJun merged 13 commits into
developfrom
feature/tool-execution-calc

Conversation

@devin-ai-integration

@devin-ai-integration devin-ai-integration Bot commented Sep 25, 2026 •

Copy link
Copy Markdown
Contributor

What and why

Builds on #573 (invocation block), #577 (reply block) and #580 (sequence-valued outputs) and must merge after them; it targets feature/tool-sequence-outputs so the diff shows only its own delta, and follows that branch into develop.

A calc def or calc usage annotated metadata AnalysisTooling::ToolExecution { toolName = "…"; uri = "…"; } is now computed by the named external tool, exactly as an annotated action is performed by it. The body — when the calc states one — is never evaluated and never compiled; a bodiless annotated calc (calc def Thermal { in m : MassValue; in p : PowerValue; return : TemperatureValue; }) is the natural form and is no longer reported as uncomputed.

  • Every in/inout parameter carrying a ToolVariable is sent under that name; every out/inout one carrying a ToolVariable is bound from the reply. The result parameter is always answered, keyed by its ToolVariable name, else its declared name, else result; a calc with no result parameter asks for none and yields its designated output as a body that returns nothing does.
  • Same guard and typed failures as performByTool: an annotation naming no tool, no manifest, or a context with no runner is ToolNotRegisteredError (tool 'X' is not registered; set OPENSYSML_TOOLS); refusal, process failure, timeout, malformed/missing/unknown outputs, wrong unit dimension and unsent inputs are the same ToolError kinds; no fallback to the body, no invented default. Reply units convert to the declared quantity kind's coherent unit through the existing ToolCall.Bind. Equal inputs answered differently are noted as ToolDivergence with the calc's location.
  • Hooks: invokeCalcShapeIn (direct invocation, before runCalcBody) and runCalcUsage (before runCalcSteps); calcRun.value refuses to fall through to a binding for an output the tool left unanswered. The compiled tier withdraws a tool-annotated shape (compileBatch.compile), so a compilable caller such as Wrapper(x) = External(x) settles to the evaluator and reaches the tool rather than compiling the callee's body.
  • An optional ([0..1]) in parameter no argument binds is not sent, as an action's unbound optional input is not; a required one bound to a value the protocol cannot carry keeps its typed refusal.
  • ToolCall no longer holds an *ActionExecutor; output conversion (toolOutput, toolOutputValue) became *Context methods taking the scope units are read in, so the analysis runner's Bind works for both. ToolCall.Action keeps its name and now holds the annotated action or calc.
  • Only calcs compute by tool; an annotated analysis or verification case runs its body as before. A bodiless annotated calc specializing a library function (calc def Custom :> abs) answers from the tool, not the built-in: resolveLibraryPerformance counts the annotation as the calc's own computation.
  • A calc declaring no result parameter yields the one output the tool bound (designatedToolOutput); several bound outputs fail with ErrAmbiguousResult as a body computing several does, and no body binding is evaluated.
  • The runner reaches every evaluation context, not only analysis plans: gRPC newRuntimeContext(ctx, model) attaches Registry.ToolRunner bound to the request context (so EvaluateCalc, feature values, documents and held objects see it, and a cancelled request ends the tool it started; the cached held-object runtime rebinds its runner per request in heldObjects.lock(ctx)), the REPL report runtime attaches the session's runner, and document queries evaluating derived features now read through runtime.NewDeclaredReaderIn(held) which seeds the runner and expression parser from the query's runtime context and forwards the notes it makes (a ToolDivergence among them) to that context's Notes(). -calc, %calc, EvaluateCalc, attribute x = toolCalc(a, b) and document formulas all reach the tool.
  • -engines/%engines are unchanged: they list registered engines, not annotated elements.

Annotating the analysis case itself is not part of this change.

Specification basis

AnalysisTooling::ToolExecution / ToolVariable (SysML v2 Systems Library, Analysis Tooling); the existing external-tool row in docs/project/spec-compliance.md is extended to name calcs.

How it was verified

  • internal/exec/runtime/tool_calc_test.go (in-process recording runner): result bound, out bound, out-only calc asks for no result, kW→W conversion and wrong-dimension ToolMalformed, derived attribute through Instantiate, library specialization by tool (direct and expression form), compiled caller of an annotated calc (compilation on and off, with and without a runner), omitted optional input not sent, declared reader forwarding a divergence note, annotated case keeps its body, body-never-runs with an impossible body (Poisoned), no runner / empty toolName → ErrToolNotRegistered, runner error propagation, ToolMissingOutput/ToolUnknownOutput, divergence note; robustness_toolcalc_test.go for typed errors under malformed inputs.
  • internal/exec/analysis/tool_calc_test.go with the new testdata/toolcalc stand-in (built via tests/testutil/gobuild): registry runner answer with unit conversion, refusal, process failure, timeout, unregistered tool, calc usage form.
  • internal/frontend/repl/tool_calc_test.go (%calc answer and the not-registered message), cmd/sysml/tool_calc_test.go (-calc answer, not-registered error, and a rendered document whose column reads a derived attribute calling the tool calc), gRPC conformance EvaluateCalc cases evaluate_calc_tool / evaluate_calc_tool_unregistered plus TestGRPCToolCalcHonoursACancelledRequest (the harness gains an EvaluateCalc RPC and a tools field that builds the stand-in and points OPENSYSML_TOOLS at a manifest before the service starts; README schema updated).
  • Every existing tool, calc and execution-conformance test passes unchanged. go build ./..., go vet ./..., gofmt -l ., make lint, go test -race ./..., the training and pilot corpus gates with both require variables, python3 scripts/changelog.py check, python3 scripts/check-doc-links.py, python3 -m mkdocs build --strict. Man pages regenerated with make man for the reworded OPENSYSML_TOOLS help.

Checklist

  • make test and make lint pass locally
  • Tests added or updated for the change
  • Documentation extended where it already covers the surface (see CONTRIBUTING.md)
  • Changelog entry added as changes/unreleased/<slug>.<section>.md, not as an edit to CHANGELOG.md
  • baselines regenerated and make docs-counts run if a gate count moved (compliance rows need nothing: the census is counted at docs build)
  • No internal work-item labels (waves, slices, F4, K5) in the body, docs, or changelog

devin-ai-integration Bot and others added 2 commits September 25, 2026 06:47
A calc def or calc usage carrying AnalysisTooling::ToolExecution is computed
by the named tool on every calc surface (sysml -calc, %calc, EvaluateCalc,
derived attributes, document formulas): its ToolVariable-named in/inout
parameters are the request's inputs, its ToolVariable-named out/inout
parameters and its result parameter — keyed by its ToolVariable name, else
its declared name, else result — bind from the reply, and the body is never
evaluated or compiled. A tool unregistered, refusing or failing fails the
calculation with the same typed errors a performance gets; equal inputs
answered differently are noted as a ToolDivergence.

Co-Authored-By: jason.han <hanhuijun@gmail.com>
calcToolCall reports resultKey empty when the calc has no result parameter,
so an out-only tool calc demands no 'result' output of the reply; the
calculation yields its designated output as a body that returns nothing does,
and a usage's trace records CalcUsageExit rather than a returned value.

Co-Authored-By: jason.han <hanhuijun@gmail.com>
@devin-ai-integration

Copy link
Copy Markdown
Contributor Author

I'll fix CI failures and address comments from users with write access. I'll skip comments containing "(aside)".

  • Disable automatic comment, CI, and merge conflict monitoring

@devin-ai-integration
devin-ai-integration Bot marked this pull request as ready for review September 25, 2026 07:52

@devin-ai-integration devin-ai-integration Bot left a comment •

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note

Newer findings are available below. Devin Review posted a newer report on this PR, in addition to the findings presented here.

Devin Review found 5 potential issues.

1 flag not posted on this PR by your GitHub settings — view it in Devin Review. (Configure)

Devin Review

Comment thread internal/exec/runtime/invoke_calc.go Outdated
Comment thread internal/frontend/grpc/service.go Outdated
Comment thread internal/exec/analysis/tool.go
Comment thread internal/exec/analysis/reply.go
Comment thread internal/exec/analysis/tool.go Outdated
devin-ai-integration Bot and others added 2 commits September 25, 2026 08:10
A calc declaring only ToolVariable-marked outputs has no declared value or
body assignment for designatedOutput to see, so invoking it yielded
ErrCalcNoReturn though the tool answered. designatedToolOutput picks among
the outputs the tool actually bound — an answered result parameter winning
outright, one answered output alone, none or several failing as
designatedOutput reports — and its value is read from the reply, never
from a body binding.

Co-Authored-By: jason.han <hanhuijun@gmail.com>
newRuntimeContext bound the runner to context.Background(), so a cancelled
RPC left a tool subprocess running — under the held-object lock for the
cached population. The request context is threaded through newRuntime,
newRuntimeOver, newRuntimeContext and newVerifyContext; model().Fresh and
OpenSession bind to the background since plans rebind the runner on the
worker and a session runtime outlives a call, and heldObjects rebinds its
cached runtime's runner to the request inside lock().

Co-Authored-By: jason.han <hanhuijun@gmail.com>

@devin-ai-integration devin-ai-integration Bot left a comment •

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note

Newer findings are available below. Devin Review posted a newer report on this PR, in addition to the findings presented here.

Devin Review found 5 new potential issues.

1 flag not posted on this PR by your GitHub settings — view it in Devin Review. (Configure)

Devin Review

Comment thread internal/exec/analysis/reply.go
Comment thread internal/exec/runtime/invoke_calc.go Outdated
Comment thread internal/exec/runtime/calc_usage.go
Comment thread internal/exec/analysis/invocation.go
Comment thread internal/exec/analysis/invocation.go
devin-ai-integration Bot and others added 4 commits September 25, 2026 08:31
A bodiless calc def specializing a library function the runtime implements
was answered by the built-in — the library-performance path ran before the
shape was built and counted no tool. resolveLibraryPerformance treats a
calc carrying ToolExecution as computing its own answer, so the shape path
runs and the tool answers; a failed annotation read computes too, letting
the error surface where the shape reports it. The same guard covers
invocation expressions, which pick the same resolution.

Co-Authored-By: jason.han <hanhuijun@gmail.com>
…r body

calcInterfaceOf attached the annotation to every calc shape, so an
analysis or verification case carrying ToolExecution would have skipped
its body and steps for a tool. shape.Tool is recorded only for shapes
whose kind is calc; cases run as before while the annotation is deferred.

Co-Authored-By: jason.han <hanhuijun@gmail.com>
Co-Authored-By: jason.han <hanhuijun@gmail.com>
…ion-calc

Co-Authored-By: jason.han <hanhuijun@gmail.com>

# Conflicts:
#	internal/exec/runtime/tool.go
@devin-ai-integration
devin-ai-integration Bot changed the base branch from develop to feature/tool-sequence-outputs September 25, 2026 19:12
devin-ai-integration[bot]

This comment was marked as resolved.

devin-ai-integration Bot and others added 2 commits September 25, 2026 19:38
The compiled tier compiled a tool-annotated callee's body into callers that
compilably invoked it, so the tool never ran: compileBatch.compile withdraws
a tool shape as computed by tool, settle() then settles every caller to the
evaluator, and invokeCalcShapeIn no longer needs the tool excluded from the
compiled fast path — compiledCalcOf answers nil for the withdrawn shape.

An optional input no argument binds arrived at the protocol as bound with a
null value, which toolInput refuses: as an action's unbound optional sends
nothing, a ToolVariable input whose held value is null on an optional
parameter is simply not sent, while a required one keeps the typed refusal.

A declared reader's context noted a runner's divergence on itself, where
nothing reads it: a context seeded from another now forwards each note it
makes to the one it was seeded from, so NewDeclaredReaderIn reports the
divergence on the held runtime's Notes as a direct evaluation does.

Co-Authored-By: jason.han <hanhuijun@gmail.com>
…ion-calc

Co-Authored-By: jason.han <hanhuijun@gmail.com>
devin-ai-integration[bot]

This comment was marked as resolved.

devin-ai-integration Bot and others added 2 commits September 25, 2026 19:59
A runner answering without passing the reply through Bind could omit an
output, and a direct invocation bound the zero Value for it; the missing
output is refused as ToolMissingOutput, the same wording the usage path
reports, before the reply binds anything.

Co-Authored-By: jason.han <hanhuijun@gmail.com>
…ion-calc

Co-Authored-By: jason.han <hanhuijun@gmail.com>
Base automatically changed from feature/tool-sequence-outputs to develop September 26, 2026 03:08
devin-ai-integration[bot]

This comment was marked as resolved.

Document-derived calculations evaluated through NewDeclaredReaderIn now run under the held runtime's configured budgets instead of the defaults.

Co-Authored-By: jason.han <hanhuijun@gmail.com>
@HuiJun
HuiJun merged commit 27e5c08 into develop Sep 26, 2026
18 checks passed
@HuiJun
HuiJun deleted the feature/tool-execution-calc branch September 26, 2026 05:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant