feat(exec): evaluate ToolExecution-annotated calc definitions through the registered tool - #581
Merged
Merged
Conversation
A calc def or calc usage carrying AnalysisTooling::ToolExecution is computed by the named tool on every calc surface (sysml -calc, %calc, EvaluateCalc, derived attributes, document formulas): its ToolVariable-named in/inout parameters are the request's inputs, its ToolVariable-named out/inout parameters and its result parameter — keyed by its ToolVariable name, else its declared name, else result — bind from the reply, and the body is never evaluated or compiled. A tool unregistered, refusing or failing fails the calculation with the same typed errors a performance gets; equal inputs answered differently are noted as a ToolDivergence. Co-Authored-By: jason.han <hanhuijun@gmail.com>
calcToolCall reports resultKey empty when the calc has no result parameter, so an out-only tool calc demands no 'result' output of the reply; the calculation yields its designated output as a body that returns nothing does, and a usage's trace records CalcUsageExit rather than a returned value. Co-Authored-By: jason.han <hanhuijun@gmail.com>
Contributor
Author
|
I'll fix CI failures and address comments from users with write access. I'll skip comments containing "(aside)".
|
Contributor
Author
There was a problem hiding this comment.
Note
Newer findings are available below. Devin Review posted a newer report on this PR, in addition to the findings presented here.
Devin Review found 5 potential issues.
1 flag not posted on this PR by your GitHub settings — view it in Devin Review. (Configure)
A calc declaring only ToolVariable-marked outputs has no declared value or body assignment for designatedOutput to see, so invoking it yielded ErrCalcNoReturn though the tool answered. designatedToolOutput picks among the outputs the tool actually bound — an answered result parameter winning outright, one answered output alone, none or several failing as designatedOutput reports — and its value is read from the reply, never from a body binding. Co-Authored-By: jason.han <hanhuijun@gmail.com>
newRuntimeContext bound the runner to context.Background(), so a cancelled RPC left a tool subprocess running — under the held-object lock for the cached population. The request context is threaded through newRuntime, newRuntimeOver, newRuntimeContext and newVerifyContext; model().Fresh and OpenSession bind to the background since plans rebind the runner on the worker and a session runtime outlives a call, and heldObjects rebinds its cached runtime's runner to the request inside lock(). Co-Authored-By: jason.han <hanhuijun@gmail.com>
Contributor
Author
There was a problem hiding this comment.
Note
Newer findings are available below. Devin Review posted a newer report on this PR, in addition to the findings presented here.
Devin Review found 5 new potential issues.
1 flag not posted on this PR by your GitHub settings — view it in Devin Review. (Configure)
A bodiless calc def specializing a library function the runtime implements was answered by the built-in — the library-performance path ran before the shape was built and counted no tool. resolveLibraryPerformance treats a calc carrying ToolExecution as computing its own answer, so the shape path runs and the tool answers; a failed annotation read computes too, letting the error surface where the shape reports it. The same guard covers invocation expressions, which pick the same resolution. Co-Authored-By: jason.han <hanhuijun@gmail.com>
…r body calcInterfaceOf attached the annotation to every calc shape, so an analysis or verification case carrying ToolExecution would have skipped its body and steps for a tool. shape.Tool is recorded only for shapes whose kind is calc; cases run as before while the annotation is deferred. Co-Authored-By: jason.han <hanhuijun@gmail.com>
Co-Authored-By: jason.han <hanhuijun@gmail.com>
…ion-calc Co-Authored-By: jason.han <hanhuijun@gmail.com> # Conflicts: # internal/exec/runtime/tool.go
devin-ai-integration
Bot
changed the base branch from
develop
to
feature/tool-sequence-outputs
September 25, 2026 19:12
The compiled tier compiled a tool-annotated callee's body into callers that compilably invoked it, so the tool never ran: compileBatch.compile withdraws a tool shape as computed by tool, settle() then settles every caller to the evaluator, and invokeCalcShapeIn no longer needs the tool excluded from the compiled fast path — compiledCalcOf answers nil for the withdrawn shape. An optional input no argument binds arrived at the protocol as bound with a null value, which toolInput refuses: as an action's unbound optional sends nothing, a ToolVariable input whose held value is null on an optional parameter is simply not sent, while a required one keeps the typed refusal. A declared reader's context noted a runner's divergence on itself, where nothing reads it: a context seeded from another now forwards each note it makes to the one it was seeded from, so NewDeclaredReaderIn reports the divergence on the held runtime's Notes as a direct evaluation does. Co-Authored-By: jason.han <hanhuijun@gmail.com>
…ion-calc Co-Authored-By: jason.han <hanhuijun@gmail.com>
A runner answering without passing the reply through Bind could omit an output, and a direct invocation bound the zero Value for it; the missing output is refused as ToolMissingOutput, the same wording the usage path reports, before the reply binds anything. Co-Authored-By: jason.han <hanhuijun@gmail.com>
…ion-calc Co-Authored-By: jason.han <hanhuijun@gmail.com>
Document-derived calculations evaluated through NewDeclaredReaderIn now run under the held runtime's configured budgets instead of the defaults. Co-Authored-By: jason.han <hanhuijun@gmail.com>
This was referenced Sep 26, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What and why
Builds on #573 (
invocationblock), #577 (replyblock) and #580 (sequence-valued outputs) and must merge after them; it targetsfeature/tool-sequence-outputsso the diff shows only its own delta, and follows that branch intodevelop.A
calc defor calc usage annotatedmetadata AnalysisTooling::ToolExecution { toolName = "…"; uri = "…"; }is now computed by the named external tool, exactly as an annotated action is performed by it. The body — when the calc states one — is never evaluated and never compiled; a bodiless annotated calc (calc def Thermal { in m : MassValue; in p : PowerValue; return : TemperatureValue; }) is the natural form and is no longer reported as uncomputed.in/inoutparameter carrying aToolVariableis sent under that name; everyout/inoutone carrying aToolVariableis bound from the reply. The result parameter is always answered, keyed by itsToolVariablename, else its declared name, elseresult; a calc with no result parameter asks for none and yields its designated output as a body that returns nothing does.performByTool: an annotation naming no tool, no manifest, or a context with no runner isToolNotRegisteredError(tool 'X' is not registered; set OPENSYSML_TOOLS); refusal, process failure, timeout, malformed/missing/unknown outputs, wrong unit dimension and unsent inputs are the sameToolErrorkinds; no fallback to the body, no invented default. Reply units convert to the declared quantity kind's coherent unit through the existingToolCall.Bind. Equal inputs answered differently are noted asToolDivergencewith the calc's location.invokeCalcShapeIn(direct invocation, beforerunCalcBody) andrunCalcUsage(beforerunCalcSteps);calcRun.valuerefuses to fall through to a binding for an output the tool left unanswered. The compiled tier withdraws a tool-annotated shape (compileBatch.compile), so a compilable caller such asWrapper(x) = External(x)settles to the evaluator and reaches the tool rather than compiling the callee's body.[0..1])inparameter no argument binds is not sent, as an action's unbound optional input is not; a required one bound to a value the protocol cannot carry keeps its typed refusal.ToolCallno longer holds an*ActionExecutor; output conversion (toolOutput,toolOutputValue) became*Contextmethods taking the scope units are read in, so the analysis runner'sBindworks for both.ToolCall.Actionkeeps its name and now holds the annotated action or calc.calc def Custom :> abs) answers from the tool, not the built-in:resolveLibraryPerformancecounts the annotation as the calc's own computation.designatedToolOutput); several bound outputs fail withErrAmbiguousResultas a body computing several does, and no body binding is evaluated.newRuntimeContext(ctx, model)attachesRegistry.ToolRunnerbound to the request context (soEvaluateCalc, feature values, documents and held objects see it, and a cancelled request ends the tool it started; the cached held-object runtime rebinds its runner per request inheldObjects.lock(ctx)), the REPL report runtime attaches the session's runner, and document queries evaluating derived features now read throughruntime.NewDeclaredReaderIn(held)which seeds the runner and expression parser from the query's runtime context and forwards the notes it makes (aToolDivergenceamong them) to that context'sNotes().-calc,%calc,EvaluateCalc,attribute x = toolCalc(a, b)and document formulas all reach the tool.-engines/%enginesare unchanged: they list registered engines, not annotated elements.Annotating the analysis case itself is not part of this change.
Specification basis
AnalysisTooling::ToolExecution/ToolVariable(SysML v2 Systems Library, Analysis Tooling); the existing external-tool row indocs/project/spec-compliance.mdis extended to name calcs.How it was verified
internal/exec/runtime/tool_calc_test.go(in-process recording runner): result bound,outbound, out-only calc asks for noresult, kW→W conversion and wrong-dimensionToolMalformed, derived attribute throughInstantiate, library specialization by tool (direct and expression form), compiled caller of an annotated calc (compilation on and off, with and without a runner), omitted optional input not sent, declared reader forwarding a divergence note, annotated case keeps its body, body-never-runs with an impossible body (Poisoned), no runner / emptytoolName→ErrToolNotRegistered, runner error propagation,ToolMissingOutput/ToolUnknownOutput, divergence note;robustness_toolcalc_test.gofor typed errors under malformed inputs.internal/exec/analysis/tool_calc_test.gowith the newtestdata/toolcalcstand-in (built viatests/testutil/gobuild): registry runner answer with unit conversion, refusal, process failure, timeout, unregistered tool, calc usage form.internal/frontend/repl/tool_calc_test.go(%calcanswer and the not-registered message),cmd/sysml/tool_calc_test.go(-calcanswer, not-registered error, and a rendered document whose column reads a derived attribute calling the tool calc), gRPC conformanceEvaluateCalccasesevaluate_calc_tool/evaluate_calc_tool_unregisteredplusTestGRPCToolCalcHonoursACancelledRequest(the harness gains anEvaluateCalcRPC and atoolsfield that builds the stand-in and pointsOPENSYSML_TOOLSat a manifest before the service starts; README schema updated).go build ./...,go vet ./...,gofmt -l .,make lint,go test -race ./..., the training and pilot corpus gates with both require variables,python3 scripts/changelog.py check,python3 scripts/check-doc-links.py,python3 -m mkdocs build --strict. Man pages regenerated withmake manfor the rewordedOPENSYSML_TOOLShelp.Checklist
make testandmake lintpass locallychanges/unreleased/<slug>.<section>.md, not as an edit toCHANGELOG.mdmake docs-countsrun if a gate count moved (compliance rows need nothing: the census is counted at docs build)F4,K5) in the body, docs, or changelog