feat(cli): preview external tool invocations and document the external-tool workflow

[devin-ai-integration]

What and why

Builds on #573 (invocation block) and #577 (reply block) and must merge after them; the branch is cut from feature/tool-reply-formats, so until those land the diff shows all three. Once they merge this PR will be rebased onto develop so only its delta shows.

With a manifest able to compose argv/env/stdin from model values and parse arbitrary replies, an author needs to see what a tool would be handed before trusting a run, and a recorded run needs to say which program produced it. This PR adds those surfaces, the documentation of the whole workflow, and a worked example.

• Dry run. sysml -tool-dry-run <case|action> and %tool <case|action>[(<args>)] [<object>] perform the target in the session's own context under a ToolRunner that, at the first tool call reached, composes the invocation from the resolved manifest entry and fails the performance with a typed analysis.ToolDryRunError the surface swallows — nothing after the call runs and no process starts. The performance runs on a runtime.Snapshot of the session's context that is restored afterwards, so the preview sees the session's current values and leaves no trace of what it performed. The DryRunner remembers the first preview it made, so a call reached only while an objective or assertion is evaluated is still reported; an exploring schedule is refused (ExplorePreviewError), since a preview shows one run's first call. The session's own tool runner is put back after the preview. Monte Carlo row records bound their tool calls at a mark taken between settling the runs and concluding the case (SettleMonteCarloRuns / MonteCarloRun.Conclude), so the conclusion's calls appear only on the sample record. The preview lists manifest, executable, protocol, each argv element as %q, the environment (inherited from this process for an entry without an invocation block), cwd, the stdin mode and its rendered bytes, the input file's bytes, the inputs and outputs of the call, and the reply mapping per output (path / column / row / unit selectors, type). {inputFile} and {outputDir} render as <inputFile> and <outputDir> since no directory is made. compose is split into callScope / InputFile.inputData / render so the preview and the real invocation share one rendering path.
  • A missing manifest, an unregistered tool, an unaccepted variable, an absent executable or an unsent input reports the same typed error the real run fails with; a run that reaches no tool says no ToolExecution-annotated action was reached; nothing to preview. A case with several tool actions shows the first (the later ones depend on outputs the dry run does not have).
  • -sweep refuses -tool-dry-run; the action path uses CreateActionExecutorFor + RunToCompletion directly so no %action debugger session is left behind.
• Provenance. analysis.Result.Tool (ToolUse: tool, version, manifest file, executable, argv — or the request read on stdin for an object entry) is filled by the tool engine; the plan's toolRunner accumulates them, each attributed to the runtime context that made the call, and Plan.Tools / Plan.ToolTexts() / Plan.ToolTextsIn(ctx) expose them in call order. A call that failed after its process was started (spawn, reply read, binding) is kept too: the engine returns a ToolUseError wrapping the failure, and its ToolUse.Failed renders as … failed: <message>. record.Run.Tools writes each run's own calls into its record as tools = ("ThermalSolver 2.3 from /etc/opensysml/tools/thermal.json: /usr/bin/python3 solve.py --mass 12.5", …);, and AnalysisRecords::RecordedRun gains attribute tools : String[0..*];. Records of runs that reached no tool are byte-for-byte unchanged (existing goldens untouched; a new tool_run.sysml.golden covers the attribute). Sweep rows and Monte Carlo runs each list only the calls their own row made; the whole-sample record lists them all.
• Protocol column. Already on the wire in ListEngines and in the Python Engine.protocol; the proto comment and the Python docstring now describe the tool spellings (object, argv+<stdin>/<reply>), Engine.explain() shows kind and protocol for tools as it does for engines, with a Python client test and a gRPC test asserting argv+none/csv for a composed entry.
• Docs. docs/manual/running-external-programs.md ("Running external programs from a model") walks manifest → model → run → record → document and is linked from the analysis and recording pages and from the External tools section of docs/reference/environment.md, which covers OPENSYSML_TOOLS, OPENSYSML_TOOL_TIMEOUT, OPENSYSML_TOOL_MAX_OUTPUT, OPENSYSML_TOOL_ENV_PASSTHROUGH and OPENSYSML_TOOL_KEEP. -tool-dry-run / %tool are in the CLI and REPL references and the regenerated manual page.
• Example. examples/external-tool-demo/: a stdlib-only Python script taking --mass --power --ambient and answering CSV, its manifest (executable: python3 looked up on PATH, cwd: "." so the script sits beside the manifest), a model whose Solve action carries ToolExecution, a Records package and a document tabulating the recorded runs, a README with the exact commands (the manifest directory must live outside every workspace, so the README copies tools/ out of the checkout), and the rendered report.md. TestExternalToolDemo in cmd/sysml runs dry run → -record-run → Markdown render end to end and skips when python3 is absent.

Known limitation: a document Project reads a record's own features, not its annotation's attributes, so tools is read in the record's source rather than tabulated (same as runAt/command today).

How it was verified

• internal/frontend/repl/tool_test.go: %tool happy path over an invocation+reply entry (argv, env, stdin, input file, <outputDir>, reply lines, the tool did not run), object-protocol entry, no manifest, unsent input, action target leaving no debugger session, the session's values, objects and trace left as the preview found them, an action's positional and named arguments bound before the preview (ctx.ActionInputs), usage and %help.
• cmd/sysml/tool_dry_run_test.go: -tool-dry-run happy path, no manifest, unsent input; cmd/sysml/external_tool_demo_test.go: the demo end to end.
• internal/exec/analysis/record: new tool_run.sysml.golden; repl record test asserting tools = (…) for a tool-computed case; internal/frontend/grpc/engines_test.go protocol assertion; client/python/tests/test_engines.py.
• go build ./..., go vet ./..., gofmt -l ., make lint, go test -race ./..., the training and pilot corpus gates with both require variables, make man-check, make docs-check, make docs-counts, make stdlib-snapshot-check, python -m mkdocs build --strict, python3 scripts/check-doc-links.py, python3 scripts/changelog.py check, pytest client/python/tests.
• Ratchets moved only by the new example file: corpus_roundtrip_expected.txt and api_json_roundtrip_expected.txt gain one stable row each; the pilot-differential baseline and doc-count blocks were regenerated for the 45th example.

Checklist

• make test and make lint pass locally
• Tests added or updated for the change
• Documentation extended where it already covers the surface (see CONTRIBUTING.md)
• Changelog entry added as changes/unreleased/<slug>.<section>.md, not as an edit to CHANGELOG.md
• baselines regenerated and make docs-counts run if a gate count moved (compliance rows need nothing: the census is counted at docs build)
• No internal work-item labels (waves, slices, F4, K5) in the body, docs, or changelog

[devin-ai-integration]

I'll fix CI failures and address comments from users with write access. I'll skip comments containing "(aside)".

• Disable automatic comment, CI, and merge conflict monitoring

[devin-ai-integration]

Note

Newer findings are available below. Devin Review posted a newer report on this PR, in addition to the findings presented here.

Devin Review found 6 new potential issues.

2 flags not posted on this PR by your GitHub settings — view them in Devin Review. (Configure)

[devin-ai-integration]

🟨 Tool preview exposes passed-through credentials

When a tool receives credentials through OPENSYSML_TOOL_ENV_PASSTHROUGH, DryRun.Lines prints their values verbatim. CLI and REPL preview output can expose those credentials.

---

Was this helpful? React with 👍 or 👎 to provide feedback.

[devin-ai-integration]

Intended: the preview's purpose is to show exactly what the process would be started with, environment included. The only values that can appear there are the manifest's own env entries and variables the operator explicitly listed in OPENSYSML_TOOL_ENV_PASSTHROUGH from their own process environment — nothing the model or another party supplies — and the preview prints to the operator's terminal, not to a file or record.

[devin-ai-integration]

🟨 Run records persist sensitive tool inputs

When a tool receives sensitive inputs, ToolUse.String embeds argv or its full JSON request in recorded provenance. Exported records then retain those values.

---

Was this helpful? React with 👍 or 👎 to provide feedback.

[devin-ai-integration]

Intended: the record already stores every input value the run was made with (inputs), and the argv / request the tool was given is composed from those same values — that is the provenance the tools attribute exists to keep, so a reader can reproduce the run. A value that must not be written into a record must not be an input of a recorded run; -record-run is an explicit, opt-in act by the modeller.

[devin-ai-integration]

Devin Review found 1 new potential issue.

1 flag not posted on this PR by your GitHub settings — view it in Devin Review. (Configure)

[devin-ai-integration]

🟡 Preview bypasses an object's existing action

When an object already performs an action, runActionToCompletion starts another performance. performanceOf instead joins the existing one or rejects explicit inputs, so the preview can describe a call the run cannot make.

Learn more

An object can already have an action performance attached to it. The ordinary action path uses performanceOf to find and continue that performance, and rejects new arguments when it exists. The preview creates a fresh executor instead, so it can preview an action with different inputs or from a different point in its execution.

Example: An object already performs Heating with mass 12.5. Previewing Heating(30) shows mass 30, although the ordinary action path refuses those explicit inputs.

Recommended fix: Make the action preview use the same existing-performance selection and input checks as the ordinary action path, while restoring the context snapshot afterward.

---

Was this helpful? React with 👍 or 👎 to provide feedback.