diff --git a/.agents/skills/testing-pilot-corpora-gate/SKILL.md b/.agents/skills/testing-pilot-corpora-gate/SKILL.md index eeeeacca8..19f20fd05 100644 --- a/.agents/skills/testing-pilot-corpora-gate/SKILL.md +++ b/.agents/skills/testing-pilot-corpora-gate/SKILL.md @@ -183,8 +183,8 @@ gate's own helpers are package-private but reusable (`pilotCorporaGate.files(t)` `actionlint`, `shellcheck`, `python3 scripts/check-doc-links.py`, `gofmt`, `go vet`, `go run -C tools ./cmd/pilot-diff` (validators pre-downloaded; ~4min, prints e.g. -the headline the committed baseline holds — `379 file(s), 347 fully agreeing; 38 agreed -diagnostic(s), 38 only ours, 1582 only the pilot's` at the `2026-08` pin, so read it from +the headline the committed baseline holds — `380 file(s), 347 fully agreeing; 38 agreed +diagnostic(s), 38 only ours, 1614 only the pilot's` at the `2026-08` pin, so read it from `docs/project/pilot-differential-baseline.json` rather than from this line) and `make lint` (staticcheck+gosec, ~2min) all work. There is **no** `yamllint` and **no** `circleci` CLI, so `.circleci/config.yml` can only be parsed as YAML, not schema-validated — say so diff --git a/.agents/skills/testing-pilot-differential/SKILL.md b/.agents/skills/testing-pilot-differential/SKILL.md index bea4275b7..033485c39 100644 --- a/.agents/skills/testing-pilot-differential/SKILL.md +++ b/.agents/skills/testing-pilot-differential/SKILL.md @@ -21,9 +21,9 @@ GNU-format diagnostics **relative to `--root`**. Consequences for testing: - The pin `tools/referee/diff` reports comes from `build/pilot-sysml-validator/pilot-pin.txt` (written by the new script), not from the DeciSym `pom.xml`. - `-validator /nonexistent` now says `run ./scripts/download-pilot-sysml-validator.sh`. -- Measured at the `2026-08` pin, with a fresh library cache: `379 file(s), 347 fully agreeing; 38 agreed, - 38 only ours, 1582 only the pilot's`, JSON totals `openSysMLDiagnostics 79 / pilotDiagnostics - 1623 / severityMismatch 3`; ~2 min wall, byte-identical across runs *and* after a from-scratch +- Measured at the `2026-08` pin, with a fresh library cache: `380 file(s), 347 fully agreeing; 38 agreed, + 38 only ours, 1614 only the pilot's`, JSON totals `openSysMLDiagnostics 79 / pilotDiagnostics + 1655 / severityMismatch 3`; ~2 min wall, byte-identical across runs *and* after a from-scratch rebuild of `build/pilot-validator`. The six `kerml-examples` pilot-only rows the `2026-07` run carried (`The opposite features 'owningType' … do not refer to each other`) are gone: the pilot fixed its `ownedDisjoining` delegate, and nothing on our side moved. `kerml-examples` carries no `syntax` diagnostic on either @@ -137,7 +137,7 @@ The harness compares OpenSysML diagnostics against the OMG SysML v2 Pilot Implem `build/pilot-diff/pilot-diff.{txt,json}`. `docs/project/pilot-differential-baseline.json` is the committed result of the *last refreshed* run, so **the harness is testable by reproduction** — but only while the baseline is current. Check that first. As of the rebaseline that came when the Legend of the Red Dragon example left for its own repository it **is** -current: a live run gives `379 file(s), 347 fully agreeing; 38 agreed, 38 only ours, 1582 only the +current: a live run gives `380 file(s), 347 fully agreeing; 38 agreed, 38 only ours, 1614 only the pilot's`, byte-identical to the committed baseline, and `docs/project/pilot-differential.md`'s "Results" table matches. The rebaseline before it, at the architecture self-model's landing, covered two rounds, because the succession-shorthand removal before it landed without refreshing the baseline; a control run of its merge commit gives diff --git a/.agents/skills/testing-pilot-execution-referee/SKILL.md b/.agents/skills/testing-pilot-execution-referee/SKILL.md index 01e344319..2448df190 100644 --- a/.agents/skills/testing-pilot-execution-referee/SKILL.md +++ b/.agents/skills/testing-pilot-execution-referee/SKILL.md @@ -148,8 +148,8 @@ pilot answers the representation's own. See `pilot-exec-diff: :: model no/such/model.sysml: stat : no such file or directory`. - **Additivity.** `go run -C tools ./cmd/pilot-diff` must still print the headline the - committed baseline holds (`379 file(s), 347 fully agreeing; 38 agreed - diagnostic(s), 38 only ours, 1582 only the pilot's` at the `2026-08` pin — read it from the baseline JSON, not from this line, since each + committed baseline holds (`380 file(s), 347 fully agreeing; 38 agreed + diagnostic(s), 38 only ours, 1614 only the pilot's` at the `2026-08` pin — read it from the baseline JSON, not from this line, since each fix round moves it) and `jq -S` diff clean against `docs/project/pilot-differential-baseline.json`; `git status --porcelain` empty at the end. diff --git a/.agents/skills/testing-pilot-xpect/SKILL.md b/.agents/skills/testing-pilot-xpect/SKILL.md index e36f10d4e..0b6739453 100644 --- a/.agents/skills/testing-pilot-xpect/SKILL.md +++ b/.agents/skills/testing-pilot-xpect/SKILL.md @@ -416,8 +416,8 @@ census in `w5c_census_test.go` is live two ways: perturb one pinned triple (e.g. ## Regression neighbour `go run -C tools ./cmd/pilot-diff` (~1m12s) must still print the headline the *committed* baseline holds — -at the `2026-08` pin that is `379 file(s), 347 fully agreeing; 38 agreed diagnostic(s), 38 -only ours, 1582 only the pilot's`. Read the number out of +at the `2026-08` pin that is `380 file(s), 347 fully agreeing; 38 agreed diagnostic(s), 38 +only ours, 1614 only the pilot's`. Read the number out of `docs/project/pilot-differential-baseline.json` rather than trusting this line, since a landing fix round moves it. When the baseline is itself stale (it was at `19a3ce03`, holding 273 / 281 / 317), a failing `cmp` against it is *not* evidence of an Xpect regression — compare the summary line, and see diff --git a/README.md b/README.md index ef255be30..f7201ff6d 100644 --- a/README.md +++ b/README.md @@ -317,11 +317,11 @@ The project is under active development, with the core infrastructure operationa **Measured against the pinned reference** (`PILOT_TAG=2026-08`, artifact `0.62.0`). Every number below is generated by `make docs-counts` from the committed baselines and gated; none of them is typed in by hand. -- **Corpus agreement:** 347 of 379 files agree diagnostic-by-diagnostic; 38 diagnostics are ours alone and 1582 the reference's alone, and the first number must be read by root: our diagnostics against the reference's own corpora fell while our non-standard-notation warnings on our own example models rose ([differential](docs/project/pilot-differential.md), `go run -C tools ./cmd/pilot-diff`). +- **Corpus agreement:** 347 of 380 files agree diagnostic-by-diagnostic; 38 diagnostics are ours alone and 1614 the reference's alone, and the first number must be read by root: our diagnostics against the reference's own corpora fell while our non-standard-notation warnings on our own example models rose ([differential](docs/project/pilot-differential.md), `go run -C tools ./cmd/pilot-diff`). - **Declared-diagnostic silence:** of the 512 declared `errors` rows in the reference's own Xpect suites, we report nothing for 0. 245 we report word-for-word; 248 wording-only and 7 location-only differences are agreement in substance and are not counted as gaps; 0 more we report as a warning and 2 elsewhere in the file ([Xpect oracle](docs/project/pilot-xpect.md), `go run -C tools ./cmd/pilot-xpect`). - **Scope agreement:** 230 of 230 declared scope assertions match exactly (same source). - **Permissiveness gaps:** of 306 invalid models we wrote ourselves, the reference rejects 4 that we accept by default, and 293 both reject; 4 further cases agree only when we are asked strictly. We authored every one of these cases ourselves, so the denominator measures the reach of our own corpus and not our conformance; agreement reached only under an opt-in strict mode is weaker evidence than agreement by default ([rejection oracle](docs/project/pilot-rejection.md), `go run -C tools ./cmd/pilot-reject`). -- **Declared errata:** the registry declares 12 defect(s) in the published reference material — 4 with a specification-derived correction, 8 documented without one, since no intended reading can be inferred ([OMG issues](docs/project/omg-issues.md), `tools/oracle/errata`). Every figure above is as published and stays the conformance statement; running the same oracles over the corrected text instead reports 348 of 379 files agreeing, 37 diagnostics ours alone and 1582 the reference's alone, 0 declared rows we are silent on, and 0 of 306 authored cases the reference alone rejects. The corrected figures are diagnostic only: an erratum never reclassifies a divergence category, and the published corpus is never edited. +- **Declared errata:** the registry declares 12 defect(s) in the published reference material — 4 with a specification-derived correction, 8 documented without one, since no intended reading can be inferred ([OMG issues](docs/project/omg-issues.md), `tools/oracle/errata`). Every figure above is as published and stays the conformance statement; running the same oracles over the corrected text instead reports 348 of 380 files agreeing, 37 diagnostics ours alone and 1614 the reference's alone, 0 declared rows we are silent on, and 0 of 306 authored cases the reference alone rejects. The corrected figures are diagnostic only: an erratum never reclassifies a divergence category, and the published corpus is never edited. - **Self-assessed surface:** the action, state-machine and classifier-behavior rows have no external referee at all — the four refereed figures above cannot see them, because the pinned artifact evaluates expressions but executes neither actions nor state machines. [Spec compliance](docs/project/spec-compliance.md) counts them. What these numbers cannot show: the OMG corpora are demonstrations rather than an official conformance suite; the differential is one-directional, comparing the diagnostics the two implementations report on the same files; the Xpect suites are the pilot authors' test intent rather than a certification oracle; and none of these is a percentage of the specification — no global compliance figure is claimed anywhere. @@ -333,7 +333,7 @@ What these numbers cannot show: the OMG corpora are demonstrations rather than a **Test coverage:** top-level `Test` functions (counted from the `_test.go` files, as `go test ./...` runs them) covering parsers, semantics, runtime (actions, states, instances, operators, validation), behind golden ASTs, negatives, execution conformance cases, golden traces, runtime robustness cases and gRPC conformance and robustness cases. The figures are counted from the tree when the documentation site is built into the test inventory of [spec compliance](docs/project/spec-compliance.md), never committed, so a branch adding a test does not rewrite this page. A test skips only for want of something the run did not provide, and says what: the held-image round trip declines a conformance case that creates no instance, a few gate on a PDF or Mermaid toolchain, a pinned pilot artifact, the PSSM suite, a locale, a case-insensitive filesystem or a live Flexo stack, and the OMG corpus gates skip until the corpora are downloaded unless asked to fail. **Parser coverage:** 105/105 bundled library files parse cleanly — the 94 official SysML v2 standard library files and the non-normative `OpenSysML Libraries/OpenSysMLMathFunctions.kerml`, `OpenSysML Libraries/DocumentQueries.sysml`, `OpenSysML Libraries/IdentityMetadata.sysml`, `OpenSysML Libraries/DiagramLayout.sysml`, `OpenSysML Libraries/OOSEM.sysml`, `OpenSysML Libraries/MOSA.sysml`, `OpenSysML Libraries/StateSpaceIntegration.sysml`, `OpenSysML Libraries/Stochastic.sysml`, `OpenSysML Libraries/RandomFunctions.kerml`, `OpenSysML Libraries/Simulation.sysml` and `OpenSysML Libraries/MigrationMetadata.sysml` extensions. Conformance verified by [stdlib_conformance_test.go](internal/workspace/libs/stdlib_conformance_test.go). Grammar reference: [OMG Xtext grammar](https://github.com/Systems-Modeling/SysML-v2-Pilot-Implementation/tree/master/org.omg.kerml.xtext/src/org/omg/kerml/xtext). **Behavioral execution:** Calc/constraint/requirement/satisfy functional. Action/state executors handle nested invocation, control flow keywords, loop and conditional statements and the send statement (every conformance case passing). Coverage is self-assessed against the specification text and the normative library: the pinned OMG pilot implementation evaluates expressions but does not execute actions or state machines headlessly, so no external implementation currently adjudicates these rows. See [spec compliance](docs/project/spec-compliance.md). -**Reference differential:** 379 files compared diagnostic-by-diagnostic against the pinned OMG pilot implementation (`2026-08`), 347 in full agreement; every divergence is enumerated and adjudicated in [the differential](docs/project/pilot-differential.md), reproducible with `go run -C tools ./cmd/pilot-diff`. +**Reference differential:** 380 files compared diagnostic-by-diagnostic against the pinned OMG pilot implementation (`2026-08`), 347 in full agreement; every divergence is enumerated and adjudicated in [the differential](docs/project/pilot-differential.md), reproducible with `go run -C tools ./cmd/pilot-diff`. **Rejection oracle:** the reverse direction — do we reject what the reference rejects? 306 hand-written invalid models validated by both implementations, 297 rejected by both, 0 the pinned pilot rejects and we accept; the remainder only we reject — the control-node succession rules the pinned pilot leaves unimplemented and a non-Boolean succession guard it accepts once the standard library types it — and every permissiveness gap is enumerated with a reproducer and likely root cause in [the rejection oracle](docs/project/pilot-rejection.md), reproducible with `go run -C tools ./cmd/pilot-reject`. We wrote every case, so the count measures our coverage of the rejection surface, not our conformance — a sample, not a proof. **Training examples:** 100/100 files clean, gated by `tests/corpus/testdata/training_examples_expected.txt`. Download with `./scripts/download-training-examples.sh` (from the [OMG training directory](https://github.com/Systems-Modeling/SysML-v2-Pilot-Implementation/tree/master/sysml/src/training)). See [training examples](docs/project/training-examples.md) for analysis. **Semantic layer:** a complete implementation of runtime operators, feature chains and validation rules. See [examples/semantic-layer/](examples/semantic-layer/) for a full demonstration. diff --git a/api/proto/sysml.pb.go b/api/proto/sysml.pb.go index 7d251ebc1..8872b6963 100644 --- a/api/proto/sysml.pb.go +++ b/api/proto/sysml.pb.go @@ -2155,8 +2155,11 @@ type EngineInfo struct { // of the manifest entry that registered it: "tool", "engine", "policy" or // "sampler". The fields below are empty for a built-in engine. Kind string `protobuf:"bytes,9,opt,name=kind,proto3" json:"kind,omitempty"` - // How the engine is spoken to: "-" for one built in, "object" for a tool's - // one JSON object each way, "/" for an engine entry. + // How the engine is spoken to: "-" for one built in; for a tool, "object" + // for the one-JSON-object exchange or "argv+" when the entry has an + // invocation block, with a "/" suffix such as "argv+none/csv" + // when the reply block reads another format (the format alone when the entry + // has no invocation); "/" for an engine entry. Protocol string `protobuf:"bytes,10,opt,name=protocol,proto3" json:"protocol,omitempty"` // The manifest entry the engine was registered from, the command it resolved // to and the version the entry declares. diff --git a/api/proto/sysml.proto b/api/proto/sysml.proto index fec77ddee..e7e995b4e 100644 --- a/api/proto/sysml.proto +++ b/api/proto/sysml.proto @@ -523,8 +523,11 @@ message EngineInfo { // of the manifest entry that registered it: "tool", "engine", "policy" or // "sampler". The fields below are empty for a built-in engine. string kind = 9; - // How the engine is spoken to: "-" for one built in, "object" for a tool's - // one JSON object each way, "/" for an engine entry. + // How the engine is spoken to: "-" for one built in; for a tool, "object" + // for the one-JSON-object exchange or "argv+" when the entry has an + // invocation block, with a "/" suffix such as "argv+none/csv" + // when the reply block reads another format (the format alone when the entry + // has no invocation); "/" for an engine entry. string protocol = 10; // The manifest entry the engine was registered from, the command it resolved // to and the version the entry declares. diff --git a/changes/unreleased/tool-dry-run.added.md b/changes/unreleased/tool-dry-run.added.md new file mode 100644 index 000000000..b423aa75e --- /dev/null +++ b/changes/unreleased/tool-dry-run.added.md @@ -0,0 +1,3 @@ +- **Tools get a dry run.** `%tool [()] []` at the prompt and `-tool-dry-run ` at the CLI show what the external tool a run's first `ToolExecution` names would be given — manifest, executable, argv, environment, working directory, standard input, input file and reply mapping — with the model's current values, without starting the process, then discards everything the preview performed; a manifest fault, an unregistered tool or an input the call does not send reports the same typed error the real run would fail with. +- **Runs record the tools they reached, and `-engines` spells the protocol.** `-record-run`/`%record` write every external tool call a run made into its `AnalysisRecords::RecordedRun` provenance as `tools`, one element per call in call order (`tool version from manifest: executable argv`); the `protocol` column a `-engines`/`%engines`/`ListEngines` listing shows for a tool spells how its manifest entry composes the process — `object` for the one-JSON-object exchange, or `argv+` with a `/` suffix such as `argv+none/csv` for an `invocation`/`reply` block. +- **A worked example walks the whole loop.** `examples/external-tool-demo/` registers a small Python solver as a tool, previews its invocation with `-tool-dry-run`, records a run — tools included — and renders the recorded run in a document; `docs/manual/running-external-programs.md` tells the same story end to end. diff --git a/client/java/opensysml-client/src/main/java/org/openmbee/opensysml/proto/EngineInfo.java b/client/java/opensysml-client/src/main/java/org/openmbee/opensysml/proto/EngineInfo.java index 2aea27338..d53b017b0 100644 --- a/client/java/opensysml-client/src/main/java/org/openmbee/opensysml/proto/EngineInfo.java +++ b/client/java/opensysml-client/src/main/java/org/openmbee/opensysml/proto/EngineInfo.java @@ -475,8 +475,11 @@ public java.lang.String getKind() { private volatile java.lang.Object protocol_ = ""; /** * 
-   * How the engine is spoken to: "-" for one built in, "object" for a tool's
-   * one JSON object each way, "<transport>/<protocol>" for an engine entry.
+   * How the engine is spoken to: "-" for one built in; for a tool, "object"
+   * for the one-JSON-object exchange or "argv+<stdin>" when the entry has an
+   * invocation block, with a "/<reply format>" suffix such as "argv+none/csv"
+   * when the reply block reads another format (the format alone when the entry
+   * has no invocation); "<transport>/<protocol>" for an engine entry.
    *
* * string protocol = 10 [json_name = "protocol"]; @@ -497,8 +500,11 @@ public java.lang.String getProtocol() { } /** * 
-   * How the engine is spoken to: "-" for one built in, "object" for a tool's
-   * one JSON object each way, "<transport>/<protocol>" for an engine entry.
+   * How the engine is spoken to: "-" for one built in; for a tool, "object"
+   * for the one-JSON-object exchange or "argv+<stdin>" when the entry has an
+   * invocation block, with a "/<reply format>" suffix such as "argv+none/csv"
+   * when the reply block reads another format (the format alone when the entry
+   * has no invocation); "<transport>/<protocol>" for an engine entry.
    *
* * string protocol = 10 [json_name = "protocol"]; @@ -2210,8 +2216,11 @@ public Builder setKindBytes( private java.lang.Object protocol_ = ""; /** * 
-     * How the engine is spoken to: "-" for one built in, "object" for a tool's
-     * one JSON object each way, "<transport>/<protocol>" for an engine entry.
+     * How the engine is spoken to: "-" for one built in; for a tool, "object"
+     * for the one-JSON-object exchange or "argv+<stdin>" when the entry has an
+     * invocation block, with a "/<reply format>" suffix such as "argv+none/csv"
+     * when the reply block reads another format (the format alone when the entry
+     * has no invocation); "<transport>/<protocol>" for an engine entry.
      *
* * string protocol = 10 [json_name = "protocol"]; @@ -2231,8 +2240,11 @@ public java.lang.String getProtocol() { } /** * 
-     * How the engine is spoken to: "-" for one built in, "object" for a tool's
-     * one JSON object each way, "<transport>/<protocol>" for an engine entry.
+     * How the engine is spoken to: "-" for one built in; for a tool, "object"
+     * for the one-JSON-object exchange or "argv+<stdin>" when the entry has an
+     * invocation block, with a "/<reply format>" suffix such as "argv+none/csv"
+     * when the reply block reads another format (the format alone when the entry
+     * has no invocation); "<transport>/<protocol>" for an engine entry.
      *
* * string protocol = 10 [json_name = "protocol"]; @@ -2253,8 +2265,11 @@ public java.lang.String getProtocol() { } /** * 
-     * How the engine is spoken to: "-" for one built in, "object" for a tool's
-     * one JSON object each way, "<transport>/<protocol>" for an engine entry.
+     * How the engine is spoken to: "-" for one built in; for a tool, "object"
+     * for the one-JSON-object exchange or "argv+<stdin>" when the entry has an
+     * invocation block, with a "/<reply format>" suffix such as "argv+none/csv"
+     * when the reply block reads another format (the format alone when the entry
+     * has no invocation); "<transport>/<protocol>" for an engine entry.
      *
* * string protocol = 10 [json_name = "protocol"]; @@ -2271,8 +2286,11 @@ public Builder setProtocol( } /** * 
-     * How the engine is spoken to: "-" for one built in, "object" for a tool's
-     * one JSON object each way, "<transport>/<protocol>" for an engine entry.
+     * How the engine is spoken to: "-" for one built in; for a tool, "object"
+     * for the one-JSON-object exchange or "argv+<stdin>" when the entry has an
+     * invocation block, with a "/<reply format>" suffix such as "argv+none/csv"
+     * when the reply block reads another format (the format alone when the entry
+     * has no invocation); "<transport>/<protocol>" for an engine entry.
      *
* * string protocol = 10 [json_name = "protocol"]; @@ -2286,8 +2304,11 @@ public Builder clearProtocol() { } /** * 
-     * How the engine is spoken to: "-" for one built in, "object" for a tool's
-     * one JSON object each way, "<transport>/<protocol>" for an engine entry.
+     * How the engine is spoken to: "-" for one built in; for a tool, "object"
+     * for the one-JSON-object exchange or "argv+<stdin>" when the entry has an
+     * invocation block, with a "/<reply format>" suffix such as "argv+none/csv"
+     * when the reply block reads another format (the format alone when the entry
+     * has no invocation); "<transport>/<protocol>" for an engine entry.
      *
* * string protocol = 10 [json_name = "protocol"]; diff --git a/client/java/opensysml-client/src/main/java/org/openmbee/opensysml/proto/EngineInfoOrBuilder.java b/client/java/opensysml-client/src/main/java/org/openmbee/opensysml/proto/EngineInfoOrBuilder.java index 3c25dceda..ec64403df 100644 --- a/client/java/opensysml-client/src/main/java/org/openmbee/opensysml/proto/EngineInfoOrBuilder.java +++ b/client/java/opensysml-client/src/main/java/org/openmbee/opensysml/proto/EngineInfoOrBuilder.java @@ -229,8 +229,11 @@ public interface EngineInfoOrBuilder extends /** * 
-   * How the engine is spoken to: "-" for one built in, "object" for a tool's
-   * one JSON object each way, "<transport>/<protocol>" for an engine entry.
+   * How the engine is spoken to: "-" for one built in; for a tool, "object"
+   * for the one-JSON-object exchange or "argv+<stdin>" when the entry has an
+   * invocation block, with a "/<reply format>" suffix such as "argv+none/csv"
+   * when the reply block reads another format (the format alone when the entry
+   * has no invocation); "<transport>/<protocol>" for an engine entry.
    *
* * string protocol = 10 [json_name = "protocol"]; @@ -239,8 +242,11 @@ public interface EngineInfoOrBuilder extends java.lang.String getProtocol(); /** * 
-   * How the engine is spoken to: "-" for one built in, "object" for a tool's
-   * one JSON object each way, "<transport>/<protocol>" for an engine entry.
+   * How the engine is spoken to: "-" for one built in; for a tool, "object"
+   * for the one-JSON-object exchange or "argv+<stdin>" when the entry has an
+   * invocation block, with a "/<reply format>" suffix such as "argv+none/csv"
+   * when the reply block reads another format (the format alone when the entry
+   * has no invocation); "<transport>/<protocol>" for an engine entry.
    *
* * string protocol = 10 [json_name = "protocol"]; diff --git a/client/java/opensysml-client/src/main/java/org/openmbee/opensysml/proto/ParseFileRequest.java b/client/java/opensysml-client/src/main/java/org/openmbee/opensysml/proto/ParseFileRequest.java index d2a1b0d87..ec7067631 100644 --- a/client/java/opensysml-client/src/main/java/org/openmbee/opensysml/proto/ParseFileRequest.java +++ b/client/java/opensysml-client/src/main/java/org/openmbee/opensysml/proto/ParseFileRequest.java @@ -206,7 +206,7 @@ public java.lang.String getContent() { * * string content_hash = 3 [json_name = "contentHash", deprecated = true]; * @deprecated sysml.ParseFileRequest.content_hash is deprecated. - * See sysml.proto;l=553 + * See sysml.proto;l=556 * @return The contentHash. */ @java.lang.Override @@ -230,7 +230,7 @@ public java.lang.String getContent() { * * string content_hash = 3 [json_name = "contentHash", deprecated = true]; * @deprecated sysml.ParseFileRequest.content_hash is deprecated. - * See sysml.proto;l=553 + * See sysml.proto;l=556 * @return The bytes for contentHash. */ @java.lang.Override @@ -942,7 +942,7 @@ public Builder setContentBytes( * * string content_hash = 3 [json_name = "contentHash", deprecated = true]; * @deprecated sysml.ParseFileRequest.content_hash is deprecated. - * See sysml.proto;l=553 + * See sysml.proto;l=556 * @return The contentHash. */ @java.lang.Deprecated public java.lang.String getContentHash() { @@ -965,7 +965,7 @@ public Builder setContentBytes( * * string content_hash = 3 [json_name = "contentHash", deprecated = true]; * @deprecated sysml.ParseFileRequest.content_hash is deprecated. - * See sysml.proto;l=553 + * See sysml.proto;l=556 * @return The bytes for contentHash. */ @java.lang.Deprecated public com.google.protobuf.ByteString @@ -989,7 +989,7 @@ public Builder setContentBytes( * * string content_hash = 3 [json_name = "contentHash", deprecated = true]; * @deprecated sysml.ParseFileRequest.content_hash is deprecated. - * See sysml.proto;l=553 + * See sysml.proto;l=556 * @param value The contentHash to set. * @return This builder for chaining. */ @@ -1009,7 +1009,7 @@ public Builder setContentBytes( * * string content_hash = 3 [json_name = "contentHash", deprecated = true]; * @deprecated sysml.ParseFileRequest.content_hash is deprecated. - * See sysml.proto;l=553 + * See sysml.proto;l=556 * @return This builder for chaining. */ @java.lang.Deprecated public Builder clearContentHash() { @@ -1026,7 +1026,7 @@ public Builder setContentBytes( * * string content_hash = 3 [json_name = "contentHash", deprecated = true]; * @deprecated sysml.ParseFileRequest.content_hash is deprecated. - * See sysml.proto;l=553 + * See sysml.proto;l=556 * @param value The bytes for contentHash to set. * @return This builder for chaining. */ diff --git a/client/java/opensysml-client/src/main/java/org/openmbee/opensysml/proto/ParseFileRequestOrBuilder.java b/client/java/opensysml-client/src/main/java/org/openmbee/opensysml/proto/ParseFileRequestOrBuilder.java index e624aae91..d6b25a918 100644 --- a/client/java/opensysml-client/src/main/java/org/openmbee/opensysml/proto/ParseFileRequestOrBuilder.java +++ b/client/java/opensysml-client/src/main/java/org/openmbee/opensysml/proto/ParseFileRequestOrBuilder.java @@ -52,7 +52,7 @@ public interface ParseFileRequestOrBuilder extends * * string content_hash = 3 [json_name = "contentHash", deprecated = true]; * @deprecated sysml.ParseFileRequest.content_hash is deprecated. - * See sysml.proto;l=553 + * See sysml.proto;l=556 * @return The contentHash. */ @java.lang.Deprecated java.lang.String getContentHash(); @@ -64,7 +64,7 @@ public interface ParseFileRequestOrBuilder extends * * string content_hash = 3 [json_name = "contentHash", deprecated = true]; * @deprecated sysml.ParseFileRequest.content_hash is deprecated. - * See sysml.proto;l=553 + * See sysml.proto;l=556 * @return The bytes for contentHash. */ @java.lang.Deprecated com.google.protobuf.ByteString diff --git a/client/node/src/generated/sysml_pb.ts b/client/node/src/generated/sysml_pb.ts index c67cec69c..ba15e134c 100644 --- a/client/node/src/generated/sysml_pb.ts +++ b/client/node/src/generated/sysml_pb.ts @@ -1219,8 +1219,11 @@ export type EngineInfo = Message<"sysml.EngineInfo"> & { kind: string; /** - * How the engine is spoken to: "-" for one built in, "object" for a tool's - * one JSON object each way, "/" for an engine entry. + * How the engine is spoken to: "-" for one built in; for a tool, "object" + * for the one-JSON-object exchange or "argv+" when the entry has an + * invocation block, with a "/" suffix such as "argv+none/csv" + * when the reply block reads another format (the format alone when the entry + * has no invocation); "/" for an engine entry. * * @generated from field: string protocol = 10; */ diff --git a/client/python/opensysml/engines.py b/client/python/opensysml/engines.py index 1bed01499..c6c7c30ad 100644 --- a/client/python/opensysml/engines.py +++ b/client/python/opensysml/engines.py @@ -107,7 +107,11 @@ class EngineInfo: ready: Whether it can run here unavailable: Why it cannot, when it cannot kind: ``built-in``, ``tool`` or ``engine`` (one registered from a manifest) - protocol: How it is spoken to: ``-`` for a built-in engine, ``object`` for a tool, + protocol: How it is spoken to: ``-`` for a built-in engine; for a tool, + ``object`` for the one-JSON-object exchange or ``argv+`` when the + entry has an ``invocation`` block, with a ``/`` suffix + such as ``argv+none/csv`` when the ``reply`` block reads another format + (the format alone when the entry has no ``invocation``); ``/`` such as ``stdio/1`` for an engine entry source: The manifest file an external engine was read from, empty for a built-in one command: The resolved command of an external engine, empty for a built-in one @@ -156,7 +160,7 @@ def explain(self): status = "ready" if self.ready else f"unavailable: {self.unavailable}" if self.kind == "engine" and not self.served: status += "; not served by this service" - kind = f" ({self.kind}, {self.protocol})" if self.kind == "engine" else "" + kind = f" ({self.kind}, {self.protocol})" if self.kind in ("engine", "tool") else "" return f"{self.name}{kind}: {self.authority}, answers {', '.join(self.answers)}; {status}" def __str__(self): diff --git a/client/python/tests/test_engines.py b/client/python/tests/test_engines.py index a379bcb6f..0d86f122d 100644 --- a/client/python/tests/test_engines.py +++ b/client/python/tests/test_engines.py @@ -106,6 +106,24 @@ def test_list_engines_reads_every_engine(): assert str(engines[1]) == "solve: proved, answers satisfiable; unavailable: z3 not found" +def test_list_engines_round_trips_a_tools_protocol(): + """A manifest tool's protocol spells its invocation and reply composition.""" + stub = Mock() + stub.ListEngines.return_value = sysml_pb2.ListEnginesResponse(engines=[ + sysml_pb2.EngineInfo( + name="tool:ThermalSolver", authority="observed", answers=["compute"], + kind="tool", protocol="argv+none/csv", ready=True, + source="/tools/thermal.json", command="/usr/bin/python3", version="1.0", + served=True, + ), + ]) + conn = make_connection(stub, CURRENT) + + engines = conn.list_engines() + assert engines[0].protocol == "argv+none/csv" + assert str(engines[0]).startswith("tool:ThermalSolver (tool, argv+none/csv): ") + + def test_list_engines_needs_the_capability(): stub = Mock() conn = make_connection(stub, OLD) diff --git a/client/rust/conformance/sysml.descriptor.binpb b/client/rust/conformance/sysml.descriptor.binpb index cdbc5bb2a..7e7a011ef 100644 Binary files a/client/rust/conformance/sysml.descriptor.binpb and b/client/rust/conformance/sysml.descriptor.binpb differ diff --git a/client/rust/opensysml/src/proto/sysml/sysml.rs b/client/rust/opensysml/src/proto/sysml/sysml.rs index 0ce9410fb..09c875281 100644 --- a/client/rust/opensysml/src/proto/sysml/sysml.rs +++ b/client/rust/opensysml/src/proto/sysml/sysml.rs @@ -546,8 +546,11 @@ pub struct EngineInfo { /// "sampler". The fields below are empty for a built-in engine. #[prost(string, tag="9")] pub kind: ::prost::alloc::string::String, - /// How the engine is spoken to: "-" for one built in, "object" for a tool's - /// one JSON object each way, "/" for an engine entry. + /// How the engine is spoken to: "-" for one built in; for a tool, "object" + /// for the one-JSON-object exchange or "argv+" when the entry has an + /// invocation block, with a "/" suffix such as "argv+none/csv" + /// when the reply block reads another format (the format alone when the entry + /// has no invocation); "/" for an engine entry. #[prost(string, tag="10")] pub protocol: ::prost::alloc::string::String, /// The manifest entry the engine was registered from, the command it resolved diff --git a/cmd/sysml/check.go b/cmd/sysml/check.go index ac2465e0f..5727f950c 100644 --- a/cmd/sysml/check.go +++ b/cmd/sysml/check.go @@ -27,6 +27,7 @@ type checks struct { satisfy optionalNames calcs stringSlice analyses stringSlice + toolDryRuns stringSlice records stringSlice recordInto string sweeps stringSlice @@ -233,7 +234,7 @@ func (a *advanceTime) Set(value string) error { func (c *checks) requested() bool { return c.validate.given || c.jsonOut || c.advance.given || c.satisfy.given || len(c.instantiate) > 0 || len(c.constraints) > 0 || len(c.requirements) > 0 || len(c.calcs) > 0 || len(c.analyses) > 0 || - len(c.records) > 0 || + len(c.toolDryRuns) > 0 || len(c.records) > 0 || len(c.queries) > 0 || len(c.actions) > 0 || len(c.states) > 0 || c.sweeping() || c.running() || c.compare != "" || c.checker.given() } @@ -358,7 +359,7 @@ func (c *checks) compareMisuse() string { switch { case len(c.states) > 0 || c.sweeping() || c.advance.given || c.checker.given() || c.validate.given || c.satisfy.given || len(c.instantiate) > 0 || len(c.constraints) > 0 || - len(c.requirements) > 0 || len(c.calcs) > 0 || len(c.analyses) > 0 || len(c.records) > 0 || len(c.queries) > 0: + len(c.requirements) > 0 || len(c.calcs) > 0 || len(c.analyses) > 0 || len(c.toolDryRuns) > 0 || len(c.records) > 0 || len(c.queries) > 0: return "-compare-results runs the migrated configurations the results index and compares the runs with the tool's; the other checks are made in a run of their own" } for _, pair := range c.observe { @@ -412,6 +413,8 @@ func (c *checks) sweepMisuse() string { } targets := len(c.calcs) + len(c.analyses) + len(c.records) switch { + case len(c.toolDryRuns) > 0: + return "-sweep runs an analysis case or a calc once per value; -tool-dry-run previews a tool call instead" case targets == 0: return "-sweep runs an analysis case or a calc; name one, as -analysis or -calc " case targets > 1: @@ -429,7 +432,7 @@ func (c *checks) sweepMisuse() string { func (c *checks) instantiatesOnly() bool { return len(c.instantiate) > 0 && !c.validate.given && !c.jsonOut && !c.advance.given && !c.satisfy.given && len(c.constraints) == 0 && len(c.requirements) == 0 && len(c.calcs) == 0 && len(c.analyses) == 0 && - len(c.records) == 0 && + len(c.toolDryRuns) == 0 && len(c.records) == 0 && len(c.queries) == 0 && len(c.actions) == 0 && len(c.states) == 0 && !c.sweeping() && !c.running() && c.compare == "" && !c.checker.given() } @@ -440,7 +443,7 @@ func (c *checks) instantiatesOnly() bool { func (c *checks) recordsOnly() bool { return len(c.records) > 0 && !c.validate.given && !c.jsonOut && !c.advance.given && !c.satisfy.given && len(c.constraints) == 0 && len(c.requirements) == 0 && len(c.calcs) == 0 && - len(c.analyses) == 0 && len(c.observe) == 0 && + len(c.analyses) == 0 && len(c.toolDryRuns) == 0 && len(c.observe) == 0 && len(c.queries) == 0 && len(c.actions) == 0 && len(c.states) == 0 && c.compare == "" && !c.checker.given() } @@ -475,7 +478,7 @@ func (c *checks) boundsMisuse() string { func (c *checks) checksOnly() bool { return len(c.validate.targets) > 0 || len(c.instantiate) > 0 || len(c.constraints) > 0 || len(c.requirements) > 0 || len(c.satisfy.targets) > 0 || len(c.calcs) > 0 || len(c.analyses) > 0 || - len(c.records) > 0 || + len(c.toolDryRuns) > 0 || len(c.records) > 0 || len(c.queries) > 0 || len(c.actions) > 0 || len(c.states) > 0 || c.compare != "" } @@ -752,6 +755,9 @@ func runChecks(files []string, exprs []string, c checks) int { rep.verdict(sess.RunAnalysis(invocation)) } } + for _, target := range c.toolDryRuns { + rep.verdict(sess.ToolDryRun(target)) + } for _, invocation := range c.records { rep.verdict(c.record(sess, invocation)) } diff --git a/cmd/sysml/external_tool_demo_test.go b/cmd/sysml/external_tool_demo_test.go new file mode 100644 index 000000000..c0e6186c2 --- /dev/null +++ b/cmd/sysml/external_tool_demo_test.go @@ -0,0 +1,66 @@ +package main + +import ( + "os" + "os/exec" + "path/filepath" + "strings" + "testing" + + "github.com/Open-MBEE/OpenSysML/internal/exec/analysis" +) + +// The external-tool demo registers a Python script as a tool, previews the +// call, records a run — tools included — and renders the record in a document. +func TestExternalToolDemo(t *testing.T) { + if _, err := exec.LookPath("python3"); err != nil { + t.Skip("the demo's tool is a Python 3 script") + } + binary := buildCLI(t) + + dir := t.TempDir() + for _, name := range []string{"thermal.sysml", filepath.Join("tools", "thermal.py"), filepath.Join("tools", "thermal.json")} { + data, err := os.ReadFile(filepath.Join("..", "..", "examples", "external-tool-demo", name)) + if err != nil { + t.Fatal(err) + } + dst := filepath.Join(dir, name) + if err := os.MkdirAll(filepath.Dir(dst), 0o700); err != nil { + t.Fatal(err) + } + if err := os.WriteFile(dst, data, 0o600); err != nil { + t.Fatal(err) + } + } + t.Setenv(analysis.ToolsEnv, filepath.Join(dir, "tools")) + model := filepath.Join(dir, "thermal.sysml") + + dry := checkPaths(t, binary, "-tool-dry-run", "ThermalDemo::heating", model) + wantReport(t, dry, 0, "dry run of tool 'ThermalSolver'", "protocol: argv+none/csv", + `"thermal.py"`, `"--mass"`, `"12.5"`, "the process was not started") + + saved := filepath.Join(dir, "recorded.sysml") + got := checkPaths(t, binary, "-record-run", "ThermalDemo::heating", "-convert", "sysml", "-o", saved, model) + wantReport(t, got, 0, "tMax = 306.0", "recorded Records::heating_run1") + data, err := os.ReadFile(saved) + if err != nil { + t.Fatal(err) + } + if !strings.Contains(string(data), `tools = ("ThermalSolver 1.0 from `) { + t.Errorf("the record names no tool:\n%s", data) + } + + report := filepath.Join(dir, "report.md") + got = checkPaths(t, binary, "-record-run", "ThermalDemo::heating", + "-render-document", "Reporting::HeatingReport", "-o", report, model) + wantReport(t, got, 0, "recorded Records::heating_run1") + data, err = os.ReadFile(report) + if err != nil { + t.Fatal(err) + } + for _, want := range []string{"heating\\_run1", "| 306 |", "satisfied"} { + if !strings.Contains(string(data), want) { + t.Errorf("report is missing %q:\n%s", want, data) + } + } +} diff --git a/cmd/sysml/tool_dry_run_test.go b/cmd/sysml/tool_dry_run_test.go new file mode 100644 index 000000000..805781c90 --- /dev/null +++ b/cmd/sysml/tool_dry_run_test.go @@ -0,0 +1,138 @@ +package main + +import ( + "fmt" + "os" + "os/exec" + "path/filepath" + "strings" + "sync" + "testing" + + "github.com/Open-MBEE/OpenSysML/internal/exec/analysis" + "github.com/Open-MBEE/OpenSysML/tests/testutil/gobuild" +) + +// toolCaseModel is a tool-computed action and the case performing it. +const toolCaseModel = `package Tools { + private import ScalarValues::Real; + private import AnalysisTooling::*; + action def Heating { + metadata ToolExecution { toolName = "Solver"; uri = "solver://eq"; } + in mass : Real = 12.5 { @ToolVariable { name = "mass"; } } + out tMax : Real { @ToolVariable { name = "tMax"; } } + } + analysis def CheckHeating { + action h : Heating; + out result : Real = h.tMax; + } +} +` + +var ( + toolOnce sync.Once + toolPath string + toolErr error +) + +// dryRunStandin builds the analysis package's tool stand-in once per test binary. +func dryRunStandin(t *testing.T) string { + t.Helper() + toolOnce.Do(func() { + dir, err := os.MkdirTemp("", "toolstandin") + if err != nil { + toolErr = err + return + } + toolPath = filepath.Join(dir, "toolstandin") + build := exec.Command("go", gobuild.Args(toolPath)...) + build.Dir = filepath.Join("..", "..", "internal", "exec", "analysis", "testdata", "toolstandin") + if out, err := build.CombinedOutput(); err != nil { + toolErr = fmt.Errorf("go build: %v\n%s", err, out) + } + }) + if toolErr != nil { + t.Fatalf("building the stand-in tool: %v", toolErr) + } + return toolPath +} + +// dryRunManifest writes the tool entries into a manifest directory of the test's +// own and points OPENSYSML_TOOLS at it. +func dryRunManifest(t *testing.T, entries ...string) { + t.Helper() + dir := t.TempDir() + for i, entry := range entries { + name := fmt.Sprintf("%02d%s", i, analysis.ManifestExt) + if err := os.WriteFile(filepath.Join(dir, name), []byte(entry), 0o600); err != nil { + t.Fatal(err) + } + } + t.Setenv(analysis.ToolsEnv, dir) +} + +// -tool-dry-run previews the process a case's tool call would start — argv, env, +// stdin and reply — and starts nothing. +func TestToolDryRunPreviewsTheComposedInvocation(t *testing.T) { + binary := buildCLI(t) + record := filepath.Join(t.TempDir(), "requests.jsonl") + t.Setenv("TOOL_STANDIN_RECORD", record) + entry := `{"kind":"tool","toolName":"Solver","version":"2.3",` + + `"executable":"` + dryRunStandin(t) + `","variables":["mass","tMax"],` + + `"invocation":{"args":["solve.py","--mass","{mass}"],"env":{"SOLVER_HOME":"/opt/solver"},"stdin":"csv"},` + + `"reply":{"format":"csv","source":"stdout","outputs":{"tMax":{"column":"tmax","type":"number","unit":"K"}}}}` + dryRunManifest(t, entry) + + got := check(t, binary, toolCaseModel, "-tool-dry-run", "Tools::CheckHeating") + wantReport(t, got, 0, + "✓ Tools::CheckHeating: dry run of tool 'Solver' for Tools::Heating", + "tool: Solver 2.3", "protocol: argv+csv/csv", "executable: "+dryRunStandin(t), + `"solve.py"`, `"--mass"`, `"12.5"`, "SOLVER_HOME=/opt/solver", "stdin: csv", + " mass", " 12.5", `tMax: column "tmax", row last, type number, unit K`, + "the process was not started") + if _, err := os.Stat(record); !os.IsNotExist(err) { + t.Fatalf("the tool recorded a request; a dry run must not start it") + } +} + +// Without the manifest the dry run is an unresolved verdict naming the tool and +// the variable registering it. +func TestToolDryRunWithoutAManifest(t *testing.T) { + binary := buildCLI(t) + t.Setenv(analysis.ToolsEnv, "") + got := check(t, binary, toolCaseModel, "-tool-dry-run", "Tools::CheckHeating") + wantReport(t, got, 2, "tool 'Solver' is not registered; set OPENSYSML_TOOLS") + rejectReport(t, got, "the process was not started") +} + +// An invocation naming an input the call does not send reports the same refusal +// the real run would. +func TestToolDryRunRefusesAnUnsentInput(t *testing.T) { + binary := buildCLI(t) + entry := `{"kind":"tool","toolName":"Solver","executable":"` + dryRunStandin(t) + `","variables":["mass","power","tMax"],` + + `"invocation":{"args":["{power}"]}}` + dryRunManifest(t, entry) + got := check(t, binary, toolCaseModel, "-tool-dry-run", "Tools::CheckHeating") + wantReport(t, got, 2, "the invocation names {power} but the call sent no value for power") + rejectReport(t, got, "the process was not started") +} + +// -help documents -tool-dry-run beside -analysis. +func TestToolDryRunIsInTheHelp(t *testing.T) { + binary := buildCLI(t) + out := run(t, binary, "-h") + if !strings.Contains(out, "-tool-dry-run") { + t.Fatalf("-help does not name -tool-dry-run:\n%s", out) + } +} + +// -tool-dry-run on an action binds the invocation's argument into the preview. +func TestToolDryRunBindsAnActionsArguments(t *testing.T) { + binary := buildCLI(t) + entry := `{"kind":"tool","toolName":"Solver","executable":"` + dryRunStandin(t) + `","variables":["mass","tMax"],` + + `"invocation":{"args":["--mass","{mass}"],"stdin":"none"}}` + dryRunManifest(t, entry) + got := check(t, binary, toolCaseModel, "-tool-dry-run", "Tools::Heating(30)") + wantReport(t, got, 0, "dry run of tool 'Solver' for Tools::Heating", `"--mass"`, `"30"`, + "inputs:", " mass = 30", "the process was not started") +} diff --git a/cmd/sysml/usage.go b/cmd/sysml/usage.go index 820a28b25..892da1470 100644 --- a/cmd/sysml/usage.go +++ b/cmd/sysml/usage.go @@ -562,6 +562,7 @@ func registerFlags(fs *flag.FlagSet) { fs.Var(&modelChecks.satisfy, "satisfy", "Evaluate every satisfaction assertion, or with -satisfy= those the named element states, and exit (repeatable)") fs.Var(&modelChecks.calcs, "calc", "Invoke this calculation and report its result, as -calc \"Fall(3, 4)\" (repeatable)") fs.Var(&modelChecks.analyses, "analysis", "Run this analysis or verification case and report its outputs and verdict, as -analysis \"Pkg::Case(3.0) Pkg::part\" (repeatable)") + fs.Var(&modelChecks.toolDryRuns, "tool-dry-run", "Show what the external tool the case's or action's ToolExecution names would be given — manifest, executable, argv, environment, standard input and reply mapping — without starting it, and discards what the run did; repeatable") fs.Var(&modelChecks.records, "record-run", "Run this analysis case as -analysis does and record the run into the model as AnalysisRecords elements, one record per -sweep value or -runs run (repeatable)") fs.StringVar(&modelChecks.recordInto, "record-into", "", "Record -record-run runs into this package instead of a Records package beside the case's") fs.Var(&modelChecks.queries, "run-query", "Execute this document query and report its rows, as -run-query \"Heavy root=telescope\" (repeatable)") @@ -683,6 +684,7 @@ func optionGroups() []usage.OptionGroup { usage.Opt("satisfy", "[=]"), usage.Opt("calc", callArg), usage.Opt("analysis", callArg), + usage.Opt("tool-dry-run", callArg), usage.Opt("record-run", callArg), usage.Opt("record-into", ""), usage.Opt("run-query", ""), diff --git a/docs/internals/architecture.md b/docs/internals/architecture.md index 8006044f8..f03ba22c1 100644 --- a/docs/internals/architecture.md +++ b/docs/internals/architecture.md @@ -840,11 +840,11 @@ Every behavioral feature must have: **Measured against the pinned reference** (`PILOT_TAG=2026-08`, artifact `0.62.0`). Every number below is generated by `make docs-counts` from the committed baselines and gated; none of them is typed in by hand. -- **Corpus agreement:** 347 of 379 files agree diagnostic-by-diagnostic; 38 diagnostics are ours alone and 1582 the reference's alone, and the first number must be read by root: our diagnostics against the reference's own corpora fell while our non-standard-notation warnings on our own example models rose ([differential](../project/pilot-differential.md), `go run -C tools ./cmd/pilot-diff`). +- **Corpus agreement:** 347 of 380 files agree diagnostic-by-diagnostic; 38 diagnostics are ours alone and 1614 the reference's alone, and the first number must be read by root: our diagnostics against the reference's own corpora fell while our non-standard-notation warnings on our own example models rose ([differential](../project/pilot-differential.md), `go run -C tools ./cmd/pilot-diff`). - **Declared-diagnostic silence:** of the 512 declared `errors` rows in the reference's own Xpect suites, we report nothing for 0. 245 we report word-for-word; 248 wording-only and 7 location-only differences are agreement in substance and are not counted as gaps; 0 more we report as a warning and 2 elsewhere in the file ([Xpect oracle](../project/pilot-xpect.md), `go run -C tools ./cmd/pilot-xpect`). - **Scope agreement:** 230 of 230 declared scope assertions match exactly (same source). - **Permissiveness gaps:** of 306 invalid models we wrote ourselves, the reference rejects 4 that we accept by default, and 293 both reject; 4 further cases agree only when we are asked strictly. We authored every one of these cases ourselves, so the denominator measures the reach of our own corpus and not our conformance; agreement reached only under an opt-in strict mode is weaker evidence than agreement by default ([rejection oracle](../project/pilot-rejection.md), `go run -C tools ./cmd/pilot-reject`). -- **Declared errata:** the registry declares 12 defect(s) in the published reference material — 4 with a specification-derived correction, 8 documented without one, since no intended reading can be inferred ([OMG issues](../project/omg-issues.md), `tools/oracle/errata`). Every figure above is as published and stays the conformance statement; running the same oracles over the corrected text instead reports 348 of 379 files agreeing, 37 diagnostics ours alone and 1582 the reference's alone, 0 declared rows we are silent on, and 0 of 306 authored cases the reference alone rejects. The corrected figures are diagnostic only: an erratum never reclassifies a divergence category, and the published corpus is never edited. +- **Declared errata:** the registry declares 12 defect(s) in the published reference material — 4 with a specification-derived correction, 8 documented without one, since no intended reading can be inferred ([OMG issues](../project/omg-issues.md), `tools/oracle/errata`). Every figure above is as published and stays the conformance statement; running the same oracles over the corrected text instead reports 348 of 380 files agreeing, 37 diagnostics ours alone and 1614 the reference's alone, 0 declared rows we are silent on, and 0 of 306 authored cases the reference alone rejects. The corrected figures are diagnostic only: an erratum never reclassifies a divergence category, and the published corpus is never edited. - **Self-assessed surface:** the action, state-machine and classifier-behavior rows have no external referee at all — the four refereed figures above cannot see them, because the pinned artifact evaluates expressions but executes neither actions nor state machines. [Spec compliance](../project/spec-compliance.md) counts them. What these numbers cannot show: the OMG corpora are demonstrations rather than an official conformance suite; the differential is one-directional, comparing the diagnostics the two implementations report on the same files; the Xpect suites are the pilot authors' test intent rather than a certification oracle; and none of these is a percentage of the specification — no global compliance figure is claimed anywhere. diff --git a/docs/manual/recording-analysis-runs.md b/docs/manual/recording-analysis-runs.md index 91a1ab3fa..e4b80a28b 100644 --- a/docs/manual/recording-analysis-runs.md +++ b/docs/manual/recording-analysis-runs.md @@ -174,7 +174,9 @@ vocabulary the records are written in: - `RecordedRun` — the metadata annotation a record carries: `runAt` (the UTC timestamp), `tool`, `command`, `kind` (`"run"`, `"trade"`, `"sweep"`, - `"runs"` or `"sample"`). + `"runs"` or `"sample"`), and `tools` (each external tool call the run made, + in call order, as `tool version from manifest: executable argv` — see + [Running external programs from a model](running-external-programs.md)). - `AnalysisRun` — the record definition's supertype: `caseName`, `kind`, `'objective'` (the run's objective verdict, `"undecided"` when the case declares none), `iteration` (its position in a sweep or sample), `'subject'` diff --git a/docs/manual/running-external-programs.md b/docs/manual/running-external-programs.md new file mode 100644 index 000000000..da9c22c68 --- /dev/null +++ b/docs/manual/running-external-programs.md @@ -0,0 +1,170 @@ +# Running external programs from a model + +An analysis case's action can be *performed by a program outside OpenSysML* — a +solver, a simulator wrapper, a script — rather than by a body written in SysML. +This chapter walks the whole loop on the worked example in +[`examples/external-tool-demo/`](https://github.com/Open-MBEE/OpenSysML/tree/develop/examples/external-tool-demo): +registering the program in a manifest, previewing what a run would give it, +running the case, recording the run with its tool provenance, and tabulating +the record in a document. The manifest members, the protocol the exchange +speaks and every failure's shape are reference material in [External +tools](../reference/environment.md#external-tools); this is the story. + +## Declaring the tool + +An action declares it is performed by a tool with the +`AnalysisTooling::ToolExecution` metadata, its `toolName` and `uri` carried to +the tool verbatim; each parameter taking part in the exchange carries +`@AnalysisTooling::ToolVariable` naming it as the tool's manifest spells it: + +```sysml +action def Solve { + metadata ToolExecution { + toolName = "ThermalSolver"; + uri = "thermal://demo/solve"; + } + in mass : MassValue { @ToolVariable { name = "mass"; } } + in power : PowerValue { @ToolVariable { name = "power"; } } + in ambient : TemperatureValue { @ToolVariable { name = "ambient"; } } + out tMax : TemperatureValue { @ToolVariable { name = "tMax"; } } + out margin : TemperatureValue { @ToolVariable { name = "margin"; } } +} +``` + +The tools a `sysml` or `sysml-grpc` process may run are the entries of the +directory `OPENSYSML_TOOLS` names, read once at startup — one JSON file per +tool, in a directory outside every workspace, writable by its owner alone: + +```json +{ + "toolName": "ThermalSolver", + "version": "1.0", + "executable": "python3", + "variables": ["mass", "power", "ambient", "tMax", "margin"], + "invocation": { + "args": ["thermal.py", "--mass", "{mass}", "--power", "{power}", "--ambient", "{ambient}"], + "cwd": ".", + "stdin": "none" + }, + "reply": { + "format": "csv", + "source": "stdout", + "outputs": { + "tMax": {"column": "tmax", "type": "number", "unitColumn": "tmaxUnit"}, + "margin": {"column": "margin", "type": "number", "unit": "K"} + } + } +} +``` + +The `invocation` block composes the command: each `args` entry renders to +exactly one argument with `{mass}` placeholders filled from the call's values, +`cwd` confined to the manifest's directory (`.` — the script sits beside the +entry), `stdin` saying what the process reads. The `reply` block says the +answer is the CSV record on standard output and where each output lives in it. +An entry with no block at all speaks the `object` protocol — one JSON object +each way on stdin/stdout. + +`-engines` lists the tool as `tool:ThermalSolver`, its `protocol` column +spelling the composition — here `argv+none/csv`: + +```text +tool:ThermalSolver tool argv+none/csv observed compute ready (ThermalSolver 1.0 at /usr/bin/python3) +``` + +## Previewing the call + +`-tool-dry-run` at the CLI, `%tool` in the REPL, runs the case or action as +`-analysis` would until the performance first reaches a tool — then prints +what the call would have been given, without starting the process. What the +preview performed is discarded: the session is as it found it. + +```text +✓ ThermalDemo::heating: dry run of tool 'ThermalSolver' for ThermalDemo::Solve + tool: ThermalSolver 1.0 + protocol: argv+none/csv + manifest: /home/me/.opensysml-tools/thermal.json + executable: /usr/bin/python3 + argv: + "thermal.py" + "--mass" + "12.5" + "--power" + "250" + "--ambient" + "290" + env: + ... + cwd: /home/me/.opensysml-tools + stdin: none + inputs: + ambient = 290 [K] + mass = 12.5 [kg] + power = 250 [W] + outputs: + margin + tMax + reply: csv from stdout, header, delimiter "," + margin: column "margin", row last, type number, unit K + tMax: column "tmax", row last, unitColumn "tmaxUnit", type number + the process was not started +``` + +A manifest fault, an unregistered tool or an input the call does not send +reports the same typed error the real run would fail with — the preview is a +diagnostic surface, not a guess. A run that reaches no tool says so. + +## Running and recording it + +`-analysis` runs the case; the tool's process is started, the inputs bound, +the reply read: + +```text +✓ ThermalDemo::heating + tMax = 306.0 [SI::K] + margin = 94.0 [SI::K] + objective marginHeld: satisfied + standing: value (observed: 1 run under reverse) +``` + +A tool's answer stands at strength *observed*: nothing in OpenSysML knows +what the tool should have computed, so equal inputs answering differently are +reported as a divergence in the run's notes. + +`-record-run` records the run as an `AnalysisRecords` element — see +[Recording analysis runs](recording-analysis-runs.md) — and its `RecordedRun` +annotation's `tools` names every external tool call the run made, one element +per call in call order (`tool version from manifest: executable argv`): + +```sysml +part heating_run1 : HeatingRun { + @AnalysisRecords::RecordedRun { + runAt = "2026-10-15T11:24:00Z"; + tool = "sysml dev"; + command = "-record-run \"ThermalDemo::heating\""; + kind = "run"; + tools = ("ThermalSolver 1.0 from /home/me/.opensysml-tools/thermal.json: /usr/bin/python3 thermal.py --mass 12.5 --power 250 --ambient 290"); + } + ... +} +``` + +## Reporting it + +Recording composes with document rendering — `-record-run` beside +`-render-document` records first, so the document's queries see the record — +and `Project` tabulates the record's own features. An annotation's attribute +values are not projectable, so `tools` stays in the record's source rather +than a table column: + +```bash +./bin/sysml examples/external-tool-demo/thermal.sysml \ + -record-run ThermalDemo::heating \ + -render-document Reporting::HeatingReport -o report.md +``` + +```markdown +| name | caseName | kind | mass | power | ambient | tMax | margin | objective | +| --- | --- | --- | --- | --- | --- | --- | --- | --- | +| heating\_run1 | ThermalDemo::heating | run | 12.5 | 250 | 290 | 306 | 94 | satisfied | +``` diff --git a/docs/project/pilot-differential-baseline.json b/docs/project/pilot-differential-baseline.json index eabddd704..cfe3e6e0e 100644 --- a/docs/project/pilot-differential-baseline.json +++ b/docs/project/pilot-differential-baseline.json @@ -60,8 +60,8 @@ "name": "examples", "dir": "examples", "origin": "ours", - "files": 44, - "digest": "sha256:7e1371db53edfdcc895af22c619a6165ab72b9eb253219bb745097379d7648fb" + "files": 45, + "digest": "sha256:dd7ea383c6ed28e47c2252c0f9a99c84e6a287c98edcf104373855cc38fe74e1" }, { "name": "probes", @@ -74,14 +74,14 @@ "recorded": "2026-09-25" }, "totals": { - "files": 379, + "files": 380, "filesFullyAgreeing": 347, "agreement": 38, "severityMismatch": 3, "openSysMLOnly": 38, - "pilotOnly": 1582, + "pilotOnly": 1614, "openSysMLDiagnostics": 79, - "pilotDiagnostics": 1623 + "pilotDiagnostics": 1655 }, "roots": [ { @@ -746,14 +746,14 @@ "name": "examples", "dir": "examples", "totals": { - "files": 44, + "files": 45, "filesFullyAgreeing": 30, "agreement": 4, "severityMismatch": 2, "openSysMLOnly": 7, - "pilotOnly": 1562, + "pilotOnly": 1594, "openSysMLDiagnostics": 13, - "pilotDiagnostics": 1568 + "pilotDiagnostics": 1600 }, "files": [ { @@ -2583,6 +2583,158 @@ } ] }, + { + "path": "external-tool-demo/thermal.sysml", + "agreement": [], + "severityMismatch": [], + "openSysMLOnly": [], + "pilotOnly": [ + { + "line": 78, + "severity": "error", + "category": "unresolved-reference", + "count": 1 + }, + { + "line": 83, + "severity": "error", + "category": "unresolved-reference", + "count": 1 + }, + { + "line": 84, + "severity": "error", + "category": "kind-mismatch", + "count": 1 + }, + { + "line": 84, + "severity": "error", + "category": "unresolved-reference", + "count": 1 + }, + { + "line": 85, + "severity": "error", + "category": "kind-mismatch", + "count": 1 + }, + { + "line": 85, + "severity": "error", + "category": "unresolved-reference", + "count": 2 + }, + { + "line": 86, + "severity": "error", + "category": "kind-mismatch", + "count": 1 + }, + { + "line": 86, + "severity": "error", + "category": "unresolved-reference", + "count": 2 + }, + { + "line": 87, + "severity": "error", + "category": "kind-mismatch", + "count": 2 + }, + { + "line": 87, + "severity": "error", + "category": "unresolved-reference", + "count": 6 + }, + { + "line": 88, + "severity": "error", + "category": "unresolved-reference", + "count": 1 + }, + { + "line": 89, + "severity": "error", + "category": "unresolved-reference", + "count": 1 + }, + { + "line": 90, + "severity": "error", + "category": "unresolved-reference", + "count": 1 + }, + { + "line": 93, + "severity": "error", + "category": "unresolved-reference", + "count": 1 + }, + { + "line": 94, + "severity": "error", + "category": "unresolved-reference", + "count": 1 + }, + { + "line": 96, + "severity": "error", + "category": "kind-mismatch", + "count": 1 + }, + { + "line": 96, + "severity": "error", + "category": "unresolved-reference", + "count": 1 + }, + { + "line": 97, + "severity": "error", + "category": "unresolved-reference", + "count": 1 + }, + { + "line": 100, + "severity": "error", + "category": "kind-mismatch", + "count": 1 + }, + { + "line": 100, + "severity": "error", + "category": "unresolved-reference", + "count": 1 + }, + { + "line": 101, + "severity": "error", + "category": "unresolved-reference", + "count": 1 + }, + { + "line": 102, + "severity": "error", + "category": "kind-mismatch", + "count": 1 + }, + { + "line": 102, + "severity": "error", + "category": "unresolved-reference", + "count": 1 + }, + { + "line": 103, + "severity": "error", + "category": "unresolved-reference", + "count": 1 + } + ] + }, { "path": "mosa-demo/mosa-demo.sysml", "agreement": [], @@ -8183,14 +8335,14 @@ } ], "totals": { - "files": 379, + "files": 380, "filesFullyAgreeing": 348, "agreement": 38, "severityMismatch": 3, "openSysMLOnly": 37, - "pilotOnly": 1582, + "pilotOnly": 1614, "openSysMLDiagnostics": 78, - "pilotDiagnostics": 1623 + "pilotDiagnostics": 1655 }, "findings": [ { diff --git a/docs/project/pilot-differential.md b/docs/project/pilot-differential.md index a7c7d5fdc..74eb2ead6 100644 --- a/docs/project/pilot-differential.md +++ b/docs/project/pilot-differential.md @@ -209,7 +209,7 @@ nor double-counted as two independent disagreements. --- -## Results (pilot `2026-08`, 379 files) +## Results (pilot `2026-08`, 380 files) | Root | Files | Fully agreeing | Ours | Pilot | Agreed | Severity-only | Only ours | Only pilot | |---|---:|---:|---:|---:|---:|---:|---:|---:| @@ -218,9 +218,9 @@ nor double-counted as two independent disagreements. | `examples/pilot-corpora/sysml-validation` | 56 | 56 | 0 | 0 | 0 | 0 | 0 | 0 | | `examples/pilot-corpora/kerml-examples` | 58 | 55 | 10 | 0 | 0 | 0 | 10 | 0 | | `tests/testdata` | 18 | 10 | 43 | 55 | 34 | 1 | 8 | 20 | -| `examples` | 44 | 30 | 13 | 1568 | 4 | 2 | 7 | 1562 | +| `examples` | 45 | 30 | 13 | 1600 | 4 | 2 | 7 | 1594 | | `tools/referee/diff/testdata` (probes) | 4 | 1 | 6 | 0 | 0 | 0 | 6 | 0 | -| **Total** | **379** | **347** | **79** | **1623** | **38** | **3** | **38** | **1582** | +| **Total** | **380** | **347** | **79** | **1655** | **38** | **3** | **38** | **1614** | **Read the `only ours` total by root, never as one number.** Step 2 removes nine resolver false positives from the reference's **own** corpora: `pilot-examples` 16 → **7** and @@ -791,8 +791,8 @@ cascades through the rest of the file. The movement is entirely one file, | Count | Before the initializer rewrite | Now | |---|---:|---:| -| only pilot | 82 | **1582** | -| pilot diagnostics | 123 | **1623** | +| only pilot | 82 | **1614** | +| pilot diagnostics | 123 | **1655** | | severity-only | 9 | **3** | The rewrite itself took only-pilot to 61 and pilot diagnostics to 101; the `Now` column states @@ -925,7 +925,7 @@ Per category, the only-ours totals are: `pilot-examples` 4 `unmapped`, 2 advisory of the [runtime showcase round](#runtime-showcase-round)); `testdata` 7 `unmapped`, 1 `multiplicity`; `probes` 6 `unmapped`. Only-pilot: `testdata` 12 `kind-mismatch`, 3 `unmapped`, 3 syntax, 2 `unresolved-reference`; -`examples` 10 syntax, 29 `unmapped`, 661 `kind-mismatch`, 862 `unresolved-reference` — of which +`examples` 10 syntax, 29 `unmapped`, 669 `kind-mismatch`, 886 `unresolved-reference` — of which `relay-probe-demo/mission.sysml` carries none: it carried a `kind-mismatch` on its send of a `Telemetry` invocation until the send-argument round above, and the demo now writes the constructor, `send new Telemetry(…) via antenna`, which both implementations accept, so the row @@ -1017,13 +1017,13 @@ page's history. | Count | Now | |---|---:| | overall: fully agreeing / only ours / our diagnostics | **347 / 38 / 79** | -| only pilot | **1582** | -| pilot diagnostics | **1623** | +| only pilot | **1614** | +| pilot diagnostics | **1655** | | severity-only | **3** | | unmapped, our side | **34** | | kerml-examples: only ours | **10** | | pilot-examples: only ours | **7** | -| examples: only pilot | **1562** | +| examples: only pilot | **1594** | The KerML root is now the *cleanest* of the three OMG roots in proportion: **10** only-ours against 6 only-pilot, with 49 of 58 files fully diff --git a/docs/reference/cli.md b/docs/reference/cli.md index 3a09dd224..8f1f799c2 100644 --- a/docs/reference/cli.md +++ b/docs/reference/cli.md @@ -274,6 +274,7 @@ written in, so the verdicts are about that object: | `-instantiate ` | Creates an object first, so the verdicts are about it; with `-run-query` or `-render-document`, so the query reads it ([Objects the session holds](../manual/query-cookbook.md#objects-the-session-holds)). Under `-schedule explore`, `-engine check`, `smt` or `all` each run creates an object of the declaration of its own before its behaviors start, one per `-instantiate` as the session holds one per `-instantiate`, which a `-state` or `-action` named alone attaches to and a path such as `Mission::mission.vehicle` walks into ([Objects an exploration runs on](#objects-an-exploration-runs-on)) | | `-calc "()"` | Invokes a calculation and reports what it computed | | `-analysis "[()] [object]"` | Runs an analysis or [verification](#verification-case-verdicts) case — a [trade study](#trade-studies) included — and reports its `out` and `return` values with their units, then the verdict of its `objective` — `satisfied`, `not satisfied` with the violated condition, or `undecided` with the reason — as `%analysis` does. An objective typed by a requirement def binds the def's subject as a requirement usage does (`subject = ship;`, `subject s = ship;` or `subject :>> s = ship;`); one binding none checks the case's result, the library's default for it, and is `undecided` naming the type when that result is not of the subject's type. Arguments bind the case's `in` parameters, positionally (`Pkg::Case(3.0)`) or by name (`Pkg::Case(limit = 3.0)`); the object, one `-instantiate` created and named as `-state` names its performer, is the case's `subject`. A usage that binds its subject (`subject s = ship;`) needs no object; a definition, or a usage that binds none, is refused by name without one. A verification case runs the same way and reports beside those verdicts the `VerdictKind` its body produced. Repeatable | +| `-tool-dry-run "[()] [object]"` | Shows what the external tool the case's or action's `ToolExecution` names would be given — manifest, executable, argv, environment, working directory, standard input, input file and reply mapping — with the model's current values and the invocation's arguments bound, without starting the process, then discards everything the run did — the session is as the preview found it — as `%tool` does. A run reaching no `ToolExecution`-annotated action reports that; a manifest fault, an unregistered tool or an input the call does not send reports the typed error the real run would fail with; refused under an exploring schedule, which runs many linearizations while a preview shows one run's first call. Repeatable. See [External tools](external-engines.md) | | `-record-run "[()] [object]"` | Runs an analysis case as `-analysis` does and records the run into the model as `AnalysisRecords` elements: a record definition named for the case in a `Records` package beside the case's, and one part under it per run carrying the inputs bound and the outputs produced, annotated `@AnalysisRecords::RecordedRun` with when the run was made, the tool and command, and its kind. With `-sweep` the case sweeps as `-sweep` makes it and one record per row is written (`kind = "sweep"`); with `-runs ` and `-seed` a `Simulation::MonteCarlo` case is sampled as `-runs` makes it and each run recorded (`kind = "runs"`). Composes with `-convert sysml -o`, which writes the session text the records joined, and with `-render-document`, whose queries then see the records; a run that fails records nothing and leaves the model untouched. Repeatable. See [Recording analysis runs](#recording-analysis-runs) | | `-record-into ` | Records the `-record-run` runs into the package named instead of a `Records` package beside the case's; refused without `-record-run` | | `-run-query " [

=...]"` | Executes a document query and reports its rows, as `%run-query` does — including any computed `Column(name = "", expression = )` and relationship-derived `RelatedColumn(...)` projections evaluated per row — a `Column` expression may be a feature chain (`stat.runs`, `'Monte Carlo'.runs`) reading a feature of a member nested in the row element, as may a `properties`/`property` string. Each binding is written as `=`; a name binds the object `-instantiate` created under it while the run holds one (`#2` and `car.wheels[2]` bind an object by id and by path), and the element otherwise. A query over `Verdicts` reports each row as ` on : ` ([Which constraints and requirements hold](../manual/query-cookbook.md#which-constraints-and-requirements-hold)). The queries run after `-state`, `-action` and `-advance` have run, so `States`, `InState` and `Events` read where the run left the objects and, with `-trace`, what it recorded — a state row as `. in `, an event row as `t= .: ` ([Where the objects stand and what they did](../manual/query-cookbook.md#where-the-objects-stand-and-what-they-did)) | diff --git a/docs/reference/environment.md b/docs/reference/environment.md index 24296069d..2a148e7b8 100644 --- a/docs/reference/environment.md +++ b/docs/reference/environment.md @@ -151,7 +151,10 @@ The reply is read as the `reply` block below says — from standard output by de the timeout, size bounds and divergence report apply unchanged; `-engines` shows the protocol as `argv+json`, `argv+none`, `argv+csv` or `argv+template` rather than `object`, with the reply's format appended when it is not `object` (`argv+none/csv`, or the format alone for an -entry with no `invocation`). +entry with no `invocation`). `-tool-dry-run` at the CLI and `%tool` in the REPL show the +composition an entry would make for a case's or action's first tool call — executable, argv, +environment, cwd, stdin, input file and reply mapping — without starting the process; see +[Running external programs from a model](../manual/running-external-programs.md). **Reply.** Without more, the reply is the protocol's one JSON object on standard output. An optional `reply` block reads a different reply for a program that answers as a JSON document, diff --git a/docs/reference/external-engines.md b/docs/reference/external-engines.md index 908cc6505..60dd0437d 100644 --- a/docs/reference/external-engines.md +++ b/docs/reference/external-engines.md @@ -64,10 +64,13 @@ directory, as a bad run bound is reported. ## Listing, probing, selecting -`-engines` and `%engines` read the manifests and print each entry with its kind, protocol -(`/`), authority, question kinds and status, then one source line per +`-engines` and `%engines` read the manifests and print each entry with its kind, protocol, +authority, question kinds and status, then one source line per manifest entry — the file, the resolved command, and `not admitted` for an engine — spawning no -process. The status is what the file can tell: `ready (spin-bridge 1.4.0 at /opt/…)` when the +process. The protocol is `/` for an engine entry; a tool's is `object` for +the one-JSON-object exchange or `argv+` when the entry has an `invocation` block, with a +`/` suffix such as `argv+none/csv` when the `reply` block reads another format — +see [External tools](environment.md#external-tools). The status is what the file can tell: `ready (spin-bridge 1.4.0 at /opt/…)` when the command resolves to an executable regular file, `unavailable: ` when it does not or the entry is not served. diff --git a/docs/reference/repl-commands.md b/docs/reference/repl-commands.md index fe566436d..e9b70ec7d 100644 --- a/docs/reference/repl-commands.md +++ b/docs/reference/repl-commands.md @@ -46,6 +46,7 @@ into the parts it holds (`car.fl.hub`, `#3.fl`, `car.wheels[2]`). | `%jobs []` | Show or set how many runs of one check asked from then on may go concurrently — the linearizations a check explores under `%schedule explore`, the engines `%engine all` consults — the rows of a `%sweep` or `%samples` — each on a worker of its own over the session's model — and how many files of one `%load` are parsed and validated at once; `OPENSYSML_JOBS`, else one per CPU the memory available allows, until set. The result of a check is the same at any count. The count bounds a plan's runs, not the session: the held context, its objects and a debugging session under way are untouched by setting it. A value that is not a positive integer is refused and the count left as it was ([Running in parallel](cli.md#running-in-parallel)) | | `%engines [probe]` | List the analysis engines of the build in name order — the kind of each, the protocol it is spoken by, the authority it carries, the question kinds it answers and its status (`ready`, `ready (z3 at …)` for one whose process was found, `unavailable: `), then one line per manifest entry naming its file and command — as the CLI's [`-engines`](cli.md#analysis-engines) does, starting nothing. `%engines probe` also starts each [external engine](external-engines.md) once, checks its `describe` against its manifest entry field by field and reports the outcome as its status, as `-engines -probe` does; any other argument is refused | | `%engine [\|auto\|all]` | Show or set the analysis engine every question asked from then on — `%constraint`, `%requirement`, `%satisfy`, `%validate`, `%calc`, `%analysis`, `%sweep`, `%samples`, `%check` and the other solver commands — is put to. `auto` (the default) picks the engine of highest authority covering the question and advances past one that refuses or answers *not covered*; a name puts it to that engine alone, whose refusal is then the verdict; `all` puts it to every covering engine, one after another in name order, and composes their answers, naming a disagreement in the interpreter's favor. Every verdict is followed by a `standing:` line — the claim, the strength of the evidence (*not covered*, *observed*, *witnessed*, *bounded*, *proved*) and what earned it — and under `all` each engine's part. A name no engine is registered under is refused and the selection left as it was. `explore` is refused at the prompt as `%schedule explore` is, since the debuggers step one run; the `%action` and `%state` debuggers keep the schedule `%schedule` set whatever the engine ([Analysis engines](cli.md#analysis-engines)). `%engine check` is the one selection that changes what `%action` does by itself: it puts the action to the `check` engine, which searches every schedule for a violation, a deadlock, a failure or a divergence and prints the verdict, instead of starting a debugging session; `%engine all` does the same, the exploration beside the checker, once a `%check-*` setting is made ([Checking every schedule](#checking-every-schedule-of-an-action-or-a-state-machine)) | +| `%tool [()] []` | Show what the external tool a case's or action's `ToolExecution` names would be given — manifest, executable, argv, environment, working directory, standard input, input file and reply mapping — with the model's current values and the invocation's arguments bound, without starting the process, then discards everything the run did — the session is as the preview found it — as `-tool-dry-run` does ([External tools](external-engines.md)). A run reaching no `ToolExecution`-annotated action reports that; a manifest fault, an unregistered tool or an input the call does not send reports the typed error the real run would fail with. Refused under an exploring schedule, which runs many linearizations while a preview shows one run's first call | | `%check-property [...\|off]` | Show or set the constraints and requirements the `check` engine evaluates at every stable state of a checked action, on its performing object where there is one; `off` (the default) names none | | `%check-diverge [...\|off]` | Show or set the features whose final values are compared across schedules, which makes the checked action's question one of *sensitivity*, answered by the `check` engine's divergence search or the `smt` engine's two-copy query, whichever `%engine` names, and by both under `%engine all` — `x` for the action's attribute, `step.out` for an output of a node it performs, `this.level` for the performing object's, a name nothing holds refused; `off` (the default) names none, and the `check` engine then compares every attribute of the action and of its performing object (an action run without one on its own attributes only). The CLI's [`-check-diverge`](cli.md#deciding-whether-the-schedule-decides-a-feature) | | `%check-input [...\|off]` | Show or set the features of a checked action the `smt` engine leaves free in their declared type's domain although the model binds them — a default, a value the performing object holds; `off` (the default) frees only the inputs the model leaves unbound and pins every bound one at its value. A name that is not a feature the action reads is refused naming it when the action is checked. The CLI's [`-check-input`](cli.md#deciding-a-property-over-the-inputs) | diff --git a/examples/README.md b/examples/README.md index 5b58c0554..8c1d61da3 100644 --- a/examples/README.md +++ b/examples/README.md @@ -33,6 +33,7 @@ Each of these is a model and a walkthrough of the commands that exercise it. | [analysis-demo/lander.sysml](analysis-demo/lander.sysml) | [analysis-demo/README.md](analysis-demo/README.md) | analysis cases, asked every way the tool answers them: an analysis whose action steps feed each other and whose objective is a requirement, run bound, with arguments and on an object; a verification case whose body decides its verdict beside its objective; a parameter sweep and a seeded sample; two trade studies choosing among three landers; an action and a state machine due at the same instant of one clock, under each scheduling policy and explored; `-trace`, `-json`, the REPL forms and [the same questions from Python](analysis-demo/lander_demo.py) | | [verdicts-demo/rover.sysml](verdicts-demo/rover.sysml) | [verdicts-demo/README.md](verdicts-demo/README.md) | `Verdicts(...)` in a document query: one row per assertion about a rover and every object it holds — its constraints, a requirement it carries, a `satisfy` of its battery with the verification case verifying it, six wheels from a multiplicity and a heater no value decides — over the declared object, then over the object a session holds after a drive; what the table sees that evaluating one expression does not | | [analysis-results-demo/lander-results.sysml](analysis-results-demo/lander-results.sysml) | [analysis-results-demo/README.md](analysis-results-demo/README.md) | saving analysis runs into the model: `-analysis` and `-sweep` print and discard, so each run is written back as a `part` usage typed by a result-record definition with `@RecordedRun` provenance metadata and a `ref part` to its subject — then a document groups the records by subject, filters the sweep rows and the trade study, lists every annotated record's command, and flags the record a later model edit made stale, beside live `Verdicts` recomputed at render time | +| [external-tool-demo/thermal.sysml](external-tool-demo/thermal.sysml) | [external-tool-demo/README.md](external-tool-demo/README.md) | an analysis action performed by a program outside the tool: a Python solver registered through `OPENSYSML_TOOLS` with an `invocation`/`reply` manifest block, previewed by `-tool-dry-run` without starting it, run by `-analysis`, recorded by `-record-run` with every tool call named in `RecordedRun.tools`, and tabulated by a rendered document | | [solver-demo.sysml](solver-demo.sysml) | [SOLVER-DEMO.md](SOLVER-DEMO.md) | `%check`, `%explain`, `%solve`, `%configure` and `%optimize` — what conditions *can* hold, which conflict, what satisfies them, which variants are permitted, what is best (needs z3 or cvc5) | | [oosem-demo/oosem-demo.sysml](oosem-demo/oosem-demo.sysml) | [oosem-demo/README.md](oosem-demo/README.md) | the `OOSEM` library on a small Earth-observation mission: as-is and to-be enterprise, causal analysis, stakeholder needs derived down to component requirements with `#moe`/`#mop`, the black-box system context and its use case, the logical scenario and components, and the physical architecture distributed over nodes | | [mosa-demo/mosa-demo.sysml](mosa-demo/mosa-demo.sysml) | [mosa-demo/README.md](mosa-demo/README.md) | the `MOSA` library on a modular ground vehicle: the major system platform, its major system components and a modular autonomy system, the modular system interfaces between them (one written as `#keyInterface`), the consensus standards they conform to, data rights and proprietary elements, interface control, MOSA requirements with their traces, a conformance assessment, the MOSA views and a generated interface control document; `-validate` reports the openness gaps the model leaves on purpose | diff --git a/examples/external-tool-demo/README.md b/examples/external-tool-demo/README.md new file mode 100644 index 000000000..173bd157c --- /dev/null +++ b/examples/external-tool-demo/README.md @@ -0,0 +1,137 @@ +# External tool demo: a model running a program beside it + +[`thermal.sysml`](thermal.sysml) asks one question: **can an analysis case hand +its work to a program that is not OpenSysML, and can the run still be previewed, +recorded and reported?** + +Yes. The action `ThermalDemo::Solve` carries `@AnalysisTooling::ToolExecution` +naming the tool `ThermalSolver`, so its performance is given to that tool +instead of a body. The tool here is a three-argument Python script, +[`tools/thermal.py`](tools/thermal.py), that computes a steady-state +temperature and answers as CSV — the manifest entry +[`tools/thermal.json`](tools/thermal.json) says how OpenSysML composes the +command from the model's values (`invocation`) and reads the reply (`reply`). + +## Setup + +```bash +make build-sysml # writes bin/sysml +``` + +A manifest directory is read only when the environment names one **outside +every workspace** — one under the working directory you load the model from is +refused, since a model must not be able to register a program — so copy the +demo's `tools/` somewhere outside the checkout before pointing +`OPENSYSML_TOOLS` at it: + +```bash +cp -r examples/external-tool-demo/tools ~/.opensysml-tools +export OPENSYSML_TOOLS=~/.opensysml-tools +``` + +The entry's `executable` is the bare name `python3`, looked up on `PATH`; its +`cwd` is `.`, the manifest directory itself, which is why `thermal.py` lives +beside `thermal.json` — a `cwd` path must stay inside the manifest directory. + +## What the manifest buys + +`-engines` lists the tool as `tool:ThermalSolver`, and the `protocol` column +says how the entry composes the process: `argv+none/csv` — arguments over +argv, nothing on stdin, a CSV reply: + +```bash +$ ./bin/sysml -engines +engine kind protocol authority answers status +... +tool:ThermalSolver tool argv+none/csv observed compute ready (ThermalSolver 1.0 at /usr/bin/python3) +tool:ThermalSolver 1.0: tool from /home/me/.opensysml-tools/thermal.json, runs python3 +``` + +## Previewing the call + +`-tool-dry-run` composes the invocation with the model's current values and +stops short of the process: + +```bash +$ ./bin/sysml examples/external-tool-demo/thermal.sysml -tool-dry-run ThermalDemo::heating +✓ ThermalDemo::heating: dry run of tool 'ThermalSolver' for ThermalDemo::Solve + tool: ThermalSolver 1.0 + protocol: argv+none/csv + manifest: /home/me/.opensysml-tools/thermal.json + executable: /usr/bin/python3 + argv: + "thermal.py" + "--mass" + "12.5" + "--power" + "250" + "--ambient" + "290" + ... + reply: csv from stdout, header, delimiter "," + margin: column "margin", row last, type number, unit K + tMax: column "tmax", row last, unitColumn "tmaxUnit", type number + the process was not started +``` + +## Running it + +```bash +$ ./bin/sysml examples/external-tool-demo/thermal.sysml -analysis ThermalDemo::heating +✓ ThermalDemo::heating + tMax = 306.0 [SI::K] + margin = 94.0 [SI::K] + objective marginHeld: satisfied + standing: value (observed: 1 run under reverse) +``` + +## Recording it + +`-record-run` writes the run into the model as an `AnalysisRecords` element in +the `Records` package beside the case, and `tools` on the `RecordedRun` +annotation names every external tool call the run made — the provenance a +printed report would have lost: + +```bash +$ ./bin/sysml examples/external-tool-demo/thermal.sysml \ + -record-run ThermalDemo::heating -convert sysml -o recorded.sysml +✓ ThermalDemo::heating + ... + recorded Records::heating_run1 (Records::HeatingRun) +wrote recorded.sysml (sysml, 6030 bytes) +``` + +```sysml +part heating_run1 : HeatingRun { + @AnalysisRecords::RecordedRun { + runAt = "2026-10-15T11:24:00Z"; + tool = "sysml dev"; + command = "-record-run \"ThermalDemo::heating\""; + kind = "run"; + tools = ("ThermalSolver 1.0 from /home/me/.opensysml-tools/thermal.json: /usr/bin/python3 thermal.py --mass 12.5 --power 250 --ambient 290"); + } + ... +} +``` + +## Reporting it + +Recording and rendering compose: one invocation records the run, then the +document's queries see the record. `Reporting::HeatingReport` projects every +`@RecordedRun`-annotated `AnalysisRun` in `Records` — an annotation's own +attributes are not projectable, so `tools` is read in the record's source +(above) rather than in the table: + +```bash +$ ./bin/sysml examples/external-tool-demo/thermal.sysml \ + -record-run ThermalDemo::heating \ + -render-document Reporting::HeatingReport -o report.md +``` + +[`report.md`](report.md) is committed as the render the commands produce. + +See [Running external programs from a +model](../../docs/manual/running-external-programs.md) for the whole story and +[the `OPENSYSML_TOOLS` +reference](../../docs/reference/environment.md#external-tools) for every +manifest member. diff --git a/examples/external-tool-demo/report.md b/examples/external-tool-demo/report.md new file mode 100644 index 000000000..f73e265d9 --- /dev/null +++ b/examples/external-tool-demo/report.md @@ -0,0 +1,11 @@ +# Heating runs an external tool answered + +Each row is a run of ThermalDemo::heating recorded with -record-run: the inputs bound, the outputs the ThermalSolver tool computed and the objective's verdict. + +## Recorded runs + +*Every @AnalysisRecords::RecordedRun-annotated AnalysisRun in the Records package* + +| name | caseName | kind | mass | power | ambient | tMax | margin | objective | +| --- | --- | --- | --- | --- | --- | --- | --- | --- | +| heating\_run1 | ThermalDemo::heating | run | 12.5 | 250 | 290 | 306 | 94 | satisfied | diff --git a/examples/external-tool-demo/thermal.sysml b/examples/external-tool-demo/thermal.sysml new file mode 100644 index 000000000..22c51828d --- /dev/null +++ b/examples/external-tool-demo/thermal.sysml @@ -0,0 +1,108 @@ +// An analysis case whose action is performed by an external program: the +// `ToolExecution` metadata on `Solve` names the tool the `ThermalSolver` +// manifest entry registers, and `ToolVariable` on each parameter names it as +// the manifest's `variables` spell it. See README.md for the manifest and +// the commands that preview, run, record and report it. +package ThermalDemo { + private import ScalarValues::*; + private import ISQ::*; + private import AnalysisTooling::*; + + part def Heater { + attribute mass : MassValue; // kg the heat spreads through + attribute power : PowerValue; // W the load draws + attribute ambient : TemperatureValue; // K it starts and is held near + } + + part box : Heater { + attribute :>> mass = 12.5 [SI::kg]; + attribute :>> power = 250.0 [SI::W]; + attribute :>> ambient = 290.0 [SI::K]; + } + + // The requirement the objective is typed by: 400 K of headroom. + requirement def ThermalMargin { + subject h : Heater; + in attribute tMax : TemperatureValue; + attribute limit : TemperatureValue = 400.0 [SI::K]; + require constraint { tMax <= limit } + } + + // The action the tool performs; its body never runs while the metadata is + // present and the tool is registered. + action def Solve { + metadata ToolExecution { + toolName = "ThermalSolver"; + uri = "thermal://demo/solve"; + } + in mass : MassValue { @ToolVariable { name = "mass"; } } + in power : PowerValue { @ToolVariable { name = "power"; } } + in ambient : TemperatureValue { @ToolVariable { name = "ambient"; } } + out tMax : TemperatureValue { @ToolVariable { name = "tMax"; } } + out margin : TemperatureValue { @ToolVariable { name = "margin"; } } + } + + analysis def Heating { + subject h : Heater; + in attribute mass : MassValue = 12.5 [SI::kg]; + in attribute power : PowerValue = 250.0 [SI::W]; + in attribute ambient : TemperatureValue = 290.0 [SI::K]; + action solve : Solve { + in mass = mass; + in power = power; + in ambient = ambient; + } + + return tMax : TemperatureValue = solve.tMax; + out margin : TemperatureValue = solve.margin; + + objective marginHeld : ThermalMargin { + subject = h; + in tMax = solve.tMax; + } + } + + analysis heating : Heating { + subject h = box; + } +} + +// The package `-record-run` writes its records into, beside the case's own; +// the report's queries walk its direct members. +package Records { + private import ScalarValues::*; +} + +package Reporting { + private import ScalarValues::*; + private import DocumentQueries::*; + + // Every recorded heating run. `Project` reads the record's own features + // only — an annotation's attribute values, `tools` among them, are not + // projectable, so the tools a run reached are read in the record's source. + calc def ThermalRuns :> Query { + Project( + source = WhereType( + source = WhereMetadata( + source = Descendants(source = Named(qualifiedName = "Records"), maxDepth = 1), + 'metadata' = "AnalysisRecords::RecordedRun"), + type = "AnalysisRecords::AnalysisRun"), + properties = ("name", "caseName", "kind", "mass", "power", "ambient", "tMax", "margin", "objective")) + } + + part def HeatingReport :> Document { + attribute redefines title = "Heating runs an external tool answered"; + + part intro : Paragraph { + attribute redefines text = "Each row is a run of ThermalDemo::heating recorded with -record-run: the inputs bound, the outputs the ThermalSolver tool computed and the objective's verdict."; + } + + part runs : Section { + attribute redefines title = "Recorded runs"; + part table : Table { + attribute redefines caption = "Every @AnalysisRecords::RecordedRun-annotated AnalysisRun in the Records package"; + calc rows : ThermalRuns; + } + } + } +} diff --git a/examples/external-tool-demo/tools/thermal.json b/examples/external-tool-demo/tools/thermal.json new file mode 100644 index 000000000..67be2779e --- /dev/null +++ b/examples/external-tool-demo/tools/thermal.json @@ -0,0 +1,19 @@ +{ + "toolName": "ThermalSolver", + "version": "1.0", + "executable": "python3", + "variables": ["mass", "power", "ambient", "tMax", "margin"], + "invocation": { + "args": ["thermal.py", "--mass", "{mass}", "--power", "{power}", "--ambient", "{ambient}"], + "cwd": ".", + "stdin": "none" + }, + "reply": { + "format": "csv", + "source": "stdout", + "outputs": { + "tMax": {"column": "tmax", "type": "number", "unitColumn": "tmaxUnit"}, + "margin": {"column": "margin", "type": "number", "unit": "K"} + } + } +} diff --git a/examples/external-tool-demo/tools/thermal.py b/examples/external-tool-demo/tools/thermal.py new file mode 100644 index 000000000..06f0a2ffe --- /dev/null +++ b/examples/external-tool-demo/tools/thermal.py @@ -0,0 +1,32 @@ +#!/usr/bin/env python3 +"""Steady-state temperature solver for the external-tool demo. + +Reads --mass (kg), --power (W) and --ambient (K) from the command line and +prints one CSV record to standard output: the temperature the mass settles +at and the margin left to the 400 K limit. +""" + +import argparse +import csv +import sys + + +def main(): + parser = argparse.ArgumentParser(description="steady-state thermal solver") + parser.add_argument("--mass", type=float, required=True, help="mass in kg") + parser.add_argument("--power", type=float, required=True, help="heat load in W") + parser.add_argument("--ambient", type=float, required=True, help="ambient temperature in K") + args = parser.parse_args() + if args.mass <= 0: + print("mass must be positive", file=sys.stderr) + return 2 + tmax = args.ambient + 0.8 * args.power / args.mass + margin = 400.0 - tmax + writer = csv.writer(sys.stdout, lineterminator="\n") + writer.writerow(["tmax", "tmaxUnit", "margin"]) + writer.writerow([tmax, "K", margin]) + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/internal/exec/analysis/dispatch.go b/internal/exec/analysis/dispatch.go index e85490fe8..0c3d7d816 100644 --- a/internal/exec/analysis/dispatch.go +++ b/internal/exec/analysis/dispatch.go @@ -8,6 +8,8 @@ import ( "strings" "sync" "time" + + "github.com/Open-MBEE/OpenSysML/internal/exec/runtime" ) // Plan is how a question was answered: the selection made, every engine @@ -24,12 +26,77 @@ type Plan struct { // Disagreements are the contradictions the composition under all resolved, // each in the interpreter's favor; empty under auto or a named engine. Disagreements []Disagreement + // Tools is every tool call the plan's runs made when the plan ended, in call order. + Tools []ToolUse + // tools is the runner the calls were made through, kept so calls a run's + // context makes after the plan ended — a Monte Carlo conclusion's — are read too. + tools *toolRunner // Workers is how many workers the plan built over every fleet its engines ran on, and // Warming the time building them took, summed. Workers int Warming time.Duration } +// uses is every tool call the plan's runs made, including the calls a run's +// context made after the plan ended. +func (p Plan) uses() []ToolUse { + if p.tools != nil { + return p.tools.used() + } + return p.Tools +} + +// ToolTexts is what every tool call the plan made ran, as ToolUse spells it, +// including calls a run's context made after the plan ended. +func (p Plan) ToolTexts() []string { + uses := p.uses() + texts := make([]string, 0, len(uses)) + for _, use := range uses { + texts = append(texts, use.String()) + } + return texts +} + +// ToolMark is the count of tool calls made so far: a mark taken before a +// point — a Monte Carlo conclusion's run in a row's context — bounds the +// calls before it. +func (p Plan) ToolMark() int { + return len(p.uses()) +} + +// ToolTextsInBefore is the texts of the calls made from ctx before mark: +// calls the same context made after it, a conclusion's, are not the run's. +func (p Plan) ToolTextsInBefore(ctx *runtime.Context, mark int) []string { + if ctx == nil { + return nil + } + uses := p.uses() + if mark > len(uses) { + mark = len(uses) + } + var texts []string + for i := 0; i < mark; i++ { + if uses[i].in == ctx { + texts = append(texts, uses[i].String()) + } + } + return texts +} + +// ToolTextsIn is the texts of the calls made from ctx, nil for a nil ctx. +func (p Plan) ToolTextsIn(ctx *runtime.Context) []string { + if ctx == nil { + return nil + } + var texts []string + for _, use := range p.uses() { + if use.in == ctx { + texts = append(texts, use.String()) + } + } + return texts +} + // Step is one engine's part in a plan: a refusal before running, the result it // answered, the fault that stopped the plan, or its cancellation under all. type Step struct { @@ -162,9 +229,13 @@ func (r *Registry) AnswerWith(ctx context.Context, model *Model, q Question, bud defer cancel() } held := model.plan() - held.compute(r.newToolRunner(ctx, held, budget, selection)) + tools := r.newToolRunner(ctx, held, budget, selection) + held.compute(tools) defer held.release() - return r.answer(ctx, held, q, budget, selection) + plan, err := r.answer(ctx, held, q, budget, selection) + plan.Tools = tools.used() + plan.tools = tools + return plan, err } // answer answers q on the plan's copy of the model, whose tool runner puts every diff --git a/internal/exec/analysis/dryrun.go b/internal/exec/analysis/dryrun.go new file mode 100644 index 000000000..94fb44c41 --- /dev/null +++ b/internal/exec/analysis/dryrun.go @@ -0,0 +1,434 @@ +package analysis + +import ( + "errors" + "fmt" + "sort" + "strconv" + "strings" + "sync" + + "github.com/Open-MBEE/OpenSysML/internal/exec/runtime" + "github.com/Open-MBEE/OpenSysML/internal/semantic/symbols" +) + +// DryRun is what one tool call would be given, composed without starting the process. +type DryRun struct { + Tool, Version, Manifest, Executable, Protocol string + // Args are the argv entries after the executable, as the invocation block renders + // them; nil for an entry without one. + Args []string + // Env is the environment in NAME=value pairs, sorted; nil when the entry has no + // invocation block, in which case the whole environment is inherited. + Env []string + // Cwd is the working directory; empty inherits this process's. + Cwd string + // Stdin says what the process reads; StdinJSON for an entry without the block. + Stdin StdinFormat + // StdinData is what standard input carries. + StdinData []byte + // InputFile is the file the block's `inputFile` writes, nil when it has none. + InputFile *DryRunFile + // OutputDir marks that the entry uses {outputDir}. + OutputDir bool + // Inputs are the values the call sent the tool. + Inputs []runtime.ToolInput + // Outputs are the bindings the call expects back, sorted by variable. + Outputs []runtime.ToolOutput + // Reply is the entry's `reply` block; nil is the object protocol. + Reply *Reply + // ReplySource is where the reply is read, rendered like argv: stdout, or + // file: