perf(model): parse, index and validate a load's files on a pool of workers

.agents/skills/testing-sysml-repl/SKILL.md

@@ -2530,9 +2530,66 @@ its output rather than in an exit code — so assert on the exact rendered text:
 
 ## Multi-file projects: `%load <path>...` and positional dirs/globs (PR #146)
 
+### Per-file document isolation probes
+
+- Give only one file a root-level `private import ScalarValues::*;`. A second
+  file's bare `Real` must stay unresolved, as must a later prompt declaration's
+  bare `Real`. A qualified expression such as `%eval A::x + 1.0` should still
+  work, proving isolation did not remove the loaded package from the index.
+- Two loaded files declaring the same root package are two root namespaces, not
+  a duplicate. References select the declaration in the document whose name
+  sorts first, independent of CLI argument order (see the CLI reference's
+  Multiple Files section). Put `A::X` in `first.sysml` and `A::Y` in
+  `second.sysml`, then reverse arguments: `A::X` must resolve and `A::Y` must
+  remain unresolved in both orders. Do not confuse reference precedence with
+  document rendering order.
+- For rendering order, `%view` takes a **view**, not an ordinary package.
+  `%render #table` renders the loaded documents without a declared view; reverse
+  two nonalphabetical package names and assert their member groups reverse.
+  A declared view with `render asElementTable;` needs `private import Views::*;`
+  in its scope.
+- `%save` passes notation through the formatter. Test source retention separately
+  from byte equality: tabs can become four spaces even while comments, members,
+  file order and typed declarations survive. Compare with a `develop` build
+  before attributing such formatting to a load-path regression.
+- Both debugger fixtures in `internal/frontend/repl/testdata/` are load-ready:
+  `action_debug.sysml` (`%action Debug::tally`, `%step`, type `part def Z;`,
+  `%continue`) ends at `total = 5`; `state_debug.sysml` (`%state Debug::Cycle`,
+  `%advance 1`, type `part def Z;`, `%advance 9`, `%advance 5`) reaches working
+  at t=10 and done at t=15. This tests symbol rebinding across prompt edits.
+
+#### Devin Secrets Needed
+
+None for local multi-file CLI/REPL tests.
+
+### Parallel file-load verification
+
+- The shared concurrency knob is `-jobs N` / `OPENSYSML_JOBS`, also observable
+  with `%jobs`. It bounds both plan execution and files parsed/validated in one
+  load. Compare stdout, stderr and exit status at 1, 2 and 8 jobs plus an
+  environment override; test invalid values against a nonexistent path to
+  distinguish startup rejection from a load failure. `-workers` is not a
+  supported replacement flag.
+- Generate a small multi-file model from the nested tools module:
+  `go run -C tools ./cmd/stress-model -planes 4 -satellites 10 -ground-stations 8 -split-planes <scratch-dir>`.
+  Verify SHA256/name manifest records, then shrink the model: unchanged surplus
+  output should disappear, while an edited surplus plane and user file survive.
+  Editing a still-current output should refuse the entire regeneration with
+  `nothing written`; compare all directory bytes before and after.
+- A load containing `part component : Needed::T;` gives a non-vacuous batch
+  diagnostic. In `%verbosity debug`, declare `package Needed { part def T; }`,
+  then `package Needed {}`, then restore `T`. Whole-buffer diagnostics must
+  change 1→0→1→0. An empty package removes its members; a nonempty declaration
+  merges with existing members and is not a suitable deletion probe.
+- For save/reload, retain comments and loaded-file declarations and assert
+  only the latest prompt redeclaration survives. Re-evaluate a compound
+  expression after `%clear` and reloading the saved file.
+
 `sysml <dir|glob|file>...` and `%load <path>...` expand to model files via
 `internal/workspace/project.Expand`, and every file is accepted before one analysis pass
-(`Session.SubmitAll`), so load order does not affect name resolution. Shapes to expect:
+(`Session.SubmitAll`), each file a workspace document of its own indexed with the
+others. Repeated root names resolve by document-name order, not load order.
+Shapes to expect:
 
 - More than one file prints a `loaded N files:` header listing each path (a single file prints no
   header — a good tell that the multi-file path was taken).

changes/unreleased/parallel-batch-validation.performance.md

@@ -0,0 +1 @@
+- **A batch of files is parsed and analysed on a pool of workers.** `sysml -validate`, `-satisfy`, `-check` and `%load` open the files they are given as one batch: the files are parsed on one worker per CPU, indexed once, wildcard imports are expanded once for the batch instead of once per file, and the documents are analysed in parallel, each with a resolver and semantic model of its own over an index nothing writes meanwhile; the facts the workspace-wide audits (OOSEM, MOSA, identity metadata) need of every document are gathered once for the batch and shared by the workers. The diagnostics are the same at any job count, in command-line order; `-jobs N` or `OPENSYSML_JOBS`, the setting that already bounds how many runs of one check go concurrently, sets the pool. On an eight-CPU machine the 1 600-satellite stress constellation split over 34 files validates in 9.2 s against 22.0 s on one job and 20.3 s as a single file, within a tenth of the single file's peak memory; a 200-satellite split in 1.19 s against 2.63 s. `stress-model -split-planes <dir>` writes the constellation one file per orbital plane, and `docs/project/satellite-network-stress-test.md` records where the split model's remaining time goes.

changes/unreleased/per-file-documents.changed.md

@@ -0,0 +1,2 @@
+- **Every file the command line or `%load` reads is a document of its own.** `sysml -validate`, `-satisfy`, `-e` and the REPL's `%load` used to join the files they were given into one buffer with the typed transcript, so a model split over files was analysed as if it were one file; each file is now a workspace document, indexed with the others and analysed on its own, exactly as the editor and the OMG corpus gates analyse it. Two things a reader will observe: a root-level import in one file (`private import ScalarValues::*;`) no longer serves the other files on the command line or the prompt after `%load` — a KerML root import surfaces its names in its own document's root namespace only, as `docs/project/spec-compliance.md` records — and two files that both declare `package A` are no longer reported as `Duplicate of other owned member name`: they are two root namespaces of one name, and a reference to `A` resolves to the declaration in the file whose name sorts first (the order the editor gives documents, whatever order the files were given in), as the pilot implementation resolves a repeated root name to the first. Root packages stay reachable from every file and from the prompt through the global namespace. A differential test runs every multi-file directory of the fixtures and of the four OMG corpora through the command line and through a workspace and asserts the same diagnostics.
+- **A file named `<repl>` is refused by every load.** `<repl>` is the name the REPL keeps the typed transcript under, so `%load <repl>` and `sysml -validate <repl>` report `cannot load <repl>: the name is reserved for the text typed at the prompt` before anything loads and exit as a read failure does; `./<repl>` loads it.

cmd/sysml/check.go

@@ -617,8 +617,8 @@ func runChecks(files []string, exprs []string, c checks) int {
 		return rep.finish()
 	}
 
-	// The files are loaded as one submission, indexed and analyzed once, and
-	// each is still summarized on its own.
+	// The files are loaded as one submission, each a document of its own indexed
+	// with the others, and each is summarized on its own.
 	loaded, err := sess.LoadFilesSummary(paths)
 	if err != nil {
 		rep.failed(err.Error())

cmd/sysml/jobs_load_test.go

@@ -0,0 +1,83 @@
+package main
+
+import (
+	"os"
+	"path/filepath"
+	"strings"
+	"testing"
+)
+
+// A model of several files whose validation crosses them: a metadata body to
+// link, a reference into another file, and an error to report in a third.
+var loadModel = map[string]string{
+	"a.sysml": "package A {\n\tmetadata def M { attribute n; }\n\tpart def X { @M { n = 1; } }\n}\n",
+	"b.sysml": "package B { part x : A::X; part y : Missing; }\n",
+	"c.sysml": "package C { part z : A::X { @A::M { n = 2; } } }\n",
+}
+
+func writeLoadModel(t *testing.T) []string {
+	t.Helper()
+	dir := t.TempDir()
+	paths := make([]string, 0, len(loadModel))
+	for _, name := range []string{"a.sysml", "b.sysml", "c.sysml"} {
+		path := filepath.Join(dir, name)
+		if err := os.WriteFile(path, []byte(loadModel[name]), 0o644); err != nil {
+			t.Fatal(err)
+		}
+		paths = append(paths, path)
+	}
+	return paths
+}
+
+// TestJobsGovernLoads: -jobs and OPENSYSML_JOBS bound how many files of a load
+// are parsed and validated at once; -validate reports the same at any count, and a
+// bad value is rejected before any file is read, in every mode that loads a model.
+func TestJobsGovernLoads(t *testing.T) {
+	binary := buildCLI(t)
+	paths := writeLoadModel(t)
+	validate := func(env []string, args ...string) runOutcome {
+		return checkPathsEnv(t, binary, env, append(append([]string{"-validate"}, args...), paths...)...)
+	}
+	want := validate(nil)
+	if want.status != 2 || !strings.Contains(want.output(), "unresolved reference: Missing") {
+		t.Fatalf("the default run should report b.sysml's unresolved name and exit 2, got %d:\n%s", want.status, want.output())
+	}
+
+	for _, jobs := range []string{"1", "2", "8"} {
+		if got := validate(nil, "-jobs", jobs); got.status != want.status || got.output() != want.output() {
+			t.Errorf("-jobs %s reported %d\n%s\nwant the default's %d\n%s", jobs, got.status, got.output(), want.status, want.output())
+		}
+		if got := validate([]string{"OPENSYSML_JOBS=" + jobs}); got.status != want.status || got.output() != want.output() {
+			t.Errorf("OPENSYSML_JOBS=%s reported %d\n%s\nwant the default's %d\n%s", jobs, got.status, got.output(), want.status, want.output())
+		}
+	}
+
+	for _, bad := range []string{"0", "-3", "two", "1.5"} {
+		got := validate(nil, "-jobs", bad)
+		if got.status != 2 || !strings.Contains(got.output(), `-jobs="`+bad+`" is not a positive integer`) || strings.Contains(got.output(), "Missing") {
+			t.Errorf("-jobs %s: status %d\n%s", bad, got.status, got.output())
+		}
+		got = validate([]string{"OPENSYSML_JOBS=" + bad})
+		if got.status != 2 || !strings.Contains(got.output(), `OPENSYSML_JOBS="`+bad+`" is not a positive integer`) || strings.Contains(got.output(), "Missing") {
+			t.Errorf("OPENSYSML_JOBS=%s: status %d\n%s", bad, got.status, got.output())
+		}
+	}
+
+	if got := validate([]string{"OPENSYSML_JOBS=nope"}, "-jobs", "2"); got.status != want.status || got.output() != want.output() {
+		t.Errorf("-jobs 2 under OPENSYSML_JOBS=nope reported %d\n%s\nwant the default's\n%s", got.status, got.output(), want.output())
+	}
+
+	// Every mode that loads a model reads the setting before loading.
+	for _, mode := range [][]string{
+		{"-satisfy"},
+		{"-query", `sysml:name="X"`},
+		{"-render", "A::X"},
+		{"-render-all", t.TempDir()},
+		{"-compile", "A::X", "-o", filepath.Join(t.TempDir(), "x")},
+	} {
+		got := checkPathsEnv(t, binary, []string{"OPENSYSML_JOBS=0"}, append(mode, paths...)...)
+		if got.status != 2 || !strings.Contains(got.output(), `OPENSYSML_JOBS="0" is not a positive integer`) || strings.Contains(got.output(), "Missing") {
+			t.Errorf("%s under OPENSYSML_JOBS=0: status %d\n%s", mode[0], got.status, got.output())
+		}
+	}
+}

cmd/sysml/load_test.go

@@ -113,11 +113,62 @@ func TestCheckGlobExitStatus(t *testing.T) {
 	wantReport(t, checkPaths(t, binary, "-validate", filepath.Join(dir, "*.kerml")), 2, "no model files match")
 }
 
+// TestLoadRefusesTheTranscriptsName checks that a file named as the session's
+// transcript is a read failure like any other: refused before anything loads,
+// named in the error, and exiting as a run that decided nothing.
+func TestLoadRefusesTheTranscriptsName(t *testing.T) {
+	const reserved = "<repl>"
+	dir := t.TempDir()
+	write(t, filepath.Join(dir, reserved), "package FromFile { part def X; }\n")
+	write(t, filepath.Join(dir, "ok.sysml"), "package OK { part def Y; }\n")
+	t.Chdir(dir)
+
+	sess := repl.NewSession()
+	status, err := loadFiles(sess, []string{"ok.sysml", reserved})
+	var named *repl.ReservedNameError
+	if !errors.As(err, &named) || named.Name != reserved {
+		t.Fatalf("loadFiles error = %v, want a *repl.ReservedNameError naming %s", err, reserved)
+	}
+	if status != exitUnevaluable {
+		t.Errorf("status = %d, want %d", status, exitUnevaluable)
+	}
+	if got := sess.List(); len(got) != 0 {
+		t.Errorf("a refused load must load nothing, got %v", got)
+	}
+
+	binary := buildCLI(t)
+	wantReport(t, checkPathsIn(t, dir, binary, "-validate", reserved), exitUnevaluable,
+		"sysml: cannot load <repl>: the name is reserved")
+	wantReport(t, checkPathsIn(t, dir, binary, "-constraint", "OK::Held", "ok.sysml", reserved), exitUnevaluable,
+		"cannot load <repl>: the name is reserved")
+	wantReport(t, checkPathsIn(t, dir, binary, "-validate", "./"+reserved), exitHolds)
+}
+
 // checkPaths runs the binary on paths the caller names, rather than on a model
 // written to a file for it as check does.
 func checkPaths(t *testing.T, binary string, args ...string) runOutcome {
+	t.Helper()
+	return checkPathsEnv(t, binary, nil, args...)
+}
+
+// checkPathsEnv is checkPaths with variables added to the binary's environment.
+func checkPathsEnv(t *testing.T, binary string, env []string, args ...string) runOutcome {
+	t.Helper()
+	return runPaths(t, "", binary, env, args...)
+}
+
+// checkPathsIn is checkPaths run from dir, so a relative path is read there.
+func checkPathsIn(t *testing.T, dir, binary string, args ...string) runOutcome {
+	t.Helper()
+	return runPaths(t, dir, binary, nil, args...)
+}
+
+// runPaths runs the binary on the paths in dir with env added to its environment.
+func runPaths(t *testing.T, dir, binary string, env []string, args ...string) runOutcome {
 	t.Helper()
 	cmd := exec.Command(binary, args...)
+	cmd.Dir = dir
+	cmd.Env = append(os.Environ(), env...)
 	var stdout, stderr bytes.Buffer
 	cmd.Stdout, cmd.Stderr = &stdout, &stderr
 	err := cmd.Run()

cmd/sysml/main.go

@@ -503,6 +503,9 @@ func runCLI() int {
 			fmt.Fprintln(os.Stderr, "sysml: -compile needs -o to name the executable (or the source file, with -source)")
 			return 2
 		}
+		if status := resolveRunBounds(); status != 0 {
+			return status
+		}
 		if err := runCompile(args); err != nil {
 			return fail(err)
 		}
@@ -592,6 +595,9 @@ func runCLI() int {
 			return refuse(modelChecks,
 				"-render-all writes views out and decides nothing about the model; check it in its own run")
 		}
+		if status := resolveRunBounds(); status != 0 {
+			return status
+		}
 		if err := runRenderAll(args); err != nil {
 			return fail(err)
 		}
@@ -635,6 +641,9 @@ func runCLI() int {
 			fmt.Fprintln(os.Stderr, "sysml: -query cannot be combined with checks, -eval, -render, -render-document, -output or -from")
 			return 2
 		}
+		if status := resolveRunBounds(); status != 0 {
+			return status
+		}
 		return runQuery(args, queryText)
 	}
 
@@ -647,6 +656,9 @@ func runCLI() int {
 			fmt.Fprintln(os.Stderr, "sysml: -render and -render-document each write a document out; ask for one per run")
 			return 2
 		}
+		if status := resolveRunBounds(); status != 0 {
+			return status
+		}
 		if err := runRender(args); err != nil {
 			return fail(err)
 		}

cmd/sysml/usage.go

@@ -104,7 +104,8 @@ func doc() usage.Doc {
 					"run, so machines on sibling parts share it and its connectors.",
 				"-jobs is how many runs of one check may go concurrently — the " +
 					"linearizations of an exploration, the engines -engine all consults " +
-					"— each on a worker of its own over the shared model; the result is " +
+					"— each on a worker of its own over the shared model, and how many " +
+					"files of one load are parsed and validated at once; the result is " +
 					"the same at any count. By default one per CPU, fewer where the memory " +
 					"available leaves less than 512 MiB per worker.",
 			},
@@ -510,7 +511,7 @@ func doc() usage.Doc {
 		}, {
 			Title:      "Environment",
 			ManOnly:    true,
-			Items:      append(append(append(usage.BudgetEnvironment(), usage.JobsEnvironment()...), usage.ToolEnvironment()...), solverEnvironment()...),
+			Items:      append(append(append(usage.BudgetEnvironment(), usage.LoadJobsEnvironment()...), usage.ToolEnvironment()...), solverEnvironment()...),
 			Paragraphs: []string{usage.LegacyPrefixNote, usage.BudgetScopeNote},
 		}, {
 			Title:   "Files",
@@ -578,7 +579,7 @@ func registerFlags(fs *flag.FlagSet) {
 	fs.Var(&modelChecks.observe, "observe", "Report this feature of the -runs action, or clock for the time it completed at; default every feature it holds and the clock. With -compare-results, the stored observable to compare, or <observable>=<feature> to read it from another feature of the run (repeatable)")
 	fs.Var(&modelChecks.sweeps, "sweep", "Run the -analysis or -calc once per value of this range, as -sweep \"n=1..8:2\"; several ranges run their cartesian product (repeatable)")
 	fs.Var(&modelChecks.samples, "samples", "Draw this many values uniformly from each -sweep range instead of stepping through it; needs -seed")
-	fs.Var(&jobsFlag, "jobs", "Runs of one check that may go concurrently, each on a worker of its own; default OPENSYSML_JOBS, else one per CPU, fewer where the memory available leaves less than 512 MiB per worker")
+	fs.Var(&jobsFlag, "jobs", "Runs of one check that may go concurrently, each on a worker of its own, and files of one load that are parsed and validated at once; the result is the same at any count. Default OPENSYSML_JOBS, else one per CPU, fewer where the memory available leaves less than 512 MiB per worker")
 
 	fs.BoolVar(&listEngines, "engines", false, "List the analysis engines this build knows and their status, spawning nothing, and exit")
 	fs.BoolVar(&probeEngines, "probe", false, "With -engines, also start each external engine once and report the outcome as its status")