perf(model): parse, index and validate a load's files on a pool of workers

[devin-ai-integration]

What and why

sysml -validate a.sysml b.sysml …, -satisfy, -check and %load now open the files they are given as one batch and validate them on a pool of workers, sized by the existing -jobs setting. This is the parallel batch pipeline of docs/project/large-model-scaling-design.md §5, built on #309 (one workspace document per loaded file), which this branch contains; merge #309 first, then this. Until #309 lands, the diff shown here includes its commits — the changes of this PR alone are git diff feature/per-file-documents...perf/parallel-batch-validation (31 files).

The pipeline, in internal/workspace/model/batch.go:

type Input struct{ Name string; Content []byte; Version int }

func (w *Workspace) OpenAll(inputs []Input)
    // parse + symbols.Build on the pool, outside the lock
    // then, under the lock, in input order: installLocked(doc)
    //   = displaceLocked + index.AddBuiltDocument(doc.Name, doc.AST, doc.Scope) + standInLocked
    // then ExpandWildcardImports() once, invalidateLocked(installed...) once

func (w *Workspace) DiagnosticsAll(names []string) [][]diag.Diagnostic
    // settleGathersLocked(); cached entries served; the rest:
    //   batch := &passes.Batch{Documents: pending, Gathers: passes.NewGathers(), Source: w.sourceText()}
    //   passes.PrepareBatch(w.index, batch)
    //   ParallelFor(workers, …, passes.AnalyzeInBatch(name, …, batch))   // private Resolver + Model per document
    //   results cached and recorded in w.batched; returned in `names` order

• Parse in parallel, index once. symbols.Index.AddBuiltDocument takes a scope tree the caller built with symbols.Build, so the batch builds its trees on the pool and the single writer only installs them. Wildcard imports are expanded once per batch rather than once per file added, which retires the quadratic per-file reindex cost the stress-test record noted.
• Analyze in parallel over a read-only index. Each document is analyzed by passes.AnalyzeInBatch in a kit.Context of its own; nothing takes a lock inside the resolver or the model. The one place resolution wrote to the shared scope tree — linking a metadata annotation body's owner (Scope.SetOwner) on first use — is done for all documents of the batch before the pool starts (resolve.(*Resolver).LinkMetadataBodies via passes.PrepareBatch, with the model-attached resolver analysis itself uses), so the workers only read it. A metadata body scope records the declaration its annotation is written on (Scope.Annotated()), set by the builder, so the link is computable outside a resolution. passes.Batch.Source carries the workspace's read-only source lookup to the preparatory linker and each worker model, so an import filtered on Comment::body is evaluated as the editor path evaluates it.
• One gather per batch. passes.Batch.Gathers carries one passes.Gathers to every context of the batch. The first context that runs a workspace-wide audit (OOSEMMethodPass, IdentityMetadataPass, MOSAPass) gathers every document's facts into it under its lock; every context afterwards reads the same union. The three passes are untouched — they already read through Context.Gathers(); Context.InBatch sets the context's gathers to the batch's. The gathers are the batch's, not the workspace's persistent ones, because a private resolver records no dependencies and facts gathered by one could not be invalidated per dependency.
• Batch-computed diagnostics drop on any edit; pending regathers settle before the cache is read. DiagnosticsAll records the names it analyzed in Workspace.batched; invalidateLocked deletes those diagCache entries before asking the resolver what a change invalidates, and invalidateAllLocked clears the set. Since the workspace settles the regathers an edit queues on the next read (settleGathersLocked) rather than inside the edit, DiagnosticsAll settles them first, so a verdict Diagnostics cached is not served once an edit to another document has undone it (TestBatchSettlesPendingGathersBeforeServingTheCache). A Diagnostics call after an edit re-analyzes on the persistent model, with dependencies, as before; resolve_race_test.go and lazy_regather_test.go pass with the pool.
• Concurrent edits win over the batch. OpenAll parses outside the lock, so it records each input name's change count (Workspace.changes, bumped on every install and removal) before parsing and installs a document only where the count is still the one it reserved; a name opened, updated, closed or removed meanwhile — including one opened and removed again, absent both before and after — keeps its newer state (TestOpenAllKeepsAChangeMadeWhileItParsed).
• Deterministic output. Diagnostics are gathered in the order the names were given and are the same at any job count; TestParallelBatchValidationMatchesSerial in internal/workspace/model runs every fixture directory, examples/ and the four OMG corpus roots at one job and at GOMAXPROCS and asserts identical diagnostics (content and order), and identical to opening the files one by one.
• -jobs sizes the pool. No new flag: -jobs N, OPENSYSML_JOBS and %jobs — the setting that already bounds how many runs of one check go concurrently (analysis.DefaultJobs, ParseJobs, JobsFromEnv) — also set how many files of one load are parsed and validated at once. Session.SetJobs sizes the workspace's pool (Workspace.SetWorkers, the internal pool size); a value below one is rejected at startup before anything loads, in every mode that loads files (TestJobsGovernLoads). Documented in sysml -help, the environment listing, docs/reference/{cli,environment,repl-commands}.md and the regenerated man page (make man).
• Streaming-ready, not streaming. Document stays immutable and analysis state lives in each context; nothing in the pipeline holds a tree past its analysis except the workspace's own docs map, so releasing trees once interface records exist (§4) is a change to that map alone. Nothing is dropped in this PR.
• Split-planes generator. tools/cmd/stress-model -split-planes <dir> writes the one SatelliteNetwork.Split() of tests/stressmodel/split.go to disk: one .sysml per orbital plane plus library.sysml and constellation.sysml. Files are staged under the output directory, recorded with their SHA-256 in .stress-model-files (itself replaced whole by rename, never truncated) before they are moved in, and a later run removes only recorded regular files that still read as recorded — a user's own plane009.sysml, an edited one, or a directory at a recorded name survives, and an interrupted generation is retried and cleaned up. TestSatelliteNetworkSplitValidates checks the split declares the single file's network, validates clean at one job and at several, through the batch and through the persistent workspace, and that every satisfy assertion holds across files; tools/cmd/stress-model/main_test.go pins the manifest behaviors.

Where the design and the implementation differ

• The CLI did not load N documents. Before feat(repl): analyze each loaded file as a workspace document of its own #309, -validate joined the files into one <repl> document, so per-document parallelism had nothing to parallelize; feat(repl): analyze each loaded file as a workspace document of its own #309 is the prerequisite this branch is stacked on.
• The split model's cost was not per-file reindexing; it was a quadratic gather. Before the gather was shared, each of the 34 analyses gathered all 34 documents afresh in its own model: 129 s on one worker, 30.9 s on eight, 75% of the CPU in the three audits, 7.24 GB peak RSS at eight workers. With one gather per batch the split validates in 22.0 s on one job and 9.24 s on eight, within a tenth of the single file's peak RSS. That figure is measured, and it is not the ~5 s of §9: see below for what still bounds it.
• What bounds the pool now is the serial gather. Eight jobs reach 365% CPU. The eight-job profile puts 4.5 s of the 9.57 s wall in the gather (the OOSEM union 3.6 s, identity 0.52 s, MOSA 0.37 s), run by whichever context asks first over all 34 documents while the other workers wait at its lock; the sibling-file dependency scan (project.Dependencies, 0.87 s) and installing the scope trees and expanding wildcard imports before the pool (commitBatch, 1.0 s) are serial too — over 6 s of the 9.24 s on one thread. Gathering on the pool — each worker gathering its own document into the union before analysis starts — is the step left to ~5 s and is left as the follow-up; it touches the gather's locking and belongs in a change of its own.
• Per-document benchmark without a knob. passes.Options has no option to disable the audit passes and none was added; BenchmarkAnalyzeSplitPerDocument builds a passes.Registry from DefaultRegistry().Passes() minus the three and runs it over one index at one job and one per CPU.
• Locking. DiagnosticsAll holds the workspace's write lock across the pool, as Diagnostics already does across one analysis. Batch parsing happens before the lock is taken.
• The generator's replaceability check and its rename are two steps. -split-planes refuses a destination it did not write or that was edited since, but a save into planeNNN.sysml between that check and the os.Rename is overwritten; an editor honours no lock, so only a lock file could serialize concurrent generators, and none can close the editor window. The output directory is documented as the generator's own.

Measurements

Intel Xeon Platinum 8559C, 8 CPUs, 31 GiB, no swap, Go 1.25.0 linux/amd64, GOMAXPROCS=8; /usr/bin/time -v sysml -validate -memstats; one run per row; CPU = (user + sys) / wall. Models from go run -C tools ./cmd/stress-model -planes 32 -satellites 50 -ground-stations 160 (1 600 satellites; 18.4 MB in one file, 34 files split) and -planes 8 -satellites 25 -ground-stations 20 (200 satellites, 10 files).

model	files	jobs	wall	user	CPU	allocated	peak RSS
200 sat, one file	1	—	2.15 s	2.6 s	126%	806 MiB	393 MB
200 sat, split	10	1	2.63 s	3.3 s	129%	957 MiB	358 MB
		2	1.69 s	3.3 s	201%	972 MiB	379 MB
		4	1.29 s	3.3 s	271%	974 MiB	414 MB
		8	1.19 s	3.9 s	341%	976 MiB	507 MB
1 600 sat, one file	1	—	20.3 s	26.1 s	133%	6.1 GiB	2.38 GB
1 600 sat, split	34	1	22.0 s	27.6 s	129%	7.5 GiB	2.27 GB
		2	14.3 s	29.5 s	213%	7.6 GiB	2.12 GB
		4	10.7 s	29.9 s	288%	7.6 GiB	2.23 GB
		8	9.24 s	32.4 s	365%	7.6 GiB	2.63 GB

The diagnostics of the 34-file run are byte-identical at 1/2/4/8 jobs (md5 f342e2dfbbe27d422794fd745aa46f08), and so are the 200-satellite split's; every run reports no errors. Before the gather was shared, the 34-file split measured 129 s / 66.1 s / 38.6 s / 30.9 s at 1/2/4/8 workers, 39.1 GiB allocated, 1.98 → 7.24 GB peak RSS.

CPU profile, split 1 600, 8 jobs (9.57 s wall, 34.2 s of samples, -cpuprofile): the three audits' gather 4.5 s (13%) — OOSEM 3.6 s, identity 0.52 s, MOSA 0.37 s; NameResolutionPass.Run 9.0 s (26%), W9CInheritedNameConflictPass.Run 4.0 s, TypeCheckPass.Run 1.2 s; repl.preparse 2.45 s and OpenAll parsing on the pool, commitBatch 1.0 s serial, project.Dependencies 0.87 s serial; runtime.gcBgMarkWorker 5.6 s (16%), runtime.scanobject 5.8 s (17%). On one job (22.0 s wall) the 34 analyses are 17.1 s of samples, 3.9 s of them the gather, so a document averages 0.39 s: no single file bounds the pool.

Benchmarks (tests/stressmodel, -benchtime 3x, four planes, six files, 512 satellites):

BenchmarkValidateSplit/satellites=512/files=6/jobs=1-8              6.30 s/op
BenchmarkValidateSplit/satellites=512/files=6/jobs=8-8              2.14 s/op
BenchmarkAnalyzeSplitPerDocument/satellites=512/files=6/jobs=1-8    3.91 s/op
BenchmarkAnalyzeSplitPerDocument/satellites=512/files=6/jobs=8-8    1.09 s/op

The per-document benchmark (audits left out) shows the pool's own speedup, 3.6× over six files, the largest file being about a quarter of the work.

Allocation follow-ups (listed, not implemented)

Parallelism leaves what a load allocates unchanged (7.6 GiB, 108 M objects at any job count). The allocation sites the pool does not help — the inherited-name conflict pass's per-question slices and per-base member lists, symbols.FQNOf building a string per lookup, specializationChain and AllSupertypes slices, the parser's per-reference qualified-name nodes — are recorded with their shares in docs/internals/performance.md, "What a batch of files costs", each to be measured on its own before it is changed.

One parse per load is spent twice, as before: the REPL parses each file to accept it (declared names, whether it closes its own text) and the workspace parses the same bytes again as the document. The 34 split files (17 MB) parse in 1.8 s serially — ~1.8 s of the one-job 22.0 s, ~0.3 s of the eight-job wall, where it runs on the pool. Carrying the accepted tree into the workspace batch is a change to what model.Input owns, listed as a follow-up in the same section.

Edits beside batch.go

internal/check/passes/kit/pass.go: the Batch type (Documents, Gathers, Source), Context.Batch and Context.InBatch. internal/check/passes/analyze.go: PrepareBatch and AnalyzeInBatch beside AnalyzeShared; all funnel through one analyze(ctx, root) that runs the registry, drops escalated warnings and sorts. internal/workspace/model/workspace.go: the pool size with DefaultWorkers() (one per CPU); the batched set and changes counters; invalidateLocked(names ...string) drops batch-computed diagnostics before asking the resolver; Open/Update/SetOnDisk and OpenAll install through one installLocked(doc) — the library displacement, AddBuiltDocument with the document's already-built scope, and the library stand-in — then ExpandWildcardImports once per call; analyze(name, doc, batch) dispatches to AnalyzeInBatch when a batch is given and AnalyzeShared on the persistent resolver, model and gathers otherwise. Serial incremental behavior is otherwise unchanged.

internal/semantic/symbols/index.go: adding a document the index already holds removes the old one without expanding wildcard imports (the public RemoveDocument still does) and records the document as changed for the persistent resolver's invalidation; a batch of N reloads expands once, as N fresh files do (TestReplacingDocumentsExpandsOnceEqualToFreshBuild). internal/check/edit/edit.go: the reindexer, which relied on that incidental expansion, now expands explicitly. internal/semantic/resolve/document.go: LinkMetadataBodies, and a body's owner refreshed — or cleared — when its definition's document is reloaded, on both paths (TestMetadataBodyOwnerFollowsTheDefinitionsDocument, TestOpenAllReloadedMetadataDefinitionReownsAnnotationBodies). internal/frontend/repl/jobs.go, cmd/sysml/{main,usage}.go, internal/frontend/usage/environment.go: -jobs also sizes the load pool.

How it was verified

• gofmt -l . — nothing; go build ./..., go vet ./... (root and the tools module) — clean; python3 scripts/changelog.py check, make docs-check, make docs-counts, make man-check — clean.
• go test ./... and go test -race ./... — all packages ok.
• Corpus gates with the corpora present and the require variables set: OPENSYSML_REQUIRE_TRAINING_CORPUS=1 OPENSYSML_REQUIRE_PILOT_CORPORA=1 go test -count=1 ./tests/corpus -run 'TestTrainingExamples|TestPilotCorpora' — 100/100 training files clean; pilot 55/58, 95/99, 56/56, unchanged; training_examples_expected.txt untouched, no pilot ratchet moved.
• New tests: TestParallelBatchValidationMatchesSerial (fixtures, examples/, four OMG corpora; one job vs N vs one-by-one), TestDiagnosticsAllAnswersInTheOrderAsked, TestDiagnosticsAllOverSomeDocumentsMatchesAskingOneByOne, TestDiagnosticsAllOverOneDocumentPreparesTheOthersBodies, TestDiagnosticsAllReadsCommentBodiesAsAnEditorDoes, TestParallelBatchLinksAnnotationsFoundThroughTheModel, TestOpenAllReplacesEarlierDocuments, TestOpenAllKeepsAChangeMadeWhileItParsed, TestOpenAllReloadedMetadataDefinitionReownsAnnotationBodies, TestWorkersSetting, TestBatchSettlesPendingGathersBeforeServingTheCache in internal/workspace/model; TestJobsGovernLoads in cmd/sysml; TestReplacingDocumentsExpandsOnceEqualToFreshBuild in internal/semantic/symbols; TestLinkMetadataBodies* and TestMetadataBodyOwnerFollowsTheDefinitionsDocument in tests/resolve; TestSatelliteNetworkSplitValidates, BenchmarkValidateSplit, BenchmarkAnalyzeSplitPerDocument in tests/stressmodel; the seven TestWriteSplit* manifest tests in tools/cmd/stress-model.
• By hand: sysml -validate over the 200- and 1 600-satellite splits at -jobs 1/2/4/8 produces byte-identical stdout per model; the single-file output is byte-identical to develop's.

Checklist

• make test and make lint pass locally
• Tests added or updated for the change
• Documentation extended where it already covers the surface (see CONTRIBUTING.md)
• Changelog entry added as changes/unreleased/<slug>.<section>.md, not as an edit to CHANGELOG.md
• baselines regenerated and make docs-counts run if a gate count moved (compliance rows need nothing: the census is counted at docs build)
• No internal work-item labels in the body, docs, or changelog

[devin-ai-integration]

🤖 Devin AI Engineer

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

• Address comments on this PR. Add '(aside)' to your comment to have me ignore it.
• Look at CI failures and help fix them

Note: I can only respond to comments from users who have write access to this repository.

⚙️ Control Options:

• Disable automatic comment, CI, and merge conflict monitoring

[devin-ai-integration]

End-to-end check of the reconciled head through the CLI and a recorded REPL session, on the -jobs interface and the nested tools/cmd/stress-model generator. Every probe passed.

Multi-file REPL and diagnostic invalidation

A %load on four jobs followed by prompt edits moves the whole-buffer diagnostics 1 → 0 → 1 → 0, so batch-computed diagnostics are dropped on edit. %print/%view/%render #table/%eval reach a second loaded file's members; %save then %clear + reload gives the same findings and values. A file's root imports stay out of the other files and the prompt (Real unresolved at the prompt; %eval A::x + 1.0 still 3.0).

Diagnostic invalidation	Loaded-symbol operations

CLI, concurrency and generator checks

• a.sysml (private import ScalarValues::*) + b.sysml (uses Real): b.sysml:1:27: error: unresolved reference: Real, exit 2, with or without a.sysml on the line.
• Two files declaring package C: no duplicate-member warning; C::X (in c1.sysml) resolves and C::Y (in c2.sysml) does not, in either argument order — document-name order, as documented.
• A four-plane split model validates identically at -jobs 1, 2, 4, 8 and OPENSYSML_JOBS=3: byte-identical stdout/stderr (stdout MD5 fe908c6ce4ae8afbd6d787f8d3f98fda).
• -jobs 0 / -jobs abc exit 2 before any file is read; -workers 2 is flag provided but not defined.
• -help and -man describe -jobs bounding how many files of one load are parsed and validated at once.
• -split-planes 4 → 2 planes removes the unchanged surplus plane002.sysml, keeps an edited plane003.sysml and a user's mine.sysml byte for byte, and the manifest matches the directory's SHA-256s; editing a still-current plane makes the next generation refuse with nothing written and leaves every file untouched.
• examples/ expressions, views, action-executor and disposal-robot models validate clean; -e "1 + 2" prints 3.