diff --git a/README.md b/README.md index 63b0e560dd..eab64788b6 100644 --- a/README.md +++ b/README.md @@ -326,7 +326,7 @@ What these numbers cannot show: the OMG corpora are demonstrations rather than a **Current commit:** All tests pass (`go test -race ./...`), builds clean (`go build ./...`). -**Test coverage:** 8,393 top-level `Test` functions (counted from the `_test.go` files, as `go test ./...` runs them) covering parsers, semantics, runtime (actions, states, instances, operators, validation). Behavioral robustness: 206 golden ASTs, 252 negatives, 940 conformance cases, 257 golden traces, 465 runtime robustness cases, 21 gRPC conformance cases and 8 gRPC robustness cases. These figures are generated by `make docs-counts` from the tree and gated. A test skips only for want of something the run did not provide, and says what: the held-image round trip declines a conformance case that creates no instance, a few gate on a PDF or Mermaid toolchain, a pinned pilot artifact, the PSSM suite, a locale, a case-insensitive filesystem or a live Flexo stack, and the OMG corpus gates skip until the corpora are downloaded unless asked to fail. +**Test coverage:** 8,419 top-level `Test` functions (counted from the `_test.go` files, as `go test ./...` runs them) covering parsers, semantics, runtime (actions, states, instances, operators, validation). Behavioral robustness: 206 golden ASTs, 252 negatives, 940 conformance cases, 257 golden traces, 465 runtime robustness cases, 21 gRPC conformance cases and 8 gRPC robustness cases. These figures are generated by `make docs-counts` from the tree and gated. A test skips only for want of something the run did not provide, and says what: the held-image round trip declines a conformance case that creates no instance, a few gate on a PDF or Mermaid toolchain, a pinned pilot artifact, the PSSM suite, a locale, a case-insensitive filesystem or a live Flexo stack, and the OMG corpus gates skip until the corpora are downloaded unless asked to fail. **Parser coverage:** 101/101 bundled library files parse cleanly — the 94 official SysML v2 standard library files and the non-normative `OpenSysML Libraries/OpenSysMLMathFunctions.kerml`, `OpenSysML Libraries/DocumentQueries.sysml`, `OpenSysML Libraries/IdentityMetadata.sysml`, `OpenSysML Libraries/DiagramLayout.sysml`, `OpenSysML Libraries/OOSEM.sysml`, `OpenSysML Libraries/MOSA.sysml` and `OpenSysML Libraries/StateSpaceIntegration.sysml` extensions. Conformance verified by [stdlib_conformance_test.go](internal/core/libs/stdlib_conformance_test.go). Grammar reference: [OMG Xtext grammar](https://github.com/Systems-Modeling/SysML-v2-Pilot-Implementation/tree/master/org.omg.kerml.xtext/src/org/omg/kerml/xtext). **Behavioral execution:** Calc/constraint/requirement/satisfy functional. Action/state executors handle nested invocation, control flow keywords, loop and conditional statements and the send statement (940/940 conformance cases passing). Coverage is self-assessed against the specification text and the normative library: the pinned OMG pilot implementation evaluates expressions but does not execute actions or state machines headlessly, so no external implementation currently adjudicates these rows. See [spec compliance](docs/project/spec-compliance.md). **Reference differential:** 377 files compared diagnostic-by-diagnostic against the pinned OMG pilot implementation (`2026-08`), 346 in full agreement; every divergence is enumerated and adjudicated in [the differential](docs/project/pilot-differential.md), reproducible with `go run ./cmd/pilot-diff`. diff --git a/changes/unreleased/persistent-semantic-model.performance.md b/changes/unreleased/persistent-semantic-model.performance.md new file mode 100644 index 0000000000..f93e04ec34 --- /dev/null +++ b/changes/unreleased/persistent-semantic-model.performance.md @@ -0,0 +1,22 @@ +- **A workspace keeps its semantic model between edits and invalidates it per document.** + `model.Workspace` owns one `resolve.Resolver` and one `semantics.Model` for its lifetime and + hands them to every analysis it runs; the resolver keeps a frame per document owning what was + memoized while that document was analyzed and records which documents it read (a namespace it + imports that another contributes to, a namespace both contribute to, a symbol of another that a + resolution returned). Replacing a document drops its frame and, transitively, its dependents' — + their memo entries, cached diagnostics and reverse references — and nothing else, where every + edit used to clear the whole workspace. The OOSEM, MOSA and identity-metadata audits and the + coherent-quantity ranking gather each document's facts once into the workspace and judge each + analyzed document over the union, where they gathered every document once per document + analyzed. `TestIncrementalEqualsFresh` replays scripted and random edit sequences over the + fixtures and the OMG corpora and compares diagnostics, resolutions and references with a fresh + workspace after every step. On the satellite-network stress test, editing a two-line file beside + 512 satellites goes from 861 ms and 327 MiB per edit to 8.7 ms and 2.0 MiB; editing the library + every file of the split network imports costs one analysis of the model (8.8 s to 5.3 s at 512 + satellites), and loading the 1 600-satellite network split into 34 files through one workspace + goes from 126 s to 18 s. A loaded workspace holds about twice the heap (254 MiB to 478 MiB at + 512 satellites), the memo tables that were allocated and discarded on every analysis, and a + thousand edits grow it by 4.5%. A one-shot `sysml -validate` pays the dependency recording it + never uses: about a sixth more wall time (1.9 s to 2.2 s at 200 satellites) and 4% more + allocation. Figures and the machine they were taken on are in `docs/internals/performance.md` + and `docs/project/satellite-network-stress-test.md`. diff --git a/docs/internals/performance.md b/docs/internals/performance.md index c863234e2b..d8434fbd32 100644 --- a/docs/internals/performance.md +++ b/docs/internals/performance.md @@ -394,6 +394,75 @@ What this says about a real workload is that the collector, not the run, is what grows: a long-lived session over a large model tunes better with `GOGC` than with a faster executor. +## What the persistent semantic model changes + +A `model.Workspace` keeps one `resolve.Resolver` and one `semantics.Model` +for its lifetime, beside its index, and hands them to every `passes.Context` +it builds; a context built outside a workspace still gets fresh ones. The +resolver keeps a frame per document that owns what was memoized while that +document was analyzed, and records which documents each frame read: a +document depends on another when it imports a namespace the other contributes +to, when both contribute to one namespace, or when a resolution from its scope +returned the other's symbol. Replacing a document drops its frame and, +transitively, its dependents' frames — their memo entries, cached diagnostics +and reverse references — and nothing else. The three workspace-wide audits +(OOSEM, MOSA, identity metadata) and the coherent-quantity ranking gather each +document's facts once into the workspace, regather a document when it changes, +and judge each analyzed document over the union. + +`TestIncrementalEqualsFresh` replays scripted and seeded random edit sequences +— edits, reverts to earlier versions, closes and opens — over the fixtures and +the four OMG corpora and, after every step, compares diagnostics, resolutions +and reverse references with a workspace built fresh from the same documents. + +Measured on the satellite-network generator (`docs/project/satellite-network-stress-test.md`, +"Editing"; Intel Xeon Platinum 8559C, 8 CPUs, 31 GiB, Go 1.25, `-benchtime=5x +-count=3` medians), rebuilt on every edit → kept: + +| measurement | rebuilt | kept | +| ----------- | ------- | ---- | +| `BenchmarkEditBeside`, 512 satellites beside a two-line file, per edit | 861 ms, 327 MiB | 8.7 ms, 2.0 MiB | +| `BenchmarkEditBeside`, 128 satellites | 189 ms, 83 MiB | 2.5 ms, 0.64 MiB | +| `BenchmarkEditBeside`, 32 satellites | 48 ms, 22 MiB | 0.82 ms, 0.31 MiB | +| `BenchmarkEditImported`, 512 satellites in 6 files, edit the library then every file's diagnostics | 8.78 s, 2.77 GiB | 5.26 s, 1.17 GiB | +| `BenchmarkLoadFiles`, 512 satellites in 6 files through one workspace | 9.20 s, 3.05 GiB | 5.26 s, 1.56 GiB | +| 1 600 satellites in 34 files through one workspace, open and analyze all | 126.5 s | 18.3 s | +| `BenchmarkLoad`, 512 satellites in one file | 5.13 s, 254 MiB held | 5.99 s, 478 MiB held | +| live heap after 1 000 edits beside 32 satellites, against after the first | — | 67.2 MB → 70.2 MB | + +Editing the library every file imports costs one analysis of the whole model, +what loading it costs; the audits no longer gather every document once per +document analyzed, which is the whole of the 34-file difference. What the +model holds between edits nearly doubles — the memo tables that were allocated +and discarded during every analysis now stay — and a thousand edits grow it by +4.5%. + +### What the bookkeeping costs a one-shot validation + +Every memoized read records that the current document depends on the owner of +the entry it read. That is what makes invalidation sound: an entry keyed by two +symbols of two documents (`composed[(S, T)]`) must go when either changes, and +the reader of a cached answer must be re-analyzed when the answer's owner is +replaced, so the dependency has to be recorded on a hit as well as on a miss. +A validation that will never edit records about 25 million such reads at +roughly 8 ns each for nothing. `sysml -validate -memstats` on the 200-satellite +constellation, one file, three runs each: + +| | rebuilt | kept | +| --- | ------- | ---- | +| wall | 1.85–1.96 s | 2.19–2.25 s | +| allocated | 738 MiB in 10.97 M allocations | 767 MiB in 10.98 M allocations | +| peak RSS (`/usr/bin/time`) | 421 MiB | 428 MiB | + +At 1 600 satellites: 17.7 s and 5.5 GiB allocated became 20.5 s and 5.8 GiB. +The cost falls on whatever analyzes through the workspace's own context: the +LSP server, a REPL session, and `sysml -validate`, which loads through a REPL +session. A batch that analyzes each document in a private `passes.Context` — +its own resolver and model over the read-only index, as a pool of workers +must — has no frames to record into and pays none of it; the workspace's +gathered facts are what such a batch should hand its workers, so that they do +not gather per worker what the workspace gathered once. + ## Notes for further work - The `about`-metadata index walks the bundled library's documents once per diff --git a/docs/project/satellite-network-stress-test.md b/docs/project/satellite-network-stress-test.md index 66ef84232f..779be5ffc2 100644 --- a/docs/project/satellite-network-stress-test.md +++ b/docs/project/satellite-network-stress-test.md @@ -86,6 +86,13 @@ validation pass, reporting nothing. *Allocated* is cumulative allocation; | 6 400 | 1 196 257 | 73.6 MB | 86 s | 22.0 GiB | 9.9 GB | | 12 800 | 2 392 417 | 147 MB | 318 s | 44.0 GiB | 20.6 GB | +These figures predate the workspace keeping its semantic model between edits, +which costs a one-shot validation the bookkeeping of what it would invalidate: +at 200 satellites 1.85–1.96 s became 2.19–2.25 s and 738 MiB allocated became +767 MiB, peak RSS 421 to 428 MiB; at 1 600 satellites 17.7 s became 20.5 s and +5.5 GiB allocated 5.8 GiB. What pays it and what does not is in +`docs/internals/performance.md`, "What the persistent semantic model changes". + Above the process floor the cost is close to linear in the model: **about 55 µs, 19 KiB allocated and 8.5 KB of peak RSS per element**, or 10–13 ms, 3.5 MiB and 1.6 MB per fully modeled satellite. Doubling the model doubles @@ -180,39 +187,77 @@ change, with the rest of the project indexed beside it. `BenchmarkEditBeside` opens the constellation as one workspace document, opens a second small file that imports it (`package Ops { private import SatelliteNetwork::Constellation::*; part spare : Sat0; }`), and measures one edit to the small file followed by its diagnostics — what the -LSP server does on `didChange`: - -| satellites in the workspace | elements | per edit of the small file | allocated per edit | -| --------------------------- | -------- | -------------------------- | ------------------ | -| 32 | 6 227 | 52 ms | 21 MiB | -| 128 | 24 167 | 263 ms | 79 MiB | -| 512 | 95 927 | 1.53 s | 311 MiB | - -**The cost of editing a two-line file grows linearly with the size of the -model it sits beside**: about 16 µs per element in the workspace, per -keystroke. The reasons are structural, not incidental: - -- `model.(*Workspace).invalidateLocked` drops every cached diagnostic and - the reverse-reference index on any change, on the correct grounds that a - change anywhere can alter what a name elsewhere resolves to. The next - diagnostics request re-analyzes from a fresh semantic model with cold - memoization. -- Several passes are workspace-wide by design: a CPU profile of the 128- - satellite case spends 24% in the OOSEM method audit (`OOSEMMethodPass`), - which gathers the kind of every symbol in every workspace document to - check derivation and satisfaction relationships across files, 10% in the - inherited-name conflict pass and 3% in the MOSA audit; between them they - compute `semantics.(*Model).FeatureTypeSet` over the whole constellation - to analyze a file that declares one part. Name resolution of the small - file's own imports is 16%. - -The interactive limit is therefore set by the workspace, not the file being -edited. With one large model file beside the one being typed in, 100–200 of -these satellites (20 000–40 000 elements) keep a keystroke under 250–500 ms; -at 500 satellites every keystroke costs 1.5 s and the editor is no longer -usable. A project that splits the constellation over many files pays the -same per-file analysis for each open file it publishes diagnostics for, so a -workspace with `n` open files costs about `n` times the figures above. +LSP server does on `didChange`. The workspace now keeps one semantic model +across edits and invalidates it per document (`docs/internals/performance.md`, +"What the persistent semantic model changes"); *rebuilt* is the model rebuilt +from cold memoization on every edit, measured on the same machine, *kept* is +the persistent one (`-benchtime=5x -count=3`, medians): + +| satellites in the workspace | elements | per edit, rebuilt | allocated, rebuilt | per edit, kept | allocated, kept | +| --------------------------- | -------- | ----------------- | ------------------ | -------------- | --------------- | +| 32 | 6 227 | 48 ms | 22 MiB | 0.82 ms | 0.31 MiB | +| 128 | 24 167 | 189 ms | 83 MiB | 2.5 ms | 0.64 MiB | +| 512 | 95 927 | 861 ms | 327 MiB | 8.7 ms | 2.0 MiB | + +Rebuilding, **the cost of editing a two-line file grew linearly with the size +of the model it sat beside**: about 9 µs per element in the workspace, per +keystroke (an earlier revision of this record measured 16 µs; resolution got +cheaper in between). The reasons were structural: `invalidateLocked` dropped +every cached diagnostic and the reverse-reference index on any change, and the +next request re-analyzed from a fresh semantic model; and the OOSEM, MOSA and +identity audits gathered the kind of every symbol in every workspace document +to check the one file's relationships. + +Kept, a keystroke costs what the two-line file costs plus what it reads of the +constellation, and grows a hundred times more slowly with the model: 512 +satellites beside the file cost 8.7 ms and 2 MiB, a hundredth of the rebuilt +figures. The edit invalidates the small document only; the constellation's +frame in the resolver, its memoized semantics and its gathered facts stay. + +### Editing a file the others import + +The worst edit is to a document everything else depends on. `Split` writes the +same network as one document per plane beside the library they build on and +the constellation joining them (six files at these sizes); `BenchmarkLoadFiles` +opens and analyzes every file through one workspace, and `BenchmarkEditImported` +edits the library and then asks every file for its diagnostics, as the editor's +refresh sweep does: + +| satellites | files | load all files, rebuilt | load all files, kept | edit the library, rebuilt | edit the library, kept | +| ---------- | ----- | ----------------------- | -------------------- | ------------------------- | ---------------------- | +| 32 | 6 | 0.55 s / 228 MiB | 0.34 s / 129 MiB | 0.63 s / 213 MiB | 0.43 s / 108 MiB | +| 128 | 6 | 2.11 s / 805 MiB | 1.27 s / 420 MiB | 2.13 s / 735 MiB | 1.31 s / 324 MiB | +| 512 | 6 | 9.20 s / 3.05 GiB | 5.26 s / 1.56 GiB | 8.78 s / 2.77 GiB | 5.26 s / 1.17 GiB | + +Editing the library invalidates every document, since each imports it, so the +edit costs one analysis of the whole model — the same 5.26 s loading the six +files costs. It cost 1.7× that rebuilt, because every file's analysis +re-gathered every other file for the audits. Loading the split model kept costs +what the single-file model costs (5.99 s for 512 satellites, below); rebuilt it +cost 1.8× as much, and the ratio grew with the file count. Over the +1 600-satellite network split into 34 files (297 429 elements), opening every +file and asking each for its diagnostics through one workspace took 126.5 s +rebuilt and 18.3 s kept, against 17.7 s for the same model as one file; the +difference is the three audits, which gathered all 34 documents once per +document analyzed and now gather each once. + +### What the model holds between edits + +Keeping the semantic model means keeping its memo tables: the supertype +closures, redefinition closures, masks, resolved parts and identities the +analysis computed. `BenchmarkLoad` reports the heap a loaded session holds per +element, and it rises from about 2.7 KiB to about 5.0 KiB — 254 MiB to 478 MiB +for 512 satellites (19.4 to 34.7 MiB at 32, 66 to 123 MiB at 128). The same +bytes were allocated and discarded during every analysis before; now they stay, +which is what makes the next edit cheap. Editing does not let them grow: +`TestEditsHoldNoStaleState` edits the small file a thousand times beside the +32-satellite network, editing the network itself every fiftieth time, and the +live heap goes from 67.2 MB after the first edit to 70.2 MB after the +thousandth — the journals drop what a replaced document owned. + +The interactive limit is therefore no longer set by the size of the workspace +a small file sits beside; it is set by the size of the document being edited, +and by the documents that import it when that document is a library. ## Where the time goes @@ -256,14 +301,16 @@ is what exposes them: | use | comfortable | slow | impractical | bound by | | --- | ----------- | ---- | ----------- | -------- | -| editing with the model open beside the file | ≤ 100 satellites (20 000 elements, ≤ 250 ms per keystroke) | 200–400 satellites (0.5–1.2 s) | ≥ 500 satellites (≥ 1.5 s per keystroke) | whole-workspace re-analysis per edit | +| editing a small file with the model open beside it | ≤ 512 satellites (96 000 elements, ≤ 9 ms per keystroke; the largest size measured) | — | — | the size of the document edited and of the documents importing it, not of the workspace | +| editing the library every file of a split model imports | ≤ 32 satellites (0.43 s per edit) | 128 satellites (1.3 s) | ≥ 512 satellites (5.3 s) | one analysis of every dependent document | | a REPL or gRPC session holding the model | ≤ 500 satellites (≤ 5 s to load, 250 MiB held) | 1 000–2 000 satellites (10–25 s to load, 0.5–1 GiB held) | limited by load time, not by memory, until the tens of thousands | load wall time; ~490 KiB held per satellite | | `sysml -validate` in a build or CI | ≤ 800 satellites (≤ 10 s, 1.4 GB) | 1 600–6 400 satellites (20–90 s, 2.7–10 GB) | 12 800 satellites at 5 min and 21 GB; 25 600 would not fit in 31 GiB | peak RSS, 8.5 KB per element | | `sysml -satisfy` over every assertion | ≤ 400 satellites (≤ 8 s, 1 GB) | 800–1 600 satellites (18–38 s, 2–4 GB) | 3 200 satellites at 83 s and 7.6 GB; memory runs out about half as far as validation | 2× the validation cost | For the question as asked — a satellite network with every component modeled -— the practical ceilings on this machine are **a few hundred satellites for -interactive editing, a few thousand for batch validation and checking, and +— the practical ceilings on this machine are **the largest model measured for +editing a file beside it, a few hundred satellites of dependents for editing +a file they all import, a few thousand for batch validation and checking, and about ten thousand (two million elements) before a 31 GiB machine cannot hold a validation**. A small constellation of a dozen satellites is well inside the interactive band at every operation measured. @@ -277,10 +324,11 @@ the interactive band at every operation measured. usage cost more per element, long documentation comments cost less — but the shape of the curve (linear load, memory-bound batch, workspace-bound editing) does not depend on the regularity. -- The whole constellation is one file. Splitting it over files changes two - things: the CLI submits files one at a time and reindexes after each, which - is quadratic in the file count (`docs/internals/performance.md`, notes for - further work), and an editor pays the per-file analysis once per open file. +- The whole constellation is one file except where a figure says it was + split. Splitting it over files changes two things: the CLI submits files one + at a time and reindexes after each, which is quadratic in the file count + (`docs/internals/performance.md`, notes for further work), and an editor pays + the per-file analysis once per open file. - `-satisfy` instantiates each satellite's tree on its own; it does not instantiate the whole `Network` as one object with 12 800 satellites and their links, and no figure here says what that would cost. @@ -298,17 +346,18 @@ validation, and one definition with many occurrences — is in [scaling to very large models](large-model-scaling-design.md). The three items below are the ones the profiles point at directly. -- **Scope the workspace-wide passes.** The OOSEM and MOSA audits and the - inherited-name conflict pass are what make one keystroke cost a - whole-workspace walk. The audits need the whole workspace only when some - document declares an artefact of the method's kinds; whether one does is - computable once per reindex and cached, and a workspace that declares none - would then pay nothing. That alone removes a quarter of the per-edit cost. -- **Keep the semantic model across edits.** Every diagnostics request after an - edit starts from cold memoization. Invalidating what a change can reach — - the documents that import the changed one, transitively — rather than - everything would let a keystroke in a small file beside a large model cost - what the small file costs. +- **Hand the gathered facts to the batch pipeline.** The OOSEM, MOSA and + identity audits now gather each workspace document once and judge each + analyzed document over the union, which is what made the 34-file split load + in 18 s rather than 126 s through one workspace. A batch that analyzes + documents on parallel workers with private contexts gathers per worker + again unless the workspace's gathers are what the batch hands them. +- **Make a one-shot validation skip the bookkeeping.** The persistent model + records, on every memoized read, which document depends on the entry's + owner, so that the owner's replacement invalidates the reader. A validation + that will never edit pays that for nothing — about a sixth of its wall time + at 200 satellites. Analyzing each document in a private context over the + read-only index, as a parallel batch does, records nothing. - **Reduce allocation per element.** Nineteen KiB allocated per element against 2.7 KiB held means a load produces seven times its own weight in garbage, and the collector's quarter of the profile is the price. The diff --git a/docs/project/spec-compliance.md b/docs/project/spec-compliance.md index 64e699b724..9b62df873e 100644 --- a/docs/project/spec-compliance.md +++ b/docs/project/spec-compliance.md @@ -133,7 +133,7 @@ what cannot be checked by anything is in - Golden traces: 257 golden execution traces under the default schedule (state×108, action×74, calc×32, clock×6, extent×6, constraint×4, string×4, three each of accept and analysis, two each of exhibited, f63 and verification, and one each of assign, f62, function, meta, object, occurrence, performed, send, two, w6e and w7d), and 54 more `.trace.golden` files pinning a case under a named policy, `.declared` or `.seed-` — entry/do/exit ordering of inline action bodies and a do body run to its end inside one round, the standard loop `until` with `then done`, a decision's guarded and `else` branches, a named flow carrying a value between action nodes, an accept with a `when` trigger, an accept subsetting an event, a send invocation through a port, a transition accepting through a port, loop and conditional bodies, one calc usage body run feeding several output reads, a usage whose outputs are read either side of an assignment to what its input named, a usage nested in a calc read for two of its outputs, calc statement bodies and their loop iterations, fork/join branch ordering, region entry/exit ordering, do behavior interleaving across orthogonal regions, send/accept, an accept parked until its message arrives, a payload read by a node declared before the accept that binds it, calc and constraint evaluation, library function invocation, the dotted-target transition, control-node and merge-body traces, and the merge loops re-entered on every pass) - Negative parser tests: 252 negative parser subtests (first-level subtests of `TestNegative`; 396 across the `TestNegative*` functions, 60 of them KerML, and 454 across every `*Negative*` parser test) - gRPC: 21 gRPC conformance cases and 8 gRPC robustness cases (`internal/grpc/testdata/conformance/`, `internal/grpc/robustness_test.go`) -- Test functions: 8,393 top-level `Test` functions across the module (`go test -count=1 ./...` runs them all, with the OMG corpora downloaded, `OPENSYSML_REQUIRE_TRAINING_CORPUS=1 OPENSYSML_REQUIRE_PILOT_CORPORA=1 OPENSYSML_REQUIRE_SMT=1` and z3 installed). The figures on this list are generated by `make docs-counts` from the tree and gated; the test and subtest total of a run is not, since it moves with every fixture and only a run can state it. A test skips only where it says why: TestHeldImageRoundTrip declines a conformance case that creates no instance, so there is no held image to round-trip. Three skip themselves: TestSubsettingTargetIsTheInheritedFeature and TestRequirementEvaluation_SubjectNotFound against a limitation they record, and TestHelperSolverProcess, which is a solver child process the parent invokes. The others skip for want of something the run did not provide, and each names it: the `weasyprint`, `pandoc` and `prince` subtests of TestRenderWithInstalledEngines and TestRenderInlineRunsWithInstalledEngines and TestRenderDiagramsWithInstalledMermaid want the PDF and Mermaid toolchain, TestExtractionMatchesBaseline and TestUpdateIsIdempotentAcrossDays the pinned pilot validator jar, TestEmitSuite, TestRefereeRowsAreWellFormed, TestSuiteRead and TestSuiteClassification the downloaded PSSM test suite, TestCRealNotationIsLocaleIndependent a non-C locale, TestRenderDocumentsRejectsCaseAliasedTargets a case-insensitive filesystem, and TestFlexoInterop and TestFlexoInteropApply a live Flexo stack. +- Test functions: 8,419 top-level `Test` functions across the module (`go test -count=1 ./...` runs them all, with the OMG corpora downloaded, `OPENSYSML_REQUIRE_TRAINING_CORPUS=1 OPENSYSML_REQUIRE_PILOT_CORPORA=1 OPENSYSML_REQUIRE_SMT=1` and z3 installed). The figures on this list are generated by `make docs-counts` from the tree and gated; the test and subtest total of a run is not, since it moves with every fixture and only a run can state it. A test skips only where it says why: TestHeldImageRoundTrip declines a conformance case that creates no instance, so there is no held image to round-trip. Three skip themselves: TestSubsettingTargetIsTheInheritedFeature and TestRequirementEvaluation_SubjectNotFound against a limitation they record, and TestHelperSolverProcess, which is a solver child process the parent invokes. The others skip for want of something the run did not provide, and each names it: the `weasyprint`, `pandoc` and `prince` subtests of TestRenderWithInstalledEngines and TestRenderInlineRunsWithInstalledEngines and TestRenderDiagramsWithInstalledMermaid want the PDF and Mermaid toolchain, TestExtractionMatchesBaseline and TestUpdateIsIdempotentAcrossDays the pinned pilot validator jar, TestEmitSuite, TestRefereeRowsAreWellFormed, TestSuiteRead and TestSuiteClassification the downloaded PSSM test suite, TestCRealNotationIsLocaleIndependent a non-C locale, TestRenderDocumentsRejectsCaseAliasedTargets a case-insensitive filesystem, and TestFlexoInterop and TestFlexoInteropApply a live Flexo stack. --- diff --git a/internal/core/model/docquery.go b/internal/core/model/docquery.go index 961efbb7ff..10b1418212 100644 --- a/internal/core/model/docquery.go +++ b/internal/core/model/docquery.go @@ -10,6 +10,8 @@ import ( "github.com/Open-MBEE/OpenSysML/internal/core/docrender" "github.com/Open-MBEE/OpenSysML/internal/core/queryexec" "github.com/Open-MBEE/OpenSysML/internal/core/queryplan" + "github.com/Open-MBEE/OpenSysML/internal/core/resolve" + "github.com/Open-MBEE/OpenSysML/internal/core/semantics" "github.com/Open-MBEE/OpenSysML/internal/core/symbols" ) @@ -23,15 +25,16 @@ type DocumentDefinition struct { // DocumentDefinitions lists the document definitions declared across the // workspace's own documents, in qualified-name order. func (w *Workspace) DocumentDefinitions() []DocumentDefinition { - w.mu.RLock() - defer w.mu.RUnlock() - _, sem := w.newResolver() + w.mu.Lock() + defer w.mu.Unlock() out := []DocumentDefinition{} for name := range w.docs { - walkScope(w.index.DocumentRoot(name), func(sym *symbols.Symbol) { - if docplan.IsDocumentDefinition(w.index, sem, sym) { - out = append(out, DocumentDefinition{FQN: notationFQN(w.index, sym), Doc: name}) - } + w.queryLocked(name, func(_ *resolve.Resolver, sem *semantics.Model) { + walkScope(w.index.DocumentRoot(name), func(sym *symbols.Symbol) { + if docplan.IsDocumentDefinition(w.index, sem, sym) { + out = append(out, DocumentDefinition{FQN: notationFQN(w.index, sym), Doc: name}) + } + }) }) } sort.Slice(out, func(i, j int) bool { return out[i].FQN < out[j].FQN }) @@ -41,9 +44,8 @@ func (w *Workspace) DocumentDefinitions() []DocumentDefinition { // RenderDocumentMarkdown compiles the named document definition, evaluates its // queries against the workspace model, and renders the result as Markdown. func (w *Workspace) RenderDocumentMarkdown(fqn string, opts docrender.MarkdownOptions) (string, error) { - w.mu.RLock() - defer w.mu.RUnlock() - resolver, sem := w.newResolver() + w.mu.Lock() + defer w.mu.Unlock() matches := symbols.PreferDeclared(w.index.LookupQualified(fqn)) if len(matches) == 0 { return "", fmt.Errorf("no element named %s", fqn) @@ -52,21 +54,28 @@ func (w *Workspace) RenderDocumentMarkdown(fqn string, opts docrender.MarkdownOp return "", fmt.Errorf("%s names %d elements; rename one so the name is unambiguous", fqn, len(matches)) } sym := matches[0] - if !docplan.IsDocumentDefinition(w.index, sem, sym) { - return "", fmt.Errorf("%s is not a document: one is a part def specializing DocumentQueries::Document", fqn) - } - plan, err := docplan.Compile(w.index, sem, resolver, sym) - if err != nil { - return "", err - } - document, err := docir.EvaluateLinked(plan, - SiblingDocumentPlans(w.index, sem, resolver, sym), - queryexec.Context{Index: w.index, Resolver: resolver, Model: sem}, - queryexec.Options{}, w.sourceTextLocked()) - if err != nil { - return "", err - } - return docrender.Markdown(document, opts) + var out string + var err error + w.queryLocked(sym.DocName, func(resolver *resolve.Resolver, sem *semantics.Model) { + if !docplan.IsDocumentDefinition(w.index, sem, sym) { + err = fmt.Errorf("%s is not a document: one is a part def specializing DocumentQueries::Document", fqn) + return + } + var plan *docplan.Plan + if plan, err = docplan.Compile(w.index, sem, resolver, sym); err != nil { + return + } + var document *docir.Document + document, err = docir.EvaluateLinked(plan, + SiblingDocumentPlans(w.index, sem, resolver, sym), + queryexec.Context{Index: w.index, Resolver: resolver, Model: sem}, + queryexec.Options{}, w.sourceText()) + if err != nil { + return + } + out, err = docrender.Markdown(document, opts) + }) + return out, err } // QueryBindingParameter resolves the parameter a document query binding names: @@ -80,57 +89,72 @@ func (w *Workspace) QueryBindingParameter(sym *symbols.Symbol) (*symbols.Symbol, if !ok || decl.Direction != ast.DirIn { return nil, false } - w.mu.RLock() - defer w.mu.RUnlock() - resolver, sem := w.newResolver() - target := docplan.QueryTarget(w.index, sem, resolver, sym.OwnerScope.Owner()) - if target == nil { - return nil, false - } - for _, member := range w.memberSymbolsLocked(resolver, sem, sym.OwnerScope, target) { - if member == nil || member.Name != sym.Name { - continue + w.mu.Lock() + defer w.mu.Unlock() + var out *symbols.Symbol + w.queryLocked(sym.DocName, func(resolver *resolve.Resolver, sem *semantics.Model) { + target := docplan.QueryTarget(w.index, sem, resolver, sym.OwnerScope.Owner()) + if target == nil { + return } - if md, ok := member.Decl.(*ast.Usage); ok && md.Direction == ast.DirIn { - return member, true + for _, member := range w.memberSymbolsLocked(resolver, sem, sym.OwnerScope, target) { + if member == nil || member.Name != sym.Name { + continue + } + if md, ok := member.Decl.(*ast.Usage); ok && md.Direction == ast.DirIn { + out = member + return + } } - } - return nil, false + }) + return out, out != nil } // QueryUsageParameters lists the `in` parameters of the query definition a calc // usage is typed by; false when the usage is not typed by one. func (w *Workspace) QueryUsageParameters(usage *symbols.Symbol) ([]*symbols.Symbol, bool) { - w.mu.RLock() - defer w.mu.RUnlock() - resolver, sem := w.newResolver() - target := docplan.QueryTarget(w.index, sem, resolver, usage) - if target == nil { + if usage == nil { return nil, false } - scope := usage.OwnerScope - if usage.Scope != nil { - scope = usage.Scope - } + w.mu.Lock() + defer w.mu.Unlock() var out []*symbols.Symbol - for _, member := range w.memberSymbolsLocked(resolver, sem, scope, target) { - if member == nil { - continue + typed := false + w.queryLocked(usage.DocName, func(resolver *resolve.Resolver, sem *semantics.Model) { + target := docplan.QueryTarget(w.index, sem, resolver, usage) + if target == nil { + return } - if md, ok := member.Decl.(*ast.Usage); ok && md.Direction == ast.DirIn { - out = append(out, member) + typed = true + scope := usage.OwnerScope + if usage.Scope != nil { + scope = usage.Scope } - } - return out, true + for _, member := range w.memberSymbolsLocked(resolver, sem, scope, target) { + if member == nil { + continue + } + if md, ok := member.Decl.(*ast.Usage); ok && md.Direction == ast.DirIn { + out = append(out, member) + } + } + }) + return out, typed } // IsDocumentDefinition reports whether sym is a native document definition: a // part def specializing DocumentQueries::Document. func (w *Workspace) IsDocumentDefinition(sym *symbols.Symbol) bool { - w.mu.RLock() - defer w.mu.RUnlock() - _, sem := w.newResolver() - return docplan.IsDocumentDefinition(w.index, sem, sym) + if sym == nil { + return false + } + w.mu.Lock() + defer w.mu.Unlock() + var out bool + w.queryLocked(sym.DocName, func(_ *resolve.Resolver, sem *semantics.Model) { + out = docplan.IsDocumentDefinition(w.index, sem, sym) + }) + return out } // QueryTypeCandidate pairs a visible spelling with the element it reaches, so @@ -168,14 +192,18 @@ func (w *Workspace) QueryTypeCandidates(scope *symbols.Scope) []QueryTypeCandida // QueryDefinitions filters syms to the query definitions among them: the calc // defs specializing DocumentQueries::Query. func (w *Workspace) QueryDefinitions(syms []*symbols.Symbol) []*symbols.Symbol { - w.mu.RLock() - defer w.mu.RUnlock() - _, sem := w.newResolver() + w.mu.Lock() + defer w.mu.Unlock() var out []*symbols.Symbol for _, sym := range syms { - if queryplan.IsQueryDefinition(w.index, sem, sym) { - out = append(out, sym) + if sym == nil { + continue } + w.queryLocked(sym.DocName, func(_ *resolve.Resolver, sem *semantics.Model) { + if queryplan.IsQueryDefinition(w.index, sem, sym) { + out = append(out, sym) + } + }) } return out } diff --git a/internal/core/model/highlight.go b/internal/core/model/highlight.go index 5f474a5776..5891648072 100644 --- a/internal/core/model/highlight.go +++ b/internal/core/model/highlight.go @@ -3,6 +3,7 @@ package model import ( "github.com/Open-MBEE/OpenSysML/internal/core/highlight" "github.com/Open-MBEE/OpenSysML/internal/core/resolve" + "github.com/Open-MBEE/OpenSysML/internal/core/semantics" "github.com/Open-MBEE/OpenSysML/internal/core/symbols" ) @@ -17,10 +18,13 @@ func (w *Workspace) HighlightTokens(name string) []highlight.Token { if doc == nil { return nil } - w.mu.RLock() - defer w.mu.RUnlock() - resolver, _ := w.newResolver() - return highlight.Tokens(doc.Content, doc.AST, doc.Scope, resolution{r: resolver}) + w.mu.Lock() + defer w.mu.Unlock() + var out []highlight.Token + w.queryLocked(name, func(resolver *resolve.Resolver, _ *semantics.Model) { + out = highlight.Tokens(doc.Content, doc.AST, doc.Scope, resolution{r: resolver}) + }) + return out } // resolution answers highlighting queries from one resolver, so the memoized diff --git a/internal/core/model/identity.go b/internal/core/model/identity.go index a094239410..9db8f4798e 100644 --- a/internal/core/model/identity.go +++ b/internal/core/model/identity.go @@ -2,6 +2,8 @@ package model import ( "github.com/Open-MBEE/OpenSysML/internal/core/identity" + "github.com/Open-MBEE/OpenSysML/internal/core/resolve" + "github.com/Open-MBEE/OpenSysML/internal/core/semantics" "github.com/Open-MBEE/OpenSysML/internal/core/symbols" ) @@ -12,8 +14,8 @@ func (w *Workspace) IdentityOf(name string, sym *symbols.Symbol) (*identity.Info if sym == nil || sym.Decl == nil { return nil, false } - w.mu.RLock() - defer w.mu.RUnlock() + w.mu.Lock() + defer w.mu.Unlock() // The annotation model is keyed by the index's symbol for the same AST node. // A library document is parsed afresh from the bytes the index read, so its // symbols share the index's spans rather than its nodes. @@ -31,6 +33,10 @@ func (w *Workspace) IdentityOf(name string, sym *symbols.Symbol) (*identity.Info if indexed == nil { return nil, false } - resolver, sem := w.newResolver() - return identity.Of(sem, resolver, indexed) + var info *identity.Info + var ok bool + w.queryLocked(name, func(resolver *resolve.Resolver, sem *semantics.Model) { + info, ok = identity.Of(sem, resolver, indexed) + }) + return info, ok } diff --git a/internal/core/model/incremental_test.go b/internal/core/model/incremental_test.go new file mode 100644 index 0000000000..d29f514937 --- /dev/null +++ b/internal/core/model/incremental_test.go @@ -0,0 +1,478 @@ +package model_test + +import ( + "fmt" + "math/rand" + "os" + "path/filepath" + "regexp" + "sort" + "strings" + "testing" + + "github.com/Open-MBEE/OpenSysML/internal/core/model" + "github.com/Open-MBEE/OpenSysML/internal/core/resolve" + "github.com/Open-MBEE/OpenSysML/internal/core/source" + "github.com/Open-MBEE/OpenSysML/internal/core/symbols" + "github.com/Open-MBEE/OpenSysML/internal/stressmodel" +) + +// A workspace edited incrementally (scripted and seeded random edits, reverts, closes, +// opens) must answer as one built fresh: same diagnostics, resolutions and references. + +// replayDoc is one document of a replay: the versions it has had, and which one +// is open, if any. +type replayDoc struct { + name string + versions [][]byte + open bool + current int +} + +// replay drives an incremental workspace and checks it against a fresh one. +type replay struct { + t *testing.T + ws *model.Workspace + docs []*replayDoc + rng *rand.Rand + steps []string + next int // version counter handed to Open/Update + extra int // numbers the declarations mutations insert + + checkHook func(*replay) +} + +func freshOf(r *replay) *model.Workspace { + fresh := model.NewWorkspace() + for _, d := range r.openDocs() { + fresh.Open(d.name, d.versions[d.current], 1) + } + return fresh +} + +func newReplay(t *testing.T, files map[string][]byte, seed int64) *replay { + names := make([]string, 0, len(files)) + for name := range files { + names = append(names, name) + } + sort.Strings(names) + r := &replay{t: t, ws: model.NewWorkspace(), rng: rand.New(rand.NewSource(seed))} + for _, name := range names { + r.docs = append(r.docs, &replayDoc{name: name, versions: [][]byte{files[name]}}) + } + for _, d := range r.docs { + r.open(d, 0) + } + r.check("initial open") + return r +} + +func (r *replay) open(d *replayDoc, version int) { + r.next++ + d.open, d.current = true, version + r.ws.Open(d.name, d.versions[version], r.next) +} + +func (r *replay) update(d *replayDoc, version int) { + r.next++ + d.current = version + r.ws.Update(d.name, d.versions[version], r.next) +} + +func (r *replay) close(d *replayDoc) { + d.open = false + r.ws.Close(d.name) +} + +func (r *replay) openDocs() []*replayDoc { + var out []*replayDoc + for _, d := range r.docs { + if d.open { + out = append(out, d) + } + } + return out +} + +func (r *replay) closedDocs() []*replayDoc { + var out []*replayDoc + for _, d := range r.docs { + if !d.open { + out = append(out, d) + } + } + return out +} + +// step performs one operation, logging it for the failure report. +func (r *replay) step(desc string, op func()) { + r.steps = append(r.steps, desc) + op() + r.check(desc) +} + +// scripted edits every document once, reverts it, then closes and reopens one. +func (r *replay) scripted() { + for _, d := range r.docs { + r.step("mutate "+d.name, func() { r.mutate(d, 0) }) + r.step("revert "+d.name, func() { r.update(d, 0) }) + } + if len(r.docs) > 1 { + last := r.docs[len(r.docs)-1] + r.step("close "+last.name, func() { r.close(last) }) + r.step("reopen "+last.name, func() { r.open(last, 0) }) + } +} + +// randomized performs n seeded operations: edits to a mutation or to an earlier +// version, closes and opens. +func (r *replay) randomized(n int) { + for i := 0; i < n; i++ { + open, closed := r.openDocs(), r.closedDocs() + switch k := r.rng.Intn(10); { + case k < 6 && len(open) > 0: + d := open[r.rng.Intn(len(open))] + r.step("mutate "+d.name, func() { r.mutate(d, r.rng.Intn(4)) }) + case k < 7 && len(open) > 0: + d := open[r.rng.Intn(len(open))] + v := r.rng.Intn(len(d.versions)) + r.step(fmt.Sprintf("edit %s to version %d", d.name, v), func() { r.update(d, v) }) + case k < 8 && len(open) > 1: + d := open[r.rng.Intn(len(open))] + r.step("close "+d.name, func() { r.close(d) }) + case len(closed) > 0: + d := closed[r.rng.Intn(len(closed))] + v := r.rng.Intn(len(d.versions)) + r.step(fmt.Sprintf("open %s at version %d", d.name, v), func() { r.open(d, v) }) + default: + if len(open) > 0 { + d := open[r.rng.Intn(len(open))] + r.step("mutate "+d.name, func() { r.mutate(d, r.rng.Intn(4)) }) + } + } + } +} + +var identifierRE = regexp.MustCompile(`\b[A-Za-z_][A-Za-z0-9_]*\b`) +var packageRE = regexp.MustCompile(`\bpackage\s+([A-Za-z_][A-Za-z0-9_]*)`) + +// mutate derives a new version of d from its current one and installs it: +// a deleted line, a renamed identifier, a declaration or an import inserted. +func (r *replay) mutate(d *replayDoc, kind int) { + src := string(d.versions[d.current]) + var out string + switch kind { + case 0: + lines := strings.Split(src, "\n") + var candidates []int + for i, line := range lines { + if strings.TrimSpace(line) != "" { + candidates = append(candidates, i) + } + } + if len(candidates) == 0 { + return + } + i := candidates[r.rng.Intn(len(candidates))] + out = strings.Join(append(lines[:i:i], lines[i+1:]...), "\n") + case 1: + locs := identifierRE.FindAllStringIndex(src, -1) + if len(locs) == 0 { + return + } + loc := locs[r.rng.Intn(len(locs))] + out = src[:loc[1]] + "_x" + src[loc[1]:] + case 2: + i := strings.IndexByte(src, '{') + if i < 0 { + return + } + braces := indexesOf(src, '{') + i = braces[r.rng.Intn(len(braces))] + r.extra++ + decl := fmt.Sprintf(" part def Extra%d; ", r.extra) + if source.KindOf(d.name) == source.KindKerML { + decl = fmt.Sprintf(" class Extra%d; ", r.extra) + } + out = src[:i+1] + decl + src[i+1:] + default: + braces := indexesOf(src, '{') + if len(braces) == 0 { + return + } + pkgs := r.packageNames() + if len(pkgs) == 0 { + return + } + i := braces[r.rng.Intn(len(braces))] + out = src[:i+1] + " import " + pkgs[r.rng.Intn(len(pkgs))] + "::*; " + src[i+1:] + } + d.versions = append(d.versions, []byte(out)) + r.update(d, len(d.versions)-1) +} + +func indexesOf(s string, c byte) []int { + var out []int + for i := 0; i < len(s); i++ { + if s[i] == c { + out = append(out, i) + } + } + return out +} + +// packageNames lists the packages the documents declare, so an inserted import +// can reach across documents. +func (r *replay) packageNames() []string { + seen := map[string]bool{} + var out []string + for _, d := range r.docs { + for _, m := range packageRE.FindAllStringSubmatch(string(d.versions[d.current]), -1) { + if !seen[m[1]] { + seen[m[1]] = true + out = append(out, m[1]) + } + } + } + sort.Strings(out) + return out +} + +// check compares the incremental workspace with one built fresh from the open +// documents' current versions. +func (r *replay) check(after string) { + r.t.Helper() + fresh := freshOf(r) + for _, d := range r.openDocs() { + got, want := workspaceAnswers(r.ws, d.name), workspaceAnswers(fresh, d.name) + if diff := firstDifference(got, want); diff != "" { + if r.checkHook != nil { + r.checkHook(r) + } + r.t.Fatalf("after %q (steps: %s)\n%s: incremental differs from fresh:\n%s", + after, strings.Join(r.steps, "; "), d.name, diff) + } + } +} + +// workspaceAnswers renders everything the workspace says about a document: +// its diagnostics, what each written reference resolves to, and the reverse +// references of each declared element. +func workspaceAnswers(ws *model.Workspace, name string) []string { + var out []string + for _, d := range ws.Diagnostics(name) { + out = append(out, fmt.Sprintf("diag %d+%d %s %s/%s: %s", + d.Span.Offset, d.Span.Len, d.Severity, d.Source, d.Code, d.Message)) + } + doc := ws.Document(name) + if doc == nil { + return append(out, "no document") + } + for _, ref := range resolve.References(doc.AST, doc.Scope) { + if ref.QN == nil || len(ref.QN.Parts) == 0 { + continue + } + sym, ok := ws.ResolveReferenceInDoc(name, ref) + line := fmt.Sprintf("ref %d+%d -> %v %s", ref.QN.Span().Offset, ref.QN.Span().Len, ok, symbolID(sym)) + for _, seg := range ws.ResolveReferenceSegmentsInDoc(name, ref) { + line += " " + symbolID(seg) + } + out = append(out, line) + } + var syms []*symbols.Symbol + walkSymbols(doc.Scope, func(sym *symbols.Symbol) { syms = append(syms, sym) }) + for _, sym := range syms { + locs := ws.ReferencesTo(sym) + if len(locs) == 0 { + continue + } + parts := make([]string, 0, len(locs)) + for _, loc := range locs { + parts = append(parts, fmt.Sprintf("%s:%d+%d", loc.Doc, loc.Span.Offset, loc.Span.Len)) + } + sort.Strings(parts) + out = append(out, "refs-to "+symbolID(sym)+" <- "+strings.Join(parts, " ")) + } + return out +} + +func symbolID(sym *symbols.Symbol) string { + if sym == nil { + return "" + } + return fmt.Sprintf("%s@%s:%d+%d", sym.Name, sym.DocName, sym.DeclSpan.Offset, sym.DeclSpan.Len) +} + +func walkSymbols(scope *symbols.Scope, visit func(*symbols.Symbol)) { + if scope == nil { + return + } + scope.ForEachMember(func(sym *symbols.Symbol) bool { + visit(sym) + return true + }) + for _, child := range scope.Children() { + walkSymbols(child, visit) + } +} + +func firstDifference(got, want []string) string { + for i := 0; i < len(got) || i < len(want); i++ { + var g, w string + if i < len(got) { + g = got[i] + } + if i < len(want) { + w = want[i] + } + if g != w { + return fmt.Sprintf("line %d\n incremental: %s\n fresh: %s", i, g, w) + } + } + return "" +} + +// fixtureSets are the repository's own multi-file models: each example +// directory, the loose example files together, and each testdata directory. +func fixtureSets(t *testing.T) map[string]map[string][]byte { + t.Helper() + sets := map[string]map[string][]byte{} + add := func(set, name, path string) { + content, err := os.ReadFile(path) + if err != nil { + t.Fatal(err) + } + if sets[set] == nil { + sets[set] = map[string][]byte{} + } + sets[set][name] = content + } + entries, err := os.ReadDir("../../../examples") + if err != nil { + t.Fatal(err) + } + for _, e := range entries { + path := filepath.Join("../../../examples", e.Name()) + switch { + case e.Name() == "sysml-v2-training" || e.Name() == "pilot-corpora": + continue + case e.IsDir(): + for _, f := range modelFiles(t, path) { + add("examples/"+e.Name(), filepath.ToSlash(f), filepath.Join(path, f)) + } + case model.IsModelSource(e.Name()): + add("examples", e.Name(), path) + } + } + dirs, err := os.ReadDir("../../../testdata") + if err != nil { + t.Fatal(err) + } + for _, e := range dirs { + if !e.IsDir() { + continue + } + path := filepath.Join("../../../testdata", e.Name()) + for _, f := range modelFiles(t, path) { + add("testdata/"+e.Name(), filepath.ToSlash(f), filepath.Join(path, f)) + } + } + src, _ := stressmodel.SatelliteNetwork{Planes: 2, Satellites: 2, GroundStations: 1}.Source() + sets["stressmodel"] = map[string][]byte{ + "satnet.sysml": []byte(src), + "ops.sysml": []byte("package Ops { private import SatelliteNetwork::Constellation::*; part spare : Sat0; }"), + } + return sets +} + +// modelFiles lists the model files under dir, relative to it and sorted. +func modelFiles(t *testing.T, dir string) []string { + t.Helper() + var files []string + err := filepath.WalkDir(dir, func(path string, entry os.DirEntry, err error) error { + if err != nil { + return err + } + if !entry.IsDir() && model.IsModelSource(path) { + rel, err := filepath.Rel(dir, path) + if err != nil { + return err + } + files = append(files, rel) + } + return nil + }) + if err != nil { + t.Fatal(err) + } + sort.Strings(files) + return files +} + +// languageSets splits files into one workspace per language: KerML and SysML +// files must not share a workspace, as the corpus gates hold. +func languageSets(files map[string][]byte) map[string]map[string][]byte { + out := map[string]map[string][]byte{} + for name, content := range files { + lang := "sysml" + if source.KindOf(name) == source.KindKerML { + lang = "kerml" + } + if out[lang] == nil { + out[lang] = map[string][]byte{} + } + out[lang][name] = content + } + return out +} + +func TestIncrementalEqualsFresh(t *testing.T) { + for set, files := range fixtureSets(t) { + for lang, docs := range languageSets(files) { + if len(docs) == 0 { + continue + } + t.Run(set+"/"+lang, func(t *testing.T) { + r := newReplay(t, docs, 1) + r.scripted() + r.randomized(12) + }) + } + } +} + +// corpusRoots are the four OMG model roots the corpus gates pin, replayed under +// the same absence policy: skipped locally, failed when the require variable is set. +var corpusRoots = []struct{ dir, requireEnv, fetch string }{ + {"../../../examples/sysml-v2-training", "OPENSYSML_REQUIRE_TRAINING_CORPUS", "./scripts/download-training-examples.sh"}, + {"../../../examples/pilot-corpora/kerml-examples", "OPENSYSML_REQUIRE_PILOT_CORPORA", "./scripts/download-pilot-corpora.sh"}, + {"../../../examples/pilot-corpora/sysml-examples", "OPENSYSML_REQUIRE_PILOT_CORPORA", "./scripts/download-pilot-corpora.sh"}, + {"../../../examples/pilot-corpora/sysml-validation", "OPENSYSML_REQUIRE_PILOT_CORPORA", "./scripts/download-pilot-corpora.sh"}, +} + +func TestIncrementalEqualsFreshCorpora(t *testing.T) { + for _, root := range corpusRoots { + t.Run(filepath.Base(root.dir), func(t *testing.T) { + if _, err := os.Stat(root.dir); os.IsNotExist(err) { + if os.Getenv(root.requireEnv) != "" { + t.Fatalf("%s is set but %s is missing (run %s)", root.requireEnv, root.dir, root.fetch) + } + t.Skipf("%s not downloaded (run %s)", root.dir, root.fetch) + } + files := map[string][]byte{} + for _, f := range modelFiles(t, root.dir) { + content, err := os.ReadFile(filepath.Join(root.dir, f)) + if err != nil { + t.Fatal(err) + } + files[filepath.ToSlash(f)] = content + } + for lang, docs := range languageSets(files) { + t.Run(lang, func(t *testing.T) { + r := newReplay(t, docs, 2) + r.randomized(6) + }) + } + }) + } +} diff --git a/internal/core/model/metadata.go b/internal/core/model/metadata.go index 589c65ecdc..e7a50c0a12 100644 --- a/internal/core/model/metadata.go +++ b/internal/core/model/metadata.go @@ -2,6 +2,8 @@ package model import ( "github.com/Open-MBEE/OpenSysML/internal/core/ast" + "github.com/Open-MBEE/OpenSysML/internal/core/resolve" + "github.com/Open-MBEE/OpenSysML/internal/core/semantics" "github.com/Open-MBEE/OpenSysML/internal/core/symbols" ) @@ -17,14 +19,14 @@ func (w *Workspace) MetadataBodyRedefines(sym *symbols.Symbol) (*symbols.Symbol, if !ok { return nil, "", false } - w.mu.RLock() - defer w.mu.RUnlock() - resolver, sem := w.newResolver() - owner := resolver.MetadataBodyOwner(sym.OwnerScope) - if owner == nil { - return nil, "", false - } - target := symbols.MetadataBodyTarget(sem, owner, usage.Ident) + w.mu.Lock() + defer w.mu.Unlock() + var target *symbols.Symbol + w.queryLocked(sym.DocName, func(resolver *resolve.Resolver, sem *semantics.Model) { + if owner := resolver.MetadataBodyOwner(sym.OwnerScope); owner != nil { + target = symbols.MetadataBodyTarget(sem, owner, usage.Ident) + } + }) if target == nil { return nil, "", false } @@ -34,15 +36,18 @@ func (w *Workspace) MetadataBodyRedefines(sym *symbols.Symbol) (*symbols.Symbol, // EnclosingMetadataBody returns the nearest scope, from scope outward, whose // declarations redefine the features of a metadata type; nil outside one. func (w *Workspace) EnclosingMetadataBody(scope *symbols.Scope) *symbols.Scope { - w.mu.RLock() - defer w.mu.RUnlock() - resolver, _ := w.newResolver() - for ; scope != nil; scope = scope.Parent() { - if resolver.MetadataBodyOwner(scope) != nil { - return scope + w.mu.Lock() + defer w.mu.Unlock() + var body *symbols.Scope + w.queryLocked(symbols.DocNameOf(scope), func(resolver *resolve.Resolver, _ *semantics.Model) { + for ; scope != nil; scope = scope.Parent() { + if resolver.MetadataBodyOwner(scope) != nil { + body = scope + return + } } - } - return nil + }) + return body } // MetadataBodyMembers returns the members of the metadata definition an @@ -50,12 +55,13 @@ func (w *Workspace) EnclosingMetadataBody(scope *symbols.Scope) *symbols.Scope { // when scope is not a metadata annotation body or its metaclass does not // resolve. func (w *Workspace) MetadataBodyMembers(scope *symbols.Scope) []*symbols.Symbol { - w.mu.RLock() - defer w.mu.RUnlock() - resolver, sem := w.newResolver() - owner := resolver.MetadataBodyOwner(scope) - if owner == nil { - return nil - } - return w.memberSymbolsLocked(resolver, sem, scope, owner) + w.mu.Lock() + defer w.mu.Unlock() + var out []*symbols.Symbol + w.queryLocked(symbols.DocNameOf(scope), func(resolver *resolve.Resolver, sem *semantics.Model) { + if owner := resolver.MetadataBodyOwner(scope); owner != nil { + out = w.memberSymbolsLocked(resolver, sem, scope, owner) + } + }) + return out } diff --git a/internal/core/model/refindex.go b/internal/core/model/refindex.go index a69bf8c623..41fddb9ce6 100644 --- a/internal/core/model/refindex.go +++ b/internal/core/model/refindex.go @@ -32,19 +32,32 @@ type refEntry struct { } // refIndex maps every element (by symbols.KeyOf) to the segments in the -// workspace's documents — never the library's — that reach it or write its name. +// workspace's documents — never the library's — that reach it or write its name, +// one table per document so a change drops only the documents it moved. type refIndex struct { - entries map[symbols.ElementKey][]refEntry + docs map[string]map[symbols.ElementKey][]refEntry +} + +func newRefIndex() *refIndex { + return &refIndex{docs: map[string]map[symbols.ElementKey][]refEntry{}} +} + +// drop forgets doc's table; the next query rebuilds it. A nil index holds none. +func (x *refIndex) drop(doc string) { + if x != nil { + delete(x.docs, doc) + } } // add records segment part of ref, which reaches element and writes name (either -// may be nil). +// may be nil), in doc's table. func (x *refIndex) add(doc *Document, ref resolve.Reference, part int, element, name *symbols.Symbol) { seg := ref.QN.Parts[part] loc := ReferenceLocation{Doc: doc.Name, Content: doc.Content, Span: seg.Span} + entries := x.docs[doc.Name] put := func(sym *symbols.Symbol, reached, named bool) { key := symbols.KeyOf(sym) - x.entries[key] = append(x.entries[key], refEntry{ReferenceLocation: loc, text: seg.Text, + entries[key] = append(entries[key], refEntry{ReferenceLocation: loc, text: seg.Text, reached: reached, named: named, ref: ref, part: part}) } switch { @@ -62,24 +75,37 @@ func (x *refIndex) add(doc *Document, ref resolve.Reference, part int, element, } } -// referenceIndexLocked returns the reverse reference index, building it over every -// document with one resolver when a change has dropped it. Caller holds the write lock. -func (w *Workspace) referenceIndexLocked() *refIndex { - if w.refs != nil { - return w.refs +// referencesLocked returns the reverse-index entries for key across every +// document, in document then position order, building the table of each +// document a change has dropped. Caller holds the write lock. +func (w *Workspace) referencesLocked(key symbols.ElementKey) []refEntry { + if w.refs == nil { + w.refs = newRefIndex() } - idx := &refIndex{entries: map[symbols.ElementKey][]refEntry{}} - r, sem := w.newResolver() names := make([]string, 0, len(w.docs)) for name := range w.docs { names = append(names, name) } sort.Strings(names) + var out []refEntry for _, name := range names { - doc := w.docs[name] - if doc.Scope == nil { - continue + if w.refs.docs[name] == nil { + w.indexReferencesLocked(w.docs[name]) } + out = append(out, w.refs.docs[name][key]...) + } + return out +} + +// indexReferencesLocked builds doc's reverse-index table, as a query owned by +// doc: the resolutions it memoizes and the table itself go when doc or a +// document it read changes. +func (w *Workspace) indexReferencesLocked(doc *Document) { + w.refs.docs[doc.Name] = map[symbols.ElementKey][]refEntry{} + if doc.Scope == nil { + return + } + w.queryLocked(doc.Name, func(r *resolve.Resolver, sem *semantics.Model) { for _, ref := range resolve.References(doc.AST, doc.Scope) { if ref.QN == nil || len(ref.QN.Parts) == 0 { continue @@ -89,12 +115,10 @@ func (w *Workspace) referenceIndexLocked() *refIndex { elements := segmentElements(r, ref, sel) written := segmentNames(r, ref, sel) for i := range ref.QN.Parts { - idx.add(doc, ref, i, elements[i], written[i]) + w.refs.add(doc, ref, i, elements[i], written[i]) } } - } - w.refs = idx - return idx + }) } // ReferencesTo returns every segment in the workspace's documents that reaches @@ -119,7 +143,7 @@ func (w *Workspace) referenceLocations(target *symbols.Symbol, keep func(refEntr } w.mu.Lock() defer w.mu.Unlock() - entries := w.referenceIndexLocked().entries[symbols.KeyOf(target)] + entries := w.referencesLocked(symbols.KeyOf(target)) out := make([]ReferenceLocation, 0, len(entries)) for _, e := range entries { if keep(e) { @@ -139,13 +163,16 @@ func (w *Workspace) RenameConflict(target *symbols.Symbol, name, newName string) w.mu.Lock() defer w.mu.Unlock() var occurrences []rename.Occurrence - for _, e := range w.referenceIndexLocked().entries[symbols.KeyOf(target)] { + for _, e := range w.referencesLocked(symbols.KeyOf(target)) { if e.named && e.text == name { occurrences = append(occurrences, rename.Occurrence{Ref: e.ref, Part: e.part}) } } - r, sem := w.newResolver() - return rename.Check(r, sem, target, name, newName, occurrences) + var conflict *rename.Conflict + w.queryLocked(target.DocName, func(r *resolve.Resolver, sem *semantics.Model) { + conflict = rename.Check(r, sem, target, name, newName, occurrences) + }) + return conflict } // segmentElements is the element each segment of a resolved ref reaches (nil where diff --git a/internal/core/model/refindex_test.go b/internal/core/model/refindex_test.go index cd67b2d20b..891b47fae1 100644 --- a/internal/core/model/refindex_test.go +++ b/internal/core/model/refindex_test.go @@ -2,6 +2,7 @@ package model import ( "fmt" + "sort" "sync" "testing" @@ -87,8 +88,9 @@ func TestReferenceIndexOrdersAcrossDocuments(t *testing.T) { } } -// The index is built on the first query after a change and dropped by every -// mutation, so a query never reads a document set it was not built over. +// The index is built on the first query, one table per document; a mutation +// drops the tables of the documents it moved and keeps the rest, so a query +// never reads a document set it was not built over. func TestReferenceIndexRebuiltLazilyAfterChanges(t *testing.T) { ws := NewWorkspace() ws.SetOnDisk("a.sysml", []byte("package A { part def X; }")) @@ -104,24 +106,35 @@ func TestReferenceIndexRebuiltLazilyAfterChanges(t *testing.T) { if ws.refs == nil { t.Fatal("an unchanged mode dropped the index") } + tables := func() []string { + var out []string + if ws.refs != nil { + for doc := range ws.refs.docs { + out = append(out, doc) + } + } + sort.Strings(out) + return out + } for _, step := range []struct { name string mutate func() + kept []string want int }{ - {"Update", func() { ws.Update("b.sysml", []byte("package B { part y : A::X; part z : A::X; }"), 2) }, 2}, - {"SetOnDisk", func() { ws.SetOnDisk("c.sysml", []byte("package C { part w : A::X; }")) }, 3}, - {"Close", func() { ws.Close("b.sysml") }, 1}, - {"SetConformanceMode", func() { ws.SetConformanceMode(conformance.ModeStrict) }, 1}, - {"DeleteOnDisk", func() { ws.DeleteOnDisk("c.sysml") }, 0}, - {"Remove", func() { ws.Remove("a.sysml") }, 0}, + {"Update", func() { ws.Update("b.sysml", []byte("package B { part y : A::X; part z : A::X; }"), 2) }, []string{"a.sysml"}, 2}, + {"SetOnDisk", func() { ws.SetOnDisk("c.sysml", []byte("package C { part w : A::X; }")) }, []string{"a.sysml", "b.sysml"}, 3}, + {"Close", func() { ws.Close("b.sysml") }, []string{"a.sysml", "c.sysml"}, 1}, + {"SetConformanceMode", func() { ws.SetConformanceMode(conformance.ModeStrict) }, nil, 1}, + {"DeleteOnDisk", func() { ws.DeleteOnDisk("c.sysml") }, []string{"a.sysml"}, 0}, + {"Remove", func() { ws.Remove("a.sysml") }, nil, 0}, } { if ws.refs == nil { t.Fatalf("%s: index not built by the query before it", step.name) } step.mutate() - if ws.refs != nil { - t.Fatalf("%s: index kept across the change", step.name) + if got := tables(); fmt.Sprint(got) != fmt.Sprint(step.kept) { + t.Fatalf("%s: tables kept across the change = %v, want %v", step.name, got, step.kept) } if n := len(ws.ReferencesTo(x)); n != step.want { t.Fatalf("%s: references = %d, want %d", step.name, n, step.want) diff --git a/internal/core/model/render.go b/internal/core/model/render.go index e916908d3d..d7800505d3 100644 --- a/internal/core/model/render.go +++ b/internal/core/model/render.go @@ -7,6 +7,7 @@ import ( "strings" "github.com/Open-MBEE/OpenSysML/internal/core/libs" + "github.com/Open-MBEE/OpenSysML/internal/core/resolve" "github.com/Open-MBEE/OpenSysML/internal/core/semantics" "github.com/Open-MBEE/OpenSysML/internal/core/source" "github.com/Open-MBEE/OpenSysML/internal/core/symbols" @@ -31,29 +32,31 @@ type ViewInfo struct { // the rendering kind it states. A recognized kind this build does not produce // is listed as unsupported with the reason. func (w *Workspace) Views(doc string) []ViewInfo { - w.mu.RLock() - defer w.mu.RUnlock() - renderer := w.rendererLocked(doc) - if renderer == nil { + w.mu.Lock() + defer w.mu.Unlock() + if w.docs[doc] == nil { return nil } out := []ViewInfo{} - for _, sym := range w.documentViewsLocked(doc) { - info := ViewInfo{Name: notationFQN(w.index, sym), Supported: true} - kind, _, err := renderer.KindOf(sym) - switch { - case err == nil: - info.Kind = kind - default: - info.Supported = false - info.Reason = err.Error() - var unsupported *view.UnsupportedKindError - if errors.As(err, &unsupported) { - info.Kind = unsupported.Kind + w.queryLocked(doc, func(*resolve.Resolver, *semantics.Model) { + renderer := w.rendererLocked(doc) + for _, sym := range w.documentViewsLocked(doc) { + info := ViewInfo{Name: notationFQN(w.index, sym), Supported: true} + kind, _, err := renderer.KindOf(sym) + switch { + case err == nil: + info.Kind = kind + default: + info.Supported = false + info.Reason = err.Error() + var unsupported *view.UnsupportedKindError + if errors.As(err, &unsupported) { + info.Kind = unsupported.Kind + } } + out = append(out, info) } - out = append(out, info) - } + }) sort.SliceStable(out, func(i, j int) bool { return out[i].Name < out[j].Name }) return out } @@ -63,21 +66,25 @@ func (w *Workspace) Views(doc string) []ViewInfo { // document returned is the one the rendering was made from, read under the same // lock, so its version, content and scope are the rendering's. func (w *Workspace) RenderView(doc, fqn string) (*view.Rendering, *Document, error) { - w.mu.RLock() - defer w.mu.RUnlock() + w.mu.Lock() + defer w.mu.Unlock() d := w.docs[doc] if d == nil { return nil, nil, fmt.Errorf("%s: no such document", doc) } - rendering, err := w.renderViewLocked(d, fqn) + var rendering *view.Rendering + var err error + w.queryLocked(doc, func(*resolve.Resolver, *semantics.Model) { + rendering, err = w.renderViewLocked(d, fqn) + }) if err != nil { return nil, nil, err } return rendering, d, nil } -// renderViewLocked renders fqn of the held document d under the read lock. +// renderViewLocked renders fqn of the held document d, as a query owned by d. func (w *Workspace) renderViewLocked(d *Document, fqn string) (*view.Rendering, error) { doc := d.Name renderer := w.rendererLocked(doc) @@ -133,34 +140,39 @@ func (w *Workspace) renderPseudoLocked(doc, spec string, renderer *view.Renderer return renderer.RenderExposed(exposed, kind, stated) } -// rendererLocked builds a renderer over the workspace index, reading the -// document's own content for the labels a rendering takes verbatim. It is the -// same construction Session.viewRenderer makes in the REPL. +// rendererLocked builds a renderer over the workspace's resolver and model, +// reading the document's own content for the labels a rendering takes verbatim. +// It is the same construction Session.viewRenderer makes in the REPL. func (w *Workspace) rendererLocked(doc string) *view.Renderer { d := w.docs[doc] if d == nil { return nil } - resolver, sem := w.newResolver() - sf := source.New(doc, d.Content) + resolver, sem := w.semanticsLocked() text := func(name string, span source.Span) string { if name != doc { return "" } - return sf.Text(span) + return d.sf.Text(span) } return view.NewRenderer(sem, resolver, text) } -// sourceTextLocked reads notation from any of the workspace's documents, and -// behind them from the library files its index holds, for the labels a -// rendering takes verbatim across document boundaries. -func (w *Workspace) sourceTextLocked() view.SourceText { - files := make(map[string]*source.SourceFile, len(w.docs)) - for name, d := range w.docs { - files[name] = source.New(name, d.Content) +// sourceText reads notation from whichever document the workspace currently +// holds under a name, and behind them from the library files its index holds, +// for the labels a rendering takes verbatim across document boundaries. It is +// read under the lock, so it always sees the document the index was built from. +func (w *Workspace) sourceText() view.SourceText { + lib := libs.Text(w.libSource) + return func(doc string, span source.Span) string { + if d := w.docs[doc]; d != nil { + return d.sf.Text(span) + } + if lib == nil { + return "" + } + return lib(doc, span) } - return source.TextOf(files, libs.Text(w.libSource)) } // documentViewsLocked are the views the document declares, outermost first, in diff --git a/internal/core/model/scope_names.go b/internal/core/model/scope_names.go index d300e76ce2..433a3ffc47 100644 --- a/internal/core/model/scope_names.go +++ b/internal/core/model/scope_names.go @@ -60,29 +60,33 @@ func (w *Workspace) VisibleNames(scope *symbols.Scope, opts VisibleNamesOptions) if scope == nil { return nil } - w.mu.RLock() - defer w.mu.RUnlock() - - r, sem := w.newResolver() - nw := &nameWalk{ - idx: w.index, - r: r, - sem: sem, - doc: symbols.DocNameOf(scope), - maxDepth: opts.MaxDepth, - library: map[string]bool{}, - seen: map[string]bool{}, - at: map[string]*symbols.Symbol{}, - } - for _, root := range opts.LibraryRoots { - nw.library[root] = true - } - if nw.maxDepth <= 0 { - nw.maxDepth = defaultVisibleNameDepth - } - nw.walk(scope, opts.Redefinition) - sort.Slice(nw.out, func(i, j int) bool { return nw.out[i].Name < nw.out[j].Name }) - return nw.out + w.mu.Lock() + defer w.mu.Unlock() + + doc := symbols.DocNameOf(scope) + var out []VisibleName + w.queryLocked(doc, func(r *resolve.Resolver, sem *semantics.Model) { + nw := &nameWalk{ + idx: w.index, + r: r, + sem: sem, + doc: doc, + maxDepth: opts.MaxDepth, + library: map[string]bool{}, + seen: map[string]bool{}, + at: map[string]*symbols.Symbol{}, + } + for _, root := range opts.LibraryRoots { + nw.library[root] = true + } + if nw.maxDepth <= 0 { + nw.maxDepth = defaultVisibleNameDepth + } + nw.walk(scope, opts.Redefinition) + out = nw.out + }) + sort.Slice(out, func(i, j int) bool { return out[i].Name < out[j].Name }) + return out } // VisibleNamesAt is VisibleNames for the deepest scope of a document that @@ -735,23 +739,26 @@ func (w *Workspace) ElementOnPath(scope *symbols.Scope, path []string) (*symbols if scope == nil || len(path) == 0 { return nil, false } - w.mu.RLock() - defer w.mu.RUnlock() + w.mu.Lock() + defer w.mu.Unlock() - r, sem := w.newResolver() - sym, ok := r.ResolveName(scope, path[0], nil) - if !ok || sym == nil { - return nil, false - } - for _, seg := range path[1:] { - if sym, ok = sem.LookupMember(sym, seg); !ok || sym == nil { - return nil, false + var out *symbols.Symbol + w.queryLocked(symbols.DocNameOf(scope), func(r *resolve.Resolver, sem *semantics.Model) { + sym, ok := r.ResolveName(scope, path[0], nil) + if !ok || sym == nil { + return } - } - if target, ok := r.ResolveAliasTarget(sym); ok && target != nil { - sym = target - } - return sym, true + for _, seg := range path[1:] { + if sym, ok = sem.LookupMember(sym, seg); !ok || sym == nil { + return + } + } + if target, ok := r.ResolveAliasTarget(sym); ok && target != nil { + sym = target + } + out = sym + }) + return out, out != nil } // FQNOf is the qualified name the index registers an element under, which is diff --git a/internal/core/model/workspace.go b/internal/core/model/workspace.go index 2418b3d85a..d2eb45e8d5 100644 --- a/internal/core/model/workspace.go +++ b/internal/core/model/workspace.go @@ -29,9 +29,16 @@ type Workspace struct { // for a caller-built index. libBase *symbols.Index diagCache map[string][]passes.Diagnostic - // refs is the reverse reference index, nil until a query after a change - // rebuilds it (see refindex.go). + // refs is the reverse reference index, built per document on demand and + // dropped per document on a change (see refindex.go). refs *refIndex + // resolver and model are the one resolver and semantic model every analysis + // and query of this workspace shares; what they memoize is owned by the + // document it was computed for and dropped when that document or one it + // read changes. Made on first use (see semanticsLocked). + resolver *resolve.Resolver + model *semantics.Model + gathers *passes.Gathers // analysis is the options every document of this workspace is analyzed under, // so one session asks one question of all its files. analysis passes.Options @@ -102,7 +109,7 @@ func (w *Workspace) SetConformanceMode(mode conformance.Mode) { return } w.analysis.Conformance = mode - w.invalidateLocked() + w.invalidateAllLocked() } // NewIndexWithStdlib returns an index carrying the standard library for a @@ -204,25 +211,78 @@ func (w *Workspace) reindexLocked(name string, content []byte, version int) { w.docs[name] = doc w.index.AddDocument(name, doc.AST) // AddDocument removes stale entries first w.index.ExpandWildcardImports() // Expand new document's wildcard imports - w.invalidateLocked() + w.invalidateLocked(name) } // removeLocked drops name from the document set and index. Caller holds the lock. func (w *Workspace) removeLocked(name string) { delete(w.docs, name) w.index.RemoveDocument(name) - w.invalidateLocked() + w.invalidateLocked(name) } -// invalidateLocked clears all cached diagnostics and the reverse reference index. -// Caller holds the write lock. -func (w *Workspace) invalidateLocked() { - // Conservative: any change clears all cached diagnostics. Correctness first; - // fine-grained cross-document dependency tracking is a later optimization. +// invalidateLocked drops what the replacement of name made stale: the resolver +// and model entries name and its dependents own, transitively, with their +// diagnostics and reverse references. Caller holds the write lock. +func (w *Workspace) invalidateLocked(name string) { + if w.resolver == nil { + w.invalidateAllLocked() + return + } + ch := w.index.TakeChanges() + if ch.Docs == nil { + ch.Docs = map[string]bool{} + } + ch.Docs[name] = true + dropped := w.resolver.Invalidate(ch) + delete(w.diagCache, name) + w.refs.drop(name) + // The gathers the drop took go again, and what they now say differently + // drops the judgments that read it, until nothing more moves. + regather := ch.Docs + for { + for _, doc := range dropped { + delete(w.diagCache, doc) + w.refs.drop(doc) + if gathered, ok := resolve.GatheredDoc(doc); ok { + regather[gathered] = true + } + } + if len(regather) == 0 { + return + } + changed := w.gathers.Regather(w.contextLocked(), regather) + if len(changed) == 0 { + return + } + names := make(map[string]bool, len(changed)) + for _, n := range changed { + names[n] = true + } + dropped = w.resolver.Invalidate(symbols.Changes{Names: names}) + regather = map[string]bool{} + } +} + +// contextLocked is a pass context over the workspace's shared semantic state, +// for work done between analyses. Caller holds the write lock. +func (w *Workspace) contextLocked() *passes.Context { + resolver, sem := w.semanticsLocked() + ctx := passes.NewContextWithOptions("", source.KindSysML, w.index, nil, w.analysis) + ctx.Share(resolver, sem, w.gathers) + return ctx +} + +// invalidateAllLocked drops every cached answer, for a change that moves them +// all: the conformance mode. Caller holds the write lock. +func (w *Workspace) invalidateAllLocked() { w.diagCache = map[string][]passes.Diagnostic{} - // A change anywhere can alter what a name elsewhere resolves to (a shadowing - // declaration, an import target, an alias, an overload), so the index goes too. w.refs = nil + if w.resolver != nil { + w.resolver.InvalidateAll() + w.gathers.Reset() + } + w.index.TakeChanges() } // Diagnostics returns the analysis diagnostics for name, computing them lazily @@ -276,7 +336,8 @@ func (w *Workspace) diagnosticsLocked(name string, doc *Document) []passes.Diagn Fixes: pw.Fixes, }) } - diags := passes.AnalyzeWithOptions(name, source.KindOf(name), doc.AST, parseDiags, w.index, w.analysis) + resolver, sem := w.semanticsLocked() + diags := passes.AnalyzeShared(name, source.KindOf(name), doc.AST, parseDiags, w.analysis, resolver, sem, w.gathers) w.diagCache[name] = diags return diags } @@ -302,10 +363,13 @@ func (w *Workspace) LookupQualified(fqn string) []*symbols.Symbol { // every document's top-level declarations. This is the read path for completion, // which offers library names that no open document declares. func (w *Workspace) TopLevelSymbols(doc string) []*symbols.Symbol { - w.mu.RLock() - defer w.mu.RUnlock() - resolver, _ := w.newResolver() - return resolver.AdmittedTopLevel(doc, w.index.TopLevelBindings(doc)) + w.mu.Lock() + defer w.mu.Unlock() + var out []*symbols.Symbol + w.queryLocked(doc, func(resolver *resolve.Resolver, _ *semantics.Model) { + out = resolver.AdmittedTopLevel(doc, w.index.TopLevelBindings(doc)) + }) + return out } // MembersOnPath returns the members visible on the element that path names from @@ -317,24 +381,25 @@ func (w *Workspace) MembersOnPath(scope *symbols.Scope, path []string) []*symbol if scope == nil || len(path) == 0 { return nil } - w.mu.RLock() - defer w.mu.RUnlock() - - resolver, sem := w.newResolver() - - sym, ok := resolver.ResolveName(scope, path[0], nil) - if !ok || sym == nil { - return nil - } - for _, seg := range path[1:] { - if sym, ok = sem.LookupMember(sym, seg); !ok || sym == nil { - return nil + w.mu.Lock() + defer w.mu.Unlock() + var out []*symbols.Symbol + w.queryLocked(symbols.DocNameOf(scope), func(resolver *resolve.Resolver, sem *semantics.Model) { + sym, ok := resolver.ResolveName(scope, path[0], nil) + if !ok || sym == nil { + return } - } - if target, ok := resolver.ResolveAliasTarget(sym); ok { - sym = target - } - return w.memberSymbolsLocked(resolver, sem, scope, sym) + for _, seg := range path[1:] { + if sym, ok = sem.LookupMember(sym, seg); !ok || sym == nil { + return + } + } + if target, ok := resolver.ResolveAliasTarget(sym); ok { + sym = target + } + out = w.memberSymbolsLocked(resolver, sem, scope, sym) + }) + return out } // memberSymbolsLocked returns the members visible on sym as seen from scope. @@ -355,17 +420,28 @@ func (w *Workspace) memberSymbolsLocked(resolver *resolve.Resolver, sem *semanti return members } -// newResolver is a resolver over the index with a semantic model attached: an -// inherited member and the element filters gating an import are both answered by -// the model, so a read path without one resolves differently to a checked one. -// Calls are selected under the checker's argument typing, as a checked document's are. -func (w *Workspace) newResolver() (*resolve.Resolver, *semantics.Model) { - resolver := resolve.New(w.index) - sem := semantics.NewModel(resolver) - resolver.SetModel(sem) - sem.SetArgumentTyper(passes.NewArgumentTyper(resolver, sem)) - sem.SetSourceText(w.sourceTextLocked()) - return resolver, sem +// semanticsLocked is the workspace's resolver with its model and argument typer, made +// on first use and kept for its life; it tracks per document what each entry read. +func (w *Workspace) semanticsLocked() (*resolve.Resolver, *semantics.Model) { + if w.resolver == nil { + resolver := resolve.New(w.index) + sem := semantics.NewModel(resolver) + resolver.SetModel(sem) + sem.SetArgumentTyper(passes.NewArgumentTyper(resolver, sem)) + sem.SetSourceText(w.sourceText()) + resolver.Track() + w.resolver, w.model, w.gathers = resolver, sem, passes.NewGathers() + } + return w.resolver, w.model +} + +// queryLocked runs f, a query made from doc ("" for none), over the workspace's +// resolver and model: quietly, so a failure it meets is still reported when doc +// is analyzed, and owned by doc, so doc's change drops what it memoized. Caller +// holds the write lock. +func (w *Workspace) queryLocked(doc string, f func(*resolve.Resolver, *semantics.Model)) { + resolver, sem := w.semanticsLocked() + resolver.Query(doc, func() { f(resolver, sem) }) } // Document returns the current parsed document for name, or nil. The document is @@ -451,14 +527,17 @@ func (w *Workspace) ResolveQualifiedInDoc(name string, scope *symbols.Scope, qn // a reference subsetting or a feature chain's member segment (see // resolve.Reference). func (w *Workspace) ResolveReferenceInDoc(name string, ref resolve.Reference) (*symbols.Symbol, bool) { - w.mu.RLock() - defer w.mu.RUnlock() - resolver, sem := w.newResolver() - sym, ok := resolver.ResolveReference(ref) - if sel := invocationSelection(resolver, sem, ref); sel != nil { - sym = calledDeclaration(sel, sym) - ok = sym != nil - } + w.mu.Lock() + defer w.mu.Unlock() + var sym *symbols.Symbol + var ok bool + w.queryLocked(name, func(resolver *resolve.Resolver, sem *semantics.Model) { + sym, ok = resolver.ResolveReference(ref) + if sel := invocationSelection(resolver, sem, ref); sel != nil { + sym = calledDeclaration(sel, sym) + ok = sym != nil + } + }) return sym, ok } @@ -466,13 +545,15 @@ func (w *Workspace) ResolveReferenceInDoc(name string, ref resolve.Reference) (* // arguments leave tied, when ref is the name it calls; nil for any other reference // and for a call that selects one declaration. func (w *Workspace) AmbiguousInvocationInDoc(name string, ref resolve.Reference) []*symbols.Symbol { - w.mu.RLock() - defer w.mu.RUnlock() - resolver, sem := w.newResolver() - if sel := invocationSelection(resolver, sem, ref); sel != nil && sel.Ambiguous { - return sel.Tied - } - return nil + w.mu.Lock() + defer w.mu.Unlock() + var tied []*symbols.Symbol + w.queryLocked(name, func(resolver *resolve.Resolver, sem *semantics.Model) { + if sel := invocationSelection(resolver, sem, ref); sel != nil && sel.Ambiguous { + tied = sel.Tied + } + }) + return tied } // invocationSelection is the overload selection for the call whose name ref is, @@ -512,11 +593,14 @@ func (w *Workspace) ResolveReferenceSegmentsInDoc(name string, ref resolve.Refer if ref.QN == nil || len(ref.QN.Parts) == 0 { return nil } - w.mu.RLock() - defer w.mu.RUnlock() - r, sem := w.newResolver() - r.ResolveReference(ref) - return segmentElements(r, ref, invocationSelection(r, sem, ref)) + w.mu.Lock() + defer w.mu.Unlock() + var out []*symbols.Symbol + w.queryLocked(name, func(r *resolve.Resolver, sem *semantics.Model) { + r.ResolveReference(ref) + out = segmentElements(r, ref, invocationSelection(r, sem, ref)) + }) + return out } // ResolveReferenceNameSegmentsInDoc is ResolveReferenceSegmentsInDoc reporting @@ -526,11 +610,14 @@ func (w *Workspace) ResolveReferenceNameSegmentsInDoc(name string, ref resolve.R if ref.QN == nil || len(ref.QN.Parts) == 0 { return nil } - w.mu.RLock() - defer w.mu.RUnlock() - r, sem := w.newResolver() - r.ResolveReference(ref) - return segmentNames(r, ref, invocationSelection(r, sem, ref)) + w.mu.Lock() + defer w.mu.Unlock() + var out []*symbols.Symbol + w.queryLocked(name, func(r *resolve.Resolver, sem *semantics.Model) { + r.ResolveReference(ref) + out = segmentNames(r, ref, invocationSelection(r, sem, ref)) + }) + return out } // selectedName is the name a call's written last segment is once selected names diff --git a/internal/core/model/workspace_test.go b/internal/core/model/workspace_test.go index d8a8b41b85..98dab8cae5 100644 --- a/internal/core/model/workspace_test.go +++ b/internal/core/model/workspace_test.go @@ -1,6 +1,11 @@ package model -import "testing" +import ( + "reflect" + "testing" + + "github.com/Open-MBEE/OpenSysML/internal/core/passes" +) func TestWorkspaceOpenIndexesDocument(t *testing.T) { ws := NewWorkspace() @@ -114,3 +119,77 @@ func TestWorkspaceRemoveDropsFromIndex(t *testing.T) { t.Fatal("document should be gone after remove") } } + +func TestWorkspaceEditDropsTheDependentsOnly(t *testing.T) { + ws := NewWorkspace() + ws.Open("a.sysml", []byte("package A { part def X; }"), 1) + ws.Open("b.sysml", []byte("package B { part def Y :> A::X; }"), 1) + ws.Open("c.sysml", []byte("package C { part def Z; part z : Z; }"), 1) + for _, name := range []string{"a.sysml", "b.sysml", "c.sysml"} { + if diags := ws.Diagnostics(name); len(diags) != 0 { + t.Fatalf("%s: %v", name, diags) + } + } + ws.Update("a.sysml", []byte("package A { part def X2; }"), 2) + if _, cached := ws.diagCache["c.sysml"]; !cached { + t.Fatal("c.sysml, which reads nothing of a.sysml, lost its diagnostics") + } + if _, cached := ws.diagCache["b.sysml"]; cached { + t.Fatal("b.sysml, which specializes A::X, kept its diagnostics") + } + if diags := ws.Diagnostics("b.sysml"); len(diags) == 0 { + t.Fatal("b.sysml still resolves A::X, which is gone") + } +} + +func TestWorkspaceEditsReleaseWhatTheyReplaced(t *testing.T) { + ws := NewWorkspace() + ws.Open("a.sysml", []byte("package A { part def X; part x : X; }"), 1) + ws.Open("b.sysml", []byte("package B { part def Y :> A::X; part y : Y; }"), 1) + ws.Diagnostics("a.sysml") + ws.Diagnostics("b.sysml") + ownedA, ownedB := ws.resolver.Owned("a.sysml"), ws.resolver.Owned("b.sysml") + if ownedA == 0 || ownedB == 0 { + t.Fatalf("owned after the first analysis: a %d, b %d; want both > 0", ownedA, ownedB) + } + for i := 2; i < 200; i++ { + ws.Update("a.sysml", []byte("package A { part def X; part x : X; }"), i) + ws.Diagnostics("a.sysml") + ws.Diagnostics("b.sysml") + if a, b := ws.resolver.Owned("a.sysml"), ws.resolver.Owned("b.sysml"); a != ownedA || b != ownedB { + t.Fatalf("edit %d: owned a %d, b %d; want %d, %d as after the first analysis", i, a, b, ownedA, ownedB) + } + } +} + +// TestWorkspaceUnionJudgmentFollowsAnotherDocument: a workspace-wide audit judges +// each document over what every document declares, so a change to one document +// moves another's verdict though the other reads nothing of it. +func TestWorkspaceUnionJudgmentFollowsAnotherDocument(t *testing.T) { + const notDerived = "oosem-requirement-not-derived" + sat := []byte("package S { private import OOSEM::*; #systemRequirement requirement sys; }") + ws := NewWorkspace() + ws.Open("hub.sysml", []byte("package M { private import OOSEM::*; #missionRequirement requirement mission; }"), 1) + ws.Open("sat.sysml", sat, 1) + if got := codesOf(ws.Diagnostics("sat.sysml")); got[notDerived] != 1 { + t.Fatalf("sat.sysml under a mission requirement: %v, want one %s", got, notDerived) + } + ws.Update("hub.sysml", []byte("package M { private import OOSEM::*; #stakeholderNeed requirement need; }"), 2) + if got := codesOf(ws.Diagnostics("sat.sysml")); got[notDerived] != 0 { + t.Fatalf("sat.sysml with no mission requirement anywhere: %v, want no %s", got, notDerived) + } + fresh := NewWorkspace() + fresh.Open("hub.sysml", ws.Document("hub.sysml").Content, 1) + fresh.Open("sat.sysml", sat, 1) + if got, want := codesOf(ws.Diagnostics("sat.sysml")), codesOf(fresh.Diagnostics("sat.sysml")); !reflect.DeepEqual(got, want) { + t.Fatalf("incremental %v, fresh %v", got, want) + } +} + +func codesOf(diags []passes.Diagnostic) map[string]int { + out := map[string]int{} + for _, d := range diags { + out[d.Code]++ + } + return out +} diff --git a/internal/core/passes/analyze.go b/internal/core/passes/analyze.go index 9228e0563a..97c0e0a7bb 100644 --- a/internal/core/passes/analyze.go +++ b/internal/core/passes/analyze.go @@ -4,6 +4,8 @@ import ( "sort" "github.com/Open-MBEE/OpenSysML/internal/core/ast" + "github.com/Open-MBEE/OpenSysML/internal/core/resolve" + "github.com/Open-MBEE/OpenSysML/internal/core/semantics" "github.com/Open-MBEE/OpenSysML/internal/core/source" "github.com/Open-MBEE/OpenSysML/internal/core/symbols" ) @@ -112,8 +114,24 @@ func dropEscalatedWarnings(diags []Diagnostic) []Diagnostic { // AnalyzeWithOptions validates a document under explicit analysis options. func AnalyzeWithOptions(name string, kind source.Kind, root *ast.RootNamespace, parseDiags []Diagnostic, idx *symbols.Index, opts Options) []Diagnostic { - ctx := NewContextWithOptions(name, kind, idx, parseDiags, opts) - diags := dropEscalatedWarnings(DefaultRegistry().Run(ctx, name, root)) + return analyze(NewContextWithOptions(name, kind, idx, parseDiags, opts), root) +} + +// AnalyzeShared validates a document over a resolver and model kept across +// analyses: what the run memoizes is owned by the document (see +// Resolver.InDocument), to be dropped when it or what it read changes. +func AnalyzeShared(name string, kind source.Kind, root *ast.RootNamespace, + parseDiags []Diagnostic, opts Options, resolver *resolve.Resolver, model *semantics.Model, + gathers *Gathers) []Diagnostic { + ctx := NewContextWithOptions(name, kind, resolver.Index(), parseDiags, opts) + ctx.Share(resolver, model, gathers) + var diags []Diagnostic + resolver.InDocument(name, func() { diags = analyze(ctx, root) }) + return diags +} + +func analyze(ctx *Context, root *ast.RootNamespace) []Diagnostic { + diags := dropEscalatedWarnings(DefaultRegistry().Run(ctx, ctx.Name, root)) sort.SliceStable(diags, func(i, j int) bool { a, b := diags[i], diags[j] if a.Span.Offset != b.Span.Offset { diff --git a/internal/core/passes/gathers.go b/internal/core/passes/gathers.go new file mode 100644 index 0000000000..af783922f9 --- /dev/null +++ b/internal/core/passes/gathers.go @@ -0,0 +1,208 @@ +package passes + +import ( + "sort" + "sync" + + "github.com/Open-MBEE/OpenSysML/internal/core/symbols" +) + +// Gathers holds what the workspace-wide audits gathered from each document, and the +// unions they judge over; kept current by Regather and readable concurrently once built. +type Gathers struct { + mu sync.Mutex + // docs is the set of workspace documents gathered; nil until first use. + docs map[string]bool + oosem *oosemUnion + mosa *mosaUnion + identity *identityUnion +} + +// NewGathers returns gathers with nothing gathered yet. +func NewGathers() *Gathers { return &Gathers{} } + +// Reset forgets every gather, for a change that moves them all. +func (g *Gathers) Reset() { + g.mu.Lock() + defer g.mu.Unlock() + g.docs, g.oosem, g.mosa, g.identity = nil, nil, nil, nil +} + +// documents lists the workspace documents gathered, sorted, learning them on +// first use; the read is untracked since Regather keeps the set current. +func (g *Gathers) documents(ctx *Context) []string { + if g.docs == nil { + g.docs = map[string]bool{} + ctx.Resolver().Untracked(func() { + for _, doc := range ctx.Index.WorkspaceDocuments() { + g.docs[doc] = true + } + }) + } + return sortedKeys(g.docs) +} + +// workspaceRoot is the root scope of a workspace document — one the index +// holds that is not bundled library content — or nil. +func (g *Gathers) workspaceRoot(ctx *Context, doc string) *symbols.Scope { + var root *symbols.Scope + ctx.Resolver().Untracked(func() { + if !ctx.Index.IsLibraryDocument(doc) { + root = ctx.Index.DocumentRoot(doc) + } + }) + return root +} + +// gather runs f over doc's root in doc's own frame, quietly and as no +// dependency of the analysis under way (see resolve.Resolver.Gather). +func (g *Gathers) gather(ctx *Context, doc string, f func(root *symbols.Scope)) { + ctx.Resolver().Gather(doc, func() { + if root := ctx.Index.DocumentRoot(doc); root != nil { + f(root) + } + }) +} + +// Regather gathers anew the documents in docs — replaced, removed, or whose +// gather the resolver dropped (see resolve.GatheredDoc) — and names what the +// unions now answer differently, for the resolver to drop the readers of. +// Nothing is gathered for a union no analysis has asked for yet. +func (g *Gathers) Regather(ctx *Context, docs map[string]bool) []string { + g.mu.Lock() + defer g.mu.Unlock() + if g.docs == nil { + return nil + } + todo := docs + changed := map[string]bool{} + about := todo[aboutGather] + for _, doc := range sortedKeys(todo) { + if doc == aboutGather { + continue + } + switch { + case g.workspaceRoot(ctx, doc) != nil: + about = about || !g.docs[doc] + g.docs[doc] = true + case g.docs[doc]: + about = true + delete(g.docs, doc) + default: + continue + } + if g.oosem != nil { + if a := newOOSEMAudit(ctx); a != nil { + g.oosem.regather(ctx, g, a, doc, changed) + } + } + if g.mosa != nil { + if a := newMOSAAudit(ctx); a != nil { + g.mosa.regather(ctx, g, a, doc, changed) + } + } + if g.identity != nil { + g.identity.regather(ctx, g, doc, changed) + } + } + if about && g.identity != nil { + g.identity.regatherAbout(ctx, g, changed) + } + return sortedKeys(changed) +} + +// has reports whether doc is among the workspace documents gathered. +func (g *Gathers) has(doc string) bool { + g.mu.Lock() + defer g.mu.Unlock() + return g.docs[doc] +} + +// oosemOf is the OOSEM facts of every workspace document, gathered on first use. +func (g *Gathers) oosemOf(ctx *Context, a *oosemAudit) *oosemUnion { + g.mu.Lock() + defer g.mu.Unlock() + if g.oosem == nil { + u := newOOSEMUnion() + for _, doc := range g.documents(ctx) { + u.regather(ctx, g, a, doc, nil) + } + g.oosem = u + } + return g.oosem +} + +// mosaOf is the MOSA facts of every workspace document, gathered on first use. +func (g *Gathers) mosaOf(ctx *Context, a *mosaAudit) *mosaUnion { + g.mu.Lock() + defer g.mu.Unlock() + if g.mosa == nil { + u := newMOSAUnion() + for _, doc := range g.documents(ctx) { + u.regather(ctx, g, a, doc, nil) + } + g.mosa = u + } + return g.mosa +} + +// identitiesOf is the identity tables of every workspace document, gathered on +// first use. +func (g *Gathers) identitiesOf(ctx *Context) *identityUnion { + g.mu.Lock() + defer g.mu.Unlock() + if g.identity == nil { + u := newIdentityUnion() + for _, doc := range g.documents(ctx) { + u.regather(ctx, g, doc, nil) + } + u.regatherAbout(ctx, g, nil) + g.identity = u + } + return g.identity +} + +// countSet is the union of per-document sets: a key is in it while any +// document states it. +type countSet[K comparable] map[K]int + +func (s countSet[K]) has(k K) bool { return s[k] > 0 } + +// move replaces one document's contribution, old by cur, naming the keys whose +// membership flipped in changed when it is not nil. +func move[K comparable](s countSet[K], old, cur map[K]bool, name func(K) string, changed map[string]bool) { + before := map[K]bool{} + for k := range old { + before[k] = s.has(k) + } + for k := range cur { + before[k] = s.has(k) + } + for k := range old { + if s[k] <= 1 { + delete(s, k) + } else { + s[k]-- + } + } + for k := range cur { + s[k]++ + } + if changed == nil { + return + } + for k, was := range before { + if s.has(k) != was { + changed[name(k)] = true + } + } +} + +func sortedKeys(m map[string]bool) []string { + out := make([]string, 0, len(m)) + for k := range m { + out = append(out, k) + } + sort.Strings(out) + return out +} diff --git a/internal/core/passes/gathers_test.go b/internal/core/passes/gathers_test.go new file mode 100644 index 0000000000..5ec76319b4 --- /dev/null +++ b/internal/core/passes/gathers_test.go @@ -0,0 +1,292 @@ +package passes + +import ( + "fmt" + "reflect" + "sort" + "sync" + "testing" + + "github.com/Open-MBEE/OpenSysML/internal/core/ast" + "github.com/Open-MBEE/OpenSysML/internal/core/parser" + "github.com/Open-MBEE/OpenSysML/internal/core/resolve" + "github.com/Open-MBEE/OpenSysML/internal/core/semantics" + "github.com/Open-MBEE/OpenSysML/internal/core/source" + "github.com/Open-MBEE/OpenSysML/internal/core/symbols" +) + +// sharedWorkspace is the shared semantic state a workspace keeps over its +// index: one tracked resolver, one model and one set of gathers. +type sharedWorkspace struct { + idx *symbols.Index + resolver *resolve.Resolver + model *semantics.Model + gathers *Gathers + docs map[string]*ast.RootNamespace +} + +func newSharedWorkspace() *sharedWorkspace { + idx := newTestIndex() + resolver := resolve.New(idx) + sem := semantics.NewModel(resolver) + resolver.SetModel(sem) + sem.SetArgumentTyper(NewArgumentTyper(resolver, sem)) + resolver.Track() + return &sharedWorkspace{idx: idx, resolver: resolver, model: sem, gathers: NewGathers(), docs: map[string]*ast.RootNamespace{}} +} + +// put indexes src under name and invalidates what the change reached, as the +// workspace does on a replacement. +func (w *sharedWorkspace) put(name, src string) { + root := parser.New(source.New(name, []byte(src))).ParseFile() + w.docs[name] = root + w.idx.AddDocument(name, root) + w.idx.ExpandWildcardImports() + w.invalidate(name) +} + +func (w *sharedWorkspace) remove(name string) { + delete(w.docs, name) + w.idx.RemoveDocument(name) + w.invalidate(name) +} + +func (w *sharedWorkspace) context() *Context { + ctx := NewContextWithOptions("", source.KindSysML, w.idx, nil, Options{}) + ctx.Share(w.resolver, w.model, w.gathers) + return ctx +} + +func (w *sharedWorkspace) invalidate(name string) { + ch := w.idx.TakeChanges() + if ch.Docs == nil { + ch.Docs = map[string]bool{} + } + ch.Docs[name] = true + dropped := w.resolver.Invalidate(ch) + regather := ch.Docs + for len(regather) > 0 { + for _, doc := range dropped { + if gathered, ok := resolve.GatheredDoc(doc); ok { + regather[gathered] = true + } + } + changed := w.gathers.Regather(w.context(), regather) + if len(changed) == 0 { + break + } + names := map[string]bool{} + for _, n := range changed { + names[n] = true + } + dropped = w.resolver.Invalidate(symbols.Changes{Names: names}) + regather = map[string]bool{} + } +} + +func (w *sharedWorkspace) analyze(name string) []Diagnostic { + return AnalyzeShared(name, source.KindSysML, w.docs[name], nil, Options{}, w.resolver, w.model, w.gathers) +} + +func (w *sharedWorkspace) analyzeAll() { + for _, name := range w.names() { + w.analyze(name) + } +} + +func (w *sharedWorkspace) names() []string { + out := make([]string, 0, len(w.docs)) + for name := range w.docs { + out = append(out, name) + } + sort.Strings(out) + return out +} + +// fresh analyzes name over a resolver and model of the run alone, the answer +// the shared analysis has to match. +func (w *sharedWorkspace) fresh(name string) []Diagnostic { + return AnalyzeWithOptions(name, source.KindSysML, w.docs[name], nil, w.idx, Options{}) +} + +// oosemSplit is a mission requirement in one document and a system requirement +// in each of n others, derived from nothing: whether that is a finding depends +// on the union of them all, though no document reads another. +func oosemSplit(n int) map[string]string { + docs := map[string]string{ + "hub.sysml": oosemModel(` + #missionRequirement requirement mission;`), + } + for i := 0; i < n; i++ { + docs[fmt.Sprintf("sat%02d.sysml", i)] = fmt.Sprintf(`package S%d { + private import OOSEM::*; + #systemRequirement requirement sys%d; + }`, i, i) + } + return docs +} + +// gatherPointers are the per-document gathers held, by identity, so a test can +// tell a gather kept from one done again. +func gatherPointers(g *Gathers) map[string][3]uintptr { + out := map[string][3]uintptr{} + for doc, f := range g.oosem.perDoc { + p := out[doc] + p[0] = reflect.ValueOf(f).Pointer() + out[doc] = p + } + for doc, f := range g.mosa.perDoc { + p := out[doc] + p[1] = reflect.ValueOf(f).Pointer() + out[doc] = p + } + for doc, f := range g.identity.perDoc { + p := out[doc] + p[2] = reflect.ValueOf(f).Pointer() + out[doc] = p + } + return out +} + +// Analyzing every document of a workspace gathers each once: a second pass +// over them all finds every gather where the first left it. +func TestGathersGatherEachDocumentOnce(t *testing.T) { + w := newSharedWorkspace() + docs := oosemSplit(6) + for name, src := range docs { + w.put(name, src) + } + w.analyzeAll() + first := gatherPointers(w.gathers) + if len(first) != len(docs) { + t.Fatalf("gathered %d documents, want %d", len(first), len(docs)) + } + for doc, p := range first { + if p[0] == 0 || p[1] == 0 || p[2] == 0 { + t.Errorf("%s: gathers missing an audit: %v", doc, p) + } + } + w.analyzeAll() + if again := gatherPointers(w.gathers); !reflect.DeepEqual(first, again) { + t.Fatalf("a second analysis of every document gathered again:\n%v\n%v", first, again) + } +} + +// A change gathers the changed document again and leaves the others' gathers +// in place; a removal drops the gather, an addition makes one. +func TestGathersRegatherOnlyTheChangedDocument(t *testing.T) { + w := newSharedWorkspace() + for name, src := range oosemSplit(4) { + w.put(name, src) + } + w.analyzeAll() + before := gatherPointers(w.gathers) + + w.put("sat01.sysml", "package S1 { private import OOSEM::*; #systemRequirement requirement other; }") + after := gatherPointers(w.gathers) + for doc, p := range before { + if doc == "sat01.sysml" { + if after[doc] == p { + t.Errorf("%s: changed but its gathers were kept", doc) + } + continue + } + if after[doc] != p { + t.Errorf("%s: unchanged but gathered again", doc) + } + } + + w.remove("sat02.sysml") + if _, ok := gatherPointers(w.gathers)["sat02.sysml"]; ok { + t.Error("a removed document keeps its gathers") + } + if w.gathers.docs["sat02.sysml"] { + t.Error("a removed document is still counted a workspace document") + } + w.put("sat09.sysml", "package S9 { private import OOSEM::*; #systemRequirement requirement added; }") + if _, ok := gatherPointers(w.gathers)["sat09.sysml"]; !ok { + t.Error("an added document has no gathers") + } + for _, name := range w.names() { + if got, want := w.analyze(name), w.fresh(name); !reflect.DeepEqual(got, want) { + t.Errorf("%s: shared analysis\n%v\nfresh analysis\n%v", name, got, want) + } + } +} + +// A change that moves the union — the mission requirement leaves, so no system +// requirement has a level above it to be derived from — reaches every document +// whose verdict read it, though none of them changed. +func TestGathersUnionChangeReachesItsReaders(t *testing.T) { + w := newSharedWorkspace() + for name, src := range oosemSplit(3) { + w.put(name, src) + } + w.analyzeAll() + if w.resolver.Owned("sat00.sysml") == 0 { + t.Fatal("the analysis of sat00.sysml memoized nothing to drop") + } + if len(only(w.analyze("sat00.sysml"), CodeOOSEMRequirementNotDerived)) == 0 { + t.Fatal("sat00.sysml: no underived system requirement while the hub declares a mission requirement") + } + before := gatherPointers(w.gathers) + // With no mission requirement above them, the system requirements are no + // longer expected to derive from one. + w.put("hub.sysml", oosemModel(`#stakeholderNeed requirement need;`)) + if w.resolver.Owned("sat00.sysml") != 0 { + t.Fatal("sat00.sysml kept what it memoized over a union that moved") + } + after := gatherPointers(w.gathers) + for doc, p := range before { + if doc == "hub.sysml" { + continue + } + if after[doc] != p { + t.Errorf("%s: read a union that moved, but was gathered again", doc) + } + } + if got := only(w.analyze("sat00.sysml"), CodeOOSEMRequirementNotDerived); len(got) != 0 { + t.Fatalf("sat00.sysml: underived system requirement reported with no mission requirement anywhere:\n%v", got) + } + for _, name := range w.names() { + if got, want := w.analyze(name), w.fresh(name); !reflect.DeepEqual(got, want) { + t.Errorf("%s: shared analysis\n%v\nfresh analysis\n%v", name, got, want) + } + } +} + +// Gathers populated once serve analyses on other goroutines, each over a +// resolver and model of its own, as a batch pool over a read-only index runs. +func TestGathersServeConcurrentAnalyses(t *testing.T) { + w := newSharedWorkspace() + docs := oosemSplit(8) + for name, src := range docs { + w.put(name, src) + } + w.analyze("hub.sysml") + want := map[string][]Diagnostic{} + for _, name := range w.names() { + want[name] = w.fresh(name) + } + var wg sync.WaitGroup + errs := make(chan string, len(docs)) + for _, name := range w.names() { + wg.Add(1) + go func(name string) { + defer wg.Done() + resolver := resolve.New(w.idx) + sem := semantics.NewModel(resolver) + resolver.SetModel(sem) + sem.SetArgumentTyper(NewArgumentTyper(resolver, sem)) + got := AnalyzeShared(name, source.KindSysML, w.docs[name], nil, Options{}, resolver, sem, w.gathers) + if !reflect.DeepEqual(got, want[name]) { + errs <- fmt.Sprintf("%s: concurrent analysis\n%v\nfresh analysis\n%v", name, got, want[name]) + } + }(name) + } + wg.Wait() + close(errs) + for e := range errs { + t.Error(e) + } +} diff --git a/internal/core/passes/identity_gather.go b/internal/core/passes/identity_gather.go new file mode 100644 index 0000000000..184e685f59 --- /dev/null +++ b/internal/core/passes/identity_gather.go @@ -0,0 +1,347 @@ +package passes + +import ( + "sort" + "strings" + + "github.com/Open-MBEE/OpenSysML/internal/core/identity" + "github.com/Open-MBEE/OpenSysML/internal/core/resolve" + "github.com/Open-MBEE/OpenSysML/internal/core/symbols" +) + +// aboutGather names the gather of the `about`-annotated elements no workspace +// document declares — bundled library ones — which join the id space too. +const aboutGather = "\x00identity" + +// identityKey is one effective id in one project scope: the unit the identity +// audit reads the union by. +type identityKey struct{ scope, id string } + +// identityJudgment names what the identity audit of doc read of the union: the +// groups its elements are filed in. A regather that moves a group names the +// judgments of every document filed there, and of every non-workspace document. +func identityJudgment(doc string) string { return "\x00identity/" + doc } + +const identityJudgments = "\x00identity/*" + +// identityHit is a declared id landing in the derived id space of the element +// with the key it is filed under. +type identityHit struct { + info *identity.Info + decl identity.Declaration + space string +} + +// derivedTarget is the base id whose derived id space (a membership id, an +// expression-node id) a declared id lands in. +type derivedTarget struct{ base, space string } + +// derivedTargets lists the derived id spaces a declared id lands in: the +// owner of `…_om`, and of each `…_p…` tail that is a chain of encoded positions. +func derivedTargets(id string) []derivedTarget { + var out []derivedTarget + if base, ok := strings.CutSuffix(id, "_om"); ok { + out = append(out, derivedTarget{base, "the owning-membership id"}) + } + for i := strings.Index(id, "_p"); i >= 0; { + if expressionPositions(id[i+2:]) { + out = append(out, derivedTarget{id[:i], "an expression-node id"}) + } + next := strings.Index(id[i+1:], "_p") + if next < 0 { + break + } + i += 1 + next + } + return out +} + +// identityIndex is a generated id space indexed by effective id: the elements +// sharing each id, and the declared ids landing in each element's derived ids. +type identityIndex struct { + byID map[identityKey][]*identity.Info + hits map[identityKey][]identityHit +} + +func newIdentityIndex() *identityIndex { + return &identityIndex{byID: map[identityKey][]*identity.Info{}, hits: map[identityKey][]identityHit{}} +} + +func keyOf(info *identity.Info) identityKey { + return identityKey{scopeKey(info), info.EffectiveID} +} + +// insert files info under its effective id and its declared ids under the +// elements whose derived id spaces they land in. +func (x *identityIndex) insert(info *identity.Info) { + x.byID[keyOf(info)] = append(x.byID[keyOf(info)], info) + scope := scopeKey(info) + for _, d := range info.Declarations { + if !d.Declared || d.ID == "" { + continue + } + for _, t := range derivedTargets(d.ID) { + k := identityKey{scope, t.base} + x.hits[k] = append(x.hits[k], identityHit{info, d, t.space}) + } + } +} + +// remove undoes insert. +func (x *identityIndex) remove(info *identity.Info) { + k := keyOf(info) + group := x.byID[k] + for i, o := range group { + if o == info { + x.byID[k] = append(group[:i:i], group[i+1:]...) + break + } + } + if len(x.byID[k]) == 0 { + delete(x.byID, k) + } + scope := scopeKey(info) + for _, d := range info.Declarations { + if !d.Declared || d.ID == "" { + continue + } + for _, t := range derivedTargets(d.ID) { + hk := identityKey{scope, t.base} + hits := x.hits[hk] + for i, h := range hits { + if h.info == info && h.decl == d { + x.hits[hk] = append(hits[:i:i], hits[i+1:]...) + break + } + } + if len(x.hits[hk]) == 0 { + delete(x.hits, hk) + } + } + } +} + +// keysOf lists the keys an element's entries are filed under. +func keysOf(info *identity.Info) []identityKey { + out := []identityKey{keyOf(info)} + scope := scopeKey(info) + for _, d := range info.Declarations { + if !d.Declared || d.ID == "" { + continue + } + for _, t := range derivedTargets(d.ID) { + out = append(out, identityKey{scope, t.base}) + } + } + return out +} + +// group is the elements sharing one effective id. +func (x *identityIndex) group(k identityKey) []*identity.Info { return x.byID[k] } + +// hitsOn is the declared ids landing in one element's derived id space. +func (x *identityIndex) hitsOn(k identityKey) []identityHit { return x.hits[k] } + +// readers names the documents whose elements are filed under k. +func (x *identityIndex) readers(k identityKey, into map[string]bool) { + for _, info := range x.byID[k] { + into[info.Symbol.DocName] = true + } + for _, h := range x.hits[k] { + into[h.info.Symbol.DocName] = true + } +} + +// identityContribution is what one gather — a document's own elements, or the +// `about`-annotated library elements — adds to the union. +type identityContribution struct { + table *identity.Table + infos []*identity.Info +} + +// contribute selects the infos of table that pass keep. +func contribute(table *identity.Table, keep func(*identity.Info) bool) *identityContribution { + c := &identityContribution{table: table} + for _, sym := range table.Symbols() { + if info, ok := table.Info(sym); ok && keep(info) { + c.infos = append(c.infos, info) + } + } + return c +} + +// facts spells, per key, what the contribution files there, so a regather +// names only the keys whose entries it moved. +func (c *identityContribution) facts() map[identityKey]string { + if c == nil { + return nil + } + lines := map[identityKey][]string{} + for _, info := range c.infos { + spelled := spellInfo(info) + for _, k := range keysOf(info) { + lines[k] = append(lines[k], spelled) + } + } + facts := make(map[identityKey]string, len(lines)) + for k, l := range lines { + sort.Strings(l) + facts[k] = strings.Join(l, "\n") + } + return facts +} + +// spellInfo spells what the union's readers see of an element. +func spellInfo(info *identity.Info) string { + var b strings.Builder + b.WriteString(info.FQN) + b.WriteString("\x01") + b.WriteString(info.EffectiveID) + if info.Annotated { + b.WriteString("\x01@") + } + for _, d := range info.Declarations { + if d.Declared { + b.WriteString("\x01") + b.WriteString(d.ID) + } + } + return b.String() +} + +// identityUnion is the identity tables of every workspace document, indexed as +// one id space per project scope and read by name, as oosemUnion is. The +// `about`-annotated elements outside every document are one more contribution. +type identityUnion struct { + *identityIndex + perDoc map[string]*identityContribution + about *identityContribution +} + +func newIdentityUnion() *identityUnion { + return &identityUnion{identityIndex: newIdentityIndex(), perDoc: map[string]*identityContribution{}} +} + +// tableOf is doc's identity table, as gathered. +func (u *identityUnion) tableOf(doc string) *identity.Table { + if c := u.perDoc[doc]; c != nil { + return c.table + } + return nil +} + +// regather replaces doc's contribution — its own elements — with a fresh +// gather, none when doc is no workspace document, naming the keys it moved. +func (u *identityUnion) regather(ctx *Context, g *Gathers, doc string, changed map[string]bool) { + var cur *identityContribution + if g.docs[doc] { + g.gather(ctx, doc, func(root *symbols.Scope) { + table := identity.Build(ctx.Model(), ctx.Resolver(), root) + cur = contribute(table, func(info *identity.Info) bool { + return info.Symbol.DocName == doc + }) + }) + } + u.replace(u.perDoc[doc], cur, changed) + if cur == nil { + delete(u.perDoc, doc) + } else { + u.perDoc[doc] = cur + } +} + +// regatherAbout replaces the contribution of the `about`-annotated elements no +// workspace document declares. +func (u *identityUnion) regatherAbout(ctx *Context, g *Gathers, changed map[string]bool) { + var cur *identityContribution + ctx.Resolver().Gather(aboutGather, func() { + table := identity.Build(ctx.Model(), ctx.Resolver()) + cur = contribute(table, func(info *identity.Info) bool { + return !g.docs[info.Symbol.DocName] + }) + }) + u.replace(u.about, cur, changed) + u.about = cur +} + +// replace swaps one contribution for another in the index, naming in changed, +// when it is not nil, the judgments that read a group the swap moved. +func (u *identityUnion) replace(old, cur *identityContribution, changed map[string]bool) { + var moved []identityKey + if changed != nil { + before, after := old.facts(), cur.facts() + for k, spelled := range before { + if after[k] != spelled { + moved = append(moved, k) + } + } + for k := range after { + if _, had := before[k]; !had { + moved = append(moved, k) + } + } + } + readers := map[string]bool{} + for _, k := range moved { + u.readers(k, readers) + } + if old != nil { + for _, info := range old.infos { + u.remove(info) + } + } + if cur != nil { + for _, info := range cur.infos { + u.insert(info) + } + } + for _, k := range moved { + u.readers(k, readers) + } + if len(moved) > 0 { + for doc := range readers { + changed[identityJudgment(doc)] = true + } + changed[identityJudgments] = true + } +} + +// judged reads the union for the identity audit of doc, whose table it returns +// when doc is a workspace document. +func (u *identityUnion) judged(r *resolve.Resolver, doc string) *identity.Table { + if t := u.tableOf(doc); t != nil { + r.ReadName(identityJudgment(doc)) + return t + } + r.ReadName(identityJudgments) + return nil +} + +// including is the union with one more table's elements filed in, for a +// document that is no workspace document — a library one — judged against +// the workspace: the union is copied, not changed. +func (u *identityUnion) including(table *identity.Table) *identityIndex { + x := newIdentityIndex() + for k, group := range u.byID { + x.byID[k] = append([]*identity.Info(nil), group...) + } + for k, hits := range u.hits { + x.hits[k] = append([]identityHit(nil), hits...) + } + for _, sym := range table.Symbols() { + if info, ok := table.Info(sym); ok && !u.holds(info) { + x.insert(info) + } + } + return x +} + +// holds reports whether the union files an element for info's symbol. +func (u *identityUnion) holds(info *identity.Info) bool { + for _, o := range u.byID[keyOf(info)] { + if o.Symbol == info.Symbol { + return true + } + } + return false +} diff --git a/internal/core/passes/identity_metadata.go b/internal/core/passes/identity_metadata.go index 705e43412e..7a2166a169 100644 --- a/internal/core/passes/identity_metadata.go +++ b/internal/core/passes/identity_metadata.go @@ -32,24 +32,21 @@ func (IdentityMetadataPass) Run(ctx *Context, name string, root *ast.RootNamespa return nil } // A project scope may span workspace documents, so uniqueness is judged - // over all of them; each document only reports its own elements. - roots := []*symbols.Scope{rootScope} - for _, doc := range ctx.Index.WorkspaceDocuments() { - if doc == name { - continue - } - if r := ctx.Index.DocumentRoot(doc); r != nil { - roots = append(roots, r) - } + // over the union of their gathers; each document only reports its own elements. + union := ctx.Gathers().identitiesOf(ctx) + c := &identityChecker{space: union.identityIndex, docRoot: rootScope} + if c.table = union.judged(ctx.Resolver(), name); c.table == nil { + c.table = identity.Build(ctx.Model(), ctx.Resolver(), rootScope) + c.space = union.including(c.table) } - table := identity.Build(ctx.Model(), ctx.Resolver(), roots...) - c := &identityChecker{table: table, docRoot: rootScope} c.check() return c.diags } type identityChecker struct { + // table is the document's own identities; space the id space they are judged in. table *identity.Table + space *identityIndex docRoot *symbols.Scope diags []Diagnostic } @@ -77,8 +74,6 @@ func (c *identityChecker) inDoc(scope *symbols.Scope) bool { } func (c *identityChecker) check() { - scopes := make(map[string][]*identity.Info) - var keys []string for _, sym := range c.table.Symbols() { info, ok := c.table.Info(sym) if !ok { @@ -97,14 +92,7 @@ func (c *identityChecker) check() { if info.Scope != nil && info.Scope.Symbol == sym { c.checkScopeConflicts(info) } - key := scopeKey(info) - if _, seen := scopes[key]; !seen { - keys = append(keys, key) - } - scopes[key] = append(scopes[key], info) - } - for _, key := range keys { - c.checkScope(scopes[key]) + c.checkIDSpace(info) } } @@ -202,53 +190,51 @@ func projectName(d identity.ScopeDeclaration) string { return fmt.Sprintf("project %q of org %q", d.ProjectID, d.Org) } -// checkScope validates the generated id space of one project scope: duplicate -// effective ids, and declared ids that land on another element's membership -// (`…_om`) or expression-node (`…_p…`) id. -func (c *identityChecker) checkScope(infos []*identity.Info) { - byID := make(map[string][]*identity.Info) - for _, info := range infos { - byID[info.EffectiveID] = append(byID[info.EffectiveID], info) - } - for _, group := range byID { - // Distinct qualified names never derive one id, so a group of derived - // ids is one name seen twice, not an identity conflict. - if len(group) < 2 || !anyAnnotated(group) { - continue - } +// checkIDSpace validates an element's place in the generated id space of its +// project scope: an effective id another element shares, a declared id that +// lands on another element's membership (`…_om`) or expression-node (`…_p…`) +// id, and another element's declared id landing on this one's. +func (c *identityChecker) checkIDSpace(info *identity.Info) { + key := keyOf(info) + // Distinct qualified names never derive one id, so a group of derived + // ids is one name seen twice, not an identity conflict. + if group := c.space.group(key); len(group) >= 2 && anyAnnotated(group) { names := make([]string, 0, len(group)) - for _, info := range group { - names = append(names, info.FQN) + for _, o := range group { + names = append(names, o.FQN) } sort.Strings(names) - for _, info := range group { - if span, ok := c.reportSite(info); ok { - c.errorf(span, duplicateIDCode, - "duplicate element id %q in one project scope: %s", - info.EffectiveID, strings.Join(names, " and ")) - } + if span, ok := c.reportSite(info); ok { + c.errorf(span, duplicateIDCode, + "duplicate element id %q in one project scope: %s", + info.EffectiveID, strings.Join(names, " and ")) } } - for _, info := range infos { - for _, d := range info.Declarations { - if !d.Declared || d.ID == "" { - continue - } - if base, ok := strings.CutSuffix(d.ID, "_om"); ok { - c.reportDerivedCollision(info, d, byID[base], "the owning-membership id") - } - for i := strings.Index(d.ID, "_p"); i >= 0; { - if expressionPositions(d.ID[i+2:]) { - c.reportDerivedCollision(info, d, byID[d.ID[:i]], "an expression-node id") - } - next := strings.Index(d.ID[i+1:], "_p") - if next < 0 { - break + for _, d := range info.Declarations { + if !d.Declared || d.ID == "" || !c.declInDocument(d) { + continue + } + for _, t := range derivedTargets(d.ID) { + for _, owner := range c.space.group(identityKey{key.scope, t.base}) { + if owner.Symbol == info.Symbol { + continue } - i += 1 + next + c.errorf(d.Span, duplicateIDCode, + "element id %q of %s collides with %s of %s", + d.ID, info.FQN, t.space, owner.FQN) } } } + for _, hit := range c.space.hitsOn(key) { + if hit.info.Symbol == info.Symbol { + continue + } + if span, ok := c.reportSite(info); ok { + c.errorf(span, duplicateIDCode, + "%s of %s collides with element id %q of %s", + hit.space, info.FQN, hit.decl.ID, hit.info.FQN) + } + } } // expressionPositions reports whether rest is a chain of `_p`-separated @@ -276,26 +262,6 @@ func anyAnnotated(group []*identity.Info) bool { return false } -// reportDerivedCollision errors on both elements when a declared id lands in -// the derived id space another element generates. -func (c *identityChecker) reportDerivedCollision(info *identity.Info, d identity.Declaration, owners []*identity.Info, space string) { - for _, owner := range owners { - if owner == info { - continue - } - if c.declInDocument(d) { - c.errorf(d.Span, duplicateIDCode, - "element id %q of %s collides with %s of %s", - d.ID, info.FQN, space, owner.FQN) - } - if span, ok := c.reportSite(owner); ok { - c.errorf(span, duplicateIDCode, - "%s of %s collides with element id %q of %s", - space, owner.FQN, d.ID, info.FQN) - } - } -} - // firstInDocument is the element's first ElementId annotation declared in the // document under validation. func (c *identityChecker) firstInDocument(info *identity.Info) (identity.Declaration, bool) { diff --git a/internal/core/passes/mosa.go b/internal/core/passes/mosa.go index d3cb79cb8d..2457aab0ed 100644 --- a/internal/core/passes/mosa.go +++ b/internal/core/passes/mosa.go @@ -43,15 +43,12 @@ func (MOSAPass) Run(ctx *Context, name string, root *ast.RootNamespace) []Diagno if a == nil { return nil } - gathered := map[*symbols.Scope]bool{} - for _, doc := range ctx.Index.WorkspaceDocuments() { - if r := ctx.Index.DocumentRoot(doc); r != nil && !gathered[r] { - gathered[r] = true - a.gather(r) - } - } - if !gathered[rootScope] { + a.union = ctx.Gathers().mosaOf(ctx, a) + if !ctx.Gathers().has(name) { + a.local = newMOSAFacts() + a.facts = a.local a.gather(rootScope) + a.facts = nil } a.check(rootScope) return a.diags @@ -132,15 +129,14 @@ type mosaAudit struct { metadata map[mosaMetadataKind]*symbols.Symbol // metadataKinds memoizes metadataKindOf by the annotation type's qualified name. metadataKinds map[string]mosaMetadataKind - // present records the kinds the workspace declares at all. - present map[mosaKind]bool - // marks caches each element's annotations; anyMarks records those stated anywhere. - marks map[*symbols.Symbol]mosaMarks - anyMarks mosaMarks - // conformant: elements at a #conformant end; satisfiers: elements a satisfy traces. - conformant map[symbols.ElementKey]bool - satisfiers map[symbols.ElementKey]bool - kinds map[*symbols.Symbol]mosaKind + // facts is what a gather under way records; a check judges over union and + // local, the facts of a root outside the gathered documents. + facts *mosaFacts + union *mosaUnion + local *mosaFacts + // marks caches each element's annotations. + marks map[*symbols.Symbol]mosaMarks + kinds map[*symbols.Symbol]mosaKind // typeKinds memoizes kindOfType: one type classifies every feature it types. typeKinds map[*symbols.Symbol]mosaKind diags []Diagnostic @@ -155,10 +151,7 @@ func newMOSAAudit(ctx *Context) *mosaAudit { definitions: map[mosaKind]*symbols.Symbol{}, metadata: map[mosaMetadataKind]*symbols.Symbol{}, metadataKinds: map[string]mosaMetadataKind{}, - present: map[mosaKind]bool{}, marks: map[*symbols.Symbol]mosaMarks{}, - conformant: map[symbols.ElementKey]bool{}, - satisfiers: map[symbols.ElementKey]bool{}, kinds: map[*symbols.Symbol]mosaKind{}, typeKinds: map[*symbols.Symbol]mosaKind{}, } @@ -291,12 +284,9 @@ func (a *mosaAudit) effectiveMarks(sym *symbols.Symbol) mosaMarks { func (a *mosaAudit) gather(root *symbols.Scope) { w8dWalkSymbols(a.ctx, root, func(sym *symbols.Symbol) { if kind := a.kindOf(sym); kind != mosaNone { - a.present[kind] = true + a.facts.present[kind] = true } - m := a.marksOf(sym) - a.anyMarks.dataRights = a.anyMarks.dataRights || m.dataRights - a.anyMarks.proprietary = a.anyMarks.proprietary || m.proprietary - a.anyMarks.interfaceControl = a.anyMarks.interfaceControl || m.interfaceControl + a.facts.note(a.marksOf(sym)) usage, ok := sym.Decl.(*ast.Usage) if !ok { return @@ -336,7 +326,7 @@ func (a *mosaAudit) gatherConformance(sym *symbols.Symbol) { return } for _, target := range conformant { - a.conformant[symbols.KeyOf(target)] = true + a.facts.conformant[symbols.KeyOf(target)] = true } } @@ -353,14 +343,14 @@ func (a *mosaAudit) gatherSatisfaction(sym *symbols.Symbol, usage *ast.Usage) { } named = true if target, ok := a.ctx.Resolver().ResolveTarget(sym.OwnerScope, rel.Target); ok && target != nil { - a.satisfiers[symbols.KeyOf(target)] = true + a.facts.satisfiers[symbols.KeyOf(target)] = true } } if named || sym.OwnerScope == nil { return } if owner := sym.OwnerScope.Owner(); owner != nil && owner.IsFeature() { - a.satisfiers[symbols.KeyOf(owner)] = true + a.facts.satisfiers[symbols.KeyOf(owner)] = true } } @@ -464,7 +454,7 @@ func (a *mosaAudit) checkComponent(sym *symbols.Symbol) { if kind != mosaMajorSystemComponent && kind != mosaModularSystem { return } - if !a.anyMarks.dataRights || a.effectiveMarks(sym).dataRights { + if !a.anyMarked(mosaMarkDataRights) || a.effectiveMarks(sym).dataRights { return } a.report(sym, CodeMOSAComponentNoDataRights, fmt.Sprintf( @@ -476,27 +466,27 @@ func (a *mosaAudit) checkComponent(sym *symbols.Symbol) { // interface control authority and satisfy a requirement, each once the model states such facts. func (a *mosaAudit) checkInterface(sym *symbols.Symbol) { marks := a.effectiveMarks(sym) - if a.present[mosaStandard] && !marks.proprietary && !a.anyOf(sym, a.conformant) { + if a.hasKind(mosaStandard) && !marks.proprietary && !a.anyOf(sym, a.isConformant) { a.report(sym, CodeMOSAInterfaceNoStandard, "This modular system interface conforms to no standard: the model declares standards, so MOSA expects a #conformance connection naming it at a #conformant end and a standard at a #conformsTo end, or a @Proprietary annotation with the rationale for it.") } - if a.anyMarks.interfaceControl && !marks.interfaceControl { + if a.anyMarked(mosaMarkInterfaceControl) && !marks.interfaceControl { a.report(sym, CodeMOSAInterfaceNoControl, "This modular system interface names no interface control authority: the model names one for other interfaces, so MOSA expects an @InterfaceControl annotation on it or its definition.") } - if a.present[mosaInterfaceRequirement] && !a.anyOf(sym, a.satisfiers) { + if a.hasKind(mosaInterfaceRequirement) && !a.anyOf(sym, a.isSatisfier) { a.report(sym, CodeMOSAInterfaceNotTraced, "This modular system interface satisfies no requirement: the model declares interface requirements, so MOSA expects a `satisfy` naming it (or its definition) after `by`.") } } -// anyOf reports whether sym or anything it specializes is in set. -func (a *mosaAudit) anyOf(sym *symbols.Symbol, set map[symbols.ElementKey]bool) bool { - if set[symbols.KeyOf(sym)] { +// anyOf reports whether sym or anything it specializes is in the set in reads. +func (a *mosaAudit) anyOf(sym *symbols.Symbol, in func(symbols.ElementKey) bool) bool { + if in(symbols.KeyOf(sym)) { return true } for _, t := range a.model.AllSupertypes(sym) { - if set[symbols.KeyOf(t)] { + if in(symbols.KeyOf(t)) { return true } } @@ -512,7 +502,7 @@ type mosaAttachment struct { // checkBoundary expects a connector joining two distinct components to be a modular // system interface once the model designates any; a party nested in another is part of it. func (a *mosaAudit) checkBoundary(sym *symbols.Symbol, usage *ast.Usage) { - if !a.present[mosaModularSystemInterface] { + if !a.hasKind(mosaModularSystemInterface) { return } parties := map[*symbols.Symbol]bool{} diff --git a/internal/core/passes/mosa_gather.go b/internal/core/passes/mosa_gather.go new file mode 100644 index 0000000000..49750b03a6 --- /dev/null +++ b/internal/core/passes/mosa_gather.go @@ -0,0 +1,148 @@ +package passes + +import ( + "strconv" + + "github.com/Open-MBEE/OpenSysML/internal/core/symbols" +) + +// mosaFacts is what the MOSA audit gathers from one document: the kinds and +// annotations it declares and the conformances and satisfactions it states. +type mosaFacts struct { + present map[mosaKind]bool + marks map[mosaMark]bool + // conformant: elements at a #conformant end; satisfiers: elements a satisfy traces. + conformant map[symbols.ElementKey]bool + satisfiers map[symbols.ElementKey]bool +} + +// mosaMark is one of the MOSA annotation families an element may carry. +type mosaMark int + +const ( + mosaMarkDataRights mosaMark = iota + mosaMarkProprietary + mosaMarkInterfaceControl +) + +func newMOSAFacts() *mosaFacts { + return &mosaFacts{ + present: map[mosaKind]bool{}, + marks: map[mosaMark]bool{}, + conformant: map[symbols.ElementKey]bool{}, + satisfiers: map[symbols.ElementKey]bool{}, + } +} + +// note records the marks an element carries among those stated anywhere. +func (f *mosaFacts) note(m mosaMarks) { + if m.dataRights { + f.marks[mosaMarkDataRights] = true + } + if m.proprietary { + f.marks[mosaMarkProprietary] = true + } + if m.interfaceControl { + f.marks[mosaMarkInterfaceControl] = true + } +} + +// mosaUnion is the MOSA facts of every workspace document, counted per document +// and read by name, as oosemUnion is. +type mosaUnion struct { + perDoc map[string]*mosaFacts + present countSet[mosaKind] + marks countSet[mosaMark] + conformant countSet[symbols.ElementKey] + satisfiers countSet[symbols.ElementKey] +} + +func newMOSAUnion() *mosaUnion { + return &mosaUnion{ + perDoc: map[string]*mosaFacts{}, + present: countSet[mosaKind]{}, + marks: countSet[mosaMark]{}, + conformant: countSet[symbols.ElementKey]{}, + satisfiers: countSet[symbols.ElementKey]{}, + } +} + +func mosaPresentName(k mosaKind) string { return "\x00mosa/present/" + strconv.Itoa(int(k)) } +func mosaMarkName(m mosaMark) string { return "\x00mosa/marks/" + strconv.Itoa(int(m)) } +func mosaConformantName(k symbols.ElementKey) string { + return "\x00mosa/conformant/" + k.String() +} +func mosaSatisfierName(k symbols.ElementKey) string { return "\x00mosa/satisfier/" + k.String() } + +// regather replaces doc's facts with a fresh gather — none when doc is no +// workspace document — naming what the union now answers differently. +func (u *mosaUnion) regather(ctx *Context, g *Gathers, a *mosaAudit, doc string, changed map[string]bool) { + old := u.perDoc[doc] + var cur *mosaFacts + if g.docs[doc] { + cur = newMOSAFacts() + a.facts = cur + g.gather(ctx, doc, a.gather) + a.facts = nil + } + move(u.present, old.presentSet(), cur.presentSet(), mosaPresentName, changed) + move(u.marks, old.markSet(), cur.markSet(), mosaMarkName, changed) + move(u.conformant, old.conformantSet(), cur.conformantSet(), mosaConformantName, changed) + move(u.satisfiers, old.satisfierSet(), cur.satisfierSet(), mosaSatisfierName, changed) + if cur == nil { + delete(u.perDoc, doc) + } else { + u.perDoc[doc] = cur + } +} + +func (f *mosaFacts) presentSet() map[mosaKind]bool { + if f == nil { + return nil + } + return f.present +} + +func (f *mosaFacts) markSet() map[mosaMark]bool { + if f == nil { + return nil + } + return f.marks +} + +func (f *mosaFacts) conformantSet() map[symbols.ElementKey]bool { + if f == nil { + return nil + } + return f.conformant +} + +func (f *mosaFacts) satisfierSet() map[symbols.ElementKey]bool { + if f == nil { + return nil + } + return f.satisfiers +} + +// The reads a judgment makes: the union by name, so a regather that flips the +// answer drops the reader, plus the facts of a root gathered for this run alone. + +func (a *mosaAudit) hasKind(k mosaKind) bool { + a.ctx.Resolver().ReadName(mosaPresentName(k)) + return a.union.present.has(k) || a.local.presentSet()[k] +} + +func (a *mosaAudit) anyMarked(m mosaMark) bool { + a.ctx.Resolver().ReadName(mosaMarkName(m)) + return a.union.marks.has(m) || a.local.markSet()[m] +} + +func (a *mosaAudit) isConformant(k symbols.ElementKey) bool { + a.ctx.Resolver().ReadName(mosaConformantName(k)) + return a.union.conformant.has(k) || a.local.conformantSet()[k] +} + +func (a *mosaAudit) isSatisfier(k symbols.ElementKey) bool { + a.ctx.Resolver().ReadName(mosaSatisfierName(k)) + return a.union.satisfiers.has(k) || a.local.satisfierSet()[k] +} diff --git a/internal/core/passes/oosem_gather.go b/internal/core/passes/oosem_gather.go new file mode 100644 index 0000000000..ef7d887e92 --- /dev/null +++ b/internal/core/passes/oosem_gather.go @@ -0,0 +1,155 @@ +package passes + +import ( + "strconv" + + "github.com/Open-MBEE/OpenSysML/internal/core/symbols" +) + +// oosemFacts is what the OOSEM audit gathers from one document: the kinds it +// declares and the derivation, satisfaction and allocation relationships it states. +type oosemFacts struct { + present map[oosemKind]bool + // derived holds each requirement at a `#derive` end with a kind of an + // `#original` end of the same `#derivation`. + derived map[oosemDerivation]bool + satisfied map[symbols.ElementKey]bool + allocated map[symbols.ElementKey]bool + // satisfies holds true when the document states any satisfy. + satisfies map[bool]bool +} + +// oosemDerivation is one requirement derived from a requirement of one kind. +type oosemDerivation struct { + requirement symbols.ElementKey + from oosemKind +} + +func newOOSEMFacts() *oosemFacts { + return &oosemFacts{ + present: map[oosemKind]bool{}, + derived: map[oosemDerivation]bool{}, + satisfied: map[symbols.ElementKey]bool{}, + allocated: map[symbols.ElementKey]bool{}, + satisfies: map[bool]bool{}, + } +} + +// oosemUnion is the OOSEM facts of every workspace document, counted per +// document so a document's regather moves only what it alone stated. Judgments +// read it by name (see resolve.Resolver.ReadName), so a regather that flips an +// answer drops exactly the documents that read it. +type oosemUnion struct { + perDoc map[string]*oosemFacts + present countSet[oosemKind] + derived countSet[oosemDerivation] + satisfied countSet[symbols.ElementKey] + allocated countSet[symbols.ElementKey] + satisfies countSet[bool] +} + +func newOOSEMUnion() *oosemUnion { + return &oosemUnion{ + perDoc: map[string]*oosemFacts{}, + present: countSet[oosemKind]{}, + derived: countSet[oosemDerivation]{}, + satisfied: countSet[symbols.ElementKey]{}, + allocated: countSet[symbols.ElementKey]{}, + satisfies: countSet[bool]{}, + } +} + +func oosemPresentName(k oosemKind) string { return "\x00oosem/present/" + strconv.Itoa(int(k)) } +func oosemDerivedName(d oosemDerivation) string { + return "\x00oosem/derived/" + d.requirement.String() +} +func oosemSatisfiedName(k symbols.ElementKey) string { return "\x00oosem/satisfied/" + k.String() } +func oosemAllocatedName(k symbols.ElementKey) string { return "\x00oosem/allocated/" + k.String() } +func oosemSatisfiesName(bool) string { return "\x00oosem/satisfies" } + +// regather replaces doc's facts with a fresh gather — none when doc is no +// workspace document — naming what the union now answers differently. +func (u *oosemUnion) regather(ctx *Context, g *Gathers, a *oosemAudit, doc string, changed map[string]bool) { + old := u.perDoc[doc] + var cur *oosemFacts + if g.docs[doc] { + cur = newOOSEMFacts() + a.facts = cur + g.gather(ctx, doc, a.gather) + a.facts = nil + } + move(u.present, old.presentSet(), cur.presentSet(), oosemPresentName, changed) + move(u.derived, old.derivedSet(), cur.derivedSet(), oosemDerivedName, changed) + move(u.satisfied, old.satisfiedSet(), cur.satisfiedSet(), oosemSatisfiedName, changed) + move(u.allocated, old.allocatedSet(), cur.allocatedSet(), oosemAllocatedName, changed) + move(u.satisfies, old.satisfiesSet(), cur.satisfiesSet(), oosemSatisfiesName, changed) + if cur == nil { + delete(u.perDoc, doc) + } else { + u.perDoc[doc] = cur + } +} + +func (f *oosemFacts) presentSet() map[oosemKind]bool { + if f == nil { + return nil + } + return f.present +} + +func (f *oosemFacts) derivedSet() map[oosemDerivation]bool { + if f == nil { + return nil + } + return f.derived +} + +func (f *oosemFacts) satisfiedSet() map[symbols.ElementKey]bool { + if f == nil { + return nil + } + return f.satisfied +} + +func (f *oosemFacts) allocatedSet() map[symbols.ElementKey]bool { + if f == nil { + return nil + } + return f.allocated +} + +func (f *oosemFacts) satisfiesSet() map[bool]bool { + if f == nil { + return nil + } + return f.satisfies +} + +// The reads a judgment makes: the union by name, so a regather that flips the +// answer drops the reader, plus the facts of a root gathered for this run alone. + +func (a *oosemAudit) hasKind(k oosemKind) bool { + a.ctx.Resolver().ReadName(oosemPresentName(k)) + return a.union.present.has(k) || a.local.presentSet()[k] +} + +func (a *oosemAudit) derivedFrom(requirement symbols.ElementKey, from oosemKind) bool { + d := oosemDerivation{requirement, from} + a.ctx.Resolver().ReadName(oosemDerivedName(d)) + return a.union.derived.has(d) || a.local.derivedSet()[d] +} + +func (a *oosemAudit) isSatisfied(k symbols.ElementKey) bool { + a.ctx.Resolver().ReadName(oosemSatisfiedName(k)) + return a.union.satisfied.has(k) || a.local.satisfiedSet()[k] +} + +func (a *oosemAudit) isAllocated(k symbols.ElementKey) bool { + a.ctx.Resolver().ReadName(oosemAllocatedName(k)) + return a.union.allocated.has(k) || a.local.allocatedSet()[k] +} + +func (a *oosemAudit) statesSatisfaction() bool { + a.ctx.Resolver().ReadName(oosemSatisfiesName(true)) + return a.union.satisfies.has(true) || a.local.satisfiesSet()[true] +} diff --git a/internal/core/passes/oosem_method.go b/internal/core/passes/oosem_method.go index 15af01c4fa..14dcc03fb3 100644 --- a/internal/core/passes/oosem_method.go +++ b/internal/core/passes/oosem_method.go @@ -41,15 +41,12 @@ func (OOSEMMethodPass) Run(ctx *Context, name string, root *ast.RootNamespace) [ if a == nil { return nil } - gathered := map[*symbols.Scope]bool{} - for _, doc := range ctx.Index.WorkspaceDocuments() { - if r := ctx.Index.DocumentRoot(doc); r != nil && !gathered[r] { - gathered[r] = true - a.gather(r) - } - } - if !gathered[rootScope] { + a.union = ctx.Gathers().oosemOf(ctx, a) + if !ctx.Gathers().has(name) { + a.local = newOOSEMFacts() + a.facts = a.local a.gather(rootScope) + a.facts = nil } a.check(rootScope) return a.diags @@ -125,17 +122,12 @@ type oosemAudit struct { model *semantics.Model // definitions holds the library definition of each OOSEM kind. definitions map[oosemKind][]*symbols.Symbol - // present records the kinds the workspace declares at all. - present map[oosemKind]bool - // derivedFrom holds, per requirement at a `#derive` end, the kinds of the - // `#original` ends of the same `#derivation`; satisfied holds the requirements - // a `satisfy` names; allocated the sources allocated to a node or physical component. - derivedFrom map[symbols.ElementKey]map[oosemKind]bool - satisfied map[symbols.ElementKey]bool - allocated map[symbols.ElementKey]bool - // statesSatisfaction reports whether the workspace states any satisfy. - statesSatisfaction bool - kinds map[*symbols.Symbol]oosemKind + // facts is what a gather under way records; a check judges over union and + // local, the facts of a root outside the gathered documents. + facts *oosemFacts + union *oosemUnion + local *oosemFacts + kinds map[*symbols.Symbol]oosemKind // typeKinds memoizes kindOfType: one type classifies every feature it types. typeKinds map[*symbols.Symbol]oosemKind diags []Diagnostic @@ -149,10 +141,6 @@ func newOOSEMAudit(ctx *Context) *oosemAudit { ctx: ctx, model: ctx.Model(), definitions: map[oosemKind][]*symbols.Symbol{}, - present: map[oosemKind]bool{}, - derivedFrom: map[symbols.ElementKey]map[oosemKind]bool{}, - satisfied: map[symbols.ElementKey]bool{}, - allocated: map[symbols.ElementKey]bool{}, kinds: map[*symbols.Symbol]oosemKind{}, typeKinds: map[*symbols.Symbol]oosemKind{}, } @@ -222,7 +210,7 @@ func (a *oosemAudit) gather(root *symbols.Scope) { w8dWalkSymbols(a.ctx, root, func(sym *symbols.Symbol) { usage, isUsage := sym.Decl.(*ast.Usage) if kind := a.kindOf(sym); kind != oosemNone { - a.present[kind] = true + a.facts.present[kind] = true } if !isUsage { return @@ -259,11 +247,8 @@ func (a *oosemAudit) gatherDerivation(sym *symbols.Symbol) { } for _, d := range derived { key := symbols.KeyOf(d) - if a.derivedFrom[key] == nil { - a.derivedFrom[key] = map[oosemKind]bool{} - } for _, o := range originals { - a.derivedFrom[key][a.kindOf(o)] = true + a.facts.derived[oosemDerivation{key, a.kindOf(o)}] = true } } } @@ -300,7 +285,7 @@ func (a *oosemAudit) gatherAllocation(sym *symbols.Symbol, usage *ast.Usage) { return } for _, s := range source { - a.allocated[symbols.KeyOf(s)] = true + a.facts.allocated[symbols.KeyOf(s)] = true } } @@ -337,8 +322,8 @@ func (a *oosemAudit) gatherSatisfaction(sym *symbols.Symbol, usage *ast.Usage) { return } if usage.DeclaresRequirement { - a.statesSatisfaction = true - a.satisfied[symbols.KeyOf(sym)] = true + a.facts.satisfies[true] = true + a.facts.satisfied[symbols.KeyOf(sym)] = true return } for _, rel := range usage.Relationships { @@ -349,8 +334,8 @@ func (a *oosemAudit) gatherSatisfaction(sym *symbols.Symbol, usage *ast.Usage) { if !ok || target == nil || isViewpoint(target) { continue } - a.statesSatisfaction = true - a.satisfied[symbols.KeyOf(target)] = true + a.facts.satisfies[true] = true + a.facts.satisfied[symbols.KeyOf(target)] = true } } @@ -424,12 +409,12 @@ func (a *oosemAudit) checkRequirement(sym *symbols.Symbol) { return } key := symbols.KeyOf(sym) - if a.present[from] && !a.derivedFrom[key][from] { + if a.hasKind(from) && !a.derivedFrom(key, from) { a.report(sym, CodeOOSEMRequirementNotDerived, fmt.Sprintf( "This %s derives from no %s: the model declares %ss, so OOSEM expects a #derivation connection naming it at a #derive end and a %s at an #original end.", oosemKindNames[kind], oosemKindNames[from], oosemKindNames[from], oosemKindNames[from])) } - if kind != oosemMissionRequirement && a.statesSatisfaction && !a.satisfied[key] { + if kind != oosemMissionRequirement && a.statesSatisfaction() && !a.isSatisfied(key) { a.report(sym, CodeOOSEMRequirementNotSatisfied, fmt.Sprintf( "This %s is satisfied by nothing: the model states satisfactions, so OOSEM expects a `satisfy` naming it.", oosemKindNames[kind])) @@ -443,19 +428,19 @@ func (a *oosemAudit) checkLogicalComponent(sym *symbols.Symbol) { if a.kindOf(sym) != oosemLogicalComponent { return } - if !a.present[oosemNode] && !a.present[oosemPhysicalComponent] { + if !a.hasKind(oosemNode) && !a.hasKind(oosemPhysicalComponent) { return } - if a.allocated[symbols.KeyOf(sym)] { + if a.isAllocated(symbols.KeyOf(sym)) { return } for _, t := range a.model.FeatureTypeSet(sym) { - if a.allocated[symbols.KeyOf(t)] { + if a.isAllocated(symbols.KeyOf(t)) { return } } for scope := sym.OwnerScope; scope != nil; scope = scope.Parent() { - if owner := scope.Owner(); owner != nil && a.allocated[symbols.KeyOf(owner)] { + if owner := scope.Owner(); owner != nil && a.isAllocated(symbols.KeyOf(owner)) { return } } diff --git a/internal/core/passes/pass.go b/internal/core/passes/pass.go index 0008902253..114d890f9d 100644 --- a/internal/core/passes/pass.go +++ b/internal/core/passes/pass.go @@ -58,6 +58,7 @@ type Context struct { resolver *resolve.Resolver model *semantics.Model + gathers *Gathers w8dCache map[*symbols.Scope][]*symbols.Symbol w8cCache map[*symbols.Scope][]*symbols.Symbol // failures is where the tiers below the pass now running found blocking @@ -89,6 +90,21 @@ func NewContextWithOptions(name string, kind source.Kind, idx *symbols.Index, return &Context{Name: name, Kind: kind, Index: idx, ParseDiagnostics: parseDiags, Options: opts} } +// Share hands the context a resolver, model and gathers that outlive it — a +// workspace's, kept across analyses — instead of the fresh ones it would make. +func (c *Context) Share(resolver *resolve.Resolver, model *semantics.Model, gathers *Gathers) { + c.resolver, c.model, c.gathers = resolver, model, gathers +} + +// Gathers is what the workspace-wide audits gathered per document, shared +// across analyses by a workspace and made afresh for a context outside one. +func (c *Context) Gathers() *Gathers { + if c.gathers == nil { + c.gathers = NewGathers() + } + return c.gathers +} + // setFailures records the blocking spans of the tiers below the pass about to // run. Only the registry calls it, once per pass. func (c *Context) setFailures(spans []source.Span) { c.failures = spans } diff --git a/internal/core/resolve/accept_payload.go b/internal/core/resolve/accept_payload.go index 8951168b42..ec43c8f90d 100644 --- a/internal/core/resolve/accept_payload.go +++ b/internal/core/resolve/accept_payload.go @@ -13,9 +13,12 @@ func (r *Resolver) acceptPayload(scope *symbols.Scope, name string) (*symbols.Sy if scope == nil || name == "" { return nil, false } + r.EnterDoc(symbols.DocNameOf(scope)) + defer r.LeaveDoc() payloads, done := r.payloads[scope] if !done { payloads = acceptPayloadsIn(scope) + journalNew(r, r.payloads, scope, scope.Node()) r.payloads[scope] = payloads } sym, ok := payloads[name] diff --git a/internal/core/resolve/alias.go b/internal/core/resolve/alias.go index 01aa63c9d6..959d23baa5 100644 --- a/internal/core/resolve/alias.go +++ b/internal/core/resolve/alias.go @@ -39,6 +39,8 @@ func (r *Resolver) ResolveAliasTarget(sym *symbols.Symbol) (*symbols.Symbol, boo if sym == nil { return nil, false } + r.EnterDoc(sym.DocName) + defer r.LeaveDoc() if cached, ok := r.aliasTargets[sym]; ok { return cached.sym, cached.ok } @@ -49,6 +51,7 @@ func (r *Resolver) ResolveAliasTarget(sym *symbols.Symbol) (*symbols.Symbol, boo defer delete(r.resolvingAlias, sym) target, ok := r.resolveAliasTarget(sym) + journalNew(r, r.aliasTargets, sym, sym.Decl) r.aliasTargets[sym] = resolution{sym: target, ok: ok} return target, ok } diff --git a/internal/core/resolve/document.go b/internal/core/resolve/document.go index 12fa3c1735..0e89840c0d 100644 --- a/internal/core/resolve/document.go +++ b/internal/core/resolve/document.go @@ -1041,6 +1041,8 @@ func (r *Resolver) resolveRedefinition(scope *symbols.Scope, qn *ast.QualifiedNa if qn == nil || len(qn.Parts) == 0 { return } + r.EnterDoc(symbols.DocNameOf(scope)) + defer r.LeaveDoc() if _, done := r.memo[qn]; done { return } @@ -1190,6 +1192,8 @@ func (r *Resolver) resolveRedefinedChain(scope *symbols.Scope, fc *ast.FeatureCh if fc == nil { return nil, false } + r.EnterDoc(symbols.DocNameOf(scope)) + defer r.LeaveDoc() if res, done := r.featureChains[featureChainKey{scope: scope, node: fc}]; done { return res.sym, res.ok } @@ -1539,6 +1543,8 @@ func (r *Resolver) resolveFeatureChain(scope *symbols.Scope, fc *ast.FeatureChai if fc == nil { return nil } + r.EnterDoc(symbols.DocNameOf(scope)) + defer r.LeaveDoc() key := featureChainKey{scope: scope, node: fc} if res, done := r.featureChains[key]; done { return res.sym diff --git a/internal/core/resolve/filter.go b/internal/core/resolve/filter.go index 63e2bb4f1b..12d839e0d3 100644 --- a/internal/core/resolve/filter.go +++ b/internal/core/resolve/filter.go @@ -58,6 +58,8 @@ func (r *Resolver) namespaceFilters(scope *symbols.Scope) []symbols.ElementFilte if scope == nil { return nil } + r.EnterDoc(symbols.DocNameOf(scope)) + defer r.LeaveDoc() if filters, ok := r.nsFilters[scope]; ok { return filters } @@ -72,6 +74,7 @@ func (r *Resolver) namespaceFilters(scope *symbols.Scope) []symbols.ElementFilte } } } + journalNew(r, r.nsFilters, scope, scope.Node()) r.nsFilters[scope] = filters return filters } @@ -83,6 +86,8 @@ func (r *Resolver) inheritedViewConditions(scope *symbols.Scope) []symbols.Eleme if scope == nil || !isViewSymbol(scope.Owner()) { return nil } + r.EnterDoc(symbols.DocNameOf(scope)) + defer r.LeaveDoc() if filters, ok := r.viewFilters[scope]; ok { return filters } @@ -111,6 +116,7 @@ func (r *Resolver) inheritedViewConditions(scope *symbols.Scope) []symbols.Eleme } queue = append(queue, supers.DirectSupertypes(super)...) } + journalNew(r, r.viewFilters, scope, scope.Node()) r.viewFilters[scope] = filters return filters } diff --git a/internal/core/resolve/frames.go b/internal/core/resolve/frames.go new file mode 100644 index 0000000000..6f5f586ad2 --- /dev/null +++ b/internal/core/resolve/frames.go @@ -0,0 +1,552 @@ +package resolve + +import ( + "reflect" + "sort" + "strings" + + "github.com/Open-MBEE/OpenSysML/internal/core/ast" + "github.com/Open-MBEE/OpenSysML/internal/core/symbols" +) + +// frame owns what the resolver and the side tables joining its lifecycle +// memoize while it is the innermost frame: a document's, or a Scratch call's. +type frame struct { + // doc names the document whose nodes, symbols and scopes the entries are + // keyed by; "" is the frame for state no document owns. + doc string + // transient is set for a Scratch frame: the nodes it disowns when it ends. + transient map[ast.Node]bool + // barrier is set for the frame Untracked pushes: nothing below it is current. + barrier bool + // journal is what the frame drops when it ends; ledgers finds the ledger + // among it for a memo table, by the table's identity. + journal []dropper + ledgers map[uintptr]dropper + // names, namespaces and docs are what the index answered about while the + // frame was innermost; all is set once it enumerated the whole name table, + // after which only names outside it (a judgment's) are worth recording. + names map[string]bool + namespaces map[string]bool + docs map[string]bool + all bool + // deps are the documents whose frames were entered from this one. + deps map[string]bool + // recent are the frames last entered from this one, consulted before the + // maps: an analysis enters a library's frame once per symbol it reads there. + recent [4]*frame + next uint8 +} + +func (f *frame) scratch() bool { return f.transient != nil } + +func (f *frame) drop() { + for _, d := range f.journal { + d.drop() + } +} + +// entries counts what the frame will drop. +func (f *frame) entries() int { + n := 0 + for _, d := range f.journal { + n += d.size() + } + return n +} + +// A dropper is an entry of a frame's journal. +type dropper interface { + drop() + size() int +} + +// dropFunc is a journaled closure. +type dropFunc func() + +func (d dropFunc) drop() { d() } +func (d dropFunc) size() int { return 1 } + +// ledger is a frame's share of one memo table: the keys it wrote there. +type ledger[K comparable, V any] struct { + table map[K]V + keys []K +} + +func (l *ledger[K, V]) drop() { + for _, k := range l.keys { + delete(l.table, k) + } +} + +func (l *ledger[K, V]) size() int { return len(l.keys) } + +// entered is the frame of doc among those recently entered from f, or nil. +func (f *frame) entered(doc string) *frame { + for _, g := range f.recent { + if g != nil && g.doc == doc { + return g + } + } + return nil +} + +func (f *frame) remember(g *frame) { + f.recent[f.next%uint8(len(f.recent))] = g + f.next++ +} + +// stale reports whether ch moved anything the frame's entries were read from. +func (f *frame) stale(ch symbols.Changes) bool { + if ch.Docs[f.doc] || (f.all && ch.Registered()) { + return true + } + if len(f.names) < len(ch.Names) { + for n := range f.names { + if ch.Names[n] { + return true + } + } + } else { + for n := range ch.Names { + if f.names[n] { + return true + } + } + } + for n := range ch.Namespaces { + if f.namespaces[n] { + return true + } + } + for n := range ch.Docs { + if f.docs[n] { + return true + } + } + return false +} + +// Track makes the resolver keep what it memoizes by owning document, so +// Invalidate can drop a document's entries and its dependents' when the index +// changes. The resolver records what it reads from the index from here on. +func (r *Resolver) Track() { + if r.owners != nil { + return + } + r.owners = map[string]*frame{} + r.dependents = map[string]map[string]bool{} + r.idx.TrackChanges() + r.idx.SetReadRecorder(r) +} + +// Tracking reports whether memo entries are journaled by owning document. +func (r *Resolver) Tracking() bool { return r != nil && r.owners != nil } + +// Regatherer is a model keeping per-document gathers: Regather recomputes those +// of the documents that changed and names the shared state whose readers have +// to be dropped (see Invalidate). +type Regatherer interface { + Regather(docs map[string]bool) (changed []string) +} + +// InDocument runs f, the analysis of doc: what it memoizes about doc's own +// nodes is owned by doc, and the documents it reads are doc's dependencies. +// Failures resolved here are reported to f and memoized as reported, so the +// analysis must be the first to resolve doc's references (see Query). +func (r *Resolver) InDocument(doc string, f func()) { + if !r.Tracking() { + f() + return + } + r.Diagnostics = nil + r.EnterDoc(doc) + defer r.LeaveDoc() + f() +} + +// Query runs f, a query made from doc — a hover, a reference, a completion — +// quietly: a failure it resolves is neither reported nor memoized, so doc's +// analysis still reports it. "" is the frame for a query made from no document. +func (r *Resolver) Query(doc string, f func()) { + if !r.Tracking() { + f() + return + } + r.EnterDoc(doc) + defer r.LeaveDoc() + r.aside(f) +} + +// gatherSuffix marks the frame a document's gather runs in (see Gather). +const gatherSuffix = "\x00gather" + +// GatherFrame names the frame doc's gather runs in: apart from the frame of +// doc's analysis, so a judgment dropped for an answer it read does not take +// the gather it had no part in with it. +func GatherFrame(doc string) string { return doc + gatherSuffix } + +// GatheredDoc is the document whose gather frame the name is, if it is one. +func GatheredDoc(frame string) (string, bool) { + return strings.CutSuffix(frame, gatherSuffix) +} + +// Gather runs f, the gathering of doc's facts for a workspace-wide judgment: +// in doc's gather frame, quiet like a Query, and no dependency of the +// enclosing document, whose judgment reads the union of gathers by name instead. +// Invalidate names the gather frames it drops, for the gathers to be redone. +func (r *Resolver) Gather(doc string, f func()) { + if !r.Tracking() { + f() + return + } + r.Untracked(func() { + r.EnterDoc(GatherFrame(doc)) + defer r.LeaveDoc() + r.aside(f) + }) +} + +// docFrame is the frame owning doc, made on first use. +func (r *Resolver) docFrame(doc string) *frame { + f := r.owners[doc] + if f == nil { + f = &frame{doc: doc, all: doc == ""} + r.owners[doc] = f + } + return f +} + +// EnterDoc makes doc the owner of what is memoized until the matching +// LeaveDoc, and records that the enclosing document depends on doc. It is a +// no-op outside Track, so callers pair it with LeaveDoc unconditionally. +func (r *Resolver) EnterDoc(doc string) { + if !r.Tracking() { + return + } + cur := r.cur + if doc == "" && cur != nil { + // Unstamped state stays with whoever computed it. + r.stack = append(r.stack, cur) + return + } + if cur != nil && cur.doc == doc { + r.stack = append(r.stack, cur) + return + } + var f *frame + if cur != nil { + f = cur.entered(doc) + } + if f == nil { + f = r.docFrame(doc) + if cur != nil { + r.depend(cur, doc) + cur.remember(f) + } + } + r.stack = append(r.stack, f) + r.cur = f +} + +// LeaveDoc ends the innermost EnterDoc. +func (r *Resolver) LeaveDoc() { + if !r.Tracking() { + return + } + n := len(r.stack) - 1 + r.stack[n] = nil + r.stack = r.stack[:n] + r.cur = r.topDoc() +} + +// topDoc is the innermost document frame on the stack, nil when none or when +// a barrier is nearer. +func (r *Resolver) topDoc() *frame { + for i := len(r.stack) - 1; i >= 0; i-- { + switch f := r.stack[i]; { + case f.barrier: + return nil + case !f.scratch(): + return f + } + } + return nil +} + +// Untracked runs f with no frame current: what it reads is recorded nowhere and +// what it memoizes is owned by no document. For reads whose answer is kept +// current by other means, such as the list of documents a gather covers. +func (r *Resolver) Untracked(f func()) { + if !r.Tracking() { + f() + return + } + r.stack = append(r.stack, &frame{barrier: true}) + r.cur = nil + defer r.LeaveDoc() + f() +} + +// Depend records that the enclosing document depends on doc: a symbol of doc +// was read, so doc's replacement invalidates what was computed from it. +func (r *Resolver) Depend(doc string) { + if r == nil { + return + } + cur := r.cur + if cur == nil || doc == "" || doc == cur.doc || cur.entered(doc) != nil { + return + } + r.depend(cur, doc) + cur.remember(r.docFrame(doc)) +} + +// found returns a resolution, making the current document depend on the one +// the symbol it found was declared in. +func (r *Resolver) found(res resolution) (*symbols.Symbol, bool) { + if res.sym != nil { + r.Depend(res.sym.DocName) + } + return res.sym, res.ok +} + +func (r *Resolver) depend(from *frame, doc string) { + if from.doc == doc { + return + } + if from.deps == nil { + from.deps = map[string]bool{} + } + if from.deps[doc] { + return + } + from.deps[doc] = true + back := r.dependents[doc] + if back == nil { + back = map[string]bool{} + r.dependents[doc] = back + } + back[from.doc] = true +} + +// ReadName, ReadNamespace, ReadDocument and ReadAllNames implement +// symbols.ReadRecorder: what the index answered is what the frame depends on. +func (r *Resolver) ReadName(fqn string) { + if r == nil { + return + } + if f := r.cur; f != nil { + if f.names == nil { + f.names = map[string]bool{} + } + f.names[fqn] = true + } +} + +func (r *Resolver) ReadNamespace(fqn string) { + if r == nil { + return + } + if f := r.cur; f != nil && !f.all { + if f.namespaces == nil { + f.namespaces = map[string]bool{} + } + f.namespaces[fqn] = true + } +} + +func (r *Resolver) ReadDocument(name string) { + if r == nil { + return + } + if f := r.cur; f != nil && !f.all && name != f.doc { + if f.docs == nil { + f.docs = map[string]bool{} + } + f.docs[name] = true + } +} + +func (r *Resolver) ReadAllNames() { + if r == nil { + return + } + if f := r.cur; f != nil { + f.all = true + } +} + +// Invalidate drops what the documents ch touched own, and what every document +// depending on them owns, transitively. It returns the documents dropped, +// sorted; each is analyzed afresh on its next request. +func (r *Resolver) Invalidate(ch symbols.Changes) []string { + if !r.Tracking() || ch.Empty() { + return nil + } + if g, ok := r.model.(Regatherer); ok && len(ch.Docs) > 0 { + for _, name := range g.Regather(ch.Docs) { + if ch.Names == nil { + ch.Names = map[string]bool{} + } + ch.Names[name] = true + } + } + if ch.Registered() { + r.names = nil + } + var work []*frame + for _, f := range r.owners { + if f.stale(ch) { + work = append(work, f) + } + } + dropped := map[string]bool{} + for len(work) > 0 { + f := work[len(work)-1] + work = work[:len(work)-1] + if dropped[f.doc] { + continue + } + dropped[f.doc] = true + for dep := range r.dependents[f.doc] { + if g := r.owners[dep]; g != nil && !dropped[dep] { + work = append(work, g) + } + } + } + out := make([]string, 0, len(dropped)) + for doc := range dropped { + r.dropFrame(doc) + out = append(out, doc) + } + sort.Strings(out) + return out +} + +// InvalidateAll drops every document's entries. +func (r *Resolver) InvalidateAll() { + if !r.Tracking() { + return + } + for doc := range r.owners { + r.dropFrame(doc) + } + r.idx.TakeChanges() +} + +func (r *Resolver) dropFrame(doc string) { + f := r.owners[doc] + if f == nil { + return + } + f.drop() + for dep := range f.deps { + delete(r.dependents[dep], doc) + } + delete(r.owners, doc) +} + +// Dependents reports the documents whose state was computed from doc's, +// directly; for tests and diagnostics of the relation. +func (r *Resolver) Dependents(doc string) []string { + out := make([]string, 0, len(r.dependents[doc])) + for d := range r.dependents[doc] { + out = append(out, d) + } + sort.Strings(out) + return out +} + +// Owned reports how many entries doc's frame will drop; for tests. +func (r *Resolver) Owned(doc string) int { + if f := r.owners[doc]; f != nil { + return f.entries() + } + return 0 +} + +// Scratch runs f, then forgets what f memoized about the transient nodes, so +// syntax the model does not own (a request expression) is not retained. +func (r *Resolver) Scratch(transient map[ast.Node]bool, f func()) { + fr := &frame{transient: transient} + r.stack = append(r.stack, fr) + r.scratching++ + defer func() { + r.scratching-- + n := len(r.stack) - 1 + r.stack[n] = nil + r.stack = r.stack[:n] + fr.drop() + }() + f() +} + +// Journal registers drop to run when the frame owning node ends: the Scratch +// that disowns node, else the innermost document frame. It is how a side +// table keyed by node joins the resolver's lifecycle. +func (r *Resolver) Journal(node ast.Node, drop func()) { + if f := r.owner(node); f != nil { + f.journal = append(f.journal, dropFunc(drop)) + } +} + +// owner is the frame whose journal an entry for node joins: the Scratch that +// disowns node, else the innermost document frame; nil when none does. +func (r *Resolver) owner(node ast.Node) *frame { + if !r.Journaling() { + return nil + } + if r.scratching > 0 { + for i := len(r.stack) - 1; i >= 0; i-- { + if f := r.stack[i]; f.scratch() && f.transient[node] { + return f + } + } + } + switch { + case r.cur != nil: + return r.cur + case len(r.stack) == 0 && r.Tracking(): + // Written outside any frame: owned by the frame every change drops. + return r.docFrame("") + } + return nil +} + +// Journaling reports whether a memo write now would be journaled, so a side +// table can skip building the drop closure when it would not. +func (r *Resolver) Journaling() bool { + return r != nil && (len(r.stack) > 0 || r.Tracking()) +} + +// JournalNew journals the deletion of m[k], about to be written for node the +// first time, with the innermost frame: in that frame's ledger for m. +func JournalNew[K comparable, V any](r *Resolver, m map[K]V, k K, node ast.Node) { + if !r.Journaling() { + return + } + if _, had := m[k]; had { + return + } + f := r.owner(node) + if f == nil { + return + } + id := reflect.ValueOf(m).Pointer() + l, ok := f.ledgers[id].(*ledger[K, V]) + if !ok { + l = &ledger[K, V]{table: m} + if f.ledgers == nil { + f.ledgers = map[uintptr]dropper{} + } + f.ledgers[id] = l + f.journal = append(f.journal, l) + } + l.keys = append(l.keys, k) +} + +// journalNew is JournalNew for the resolver's own tables. +func journalNew[K comparable, V any](r *Resolver, m map[K]V, k K, node ast.Node) { + JournalNew(r, m, k, node) +} diff --git a/internal/core/resolve/frames_test.go b/internal/core/resolve/frames_test.go new file mode 100644 index 0000000000..9614aa87fe --- /dev/null +++ b/internal/core/resolve/frames_test.go @@ -0,0 +1,220 @@ +package resolve + +import ( + "reflect" + "testing" + + "github.com/Open-MBEE/OpenSysML/internal/core/ast" + "github.com/Open-MBEE/OpenSysML/internal/core/symbols" +) + +// trackedIndex is an index with a tracking resolver over it and the roots of +// its documents, which the tests analyze and edit. +type trackedIndex struct { + t *testing.T + idx *symbols.Index + r *Resolver + roots map[string]*ast.RootNamespace +} + +func newTrackedIndex(t *testing.T, docs map[string]string) *trackedIndex { + t.Helper() + w := &trackedIndex{t: t, idx: symbols.NewIndex(), roots: map[string]*ast.RootNamespace{}} + for name, src := range docs { + w.roots[name] = parsedRoot(t, name, src) + w.idx.AddDocument(name, w.roots[name]) + } + w.r = New(w.idx) + w.r.Track() + return w +} + +// analyze resolves doc's references as its analysis would. +func (w *trackedIndex) analyze(doc string) { + w.r.InDocument(doc, func() { w.r.ResolveDocument(doc, w.roots[doc]) }) +} + +func (w *trackedIndex) analyzeAll() { + for name := range w.roots { + w.analyze(name) + } +} + +// put replaces doc and invalidates through the changes the index recorded, as +// the workspace does; it returns the documents dropped. +func (w *trackedIndex) put(doc, src string) []string { + w.t.Helper() + w.roots[doc] = parsedRoot(w.t, doc, src) + w.idx.AddDocument(doc, w.roots[doc]) + ch := w.idx.TakeChanges() + if ch.Docs == nil { + ch.Docs = map[string]bool{} + } + ch.Docs[doc] = true + return w.r.Invalidate(ch) +} + +func (w *trackedIndex) owned(doc string) bool { return w.r.Owned(doc) > 0 } + +func TestFramesOwnWhatAnAnalysisMemoizes(t *testing.T) { + w := newTrackedIndex(t, map[string]string{ + "a.sysml": "package A { part def X; part x : X; }", + "b.sysml": "package B { part def Y; part y : Y; }", + }) + w.analyzeAll() + if !w.owned("a.sysml") || !w.owned("b.sysml") { + t.Fatalf("owned: a %d, b %d; want both > 0", w.r.Owned("a.sysml"), w.r.Owned("b.sysml")) + } + if deps := w.r.Dependents("a.sysml"); len(deps) != 0 { + t.Fatalf("a.sysml has dependents %v, want none", deps) + } +} + +func TestFramesDependOnTheDocumentASymbolCameFrom(t *testing.T) { + w := newTrackedIndex(t, map[string]string{ + "lib.sysml": "package Lib { part def Base; }", + "user.sysml": "package User { part def Derived :> Lib::Base; }", + "other.sysml": "package Other { part def Alone; part alone : Alone; }", + }) + w.analyzeAll() + if got := w.r.Dependents("lib.sysml"); !reflect.DeepEqual(got, []string{"user.sysml"}) { + t.Fatalf("dependents of lib.sysml = %v, want [user.sysml]", got) + } + dropped := w.put("lib.sysml", "package Lib { part def Base; part def More; }") + if !reflect.DeepEqual(dropped, []string{"lib.sysml", "user.sysml"}) { + t.Fatalf("dropped %v, want [lib.sysml user.sysml]", dropped) + } + if !w.owned("other.sysml") { + t.Fatal("other.sysml, which read nothing of lib.sysml, lost what it memoized") + } + if w.owned("user.sysml") { + t.Fatal("user.sysml, which specializes Lib::Base, kept what it memoized") + } +} + +func TestFramesDependOnAnImportedNamespace(t *testing.T) { + w := newTrackedIndex(t, map[string]string{ + "lib.sysml": "package Lib { part def Base; }", + "user.sysml": "package User { private import Lib::*; part def Derived :> Base; }", + }) + w.analyzeAll() + // A name added to the imported namespace could shadow or newly resolve + // what the importer sees, so the importer is dropped. + dropped := w.put("lib.sysml", "package Lib { part def Base; part def Derived; }") + if !reflect.DeepEqual(dropped, []string{"lib.sysml", "user.sysml"}) { + t.Fatalf("dropped %v, want [lib.sysml user.sysml]", dropped) + } +} + +func TestFramesDependOnAPackageSpanningDocuments(t *testing.T) { + w := newTrackedIndex(t, map[string]string{ + "a.sysml": "package P { part def X; }", + "b.sysml": "package P { part y : X; }", + "c.sysml": "package Q { part def Z; part z : Z; }", + }) + w.analyzeAll() + dropped := w.put("a.sysml", "package P { part def X2; }") + if !reflect.DeepEqual(dropped, []string{"a.sysml", "b.sysml"}) { + t.Fatalf("dropped %v, want [a.sysml b.sysml]", dropped) + } + if !w.owned("c.sysml") { + t.Fatal("c.sysml, in another package, lost what it memoized") + } +} + +func TestFramesInvalidateDependentsTransitively(t *testing.T) { + w := newTrackedIndex(t, map[string]string{ + "a.sysml": "package A { part def X; }", + "b.sysml": "package B { part def Y :> A::X; }", + "c.sysml": "package C { part def Z :> B::Y; }", + "d.sysml": "package D { part def W; part w : W; }", + }) + w.analyzeAll() + dropped := w.put("a.sysml", "package A { part def X { attribute a; } }") + if !reflect.DeepEqual(dropped, []string{"a.sysml", "b.sysml", "c.sysml"}) { + t.Fatalf("dropped %v, want [a.sysml b.sysml c.sysml]", dropped) + } + if !w.owned("d.sysml") { + t.Fatal("d.sysml, reading nothing of the others, lost what it memoized") + } + // Re-analysis restores the relation, so a second edit finds it again. + w.analyzeAll() + if got := w.r.Dependents("a.sysml"); !reflect.DeepEqual(got, []string{"b.sysml"}) { + t.Fatalf("dependents of a.sysml after re-analysis = %v, want [b.sysml]", got) + } +} + +func TestFramesReleaseAReplacedDocumentsEntries(t *testing.T) { + w := newTrackedIndex(t, map[string]string{ + "a.sysml": "package A { part def X; part x : X; part y : X; }", + }) + w.analyze("a.sysml") + before := w.r.Owned("a.sysml") + for i := 0; i < 50; i++ { + w.put("a.sysml", "package A { part def X; part x : X; part y : X; }") + w.analyze("a.sysml") + if got := w.r.Owned("a.sysml"); got != before { + t.Fatalf("edit %d: a.sysml owns %d entries, want %d as after the first analysis", i, got, before) + } + } +} + +func TestFramesGatherIsNoDependencyOfTheAnalysis(t *testing.T) { + w := newTrackedIndex(t, map[string]string{ + "a.sysml": "package A { part def X; }", + "b.sysml": "package B { part def Y :> A::X; }", + }) + w.analyze("a.sysml") + w.r.InDocument("b.sysml", func() { + w.r.Gather("a.sysml", func() { w.r.ResolveDocument("a.sysml", w.roots["a.sysml"]) }) + }) + if deps := w.r.Dependents("a.sysml"); len(deps) != 0 { + t.Fatalf("a gather from b.sysml made it depend on a.sysml: %v", deps) + } + if got := w.r.Dependents(GatherFrame("a.sysml")); len(got) != 0 { + t.Fatalf("the gather frame has dependents %v, want none", got) + } + if doc, ok := GatheredDoc(GatherFrame("a.sysml")); !ok || doc != "a.sysml" { + t.Fatalf("GatheredDoc(GatherFrame(a.sysml)) = %q, %v", doc, ok) + } + dropped := w.put("a.sysml", "package A { part def X2; }") + if !reflect.DeepEqual(dropped, []string{"a.sysml", GatherFrame("a.sysml")}) { + t.Fatalf("dropped %v, want the document and its gather frame", dropped) + } +} + +func TestFramesReadingTheWholeIndexSurviveAJudgmentChange(t *testing.T) { + w := newTrackedIndex(t, map[string]string{ + "a.sysml": "package A { part def X; }", + "b.sysml": "package B { part def Y :> A::X; }", + }) + w.analyze("a.sysml") + w.r.InDocument("b.sysml", func() { + w.r.ReadAllNames() + w.r.ReadName("\x00judgment/b") + w.r.suggestTable() + }) + table := w.r.names + if table == nil { + t.Fatal("the analysis built no suggestion table") + } + if dropped := w.r.Invalidate(symbols.Changes{Names: map[string]bool{"\x00judgment/a": true}}); len(dropped) != 0 { + t.Fatalf("a judgment b.sysml never read dropped %v", dropped) + } + if w.r.names != table { + t.Fatal("a judgment change rebuilt the suggestion table") + } + if dropped := w.r.Invalidate(symbols.Changes{Names: map[string]bool{"\x00judgment/b": true}}); !reflect.DeepEqual(dropped, []string{"b.sysml"}) { + t.Fatalf("the judgment b.sysml read dropped %v, want b.sysml", dropped) + } + if w.r.names != table { + t.Fatal("a judgment change rebuilt the suggestion table") + } + w.r.InDocument("b.sysml", func() { w.r.ReadAllNames() }) + if dropped := w.put("a.sysml", "package A { part def X2; }"); !reflect.DeepEqual(dropped, []string{"a.sysml", "b.sysml"}) { + t.Fatalf("a registration dropped %v, want the whole-index reader too", dropped) + } + if w.r.names != nil { + t.Fatal("a registration kept the suggestion table") + } +} diff --git a/internal/core/resolve/metadata_scope.go b/internal/core/resolve/metadata_scope.go index d2661f5e4d..d3bb17d347 100644 --- a/internal/core/resolve/metadata_scope.go +++ b/internal/core/resolve/metadata_scope.go @@ -26,9 +26,12 @@ func (r *Resolver) MetadataBodyOwner(scope *symbols.Scope) *symbols.Symbol { if !ok { return nil } + r.EnterDoc(symbols.DocNameOf(scope)) + defer r.LeaveDoc() if def, done := r.bodyOwners[scope]; done { return def } + journalNew(r, r.bodyOwners, scope, scope.Node()) r.bodyOwners[scope] = nil var resolved *symbols.Symbol if owner.Kind == symbols.SymbolMetadataUsage { @@ -78,10 +81,13 @@ func (r *Resolver) scopeOwner(scope *symbols.Scope) *symbols.Symbol { if !ok || !scope.BodyLocal() { return nil } + r.EnterDoc(symbols.DocNameOf(scope)) + defer r.LeaveDoc() if owner, done := r.bodyOwners[scope]; done { return owner } // Break cycles while the metaclass itself resolves. + journalNew(r, r.bodyOwners, scope, scope.Node()) r.bodyOwners[scope] = nil var resolved *symbols.Symbol r.aside(func() { diff --git a/internal/core/resolve/reading.go b/internal/core/resolve/reading.go index 92bce3b8cd..857bf41947 100644 --- a/internal/core/resolve/reading.go +++ b/internal/core/resolve/reading.go @@ -59,6 +59,8 @@ func (r *Resolver) ReadQualified(scope *symbols.Scope, qn *ast.QualifiedName) Re if qn == nil { return Reading{} } + r.EnterDoc(symbols.DocNameOf(scope)) + defer r.LeaveDoc() key := readingKey{scope: scope, qn: qn} if rd, done := r.readings[key]; done { return rd diff --git a/internal/core/resolve/redefined_nesting.go b/internal/core/resolve/redefined_nesting.go index 9ebfa1a0fa..9e3e5224d3 100644 --- a/internal/core/resolve/redefined_nesting.go +++ b/internal/core/resolve/redefined_nesting.go @@ -37,9 +37,12 @@ func (r *Resolver) nestedInRedefined(scope *symbols.Scope, name string, hide *re // redefinedFeatures returns the features sym redefines, explicitly, implicitly // in a metadata body, or as an association or connector end. func (r *Resolver) redefinedFeatures(sym *symbols.Symbol) []*symbols.Symbol { + r.EnterDoc(sym.DocName) + defer r.LeaveDoc() if cached, done := r.redefined[sym]; done { return cached } + journalNew(r, r.redefined, sym, sym.Decl) r.redefined[sym] = nil out := r.explicitRedefinitions(sym) if model, ok := r.model.(endRedefinitionLookup); ok { diff --git a/internal/core/resolve/resolver.go b/internal/core/resolve/resolver.go index ade0d99606..8ceab6dedf 100644 --- a/internal/core/resolve/resolver.go +++ b/internal/core/resolve/resolver.go @@ -186,43 +186,15 @@ type Resolver struct { // invocationNames are the names invocations call, whose last segment may // denote several declarations: see ResolveInvocationName. invocationNames map[*ast.QualifiedName]bool - // scratch are the Scratch calls in progress, innermost last. - scratch []*scratchFrame -} - -// scratchFrame is one Scratch call: the nodes it disowns and the entries -// first memoized under it. -type scratchFrame struct { - transient map[ast.Node]bool - journal []journaled -} - -type journaled struct { - node ast.Node - drop func() -} - -// Scratch runs f, then forgets what f memoized about the transient nodes, so -// syntax the model does not own (a request expression) is not retained. -func (r *Resolver) Scratch(transient map[ast.Node]bool, f func()) { - frame := &scratchFrame{transient: transient} - r.scratch = append(r.scratch, frame) - defer func() { - r.scratch = r.scratch[:len(r.scratch)-1] - var parent *scratchFrame - if n := len(r.scratch); n > 0 { - parent = r.scratch[n-1] - } - for _, j := range frame.journal { - switch { - case frame.transient[j.node]: - j.drop() - case parent != nil: - parent.journal = append(parent.journal, j) - } - } - }() - f() + // stack are the frames in progress, innermost last; cur is the innermost + // document frame, which reads and dependencies are recorded on (frames.go). + stack []*frame + cur *frame + scratching int + // owners are the documents' frames once Track was called, nil before; + // dependents[d] are the documents whose frames depend on d's. + owners map[string]*frame + dependents map[string]map[string]bool } // MemoSize is the number of resolutions this resolver retains. @@ -233,30 +205,6 @@ func (r *Resolver) MemoSize() int { len(r.initials) + len(r.imports) + len(r.suggestions) } -// journalNew lets the enclosing Scratch drop m[k], about to be written for -// node the first time. -func journalNew[K comparable, V any](r *Resolver, m map[K]V, k K, node ast.Node) { - if len(r.scratch) == 0 { - return - } - if _, had := m[k]; had { - return - } - r.Journal(node, func() { delete(m, k) }) -} - -// Journal registers drop to run when the enclosing Scratch, if any, ends with -// node transient: how a side table keyed by node joins the resolver's lifecycle. -func (r *Resolver) Journal(node ast.Node, drop func()) { - if r == nil { - return - } - if n := len(r.scratch); n > 0 { - frame := r.scratch[n-1] - frame.journal = append(frame.journal, journaled{node: node, drop: drop}) - } -} - // New creates a resolver over the given index. func New(idx *symbols.Index) *Resolver { return &Resolver{ @@ -525,6 +473,8 @@ func (r *Resolver) resolveQualified(scope *symbols.Scope, qn *ast.QualifiedName, if qn == nil { return nil, false } + r.EnterDoc(symbols.DocNameOf(scope)) + defer r.LeaveDoc() if r.foreignScope(scope) { var sym *symbols.Symbol var ok bool @@ -534,18 +484,18 @@ func (r *Resolver) resolveQualified(scope *symbols.Scope, qn *ast.QualifiedName, cacheMain := hide == nil || hide.skipNamingTarget || hide.skipBorrowedName if cacheMain { if res, done := r.memo[qn]; done { - return res.sym, res.ok + return r.found(res) } } else if res, done := r.filtered[filteredMemoKey{ qn: qn, decl: hide.decl, prefix: hide.skipNamingTarget, skipBorrowedName: hide.skipBorrowedName, }]; done { - return res.sym, res.ok + return r.found(res) } mode, keyed := r.modeKey(qn, hide) if keyed { if res, done := r.modeMemo[mode]; done { - return res.sym, res.ok + return r.found(res) } } if depth := r.resolving[qn]; depth != 0 { @@ -557,7 +507,7 @@ func (r *Resolver) resolveQualified(scope *symbols.Scope, qn *ast.QualifiedName, res := r.walkQualified(scope, qn, hide.hiding(qn)) delete(r.resolving, qn) if !r.Leave() { - return res.sym, res.ok + return r.found(res) } // A failure met during a semantic query is not memoized: the reference it // belongs to must still report when its own document is resolved. @@ -581,12 +531,14 @@ func (r *Resolver) resolveQualified(scope *symbols.Scope, qn *ast.QualifiedName, r.filtered[key] = res } } - return res.sym, res.ok + return r.found(res) } // ResolveName resolves a single-segment (unqualified) reference from the given // scope. The at node keys the memo table. func (r *Resolver) ResolveName(scope *symbols.Scope, name string, at ast.Node) (*symbols.Symbol, bool) { + r.EnterDoc(symbols.DocNameOf(scope)) + defer r.LeaveDoc() if r.foreignScope(scope) { var sym *symbols.Symbol var ok bool @@ -595,13 +547,13 @@ func (r *Resolver) ResolveName(scope *symbols.Scope, name string, at ast.Node) ( } if at != nil { if res, done := r.memo[at]; done { - return res.sym, res.ok + return r.found(res) } } mode, keyed := r.modeKey(at, nil) if keyed { if res, done := r.modeMemo[mode]; done { - return res.sym, res.ok + return r.found(res) } } r.Enter() @@ -627,7 +579,7 @@ func (r *Resolver) ResolveName(scope *symbols.Scope, name string, at ast.Node) ( Fixes: r.unresolvedFixes(scope, name, at), }) } - return res.sym, res.ok + return r.found(res) } // report records a diagnostic, unless the lookup that produced it was made for @@ -642,11 +594,27 @@ func (r *Resolver) report(d Diagnostic) { // foreignScope reports whether scope belongs to a document other than the one // being resolved, so a reference read there is that document's to report. func (r *Resolver) foreignScope(scope *symbols.Scope) bool { - if r.document == "" || r.quiet > 0 || scope == nil { + if r.quiet > 0 || scope == nil { + return false + } + analyzing := r.analyzing() + if analyzing == "" { return false } doc := r.documentOf(scope) - return doc != "" && doc != r.document + return doc != "" && doc != analyzing +} + +// analyzing is the document whose references are being resolved for report: +// the one ResolveDocument is walking, else the one InDocument is running for. +func (r *Resolver) analyzing() string { + if r.document != "" { + return r.document + } + if len(r.stack) > 0 && !r.stack[0].scratch() { + return r.stack[0].doc + } + return "" } // aside runs a lookup made for a semantic query, whose diagnostics belong to diff --git a/internal/core/resolve/transition.go b/internal/core/resolve/transition.go index ae0f6023aa..a8b29c15aa 100644 --- a/internal/core/resolve/transition.go +++ b/internal/core/resolve/transition.go @@ -18,6 +18,8 @@ func (r *Resolver) ResolveEndpoint(scope *symbols.Scope, qn *ast.QualifiedName) if qn == nil || len(qn.Parts) == 0 { return nil, false } + r.EnterDoc(symbols.DocNameOf(scope)) + defer r.LeaveDoc() if res, done := r.endpoints[qn]; done { return res.sym, res.ok } @@ -78,6 +80,8 @@ func (r *Resolver) resolveEndpointChain(scope *symbols.Scope, chain *ast.Feature return nil, false } member := chain.Member + r.EnterDoc(symbols.DocNameOf(scope)) + defer r.LeaveDoc() if res, done := r.endpoints[member]; done { return res.sym, res.ok } diff --git a/internal/core/resolve/unqualified.go b/internal/core/resolve/unqualified.go index ea96e7e47c..0981d9a56e 100644 --- a/internal/core/resolve/unqualified.go +++ b/internal/core/resolve/unqualified.go @@ -124,6 +124,8 @@ func (r *Resolver) BindsName(sym *symbols.Symbol) bool { if r == nil || sym == nil || sym.Naming == symbols.NamedByDeclaration { return true } + r.EnterDoc(sym.DocName) + defer r.LeaveDoc() if named, done := r.effNames[sym]; done { return named } @@ -143,6 +145,7 @@ func (r *Resolver) BindsName(sym *symbols.Symbol) bool { }) // A lookup cut short by a guard answered for an enclosing one, not for good. if r.Leave() { + journalNew(r, r.effNames, sym, sym.Decl) r.effNames[sym] = named } return named @@ -253,6 +256,8 @@ func (r *Resolver) implicitlyNamedMember(scope *symbols.Scope, name string, hide // and a scope holding many anonymous connections or assertions would otherwise // be rescanned by every name resolved through it. func (r *Resolver) implicitParameters(scope *symbols.Scope) []*symbols.Symbol { + r.EnterDoc(symbols.DocNameOf(scope)) + defer r.LeaveDoc() params, done := r.implicitParams[scope] if done { return params diff --git a/internal/core/semantics/annotations.go b/internal/core/semantics/annotations.go index f105a93b4f..193ab33bd0 100644 --- a/internal/core/semantics/annotations.go +++ b/internal/core/semantics/annotations.go @@ -44,11 +44,13 @@ func (m *Model) annotationsOf(sym *symbols.Symbol) []annotation { if sym == nil { return nil } + defer m.own(sym).LeaveDoc() if cached, ok := m.annotations[sym]; ok { return cached } // Recorded first so that a value that resolves back to this element cannot // re-enter the collection of its own annotations. + journal(m, m.annotations, sym, sym.Decl) m.annotations[sym] = nil var out []annotation @@ -185,18 +187,17 @@ func (m *Model) ElementMetadataOf(sym *symbols.Symbol) []ElementMetadata { // documentRanks orders the documents of the index as Index.Documents lists // them; a document the index does not hold sorts after every one it does. func (m *Model) documentRanks() map[string]int { - if m.docRanks != nil { - return m.docRanks + if m.resolver == nil || m.resolver.Index() == nil { + return nil } - ranks := make(map[string]int) - if m.resolver != nil && m.resolver.Index() != nil { - docs := m.resolver.Index().Documents() + m.shared(sharedDocs, func() bool { return m.docRanks != nil }, func() { + docs := m.gatheredDocs() + m.docRanks = make(map[string]int, len(docs)) for i, doc := range docs { - ranks[doc] = i - len(docs) + m.docRanks[doc] = i - len(docs) } - } - m.docRanks = ranks - return ranks + }, func() { m.docRanks = nil }) + return m.docRanks } // metadataBody is the body an annotation node binds feature values in. @@ -406,53 +407,26 @@ func (m *Model) AboutAnnotatedSymbols() []*symbols.Symbol { // from the element it applies to, so there is no way to it from the element // itself. func (m *Model) annotationsAbout() map[*symbols.Symbol][]annotation { - if m.aboutAnnots != nil { - return m.aboutAnnots - } - m.aboutAnnots = make(map[*symbols.Symbol][]annotation) - m.aboutByDecl = make(map[ast.Node][]annotation) if m.resolver == nil || m.resolver.Index() == nil { + if m.aboutAnnots == nil { + m.aboutAnnots = make(map[*symbols.Symbol][]annotation) + m.aboutByDecl = make(map[ast.Node][]annotation) + } return m.aboutAnnots } - idx := m.resolver.Index() - var seen map[*symbols.Symbol]bool - for _, doc := range idx.Documents() { - // A frozen document — the shared standard library above all — cached - // its `about` usages when it froze, so its tree is not walked here. - if usages, cached := idx.FrozenAboutUsages(doc); cached { - for _, sym := range usages { + m.shared(sharedAbout, func() bool { return m.aboutAnnots != nil }, func() { + m.aboutAnnots = make(map[*symbols.Symbol][]annotation) + m.aboutByDecl = make(map[ast.Node][]annotation) + gathers := m.gathers() + for _, doc := range m.gatheredDocs() { + for _, sym := range gathers[doc].about { m.indexAboutUsage(sym) } - continue } - if seen == nil { - seen = make(map[*symbols.Symbol]bool) - } - m.collectAboutAnnotations(idx.DocumentRoot(doc), seen) - } + }, func() { m.aboutAnnots, m.aboutByDecl, m.aboutOrder = nil, nil, nil }) return m.aboutAnnots } -// collectAboutAnnotations walks a scope tree — anonymous members included, so -// `metadata : T about x;` counts like a named usage — indexing every `about` -// metadata usage by the elements it annotates. -func (m *Model) collectAboutAnnotations(scope *symbols.Scope, seen map[*symbols.Symbol]bool) { - if scope == nil { - return - } - scope.ForEachMember(func(sym *symbols.Symbol) bool { - if sym == nil || seen[sym] { - return true - } - seen[sym] = true - if sym.Kind == symbols.SymbolMetadataUsage { - m.indexAboutUsage(sym) - } - m.collectAboutAnnotations(sym.Scope, seen) - return true - }) -} - // indexAboutUsage records one `about` metadata usage against every element it // annotates. func (m *Model) indexAboutUsage(sym *symbols.Symbol) { diff --git a/internal/core/semantics/bodyparam.go b/internal/core/semantics/bodyparam.go index b507b7cd89..ca1121c2e8 100644 --- a/internal/core/semantics/bodyparam.go +++ b/internal/core/semantics/bodyparam.go @@ -140,6 +140,7 @@ func (m *Model) appliesOverElements(fn *symbols.Symbol) bool { // bodyApplicationOf is the operation body is the argument of, indexing the document holding // bodyScope on first query: no scope records the expression a body sits in. func (m *Model) bodyApplicationOf(bodyScope *symbols.Scope, body *ast.BodyExpr) (bodyApplication, bool) { + defer m.ownScope(bodyScope).LeaveDoc() if app, ok := m.bodyApplications[body]; ok { return app, true } @@ -150,6 +151,7 @@ func (m *Model) bodyApplicationOf(bodyScope *symbols.Scope, body *ast.BodyExpr) if m.bodyIndexed[root] { return bodyApplication{}, false } + journal(m, m.bodyIndexed, root, root.Node()) m.bodyIndexed[root] = true doc, ok := root.Node().(*ast.RootNamespace) if !ok { @@ -160,6 +162,7 @@ func (m *Model) bodyApplicationOf(bodyScope *symbols.Scope, body *ast.BodyExpr) Body: symbols.BodyExprScope, Members: func(scope *symbols.Scope, members []ast.Node) { w.WalkMembers(scope, members) }, Applied: func(scope *symbols.Scope, op ast.Node, applied *ast.BodyExpr) { + journal(m, m.bodyApplications, applied, applied) m.bodyApplications[applied] = bodyApplication{scope: scope, op: op} }, } diff --git a/internal/core/semantics/cast.go b/internal/core/semantics/cast.go index 2a59a770a5..766737f243 100644 --- a/internal/core/semantics/cast.go +++ b/internal/core/semantics/cast.go @@ -138,6 +138,7 @@ func (m *Model) excludes( // subtracts reports whether target, or a supertype, intersection or union operand excludes // reads through, is a difference subtracting a type; memoized once the closure is settled. func (m *Model) subtracts(target *symbols.Symbol) bool { + defer m.own(target).LeaveDoc() if cached, ok := m.subtracting[target]; ok { return cached } @@ -161,6 +162,7 @@ func (m *Model) subtracts(target *symbols.Symbol) bool { } } if m.resolver.Leave() && !provisional { + journal(m, m.subtracting, target, target.Decl) m.subtracting[target] = found } return found diff --git a/internal/core/semantics/coherent_quantity.go b/internal/core/semantics/coherent_quantity.go index 0318a66cf9..601bdb1314 100644 --- a/internal/core/semantics/coherent_quantity.go +++ b/internal/core/semantics/coherent_quantity.go @@ -142,9 +142,9 @@ func qualifiedUnitName(unit *symbols.Symbol) string { // coherentUnitsFor lists the declared units reducing to exactly the coherent term, // library ones first, each rank in declaration order. func (m *Model) coherentUnitsFor(coherent UnitTerm) []coherentUnit { - if m.coherentUnits == nil { + m.shared(sharedUnits, func() bool { return m.coherentUnits != nil }, func() { m.coherentUnits = m.indexCoherentUnits() - } + }, func() { m.coherentUnits = nil }) var out []coherentUnit for _, candidate := range m.coherentUnits[coherent.DimensionKey()] { if term, err := m.UnitTermOf(candidate.sym); err == nil && term.Same(coherent) { @@ -161,50 +161,40 @@ func (m *Model) indexCoherentUnits() map[string][]coherentUnit { if m.resolver == nil || m.resolver.Index() == nil { return out } - idx := m.resolver.Index() - var roots []*symbols.Scope if system := m.libSymbol(fqnSystemOfUnitsSI); system != nil { - roots = append(roots, rootScopeOf(system.OwnerScope)) - } - for _, doc := range idx.WorkspaceDocuments() { - roots = append(roots, idx.DocumentRoot(doc)) - } - for rank, root := range roots { - if root == nil { - continue + var g docGather + collectUnitCandidates(rootScopeOf(system.OwnerScope), &g) + m.judgeCoherentUnits(g.units, 0, out) + } + gathers := m.gathers() + for _, doc := range m.gatheredDocs() { + if g := gathers[doc]; !g.library { + m.judgeCoherentUnits(g.units, 1, out) } - m.gatherCoherentUnits(root, min(rank, 1), out) } return out } -// gatherCoherentUnits walks a scope tree's packages for the units they declare, leaving -// out those whose declared kind disagrees with their definition or that compose a dimension-one unit. -func (m *Model) gatherCoherentUnits(scope *symbols.Scope, rank int, out map[string][]coherentUnit) { - for _, sym := range scope.Members() { - switch sym.Kind { - case symbols.SymbolPackage, symbols.SymbolNamespace: - if sym.Scope != nil { - m.gatherCoherentUnits(sym.Scope, rank, out) - } - case symbols.SymbolAttributeUsage: - if !m.IsMeasurementUnit(sym) { - continue - } - term, err := m.UnitTermOf(sym) - if err != nil || term.Dimensionless() { - continue - } - if slices.ContainsFunc(m.definitionPowers(sym), func(p UnitPower) bool { return p.DimensionOne }) { +// judgeCoherentUnits indexes the candidates that are units, leaving out those whose +// declared kind disagrees with their definition or that compose a dimension-one unit. +func (m *Model) judgeCoherentUnits(candidates []*symbols.Symbol, rank int, out map[string][]coherentUnit) { + for _, sym := range candidates { + if !m.IsMeasurementUnit(sym) { + continue + } + term, err := m.UnitTermOf(sym) + if err != nil || term.Dimensionless() { + continue + } + if slices.ContainsFunc(m.definitionPowers(sym), func(p UnitPower) bool { return p.DimensionOne }) { + continue + } + if declared, ok := m.dimensionOf(sym); ok { + if reduced, ok := m.dimensionOfUnitTerm(term); !ok || !declared.Commensurable(reduced) { continue } - if declared, ok := m.dimensionOf(sym); ok { - if reduced, ok := m.dimensionOfUnitTerm(term); !ok || !declared.Commensurable(reduced) { - continue - } - } - out[term.DimensionKey()] = append(out[term.DimensionKey()], coherentUnit{sym: sym, rank: rank}) } + out[term.DimensionKey()] = append(out[term.DimensionKey()], coherentUnit{sym: sym, rank: rank}) } } diff --git a/internal/core/semantics/coherent_unit.go b/internal/core/semantics/coherent_unit.go index 95e967b54e..970ca50313 100644 --- a/internal/core/semantics/coherent_unit.go +++ b/internal/core/semantics/coherent_unit.go @@ -54,31 +54,30 @@ func (m *Model) CoherentUnitFor(dim Dimension, declared *symbols.Symbol) (Unit, // systemBaseUnits maps each base quantity to the base unit SI::si measures it in, // read once from the library's `baseUnits` list. func (m *Model) systemBaseUnits() map[*symbols.Symbol]*symbols.Symbol { - if m.baseUnits != nil { - return m.baseUnits - } - m.baseUnits = make(map[*symbols.Symbol]*symbols.Symbol) - system := m.libSymbol(fqnSystemOfUnitsSI) - listed, ok := m.LookupMember(system, memberBaseUnits) - if !ok || m.resolver == nil { - return m.baseUnits - } - value := usageValue(listed) - if value == nil { - return m.baseUnits - } - scope := scopeOf(listed) - for _, ref := range sequenceElements(value) { - unit, ok := m.resolver.ResolveTarget(scope, ref) - if !ok || unit == nil { - continue + m.shared(sharedBaseUnits, func() bool { return m.baseUnits != nil }, func() { + m.baseUnits = make(map[*symbols.Symbol]*symbols.Symbol) + system := m.libSymbol(fqnSystemOfUnitsSI) + listed, ok := m.LookupMember(system, memberBaseUnits) + if !ok || m.resolver == nil { + return } - term, ok := m.dimensionOf(unit) - if !ok || len(term.Factors) != 1 || term.Factors[0].Exponent != 1 { - continue + value := usageValue(listed) + if value == nil { + return } - m.baseUnits[term.Factors[0].Unit] = unit - } + scope := scopeOf(listed) + for _, ref := range sequenceElements(value) { + unit, ok := m.resolver.ResolveTarget(scope, ref) + if !ok || unit == nil { + continue + } + term, ok := m.dimensionOf(unit) + if !ok || len(term.Factors) != 1 || term.Factors[0].Exponent != 1 { + continue + } + m.baseUnits[term.Factors[0].Unit] = unit + } + }, func() { m.baseUnits = nil }) return m.baseUnits } diff --git a/internal/core/semantics/conjugation.go b/internal/core/semantics/conjugation.go index 3bb07f1c83..d658d15228 100644 --- a/internal/core/semantics/conjugation.go +++ b/internal/core/semantics/conjugation.go @@ -101,9 +101,11 @@ func (m *Model) superEdges(sym *symbols.Symbol) []superEdge { if sym == nil { return nil } + defer m.own(sym).LeaveDoc() if cached, ok := m.superEdgeCache[sym]; ok { return cached } + journal(m, m.superEdgeCache, sym, sym.Decl) m.superEdgeCache[sym] = nil var out []superEdge @@ -152,9 +154,11 @@ func (m *Model) conjugatedSupertypes(sym *symbols.Symbol) []conjugatedType { if sym == nil { return nil } + defer m.own(sym).LeaveDoc() if cached, ok := m.conjSupers[sym]; ok { return cached } + journal(m, m.conjSupers, sym, sym.Decl) m.conjSupers[sym] = nil out := []conjugatedType{{sym: sym}} diff --git a/internal/core/semantics/connector.go b/internal/core/semantics/connector.go index 2181625f26..fc27bde7e4 100644 --- a/internal/core/semantics/connector.go +++ b/internal/core/semantics/connector.go @@ -110,13 +110,18 @@ func endFeatures(ends []connectorEnd) []*symbols.Symbol { // effectiveEnds returns sym's owned ends in declaration order, then the unredefined // ends of its generals, each once however many paths inherit it. Memoized. func (m *Model) effectiveEnds(sym *symbols.Symbol) []connectorEnd { + defer m.own(sym).LeaveDoc() if cached, ok := m.ends[sym]; ok { return cached } + if sym == nil { + return nil + } // Guard against re-entrancy on cyclic specialization graphs. + journal(m, m.ends, sym, sym.Decl) m.ends[sym] = nil - if sym != nil && sym.Decl == nil && m.IsBinaryConnector(sym) { + if sym.Decl == nil && m.IsBinaryConnector(sym) { var out []connectorEnd for i, name := range binaryConnectorEndNames { end, ok := m.LookupMember(sym, name) diff --git a/internal/core/semantics/dimension.go b/internal/core/semantics/dimension.go index 9f5e3f11ea..cbf9c2db92 100644 --- a/internal/core/semantics/dimension.go +++ b/internal/core/semantics/dimension.go @@ -414,6 +414,7 @@ func (m *Model) dimensionOf(sym *symbols.Symbol) (UnitTerm, bool) { if m == nil || sym == nil { return UnitTerm{}, false } + defer m.own(sym).LeaveDoc() if cached, ok := m.dimensions[sym]; ok { return cached.term, cached.ok } @@ -423,6 +424,7 @@ func (m *Model) dimensionOf(sym *symbols.Symbol) (UnitTerm, bool) { m.dimensioning[sym] = true term, ok := m.deriveDimension(sym) delete(m.dimensioning, sym) + journal(m, m.dimensions, sym, sym.Decl) m.dimensions[sym] = dimensionResult{term: term, ok: ok} return term, ok } diff --git a/internal/core/semantics/exprtype.go b/internal/core/semantics/exprtype.go index daad75c6b4..95c6c1e2a2 100644 --- a/internal/core/semantics/exprtype.go +++ b/internal/core/semantics/exprtype.go @@ -106,24 +106,25 @@ var scalarFQNs = map[string]PrimType{ "ScalarValues::Number": PrimNumber, } -// scalarTable resolves the stdlib scalar symbols once per model, by identity, -// so a user-declared type merely named "Integer" is never mistaken for one. +// scalarTable resolves the stdlib scalar symbols by identity, so a user-declared +// type merely named "Integer" is never mistaken for one; a document declaring +// one of the scalar names drops the table with its readers when it changes. func (m *Model) scalarTable() map[*symbols.Symbol]PrimType { - if m.scalars != nil { - return m.scalars - } - table := make(map[*symbols.Symbol]PrimType, len(scalarFQNs)) - if m.resolver != nil && m.resolver.Index() != nil { - for fqn, prim := range scalarFQNs { - for _, sym := range m.resolver.Index().LookupQualified(fqn) { - if sym != nil { - table[sym] = prim + m.shared(sharedScalars, func() bool { return m.scalars != nil }, func() { + table := make(map[*symbols.Symbol]PrimType, len(scalarFQNs)) + if m.resolver != nil && m.resolver.Index() != nil { + idx := m.resolver.Index() + for fqn, prim := range scalarFQNs { + for _, sym := range idx.LookupQualified(fqn) { + if sym != nil { + table[sym] = prim + } } } } - } - m.scalars = table - return table + m.scalars = table + }, func() { m.scalars = nil }) + return m.scalars } // ScalarLatticeElement is the lattice element sym is, as opposed to one it @@ -178,11 +179,13 @@ func (m *Model) PrimTypeOf(sym *symbols.Symbol) PrimType { if m == nil || sym == nil { return PrimUnknown } + defer m.own(sym).LeaveDoc() if cached, ok := m.primTypes[sym]; ok { return cached } table := m.scalarTable() prim := PrimUnknown + m.resolver.Enter() if p, ok := table[sym]; ok { prim = p } else { @@ -195,9 +198,14 @@ func (m *Model) PrimTypeOf(sym *symbols.Symbol) PrimType { } } } + // A walk a re-entrant supertype query cut short is provisional, not memoized. + if !m.resolver.Leave() { + return prim + } if m.primTypes == nil { m.primTypes = make(map[*symbols.Symbol]PrimType) } + journal(m, m.primTypes, sym, sym.Decl) m.primTypes[sym] = prim return prim } diff --git a/internal/core/semantics/filter.go b/internal/core/semantics/filter.go index e4f6d78310..bd1cc59116 100644 --- a/internal/core/semantics/filter.go +++ b/internal/core/semantics/filter.go @@ -99,6 +99,7 @@ func (m *Model) EvalElementFilter(f symbols.ElementFilter, cand *symbols.Symbol) if pred == nil { return true, &FilterError{Err: ErrFilterUnevaluable, Reason: "the condition is empty", Span: f.Span} } + defer m.own(cand).LeaveDoc() key := filterKey{pred: pred, cand: cand} if v, ok := m.filterVerdicts[key]; ok { return v.value, v.err @@ -116,6 +117,7 @@ func (m *Model) EvalElementFilter(f symbols.ElementFilter, cand *symbols.Symbol) // while the index is still filling — and re-deciding it later is what keeps // the answer from depending on when it was first asked. if err == nil { + journal(m, m.filterVerdicts, key, cand.Decl) m.filterVerdicts[key] = verdict } return verdict.value, verdict.err @@ -128,6 +130,7 @@ func (m *Model) CompileElementFilter(f symbols.ElementFilter) *symbols.FilterPre if f.Expr == nil { return nil } + defer m.ownScope(f.Scope).LeaveDoc() if pred, ok := m.filterPreds[f.Expr]; ok { return pred } @@ -135,6 +138,7 @@ func (m *Model) CompileElementFilter(f symbols.ElementFilter) *symbols.FilterPre // they resolve through the namespace's imports unfiltered. var pred *symbols.FilterPredicate m.resolver.InCondition(func() { pred = m.compileCondition(f.Scope, f.Expr) }) + journal(m, m.filterPreds, f.Expr, f.Expr) m.filterPreds[f.Expr] = pred return pred } @@ -933,6 +937,7 @@ func (m *Model) symbolByFQN(fqn string) *symbols.Symbol { if fqn == "" || m.resolver == nil || m.resolver.Index() == nil { return nil } + m.resolver.ReadName(fqn) if sym, ok := m.filterTypes[fqn]; ok { return sym } @@ -940,6 +945,7 @@ func (m *Model) symbolByFQN(fqn string) *symbols.Symbol { if syms := m.resolver.Index().LookupQualified(fqn); len(syms) == 1 { found = syms[0] } + journal(m, m.filterTypes, fqn, nil) m.filterTypes[fqn] = found return found } diff --git a/internal/core/semantics/gather.go b/internal/core/semantics/gather.go new file mode 100644 index 0000000000..d2189b1c8d --- /dev/null +++ b/internal/core/semantics/gather.go @@ -0,0 +1,158 @@ +package semantics + +import ( + "sort" + + "github.com/Open-MBEE/OpenSysML/internal/core/ast" + "github.com/Open-MBEE/OpenSysML/internal/core/symbols" +) + +// The names of the shared frames the workspace-wide indexes are built in (see +// Model.shared). A frame is named so Regather can drop it by name when a gather +// it was built from changed; the names cannot be a document's. +const ( + sharedAbout = "\x00about" + sharedUnits = "\x00units" + sharedDocs = "\x00docs" + sharedScalars = "\x00scalars" + sharedBaseUnits = "\x00baseUnits" +) + +// docGather is what a workspace-wide index is built from per writable document, +// collected syntactically so it is cheap to keep current: the `about` metadata +// usages it declares and the attribute usages its packages declare, the +// candidates for measurement units. +type docGather struct { + about []*symbols.Symbol + units []*symbols.Symbol + library bool +} + +// gatherOf collects doc's gather from the tree the index holds for it; a +// document the index does not hold gathers nothing. +func (m *Model) gatherOf(doc string) *docGather { + root := m.resolver.Index().DocumentRoot(doc) + if root == nil { + return nil + } + g := &docGather{library: m.resolver.Index().IsLibraryDocument(doc)} + m.collectAbout(root, g, make(map[*symbols.Symbol]bool)) + collectUnitCandidates(root, g) + return g +} + +// collectAbout walks a scope tree — anonymous members included, so `metadata : +// T about x;` counts like a named usage — for its `about` metadata usages. +func (m *Model) collectAbout(scope *symbols.Scope, g *docGather, seen map[*symbols.Symbol]bool) { + if scope == nil { + return + } + scope.ForEachMember(func(sym *symbols.Symbol) bool { + if sym == nil || seen[sym] { + return true + } + seen[sym] = true + if usage, ok := sym.Decl.(*ast.Usage); ok && sym.Kind == symbols.SymbolMetadataUsage && annotatesOthers(usage) { + g.about = append(g.about, sym) + } + m.collectAbout(sym.Scope, g, seen) + return true + }) +} + +// collectUnitCandidates walks a scope tree's packages for the attribute usages +// they declare, which indexCoherentUnits judges as units. +func collectUnitCandidates(scope *symbols.Scope, g *docGather) { + for _, sym := range scope.Members() { + switch sym.Kind { + case symbols.SymbolPackage, symbols.SymbolNamespace: + if sym.Scope != nil { + collectUnitCandidates(sym.Scope, g) + } + case symbols.SymbolAttributeUsage: + g.units = append(g.units, sym) + } + } +} + +// gathers returns the per-document gathers, collecting them from every document +// the index holds on first use; Regather keeps them current from then on. The +// frozen documents' `about` usages come from the cache built when they froze. +func (m *Model) gathers() map[string]*docGather { + if m.docGathers != nil { + return m.docGathers + } + m.docGathers = map[string]*docGather{} + m.resolver.Untracked(func() { + idx := m.resolver.Index() + for _, doc := range idx.Documents() { + if usages, cached := idx.FrozenAboutUsages(doc); cached { + m.docGathers[doc] = &docGather{about: usages, library: true} + continue + } + if g := m.gatherOf(doc); g != nil { + m.docGathers[doc] = g + } + } + }) + return m.docGathers +} + +// gatheredDocs lists the gathered documents in index order. +func (m *Model) gatheredDocs() []string { + gathers := m.gathers() + out := make([]string, 0, len(gathers)) + for doc := range gathers { + out = append(out, doc) + } + sort.Strings(out) + return out +} + +// Regather implements resolve.Regatherer: it collects the gathers of the +// documents that changed anew and names the shared frames whose indexes they +// were built into, for the resolver to drop with their dependents. +func (m *Model) Regather(docs map[string]bool) []string { + if m.docGathers == nil { + return nil + } + changed := map[string]bool{} + for doc := range docs { + old, had := m.docGathers[doc] + g := m.gatherOf(doc) + if had != (g != nil) { + changed[sharedDocs] = true + } + if len(old.aboutOf()) > 0 || len(g.aboutOf()) > 0 { + changed[sharedAbout] = true + } + if len(old.unitsOf()) > 0 || len(g.unitsOf()) > 0 { + changed[sharedUnits] = true + } + if g == nil { + delete(m.docGathers, doc) + } else { + m.docGathers[doc] = g + } + } + out := make([]string, 0, len(changed)) + for name := range changed { + out = append(out, name) + } + sort.Strings(out) + return out +} + +func (g *docGather) aboutOf() []*symbols.Symbol { + if g == nil { + return nil + } + return g.about +} + +func (g *docGather) unitsOf() []*symbols.Symbol { + if g == nil { + return nil + } + return g.units +} diff --git a/internal/core/semantics/implicit.go b/internal/core/semantics/implicit.go index 7d62d14e9d..a910e1466b 100644 --- a/internal/core/semantics/implicit.go +++ b/internal/core/semantics/implicit.go @@ -344,12 +344,14 @@ func (m *Model) implicitBases(sym *symbols.Symbol) []*symbols.Symbol { if m.resolver == nil || m.resolver.Index() == nil { return nil } + defer m.own(sym).LeaveDoc() if cached, ok := m.implicitBase[sym]; ok { return cached } m.resolver.Enter() out := m.computeImplicitBases(sym) if m.resolver.Leave() { + journal(m, m.implicitBase, sym, sym.Decl) m.implicitBase[sym] = out } return out diff --git a/internal/core/semantics/invocation.go b/internal/core/semantics/invocation.go index 40a9a93b6c..dbf7ae60c2 100644 --- a/internal/core/semantics/invocation.go +++ b/internal/core/semantics/invocation.go @@ -229,12 +229,13 @@ func (m *Model) SelectInvocation(scope *symbols.Scope, e *ast.InvocationExpr, ar if m == nil || e == nil || e.Type == nil { return &InvocationSelection{} } + defer m.ownScope(scope).LeaveDoc() key := invocationKey{node: e, scope: scope, performs: performs} if sel, ok := m.invocations[key]; ok { return sel } sel := m.selectAmong(scope, m.resolver.InvocationCandidates(scope, e.Type), args, performs) - m.resolver.Journal(e, func() { delete(m.invocations, key) }) + journal(m, m.invocations, key, e) m.invocations[key] = sel return sel } diff --git a/internal/core/semantics/layout.go b/internal/core/semantics/layout.go index f00423d9e6..1230871d4f 100644 --- a/internal/core/semantics/layout.go +++ b/internal/core/semantics/layout.go @@ -107,6 +107,7 @@ func (m *Model) LayoutSitesOf(sym *symbols.Symbol) []*LayoutSite { if m == nil || sym == nil { return nil } + defer m.own(sym).LeaveDoc() if cached, ok := m.layoutSites[sym]; ok { return cached } @@ -137,6 +138,7 @@ func (m *Model) LayoutSitesOf(sym *symbols.Symbol) []*LayoutSite { } out = append(out, site) } + journal(m, m.layoutSites, sym, sym.Decl) m.layoutSites[sym] = out return out } @@ -222,10 +224,12 @@ func (m *Model) symbolDeclaringUnder(scope *symbols.Scope, decl ast.Node) (*symb if scope == nil { return nil, false } + defer m.ownScope(scope).LeaveDoc() index, ok := m.declSymbols[scope] if !ok { index = make(map[ast.Node]*symbols.Symbol) indexDeclarations(scope, index, make(map[*symbols.Scope]bool)) + journal(m, m.declSymbols, scope, scope.Node()) m.declSymbols[scope] = index } sym, ok := index[decl] diff --git a/internal/core/semantics/lifecycle.go b/internal/core/semantics/lifecycle.go new file mode 100644 index 0000000000..3a8d72ab34 --- /dev/null +++ b/internal/core/semantics/lifecycle.go @@ -0,0 +1,45 @@ +package semantics + +import ( + "github.com/Open-MBEE/OpenSysML/internal/core/ast" + "github.com/Open-MBEE/OpenSysML/internal/core/resolve" + "github.com/Open-MBEE/OpenSysML/internal/core/symbols" +) + +// own makes the document declaring sym the owner of what is memoized until +// LeaveDoc is called on the returned resolver (see resolve.Resolver.EnterDoc): +// a method memoizing per symbol computes under `defer m.own(sym).LeaveDoc()`. +func (m *Model) own(sym *symbols.Symbol) *resolve.Resolver { + doc := "" + if sym != nil { + doc = sym.DocName + } + m.resolver.EnterDoc(doc) + return m.resolver +} + +// ownScope is own for a method memoizing per scope. +func (m *Model) ownScope(scope *symbols.Scope) *resolve.Resolver { + m.resolver.EnterDoc(symbols.DocNameOf(scope)) + return m.resolver +} + +// journal registers the deletion of table[key], about to be written for the +// first time, with the frame owning node (see resolve.JournalNew). +func journal[K comparable, V any](m *Model, table map[K]V, key K, node ast.Node) { + resolve.JournalNew(m.resolver, table, key, node) +} + +// shared runs build inside the frame named name unless ready reports its result +// still stands, and journals reset to run when that frame is dropped. The +// caller's frame depends on name's, so a reader is dropped with the result. +func (m *Model) shared(name string, ready func() bool, build func(), reset func()) { + m.resolver.EnterDoc(name) + defer m.resolver.LeaveDoc() + m.resolver.ReadName(name) + if ready() { + return + } + build() + m.resolver.Journal(nil, reset) +} diff --git a/internal/core/semantics/lifecycle_test.go b/internal/core/semantics/lifecycle_test.go new file mode 100644 index 0000000000..277550bbf9 --- /dev/null +++ b/internal/core/semantics/lifecycle_test.go @@ -0,0 +1,144 @@ +package semantics + +import ( + "testing" + + "github.com/Open-MBEE/OpenSysML/internal/core/parser" + "github.com/Open-MBEE/OpenSysML/internal/core/resolve" + "github.com/Open-MBEE/OpenSysML/internal/core/source" + "github.com/Open-MBEE/OpenSysML/internal/core/symbols" +) + +// trackedModel holds idx in a tracking resolver, the way a workspace holds its +// persistent model, and returns a function replacing the document called name. +func trackedModel(t *testing.T, idx *symbols.Index, name string) (*Model, *resolve.Resolver, func(src string) *symbols.Scope) { + t.Helper() + r := resolve.New(idx) + m := NewModel(r) + r.SetModel(m) + r.Track() + replace := func(src string) *symbols.Scope { + p := parser.New(source.New(name, []byte(src))) + root := p.ParseFile() + if len(p.Diagnostics) != 0 { + t.Fatalf("parse diagnostics: %v", p.Diagnostics) + } + idx.AddDocument(name, root) + r.Invalidate(idx.TakeChanges()) + r.InDocument(name, func() { r.ResolveDocument(name, root) }) + return idx.DocumentRoot(name) + } + return m, r, replace +} + +// Replacing a document drops the body-application index built from its previous +// tree, so neither its old root nor its old body expressions stay referenced. +func TestBodyApplicationIndexFollowsItsDocument(t *testing.T) { + const src = `package P { + part def C; + part cs : C[*]; + attribute picked = cs.{ in x; x }; + }` + m, r, replace := trackedModel(t, stdlibIndex(t), "bodies.sysml") + query := func(scope *symbols.Scope) []*symbols.Symbol { + p := sym(t, scope, "P") + body := appliedBody(t, valueOf(t, p.Scope, "picked")) + x := sym(t, symbols.BodyExprScope(p.Scope, body), "x") + var types []*symbols.Symbol + r.InDocument("bodies.sysml", func() { types = m.BodyParameterElementTypes(x) }) + return types + } + for i := 0; i < 3; i++ { + scope := replace(src) + if types := query(scope); len(types) != 1 || types[0].Name != "C" { + t.Fatalf("edit %d: x typed %v, want C", i, types) + } + if len(m.bodyIndexed) != 1 || len(m.bodyApplications) != 1 { + t.Fatalf("edit %d: %d roots and %d bodies indexed, want 1 and 1", i, len(m.bodyIndexed), len(m.bodyApplications)) + } + for root := range m.bodyIndexed { + if root != scope { + t.Fatalf("edit %d: index keyed by a previous root", i) + } + } + } +} + +// A document declaring a scalar name is part of the scalar table's input: +// replacing it drops the table, and the next query maps the new symbol. +func TestScalarTableFollowsItsDeclaringDocument(t *testing.T) { + const src = `package ScalarValues { datatype Integer; } + package P { attribute n : ScalarValues::Integer; }` + m, r, replace := trackedModel(t, symbols.NewIndex(), "scalars.sysml") + var previous *symbols.Symbol + for i := 0; i < 3; i++ { + scope := replace(src) + integer := sym(t, sym(t, scope, "ScalarValues").Scope, "Integer") + var prim PrimType + var ok bool + r.InDocument("scalars.sysml", func() { prim, ok = m.ScalarLatticeElement(integer) }) + if !ok || prim != PrimInteger { + t.Fatalf("edit %d: Integer classified %v %v, want PrimInteger", i, prim, ok) + } + if _, stale := m.scalars[previous]; stale || len(m.scalars) != 1 { + t.Fatalf("edit %d: table holds %d symbols, want the current Integer only", i, len(m.scalars)) + } + previous = integer + } +} + +// A memoized member-source closure is owned by the document declaring its +// symbol, and a reader served from the memo depends on that document: editing +// it reaches what the reader derived from the closure. +func TestMemberSourcesReadersDependOnTheDeclaringDocument(t *testing.T) { + idx := symbols.NewIndex() + r := resolve.New(idx) + m := NewModel(r) + r.SetModel(m) + r.Track() + add := func(name, src string) { + p := parser.New(source.New(name, []byte(src))) + root := p.ParseFile() + if len(p.Diagnostics) != 0 { + t.Fatalf("parse diagnostics: %v", p.Diagnostics) + } + idx.AddDocument(name, root) + } + add("a.sysml", "package A { part def Base; part def Derived :> Base; }") + add("b.sysml", "package B { part def Other; }") + r.Invalidate(idx.TakeChanges()) + derived := sym(t, sym(t, idx.DocumentRoot("a.sysml"), "A").Scope, "Derived") + sources := func(from string) []*symbols.Symbol { + var out []*symbols.Symbol + r.InDocument(from, func() { out = m.MemberSources(derived) }) + return out + } + if got := sources("a.sysml"); len(got) != 1 || got[0].Name != "Base" { + t.Fatalf("MemberSources(Derived) = %v, want Base", got) + } + if got := sources("b.sysml"); len(got) != 1 || got[0].Name != "Base" { + t.Fatalf("memoized MemberSources(Derived) = %v, want Base", got) + } + if deps := r.Dependents("a.sysml"); len(deps) != 1 || deps[0] != "b.sysml" { + t.Fatalf("a.sysml's dependents = %v, want b.sysml, served from the memo", deps) + } + var lookup []lookupSource + r.InDocument("b.sysml", func() { lookup = m.lookupSources(derived) }) + if len(lookup) != 1 || lookup[0].sym.Name != "Base" { + t.Fatalf("lookupSources(Derived) = %v, want Base", lookup) + } + add("a.sysml", "package A { part def Base; part def Derived; }") + ch := idx.TakeChanges() + ch.Docs = map[string]bool{"a.sysml": true} + dropped := r.Invalidate(ch) + if len(dropped) != 2 || dropped[0] != "a.sysml" || dropped[1] != "b.sysml" { + t.Fatalf("editing a.sysml dropped %v, want a.sysml and its reader b.sysml", dropped) + } + if len(m.memberSources) != 0 || len(m.lookupOrder) != 0 { + t.Fatalf("%d member-source and %d lookup-order closures survive the edit, want none", len(m.memberSources), len(m.lookupOrder)) + } + derived = sym(t, sym(t, idx.DocumentRoot("a.sysml"), "A").Scope, "Derived") + if got := sources("b.sysml"); len(got) != 0 { + t.Fatalf("MemberSources(Derived) after the edit = %v, want none", got) + } +} diff --git a/internal/core/semantics/masking.go b/internal/core/semantics/masking.go index b915ca355c..1ddd032253 100644 --- a/internal/core/semantics/masking.go +++ b/internal/core/semantics/masking.go @@ -21,6 +21,7 @@ func (m *Model) RedefinedFeatures(sym *symbols.Symbol) []*symbols.Symbol { if m == nil || sym == nil { return nil } + defer m.own(sym).LeaveDoc() if cached, ok := m.redefined[sym]; ok { // The seed answers a re-entrant query with nothing, cutting that query short. if depth := m.computingRedefined[sym]; depth != 0 { @@ -28,6 +29,7 @@ func (m *Model) RedefinedFeatures(sym *symbols.Symbol) []*symbols.Symbol { } return cached } + journal(m, m.redefined, sym, sym.Decl) m.redefined[sym] = nil // re-entrancy guard for cyclic declarations m.computingRedefined[sym] = m.resolver.Enter() defer delete(m.computingRedefined, sym) @@ -285,9 +287,11 @@ func (m *Model) memoizedMask( if m == nil || sym == nil { return nil } + defer m.own(sym).LeaveDoc() if cached, ok := cache[sym]; ok { return cached } + journal(m, cache, sym, sym.Decl) cache[sym] = nil // re-entrancy guard: a nested query sees no mask m.resolver.Enter() mask := m.buildMaskFromCandidates(sym, iterate) @@ -406,6 +410,7 @@ func (m *Model) redefinitionClosure(candidate *symbols.Symbol) (map[*symbols.Sym if candidate == nil { return nil, false } + defer m.own(candidate).LeaveDoc() if cached, ok := m.redefClosure[candidate]; ok { return cached, false } @@ -434,6 +439,7 @@ func (m *Model) redefinitionClosure(candidate *symbols.Symbol) (map[*symbols.Sym return out, true } if settled && m.computingRedefinedFeatures == 0 { + journal(m, m.redefClosure, candidate, candidate.Decl) m.redefClosure[candidate] = out } return out, false diff --git a/internal/core/semantics/members.go b/internal/core/semantics/members.go index 742dd16b9e..80d265328c 100644 --- a/internal/core/semantics/members.go +++ b/internal/core/semantics/members.go @@ -61,6 +61,7 @@ func (m *Model) membersOf(sym *symbols.Symbol, view memberView, declaring *symbo sym = target } } + defer m.own(sym).LeaveDoc() key := memberKey{sym: sym, view: view} if view != memberViewDeclaring { if cached, ok := m.members[key]; ok { @@ -71,6 +72,7 @@ func (m *Model) membersOf(sym *symbols.Symbol, view memberView, declaring *symbo // Memoized once the sources are complete and no redefinition is mid-resolution, // the same condition MemberSources and the constructor slots memoize under. if view != memberViewDeclaring && m.MemberSourcesStable(sym) && m.computingRedefinedFeatures == 0 { + journal(m, m.members, key, sym.Decl) m.members[key] = out } return out diff --git a/internal/core/semantics/model.go b/internal/core/semantics/model.go index dd085c567f..a258429ca0 100644 --- a/internal/core/semantics/model.go +++ b/internal/core/semantics/model.go @@ -83,6 +83,9 @@ type Model struct { aboutByDecl map[ast.Node][]annotation // aboutOrder lists aboutAnnots' targets in first-annotation order. aboutOrder []*symbols.Symbol + // docGathers holds what the workspace-wide indexes are built from, per + // document (gather.go); nil until first needed. + docGathers map[string]*docGather // layoutSites memoizes the DiagramLayout annotations of each element, and // declSymbols the symbol each declaration under a scope registers (layout.go). layoutSites map[*symbols.Symbol][]*LayoutSite @@ -268,6 +271,7 @@ func (m *Model) DirectSupertypes(sym *symbols.Symbol) []*symbols.Symbol { if sym == nil { return nil } + defer m.own(sym).LeaveDoc() // An assumption answers every query under it, which it cuts short so nothing is memoized. if assumed, ok := m.assumedSupers[sym]; ok { m.resolver.CutShort(assumed.depth) @@ -281,6 +285,7 @@ func (m *Model) DirectSupertypes(sym *symbols.Symbol) []*symbols.Symbol { return cached } // Guard against re-entrancy on cyclic graphs: seed with an empty slice. + journal(m, m.directSupers, sym, sym.Decl) m.directSupers[sym] = nil m.computingSupers[sym] = m.resolver.Enter() defer delete(m.computingSupers, sym) @@ -496,6 +501,7 @@ func (m *Model) DirectSupertypes(sym *symbols.Symbol) []*symbols.Symbol { // the finished model has, so it is recomputed on the next query, not memoized. if !m.resolver.Leave() || !metadataComplete { delete(m.directSupers, sym) + journal(m, m.provisionalSupers, sym, sym.Decl) m.provisionalSupers[sym] = true return out } @@ -666,6 +672,7 @@ func (m *Model) AllSupertypes(sym *symbols.Symbol) []*symbols.Symbol { if sym == nil { return nil } + defer m.own(sym).LeaveDoc() if cached, ok := m.allSupers[sym]; ok { return cached } @@ -690,6 +697,7 @@ func (m *Model) AllSupertypes(sym *symbols.Symbol) []*symbols.Symbol { delete(m.allSupers, sym) return order } + journal(m, m.allSupers, sym, sym.Decl) m.allSupers[sym] = order return order } @@ -922,10 +930,12 @@ func (m *Model) composedOperands( if sym == nil { return nil } + defer m.own(sym).LeaveDoc() key := composedKey{sym: sym, kind: kind} if cached, ok := m.composed[key]; ok { return cached } + journal(m, m.composed, key, sym.Decl) m.composed[key] = nil var out []*symbols.Symbol diff --git a/internal/core/semantics/redefinition.go b/internal/core/semantics/redefinition.go index fdb0d8611c..b74dd219a8 100644 --- a/internal/core/semantics/redefinition.go +++ b/internal/core/semantics/redefinition.go @@ -131,10 +131,12 @@ func (m *Model) BehaviorParametersOf(sym *symbols.Symbol) []BehaviorParameter { // behavior beyond those redefined are inherited, ordered after the owned ones). // The result is memoized. func (m *Model) parametersOf(sym *symbols.Symbol) behaviorParameters { + defer m.own(sym).LeaveDoc() if cached, ok := m.params[sym]; ok { return cached } // Guard against re-entrancy on cyclic specialization graphs. + journal(m, m.params, sym, sym.Decl) m.params[sym] = behaviorParameters{} owned := ownedParameters(sym) diff --git a/internal/core/semantics/reference.go b/internal/core/semantics/reference.go index bd44deeba0..91b32afd3d 100644 --- a/internal/core/semantics/reference.go +++ b/internal/core/semantics/reference.go @@ -23,6 +23,7 @@ func (m *Model) ReferencedFeature(sym *symbols.Symbol) *symbols.Symbol { if sym == nil { return nil } + defer m.own(sym).LeaveDoc() if cached, ok := m.referenced[sym]; ok { return cached } @@ -48,6 +49,7 @@ func (m *Model) ReferencedFeature(sym *symbols.Symbol) *symbols.Symbol { // truncated member view (that symbol's own reference was hidden), so it is // provisional and must not be cached. if len(m.resolvingRef) == 1 { + journal(m, m.referenced, sym, sym.Decl) m.referenced[sym] = out } return out @@ -105,6 +107,7 @@ func (m *Model) MemberSources(sym *symbols.Symbol) []*symbols.Symbol { if sym == nil { return nil } + defer m.own(sym).LeaveDoc() if cached, ok := m.memberSources[sym]; ok { return cached } @@ -128,6 +131,7 @@ func (m *Model) MemberSources(sym *symbols.Symbol) []*symbols.Symbol { // supertype query it depends on is itself unresolved, is provisional: those // guards report fewer sources than the finished model has, so it is not cached. if len(m.resolvingRef) == 0 && !provisional { + journal(m, m.memberSources, sym, sym.Decl) m.memberSources[sym] = order } return order @@ -149,6 +153,7 @@ func (m *Model) lookupSources(sym *symbols.Symbol) []lookupSource { if sym == nil { return nil } + defer m.own(sym).LeaveDoc() if cached, ok := m.lookupOrder[sym]; ok { return cached } @@ -170,6 +175,7 @@ func (m *Model) lookupSources(sym *symbols.Symbol) []lookupSource { } walk(sym, nil) if len(m.resolvingRef) == 0 && !provisional { + journal(m, m.lookupOrder, sym, sym.Decl) m.lookupOrder[sym] = order } return order @@ -226,6 +232,7 @@ func (m *Model) DirectMemberSources(sym *symbols.Symbol) []*symbols.Symbol { // KerML feature keyword implies, then the feature it reference-subsets. The two // bases contribute members only, not conformance. func (m *Model) contributors(sym *symbols.Symbol) []*symbols.Symbol { + defer m.own(sym).LeaveDoc() if cached, ok := m.contributed[sym]; ok { return cached } @@ -233,6 +240,7 @@ func (m *Model) contributors(sym *symbols.Symbol) []*symbols.Symbol { // Memoized under the condition MemberSources memoizes its closure: no reference // mid-resolution and sym's own supertypes settled. if len(m.resolvingRef) == 0 && !m.supersUnstable(sym) { + journal(m, m.contributed, sym, sym.Decl) m.contributed[sym] = out } return out diff --git a/internal/core/semantics/shape.go b/internal/core/semantics/shape.go index d70d086a25..fb12d7fd3f 100644 --- a/internal/core/semantics/shape.go +++ b/internal/core/semantics/shape.go @@ -39,6 +39,7 @@ func (m *Model) ShapeFeatures(typ *symbols.Symbol) []ShapeFeature { if m == nil || typ == nil { return nil } + defer m.own(typ).LeaveDoc() if cached, ok := m.shapes[typ]; ok { return cached } @@ -46,6 +47,7 @@ func (m *Model) ShapeFeatures(typ *symbols.Symbol) []ShapeFeature { return !m.FrameFeature(member) || m.DescribesReference(typ, member) }) if m.MemberSourcesStable(typ) && m.computingRedefinedFeatures == 0 { + journal(m, m.shapes, typ, typ.Decl) m.shapes[typ] = out } return out @@ -109,11 +111,13 @@ func (m *Model) constructorSlots(typ *symbols.Symbol) constructorSlots { if m == nil || typ == nil || m.resolver == nil || m.resolver.Index() == nil { return constructorSlots{} } + defer m.own(typ).LeaveDoc() if cached, ok := m.ctorSlots[typ]; ok { return cached } slots := m.computeConstructorSlots(typ) if m.computingRedefinedFeatures == 0 { + journal(m, m.ctorSlots, typ, typ.Decl) m.ctorSlots[typ] = slots } return slots diff --git a/internal/core/semantics/uniqueness.go b/internal/core/semantics/uniqueness.go index 52fc34593e..a6885eb5a1 100644 --- a/internal/core/semantics/uniqueness.go +++ b/internal/core/semantics/uniqueness.go @@ -22,11 +22,13 @@ func (m *Model) IsUnique(sym *symbols.Symbol) bool { if m == nil || sym == nil { return true } + defer m.own(sym).LeaveDoc() if cached, ok := m.unique[sym]; ok { return cached } unique := m.isUnique(sym, map[*symbols.Symbol]bool{sym: true}) if m.computingRedefinedFeatures == 0 { + journal(m, m.unique, sym, sym.Decl) m.unique[sym] = unique } return unique diff --git a/internal/core/semantics/units.go b/internal/core/semantics/units.go index d013ff5575..9dfb572344 100644 --- a/internal/core/semantics/units.go +++ b/internal/core/semantics/units.go @@ -355,6 +355,7 @@ func (m *Model) UnitTermOf(sym *symbols.Symbol) (UnitTerm, error) { if m == nil || sym == nil { return UnitTerm{}, ErrNotAUnit } + defer m.own(sym).LeaveDoc() if cached, ok := m.unitTerms[sym]; ok { return cached, nil } @@ -371,6 +372,7 @@ func (m *Model) UnitTermOf(sym *symbols.Symbol) (UnitTerm, error) { if err != nil { return UnitTerm{}, err } + journal(m, m.unitTerms, sym, sym.Decl) m.unitTerms[sym] = term return term, nil } @@ -838,6 +840,7 @@ func (m *Model) libSymbol(fqn string) *symbols.Symbol { if m.resolver == nil || m.resolver.Index() == nil { return nil } + m.resolver.ReadName(fqn) if cached, ok := m.libSymbols[fqn]; ok { return cached } @@ -857,6 +860,7 @@ func (m *Model) libSymbol(fqn string) *symbols.Symbol { if found == nil && len(matches) == 1 { found = matches[0] } + journal(m, m.libSymbols, fqn, nil) m.libSymbols[fqn] = found return found } diff --git a/internal/core/symbols/changes.go b/internal/core/symbols/changes.go new file mode 100644 index 0000000000..e15bacbea1 --- /dev/null +++ b/internal/core/symbols/changes.go @@ -0,0 +1,103 @@ +package symbols + +// Changes is what a run of writes to an index changed, at the granularity a ReadRecorder +// records reads at: names, namespaces and documents. Reads meeting none saw nothing move. +type Changes struct { + Names map[string]bool + Namespaces map[string]bool + Docs map[string]bool +} + +// Empty reports whether nothing changed. +func (c Changes) Empty() bool { + return len(c.Names) == 0 && len(c.Namespaces) == 0 && len(c.Docs) == 0 +} + +// Registered reports whether the index's own tables changed: every write to +// them records a namespace or a document, so names alone are a caller's. +func (c Changes) Registered() bool { + return len(c.Namespaces) > 0 || len(c.Docs) > 0 +} + +func newChanges() *Changes { + return &Changes{Names: map[string]bool{}, Namespaces: map[string]bool{}, Docs: map[string]bool{}} +} + +// TrackChanges starts recording what writes to the index change, for +// TakeChanges to hand out. An index that is never asked records nothing. +func (idx *Index) TrackChanges() { + if idx.changes == nil { + idx.changes = newChanges() + } +} + +// TakeChanges returns what changed since the last call and starts over. +func (idx *Index) TakeChanges() Changes { + if idx.changes == nil { + return Changes{} + } + out := *idx.changes + idx.changes = newChanges() + return out +} + +func (idx *Index) changedName(fqn string) { + if idx.changes == nil { + return + } + idx.changes.Names[fqn] = true + parent, _ := splitFQN(fqn) + idx.changes.Namespaces[parent] = true +} + +func (idx *Index) changedNamespace(fqn string) { + if idx.changes == nil { + return + } + idx.changes.Namespaces[fqn] = true +} + +func (idx *Index) changedDoc(name string) { + if idx.changes == nil { + return + } + idx.changes.Docs[name] = true +} + +// A ReadRecorder is told what an index read is about — a name looked up, a namespace +// enumerated or its children read, a document's root or kind, the whole table. +type ReadRecorder interface { + ReadName(fqn string) + ReadNamespace(fqn string) + ReadDocument(name string) + ReadAllNames() +} + +// SetReadRecorder installs the recorder the index's reads report to, or none. +func (idx *Index) SetReadRecorder(r ReadRecorder) { + idx.reads = r +} + +func (idx *Index) readName(fqn string) { + if idx.reads != nil { + idx.reads.ReadName(fqn) + } +} + +func (idx *Index) readNamespace(fqn string) { + if idx.reads != nil { + idx.reads.ReadNamespace(fqn) + } +} + +func (idx *Index) readDocument(name string) { + if idx.reads != nil { + idx.reads.ReadDocument(name) + } +} + +func (idx *Index) readAllNames() { + if idx.reads != nil { + idx.reads.ReadAllNames() + } +} diff --git a/internal/core/symbols/changes_test.go b/internal/core/symbols/changes_test.go new file mode 100644 index 0000000000..8e0dbeb62c --- /dev/null +++ b/internal/core/symbols/changes_test.go @@ -0,0 +1,104 @@ +package symbols + +import ( + "reflect" + "sort" + "testing" +) + +// recordedReads collects what an index reports read, as a resolver would. +type recordedReads struct { + names, namespaces, docs []string + all bool +} + +func (r *recordedReads) ReadName(fqn string) { r.names = append(r.names, fqn) } +func (r *recordedReads) ReadNamespace(fqn string) { r.namespaces = append(r.namespaces, fqn) } +func (r *recordedReads) ReadDocument(name string) { r.docs = append(r.docs, name) } +func (r *recordedReads) ReadAllNames() { r.all = true } +func (r *recordedReads) sorted(s []string) []string { sort.Strings(s); return s } + +func keys(m map[string]bool) []string { + out := make([]string, 0, len(m)) + for k := range m { + out = append(out, k) + } + sort.Strings(out) + return out +} + +func TestTakeChangesNamesWhatAReplacementMoved(t *testing.T) { + idx := buildIndex(t, map[string]string{ + "a.sysml": "package P { part def X; }", + "b.sysml": "package Q { part def Y; }", + }) + idx.TrackChanges() + if ch := idx.TakeChanges(); !ch.Empty() { + t.Fatalf("changes before any write: %v", ch) + } + addDoc(t, idx, "a.sysml", "package P { part def Z; }") + ch := idx.TakeChanges() + if got := keys(ch.Names); !reflect.DeepEqual(got, []string{"P", "P::X", "P::Z"}) { + t.Fatalf("Names = %v, want [P P::X P::Z]", got) + } + if !ch.Namespaces["P"] || ch.Namespaces["Q"] { + t.Fatalf("Namespaces = %v, want P and not Q", keys(ch.Namespaces)) + } + if got := keys(ch.Docs); !reflect.DeepEqual(got, []string{"a.sysml"}) { + t.Fatalf("Docs = %v, want [a.sysml]", got) + } + if ch := idx.TakeChanges(); !ch.Empty() { + t.Fatalf("changes were not taken: %v", ch) + } +} + +func TestTakeChangesNamesWhatARemovalMoved(t *testing.T) { + idx := buildIndex(t, map[string]string{ + "a.sysml": "package P { part def X; }", + }) + idx.TrackChanges() + idx.RemoveDocument("a.sysml") + ch := idx.TakeChanges() + if !ch.Names["P::X"] || !ch.Docs["a.sysml"] { + t.Fatalf("removal recorded names %v, docs %v; want P::X and a.sysml", keys(ch.Names), keys(ch.Docs)) + } +} + +func TestTakeChangesNamesALibraryMarking(t *testing.T) { + idx := buildIndex(t, map[string]string{"lib.sysml": "package L { part def X; }"}) + idx.TrackChanges() + idx.MarkLibrary("lib.sysml") + if ch := idx.TakeChanges(); !ch.Docs["lib.sysml"] { + t.Fatalf("marking a document as library recorded %v, want the document", ch) + } +} + +func TestReadRecorderHearsWhatIsRead(t *testing.T) { + idx := buildIndex(t, map[string]string{ + "a.sysml": "package P { part def X; }", + }) + rec := &recordedReads{} + idx.SetReadRecorder(rec) + idx.LookupQualified("P::X") + idx.LookupQualified("P::Missing") + idx.DocumentRoot("a.sysml") + if got := rec.sorted(rec.names); !reflect.DeepEqual(got, []string{"P::Missing", "P::X"}) { + t.Fatalf("names read = %v, want [P::Missing P::X]", got) + } + if got := rec.sorted(rec.docs); !reflect.DeepEqual(got, []string{"a.sysml"}) { + t.Fatalf("documents read = %v, want [a.sysml]", got) + } + if rec.all { + t.Fatal("a lookup by name reported a scan of the whole name table") + } + idx.WorkspaceDocuments() + if !rec.all { + t.Fatal("listing the documents did not report a scan of the whole name table") + } + idx.SetReadRecorder(nil) + rec.names = nil + idx.LookupQualified("P::X") + if len(rec.names) != 0 { + t.Fatalf("reads reported after the recorder was removed: %v", rec.names) + } +} diff --git a/internal/core/symbols/filter.go b/internal/core/symbols/filter.go index d6e81229c6..0824f28c12 100644 --- a/internal/core/symbols/filter.go +++ b/internal/core/symbols/filter.go @@ -173,6 +173,7 @@ func (f ElementFilter) Same(g ElementFilter) bool { // NamespaceFiltersOf returns the filter conditions declared by the namespace // registered under fqn, over the documents declaring it in name order. func (idx *Index) NamespaceFiltersOf(fqn string) []ElementFilter { + idx.readNamespace(fqn) byDoc := idx.nsFilters.at(fqn) if len(byDoc) == 0 { return nil @@ -233,6 +234,7 @@ func (idx *Index) forgetNamespaceFilters(fqn, doc string) { // under the filters the namespace now declares, and the members it takes back // meanwhile mark the namespaces importing it onward for expansion too. func (idx *Index) refilter(fqn string) { + idx.changedNamespace(fqn) idx.lastTargets.del(fqn) idx.purgeReexportsUnder(fqn) } diff --git a/internal/core/symbols/identity.go b/internal/core/symbols/identity.go index 0073e093c2..0de3bec826 100644 --- a/internal/core/symbols/identity.go +++ b/internal/core/symbols/identity.go @@ -1,6 +1,10 @@ package symbols -import "github.com/Open-MBEE/OpenSysML/internal/core/source" +import ( + "fmt" + + "github.com/Open-MBEE/OpenSysML/internal/core/source" +) // ElementKey identifies the declaration a symbol was built from, since a document // and the global index build their own symbol for one declaration. A symbol with no @@ -22,6 +26,14 @@ func KeyOf(sym *Symbol) ElementKey { return ElementKey{doc: sym.DocName, span: sym.DeclSpan} } +// String spells the key, distinct for distinct elements, for use in a name. +func (k ElementKey) String() string { + if k.doc == "" { + return fmt.Sprintf("%p", k.sym) + } + return fmt.Sprintf("%s\x00%d\x00%d", k.doc, k.span.Offset, k.span.Len) +} + // SameElement reports whether a and b denote one element, whichever scope tree // each was reached through. A nil symbol denotes no element, itself included. func SameElement(a, b *Symbol) bool { diff --git a/internal/core/symbols/index.go b/internal/core/symbols/index.go index 65f4b98acd..0f133dc976 100644 --- a/internal/core/symbols/index.go +++ b/internal/core/symbols/index.go @@ -128,6 +128,11 @@ type Index struct { // usages with an `about` clause the document declares, so a model over a // shared frozen index reads them instead of walking its scope trees. aboutUsages map[string][]*Symbol + + // changes accumulates what writes changed since TakeChanges, once tracked, + // and reads is told what each read is about (see changes.go). + changes *Changes + reads ReadRecorder } // reexportClaim is one document's claim on a re-export: whether its imports @@ -356,6 +361,7 @@ func (idx *Index) AddDocumentWithKind(name string, root *ast.RootNamespace, kind func (idx *Index) addDocument(name string, root *ast.RootNamespace, kind source.Kind, explicitKind bool) { idx.mustBeWritable("AddDocument") idx.RemoveDocument(name) + idx.changedDoc(name) rs := Build(root) SetDocName(rs, name) idx.docRoots.set(name, rs) @@ -378,6 +384,7 @@ func (idx *Index) addDocument(name string, root *ast.RootNamespace, kind source. func (idx *Index) setWildcardImports(pkgFQN, doc string, imports []WildcardImport) { writableMap(idx.wildcardMeta, pkgFQN)[doc] = imports idx.lastTargets.del(pkgFQN) // its import set changed: expand it again + idx.changedNamespace(pkgFQN) } // ExpandWildcardImports adds re-exported symbols for every package with a @@ -672,9 +679,16 @@ func (idx *Index) register(fqn string, sym *Symbol) { } // link is register without noting the change to the parent namespace, for a -// caller that records it itself. +// caller that records it itself. Over a frozen library the symbols under a name +// are kept in declaration order; the library itself keeps the order its +// snapshot pins. func (idx *Index) link(fqn string, sym *Symbol) { - appendSlice(idx.fqn, fqn, sym) + if idx.base != nil { + insertSymbol(idx.fqn, fqn, sym) + } else { + appendSlice(idx.fqn, fqn, sym) + } + idx.changedName(fqn) parent, last := splitFQN(fqn) insertSorted(idx.children, parent, fqn) if parent != "" { @@ -712,6 +726,7 @@ func (idx *Index) unregisterSegment(fqn string) { // entirely once it names nothing. It leaves declaredAt alone: only the symbol's // own declaration owns that entry. func (idx *Index) deregister(fqn string, sym *Symbol) { + idx.changedName(fqn) syms := writableSlice(idx.fqn, fqn) for i, s := range syms { if s == sym { @@ -797,6 +812,7 @@ func (idx *Index) RemoveDocument(name string) { if !idx.knows(name) { return } + idx.changedDoc(name) library := idx.libraryDocs.at(name).Tier.Library() for _, e := range idx.contributions.at(name) { @@ -826,6 +842,7 @@ func (idx *Index) RemoveDocument(name string) { idx.wildcardMeta.del(pkgFQN) } idx.lastTargets.del(pkgFQN) // its import set changed: expand it again + idx.changedNamespace(pkgFQN) } for key := range idx.docReexports.at(name) { @@ -858,6 +875,7 @@ func (idx *Index) MarkLibraryTier(name string, tier LibraryTier) { // for when to call it. func (idx *Index) MarkLibraryDocument(name string, doc LibraryDocument) { idx.mustBeWritable("MarkLibraryDocument") + idx.changedDoc(name) if doc.Tier == TierNone { idx.libraryDocs.del(name) } else { @@ -1093,6 +1111,9 @@ func (idx *Index) reexportGated(fqn string, sym *Symbol, doc string, private boo for _, gate := range gates { widened = claim.record(gateRoute{private: private, filters: gate}) || widened } + if widened { + idx.changedName(fqn) + } // A namespace importing this one onward copied the narrower routes, so a // widened claim has to reach it too (see routesOnward). if parent, _ := splitFQN(fqn); widened && parent != "" { @@ -1135,6 +1156,7 @@ func (c *reexportClaim) record(route gateRoute) bool { // A private import's route only answers a lookup made from within the importing // namespace, which from names ("" for one made from anywhere else). func (idx *Index) ReexportGates(doc, fqn string, sym *Symbol, from string) [][]ElementFilter { + idx.readName(fqn) claims := idx.reexportDocs.at(reexportKey{fqn: fqn, sym: sym}) parent, _ := splitFQN(fqn) if parent == "" { @@ -1159,6 +1181,7 @@ func (idx *Index) ReexportVisible(doc, fqn string, sym *Symbol) bool { if parent, _ := splitFQN(fqn); parent != "" { return true } + idx.readName(fqn) if !idx.reexported.at(fqn).has(sym) { return true // declared under this name rather than borrowed } @@ -1220,6 +1243,7 @@ func (idx *Index) claimReexport(key reexportKey, doc string, public bool) *reexp docs[doc] = claim } claim.public = claim.public || public + idx.changedName(key.fqn) writableMap(idx.docReexports, doc)[key] = true idx.applyReexportMarks(key, docs) parent, _ := splitFQN(key.fqn) @@ -1236,6 +1260,7 @@ func (idx *Index) dropClaim(key reexportKey, doc string) { } docs := idx.writableClaims(key) delete(docs, doc) // the routes this document recorded go with its claim + idx.changedName(key.fqn) if len(docs) == 0 { idx.reexportDocs.del(key) idx.deregister(key.fqn, key.sym) @@ -1459,6 +1484,7 @@ func (idx *Index) LookupQualified(fqn string) []*Symbol { // fromFQN is the FQN of the referring namespace; "" means "from outside", which // is what an ordinary qualified reference elsewhere in the workspace gets. func (idx *Index) LookupQualifiedFrom(fqn, fromFQN string) []*Symbol { + idx.readName(fqn) syms := idx.fqn.at(fqn) imported := idx.reexported.at(fqn) if len(imported) == 0 { @@ -1507,6 +1533,7 @@ func (idx *Index) Declaring(fqn string) *Symbol { // which reaches cached symbols through LookupDirectChildren — asks here first // and stops, rather than resurfacing a name KerML 8.2.3.3 hides. func (idx *Index) HiddenFrom(fqn, fromFQN string) bool { + idx.readName(fqn) hidden := idx.hidden.at(fqn) if len(hidden) == 0 || withinNamespace(fromFQN, namespaceOf(fqn)) { return false @@ -1549,6 +1576,7 @@ func withinNamespace(fromFQN, ns string) bool { // FQNs returns every fully-qualified name registered in the index, sorted. func (idx *Index) FQNs() []string { + idx.readAllNames() out := idx.fqn.keys() sort.Strings(out) return out @@ -1566,6 +1594,7 @@ func (idx *Index) Registered(fn func(fqn string, syms []*Symbol)) { // segment is name, in name order. Used to suggest a candidate for a reference // whose qualifying namespace is not loaded. func (idx *Index) FQNsEndingIn(name string, limit int) []string { + idx.readAllNames() if name == "" || limit <= 0 { return nil } @@ -1580,6 +1609,7 @@ func (idx *Index) FQNsEndingIn(name string, limit int) []string { // namespace registered under fqn ("" for a document root), over the documents // declaring it in name order. func (idx *Index) WildcardImportsOf(fqn string) []WildcardImport { + idx.readNamespace(fqn) byDoc := idx.wildcardMeta.at(fqn) if len(byDoc) == 0 { return nil @@ -1627,6 +1657,7 @@ func (idx *Index) LookupDirectChildren(prefix string) []*Symbol { } func (idx *Index) lookupDirectChildren(key directChildrenKey) []*Symbol { + idx.readNamespace(key.prefix) generation := idx.generation.get() idx.directChildrenMu.Lock() idx.resetDirectChildrenCachesLocked(generation) @@ -1692,6 +1723,7 @@ func (idx *Index) LookupDirectChildrenNamedFrom(prefix, fromFQN, name string) [] } func (idx *Index) lookupDirectChildrenNamed(key directChildrenKey, name string) []*Symbol { + idx.readNamespace(key.prefix) generation := idx.generation.get() idx.directChildrenMu.Lock() idx.resetDirectChildrenCachesLocked(generation) @@ -1739,6 +1771,7 @@ type RootBinding struct { // 8.2.3.3). A caller gating those names by their element filters needs the name, // since a borrowed symbol's own name is not the root name it appears under. func (idx *Index) TopLevelBindings(doc string) []RootBinding { + idx.readNamespace("") claimed := idx.docReexports.at(doc) var out []RootBinding seen := make(map[*Symbol]bool) @@ -1871,17 +1904,20 @@ func namedOwner(scope *Scope) *Symbol { // DocumentOfRoot returns the name of the document whose root scope this is, or // "" for any other scope. func (idx *Index) DocumentOfRoot(scope *Scope) string { + idx.readDocument(idx.docOfRoot.at(scope)) return idx.docOfRoot.at(scope) } // DocumentRoot returns the root scope for the named document, or nil. func (idx *Index) DocumentRoot(name string) *Scope { + idx.readDocument(name) return idx.docRoots.at(name) } // Documents returns the names of every document with a root scope, bundled // library content included, sorted for deterministic iteration. func (idx *Index) Documents() []string { + idx.readAllNames() out := append([]string(nil), idx.docRoots.keys()...) sort.Strings(out) return out @@ -1891,6 +1927,7 @@ func (idx *Index) Documents() []string { // content — every document with a root scope that is not marked as bundled // library content — sorted for deterministic iteration. func (idx *Index) WorkspaceDocuments() []string { + idx.readAllNames() var out []string for _, name := range idx.docRoots.keys() { if !idx.libraryDocs.at(name).Tier.Library() { @@ -1904,6 +1941,7 @@ func (idx *Index) WorkspaceDocuments() []string { // DocumentKind returns a document's recorded language, or infers it from its // name when the document was added without an explicit language. func (idx *Index) DocumentKind(name string) source.Kind { + idx.readDocument(name) if kind, ok := idx.docKinds.get(name); ok { return kind } diff --git a/internal/core/symbols/layer.go b/internal/core/symbols/layer.go index 2b625fefd4..e16e8eca0c 100644 --- a/internal/core/symbols/layer.go +++ b/internal/core/symbols/layer.go @@ -170,6 +170,28 @@ func appendSlice[K comparable, S ~[]E, E any](l *layer[K, S], k K, e E) { l.set(k, append(out, e)) } +// insertSymbol adds sym to the symbols under fqn in declaration order (document +// name, then offset) after any the layer below holds, so what a name resolves +// to does not depend on the order the documents were indexed in. +func insertSymbol(l *layer[string, []*Symbol], fqn string, sym *Symbol) { + shared, _ := l.below(fqn) + syms := writableSlice(l, fqn) + i := len(syms) + for i > 0 && !(i-1 < len(shared) && shared[i-1] == syms[i-1]) && declaredAfter(syms[i-1], sym) { + i-- + } + l.set(fqn, slices.Insert(syms, i, sym)) +} + +// declaredAfter reports whether a is declared after b: in a later document, or +// later in the same one. +func declaredAfter(a, b *Symbol) bool { + if a.DocName != b.DocName { + return a.DocName > b.DocName + } + return a.DeclSpan.Offset > b.DeclSpan.Offset +} + // insertSorted adds s to the sorted, duplicate-free slice under k, copying a // slice the layer below owns first (see writableSlice). func insertSorted(l *layer[string, []string], k, s string) { diff --git a/internal/lsp/files.go b/internal/lsp/files.go index 0e05dc454d..5519ee8368 100644 --- a/internal/lsp/files.go +++ b/internal/lsp/files.go @@ -260,7 +260,8 @@ func (s *Server) refreshOpenDiagnostics(ctx context.Context, except string) { } // queueOpenDiagnostics is refreshOpenDiagnostics once an editor burst settles; -// a refresh re-analyzes every open document, too much to pay per keystroke. +// a refresh re-analyzes every open document the edit reached and republishes +// the others from the workspace's cache, still too much to pay per keystroke. func (s *Server) queueOpenDiagnostics(ctx context.Context, except string) { if s.crossDoc == nil { s.refreshOpenDiagnostics(ctx, except) diff --git a/internal/stressmodel/bench_test.go b/internal/stressmodel/bench_test.go index 44dacd05e5..bf042b39e2 100644 --- a/internal/stressmodel/bench_test.go +++ b/internal/stressmodel/bench_test.go @@ -114,3 +114,89 @@ func BenchmarkEditBeside(b *testing.B) { }) } } + +// openFiles opens every file of a network in a workspace and analyzes each, +// failing on any diagnostic. +func openFiles(tb testing.TB, ws *model.Workspace, files []File) { + tb.Helper() + for _, f := range files { + ws.Open(f.Name, []byte(f.Source), 1) + } + for _, f := range files { + for _, d := range ws.Diagnostics(f.Name) { + tb.Fatalf("%s: %s", f.Name, d.Message) + } + } +} + +// BenchmarkLoadFiles measures analyzing the network split into one document per +// plane, every document through one workspace: what a multi-file model costs +// over the same model in one file. +func BenchmarkLoadFiles(b *testing.B) { + for _, n := range networkSizes { + files, stats := network(n).Split() + b.Run(fmt.Sprintf("satellites=%d/files=%d", stats.Satellites, len(files)), func(b *testing.B) { + b.ReportAllocs() + for i := 0; i < b.N; i++ { + openFiles(b, model.NewWorkspace(), files) + } + }) + } +} + +// BenchmarkEditImported measures the worst edit in a multi-file network: a +// keystroke in the library every plane imports, followed by the diagnostics of +// every open file, as the editor's sweep asks for them. +func BenchmarkEditImported(b *testing.B) { + for _, n := range networkSizes { + files, stats := network(n).Split() + lib := files[0] + b.Run(fmt.Sprintf("satellites=%d/files=%d", stats.Satellites, len(files)), func(b *testing.B) { + ws := model.NewWorkspace() + openFiles(b, ws, files) + b.ReportAllocs() + b.ResetTimer() + for i := 0; i < b.N; i++ { + ws.Update(lib.Name, []byte(lib.Source+fmt.Sprintf("\n// %d\n", i)), i+2) + for _, f := range files { + _ = ws.Diagnostics(f.Name) + } + } + }) + } +} + +// TestEditsHoldNoStaleState checks that a workspace edited many times keeps no +// more of the replaced documents than one edited once: the heap a long editing +// session holds is that of the current model, not of every version it saw. +func TestEditsHoldNoStaleState(t *testing.T) { + if testing.Short() { + t.Skip("edits a 32-satellite network a thousand times") + } + const small = "package Ops { private import SatelliteNetwork::Constellation::*; part spare : Sat0; }" + src, _ := network(networkSizes[0]).Source() + ws := model.NewWorkspace() + ws.Open("satnet.sysml", []byte(src), 1) + ws.Open("ops.sysml", []byte(small), 1) + edit := func(i int) { + ws.Update("ops.sysml", []byte(small+fmt.Sprintf("\n// %d\n", i)), i+2) + _ = ws.Diagnostics("ops.sysml") + if i%50 == 0 { + ws.Update("satnet.sysml", []byte(src+fmt.Sprintf("\n// %d\n", i)), i+2) + } + _ = ws.Diagnostics("satnet.sysml") + } + edit(0) + warm := liveHeap() + for i := 1; i <= 1000; i++ { + edit(i) + } + after := liveHeap() + runtime.KeepAlive(ws) + // Memo tables grow a bounded amount under delete-and-reinsert churn; a + // quarter more is state that outlived the document it was computed for. + if after > warm+warm/4 { + t.Fatalf("live heap grew from %d to %d bytes over 1000 edits", warm, after) + } + t.Logf("live heap after 1 edit %d bytes, after 1000 edits %d bytes", warm, after) +} diff --git a/internal/stressmodel/satnet.go b/internal/stressmodel/satnet.go index dad0df147f..6d11ce4d1d 100644 --- a/internal/stressmodel/satnet.go +++ b/internal/stressmodel/satnet.go @@ -357,6 +357,14 @@ func (g *generator) constellation(n SatelliteNetwork) { for k := 0; k < n.GroundStations; k++ { g.groundStation(k) } + g.network(n) + g.line(1, "}") + g.line(0, "}") +} + +// network writes the part definition joining every satellite and ground +// station, and its one usage. +func (g *generator) network(n SatelliteNetwork) { g.line(0, "") g.decl(2, "part def Network {") total := n.Planes * n.Satellites @@ -388,8 +396,6 @@ func (g *generator) constellation(n SatelliteNetwork) { g.decl(3, "attribute satelliteCount : Integer = %d;", total) g.line(2, "}") g.decl(2, "part network : Network;") - g.line(1, "}") - g.line(0, "}") } // link writes a crosslink between two satellites' crosslink terminals. diff --git a/internal/stressmodel/satnet_test.go b/internal/stressmodel/satnet_test.go index c6e39bb86f..b57fa9e0a3 100644 --- a/internal/stressmodel/satnet_test.go +++ b/internal/stressmodel/satnet_test.go @@ -5,6 +5,7 @@ import ( "testing" "github.com/Open-MBEE/OpenSysML/internal/core/conformance" + "github.com/Open-MBEE/OpenSysML/internal/core/model" "github.com/Open-MBEE/OpenSysML/internal/repl" ) @@ -59,3 +60,46 @@ func TestSatelliteNetworkScales(t *testing.T) { t.Errorf("first satellite definition missing from:\n%s", one) } } + +// TestSatelliteNetworkFilesValidate keeps the multi-file form in step with the +// single one: the same constellation split by plane loads clean under strict +// conformance, one document per file, and every satisfy assertion holds. +func TestSatelliteNetworkFilesValidate(t *testing.T) { + n := SatelliteNetwork{Planes: 2, Satellites: 2, GroundStations: 1} + files, stats := n.Split() + if len(files) != n.Planes+2 { + t.Fatalf("got %d files, want the library, %d planes and the network", len(files), n.Planes) + } + _, whole := n.Source() + if stats.Satellites != whole.Satellites || stats.Requirements != whole.Requirements || stats.Connections != whole.Connections { + t.Fatalf("split stats %+v, single-file stats %+v", stats, whole) + } + ws := model.NewWorkspace(model.WithConformanceMode(conformance.ModeOf(true))) + for _, f := range files { + ws.Open(f.Name, []byte(f.Source), 1) + } + for _, f := range files { + for _, d := range ws.Diagnostics(f.Name) { + t.Errorf("%s: %s", f.Name, d.Message) + } + } + + s := repl.NewSession() + s.SetConformanceMode(conformance.ModeOf(true)) + sources := make([]repl.SourceFile, 0, len(files)) + for _, f := range files { + sources = append(sources, repl.SourceFile{Name: f.Name, Text: f.Source}) + } + for _, d := range s.SubmitFiles(sources).Diagnostics { + t.Errorf("diagnostic: %s", d.Message) + } + verdicts := s.CheckSatisfy("") + if len(verdicts) != stats.Requirements { + t.Fatalf("got %d satisfy verdicts, want %d", len(verdicts), stats.Requirements) + } + for _, v := range verdicts { + if !v.Holds() { + t.Errorf("%s: %v", v.Subject, v.Lines) + } + } +} diff --git a/internal/stressmodel/split.go b/internal/stressmodel/split.go new file mode 100644 index 0000000000..658341eae4 --- /dev/null +++ b/internal/stressmodel/split.go @@ -0,0 +1,69 @@ +package stressmodel + +import ( + "fmt" + "strings" +) + +// File is one document of a network split across files. +type File struct { + Name, Source string +} + +// Split generates the network Generate writes as one document per orbital plane +// beside the shared library and the constellation joining the planes. Each plane +// imports the library by qualified name, so a file resolves against the others. +func (n SatelliteNetwork) Split() ([]File, Stats) { + var b strings.Builder + g := &generator{b: &b} + take := func(name string) File { + f := File{Name: name, Source: b.String()} + g.stats.Bytes += b.Len() + b.Reset() + return f + } + files := make([]File, 0, n.Planes+2) + + g.library() + g.line(0, "}") + files = append(files, take("library.sysml")) + + id := 0 + for p := 0; p < n.Planes; p++ { + g.decl(0, "package Plane%d {", p) + g.planeImports() + g.line(0, "") + for s := 0; s < n.Satellites; s++ { + g.satellite(id, p, s) + id++ + } + g.line(0, "}") + files = append(files, take(fmt.Sprintf("plane%03d.sysml", p))) + } + + g.decl(0, "package Constellation {") + g.planeImports() + g.line(1, "private import SatelliteNetwork::*;") + for p := 0; p < n.Planes; p++ { + g.line(1, "private import Plane%d::*;", p) + } + g.line(0, "") + for k := 0; k < n.GroundStations; k++ { + g.groundStation(k) + } + g.network(n) + g.line(0, "}") + files = append(files, take("constellation.sysml")) + return files, g.stats +} + +// planeImports writes what a file outside the library package imports of it. +func (g *generator) planeImports() { + g.line(1, "private import ScalarValues::*;") + g.line(1, "private import ISQ::*;") + g.line(1, "private import SI::*;") + g.line(1, "private import SatelliteNetwork::Interfaces::*;") + g.line(1, "private import SatelliteNetwork::Platform::*;") + g.line(1, "private import SatelliteNetwork::Requirements::*;") + g.line(1, "private import SatelliteNetwork::Behavior::*;") +}