Repository navigation
Add unattended scalability and generation fidelity benchmarks - #49
Merged
Merged
Conversation
Spatial geometry now honours rotationZ. A shared oriented-box module (shared/spatial-geometry.ts, mirrored in frontend/src/services/spatial) builds each placed element's rotated footprint extruded over its vertical interval. The JS/OCL spatial helpers keep their names and signatures but use separating-axis overlap tests and exact convex-polygon clearance, so rotated elements no longer get axis-aligned answers. Metamodels gain spatialRules: noOverlap, minClearance, maxDistance, within, and adjacentTo over placed instances of a source and a target metaclass (subclasses included), in plan or volume mode. A pure evaluator (shared/spatial-rules.ts plus the frontend mirror) runs in browser validation, which marks both elements of an offending pair in 2D and 3D, and in the backend validation engine used by the headless pipeline. A parity test keeps the two copies identical and checks that they report the same issues. Rules are persisted in a new metamodels.spatialRules column, validated on create and update, carried through checkpoints and metamodel evolution (a rule on a removed class blocks the evolution), removed with a deleted class, and exported to Ecore as the EPackage annotation spatialdsl/spatialRules. The metamodel editor gains a Spatial Rules section with an add/edit dialog. The Smart Warehouse example declares a FixedEquipment supertype and four rules that its bundled model satisfies, and its unbounded controller references now use '*'.
The UsdGeom schema declares Cube.size, Sphere.radius, and Cylinder.radius and height as double. The generated layout and physics layers and the project-authored inspection drone and F1TENTH layers authored them as float, which UsdValidation reports as AttributeTypeMismatch on the composed warehouse stage. Values are unchanged.
benchmarks/scalability.ts generates seeded synthetic Smart Warehouse models (100 to 10,000 placed elements, random yaw, about 5 percent seeded rule violations) and reports median timings of spatial rule evaluation, backend model validation, and OpenUSD code generation, plus the model size and machine details. It fails unless every seeded violation yields exactly one rule issue. benchmarks/fidelity.py applies 200 seeded pose edits to the bundled model, regenerates through the backend generation engine after each edit, and compares every element's pose in the composed USD stage (read with pxr), the Visual Components location files, and the Omniverse layout JSON with the model. It also validates the composed stages with UsdValidation and checks that generation is byte-for-byte deterministic. Both scripts run in-process without a browser, server, or database, and write CSV, JSON, and Markdown to the git-ignored benchmarks/results/.
Contributor
There was a problem hiding this comment.
🟡 Changes recommended
Fidelity checks can pass despite incomplete or failed validation, and metamodel import/update paths can retain invalid spatial rules.
4 open findings
What changed in this PR
Adds unattended scalability and generation-fidelity benchmarks, incorporating the dependent spatial-rules functionality and OpenUSD schema fixes.
Changes:
- Adds rotation-aware geometry and declarative spatial-rule evaluation across frontend and backend.
- Adds scalability and fidelity benchmark suites.
- Corrects OpenUSD primitive attribute types and expands persistence, interoperability, documentation, and tests.
| File | Description |
|---|---|
.gitignore |
Ignores benchmark artifacts. |
README.md |
Documents spatial rules. |
shared/types/index.ts |
Adds shared spatial-rule types. |
shared/spatial-rules.ts |
Implements rule validation and evaluation. |
shared/spatial-geometry.ts |
Implements oriented spatial geometry. |
frontend/src/models/types.ts |
Adds frontend rule types. |
frontend/src/services/spatial/index.ts |
Exports spatial modules. |
frontend/src/services/spatial/spatial-rules.ts |
Mirrors rule evaluation. |
frontend/src/services/spatial/spatial-geometry.ts |
Mirrors spatial geometry. |
frontend/src/services/model/model-validation.service.ts |
Runs spatial rules during validation. |
frontend/src/services/metamodel/metamodel.service.ts |
Validates imported rules. |
frontend/src/services/metamodel/metamodel-api-sync.service.ts |
Synchronizes rules through the API. |
frontend/src/services/metamodel/metaclass-crud.service.ts |
Removes rules for deleted classes. |
frontend/src/services/metamodel/ecore.service.ts |
Adds Ecore rule interoperability. |
frontend/src/services/diagram/view-validation.service.ts |
Marks all related elements. |
frontend/src/services/constraint/spatial.helpers.ts |
Makes helpers rotation-aware. |
frontend/src/examples/data/smart-warehouse-omniverse-project.json |
Fixes generated USD types. |
frontend/src/examples/data/smart-warehouse-metamodel.json |
Adds example rules and inheritance. |
frontend/src/components/metamodel/SpatialRulesEditor.tsx |
Adds rule-authoring UI. |
frontend/src/components/metamodel/components/sidebar/MetamodelSidebar.tsx |
Integrates the rules editor. |
frontend/src/__tests__/services/view-validation.test.ts |
Tests related-element markers. |
frontend/src/__tests__/services/spatial-rules.test.ts |
Tests rule semantics. |
frontend/src/__tests__/services/spatial-helpers.test.ts |
Tests rotated geometry. |
frontend/src/__tests__/services/ecore-xmi-interop.test.ts |
Tests Ecore rule round trips. |
frontend/src/__tests__/components/SpatialRulesEditor.test.tsx |
Tests rule editor behavior. |
examples/omniverse-assets/warehouse-kit/inspection_drone.usda |
Corrects USD schema types. |
examples/omniverse-assets/nvidia-f1tenth-amr/layers/visual.usda |
Corrects visual-layer types. |
examples/omniverse-assets/nvidia-f1tenth-amr/layers/collision.usda |
Corrects collision-layer types. |
examples/omniverse-assets/nvidia-f1tenth-amr/layers/articulation.usda |
Corrects articulation-layer types. |
examples/codegen-projects/smart-warehouse-omniverse-project.json |
Corrects USD generation templates. |
docs/user-guide/spatial-rules.md |
Documents rule semantics. |
docs/user-guide/metamodels.md |
Documents metamodel integration. |
docs/reference/data-model.md |
Documents stored rule data. |
docs/reference/api.md |
Documents rule API fields. |
docs/README.md |
Links spatial-rule documentation. |
benchmarks/scalability.ts |
Benchmarks evaluation and generation. |
benchmarks/README.md |
Documents benchmark execution. |
benchmarks/package.json |
Defines benchmark commands. |
benchmarks/lib/synthetic-warehouse.ts |
Generates seeded benchmark models. |
benchmarks/lib/offline-env.ts |
Configures offline backend imports. |
benchmarks/fidelity.py |
Checks generated-output fidelity. |
benchmarks/fidelity-generate.ts |
Generates edited benchmark bundles. |
backend/src/services/sirius-interop.service.ts |
Exports rules in Sirius Ecore. |
backend/src/services/pipeline/model-validation.engine.ts |
Evaluates rules headlessly. |
backend/src/services/pipeline/headless-pipeline.service.ts |
Maps persisted rules. |
backend/src/services/metamodel.service.ts |
Persists and validates rules. |
backend/src/services/lifecycle/project-checkpoint.service.ts |
Checkpoints spatial rules. |
backend/src/services/lifecycle/metamodel-evolution.service.ts |
Handles rules during evolution. |
backend/src/__tests__/services/spatial-rules-parity.test.ts |
Tests frontend/backend parity. |
backend/src/__tests__/services/sirius-interop.service.test.ts |
Tests Sirius rule export. |
backend/src/__tests__/services/sharing.service.test.ts |
Updates persisted fixtures. |
backend/src/__tests__/services/project-checkpoint.service.test.ts |
Tests rule checkpointing. |
backend/src/__tests__/services/pipeline-engines.test.ts |
Tests pipeline evaluation. |
backend/src/__tests__/services/metamodel.service.test.ts |
Tests rule persistence. |
backend/src/__tests__/services/metamodel-evolution.service.test.ts |
Tests evolution behavior. |
backend/prisma/schema.prisma |
Adds the rule JSON column. |
backend/prisma/migrations/20261011000000_add_metamodel_spatial_rules/migration.sql |
Creates the database column. |
🧠 Review effort: Balanced
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Comment on lines
213
to
+221
| if (data.classes !== undefined) { | ||
| this.assertValidClassesVerticalPlacement(data.classes); | ||
| } | ||
| if (data.spatialRules !== undefined) { | ||
| const classes = data.classes !== undefined | ||
| ? data.classes | ||
| : (await prisma.metamodel.findUnique({ where: { id }, select: { classes: true } }))?.classes; | ||
| this.assertValidSpatialRules(data.spatialRules, classes); | ||
| } |
Comment on lines
+144
to
+145
| if "yaw" in row: | ||
| tally.compare("omniverse-layout-json", edit, element["name"], "yaw", row["yaw"], element["rotationZ"]) |
| lines += [f"Checker messages for {check['edit']}:", "", *[f"- {message}" for message in details], ""] | ||
| (RESULTS / "fidelity.md").write_text("\n".join(lines)) | ||
| print("\n".join(lines)) | ||
| return 0 if not tally.mismatches and manifest["determinism"]["byteIdentical"] else 1 |
Comment on lines
+154
to
+165
| rules.push({ | ||
| id: uuidv4(), | ||
| name, | ||
| kind: parsed.kind, | ||
| sourceClassId, | ||
| targetClassId, | ||
| ...(parsed.distanceMm !== undefined && { distanceMm: parsed.distanceMm }), | ||
| ...(parsed.mode !== undefined && { mode: parsed.mode }), | ||
| severity: parsed.severity || 'error', | ||
| ...(parsed.message !== undefined && { message: parsed.message }), | ||
| ...(parsed.enabled !== undefined && { enabled: parsed.enabled }), | ||
| }); |
The warehouse example said a violation could be provoked in the 2D view, but 2D moves only change the 3D pose that rules check when an element's 2D and 3D plan positions coincide, which is not the case for the bundled examples. Say so, and point to the 3D view and its Position & Rotation fields instead.
The backend Dockerfile compiles src/, tests included, with only backend/ and shared/ in the image. The spatial rules parity test imported the frontend mirror and the warehouse fixtures statically, so the image build failed. Load those files at run time instead, as the Sirius fixture test does.
Metamodel evolution now runs the full rule validation (kind, distance, mode, severity, duplicate IDs) before a preview can be applied, a classes-only update checks the stored rules against the new classes, and Ecore import skips annotation rules that are not valid rules with a warning instead of importing them.
…ut yaw scope The script now exits non-zero when UsdValidation reports an error or a warning, not only on mismatches or nondeterminism. warehouse_layout.json gives a yaw only for drones, so the check requires it there and the results and README say that the other elements' yaw is compared in USD and Visual Components.
…ure/evaluation-benchmarks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Depends on #45 (spatial rules); this branch contains its commit. Review the commits after "Add declarative spatial rules and yaw-aware spatial geometry".
Two unattended evaluation scripts under
benchmarks/, run in-process without a browser, server, or database (benchmarks/README.md):npm run scalability: seeded synthetic Smart Warehouse models with 100 to 10,000 placed elements (random yaw, about 5 percent seeded violations). Reports the median of 5 runs of spatial rule evaluation, full backend model validation, and OpenUSD code generation, plus model size and machine details, as CSV, JSON, and Markdown. It fails unless every seeded violation yields exactly one rule issue.npm run fidelity: 200 seeded pose edits (move, rotate, elevate) on the bundled warehouse model, regenerating after each. Every element's pose in the composed USD stage (read withpxrfromusd-core), the Visual Components location files, and the Omniverse layout JSON is compared with the model (tolerance 1e-6). The first and last stages are checked withUsdValidation(UsdUtils.ComplianceCheckeris not available in usd-core 26), and generation is checked for byte-for-byte determinism.benchmarks/results/andbenchmarks/.venv/are git-ignored.The validation run found a real defect, fixed in a separate commit: the generated layout and physics layers and the project-authored drone and F1TENTH asset layers authored
Cube.size,Sphere.radius, andCylinder.radius/heightasfloat, but the UsdGeom schema declares themdouble(101AttributeTypeMismatcherrors on the composed stage). Values are unchanged; the stage now validates with 0 errors and 0 warnings.Results (Apple M2, 16 GiB, macOS 26 / Darwin 25.5, Node 24.14)
Every seeded violation produced exactly one rule issue at every size. Rule evaluation stays under one second at 10,000 elements, so no broad phase was added (the plan's threshold was about 10 s).
Fidelity: 105,927 values compared across 201 generated bundles, 0 mismatches;
UsdValidation(24 validators) passes on both checked stages with 0 errors and 0 warnings; generating the unedited model twice gives byte-identical output (14 files). Total runtime is under 20 s for scalability and under 10 s for fidelity.