Add standard performance regression checks to Process-PSModule workflows #404

Teams using Process-PSModule need a standard way to detect and surface performance regressions during pull requests so slower behavior is visible before merge. Recent benchmark work in PSModule/Sodium#69 showed measurable startup and operation deltas, but each repository currently has to build custom wiring to compare runs and enforce thresholds.

Request

Desired capability

Process-PSModule should provide a built-in performance testing capability that can run benchmark scripts, compare results against a baseline, and clearly flag regressions to the user in GitHub Actions.

Acceptance criteria

• A repository using Process-PSModule can enable performance checks through configuration without custom workflow authoring.
• Pull request runs compare candidate results to a baseline and compute per-metric deltas.
• Slowdowns above configured thresholds are flagged in the workflow output and in a user-visible summary.
• Configured blocking thresholds fail the check when exceeded so regressions are gated.
• Non-blocking degradations are still surfaced as warnings with metric-level detail.
• Baseline, candidate, and comparison artifacts are persisted so results are auditable.
• Documentation explains the configuration contract, expected benchmark output schema, and regression policy behavior.

Technical decisions

Integration point: Add a first-class performance stage in the shared Process-PSModule pipeline so repositories get consistent behavior from the reusable workflow.

Benchmark contract: Define a standard JSON result schema (scenario, metric, unit, value, sample-count metadata) that repository benchmark scripts emit.

Baseline strategy: Resolve baseline data from a successful run on the default branch, with explicit handling when no baseline exists yet.

Regression policy: Support configurable warning and failure thresholds (percentage-based by default), with repository-level overrides in PSModule.yml.

User signaling: Publish a markdown summary table with before/after/delta values and emit explicit warnings/errors for degraded metrics.

Noise handling: Require repeated samples and a deterministic aggregation rule (for example median) to reduce flakiness from transient runner variance.

Backward compatibility: Keep the capability opt-in initially to avoid breaking existing repositories; allow later migration to default-on once stability is proven.

Implementation plan

Specification

• Define performance configuration keys in PSModule.yml (enable flag, baseline source, warning/fail thresholds, artifact retention).
• Define and document the benchmark result schema consumed by the comparison stage.

Workflow implementation

• Add a performance stage/job to the reusable Process-PSModule workflow.
• Add baseline fetch, candidate execution, and comparison steps.
• Add deterministic handling for first-run/no-baseline scenarios.

Reporting and gating

• Generate a markdown summary with per-metric before/after/delta.
• Emit warnings for non-blocking degradations.
• Fail the job when blocking thresholds are exceeded.

Validation and docs

• Add tests for comparison logic and threshold evaluation.
• Add fixture-based tests for baseline-missing and mixed-improvement/regression scenarios.
• Update Process-PSModule documentation with setup, schema, and troubleshooting guidance.