Repository navigation
Move main desktop performance measurements to SDK 12 and net12.0 - #5344
LoopedBard3 wants to merge 4 commits into
Conversation
Keep MAUI/mobile on net11 with SDK12 and preserve older channel runner versions. Add net12 framework, package, tooling and pipeline support with focused regression coverage. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
There was a problem hiding this comment.
🟡 Changes recommended
Unresolved SDK pinning and framework defaults break supported net11 and Mono WASM workflows, while SDK metadata and mixed-target runner selection remain incorrect.
4 open issues
What changed in this PR
Moves main-channel desktop performance measurements to SDK 12 and net12.0, with compatibility handling for older frameworks and mobile targets.
Changes:
- Updates SDK pins, framework defaults, scenario coverage, and tooling test targets.
- Adds a net12-specific BenchmarkDotNet runner, AOT filtering compatibility, and CoreCLR WASM selection.
- Separates SDK acquisition from target selection and adds regression tests and documentation.
| File | Description |
|---|---|
| src/tools/ResultsComparer/Data.cs | Recognizes net12 result paths. |
| src/tools/ResultsComparer.Tests/ResultsComparer.Tests.csproj | Targets net12. |
| src/tools/ResultsComparer.Tests/DataTests.cs | Tests extraction across framework versions. |
| src/tools/Reporting/Reporting.Tests/Reporting.Tests.csproj | Targets net12. |
| src/tests/harness/BenchmarkDotNet.Extensions.Tests/BenchmarkDotNet.Extensions.Tests.csproj | Tests the net12 runner and benchmarks. |
| src/tests/harness/BenchmarkDotNet.Extensions.Tests/AotFilterTests.cs | Tests runtime-specific AOT exclusions. |
| src/scenarios/build-common/Blazor.PackageVersions.props | Adds net12 Blazor package selection. |
| src/scenarios/build-common/Blazor.Directory.Build.targets | Advances the framework warning threshold. |
| src/harness/BenchmarkDotNet.Extensions/BenchmarkDotNet.Extensions.csproj | Adds conditional net12 runner dependencies. |
| src/harness/BenchmarkDotNet.Extensions/AotFilterAttribute.cs | Restores AOT exclusion support. |
| src/benchmarks/micro/README.md | Documents net12 execution and compatibility limits. |
| src/benchmarks/micro/MicroBenchmarks.csproj | Adds net12 support and runner selection. |
| src/benchmarks/micro/libraries/System.Runtime/Perf.Uri.cs | Strongly types URI argument sources. |
| src/benchmarks/micro/libraries/System.Net.Security/SslStreamTests.Protocols.cs | Strongly types TLS argument sources. |
| scripts/tests/test_channel_map.py | Adds channel, framework, and package-selection tests. |
| scripts/run_performance_job.py | Adds independent target selection and mobile fallback. |
| scripts/micro_benchmarks.py | Selects net12 CoreCLR WASM and rejects Mono WASM. |
| scripts/ci_setup.py | Adds framework overrides and net11 SDK pin handling. |
| scripts/channel_map.py | Maps main to SDK/runtime 12. |
| global.net11.json | Preserves the SDK 11 configuration. |
| global.json | Pins the root SDK to version 12. |
| eng/Versions.props | Defines the net12-specific BenchmarkDotNet version. |
| eng/pipelines/templates/runtime-perf-job.yml | Changes the default framework to net12. |
| eng/pipelines/sdk-perf-jobs.yml | Updates SDK installation and tooling test commands. |
| eng/performance/sdk_scenarios.proj | Adds net12 to SDK scenarios. |
| docs/scenarios-workflow.md | Documents desktop and mobile target selection. |
Review effort: Balanced
| steps: [] | ||
| variables: [] | ||
| framework: net11.0 # Specify the appropriate framework when running release branches (ie net6.0 for release/6.0) | ||
| framework: net12.0 # Specify the appropriate framework when running release branches (ie net6.0 for release/6.0) |
There was a problem hiding this comment.
Fixed the Android/iOS part in 6dbc9d5: the runtime wrapper defaults those jobs to net11, keeps desktop/CoreCLR defaults at net12, and still gives an explicit framework override precedence.
The Mono WASM part remains open. Just setting net11 does not establish a coherent match with the SDK/runtime packs in a main-runtime artifact. The latest BDN660 source also explicitly rejects MonoCore/MonoWasm/MonoWasmAot targets >=12. The separate BDN work is evaluating the earlier implementation in #5279 and a staged-update path. I have not disabled any existing lanes or substituted a different measured runtime, and I am not treating passing Python argument tests as actual WASM execution evidence.
| <PropertyGroup Condition="'$(TargetFramework)' == 'net12.0' or $(TargetFrameworks.Contains('net12.0'))"> | ||
| <BenchmarkDotNetVersion>$(BenchmarkDotNetNet12Version)</BenchmarkDotNetVersion> |
There was a problem hiding this comment.
Agreed; the aggregate property can select the newer runner for the net11 inner build too. The requested separate global BDN update should remove this per-framework runner split altogether, including the version propagation/output-isolation problem, instead of adding more two-runner machinery here. Its owner has the exact extraction boundary and is evaluating compatibility with the historical Mono/WASM configurations. This remains unresolved until that prerequisite exists; the current mixed-target behavior is called out in the draft body.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
|
Follow-up on the two design questions, alongside 6dbc9d5/9c916affb:
The net12 WASM case is now after net11 and still passes the requested historical |
| - ${{ else }}: | ||
| - net12.0 |
There was a problem hiding this comment.
This is the same unresolved Mono-WASM/cohort issue tracked in #5344 (comment). Android/iOS defaults are fixed, but I have not claimed the main Mono lane is fixed merely by generating net11 arguments against an unverified SDK/runtime pack cohort. The separate prerequisite is now evaluating a staged BDN641 update based on #5279. Per the review direction, no existing lanes are disabled or silently substituted while that supported path is established.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
There was a problem hiding this comment.
🟡 Changes recommended
Critical net12/BDN649 compatibility gaps cause WASM argument parsing failures, prevent Mono AOT jobs from running, and make the R2R interpreter lane target net11 instead of net12.
6 open issues
Review effort: Balanced
| <BenchmarkDotNetVersion>0.16.0-nightly.20260801.595</BenchmarkDotNetVersion> | ||
| <BenchmarkDotNetVersion Condition="'$(BenchmarkDotNetVersion)' == ''">0.16.0-nightly.20260801.595</BenchmarkDotNetVersion> | ||
| <!-- Keep older measurements on their existing runner; net12 requires the new runtime model. --> | ||
| <BenchmarkDotNetNet12Version>0.16.0-nightly.20261014.649</BenchmarkDotNetNet12Version> |
There was a problem hiding this comment.
Agreed; this stays open with the separate global BDN prerequisite. Its first stage is now BDN641 plus the scoped native MonoAotLLVM compatibility implementation from #5279. The owner is porting the real runtime/generator/toolchain rather than renaming this lane to ordinary monoaot or dropping its compiler-mode/custom-pack inputs. I have not claimed the native Mono lane executes on the published runner as-is. The per-framework runner split and duplicate compatibility code here will be extracted once that prerequisite is established and validated.
| elif framework == "net12.0": | ||
| if args.wasm_runtime_flavor != 'CoreCLR': | ||
| raise ArgumentTypeError('Mono WebAssembly is not supported on net12.0; use net11.0 or --wasm-runtime-flavor CoreCLR') | ||
| run_args += ['--runtimes', 'corewasm12.0'] |
There was a problem hiding this comment.
Agreed that the current published runner alone cannot execute this argument shape; it is an explicit prerequisite, not a passing WASM result. Keeping the public CoreCLR flavor argument is user-directed. The separate global BDN work is adding a shared harness argument bridge, called before Program forwards to BDN, that validates and consumes the historical option while preserving its runtime-selection meaning. It must not silently ignore the flavor or change the older configurations. This thread stays open until that bridge and the supported workload path are validated; the draft body records the current parser/execution limitation.
There was a problem hiding this comment.
🔵 Needs a closer look
The newer runner rejects unhandled legacy arguments in both native Mono-AOT and net12 CoreCLR WASM jobs, causing argument-parsing failures before benchmarks can run.
5 open issues
1 resolved since last review
Review effort: Balanced
|
Local candidate integration is now checked against the exact global641 commit plus migration head The composition removes the migration's595/649 split, runner property/reference propagation and duplicate compatibility, taking the prerequisite's typed async sources, LINQ10.0.9 and AOT/LLVM compatibility unchanged. The SDK-only delta is22files. Its saved patch was replayed against the candidate in an isolated Git index and reproduced the validated tree Windows x64, private SDK12.0.100-alpha.1.26527.106;
Real Program listing checks also accepted historical CoreCLR flavor and Mono12 arguments, while explicitly rejecting conflicting flavor/runtime inputs. Listing returns1 even for the no-runtime-option control because the existing empty-summary exit policy applies; this is parser evidence, not WASM execution. The local SDK patch removes the earlier649-era Python Mono12 rejection rather than preemptively gate the lane; actual matching private Mono12 SDK/runtime/workload-pack execution remains unproven. Actual MonoAOT/LLVM, WASM, MAUI/device and R2R-interpreter workloads, and full paired/noise validation, remain outstanding. This is not a global SAFE verdict or merge approval. The separate BDN PR does not exist yet, so the published draft retains its explicit temporary overlap and unresolved prerequisite threads. After checking the composition I restored the published source and rebuilt its original net12 configuration successfully; the worktree is clean. The replayable SDK-only patch, exact validation manifest and logs are retained in the owning Copilot session. Root-fixed SDK12 follow-up: Locally recomposed against edf41b94, including its package-root alignment before Arcade imports. The same Python command above now passes 192 tests, including the two new package-root tests. The SDK-only patch is byte-identical across e1/edf and isolated-index replay reproduces the new composed tree An isolated net12 CoreRun publish-path diagnostic driven by the .NET CLI host then completed with exit 0 and 4 actual workloads. Restore used a new private package cache; launch used a private fallback containing only junctions to that cache's non-BDN packages, with 0 BenchmarkDotNet packages. Removing exactly the two owned TFM-less DLLs after the woven host build forced generated recompilation rather than reusing a stale woven binary. Raw PE metadata confirms This closes the local SDK12 custom-root diagnostic gap for the edf composition only. It is not actual Mono execution or a global SAFE verdict. Actual Mono/WASM/device/cohort and paired/noise limitations above remain. Published source/index are restored cleanly; rebuilding the original published net12/BDN649 configuration passed with 0 warnings/errors. No global patch was pushed into this draft, no lanes were removed or measured runtimes substituted, and publication of the SDK-only layer still awaits the separately owned BDN prerequisite. Rebased prerequisite follow-up (October 28): Fetched canonical main The same commands in the table above now pass 206 Python / 102 harness / 13 Reporting / 24 ResultsComparer tests, both net11/net12 builds with 0 warnings/errors, and 1 actual net12 Guid benchmark on BDN These are local SDK integration results, not actual Mono/WASM/device/cohort or paired/noise SAFE evidence. The separately owned campaign remains open. Published source/index are again restored cleanly at Enum compatibility follow-up: Recomposition onto eabc68fa applies the identical SDK-only patch without conflicts and reproduces tree The full net12 harness command above now passes 104 tests. Additional command WASM reference-gather follow-up: Exact 3c025ca8 composes without conflicts using the identical SDK-only patch; replay reproduces tree Published source/index are restored cleanly at |
Summary
Move desktop
mainmeasurements to SDK12 andnet12.0, rather than building net11 benchmarks with an installation containing only the .NET12 runtime.net12.0andnativeaot12.0.global.net11.json; older-onlybenchmarks_ci.pyruns temporarily select the newest requested framework's versioned pin and restore the original root file even on failure.--wasmRuntimeFlavor CoreCLRargument. No measurement lanes have been disabled or silently substituted.r2r11_0/r2r12_0), preserving its custom runtime pack, compiler path and disabled-runtime-code-generation setting.No MAUI ACES pool changes are included. Based on freshly fetched upstream main
bee760ce03eda6021a1466486e067c7c7e76a4ef. Review revisions:6dbc9d542,9c916affb,9c0b441ad.BDN prerequisite and review status
Still a draft, not a merge-ready all-framework migration. Following the user's incremental-update direction, the separate prerequisite is evaluating a global BDN641 update based on dotnet/performance#5279, with later BDN660 work kept separate. BDN641 already includes net12 CoreCLR support;649 is not a minimum requirement. Compatibility extraction and the required configuration-matched campaign remain pending. Keep this PR targeting
main; no replacement or retargeting is intended.The original implementation temporarily uses BDN649 for net12 because existing BDN595 restores/builds net12 but crashes before execution with
GetRuntimeVersion not implemented for NotRecognized. That split and compatibility code remain only until extraction into the separate global update is established: version selection/propagation, async-LINQ dependency, AOT filter and tests, and TLS/URI typed argument sources. The typed sources fix actual BDN1507 errors (object[]yielded whereSslProtocols/stringis taken), without changing argument values or measured methods.Open integration issues are not presented as fixed:
The R2R-interpreter target mismatch is fixed in
9c0b441ad: both OS paths use a framework-aware runtime moniker and retain the existing custom pack, Crossgen2 path and interpreter flag. The prerequisite owner verified candidate641's realR2RRuntime.Net12_0parser/toolchain support. Full R2R payload execution remains unvalidated, not implied by argument-generation tests.Historical framework-selection alternatives and the argument-source rationale are documented in the review follow-up. Addressed SDK-pin, SDK-metadata, library-condition and R2R-target threads are resolved; Mono/mixed-runner compatibility threads remain open.
Validation
Windows x64; pytest used this checkout's
scriptsdirectory onPYTHONPATH. The complete focused Python suite and serialized harness/Reporting tests were rerun on final review head9c0b441ad; other results below are recorded across the incremental review revisions. The real net12 rerun used$sdk = (Resolve-Path tools\dotnet\x64-net12-review).Path, an isolated private SDK12 installation subsequently restored to normaltools\dotnet\x64. The net11 auto-acquisition control used the normal installation directory with no private SDK12 present there. Temporary test/control files were cleaned up.python -m pytest scripts\tests\test_channel_map.py scripts\tests\test_dotnet.py scripts\tests\test_run_performance_job.py scripts\tests\test_blazor_workload.py scripts\tests\test_wasm_coreclr_r2r.py -q9c0b441ad(44.12s), including SDK-pin restoration on success/failure, mixed-SDK metadata selection, mobile defaults, historical CoreCLR flavor arguments, future library-pin isolation and framework-aware R2R argument selection.python -m pytest scripts\tests\test_channel_map.py -k 'microbenchmarks_select_intended_packages or next_library_pins' -qpython -m pytest scripts\tests\test_run_performance_job.py -qdotnet test src\tools\ResultsComparer.Tests\ResultsComparer.Tests.csproj --configuration Release --framework net12.0 --nologo --verbosity minimaldotnet test src\tests\harness\BenchmarkDotNet.Extensions.Tests\BenchmarkDotNet.Extensions.Tests.csproj --configuration Release --framework net12.0 -p:PERFLAB_TARGET_FRAMEWORKS=net12.0 --nologo --verbosity minimalartifacts\obj\Reporting\Release\netstandard2.0\Reporting.dll. The serialized command below passes; this failure is not counted as a passing default parallel build.dotnet test src\tests\harness\BenchmarkDotNet.Extensions.Tests\BenchmarkDotNet.Extensions.Tests.csproj --configuration Release --framework net12.0 -p:PERFLAB_TARGET_FRAMEWORKS=net12.0 -m:1 --nologo --verbosity minimaldotnet test src\tools\Reporting\Reporting.Tests\Reporting.Tests.csproj --configuration Release --framework net12.0 -m:1 --nologo --verbosity minimalpython scripts\benchmarks_ci.py --frameworks net11.0 --dotnet-versions 11.0.100-rc.2.26504.105 --build-only --filter '*Perf_Guid.NewGuid*'global.jsonSHA256 was unchanged afterward.python scripts\benchmarks_ci.py --frameworks net11.0 --build-only --filter '*Perf_Guid.NewGuid*'python scripts\benchmarks_ci.py --frameworks net12.0 --dotnet-path $sdk --filter '*Perf_Guid.NewGuid*' --bdn-arguments '--job Dry'git diff --checkThe successful real net12 rerun's installation contained only .NET12 runtimes, so it did not implicitly roll a net11 benchmark forward. This is execution evidence, not a performance comparison or a SAFE verdict on a global BDN update.
Remaining validation limits