Skip to content

Add schema compatibility negotiation for plugin EPs - #32230

Draft
Tianlei Wu (tianleiwu) wants to merge 24 commits into
mainfrom
tlwu/ms_ops_versioning
Draft

Add schema compatibility negotiation for plugin EPs#32230
Tianlei Wu (tianleiwu) wants to merge 24 commits into
mainfrom
tlwu/ms_ops_versioning

Conversation

@tianleiwu

@tianleiwu Tianlei Wu (tianleiwu) commented Aug 24, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Description

Add operator-schema compatibility negotiation between ONNX Runtime core and separately shipped plugin EPs.

  • Add an API 30 callback through which a plugin publishes the exact operator contracts it was built against.
  • Compute deterministic SHA-256 schema ABI digests in core and maintain a checked-in com.microsoft schema manifest for inference plugin builds.
  • Negotiate each contributed operator independently and quarantine incompatible plugin kernels and capability groups before partitioning.
  • Make GQA fusion honor the negotiated kernel contract, so core does not produce a fused node that the loaded plugin cannot execute safely.
  • Keep standard ONNX-domain compatibility governed by ONNX opset versions; add the com.microsoft opset locking/versioning policy for contributed operators.
  • Protect ORT-owned domains from external custom-schema replacement and publish the manifest from the CUDA plugin EP.
  • Add a synthetic v1/v2 compatibility matrix covering legacy core/plugin behavior, matching and mismatched schemas, missing entries, duplicate entries, and fusion gating.

No C++ API wrapper is added for the new callback because it is an implementation hook populated by an EP factory and consumed internally by ORT; applications do not call it or manage its returned data.

Motivation and Context

ONNX Runtime core and a plugin EP can be built from different revisions. A contributed operator such as GroupQueryAttention may gain an input, output, attribute, or supported type without changing its historical schema version. In that situation, core could previously fuse or validate a node using one contract and dispatch it to a plugin kernel compiled against another contract, potentially causing incorrect execution or a crash.

This change makes compatibility explicit at the individual-operator level. New plugins publish schema identities and digests; core compares them with its live registry and removes incompatible contributed kernels before graph partitioning. Legacy plugins without the new callback retain the documented transition behavior. ONNX-domain operators continue to use their standard opset mechanism.

Testing

  • After merging the latest main, built onnxruntime_test_all and onnxruntime_autoep_test from a fresh CPU configuration.
  • SchemaAbiDigestTest.*, ORT-owned-domain rejection, and the affected GQA fusion gates: 7 tests passed in the fresh CPU build.
  • PluginEpSchemaCompatibilityTest.*: 8 tests passed in the fresh CPU build, including the v1/v2 and open-ended exact-start matrix.
  • PluginEpSchemaCompatibilityTest.*: 8 tests passed in an enabled CUDA build.
  • PluginEpSchemaCompatibilityTest.*: 3 applicable tests passed in a fresh --disable_contrib_ops CPU build.
  • Built the schema ABI digest test with default, DISABLE_CONTRIB_OPS, ENABLE_TRAINING_OPS, and ORT_USE_NCCL configurations.
  • Compiled the affected shape-inference and GQA optimizer tests in the enabled build.
  • Ran lintrunner on all changed files; no lint issues.

Copilot AI balanced review requested due to automatic review settings August 24, 2026 06:19
Copilot AI reviewed Aug 24, 2026
View reviewed changes

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds per-operator schema compatibility negotiation between ONNX Runtime and independently shipped plugin execution providers.

Changes:

  • Adds API 30 schema manifests and deterministic SHA-256 contract digests.
  • Quarantines incompatible plugin kernels/capabilities and gates GQA fusions.
  • Protects ORT-owned domains and adds compatibility-matrix coverage.

Reviewed changes

Copilot reviewed 40 out of 40 changed files in this pull request and generated no comments.

Show a summary per file
File Description
plugin-ep-cuda/MIN_ONNXRUNTIME_VERSION Raises the CUDA plugin runtime floor.
onnxruntime/test/optimizer/group_query_attention_pre_norm_fusion_test.cc Tests pre-norm fusion negotiation.
onnxruntime/test/optimizer/graph_transform_test_layernorm.cc Tests GQA fusion quarantine.
onnxruntime/test/ir/schema_abi_digest_test.cc Tests digests and manifest consistency.
onnxruntime/test/framework/shape_inference_test.cc Tests protected-domain rejection.
onnxruntime/test/framework/ep_plugin_provider_test.cc Updates plugin provider construction.
onnxruntime/test/autoep/test_ep_schema_compatibility.cc Adds compatibility matrix tests.
onnxruntime/core/session/provider_bridge_ort.cc Blocks external ORT-domain schemas.
onnxruntime/core/session/plugin_ep/ep_schema_compatibility.h Declares negotiation results.
onnxruntime/core/session/plugin_ep/ep_schema_compatibility.cc Implements schema negotiation.
onnxruntime/core/session/plugin_ep/ep_plugin_provider_interfaces.h Stores negotiated compatibility.
onnxruntime/core/session/plugin_ep/ep_plugin_provider_interfaces.cc Filters incompatible capabilities.
onnxruntime/core/session/plugin_ep/ep_kernel_registration.h Extends registry-import API.
onnxruntime/core/session/plugin_ep/ep_kernel_registration.cc Filters incompatible kernel ranges.
onnxruntime/core/session/inference_session.cc Supplies registries to optimizers.
onnxruntime/core/session/environment.cc Centralizes Microsoft opset registration.
onnxruntime/core/session/custom_ops.cc Rejects protected custom domains.
onnxruntime/core/providers/vitisai/vitisai_provider_factory.cc Zero-initializes the factory ABI.
onnxruntime/core/providers/qnn/qnn_provider_factory.cc Zero-initializes the factory ABI.
onnxruntime/core/providers/openvino/ov_factory.cc Zero-initializes the factory ABI.
onnxruntime/core/providers/nv_tensorrt_rtx/nv_provider_factory.cc Zero-initializes the factory ABI.
onnxruntime/core/providers/migraphx/migraphx_provider_factory.cc Zero-initializes the factory ABI.
onnxruntime/core/providers/cuda/plugin/cuda_ep_factory.h Declares manifest publication.
onnxruntime/core/providers/cuda/plugin/cuda_ep_factory.cc Publishes CUDA schema contracts.
onnxruntime/core/providers/cuda/cuda_provider_factory.cc Initializes the API 30 factory.
onnxruntime/core/optimizer/group_query_attention_pre_norm_fusion.h Adds a kernel-support checker.
onnxruntime/core/optimizer/group_query_attention_pre_norm_fusion.cc Gates pre-norm fusion.
onnxruntime/core/optimizer/group_query_attention_fusion.h Adds a kernel-support checker.
onnxruntime/core/optimizer/group_query_attention_fusion.cc Gates GQA fusion.
onnxruntime/core/optimizer/graph_transformer_utils.cc Wires effective kernel lookup.
onnxruntime/core/graph/schema_abi_digest.h Declares schema digest computation.
onnxruntime/core/graph/schema_abi_digest.cc Implements canonical SHA-256 digests.
onnxruntime/core/graph/contrib_ops/ms_schema_abi_manifest.inc Contains generated schema digests.
onnxruntime/core/graph/contrib_ops/ms_schema_abi_manifest.h Exposes the manifest array.
onnxruntime/core/common/sha256.h Provides the runtime SHA-256 alias.
include/onnxruntime/core/session/onnxruntime_ep_c_api.h Adds the API 30 callback and descriptor.
include/onnxruntime/core/optimizer/graph_transformer.h Defines the reusable support checker.
include/onnxruntime/core/optimizer/graph_transformer_utils.h Extends transformer generation.
include/onnxruntime/core/graph/constants.h Adds opset policy and domain ownership.
docs/plugin_ep/schema_compatibility_design.md Documents design, rollout, and testing.

💡 Configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

edgchen1
Edward Chen (edgchen1) reviewed Sep 3, 2026
View reviewed changes
Comment thread docs/plugin_ep/schema_compatibility_design.md Resolved
Comment thread docs/plugin_ep/schema_compatibility_design.md Resolved
@tianleiwu
Regenerate com.microsoft schema ABI manifest and add generator
faa6880 
Merging main added EngramGate, GatedDeltaNet, NGramHashMapping and
VarlenCausalConvWithState, and changed the CausalConvWithState and
PagedAttention schemas, which made the checked-in manifest stale and
failed SchemaAbiDigestTest.MSManifestMatchesSchemas plus
PluginEpSchemaCompatibilityTest.AcceptsPublishedMSDomainManifest in
every CI leg that runs them.

Add a disabled generator test that emits the manifest body, point the
assertion messages at it, and regenerate the checked-in .inc.

Also address design review feedback: new contrib operators are
introduced at the currently open com.microsoft opset, consistent with
ONNX and other domains, and document why the per-operator digest is
needed in addition to since_version.
@tianleiwu
Tianlei Wu (tianleiwu) marked this pull request as draft October 1, 2026 01:09
Tianlei Wu (tianleiwu) and others added 5 commits October 14, 2026 22:38
@tianleiwu
Merge origin/main, regenerate ms_schema_abi_manifest.inc, and address…
a6aa01b 
… review feedback

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@tianleiwu
Merge origin/main into tlwu/ms_ops_versioning
a9d21be 
Resolve conflicts with main:
- onnxruntime_ep_c_api.h: append GetOperatorCompatibilityInfo after main's GetDeviceMemoryCapacity and merge the optional-callback and zero-initialization documentation.
- graph_transformer_utils.cc: re-apply the kernel-support filtering on main's ACL-free EP sets and cover the new SwiGluFusion.
- group_query_attention_pre_norm_fusion.cc: keep main's separate q/k epsilon handling plus the kernel support check.
- cuda_provider_factory.cc: take main's identical OrtEpFactory zero-initialization.
- plugin cuda_ep_factory.cc: keep both the memory-capacity and operator-compatibility callbacks.
- nv_provider_factory.cc: accept main's removal of the legacy provider-bridge TensorRT RTX EP.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@tianleiwu
fix(optimizer): gate MatmulTransposeFusion on negotiated FusedMatMul …
66b0a56 
…kernels

MatmulTransposeFusion emits com.microsoft FusedMatMul for CPU, CUDA and DML, but it still used a provider-name allowlist. A plugin EP whose FusedMatMul kernel was quarantined by schema negotiation could therefore be given a node it cannot execute. Filter its allowlist like the other EP-specific contrib fusions.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@tianleiwu
build(graph): exclude the schema ABI digest from minimal builds
e7817ed 
Minimal builds do not carry the ONNX schema registry and onnx_protobuf.h deliberately skips onnx/defs/schema.h there. The digest is only consumed by plugin EP schema negotiation, which minimal builds do not compile, so drop it next to the other schema-registry sources.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@tianleiwu
fix(plugin-ep): gate schema negotiation on API 32 and raise the CUDA …
8a4a799 
…plugin floor

ORT 1.30 and 1.31 shipped without OrtEpFactory::GetOperatorCompatibilityInfo, and the field now follows GetDeviceMemoryCapacity (API 32). A factory built against an older header has a shorter structure, so reading the field when ort_version_supported is 30 or 31 would run past its end. Gate the callback on API 32, document it as \since 1.32, raise plugin-ep-cuda/MIN_ONNXRUNTIME_VERSION to 1.32.0, and cover the boundary with a factory whose callback pointer is non-null but older than the gate.

Also update the design document to the API 32 numbering and to describe the optimizer allowlist filtering that is already implemented.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Tianlei Wu (tianleiwu) and others added 5 commits November 10, 2026 05:52
@tianleiwu
chore(plugin-ep): regenerate the com.microsoft schema ABI manifest fo…
6ac0002 
…r main

Merging main changed six com.microsoft v1 schemas in place (CausalConvWithState, DynamicSparseAttention, GatedAdd, GatedDeltaNet, GroupQueryAttention, PackedSparseAttentionIndexer) and added thirteen operators (ConvStateReplay, GatedDeltaNetStateReplay, GatherPacked, HyperConnectionLowRankMix, MatMulNBitsQLoRA, MatMulPacked, MoEGroupRouter, MoEPacked, PackedSparseAttentionIndexerMerge, RMSNormRotary, RouterGateUpProj, SpeculativePrefixCommit, SwiGLU). SchemaAbiDigestTest.MSManifestMatchesSchemas reported all of them; regenerate the manifest with SchemaAbiDigestTest.DISABLED_GenerateMSManifest. The digests of the other 130 entries are unchanged.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@tianleiwu
docs(cuda-plugin): document the 1.32.0 runtime floor
365be93 
plugin-ep-cuda/MIN_ONNXRUNTIME_VERSION is now 1.32.0, the first core that negotiates com.microsoft operator schemas with plugins. Update the two places that state the current floor and separate the API-usage floor (1.24.4) from the declared floor.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@tianleiwu
Merge origin/main into tlwu/ms_ops_versioning
22f2236 
Merge main at d65da43 (#33954), which adds attributes to the MoE and QMoE schemas and is part of the pull request merge ref that CI validates.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@tianleiwu
chore(plugin-ep): regenerate the schema ABI manifest for MoE clipping
41a65c1 
main commit d65da43 (#33954) adds post-SiLU clipping attributes to the MoE and QMoE schemas in place. The pull request merge ref that CI builds contains it, so SchemaAbiDigestTest.MSManifestMatchesSchemas and PluginEpSchemaCompatibilityTest.AcceptsPublishedMSDomainManifest failed on every platform for MoE@1 and QMoE@1. Regenerate the manifest with SchemaAbiDigestTest.DISABLED_GenerateMSManifest; only those two digests change.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@tianleiwu
test(plugin-ep): compare manifest digests only in full-type builds
00acb86 
Every Android build, and any build that passes --disable_types float8 or float4, defines DISABLE_FLOAT8_TYPES and DISABLE_FLOAT4_TYPES. contrib_defs.cc then registers narrower type constraints for GemmFloat8 and GatherBlockQuantized, so their digests differ from the full-type manifest and SchemaAbiDigestTest.MSManifestMatchesSchemas failed on Android CI. Compare digests only in full-type builds, refuse to generate the manifest in reduced-type builds, and let AcceptsPublishedMSDomainManifest expect quarantine exactly for the entries whose digests differ there.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@skottmckay Scott McKay (skottmckay) skottmckay left review comments
@edgchen1 Edward Chen (edgchen1) edgchen1 left review comments
@kunal-vaishnavi kunal-vaishnavi kunal-vaishnavi left review comments
Copilot code review Copilot
Copilot review effort
Applies to this pull request for everyone.Learn more about Copilot code review.
Copilot left review comments

At least 1 approving review is required to merge this pull request.

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@tianleiwu @skottmckay @edgchen1 @kunal-vaishnavi