Repository navigation
Document generic CUDA operators and complete build compatibility - #33465
Closed
Akshay Sonawane (apsonawane) wants to merge 4 commits into
Closed
Akshay Sonawane (apsonawane) wants to merge 4 commits into
Akshay Sonawane (apsonawane) wants to merge 4 commits into
Conversation
Part of the grouped generic CUDA operator stack. Preserve the qualified source snapshot.
This was referenced Oct 19, 2026
Contributor
Author
|
Closing this draft to replace the feature-based stack with focused branches rebuilt from current main. Existing branches are preserved; replacement PRs will be linked here. |
This was referenced Oct 19, 2026
Merged
Contributor
Author
Akshay Sonawane (apsonawane)
added a commit
that referenced
this pull request
Oct 24, 2026
## Summary - Add opt-in explicit FP32 arithmetic and packed input slicing for BranchwiseRMSNorm. - Add versioned RotaryEmbedding exact arithmetic while retaining existing default dispatch. - Keep RMS-chain regression coverage with this PR, independent of tensor fusions. ## Review follow-up - Honor deterministic CUDA sessions by using the existing fixed-order column reduction instead of the single-row floating-point atomic reduction. Default dispatch remains unchanged. The schema documents that deterministic summation may round differently from the default path or deterministic ReduceMean. - Add CUDA exact-mode branch-scale parity coverage for distinct `(C, H)` and flattened `(C * H)` scales over multiple outer rows, covering FP32, FP16, and BF16 input and scale storage. - Add a width-8192 single-row deterministic-session regression with a numerical reference, repeated bitwise comparison, and widened-output checks. - Prevent GroupQueryAttentionFusion from removing either RotaryEmbedding node when `use_fp32_arithmetic` is requested. Add a version-2 graph-transformer regression for query-only, key-only, both-exact, and ordinary fusion. - Regenerate the affected contrib and kernel documentation using the official renderers with source-derived metadata, including version-2 availability/history, optional widened outputs, type constraints, and CPU/CUDA/DML registrations. Full native `--gen_doc validate` remains a CI check. ## Clean main-based replacement Supersedes #33459. Reconstructed directly on `main` (`28225b8583474cf6a6e7f97222227ae668f80e8b`), without the old divergent feature-branch history. This PR targets `main` directly and has no dependency on another operator PR. Family documentation is bundled here; the separate old documentation PR #33465 is retired. ## Validation - Used the `qwen-flash` Conda environment. - Ran `lintrunner -a` before committing, followed by a passing check-only run on the full PR delta and `git diff --check`. - Per request, the review follow-up does **not** rebuild ORT/GenAI or execute the new native/GPU tests. Their execution and full generated-documentation validation are left to CI. - The original branch was previously checked with schema syntax checks and affected CUDA/host/test translation-unit compilation (SM90); this historical compile evidence does not validate the new review follow-up. - The previous head's React Native Android failure was an npm `ETIMEDOUT` downloading `supports-color`, not a source compilation failure. CUDA builds succeeded, but CUDA test jobs were canceled. The new push starts fresh CI; no claim that all pipelines are green. - Full main-based model/GPU requalification remains outstanding; keeping this PR in draft. - No benchmark/profiling/evaluation scripts or local task metadata are included.
Akshay Sonawane (apsonawane)
added a commit
that referenced
this pull request
Oct 28, 2026
## Summary - Support activation-type GDN gates and fused packed projection handling. - Extend causal convolution state/normalization handling with schema and provider validation. - Keep existing default modes and upstream regression coverage. ## Clean main-based replacement Supersedes #33461. Reconstructed directly on `main` (`28225b8583474cf6a6e7f97222227ae668f80e8b`), without the old divergent feature-branch history. This PR targets `main` directly and has no dependency on another operator PR. Family documentation is bundled here; the separate old documentation PR #33465 is retired. ## Review and pipeline fixes - Initialize `mid_state` for empty ragged requests from the incoming state, or zero when absent, in both warp-specialized and generic recurrent kernels. - Add regression coverage for `cu_seqlens={0,0,1}`, head sizes 128 and 32, and both initial-state cases. - Apply the same fused-decay reference and input setup for the `softplus` and `qwen` aliases. The existing first-row-state regression retains the `softplus` alias. - Replace constructor use of `Node::OutputDefs()` with `ComputeContext::OutputCount()` validation. The former API is absent in the WebGPU plugin adapter and caused the Linux static-plugin and Windows plugin compile failures. - Synchronize the operator documentation with the extended schemas and CPU/CUDA/DML registrations. These edits are source-derived; exact generated-document validation remains with CI. - The earlier React Native Android failure was an npm feed download `ETIMEDOUT`, not an operator compilation failure. ## Validation - Used the `qwen-flash` Conda environment. - Ran `lintrunner -a` before committing, followed by a passing check-only run over the full PR delta. - `git diff --check` passed. - Before the request to skip further builds, the changed GDN CUDA and C++ test translation units compiled successfully. An isolated GPU3 harness using the current CUDA kernel bodies passed empty-request state initialization for both warp/generic geometries, with and without initial state. - No full ONNX Runtime or GenAI rebuild was performed for these review fixes. The isolated harness is not an end-to-end operator test run. - No benchmark/profiling/evaluation scripts or local task metadata are included. ### Runtime qualification The earlier accumulated-build failure in `GatedDeltaNetTest.RecurrentOutputsIncludeFirstRowState` compared different recurrences because the reference did not recognize the `softplus` alias. The reference has been corrected. A fresh full native test-suite/model run has not been performed; keeping this PR in draft pending CI and runtime qualification.
Akshay Sonawane (apsonawane)
added a commit
that referenced
this pull request
Oct 28, 2026
## Summary - Extend DynamicSparseAttention cache and grouped CUDA paths. - Add FP32 key-state support for BF16 QSA, optional mid-state/status outputs and deterministic tiled selection. - Align optional QSA positions across schema, CUDA and WebGPU; explicitly reject CUDA-only modes on WebGPU. - Keep experimental radix/warp-head controls default-off and document their scope. ## Review follow-up and pipeline fixes - Allocate grouped floating-cache decode partial buffers even for a single split; only the generic split path can omit them. - Add FP16/BF16 grouped prefill/decode reference-parity cases with nonzero logits, partial query-head groups, empty/partial selection rows, auxiliary selections/local windows, and sink logits. Include the batch-128, four-head, width-256, max-selected-512 single-split decode regression. Inputs use exactly representable binary values so the reference sees the actual stored values. - Reject connected optional outputs 8-10 in CUDA IndexShare reuse instead of returning with unproduced outputs; document the restriction and test paired mid-state and validation outputs. - Reject CUDA-only committed_length and validation_status on WebGPU DynamicSparseAttention, with model-level rejection tests. - Count optional QSA position_ids only when supplied, retaining strict checks for required bindings. Existing no-position regressions cover the eight-binding case; add explicit-position tests for nine-binding success and eight-binding rejection. - Infer state_update activation type from past_kv_buffer, not FP32 past_key_state. Add BF16-activation/FP32-state capture coverage with raw keys and compression-block transitions. - Fix Windows C4244 warning-as-error failures in the test reference: preserve the int64 KV-head index and explicitly convert INT8 test data. - Regenerate the affected contrib/operator-kernel documentation using official renderers with source-derived CPU/CUDA/DML metadata, including current signatures, T_KEY, and optional-output descriptions. ## Clean main-based replacement Supersedes #33462. Reconstructed directly on `main` (`28225b8583474cf6a6e7f97222227ae668f80e8b`), without the old divergent feature-branch history. This PR targets `main` directly and has no dependency on another operator PR. Family documentation is bundled here; the separate old documentation PR #33465 is retired. ## Validation - Used the `qwen-flash` Conda environment. - Ran `lintrunner -a` before committing, followed by a passing check-only run on the full PR delta, C++/CUDA formatting checks, and source whitespace checks. - Verified affected generated sections/rows against official renderer output using source-derived metadata. Exact renderer-generated paragraph/EOF whitespace is retained; full native `--gen_doc validate` remains a CI check. - Per request, no ORT/GenAI rebuild or new native/GPU/WebGPU test execution was performed for this follow-up. Execution is left to CI. - Original-branch schema syntax and affected CUDA/host/test translation-unit compilation (SM90) are historical evidence, not validation of this follow-up. - Inspected Windows CUDA/plugin/TensorRT logs identify the C4244 failures fixed above; macOS WebGPU reports eight QSA failures from the missing optional-position binding. React Native Android npm ETIMEDOUT and CodeQL Java Gradle-download timeout are infrastructure failures, not source errors. - The new push starts fresh CI; all pipelines are not yet claimed green. Full main-based model/GPU requalification remains outstanding; keeping this PR in draft. - No benchmark/profiling/evaluation scripts or local task metadata are included. ### Runtime qualification notes Earlier accumulated-build testing excluded malformed-state expectations `QsaMalformedStateLengthsAreClamped` and `CsaMalformedStateLengthsAreClamped`; their upstream-baseline status is not established. Those tests are not disabled by this follow-up. The current WebGPU QSA log fails before those numerical assertions due to the optional-position binding bug fixed here. The output-count regression expectation is updated for the intentional optional-output extension to 11 slots. WebGPU changes have not been built or executed locally.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Main integration update
Merged repository
mainat9a662b9395c7c5dc619ae4d050f2c9362d88eae3. Published head:e15aebba7f976b58f4826a06461ad0f6d103c5c8. Existing PR bases remain unchanged.qwen-flashconda environment; ranlintrunner -abefore each new commit, followed by clean lint checks. Final branch-vs-main lint passes.This update supersedes earlier notes saying main has not been integrated, but does not remove the draft/build/runtime qualification requirements.
Summary
Update contrib operator/kernel documentation for the stack and include the remaining compatibility/formatting adjustments.
Stack and main target
Depends on #33464.
This is layer 7/7 of the microsoft/onnxruntime stack. The current review base is
asonawane/ported-ops-mtp-fold-stateso this PR shows only its incremental group. The final destination is upstreammain: land/reconcile prerequisites first, then rebase and retarget this layer tomain. Keep this PR in draft until that base-specific validation succeeds. Do not merge the entire stack through an unvalidated intermediate base.Cross-repository dependency: GenAI package/runtime integration requires the companion ORT contrib operators.
Validation and blockers
GatedDeltaNetTest.RecurrentOutputsIncludeFirstRowStatefails in the existing combined workspace.RejectsWrongOutputCount,QsaMalformedStateLengthsAreClamped,CsaMalformedStateLengthsAreClamped.Scope
Stack navigation
Companion repository stack: