You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
feat: add unified AI prompting and native decision model support - #2811
Users currently need different Spark stages and provider-specific handling for text, embeddings and native decision models. This adds AIPrompt, built on the existing OpenAIPrompt interface, so callers can select the API explicitly, pass supported request options and inspect typed results, errors and complete response JSON without adding another inference transport.
Related Issues/PRs
Implements the native-inference portion of #2759. The umbrella remains open: recoverable Delta results are tracked separately in #2800 and the unqualified draft #2813.
The five original prerequisite PRs are merged: #2806, #2807, #2808, #2809 and #2810. Later review fixes are preserved in separate draft follow-ups: #2818 (nested assigned-session persistence), #2819 (Python method bindings and alias identity), and #2820 (wrapped attachment cancellation).
Merge order: qualify and merge those three independent follow-ups in any order, then integrate them into this PR and complete final-head validation before merging it. Optional recovery in #2813 comes last and remains blocked on managed cross-application restart qualification. No PR in this remaining set is being presented as merge-ready. The local rebase onto merged master is complete; its reviewed changes have not yet been pushed to this PR.
Live OpenAIDecisions qualification remains #2760. It is not supported in Azure OpenAI/Foundry; the notebook leaves its examples commented out.
What changes are proposed in this pull request?
Add AIPrompt for explicit Chat Completions, Responses, embeddings and native API selection, plus the small AISystemOne and OpenAIDecisions specializations.
Reuse existing request/authentication infrastructure while preserving the published legacy classes and parameter shapes. Native options are forwarded or explicitly rejected, not silently discarded.
Support scalar and column-driven JSON body extensions, correct output schemas, lossless native observations, usage and distinct failure diagnostics.
Preserve native parameter/default behavior through generated Python, copy, save/load and pipelines.
Add AIPrompt.ipynb, following the existing OpenAI.ipynb tutorial style, and a focused usage guide. All 32 supported examples executed on the packaged artifacts; Decisions examples remain comments.
This PR does not add Delta recovery, response-size configuration, old-reader/new-class interoperability, a tool-execution loop or an exactly-once inference guarantee. Default Spark execution remains lazy and may replay model calls.
How is this patch tested?
I have written tests (not required for typo or doc fix) and confirmed the proposed feature/bug-fix/change works.
Current rebase validation (November 9): local head c3ab8d96582e0f690885f9c78c660ea982fc4fea with pending tree f4f88f80beacff35a6c84d502c0c74a64ac9d707 integrates master 6c2699defcfb24d5bff5ba55f2252ba2d8cbea24. sbt "set core / Test / fork := false" core/compile core/Test/compile "core/testOnly com.microsoft.azure.synapse.ml.core.serialize.ValidateComplexParamSerializer com.microsoft.azure.synapse.ml.stages.VerifyUDFTransformerPersistence com.microsoft.azure.synapse.ml.io.split1.VerifyResponseBodyInspection com.microsoft.azure.synapse.ml.io.split1.VerifyCancellableRequests com.microsoft.azure.synapse.ml.io.split1.VerifyBoundedRequests com.microsoft.azure.synapse.ml.param.VerifyGlobalParams com.microsoft.azure.synapse.ml.codegen.PyCodegenSuite" core/scalastyle core/Test/scalastylepassed: 183 tests, zero skips, both compilation and style gates. python -B -m pytest tools\ci\tests\test_aiprompt_notebook.py -qpassed: 6 tests, zero skips. python -m black --check core\src\main\python\synapse\ml\core\schema\Utils.py cognitive\src\test\python\synapsemltest\services\test_AIPrompt.py cognitive\src\test\python\synapsemltest\services\test_AIPromptLive.py cognitive\src\test\python\synapsemltest\services\aiprompt_fabric_e2e.py tools\ci\tests\test_aiprompt_notebook.pypassed with Black 22.3.0. Rebased cognitive integration and installed-package validation have now passed on that same source tree. Complete content reviews, publication and final-head CI remain pending.
Passed: current-source generated wrappers, production JARs and Compile classpath export.
Installed-artifact harness, calling unittest discovery for cognitive/src/test/python/synapsemltest/services/test_AIPrompt.py
Passed: all 33 offline native/legacy Python cases, zero skips, on Python 3.11.8. Both wheels were built from this source and installed without dependencies. Exact loaded JAR paths and all 452 installed Python source/stub files matched these artifacts. This is a task-local harness, not a claimed standalone repository command; its exact invocation and artifact hashes are retained with the local validation evidence.
The complete current patch is frozen into 24 full-file review slices with roundtrip coverage of all 86 changed files. That preparation is not a completed review or a green CI result.
Historical evidence below: these results retain their original revision scope; they do not qualify the new rebased head automatically.
Validated the focused source against master at d3f3af6bd3e4b7d191f51c9eb29f9aeffe788193, using Spark 3.5.0, Scala 2.12.17, JDK 11 and Python 3.11.8. The local SBT launcher redirects build output and caches outside the checkout; the task arguments are listed below.
Passed, 1 test. Codecov now expects 55 uploads for the additional AI test lane; the assertion is unchanged.
npm run build in website
Passed after regenerating docs. Three unchanged generated MDX files required local LF normalization on Windows; no unrelated source changes were included.
Focused transport/legacy and core persistence/codegen/notebook-selection runs passed 49 and 61 tests respectively. The native run initially passed 442 of 444 tests; two extraction-related test defects were corrected and the affected suites passed 18/18. The live-service and benchmark tests were opt-in cancellations in that offline run, not counted as passes; the benchmark and live notebook were subsequently run explicitly.
Installed-wheel tests exposed a native Python loader regression during the split. After repairing the native reader, its six persistence/loader cases passed; 29 legacy parameter/response-schema cases also passed. Earlier combined-tree evidence is not substituted for these focused-tree results.
After the 32-test stub-extractor revision, compilation, both core style gates, code generation and JAR packaging passed again. Subsequent string-state repairs add eight regressions for skipped function/class bodies and ordinary class/control-block statements. Compilation, all 40 generator tests, both core style gates and sbt core/codegen cognitive/codegen passed on the latest integrated revision. All 451 final generated runtime/stub Python files match the notebook-tested wheels after normalizing only generated UIDs, exact build-version literals and line endings; none were added or removed.
Local Spark 4.0.1 and 4.1.1 compatibility replays also passed compilation, all four style gates and code generation after port-specific adaptations for Scala 2.13 ordered-map serialization, Spark column APIs and the ports' existing Python-wrapper conventions. These adaptations are preserved separately for port integration, not added to this master PR.
python .validation\aiprompt-core-port-compat99\python-smoke.py --port 40 and python .validation\aiprompt-core-port-compat99\python-smoke.py --port 41 using the local installed-artifact harness
6 passed per port, zero failures/skips, with matching PySpark 4.0.1/4.1.1, Python 3.11.8 and JDK 17.
The port JVM evidence totals 449 passed and one intentionally disabled benchmark per port, combining the full run with targeted reruns of repaired suites—not a fresh full sweep. Installed-wheel smoke covers public imports/construction, scalar/column JSON bindings, copy/save/load, loader identity/rejection and loopback AIPrompt transformation. No live endpoint calls were made by these port runs. This bounded local evidence does not substitute for full port CI on its pinned Python versions or managed-runtime qualification.
The final notebook run passed all 18 code cells covering 32 examples: chat, Responses, structured output, postprocessing, image/file attachments, stored-response chaining, embeddings, direct/facade SystemOne, per-row errors, parameters/copy, save/load, pipelines and pandas conversion. It used the newly packaged JARs and installed wrappers with real Azure OpenAI and Foundry Decision-1 calls. OpenAIDecisions was not executed. Notebook outputs and credentials are not committed.
The controlled 4,096-row loopback benchmark passed with all expected requests. At 4 partitions/concurrency 1, legacy/native elapsed times were 29.258s/7.885s; at 16 partitions/concurrency 4 they were 22.203s/2.881s. This is a fixed 5ms-provider-latency transport comparison, not a real-service speed guarantee.
Still required before merge: current-head automated review, Azure Pipelines/GitHub checks and integration of any prerequisite follow-ups. Historical monolithic runs are not used to qualify this split. Full current-head CI and managed-runtime qualification are not claimed complete.
Does this PR change any dependencies?
No. You can skip this section.
Yes. Make sure the dependencies are resolved correctly, and list changes here.
Does this PR add a new feature? If so, have you added samples on website?
No. You can skip this section.
Yes. Make sure you have added samples following below steps.
Find the corresponding markdown file for your new feature in website/docs/documentation folder.
Make sure you choose the correct class estimators/transformers and namespace.
Follow the pattern in markdown file and add another section for your new API, including pyspark, scala (and .NET potentially) samples.
Make sure the DocTable points to correct API link.
Navigate to website folder, and run npm start to make sure the website renders correctly.
Don't forget to add <!--pytest-codeblocks:cont--> before each python code blocks to enable auto-tests for python samples.
Make sure the WebsiteSamplesTests job pass in the pipeline.
## Summary
Protect global parameter registration, value updates, lookup and reset with one
shared monitor. Add bounded child-JVM regressions for concurrent stage use,
including the existing present-null and reset semantics.
## Prompting Intent
Keep the AIPrompt feature focused and publish independently demonstrated
pre-existing defects as separate fixes. Reproduce the concurrency failure on
baseline without allowing a corrupted map to hang the test runner.
## Linked Sources
- Bug: microsoft#2804
- Scope: microsoft#2759
- Deferred recovery: microsoft#2800
## Rationale
A shared monitor makes the two-map lookup consistent and preserves existing
signatures and Scala Map null/Option behavior. Replacing only one map or using
a null-rejecting concurrent collection would not preserve that contract.
Baseline concurrency probes failed; the fixed probes and existing global
parameter tests passed.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 5ba39157-c8dd-4adb-90e0-ff963a1e6733
## Summary
Pass the MLReader or MLWriter session into complex parameter serialization
and deserialization. Retain the existing helper overloads and stored format.
Cover writer selection and both directions of reader deserialization policy.
## Prompting Intent
Separate unrelated persistence repairs from the native AIPrompt feature.
Prove that a caller-selected Spark session governs complex payload handling
rather than inheriting an unrelated active session's configuration.
## Linked Sources
- Bug: microsoft#2803
- Scope: microsoft#2759
## Rationale
Threading the already assigned session through overloads fixes the actual
boundary without changing global Spark state, serialized parameter shapes or
unsafe-deserialization defaults. The three regressions failed on baseline and
passed with the existing serialization and UDF persistence suites.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 5ba39157-c8dd-4adb-90e0-ff963a1e6733
## Summary
Resolve the two generator review findings by sharing scope-aware method
extraction between stub generation and ServiceParam alias validation.
Preserve binding decorators across comments, blank lines and stacked decorators.
## Prompting Intent
Keep AIPrompt focused, isolate independent defects into their own PRs, and
resolve review findings with regression evidence rather than unrelated changes.
## Linked Sources
- microsoft#2805
- microsoft#2810
- microsoft#2810 (comment)
- microsoft#2810 (comment)
- microsoft#2759
## Rationale
Keep the existing synchronous single-line method-signature grammar while
tracking enclosing function/class scopes and complete decorator blocks.
Extract the helper and fixture object to respect the repository's file-length
limit without changing fixture JVM names or adding Python parser dependencies.
Three regressions fail on the preceding PR implementation, and two multiline
header regressions fail before the scope-continuation repair. All 26 codegen
tests and both Scala style gates pass in the isolated and integration trees.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 5ba39157-c8dd-4adb-90e0-ff963a1e6733
## Summary
Preserve actual thread and Spark-task cancellation through attachment reads
and prompt message construction. Keep ordinary timeout and IO failures as
sanitized row diagnostics. Cover both legacy generative APIs through public
transform actions and retain existing request and persistence cases.
## Prompting Intent
Repair the independently reproduced legacy cancellation defect in its own PR,
without bundling native AIPrompt adapters, bounded attachment features or
optional recovery machinery.
## Linked Sources
- Bug: microsoft#2801
- Scope: microsoft#2759
## Rationale
Cancellation is control flow and must unwind inference rather than become a
normal content-row error. Recognize actual task/thread interruption while
preserving sanitized behavior for unrelated failures. All three new regression
cases failed on baseline; the complete 25-test legacy multimodal suite passed
with the fix.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 5ba39157-c8dd-4adb-90e0-ff963a1e6733
## Summary
Close responses in a finally block when converting HTTP response data.
Add a regression that propagates the conversion failure while checking that
the response is closed exactly once.
## Prompting Intent
Publish the independent response-resource cleanup defect separately from
AIPrompt's new opt-in transport behavior. Keep the fix narrow and prove the
baseline failure directly.
## Linked Sources
- Bug: microsoft#2802
- Scope: microsoft#2759
## Rationale
The existing success-only close skipped cleanup when body conversion threw.
A try/finally fixes that path without changing retry accounting, adding response
limits or depending on the new native buffering implementation. The regression
failed on baseline and passed with the existing response inspection tests.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 5ba39157-c8dd-4adb-90e0-ff963a1e6733
## Summary
Add AIPrompt on the existing OpenAIPrompt interface for explicit chat,
Responses, embeddings and native decision-model APIs. Preserve request
options, typed results, errors and raw responses through Scala and generated
Python, copy, persistence and pipelines. Add the AIPrompt notebook and guide.
## Prompting Intent
Keep the contribution focused on AIPrompt and small native specializations.
Separate independent legacy defects and optional Delta recovery into their
own issues and PRs. Exercise supported notebook examples on packaged
artifacts, but leave OpenAIDecisions examples commented because Azure
OpenAI/Foundry does not support that API.
## Linked Sources
- Native inference proposal: microsoft#2759
- Deferred recovery: microsoft#2800
- Deferred live Decisions qualification: microsoft#2760
- Global parameter prerequisite: microsoft#2806
- Attachment cancellation prerequisite: microsoft#2807
- Response cleanup prerequisite: microsoft#2808
- Assigned-session prerequisite: microsoft#2809
- Python stub prerequisite: microsoft#2810
- Existing tutorial: docs/Explore Algorithms/OpenAI/OpenAI.ipynb
## Rationale
Reuse existing authentication and HTTP infrastructure rather than add a
second inference transport. Keep public behavior in Scala and generate
Python conveniences without changing legacy public JVM signatures or
serialized parameter shapes. Explicit API selection and lossless JSON avoid
silent option loss; bounded cancellation handles interrupted Spark work.
Keep ordinary inference lazy and replayable, without Delta, an automatic
tool loop, response-size configuration or an exactly-once claim.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 5ba39157-c8dd-4adb-90e0-ff963a1e6733
Hey Rana Singh (@ranadeepsingh) 👋!
Thank you so much for contributing to our repository 🙌.
Someone from SynapseML Team will be reviewing this pull request soon.
We use semantic commit messages to streamline the release process.
Before your pull request can be merged, you should make sure your first commit and PR title start with a semantic prefix.
This helps us to create release messages and credit you for your hard work!
Examples of commit messages with semantic prefixes:
fix: Fix LightGBM crashes with empty partitions
feat: Make HTTP on Spark back-offs configurable
docs: Update Spark Serving usage
build: Add codecov support
perf: improve LightGBM memory usage
refactor: make python code generation rely on classes
style: Remove nulls from CNTKModel
test: Add test coverage for CNTKModel
To test your commit locally, please follow our guild on building from source.
Check out the developer guide for additional guidance on testing your change.
The reason will be displayed to describe this comment to others. Learn more.
🔵 Needs a closer look
Spark-internal APIs and cross-runtime persistence contracts require maintainer validation, with final-head runtime qualification, live-provider testing, and prerequisite integration still pending.
No open issues 🎉
What changed in this PR
This PR introduces unified AI prompting on SynapseML’s existing OpenAI infrastructure, covering text generation, embeddings, and native decision models. It addresses the native-inference portion of #2759; Delta recovery remains separate in #2800.
Changes:
Add native API parameters, System One and Decisions specializations, request validation, and schema-preserving output support.
Extend shared transport, persistence, and Python generation infrastructure for cancellation, explicit defaults, and native JSON values.
Add regression tests, opt-in live and managed-runtime checks, benchmark coverage, and documentation navigation.
Summary
- Preserve quote, escape, delimiter and continuation state across multiline
headers and decorators in the shared Python method extractor.
- Recognize binding decorators with trailing comments and retain their binding
across nonbinding multiline decorators.
- Cover six additional regressions with Python syntax-checked fixtures.
Prompting Intent
- Keep the AIPrompt contribution focused and repair independent generator
regressions in their own prerequisite pull request.
Linked Sources
- microsoft#2805
- microsoft#2810
- microsoft#2759
Rationale
- Punctuation inside strings and comments must not change method scope or
accidentally discard a pending static/class binding.
- All six new regressions fail on the preceding extractor. All 32 generator
tests, compilation and both style gates pass with this repair.
- Only the private extractor and its existing fixture/suite files change.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 5ba39157-c8dd-4adb-90e0-ff963a1e6733
Summary
- Apply the pinned Black formatter to the AIPrompt regression suite and its
Fabric validation helper.
Prompting Intent
- Keep the focused AIPrompt pull request limited to required feature work
and repairs to its own validation gates.
Linked Sources
- microsoft#2811
- https://github.com/microsoft/SynapseML/actions/runs/42014432464
Rationale
- The Python style gate identified formatting in two new test files.
- The changes preserve both files' Python ASTs, and the repository's
tracked Python files pass Black 22.3.0.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 5ba39157-c8dd-4adb-90e0-ff963a1e6733
The reason will be displayed to describe this comment to others. Learn more.
🟡 Changes recommended
The extractor misreads declarations inside skipped triple-quoted docstrings, causing real methods to disappear from stubs and allowing a getModelCol override to bypass alias validation.
## Summary
Keep the existing lexical state current while skipping nested function and
class bodies. Triple-quoted body strings must not create false public accessor
collisions or hide a real accessor defined after the skipped scope.
## Prompting Intent
Resolve the current-head review findings for the independent Python stub
scope fix and integrate the verified repair into the focused AIPrompt work,
without expanding the native feature or optional recovery scope.
## Linked Sources
- microsoft#2805
- microsoft#2810 (review)
- microsoft#2811 (comment)
- microsoft#2759
## Rationale
Advance the existing expression scanner instead of adding another parser.
Four syntax-valid function/class regressions failed before the repair.
All 36 PyCodegenSuite cases, main/test compilation and both style gates pass
in the isolated prerequisite and integrated native-feature worktrees.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 5ba39157-c8dd-4adb-90e0-ff963a1e6733
## Summary
Keep the existing lexical state current while skipping nested function and
class bodies. Triple-quoted body strings must not create false public accessor
collisions or hide a real accessor defined after the skipped scope.
## Prompting Intent
Resolve the current-head review findings for the independent Python stub
scope fix and integrate the verified repair into the focused AIPrompt work,
without expanding the native feature or optional recovery scope.
## Linked Sources
- microsoft#2805
- microsoft#2810 (review)
- microsoft#2811 (comment)
- microsoft#2759
## Rationale
Advance the existing expression scanner instead of adding another parser.
Four syntax-valid function/class regressions failed before the repair.
All 36 PyCodegenSuite cases, main/test compilation and both style gates pass
in the isolated prerequisite and integrated native-feature worktrees.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 5ba39157-c8dd-4adb-90e0-ff963a1e6733
## Summary
Account for the additional AI test lane in Codecov's expected upload count
and apply the pinned notebook formatter to the new AIPrompt tutorial.
## Prompting Intent
Repair the deterministic checks introduced by the focused AIPrompt change
without changing dependency pins, workflows, release tooling or unrelated
notebooks, and preserve the already-executed tutorial behavior.
## Linked Sources
- microsoft#2759
- microsoft#2811
## Rationale
The authorized AI matrix lane raises expected uploads from 54 to 55.
Keep the existing coverage-count assertion unchanged and fix its configuration.
Black 22.3.0 with Jupyter support changes formatting only: all 18 code-cell
ASTs, metadata and markdown remain identical, outputs stay cleared and
OpenAIDecisions remains commented. The full CI Black command and the targeted
coverage-upload selection test pass locally.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 5ba39157-c8dd-4adb-90e0-ff963a1e6733
The reason will be displayed to describe this comment to others. Learn more.
🔵 Needs a closer look
The extractor misreads declaration-like text inside class-body multiline strings, which can omit real methods from generated stubs and leave alias collisions undetected.
No open issues 🎉
1 previously missed issue
In code that hasn't changed since last review
Scan ordinary class-body statements to avoid hiding methods in stubs
A class-body multiline string containing def notAHeader( is treated as a declaration because ordinary statements are never scanned, leaving the extractor's expression state open. Subsequent real methods can then disappear from stubs and alias-conflict checks, allowing a getModelCol collision to go undetected. Scan every class-body statement by moving expression.scan(line) outside this condition, and add a regression for a multiline string inside a control block.
## Summary
Track multiline strings and expressions in ordinary class statements so
declaration-like string content cannot hide subsequent methods or bypass
generated service-parameter accessor collision checks.
## Prompting Intent
Keep AIPrompt focused, fix regressions, and publish unrelated fixes separately.
Resolve current-head review findings without expanding the feature scope.
## Linked Sources
- Stub fix: microsoft#2810
- Stub issue: microsoft#2805
- Review: microsoft#2810 (review)
- Duplicate review: microsoft#2811 (review)
## Rationale
Reuse the existing lexical state for all class statements instead of adding
another parsing path. Four public code-generation regressions fail before
the repair and pass afterward, covering direct class statements and control
blocks. All 40 PyCodegenSuite cases, core compile/test compile, and both
Scala style checks pass. Share the existing alias-conflict assertions to
keep the suite within its file-length limit without weakening coverage.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 5ba39157-c8dd-4adb-90e0-ff963a1e6733
## Summary
Track multiline strings and expressions in ordinary class statements so
declaration-like string content cannot hide subsequent methods or bypass
generated service-parameter accessor collision checks.
## Prompting Intent
Keep AIPrompt focused, fix regressions, and publish unrelated fixes separately.
Resolve current-head review findings without expanding the feature scope.
## Linked Sources
- Stub fix: microsoft#2810
- Stub issue: microsoft#2805
- Review: microsoft#2810 (review)
- Duplicate review: microsoft#2811 (review)
## Rationale
Reuse the existing lexical state for all class statements instead of adding
another parsing path. Four public code-generation regressions fail before
the repair and pass afterward, covering direct class statements and control
blocks. All 40 PyCodegenSuite cases, core compile/test compile, and both
Scala style checks pass. Share the existing alias-conflict assertions to
keep the suite within its file-length limit without weakening coverage.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 5ba39157-c8dd-4adb-90e0-ff963a1e6733
## Summary
Track multiline strings and expressions in ordinary class statements so
declaration-like string content cannot hide subsequent methods or bypass
generated service-parameter accessor collision checks.
## Prompting Intent
Keep AIPrompt focused, fix regressions, and publish unrelated fixes separately.
Resolve current-head review findings without expanding the feature scope.
## Linked Sources
- Stub fix: microsoft#2810
- Stub issue: microsoft#2805
- Review: microsoft#2810 (review)
- Duplicate review: microsoft#2811 (review)
## Rationale
Reuse the existing lexical state for all class statements instead of adding
another parsing path. Four public code-generation regressions fail before
the repair and pass afterward, covering direct class statements and control
blocks. All 40 PyCodegenSuite cases, core compile/test compile, and both
Scala style checks pass. Share the existing alias-conflict assertions to
keep the suite within its file-length limit without weakening coverage.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 5ba39157-c8dd-4adb-90e0-ff963a1e6733
Addressed the body-only finding in review 5784252386 by integrating the separately reviewed stub repair from #2810 as 5250eb2. All 40 PyCodegenSuite cases, core compilation/test compilation, both core style gates and core/cognitive code generation pass. All 451 generated runtime/stub Python files still match the notebook-tested wheels after normalizing only generated UIDs, exact build-version literals and line endings. AIPrompt behavior and the notebook are unchanged; current-head CI and release-port validation remain pending.
The reason will be displayed to describe this comment to others. Learn more.
🔵 Needs a closer look
The changes span provider contracts and Spark-internal cancellation and persistence paths, while current-head CI, managed-runtime qualification, and prerequisite integration remain unverified.
## Summary
Keep the original response conversion failure when closing the response also
fails. Attach a distinct cleanup failure as suppressed, avoid self-suppression,
and continue propagating cleanup failures after successful conversion.
Make the regression fixtures independent of the response's default locale.
## Prompting Intent
Complete validation, reviews, CI, comment resolution, and merge ordering for
the AIPrompt prerequisite pull requests. Preserve existing public behavior,
prove fixes with regressions, and do not merge any pull request.
## Linked Sources
- HTTP response ownership fix: microsoft#2808
- Dependent AIPrompt integration: microsoft#2811
- Preserved review findings and dispositions: reviews/pr-2808/
## Rationale
Unconditional cleanup must not hide the exception that caused conversion to
fail. Keeping the same exception instance preserves existing caller behavior;
suppressed exceptions retain cleanup diagnostics without duplicate logging.
Focused tests cover conversion failure, distinct and identical cleanup
failures, and cleanup failure following successful conversion.
Java 8 compilation, test compilation, all six response-inspection tests, and
both core style checks passed. Integrated Spark port checks also passed.
All eighteen reviews in the six-stage pre-commit pass qualified on the same
two-file patch. Original superseded feedback and resolutions are retained for
archival; current-head CI and the final CI-qualified pass remain pending.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 5ba39157-c8dd-4adb-90e0-ff963a1e6733
* fix: close HTTP responses when conversion fails
## Summary
Close responses in a finally block when converting HTTP response data.
Add a regression that propagates the conversion failure while checking that
the response is closed exactly once.
## Prompting Intent
Publish the independent response-resource cleanup defect separately from
AIPrompt's new opt-in transport behavior. Keep the fix narrow and prove the
baseline failure directly.
## Linked Sources
- Bug: #2802
- Scope: #2759
## Rationale
The existing success-only close skipped cleanup when body conversion threw.
A try/finally fixes that path without changing retry accounting, adding response
limits or depending on the new native buffering implementation. The regression
failed on baseline and passed with the existing response inspection tests.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 5ba39157-c8dd-4adb-90e0-ff963a1e6733
* fix: preserve response conversion errors during cleanup
## Summary
Keep the original response conversion failure when closing the response also
fails. Attach a distinct cleanup failure as suppressed, avoid self-suppression,
and continue propagating cleanup failures after successful conversion.
Make the regression fixtures independent of the response's default locale.
## Prompting Intent
Complete validation, reviews, CI, comment resolution, and merge ordering for
the AIPrompt prerequisite pull requests. Preserve existing public behavior,
prove fixes with regressions, and do not merge any pull request.
## Linked Sources
- HTTP response ownership fix: #2808
- Dependent AIPrompt integration: #2811
- Preserved review findings and dispositions: reviews/pr-2808/
## Rationale
Unconditional cleanup must not hide the exception that caused conversion to
fail. Keeping the same exception instance preserves existing caller behavior;
suppressed exceptions retain cleanup diagnostics without duplicate logging.
Focused tests cover conversion failure, distinct and identical cleanup
failures, and cleanup failure following successful conversion.
Java 8 compilation, test compilation, all six response-inspection tests, and
both core style checks passed. Integrated Spark port checks also passed.
All eighteen reviews in the six-stage pre-commit pass qualified on the same
two-file patch. Original superseded feedback and resolutions are retained for
archival; current-head CI and the final CI-qualified pass remain pending.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 5ba39157-c8dd-4adb-90e0-ff963a1e6733
* chore: publish HTTP response cleanup review evidence
## Summary
Retain the complete pre-commit and CI-qualified final review sets for the HTTP
response cleanup fix. Index superseded reports in their immutable published
archive instead of keeping every historical attempt in the final PR diff.
## Prompting Intent
Complete missing validation, reviews, CI, and comment resolution for the
SynapseML PR stack before describing it as merge-ready. Preserve original
feedback and do not merge or enable auto-merge.
## Linked Sources
- Pull request: #2808
- Archived evidence: https://github.com/microsoft/SynapseML/tree/5a756d586411d0fc5bb4f7825c55972c18ee1803/reviews/pr-2808
- Review and retention rules: .github/skills/synapseml-pr-loop/references/review-gauntlet.md and references/loop-control.md
## Rationale
All six final rounds completed with clean results from all three required
reviewers on the same product patch. This artifact-only publication preserves
that fingerprint and avoids recursively reviewing review text. Final-commit CI
and remote review remain required; no product code, tests, or dependencies change.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 5ba39157-c8dd-4adb-90e0-ff963a1e6733
* chore: clean whitespace in HTTP cleanup review evidence
## Summary
Remove one trailing-whitespace line from a preserved PR review artifact.
No production code, tests, build inputs, or review conclusions change.
## Prompting Intent
Make the necessary minor fixes to the reviewed SynapseML PRs, address review
feedback, and verify full CI against each final head without merging.
## Linked Sources
- PR: #2808
- Issue: #2802
- Repository artifact policy: AGENTS.md, Pull requests and CI
## Rationale
Keep the numbered review artifacts and their provenance as required by the
repository guide while making the complete PR diff whitespace-clean.
The verified HTTP response ownership fix and its tests are untouched.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 0e8fdfb7-dcdd-4074-aabb-8634695107f2
---------
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 5ba39157-c8dd-4adb-90e0-ff963a1e6733
Copilot-Session: 0e8fdfb7-dcdd-4074-aabb-8634695107f2
* fix: preserve method scope and binding in Python stubs
## Summary
Recognize only class-level additional Python methods when generating stubs
and validating service-parameter aliases. Preserve staticmethod/classmethod
decorators and omit nested implementation helpers from the public stub.
## Prompting Intent
Keep generic generator defects out of the AIPrompt feature diff by publishing
a small standalone fix, with a supported custom-stage input that fails before
the change and retains existing public collision checks.
## Linked Sources
- Bug: #2805
- Scope: #2759
## Rationale
Use the generator's class-method indentation boundary for both alias checks
and stub emission, rather than treating every nested def as a public method.
The custom fixture failed on baseline; the full code-generation suite passed
after the fix. This does not claim that every shipped wrapper used the pattern.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 5ba39157-c8dd-4adb-90e0-ff963a1e6733
* fix: retain conditional Python methods and binding decorators
## Summary
Resolve the two generator review findings by sharing scope-aware method
extraction between stub generation and ServiceParam alias validation.
Preserve binding decorators across comments, blank lines and stacked decorators.
## Prompting Intent
Keep AIPrompt focused, isolate independent defects into their own PRs, and
resolve review findings with regression evidence rather than unrelated changes.
## Linked Sources
- #2805
- #2810
- #2810 (comment)
- #2810 (comment)
- #2759
## Rationale
Keep the existing synchronous single-line method-signature grammar while
tracking enclosing function/class scopes and complete decorator blocks.
Extract the helper and fixture object to respect the repository's file-length
limit without changing fixture JVM names or adding Python parser dependencies.
Three regressions fail on the preceding PR implementation, and two multiline
header regressions fail before the scope-continuation repair. All 26 codegen
tests and both Scala style gates pass in the isolated and integration trees.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 5ba39157-c8dd-4adb-90e0-ff963a1e6733
* fix: retain lexical state across Python method headers
Summary
- Preserve quote, escape, delimiter and continuation state across multiline
headers and decorators in the shared Python method extractor.
- Recognize binding decorators with trailing comments and retain their binding
across nonbinding multiline decorators.
- Cover six additional regressions with Python syntax-checked fixtures.
Prompting Intent
- Keep the AIPrompt contribution focused and repair independent generator
regressions in their own prerequisite pull request.
Linked Sources
- #2805
- #2810
- #2759
Rationale
- Punctuation inside strings and comments must not change method scope or
accidentally discard a pending static/class binding.
- All six new regressions fail on the preceding extractor. All 32 generator
tests, compilation and both style gates pass with this repair.
- Only the private extractor and its existing fixture/suite files change.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 5ba39157-c8dd-4adb-90e0-ff963a1e6733
* fix: preserve Python string state inside skipped method bodies
## Summary
Keep the existing lexical state current while skipping nested function and
class bodies. Triple-quoted body strings must not create false public accessor
collisions or hide a real accessor defined after the skipped scope.
## Prompting Intent
Resolve the current-head review findings for the independent Python stub
scope fix and integrate the verified repair into the focused AIPrompt work,
without expanding the native feature or optional recovery scope.
## Linked Sources
- #2805
- #2810 (review)
- #2811 (comment)
- #2759
## Rationale
Advance the existing expression scanner instead of adding another parser.
Four syntax-valid function/class regressions failed before the repair.
All 36 PyCodegenSuite cases, main/test compilation and both style gates pass
in the isolated prerequisite and integrated native-feature worktrees.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 5ba39157-c8dd-4adb-90e0-ff963a1e6733
* fix: scan class statements when extracting Python methods
## Summary
Track multiline strings and expressions in ordinary class statements so
declaration-like string content cannot hide subsequent methods or bypass
generated service-parameter accessor collision checks.
## Prompting Intent
Keep AIPrompt focused, fix regressions, and publish unrelated fixes separately.
Resolve current-head review findings without expanding the feature scope.
## Linked Sources
- Stub fix: #2810
- Stub issue: #2805
- Review: #2810 (review)
- Duplicate review: #2811 (review)
## Rationale
Reuse the existing lexical state for all class statements instead of adding
another parsing path. Four public code-generation regressions fail before
the repair and pass afterward, covering direct class statements and control
blocks. All 40 PyCodegenSuite cases, core compile/test compile, and both
Scala style checks pass. Share the existing alias-conflict assertions to
keep the suite within its file-length limit without weakening coverage.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 5ba39157-c8dd-4adb-90e0-ff963a1e6733
---------
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 5ba39157-c8dd-4adb-90e0-ff963a1e6733
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Users currently need different Spark stages and provider-specific handling for text, embeddings and native decision models. This adds
AIPrompt, built on the existingOpenAIPromptinterface, so callers can select the API explicitly, pass supported request options and inspect typed results, errors and complete response JSON without adding another inference transport.Related Issues/PRs
Implements the native-inference portion of #2759. The umbrella remains open: recoverable Delta results are tracked separately in #2800 and the unqualified draft #2813.
The five original prerequisite PRs are merged: #2806, #2807, #2808, #2809 and #2810. Later review fixes are preserved in separate draft follow-ups: #2818 (nested assigned-session persistence), #2819 (Python method bindings and alias identity), and #2820 (wrapped attachment cancellation).
Merge order: qualify and merge those three independent follow-ups in any order, then integrate them into this PR and complete final-head validation before merging it. Optional recovery in #2813 comes last and remains blocked on managed cross-application restart qualification. No PR in this remaining set is being presented as merge-ready. The local rebase onto merged master is complete; its reviewed changes have not yet been pushed to this PR.
Live OpenAIDecisions qualification remains #2760. It is not supported in Azure OpenAI/Foundry; the notebook leaves its examples commented out.
What changes are proposed in this pull request?
AIPromptfor explicit Chat Completions, Responses, embeddings and native API selection, plus the smallAISystemOneandOpenAIDecisionsspecializations.AIPrompt.ipynb, following the existingOpenAI.ipynbtutorial style, and a focused usage guide. All 32 supported examples executed on the packaged artifacts; Decisions examples remain comments.This PR does not add Delta recovery, response-size configuration, old-reader/new-class interoperability, a tool-execution loop or an exactly-once inference guarantee. Default Spark execution remains lazy and may replay model calls.
How is this patch tested?
Current rebase validation (November 9): local head
c3ab8d96582e0f690885f9c78c660ea982fc4feawith pending treef4f88f80beacff35a6c84d502c0c74a64ac9d707integrates master6c2699defcfb24d5bff5ba55f2252ba2d8cbea24.sbt "set core / Test / fork := false" core/compile core/Test/compile "core/testOnly com.microsoft.azure.synapse.ml.core.serialize.ValidateComplexParamSerializer com.microsoft.azure.synapse.ml.stages.VerifyUDFTransformerPersistence com.microsoft.azure.synapse.ml.io.split1.VerifyResponseBodyInspection com.microsoft.azure.synapse.ml.io.split1.VerifyCancellableRequests com.microsoft.azure.synapse.ml.io.split1.VerifyBoundedRequests com.microsoft.azure.synapse.ml.param.VerifyGlobalParams com.microsoft.azure.synapse.ml.codegen.PyCodegenSuite" core/scalastyle core/Test/scalastylepassed: 183 tests, zero skips, both compilation and style gates.python -B -m pytest tools\ci\tests\test_aiprompt_notebook.py -qpassed: 6 tests, zero skips.python -m black --check core\src\main\python\synapse\ml\core\schema\Utils.py cognitive\src\test\python\synapsemltest\services\test_AIPrompt.py cognitive\src\test\python\synapsemltest\services\test_AIPromptLive.py cognitive\src\test\python\synapsemltest\services\aiprompt_fabric_e2e.py tools\ci\tests\test_aiprompt_notebook.pypassed with Black 22.3.0. Rebased cognitive integration and installed-package validation have now passed on that same source tree. Complete content reviews, publication and final-head CI remain pending.sbt "set cognitive / Test / fork := false" cognitive/compile cognitive/Test/compile "cognitive/testOnly com.microsoft.azure.synapse.ml.services.ai.AIJsonSuite com.microsoft.azure.synapse.ml.services.ai.AIPromptAttachmentSuite com.microsoft.azure.synapse.ml.services.ai.AIPromptCancellationSuite com.microsoft.azure.synapse.ml.services.ai.AIPromptEndpointSuite com.microsoft.azure.synapse.ml.services.ai.AIPromptFabricSuite com.microsoft.azure.synapse.ml.services.ai.AIPromptHttpPolicySuite com.microsoft.azure.synapse.ml.services.ai.AIPromptInputSuite com.microsoft.azure.synapse.ml.services.ai.AIPromptLoggingSuite com.microsoft.azure.synapse.ml.services.ai.AIPromptMediaInputSuite com.microsoft.azure.synapse.ml.services.ai.AIPromptMessageSchemaSuite com.microsoft.azure.synapse.ml.services.ai.AIPromptPersistenceSuite com.microsoft.azure.synapse.ml.services.ai.AIPromptPostProcessingSuite com.microsoft.azure.synapse.ml.services.ai.AIPromptPreflightSuite com.microsoft.azure.synapse.ml.services.ai.AIPromptResponseFormatSuite com.microsoft.azure.synapse.ml.services.ai.AIPromptResponseShapeSuite com.microsoft.azure.synapse.ml.services.ai.AIPromptSuite com.microsoft.azure.synapse.ml.services.ai.AIPromptUnicodeSuite com.microsoft.azure.synapse.ml.services.ai.AIResponseCodecSuite com.microsoft.azure.synapse.ml.services.ai.AISystemOneSuite com.microsoft.azure.synapse.ml.services.ai.AIValueCodecSuite com.microsoft.azure.synapse.ml.services.ai.NativeQuestionsSuite com.microsoft.azure.synapse.ml.services.openai.OpenAIDecisionsSuite com.microsoft.azure.synapse.ml.services.openai.OpenAIPromptMultimodalRequestSuite com.microsoft.azure.synapse.ml.services.openai.OpenAIPromptParserSuite" cognitive/scalastyle cognitive/Test/scalastylesbt core/codegen cognitive/codegen core/packageBin cognitive/packageBin "export cognitive/Compile/fullClasspath"cognitive/src/test/python/synapsemltest/services/test_AIPrompt.pyThe complete current patch is frozen into 24 full-file review slices with roundtrip coverage of all 86 changed files. That preparation is not a completed review or a green CI result.
Historical evidence below: these results retain their original revision scope; they do not qualify the new rebased head automatically.
Validated the focused source against
masteratd3f3af6bd3e4b7d191f51c9eb29f9aeffe788193, using Spark 3.5.0, Scala 2.12.17, JDK 11 and Python 3.11.8. The local SBT launcher redirects build output and caches outside the checkout; the task arguments are listed below.sbt core/compile core/Test/compile cognitive/compile cognitive/Test/compilesbt core/scalastyle core/Test/scalastyle cognitive/scalastyle cognitive/Test/scalastylesbt core/codegen cognitive/codegen core/packageBin cognitive/packageBinsbt "core/testOnly com.microsoft.azure.synapse.ml.codegen.PyCodegenSuite"python -m black --check --extend-exclude 'docs/'over all 218 tracked Python filespython -m black --check -q .python -B -m pytest tools\ci\tests\test_e2e_impact.py::test_selection_preserves_all_expected_coverage_uploads -v -rsnpm run buildinwebsiteFocused transport/legacy and core persistence/codegen/notebook-selection runs passed 49 and 61 tests respectively. The native run initially passed 442 of 444 tests; two extraction-related test defects were corrected and the affected suites passed 18/18. The live-service and benchmark tests were opt-in cancellations in that offline run, not counted as passes; the benchmark and live notebook were subsequently run explicitly.
Installed-wheel tests exposed a native Python loader regression during the split. After repairing the native reader, its six persistence/loader cases passed; 29 legacy parameter/response-schema cases also passed. Earlier combined-tree evidence is not substituted for these focused-tree results.
After the 32-test stub-extractor revision, compilation, both core style gates, code generation and JAR packaging passed again. Subsequent string-state repairs add eight regressions for skipped function/class bodies and ordinary class/control-block statements. Compilation, all 40 generator tests, both core style gates and
sbt core/codegen cognitive/codegenpassed on the latest integrated revision. All 451 final generated runtime/stub Python files match the notebook-tested wheels after normalizing only generated UIDs, exact build-version literals and line endings; none were added or removed.Local Spark 4.0.1 and 4.1.1 compatibility replays also passed compilation, all four style gates and code generation after port-specific adaptations for Scala 2.13 ordered-map serialization, Spark column APIs and the ports' existing Python-wrapper conventions. These adaptations are preserved separately for port integration, not added to this master PR.
sbt "core/testOnly *PyCodegenSuite"sbt "cognitive/testOnly *AIPromptPythonPortSuite *AIJsonSerializationSuite *AIPromptInputSuite *AIPromptPostProcessingSuite *AIPromptPreflightSuite *AIValueCodecSuite"python .validation\aiprompt-core-port-compat99\python-smoke.py --port 40andpython .validation\aiprompt-core-port-compat99\python-smoke.py --port 41using the local installed-artifact harnessThe port JVM evidence totals 449 passed and one intentionally disabled benchmark per port, combining the full run with targeted reruns of repaired suites—not a fresh full sweep. Installed-wheel smoke covers public imports/construction, scalar/column JSON bindings, copy/save/load, loader identity/rejection and loopback AIPrompt transformation. No live endpoint calls were made by these port runs. This bounded local evidence does not substitute for full port CI on its pinned Python versions or managed-runtime qualification.
The final notebook run passed all 18 code cells covering 32 examples: chat, Responses, structured output, postprocessing, image/file attachments, stored-response chaining, embeddings, direct/facade SystemOne, per-row errors, parameters/copy, save/load, pipelines and pandas conversion. It used the newly packaged JARs and installed wrappers with real Azure OpenAI and Foundry Decision-1 calls.
OpenAIDecisionswas not executed. Notebook outputs and credentials are not committed.The controlled 4,096-row loopback benchmark passed with all expected requests. At 4 partitions/concurrency 1, legacy/native elapsed times were 29.258s/7.885s; at 16 partitions/concurrency 4 they were 22.203s/2.881s. This is a fixed 5ms-provider-latency transport comparison, not a real-service speed guarantee.
Still required before merge: current-head automated review, Azure Pipelines/GitHub checks and integration of any prerequisite follow-ups. Historical monolithic runs are not used to qualify this split. Full current-head CI and managed-runtime qualification are not claimed complete.
Does this PR change any dependencies?
Does this PR add a new feature? If so, have you added samples on website?
website/docs/documentationfolder.Make sure you choose the correct class
estimators/transformersand namespace.DocTablepoints to correct API link.npm startto make sure the website renders correctly.<!--pytest-codeblocks:cont-->before each python code blocks to enable auto-tests for python samples.WebsiteSamplesTestsjob pass in the pipeline.