[Rana Singh (ranadeepsingh)]

SynapseML has no Azure Content Understanding stage, so users have to write service calls and job recovery themselves. This PR adds a direct REST transformer and a journal that saves accepted operation handles before polling. It supports explicit Foundry resources and uses the runtime endpoint and identity for Fabric-managed analysis.

Related Issues/PRs

Related implementation and review: #2815. This update applies the retry, API-option, cancellation, and example findings to the existing implementation here.

What changes are proposed in this pull request?

ContentUnderstanding calls the REST API from Scala and exposes a generated Python API. It needs no Content Understanding SDK. It uses SynapseML's shared HTTP transformer for Spark concurrency, with a CU-specific protocol that distinguishes rejected submissions from work the service may already have accepted.

• Analyze one URL or binary document per row. Options include analyzer ID, API version, name, MIME type, range, model deployments, string encoding, processing location, and preview allowInputTruncation. Service options have scalar and column setters.
• Support analyze, submit, and poll modes. The service processes accepted jobs asynchronously. SynapseML can await their results or return a handle for later polling; it does not leave a detached polling daemon running.
• Default polling to six seconds when Retry-After is absent. setPollingIntervalSeconds(6) configures seconds in Python and Scala, including fractional values. The existing millisecond setter and saved pollingDelay parameter remain compatible.
• Retry definite HTTP 429 responses indefinitely by default, honoring Retry-After without consuming the ordinary poll budget. setRetryOnThrottling(False) restores bounded handling.
• Never replay a POST or PUT with an operation handle, including an invalid handle. Do not retry an uncertain submission after a transport failure or HTTP 408/5xx.
• Stop retry waits when the owning Spark task is cancelled or completes, including concurrent-result timeout. The shared HTTP worker carries its task context. Interrupted futures complete as failures instead of leaving their promises pending.
• Add writeToTable and writeToPath with an append-only Delta or Parquet journal. Stable IDs and request hashes prevent accidental reuse with different inputs. Completed records are skipped; saved handles resume without a new POST. readTable and readPath contact storage only.
• Keep the full service response JSON, including fields, confidence, grounding, usage, errors, and unknown preview fields. Analyzer creation and lookup are explicit driver calls for BYO resources. Transforms do not provision resources or change model defaults.
• Configure the Fabric AI gateway and MWC authentication when no endpoint or credentials are supplied. Keep the workspace/capacity prefix during polling and reject out-of-scope operation URLs. Explicit credentials take precedence; BYO endpoints never inherit Fabric tokens.
• Refresh implicit Fabric authentication once after a handle-free HTTP 401. A token-acquisition exception during polling retains a resumable operation rather than causing another submission or recording a terminal service failure.
• Preserve Python binary getters and use the shared setter helper to reject conflicting scalar/column arguments before changing the stage.

The default API version is 2025-11-01. Explicit allowInputTruncation=True or False requires 2026-06-01-preview or newer. The check uses each row's effective API version before submission. A null truncation value omits the option.

Limits that matter

setConcurrency(n) limits concurrent transform operations per active Spark partition. In analyze mode, a slot covers submission and polling until completion or the polling attempt limit. This uses the existing shared asynchronous scheduler, not a new completion-order scheduler. Durable writers are sequential and single-owner; batchSize controls row collection, not request concurrency. Their keyset batches can rescan and sort the input manifest. For large manifests, use controlled batches or foreachBatch with stable IDs and the same journal.

The Fabric-managed route supports analysis and result polling, not createAnalyzer or getAnalyzer. Availability depends on the Fabric service, capacity, and region. This client change does not enable service features or make Foundry LLM deployments available through Fabric. The managed notebook uses prebuilt-read for OCR. Use BYO Foundry for the LLM-backed invoice example or custom analyzer administration.

timeout governs HTTP connection/read timeouts, maxPollAttempts bounds ordinary polling attempts, and concurrentTimeout bounds waiting for concurrent results. A blocked HTTP read may need to reach its socket timeout before cancellation is observed. Driver-side writes stop on JVM interruption; stopping only Python's wait may not stop that JVM call.

There is a crash window between service acceptance and the first journal commit. This is not an exactly-once billing guarantee. Service results also expire. Keep document IDs and source content immutable when resuming.

Examples and API references

The guide covers Python and Scala APIs, authentication, request options, recovery, and limits.

The Foundry invoice notebook generates a synthetic PDF, extracts invoice fields with prebuilt-invoice, saves the handle, and resumes polling.

The keyless Fabric notebook generates a synthetic PDF and uses managed prebuilt-read with a Delta journal. Repeated display reads saved results rather than submitting again.

References: REST analyze, Python SDK.

How is this patch tested?

• I have written tests (not required for typo or doc fix) and confirmed the proposed feature/bug-fix/change works.

Polling interval follow-up

Validated the source committed as 13b12d4a855ed456057854822861772f5e3274fc, using JDK 11, Spark 3.5.0, and Python 3.11.15.

sbt scalastyle "Test / scalastyle" compile test:compile "cognitive/testOnly *ContentUnderstanding*Suite" core/codegen cognitive/codegen core/package cognitive/package "export cognitive/Compile/fullClasspath"

Passed. All 76 tests in seven CU suites passed, with no failures or skips. Style checks, production/test compilation, wrapper generation, and both packages passed.

python "$files/validate_cu_python.py" --repo "$repo" --classpath-log "$files/cu-poll-seconds-validation.log"
python -m black --check --extend-exclude docs/ .
python -m black --check "docs/Explore Algorithms/AI Services/Quickstart - Content Understanding.ipynb" "docs/Explore Algorithms/AI Services/Quickstart - Content Understanding in Fabric.ipynb"

Passed. The session runner loaded the newly built jars and generated wrappers, ran all eight checked-in Python tests, and executed the invoice notebook twice against a loopback service with one POST and one GET total. Black 22.3.0 passed for 219 Python files and both notebooks. Notebook JSON and code cells are valid, with no saved outputs.

Coverage includes the six-second default, fractional seconds, zero, invalid and overflowing values, legacy setter overrides, setParams, copy, clear, and save/load. All previously declared public JVM methods on ContentUnderstanding and ContentUnderstandingParams remain present.

The earlier real-service evidence below applies to a154713186. Foundry and Fabric live tests were not rerun for this interval change.

Azure build 245852861 has started for 13b12d4a855ed456057854822861772f5e3274fc. This is a new validation run, not a claim that the earlier CI failures are fixed.

Earlier validation at a154713

Rebased onto master 37948d7d03ef0fa7f1aa24c54c2dd4d8a29c8dc0. The following local and live runs validated a154713186a296a9e9d8d82fbca1abb1f2baf2b0.

Local validation used JDK 11, Spark 3.5.0, Python 3.11.15, and Black 22.3.0.

sbt scalastyle "Test / scalastyle" compile test:compile "cognitive/testOnly *ContentUnderstanding*Suite" core/codegen cognitive/codegen core/package cognitive/package "export cognitive/Compile/fullClasspath"

Passed. All 74 tests in seven CU suites ran, with no failures or skipped tests. Style, production/test compilation, wrapper generation, and both packages passed.

python -m black --check --extend-exclude docs/ .
python -m black --check "docs/Explore Algorithms/AI Services/Quickstart - Content Understanding.ipynb" "docs/Explore Algorithms/AI Services/Quickstart - Content Understanding in Fabric.ipynb"

Passed for 219 Python files and both notebooks. Both notebooks also passed JSON and code-cell syntax checks and contain no saved outputs.

Two session-only runners used the exported classpath and current generated Python packages. Here $repo is the tested worktree and $files is the session's validation directory.

python "$files/validate_cu_python.py" --repo "$repo" --classpath-log "$files/cu-fabric-final-validation.log"
python "$files/validate_cu_live.py" --repo "$repo" --git-dir "$git_dir" --classpath-log "$files/cu-fabric-final-validation.log" --evidence-dir "$files/cu-fabric-byo-foundry" --completion-model gpt-6.1-sol --expected-head a154713186a296a9e9d8d82fbca1abb1f2baf2b0

Both passed. The first ran all seven checked-in Python tests and the invoice notebook twice against a loopback service, with one POST and one GET total.

The second used local Spark against real Foundry, not a mock or Fabric runtime. All three analyses used GPT-6.1 Sol and extracted the expected vendor, invoice ID, and total. Running the notebook again left its saved result and two journal records unchanged. Two additional documents succeeded in one partition with concurrency 2. Per-request model mappings were applied after the notebook setup cell; shared resource defaults were unchanged.

The live run verified both JVM class sources and the generated wrapper. Artifact SHA-256 values were cc4ebd4d00b6507fb9190503e5e04c276057529eb23667890f7ec5131dddd217 for core and 5570ffab9ebcfe09861c75374869b2c411861bd11d8ea7b1e34a3303e9556ee3 for cognitive. Evidence is retained in cu-fabric-byo-foundry/evidence.json.

The session-only cu-probe scenario used the checked-in Fabric E2E runner. $workspace identifies the test workspace; $coreJar and $cognitiveJar are hash-qualified copies of the jars above.

python "$files/run_cu_fabric_scenario.py" cu_fabric_exact.py --scenario cu-probe --workspace "$workspace" --env msit --source-repo "$repo" --extra-jar "$coreJar" --extra-jar "$cognitiveJar" --output-dir "$files/cu-fabric-managed-a154"

Passed on Fabric Spark 3.5.5.5.4.20260921.11, application application_1794291062900_0001. All code cells in the keyless notebook ran with no explicit endpoint or credentials. Repeating the submit/poll cell reused the saved Delta result. Two further OCR analyses succeeded with concurrency 2. The scenario verified both jar sources and the exact generated core/cognitive Python packages, including the CU wrapper hash. This proves managed prebuilt-read, not managed LLM extraction.

Evidence, runtime markers, and JUnit results are retained under cu-fabric-managed-a154. The runner deleted its scratch notebook and lakehouse successfully.

The eight Fabric tests cover gateway paths, implicit authentication, bounded refresh, explicit credentials, BYO isolation, cross-workspace rejection, durable recovery, and unsupported analyzer management. Token-refresh failure coverage checks both transform output and writer recovery without another POST.

The shared-client changes also passed 2 cancellation tests, 14 shared-client/pipeline-selection tests, and 10 framework tests earlier in this PR. That framework run retained one pre-existing ignored getter/setter test. The Fabric follow-up does not change those shared-client implementations.

Website rendering has not been run locally. Azure build 245807451 for a154713186 failed Core unit tests and Databricks CPU jobs 1, 2, and 5. Their causes have not been diagnosed, and the polling interval follow-up does not claim to fix them.

Does this PR change any dependencies?

• No. You can skip this section.
• Yes. Make sure the dependencies are resolved correctly, and list changes here.

No SDK dependency, dependency-pin change, workflow change, or release-tooling change.

Does this PR add a new feature? If so, have you added samples on website?

• No. You can skip this section.
• Yes. Make sure you have added samples following below steps.

The website sidebar links to the guide, which links to both runnable notebooks. Neither notebook contains credentials or saved execution outputs.

1. Find the corresponding markdown file for your new feature in website/docs/documentation folder.
   Make sure you choose the correct class estimators/transformers and namespace.
2. Follow the pattern in markdown file and add another section for your new API, including pyspark, scala (and .NET potentially) samples.
3. Make sure the DocTable points to correct API link.
4. Navigate to website folder, and run npm start to make sure the website renders correctly.
5. Don't forget to add <!--pytest-codeblocks:cont--> before each python code blocks to enable auto-tests for python samples.
6. Make sure the WebsiteSamplesTests job pass in the pipeline.