Repository navigation
[dataprotection] Add Cosmos DB item-level recovery - #10486
JaysonTaiMicrosoft merged 8 commits into
Conversation
microsoft-github-policy-service
Bot
added
the
customer-reported
|
Thank you for your contribution Shashank Singh (@Shashank1306s)! We will review the pull request and get back to you soon. |
microsoft-github-policy-service
Bot
added
CosmosDB
Auto-Assign
|
/azp run |
|
Azure Pipelines: Successfully started running 2 pipeline(s). |
Final validation update (commit 3230471)Upstream azdev-style, azdev-linter, release-version calculation/output and CLA checks are now successful. Credential Scan and Policy Check are successful too. The remaining Azure integration/build checks are still pending/unknown; the PR remains draft. Addressed the initial missing-final-newline finding and two pre-existing chained-comparison warnings. The latter change is behavior-equivalent: the actual old/new Blob prefix validators returned identical acceptance/errors across 225 range combinations. Revalidated the published code:
Live native CLI restore/data-exclusion acceptance is still not executed: exact fresh-target/RBAC/cost authorization was not supplied. No live resource/role/data/vault-setting writes were performed. Four old recorded modules still have the same bounded-timeout outcome on both untouched upstream and this branch; no full-suite-green claim is made. |
Real-data wheel acceptance: all seven cases passedTested the local dataprotection 1.14.0 wheel, built from current PR head bc67e9b, against the existing real-data NoSQL fixture in East Asia. Isolated wheel import/discovery and SHA256 were verified; no source shadowing.
Every lifecycle step used native Verified exact selected inventory, partition keys, document IDs/values/nested objects/array order and exclusions against the frozen recovery point. Post-backup data was excluded. Source remained unchanged (3 DBs / 8 containers / 15 documents). Completed 2026-10-19 12:25 UTC. Only fresh tagged targets and narrow target/verification roles were created. No source, existing vault/policy/retention/network/identity changes or cleanup. Targets remain billable. Detailed evidence stays local, not added to the product diff. Scope: seven positive NoSQL full/ILR cases in one region. No new backup/protection lifecycle or live negative/50-selector/other-region/API certification is claimed. This supersedes earlier comments saying live acceptance was pending. |
There was a problem hiding this comment.
🔵 Needs a closer look
The data-recovery workflow needs human sign-off and confirmation of pending upstream validation for the current revision.
No open issues 🎉
What changed in this PR
Adds Cosmos DB for NoSQL database/container recovery to the existing dataprotection restore workflow, reusing shared identity, permission, and transport handling.
Changes:
- Enables item-level recovery with validated selectors and alternate-account targets.
- Adds focused configuration, request, and validation tests.
- Documents recovery workflows and bumps the extension to 1.14.0.
| File | Description |
|---|---|
| src/dataprotection/setup.py | Bumps version to 1.14.0. |
| src/dataprotection/README.md | Documents Cosmos recovery and identity usage. |
| src/dataprotection/HISTORY.rst | Records the new recovery capability. |
| src/dataprotection/azext_dataprotection/tests/latest/test_dataprotection_cosmosdb.py | Tests selectors, configuration, requests, and validation. |
| src/dataprotection/azext_dataprotection/manual/Manifests/AzureCosmosDB.py | Enables item-level recovery. |
| src/dataprotection/azext_dataprotection/manual/helpers.py | Validates Cosmos selections and constructs restore criteria. |
| src/dataprotection/azext_dataprotection/manual/custom.py | Integrates Cosmos configuration and target checks. |
| src/dataprotection/azext_dataprotection/manual/_params.py | Updates restore argument descriptions. |
| src/dataprotection/azext_dataprotection/manual/_help.py | Adds Cosmos command guidance and examples. |
Review effort: Balanced
…ing the AzureElasticSAN pattern - Remove client-side checks not backed by the service or live ILR testing: selector count cap, character/duplicate/overlap rules, strict restore-configuration schema checks and target ARM-type checks. - initialize-restoreconfig and get_resource_criteria_list now mirror AzureElasticSAN: require --resource-identifiers / --restore-configuration and pass them through. - Revert shared-path changes: Cosmos rejection in initialize-for-data-recovery, unsupported-datasource error text, and the Blob prefix-range comparison (byte-identical to main; pre-existing chained-comparison lint suppressed at file level). - Generic help, README and HISTORY wording; tests cover the live-validated request shape and the required inputs only.
…estore inputs - initialize-restoreconfig (AzureCosmosDB): reject AKS-only arguments and --resource-name-overrides; reject empty/whitespace identifiers and empty path segments (e.g. "db1/ "); strip identifiers (AzureElasticSAN parity). - initialize-for-item-recovery (AzureCosmosDB): reject --container-list, --from/--to-prefix-pattern and --vaulted-blob-prefix-pattern; require a GenericRestoreDatasourceCriteria restore configuration. - --resource-identifiers help: datasource-neutral first sentence. - helpers.py: scope the chained-comparison suppression to the existing Blob prefix-overlap check instead of the whole module. - Tests: add a Cosmos input-validation test.
|
Hi Yong Zhang (@yonzhan) , please help merge this PR at the earliest, I've addressed the comments and have tested the commands e2e. We want this to be released in the current milestone. Please let me know if anything is needed |
JaysonTaiMicrosoft
added
the
Request X Engineering Agent
|
Shashank Singh (@Shashank1306s) No significant defects confirmedThe Cosmos DB item-level request preserves the selected resources through the existing REST serialization and targets MSI permissions at the destination account. The added scenario exercises the item-level path and checks both selected resources and exclusions. Upstream CIThe supplied upstream snapshot has 23 completed checks: 22 passed, 0 failed, 0 pending. This does not establish execution of the live-only scenario. Test validation
Semantic coverage clarification: The automated applicability classification above does not describe the extension's actual tests. At the reviewed head, test_dataprotection_cosmosdb.py:188–260 asserts exact selection/configuration, request target and identity, and missing/invalid-input errors. These are initializer-output assertions, not HTTP-body assertions. The item-level scenario:492–569 invokes MSI permissions, restore validation and restore trigger, waits for job completion, then checks the exact restored database/container inventory. It is live-only and has no replay recording. The author reports that this added scenario passed on ⚖️ Risk assessment — 37/100 · MediumThe Medium rating is driven by public CLI behavior, dependency or supply chain.
|
x-engineering-agent
Bot
added
the
X Engineering Agent Reviewed
Add test_dataprotection_backup_and_restore_cosmosdb_item_level (@live_only), modeled on the AzureElasticSAN live scenario. It takes an adhoc backup, restores db1/col1 and db2 to an empty target account through initialize-restoreconfig, initialize-for-item-recovery, validate-for-restore and restore trigger, asserts that only the selected resources were restored, and empties the target afterwards.
|
Replying to the X Engineering Agent report above. F1 – identifier validation: Not changing this. Identifier validation is left to the service, which rejects invalid identifiers with a user error, so the CLI does not duplicate those rules. The CLI only rejects empty or whitespace identifiers (added for an earlier review comment). This also matches Az.DataProtection (Azure/azure-powershell#30300), which passes identifiers through unchanged. F2 – item-level coverage: Added Ran it live in eastus2euap: passed in 23m50s (backup job |
![x-engineering-agent[bot]](https://avatars.githubusercontent.com/in/4205534?s=60&v=4){kind=small}
There was a problem hiding this comment.
Shashank Singh (@Shashank1306s) Ready for human review. See the X Engineering Agent Report.
x-engineering-agent
Bot
removed
the
Request X Engineering Agent
…ve test Build the target account ID from the subscription the test runs under instead of a hard-coded subscription, and remove the subscription from the fixture comment.
|
[Release] Update index.json for extension [ dataprotection-1.14.0 ] : https://dev.azure.com/msazure/One/_build/results?buildId=187700007&view=results |
🤖 PR Validation —⚠️ Review suggested
This PR adds warning or check suppressions. Fix the underlying warning where possible and justify each remaining suppression in a code comment or the PR description.
1 added suppression
src/dataprotection/azext_dataprotection/manual/helpers.py# pylint: disable=chained-comparisonRelated commands
az dataprotection backup-instance initialize-restoreconfig --datasource-type AzureCosmosDBaz dataprotection backup-instance restore initialize-for-item-recovery --datasource-type AzureCosmosDBSummary
Adds AzureCosmosDB item-level (database/container) restore, following the existing AzureElasticSAN pattern:
initialize-restoreconfig --datasource-type AzureCosmosDB --resource-identifiers ...returnsGenericRestoreDatasourceCriteriawithResourceListSelectionCriteria; identifiers are passed through unchanged.--resource-identifiersis required, as for AzureElasticSAN.restore initialize-for-item-recovery --datasource-type AzureCosmosDB --restore-configuration ...uses that configuration as the restore criteria.--restore-configurationis required, as for AzureElasticSAN.No client-side limits, naming rules or target-type checks are added; the selection and target are validated by the service (
validate-for-restore). Identity, MSI permission and restore transport handling are reused unchanged.Other datasources' logic is unchanged.
helpers.pyonly gains a file-level# pylint: disable=chained-comparisonbecause CI lints the whole module and flags pre-existing Blob prefix-range lines; the Blob code is identical tomain.Validation
4ebb7cca): all checks pass, including azdev style/linter, the breaking change test and integration tests on Python 3.10-3.14.db1/col1) from a VaultStore recovery point into an empty account restored only that container; its documents match the recovery point exactly, and the unselected containers and databases were not restored.General Guidelines