Repository navigation
Update Nexus to 3.97.2 and acknowledge the required base telemetry - #5278
Open
Jay W (JayDoubleu) wants to merge 4 commits into
Open
Jay W (JayDoubleu) wants to merge 4 commits into
Jay W (JayDoubleu) wants to merge 4 commits into
Conversation
Nexus 3.97.2 stops trusting the eMudhra certificate chains that Sonatype reports as compromised. Nexus 3.97 and later must upload telemetry to nxrm.telemetry.sonatype.com. If uploads fail for 10 days, Nexus Community Edition becomes read-only. - Update the Nexus image from 3.94.0 to 3.97.2. - Acknowledge the required base telemetry through the REST API, with retries and time limits. Stop the deployment if it fails. - Allow nxrm.telemetry.sonatype.com in the Nexus firewall rule. - Test the acknowledgement in the Nexus script unit tests. - Document the telemetry requirement, the egress rule, the read-only effect, and that the upgrade is one-way. Fixes microsoft#5276 Part of microsoft#5271
A live upgrade test showed that the Baseline Telemetry check shows "Last successful upload: never" after the upgrade. Nexus 3.97.2 uploads every 24 hours and when it stops, so the first upload can take up to 24 hours. - Tell administrators that this status is normal after a deployment or an upgrade. - Say that the Telemetry upload retry task sends the failed uploads again, which is what the task does.
The one-way upgrade was a sentence at the end of the telemetry section. Operators can miss it, and the changelog could not link to it. - Add an "Upgrade to Nexus bundle 3.12.0" section with a warning that the upgrade is one-way, and the steps to do before the upgrade. - Link the breaking change entry in the changelog to this section.
The repository versioning guide asks for a major version for changes that do not upgrade automatically, or that need someone to take action first. The Nexus 3.97.2 upgrade is one-way and can need egress changes outside the TRE. If telemetry uploads fail for 10 days, Nexus becomes read-only. The API blocks a major upgrade unless the request sets force_version_update=true, and the UI does not offer it. This makes operators read the upgrade steps before they upgrade. - Change the bundle version from 3.12.0 to 4.0.0. - Rename the docs section and its anchor, and update the changelog. - Tell operators to upgrade through the API with force_version_update=true after the steps before the upgrade.
Unit Test Results0 tests 0 ✅ 0s ⏱️ Results for commit d1a387c. |
Contributor
There was a problem hiding this comment.
🔵 Needs a closer look
The irreversible database migration and mandatory telemetry dependency warrant human sign-off, with the final forced-major upgrade path still unverified in Azure.
No open issues 🎉
What changed in this PR
Updates the Azure TRE Nexus shared service to 3.97.2 to remove trust in compromised eMudhra certificate chains and support mandatory Sonatype telemetry.
Changes:
- Bumps the Nexus bundle to 4.0.0 and documents the irreversible upgrade.
- Adds telemetry egress access and acknowledgement with retries and failure handling.
- Adds acknowledgement tests, operator guidance and release notes.
| File | Description |
|---|---|
templates/shared_services/sonatype-nexus-vm/terraform/locals.tf |
Adds the required telemetry host to the Nexus egress allowlist. |
templates/shared_services/sonatype-nexus-vm/template_schema.json |
Adds telemetry information to EULA help text. |
templates/shared_services/sonatype-nexus-vm/scripts/configure_nexus_repos.sh |
Acknowledges telemetry before configuring repositories. |
templates/shared_services/sonatype-nexus-vm/porter.yaml |
Updates the Nexus image and major bundle version. |
docs/tre-templates/shared-services/nexus.md |
Documents upgrade precautions, telemetry requirements and recovery. |
devops/tests/test_nexus_repos.py |
Tests acknowledgement success, retries and persistent failures. |
CHANGELOG.md |
Records the security update and breaking upgrade requirements. |
Review effort: Balanced
Collaborator
Author
|
/test-shared-services d1a387c accept_nexus_eula |
|
🤖 pr-bot 🤖 🏃 Running shared service tests: https://github.com/microsoft/AzureTRE/actions/runs/40027819217 (in response to this comment from Jay W (@JayDoubleu)) |
Jay W (JayDoubleu)
marked this pull request as ready for review
October 24, 2026 11:00
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Resolves #5276 and part of #5271
This PR updates the Nexus shared service from Nexus 3.94.0 to Nexus 3.97.2. The Nexus bundle version changes from 3.11.2 to 4.0.0.
What is being addressed
The JVM in Nexus 3.94.0 trusts certificates from 8 eMudhra intermediate CAs. The private keys of these CAs are compromised (Sonatype advisory). The CA generated the keys incorrectly, and anyone can recover them from the public certificates (incident page, Mozilla bug 2083449). The Docker images of Nexus 3.97.2 remove the trust in these CAs.
In short, the certificate part of an attack is easy, but the network part is difficult. An attacker cannot do this attack only by sending requests to Nexus or to the TRE from the internet.
What an attacker needs
To change the packages that Nexus serves, an attacker needs both of these:
pypi.org. This part is easy, because anyone can recover the compromised keys.The second part needs control of network infrastructure, not only access to the internet. Examples:
What an attacker can do with both parts
This applies to the 21 Nexus proxies that use an HTTPS upstream, for example PyPI, conda, CRAN and Docker Hub. It also applies to the proxies that send the apt GPG keys to workspace VMs, so the attacker can also replace these keys. Nexus asks the upstream again for cached content after 24 hours (
contentMaxAge1440), so the cache does not stop the attack.What an attacker cannot do
The incident page says that it is not known if attackers used these keys.
Why Nexus is affected but browsers are not
Browsers enforce Certificate Transparency (CT), a public log of trusted certificates. The incident page says that browsers that enforce CT were not at risk. The JVM in Nexus does not enforce CT.
I ran a TLS test with the JVM and the default trust store of each Nexus image. The test connected to the 14 test hosts of the incident page:
pypi.organdfiles.pythonhosted.orgdv-g3b-emudhra.we-have-a-problem.comdv-g3b-noct-emudhra.we-have-a-problem.com, with a certificate that is not in a CT log, like a certificate that an attacker makes in secretdv-g3b-aaa-emudhra.we-have-a-problem.comThe 3.94.0 JVM connected although the CA revoked the intermediates.
The packages that Nexus serves are a separate issue
This PR changes only the trust store of Nexus. It does not change the packages that Nexus serves. Many packages contain their own list of trusted CAs. Examples are
certifiandpipon PyPI, theca-certificatespackages of Ubuntu and conda, Java and Node.js runtimes, VS Code and Docker images.These packages continue to trust the compromised CAs until these two things happen:
certifi2026.10.19 andpip26.2.2. Some fixes are still pending (list of fixed versions).Nexus continues to serve the old versions. For example, a project that pins an old version of
certificontinues to get the old list of trusted CAs. This risk applies to the connections that software in a workspace makes. It does not apply to the connection between Nexus and its upstreams.Required base telemetry
Nexus 3.97.0 and later send required base telemetry to
nxrm.telemetry.sonatype.com. An administrator must acknowledge this requirement. If the uploads fail for 10 days, Nexus Community Edition becomes read-only. In read-only mode, Nexus serves only the packages that are already in its cache.How is this addressed
porter.yaml: change the image tag from3.94.0to3.97.2and the bundle version from3.11.2to4.0.0.terraform/locals.tf: addnxrm.telemetry.sonatype.comtonexus_allowed_fqdns. Thenexus-package-sourcesfirewall rule allows this list only from the private IP address of the Nexus VM.scripts/configure_nexus_repos.sh: after the EULA step, sendPOST /service/rest/v1/system/mandatory-telemetry-acknowledgementwith{"acknowledgement":true}. The script retries with backoff. If Nexus does not return HTTP 204, the script stops before it configures the repositories, and the deployment fails.template_schema.json: add a link to the Sonatype telemetry page in the help text ofaccept_nexus_eula.devops/tests/test_nexus_repos.py: add 3 tests for the acknowledgement. The tests cover success, a retry after temporary failures, and a stop after persistent failures.docs/tre-templates/shared-services/nexus.md:force_version_update=true.CHANGELOG.md: add a BREAKING CHANGES entry that links to the upgrade section, and a BUG FIXES entry.The bundle acknowledges the telemetry automatically. The operator accepts the Nexus EULA with
accept_nexus_eula, and the help text of that property now links to the telemetry page. Sonatype gives no option to turn off this telemetry in Community Edition.The bundle version is a major version. The versioning guide asks for a major version for changes that do not upgrade automatically, or that need someone to take action first. This change has these properties:
The API blocks a major upgrade unless the request sets
force_version_update=true, and the UI does not offer it. So an operator cannot start this one-way upgrade from the UI by accident.Known issue that this PR does not fix
If an upgrade fails and the operator tries again at the same template version, Terraform does not run
azurerm_virtual_machine_run_command.configure_nexus_reposagain. The new image can then stay missing without an error. This behaviour exists since #4983. A separate issue can fix it.Test results
Local tests
python3 -m unittest discover -s devops/testspre-commit, all hooksmkdocs build --strictGETreturns{"acknowledgement": false}ortrue.POSTreturns 204 and can run again.POSTwithfalsereturns 400.configure_nexus_repos.sh(Podman)Live upgrade test in a CI environment in another fork
main(a5252e7b) passed.six-1.17.0wheel through the PyPI proxy of Nexus.azuremlbundle failed, at the Azure login step (JSON is invalid). This PR does not change that bundle.PATCHrequest through the API. The upgrade took 6 minutes.linux_vmE2E test passed against the upgraded Nexus.The later commits change the docs and
CHANGELOG.md. The last commit also changes the bundle version from 3.12.0 to 4.0.0. The bundle content is otherwise the same as in the test. The same commits passed lint, CodeQL and the Nexus bundle build in another fork.Nexus/3.94.0-12 (COMMUNITY)Nexus/3.97.2-02 (COMMUNITY){"acknowledgement":true}sixwheelnexus-package-sourcesfirewall ruleOther results:
nxrm.telemetry.sonatype.comis the only new outbound host. Nexus 3.94.0 and 3.97.2 also callrhc.sonatype.comfor the Repository Health Check. The firewall denied this host before and after the upgrade, and Nexus works without it.Not tested in Azure:
force_version_update=true. The test used the version 3.12.0, which did not need this parameter.