Skip to content

Update Nexus to 3.97.2 and acknowledge the required base telemetry - #5278

Open
Jay W (JayDoubleu) wants to merge 4 commits into
microsoft:mainfrom
JayDoubleu:fix/nexus-3.97.2-telemetry
Open

Jay W (JayDoubleu) wants to merge 4 commits into
microsoft:mainfrom
JayDoubleu:fix/nexus-3.97.2-telemetry

Conversation

@JayDoubleu

Copy link
Copy Markdown
Collaborator

Resolves #5276 and part of #5271

This PR updates the Nexus shared service from Nexus 3.94.0 to Nexus 3.97.2. The Nexus bundle version changes from 3.11.2 to 4.0.0.

What is being addressed

The JVM in Nexus 3.94.0 trusts certificates from 8 eMudhra intermediate CAs. The private keys of these CAs are compromised (Sonatype advisory). The CA generated the keys incorrectly, and anyone can recover them from the public certificates (incident page, Mozilla bug 2083449). The Docker images of Nexus 3.97.2 remove the trust in these CAs.

In short, the certificate part of an attack is easy, but the network part is difficult. An attacker cannot do this attack only by sending requests to Nexus or to the TRE from the internet.

What an attacker needs

To change the packages that Nexus serves, an attacker needs both of these:

  1. A false certificate for an upstream host, for example pypi.org. This part is easy, because anyone can recover the compromised keys.
  2. A position on the network path between the Nexus VM and that upstream. In a default TRE, this path goes from the Nexus VM through the Azure Firewall to the internet.

The second part needs control of network infrastructure, not only access to the internet. Examples:

  • Control of a network between Azure and the upstream.
  • False internet routes (a BGP hijack) for the addresses of the upstream.
  • A false DNS answer for the upstream host.
  • If the TRE uses forced tunnelling, control of the network of the operator, because the traffic of Nexus goes through it.

What an attacker can do with both parts

  • Send changed packages or new false package versions to Nexus. Nexus caches this content and serves it to all workspaces.
  • Run code in each workspace VM that installs a false package. The code runs with the permissions of the user or the process that installs the package.
  • See which packages Nexus requests from the upstream.

This applies to the 21 Nexus proxies that use an HTTPS upstream, for example PyPI, conda, CRAN and Docker Hub. It also applies to the proxies that send the apt GPG keys to workspace VMs, so the attacker can also replace these keys. Nexus asks the upstream again for cached content after 24 hours (contentMaxAge 1440), so the cache does not stop the attack.

What an attacker cannot do

  • The attacker cannot connect to Nexus or to the TRE directly. The only way into the TRE is through a false package that a workspace installs (see the list above).
  • The attacker cannot read data in the TRE without such a package.
  • A workspace user cannot do this attack from a workspace. A workspace has no access to the network path between Nexus and the upstreams.
  • The connection from workspaces to Nexus is not affected. It uses the TRE certificate for Nexus, not an eMudhra certificate.

The incident page says that it is not known if attackers used these keys.

Why Nexus is affected but browsers are not

Browsers enforce Certificate Transparency (CT), a public log of trusted certificates. The incident page says that browsers that enforce CT were not at risk. The JVM in Nexus does not enforce CT.

I ran a TLS test with the JVM and the default trust store of each Nexus image. The test connected to the 14 test hosts of the incident page:

Hosts Nexus 3.94.0 JVM Nexus 3.97.2 JVM
pypi.org and files.pythonhosted.org Connects Connects
6 hosts with a direct eMudhra chain, for example dv-g3b-emudhra.we-have-a-problem.com Connects Rejects
dv-g3b-noct-emudhra.we-have-a-problem.com, with a certificate that is not in a CT log, like a certificate that an attacker makes in secret Connects Rejects
7 hosts with the Sectigo cross-sign, for example dv-g3b-aaa-emudhra.we-have-a-problem.com Rejects Rejects

The 3.94.0 JVM connected although the CA revoked the intermediates.

The packages that Nexus serves are a separate issue

This PR changes only the trust store of Nexus. It does not change the packages that Nexus serves. Many packages contain their own list of trusted CAs. Examples are certifi and pip on PyPI, the ca-certificates packages of Ubuntu and conda, Java and Node.js runtimes, VS Code and Docker images.

These packages continue to trust the compromised CAs until these two things happen:

  1. The maintainers of the package release a fixed version. Many maintainers did this already, for example for certifi 2026.10.19 and pip 26.2.2. Some fixes are still pending (list of fixed versions).
  2. The workspaces install the fixed version.

Nexus continues to serve the old versions. For example, a project that pins an old version of certifi continues to get the old list of trusted CAs. This risk applies to the connections that software in a workspace makes. It does not apply to the connection between Nexus and its upstreams.

Required base telemetry

Nexus 3.97.0 and later send required base telemetry to nxrm.telemetry.sonatype.com. An administrator must acknowledge this requirement. If the uploads fail for 10 days, Nexus Community Edition becomes read-only. In read-only mode, Nexus serves only the packages that are already in its cache.

How is this addressed

  • porter.yaml: change the image tag from 3.94.0 to 3.97.2 and the bundle version from 3.11.2 to 4.0.0.
  • terraform/locals.tf: add nxrm.telemetry.sonatype.com to nexus_allowed_fqdns. The nexus-package-sources firewall rule allows this list only from the private IP address of the Nexus VM.
  • scripts/configure_nexus_repos.sh: after the EULA step, send POST /service/rest/v1/system/mandatory-telemetry-acknowledgement with {"acknowledgement":true}. The script retries with backoff. If Nexus does not return HTTP 204, the script stops before it configures the repositories, and the deployment fails.
  • template_schema.json: add a link to the Sonatype telemetry page in the help text of accept_nexus_eula.
  • devops/tests/test_nexus_repos.py: add 3 tests for the acknowledgement. The tests cover success, a retry after temporary failures, and a stop after persistent failures.
  • docs/tre-templates/shared-services/nexus.md:
    • Add an "Upgrade to Nexus bundle 4.0.0" section. A warning box says that the upgrade is one-way. The section lists the steps to do before the upgrade. Then it tells operators to upgrade through the API with force_version_update=true.
    • Add the firewall host and a "Required base telemetry" section. The section tells operators about TLS inspection, forced tunnelling, the read-only mode and the status check. It also says when Nexus uploads the telemetry.
  • CHANGELOG.md: add a BREAKING CHANGES entry that links to the upgrade section, and a BUG FIXES entry.

The bundle acknowledges the telemetry automatically. The operator accepts the Nexus EULA with accept_nexus_eula, and the help text of that property now links to the telemetry page. Sonatype gives no option to turn off this telemetry in Community Edition.

The bundle version is a major version. The versioning guide asks for a major version for changes that do not upgrade automatically, or that need someone to take action first. This change has these properties:

  • The upgrade is one-way. Nexus 3.94.0 does not start on a database that Nexus 3.97.2 upgraded. It fails with "Missing migrations".
  • The deployment can need egress changes outside the TRE. If the telemetry uploads fail for 10 days, Nexus becomes read-only.

The API blocks a major upgrade unless the request sets force_version_update=true, and the UI does not offer it. So an operator cannot start this one-way upgrade from the UI by accident.

Known issue that this PR does not fix

If an upgrade fails and the operator tries again at the same template version, Terraform does not run azurerm_virtual_machine_run_command.configure_nexus_repos again. The new image can then stay missing without an error. This behaviour exists since #4983. A separate issue can fix it.

Test results

Local tests

Test Result
python3 -m unittest discover -s devops/tests 448 tests pass, 4 PowerShell tests skipped
pre-commit, all hooks Pass
mkdocs build --strict Pass
Nexus 3.94.0 to 3.97.2 on the same data volume (Podman) The cached package and all repositories stay
Acknowledgement API on 3.97.2 GET returns {"acknowledgement": false} or true. POST returns 204 and can run again. POST with false returns 400.
Nexus 3.94.0 on the upgraded database Does not start ("Missing migrations")
New installation of Nexus 3.97.2 with the real configure_nexus_repos.sh (Podman) The script accepts the EULA, acknowledges the telemetry (HTTP 204) and creates the repositories

Live upgrade test in a CI environment in another fork

  1. A branch deployment of main (a5252e7b) passed.
  2. I created the certs shared service and Nexus bundle 3.11.2 through the API. Then I downloaded the six-1.17.0 wheel through the PyPI proxy of Nexus.
  3. A second branch deployment registered the Nexus bundle from the first commit of this PR. At that time, the bundle version was 3.12.0. Only the publish job of the azureml bundle failed, at the Azure login step (JSON is invalid). This PR does not change that bundle.
  4. I upgraded Nexus with a PATCH request through the API. The upgrade took 6 minutes.
  5. The linux_vm E2E test passed against the upgraded Nexus.

The later commits change the docs and CHANGELOG.md. The last commit also changes the bundle version from 3.12.0 to 4.0.0. The bundle content is otherwise the same as in the test. The same commits passed lint, CodeQL and the Nexus bundle build in another fork.

Check Before the upgrade (bundle 3.11.2) After the upgrade
Server header Nexus/3.94.0-12 (COMMUNITY) Nexus/3.97.2-02 (COMMUNITY)
Acknowledgement API HTTP 404 {"acknowledgement":true}
Repositories 23 23
Cached six wheel Present Present
HTTPS on port 443 with a certificate check HTTP 200 HTTP 200
Telemetry host in the nexus-package-sources firewall rule No Yes
Telemetry upload The firewall denied it "Telemetry upload successfully with valid signature"
Linux VM in a workspace Not tested 795 requests to Nexus, all HTTP 200 or 304

Other results:

  • The script acknowledged the telemetry 10 seconds after Nexus 3.97.2 started.
  • Nexus 3.97.2 uploads the telemetry every 24 hours and when it stops. After the upgrade, the status check showed "Last successful upload: never" until I restarted the container.
  • After the upgrade, nxrm.telemetry.sonatype.com is the only new outbound host. Nexus 3.94.0 and 3.97.2 also call rhc.sonatype.com for the Repository Health Check. The firewall denied this host before and after the upgrade, and Nexus works without it.

Not tested in Azure:

  • A new installation of the Nexus bundle. The same script runs for a new installation and for an upgrade, and the Podman test of a new installation passed.
  • The upgrade request with force_version_update=true. The test used the version 3.12.0, which did not need this parameter.
  • A TRE with forced tunnelling or with a proxy that inspects TLS.

Nexus 3.97.2 stops trusting the eMudhra certificate chains that
Sonatype reports as compromised. Nexus 3.97 and later must upload
telemetry to nxrm.telemetry.sonatype.com. If uploads fail for 10 days,
Nexus Community Edition becomes read-only.

- Update the Nexus image from 3.94.0 to 3.97.2.
- Acknowledge the required base telemetry through the REST API, with
  retries and time limits. Stop the deployment if it fails.
- Allow nxrm.telemetry.sonatype.com in the Nexus firewall rule.
- Test the acknowledgement in the Nexus script unit tests.
- Document the telemetry requirement, the egress rule, the read-only
  effect, and that the upgrade is one-way.

Fixes microsoft#5276
Part of microsoft#5271
A live upgrade test showed that the Baseline Telemetry check shows
"Last successful upload: never" after the upgrade. Nexus 3.97.2 uploads
every 24 hours and when it stops, so the first upload can take up to
24 hours.

- Tell administrators that this status is normal after a deployment or
  an upgrade.
- Say that the Telemetry upload retry task sends the failed uploads
  again, which is what the task does.
The one-way upgrade was a sentence at the end of the telemetry section.
Operators can miss it, and the changelog could not link to it.

- Add an "Upgrade to Nexus bundle 3.12.0" section with a warning that
  the upgrade is one-way, and the steps to do before the upgrade.
- Link the breaking change entry in the changelog to this section.
The repository versioning guide asks for a major version for changes
that do not upgrade automatically, or that need someone to take action
first. The Nexus 3.97.2 upgrade is one-way and can need egress changes
outside the TRE. If telemetry uploads fail for 10 days, Nexus becomes
read-only.

The API blocks a major upgrade unless the request sets
force_version_update=true, and the UI does not offer it. This makes
operators read the upgrade steps before they upgrade.

- Change the bundle version from 3.12.0 to 4.0.0.
- Rename the docs section and its anchor, and update the changelog.
- Tell operators to upgrade through the API with
  force_version_update=true after the steps before the upgrade.
@github-actions

Copy link
Copy Markdown

Unit Test Results

0 tests   0 ✅  0s ⏱️
0 suites  0 💤
0 files    0 ❌

Results for commit d1a387c.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

The irreversible database migration and mandatory telemetry dependency warrant human sign-off, with the final forced-major upgrade path still unverified in Azure.

No open issues 🎉

What changed in this PR

Updates the Azure TRE Nexus shared service to 3.97.2 to remove trust in compromised eMudhra certificate chains and support mandatory Sonatype telemetry.

Changes:

  • Bumps the Nexus bundle to 4.0.0 and documents the irreversible upgrade.
  • Adds telemetry egress access and acknowledgement with retries and failure handling.
  • Adds acknowledgement tests, operator guidance and release notes.
File Description
templates/​shared_services/​sonatype-nexus-vm/​terraform/​locals.tf Adds the required telemetry host to the Nexus egress allowlist.
templates/​shared_services/​sonatype-nexus-vm/​template_schema.json Adds telemetry information to EULA help text.
templates/​shared_services/​sonatype-nexus-vm/​scripts/​configure_nexus_repos.sh Acknowledges telemetry before configuring repositories.
templates/​shared_services/​sonatype-nexus-vm/​porter.yaml Updates the Nexus image and major bundle version.
docs/​tre-templates/​shared-services/​nexus.md Documents upgrade precautions, telemetry requirements and recovery.
devops/​tests/​test_nexus_repos.py Tests acknowledgement success, retries and persistent failures.
CHANGELOG.md Records the security update and breaking upgrade requirements.

Review effort: Balanced

@JayDoubleu

Copy link
Copy Markdown
Collaborator Author

/test-shared-services d1a387c accept_nexus_eula

@github-actions

Copy link
Copy Markdown

🤖 pr-bot 🤖

🏃 Running shared service tests: https://github.com/microsoft/AzureTRE/actions/runs/40027819217

(in response to this comment from Jay W (@JayDoubleu))

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Nexus needs updating to latest 3.97.2

2 participants