From 0b47c4e56cfe4e6177c65c698e66059b2da64972 Mon Sep 17 00:00:00 2001 From: Robin Andersson Date: Mon, 9 Nov 2026 15:16:23 +0100 Subject: [PATCH 1/6] [HWORKS-3747] Upgrade note: Prometheus scrape jobs follow the Consul domain From 5.3 Prometheus, the Grafana data source, the Prometheus adapter and the VPA recommender reach the services in global._hopsworks.consulDomainName. The pushgateway and minio-* jobs move out of serverFiles, so values that restate scrape_configs with one of them must drop it before upgrading. Co-Authored-By: Claude Opus 5.5 --- docs/setup_installation/admin/upgrade.md | 19 +++++++++++++++++++ 1 file changed, 19 insertions(+) diff --git a/docs/setup_installation/admin/upgrade.md b/docs/setup_installation/admin/upgrade.md index e0124cc363..037c1d76b5 100644 --- a/docs/setup_installation/admin/upgrade.md +++ b/docs/setup_installation/admin/upgrade.md @@ -305,6 +305,25 @@ Pipelines, outputs and plugins you added through `logstash.extraPipelines`, `log - **The kafka integration, the `azure_event_hubs` input and the `webhdfs` output are no longer in the image.** A pipeline that uses one of them needs it in `logstash.plugins`, which installs it when the Logstash pod starts, so it needs access to `rubygems.org` and cannot run on an airgapped cluster. +### Prometheus scrape jobs and the Consul domain { #upgrade-prometheus-consul-domain } + +From 5.3, Prometheus, the Prometheus data source of Grafana, the Prometheus adapter and the VPA recommender reach the Hopsworks services in `global._hopsworks.consulDomainName`, so an installation with its own Consul domain needs no Prometheus values to point them there. +With the default domain, `consul`, nothing changes. + +The `pushgateway`, `minio-job`, `minio-bucket`, `minio-resource` and `minio-node` scrape jobs move out of `prometheus.prometheus.serverFiles."prometheus.yml".scrape_configs` into a file Prometheus loads next to it, because only that file can follow the Consul domain. +Values that restate `scrape_configs` replace the chart's list, so a restated list that still holds one of these jobs gives Prometheus the job twice, and Prometheus refuses its configuration with `found multiple scrape configs with job name`. +Before upgrading, list the jobs your values restate: + +```bash +helm get values -n -o yaml | yq '.prometheus.prometheus.serverFiles."prometheus.yml".scrape_configs[].job_name' +``` + +If the list exists only to change the domain of these targets, remove it from your values, which also brings back every job the chart has added since. +Otherwise remove these five jobs from it. + +The `trino` job now reads its user from the `trino-monitoring-credentials` Secret, so a changed `trino.auth.monitoringUser` no longer needs the list restated. +Overrides that only moved `prometheus.prometheus-adapter.prometheus.url`, `prometheus.prometheus-adapter.dnsConfig` or the Grafana data source to the installation's Consul domain are no longer needed either. + ### Agent deployments reserve 2 GiB for their OTLP sidecar { #upgrade-agent-otlp-sidecar } An agent deployment with tracing enabled runs a `hops-otlp` sidecar next to the agent. From 369c943aa873dcf4e4327895c3b8f4b345cce84b Mon Sep 17 00:00:00 2001 From: Robin Andersson Date: Mon, 9 Nov 2026 21:49:08 +0100 Subject: [PATCH 2/6] [HWORKS-3747] Explain retained monitoring jobs during upgrades --- docs/setup_installation/admin/upgrade.md | 20 +++++++++++++------- 1 file changed, 13 insertions(+), 7 deletions(-) diff --git a/docs/setup_installation/admin/upgrade.md b/docs/setup_installation/admin/upgrade.md index 037c1d76b5..3dcf70163d 100644 --- a/docs/setup_installation/admin/upgrade.md +++ b/docs/setup_installation/admin/upgrade.md @@ -311,15 +311,21 @@ From 5.3, Prometheus, the Prometheus data source of Grafana, the Prometheus adap With the default domain, `consul`, nothing changes. The `pushgateway`, `minio-job`, `minio-bucket`, `minio-resource` and `minio-node` scrape jobs move out of `prometheus.prometheus.serverFiles."prometheus.yml".scrape_configs` into a file Prometheus loads next to it, because only that file can follow the Consul domain. -Values that restate `scrape_configs` replace the chart's list, so a restated list that still holds one of these jobs gives Prometheus the job twice, and Prometheus refuses its configuration with `found multiple scrape configs with job name`. -Before upgrading, list the jobs your values restate: +A job of the same name in `scrape_configs` takes precedence and keeps its own targets and settings, because Prometheus refuses two jobs with one name. +`scrape_configs` still holds these jobs in two cases: -```bash -helm get values -n -o yaml | yq '.prometheus.prometheus.serverFiles."prometheus.yml".scrape_configs[].job_name' -``` +- **Your values restate the list.** + Helm replaces the chart's list with yours, including any of these jobs you customized. + List the jobs your values restate: + + ```bash + helm get values -n -o yaml | yq '.prometheus.prometheus.serverFiles."prometheus.yml".scrape_configs[].job_name' + ``` -If the list exists only to change the domain of these targets, remove it from your values, which also brings back every job the chart has added since. -Otherwise remove these five jobs from it. + If the list exists only to point these targets at your Consul domain, remove it from your values, which also brings back every job the chart has added since. +- **You upgrade with `helm upgrade --reuse-values`.** + It keeps the chart defaults of the release it upgrades, including the previous list with these jobs, so they keep their previous targets. + Upgrade with `--reset-then-reuse-values` instead, which takes the new chart's defaults and keeps the values you supplied. The `trino` job now reads its user from the `trino-monitoring-credentials` Secret, so a changed `trino.auth.monitoringUser` no longer needs the list restated. Overrides that only moved `prometheus.prometheus-adapter.prometheus.url`, `prometheus.prometheus-adapter.dnsConfig` or the Grafana data source to the installation's Consul domain are no longer needed either. From c6aafff37792322119beec6df0d8820a256168c4 Mon Sep 17 00:00:00 2001 From: Robin Andersson Date: Wed, 11 Nov 2026 10:06:55 +0100 Subject: [PATCH 3/6] [HWORKS-3747] Shorten the Prometheus upgrade note Co-Authored-By: Claude Opus 5.5 --- docs/setup_installation/admin/upgrade.md | 28 ++++++++---------------- 1 file changed, 9 insertions(+), 19 deletions(-) diff --git a/docs/setup_installation/admin/upgrade.md b/docs/setup_installation/admin/upgrade.md index 3dcf70163d..b228f210ff 100644 --- a/docs/setup_installation/admin/upgrade.md +++ b/docs/setup_installation/admin/upgrade.md @@ -305,30 +305,20 @@ Pipelines, outputs and plugins you added through `logstash.extraPipelines`, `log - **The kafka integration, the `azure_event_hubs` input and the `webhdfs` output are no longer in the image.** A pipeline that uses one of them needs it in `logstash.plugins`, which installs it when the Logstash pod starts, so it needs access to `rubygems.org` and cannot run on an airgapped cluster. -### Prometheus scrape jobs and the Consul domain { #upgrade-prometheus-consul-domain } +### Prometheus follows the Consul domain { #upgrade-prometheus-consul-domain } -From 5.3, Prometheus, the Prometheus data source of Grafana, the Prometheus adapter and the VPA recommender reach the Hopsworks services in `global._hopsworks.consulDomainName`, so an installation with its own Consul domain needs no Prometheus values to point them there. +From 5.3, Prometheus, the Grafana data source, the Prometheus adapter and the VPA recommender use `global._hopsworks.consulDomainName`. With the default domain, `consul`, nothing changes. -The `pushgateway`, `minio-job`, `minio-bucket`, `minio-resource` and `minio-node` scrape jobs move out of `prometheus.prometheus.serverFiles."prometheus.yml".scrape_configs` into a file Prometheus loads next to it, because only that file can follow the Consul domain. -A job of the same name in `scrape_configs` takes precedence and keeps its own targets and settings, because Prometheus refuses two jobs with one name. -`scrape_configs` still holds these jobs in two cases: +Values that only point these components at your Consul domain are no longer needed, and you can remove them: -- **Your values restate the list.** - Helm replaces the chart's list with yours, including any of these jobs you customized. - List the jobs your values restate: +- `prometheus.prometheus.serverFiles."prometheus.yml".scrape_configs`. + A restated list keeps its own `pushgateway` and `minio-*` targets and lacks the jobs the chart has added since. + The same holds for a list restated for a changed `trino.auth.monitoringUser`, because the `trino` job now reads the user from the `trino-monitoring-credentials` Secret. +- `prometheus.prometheus-adapter.prometheus.url` and `prometheus.prometheus-adapter.dnsConfig`. +- The Prometheus data source in `grafana.grafana.datasources`. - ```bash - helm get values -n -o yaml | yq '.prometheus.prometheus.serverFiles."prometheus.yml".scrape_configs[].job_name' - ``` - - If the list exists only to point these targets at your Consul domain, remove it from your values, which also brings back every job the chart has added since. -- **You upgrade with `helm upgrade --reuse-values`.** - It keeps the chart defaults of the release it upgrades, including the previous list with these jobs, so they keep their previous targets. - Upgrade with `--reset-then-reuse-values` instead, which takes the new chart's defaults and keeps the values you supplied. - -The `trino` job now reads its user from the `trino-monitoring-credentials` Secret, so a changed `trino.auth.monitoringUser` no longer needs the list restated. -Overrides that only moved `prometheus.prometheus-adapter.prometheus.url`, `prometheus.prometheus-adapter.dnsConfig` or the Grafana data source to the installation's Consul domain are no longer needed either. +Upgrade with `--reset-then-reuse-values`, not `--reuse-values`, which keeps the previous chart's scrape jobs and their old targets. ### Agent deployments reserve 2 GiB for their OTLP sidecar { #upgrade-agent-otlp-sidecar } From 32e4f43ab4dfd8479028c56b0d8f8fc5dcae8fab Mon Sep 17 00:00:00 2001 From: Robin Andersson Date: Wed, 11 Nov 2026 10:23:40 +0100 Subject: [PATCH 4/6] [HWORKS-3747] Upgrade note: list the values to remove Co-Authored-By: Claude Opus 5.5 --- docs/setup_installation/admin/upgrade.md | 13 +++++-------- 1 file changed, 5 insertions(+), 8 deletions(-) diff --git a/docs/setup_installation/admin/upgrade.md b/docs/setup_installation/admin/upgrade.md index b228f210ff..f4ae49afc1 100644 --- a/docs/setup_installation/admin/upgrade.md +++ b/docs/setup_installation/admin/upgrade.md @@ -308,17 +308,14 @@ Pipelines, outputs and plugins you added through `logstash.extraPipelines`, `log ### Prometheus follows the Consul domain { #upgrade-prometheus-consul-domain } From 5.3, Prometheus, the Grafana data source, the Prometheus adapter and the VPA recommender use `global._hopsworks.consulDomainName`. -With the default domain, `consul`, nothing changes. +If your values set any of the following only to point these components at your Consul domain, or only for a changed `trino.auth.monitoringUser`, remove them before the upgrade: -Values that only point these components at your Consul domain are no longer needed, and you can remove them: - -- `prometheus.prometheus.serverFiles."prometheus.yml".scrape_configs`. - A restated list keeps its own `pushgateway` and `minio-*` targets and lacks the jobs the chart has added since. - The same holds for a list restated for a changed `trino.auth.monitoringUser`, because the `trino` job now reads the user from the `trino-monitoring-credentials` Secret. -- `prometheus.prometheus-adapter.prometheus.url` and `prometheus.prometheus-adapter.dnsConfig`. +- `prometheus.prometheus.serverFiles."prometheus.yml".scrape_configs`, because it replaces the chart's list and so misses the scrape jobs the chart adds. +- `prometheus.prometheus-adapter.prometheus.url`. +- `prometheus.prometheus-adapter.dnsConfig`. - The Prometheus data source in `grafana.grafana.datasources`. -Upgrade with `--reset-then-reuse-values`, not `--reuse-values`, which keeps the previous chart's scrape jobs and their old targets. +Upgrade with `--reset-then-reuse-values`, not `--reuse-values`, which keeps the previous chart's scrape jobs and their `service.consul` targets. ### Agent deployments reserve 2 GiB for their OTLP sidecar { #upgrade-agent-otlp-sidecar } From d13e16b20b23a968590c1e45a6172aac8dc6f078 Mon Sep 17 00:00:00 2001 From: Robin Andersson Date: Wed, 11 Nov 2026 10:24:58 +0100 Subject: [PATCH 5/6] [HWORKS-3747] Upgrade note: removed values need an upgrade without reused values Co-Authored-By: Claude Opus 5.5 --- docs/setup_installation/admin/upgrade.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/setup_installation/admin/upgrade.md b/docs/setup_installation/admin/upgrade.md index f4ae49afc1..d736edb89b 100644 --- a/docs/setup_installation/admin/upgrade.md +++ b/docs/setup_installation/admin/upgrade.md @@ -315,7 +315,7 @@ If your values set any of the following only to point these components at your C - `prometheus.prometheus-adapter.dnsConfig`. - The Prometheus data source in `grafana.grafana.datasources`. -Upgrade with `--reset-then-reuse-values`, not `--reuse-values`, which keeps the previous chart's scrape jobs and their `service.consul` targets. +Upgrade with your values files and without `--reuse-values` or `--reset-then-reuse-values`, because both carry the removed values over from the previous release. ### Agent deployments reserve 2 GiB for their OTLP sidecar { #upgrade-agent-otlp-sidecar } From 0bd87735e0f96dde63b9b26b5d09dcfbaedc020e Mon Sep 17 00:00:00 2001 From: Robin Andersson Date: Wed, 11 Nov 2026 10:31:05 +0100 Subject: [PATCH 6/6] [HWORKS-3747] Upgrade note: drop the upgrade flag line Co-Authored-By: Claude Opus 5.5 --- docs/setup_installation/admin/upgrade.md | 2 -- 1 file changed, 2 deletions(-) diff --git a/docs/setup_installation/admin/upgrade.md b/docs/setup_installation/admin/upgrade.md index d736edb89b..54112c73cc 100644 --- a/docs/setup_installation/admin/upgrade.md +++ b/docs/setup_installation/admin/upgrade.md @@ -315,8 +315,6 @@ If your values set any of the following only to point these components at your C - `prometheus.prometheus-adapter.dnsConfig`. - The Prometheus data source in `grafana.grafana.datasources`. -Upgrade with your values files and without `--reuse-values` or `--reset-then-reuse-values`, because both carry the removed values over from the previous release. - ### Agent deployments reserve 2 GiB for their OTLP sidecar { #upgrade-agent-otlp-sidecar } An agent deployment with tracing enabled runs a `hops-otlp` sidecar next to the agent.