Optimize Prometheus reporter snapshot updates by jt2594838 · Pull Request #18562 · apache/iotdb

jt2594838 · 2026-09-01T09:36:37Z

Summary

refresh Prometheus metric snapshots asynchronously every 15 seconds and serve the cached snapshot from HTTP requests
add the default-enabled prometheus_reporter_async_update switch with synchronous fallback
use IoTDBThreadPoolFactory and a named Prometheus snapshot updater thread only when async updates are enabled

Validation

mvn spotless:apply -pl iotdb-core/node-commons
mvn compile -pl iotdb-core/node-commons -DskipTests
mvn test -pl iotdb-core/metrics/interface -Dtest=MetricConfigTest
mvn test-compile -pl iotdb-core/metrics/interface -P with-zh-locale -DskipTests
git diff --check

jt2594838 · 2026-09-01T09:40:04Z

  private Integer prometheusReporterPort = 9091;

+  /** Whether Prometheus metrics are collected asynchronously into a cached snapshot. */
+  private boolean prometheusReporterAsyncUpdate = true;


Adds the default-on feature flag to the shared metric configuration so async snapshot behavior is explicit, reloadable, and included in configuration equality and hash state.

jt2594838 · 2026-09-01T09:40:06Z

                properties,
                isConfigNode)));

+    loadConfig.setPrometheusReporterAsyncUpdate(


Loads the new switch while accepting the global and legacy metric-prefixed forms, preserving compatibility with existing node configuration conventions.

jt2594838 · 2026-09-01T09:40:07Z

        serverTransport = serverTransport.secure(spec -> spec.sslContext(sslContext));
      }
      httpServer = serverTransport.bindNow();
+      if (METRIC_CONFIG.isPrometheusReporterAsyncUpdate()) {


Starts the periodic snapshot task only for async mode; synchronous mode continues to scrape on demand, while the cached response keeps Reactor request threads independent of expensive metric collection.

jt2594838 · 2026-09-01T09:40:09Z

  @Override
-  public boolean stop() {
+  public synchronized boolean stop() {
+    stopSnapshotUpdater();


Stops and disposes the snapshot task with the reporter so reloads, shutdown, and failed starts do not leave a scheduler behind.

jt2594838 · 2026-09-01T09:40:10Z

          break;
        case PROMETHEUS:
-          reporter = new PrometheusReporter(metricManager);
+          if (METRIC_CONFIG.isPrometheusReporterAsyncUpdate()) {


Creates the IoTDB-managed scheduler only when async snapshot updates are enabled; the disabled path now constructs the synchronous reporter without allocating a thread pool.

jt2594838 · 2026-09-01T09:40:12Z

  // -------------------------- Metrics --------------------------
  SYSTEM_SCHEDULE_METRICS("SystemScheduleMetrics"),
  RESOURCE_CONTROL_DISK_STATISTIC("ResourceControl-DataRegionDiskStatistics"),
+  PROMETHEUS_REPORTER_SNAPSHOT_UPDATER("PrometheusReporter-Snapshot-Updater"),


Registers a stable thread name for the snapshot updater and includes it in the metrics thread classification so the new background work remains observable.

jt2594838 · 2026-09-01T09:40:14Z

    assertEquals(MetricLevel.ALL, metricConfig.getMetricLevel());
    assertEquals(10, (int) metricConfig.getAsyncCollectPeriodInSecond());
    assertEquals(9090, (int) metricConfig.getPrometheusReporterPort());
+    assertEquals(false, metricConfig.isPrometheusReporterAsyncUpdate());


Covers parsing the explicitly disabled setting; the companion DataNode assertion verifies the enabled value and protects the default/alias behavior.

jt2594838 · 2026-09-01T09:40:15Z

+# The snapshot is refreshed every 15 seconds, matching Prometheus's default scrape interval.
+# effectiveMode: restart
+# Datatype: boolean
+prometheus_reporter_async_update=true


Documents the default-enabled switch, its 15-second refresh cadence, and restart semantics in the shipped configuration template.

jt2594838 · 2026-09-01T09:40:17Z

 | `dn(cn)_metric_level` | Initial metric level. | `OFF`, `CORE`, `IMPORTANT`, `NORMAL`, `ALL` |
 | `cn_metric_prometheus_reporter_port` | Prometheus HTTP port for ConfigNode. | `9091` |
 | `dn_metric_prometheus_reporter_port` | Prometheus HTTP port for DataNode. | `9092` |
+| `prometheus_reporter_async_update` | Serve a cached Prometheus snapshot refreshed every 15 seconds. | `true` |


Exposes the new operator-facing setting in the metrics documentation and records the default behavior.

jt2594838 · 2026-09-01T09:40:18Z

  public static final String LOG_IOTDBSESSIONREPORTER_START_WRITE_ARG_ARG_E79CDDAE = "IoTDBSessionReporter start, write to {}:{}";
  public static final String LOG_PROMETHEUSREPORTER_STARTED_USE_PORT_ARG_A688FFC8 = "PrometheusReporter started, use port {}";
  public static final String LOG_DETECTED_ERROR_TAKING_METRIC_TIMER_SNAPSHOT_WILL_DISCARD_METRIC_B7154169 = "Detected an error when taking metric timer snapshot, will discard this metric";
+  public static final String LOG_PROMETHEUSREPORTER_FAILED_TO_UPDATE_METRICS_SNAPSHOT_ASYNCHRONOUSLY_F19FE4E3 = "PrometheusReporter failed to update metrics snapshot asynchronously";


Adds the asynchronous snapshot failure message to the i18n catalog so updater errors use the project localization mechanism.

jt2594838 · 2026-09-01T09:40:20Z

  public static final String LOG_IOTDBSESSIONREPORTER_START_WRITE_ARG_ARG_E79CDDAE = "IoTDBSessionReporter 启动，写入 {}:{}";
  public static final String LOG_PROMETHEUSREPORTER_STARTED_USE_PORT_ARG_A688FFC8 = "PrometheusReporter 已启动，使用端口 {}";
  public static final String LOG_DETECTED_ERROR_TAKING_METRIC_TIMER_SNAPSHOT_WILL_DISCARD_METRIC_B7154169 = "获取 metric timer 快照时检测到错误，将丢弃该 metric";
+  public static final String LOG_PROMETHEUSREPORTER_FAILED_TO_UPDATE_METRICS_SNAPSHOT_ASYNCHRONOUSLY_F19FE4E3 = "PrometheusReporter 异步更新监控项快照失败";


Keeps the Chinese i18n catalog in key parity with the English catalog for the new updater failure message.

jt2594838 · 2026-09-02T03:25:15Z

                            }
+                            String metrics =
+                                METRIC_CONFIG.isPrometheusReporterAsyncUpdate()
+                                    ? getMetricsSnapshot()


The cache now uses a null value as an explicit uninitialized state, and this request path synchronously scrapes on the first request. This closes the startup window where AbstractMetricService has started the reporter but has not yet bound metric sets, which previously returned HTTP 200 with an empty body. The fallback is covered by IoTDBMetricIT.testPrometheusReporterWithoutAuth.

jt2594838 · 2026-09-02T03:27:34Z

+    }
+    // Delay the first background scrape until metric sets have been bound by the metric service.
+    snapshotUpdateFuture =
+        snapshotUpdateExecutor.scheduleAtFixedRate(


The first scheduled scrape is delayed until the normal 15-second refresh interval. MetricService starts reporters before AbstractMetricService binds metric sets, so scheduling at delay 0 creates the empty-snapshot race reported by CI. The delayed initial run preserves the existing cadence while the request fallback above covers requests arriving sooner.

jt2594838 · 2026-09-02T03:29:11Z

+      // Do not publish an empty scrape taken before metric sets are bound. The request path will
+      // synchronously scrape until the first complete snapshot is available. Empty snapshots are
+      // published after initialization so removed metrics are not kept in the cache indefinitely.
+      if (!snapshot.isEmpty() || metricsSnapshot != null) {


The updater ignores only the pre-initialization empty scrape, identified by a still-null cache. Once any snapshot has been published, an empty result is retained so metrics removed at runtime are reflected instead of leaving stale series in the cached response. The asynchronous path remains off Reactor request threads.

HTHou

Re-reviewed the latest revision. The initial snapshot race is addressed by the explicit uninitialized state, first-request fallback, and delayed background refresh. I found no remaining blocking issues.

Caideyipi · 2026-09-02T04:11:00Z

+      snapshotUpdateFuture = null;
+    }
+    if (snapshotUpdateExecutor != null) {
+      snapshotUpdateExecutor.shutdownNow();


[P2] Recreate the managed scheduler when the reporter restarts

stopSnapshotUpdater() always shuts down and clears the executor supplied by MetricService. If this reporter is started again through the public AbstractMetricService.stop/start(ReporterType) APIs, or retried after start() failed, startSnapshotUpdater() falls back to a raw daemon executor. That replacement is not registered with JMX or ThreadPoolMetrics, and its lowercase name does not match ThreadName.PROMETHEUS_REPORTER_SNAPSHOT_UPDATER, so CPU metrics classify it as UNKNOWN. Please retain an executor supplier/factory and create a new IoTDB-managed executor on every start (or recreate the reporter) rather than losing the managed executor after the first stop or failed start.

Caideyipi · 2026-09-02T04:11:02Z

+            reporter =
+                new PrometheusReporter(
+                    metricManager,
+                    IoTDBThreadPoolFactory.newSingleThreadScheduledExecutor(


[P3] Avoid registering the scheduler before duplicate reporters are rejected

setMetricReporterList() preserves duplicates, while CompositeReporter.addReporter() rejects them only after this constructor has run. With PROMETHEUS,PROMETHEUS, the second IoTDBThreadPoolFactory executor registers under the same pool name and overwrites the first entry in ThreadPoolMetrics; the second reporter is then discarded. The active updater's thread_pool_* gauges therefore point at the never-started executor. Please deduplicate before constructing reporters, make addReporter() report acceptance, or allocate the scheduler lazily in start().

jt2594838 · 2026-09-02T08:42:04Z

+    if (snapshotUpdateExecutor == null || snapshotUpdateExecutor.isShutdown()) {
+      // Create a fresh executor for every start so a stopped reporter can be started again with
+      // the same managed thread-pool factory.
+      snapshotUpdateExecutor = Objects.requireNonNull(snapshotUpdateExecutorSupplier.get());


Caideyipi P2 fixed: the reporter now stores a scheduler factory and invokes it on every start, so stop/start and failed-start retries receive a fresh executor. The service supplies an IoTDB-managed factory, preserving JMX/ThreadPoolMetrics registration and the Prometheus thread classification across restarts; the standalone constructor retains its daemon executor fallback.

jt2594838 · 2026-09-02T08:43:01Z

        case PROMETHEUS:
-          reporter = new PrometheusReporter(metricManager);
+          if (METRIC_CONFIG.isPrometheusReporterAsyncUpdate()) {
+            // Defer pool creation until start so duplicate reporters rejected below do not


Caideyipi P3 fixed: pool creation is now deferred into the reporter start lifecycle. If duplicate PROMETHEUS entries are rejected by CompositeReporter, no IoTDB thread pool has been created or registered for the discarded reporter, so thread-pool gauges continue to describe the active updater.

Optimize Prometheus reporter snapshot updates

650e9ae

jt2594838 commented Sep 1, 2026

View reviewed changes

jt2594838 added 2 commits September 1, 2026 18:01

Merge branch 'master' into optimize_prometheus_report

4b89921

fix compilation

17c1151

HTHou reviewed Sep 2, 2026

View reviewed changes

Comment thread ...interface/src/main/java/org/apache/iotdb/metrics/reporter/prometheus/PrometheusReporter.java Outdated

Fix Prometheus reporter initial snapshot race

8b6d0f6

jt2594838 commented Sep 2, 2026

View reviewed changes

HTHou approved these changes Sep 2, 2026

View reviewed changes

Caideyipi reviewed Sep 2, 2026

View reviewed changes

Fix Prometheus scheduler lifecycle and duplicate allocation

63295e6

jt2594838 commented Sep 2, 2026

View reviewed changes

Uh oh!

Conversation

jt2594838 commented Sep 1, 2026

Summary

Validation

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

HTHou left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants