Repository navigation
atecontroller metrics are never scraped #933
Description
atecontroller is the only control plane component with no Prometheus target. It uses
InitMetricsPushOnly (cmd/atecontroller/main.go:112) and its pod carries no prometheus.io
annotations, so the manager's :8080 is never scraped, while ateapi, atelet and atenet-router
serve /metrics on 9090 and are. docs/observability.md:178 assumes every component is scraped.
Should atecontroller move to the InitMetrics plus StartMetricsServer shape on 9090, or is
push-only intended, in which case the dead :8080 port should go instead?
Activity
YQ-Wang commentedon Aug 15, 2026
Hi, I’d be happy to work on this if no one else has already started. I checked the metrics wiring introduced in #754, and the push-only path appears intentional: atecontroller bridges controller-runtime’s registry into its OTLP reader, and the metrics e2e verifies that
controller_runtime_*reaches the collector. A direct switch toInitMetricsplusStartMetricsServerwould split the registries and could regress that OTLP path.My proposed scope is therefore to disable controller-runtime’s unused
:8080metrics listener, remove the corresponding metrics-only Service/container port, and updatedocs/observability.mdso it distinguishes direct Prometheus scrape targets from atecontroller’s OTLP path. I would keep the existing bridge/e2e coverage and add focused coverage for the disabled manager endpoint. Please let me know if this direction sounds reasonable.ericcurtin commentedon Sep 29, 2026
Sent #1976: push-only is intended, so it disables the manager's :8080 listener and drops the Service and port. Yiqing Wang (@YQ-Wang), apologies if you already had this in flight.
JeffLuoo commentedon Oct 1, 2026
Thanks for raising this. I want to answer the broader question first: should the substrate metrics pipeline be push-only? My answer is no. Substrate has two kinds of metrics, and each kind needs a different policy.
Actor metrics: push. We recommend OTLP push for metrics from an Actor. An Actor can start, do its work, and suspend between two scrapes, so a scrape can miss its samples. An Actor also moves between Workers, so it has no stable address for a scrape target. A suspended Actor cannot answer a scrape at all.
Component metrics: both paths are supported.
Comments are welcome if you see a problem with this direction.
Da Huang (@git286) Tim Bai (@baizhenyu) Krisztian F (@krisztianfekete) Julian Gutierrez Oschmann (@juli4n)
JeffLuoo commentedon Oct 1, 2026
With my comment above, we will need to revisit #1976
krisztianfekete commentedon Oct 6, 2026
This is related to / blocked on #2067
JeffLuoo commentedon Oct 6, 2026
By default, no component is scraped now. #2067 removes the scrape annotations, so a scrape occurs only when the operator adds an annotation or a
PodMonitor.A correction to my comment above: to scrape only, set
OTEL_METRICS_EXPORTER=prometheus.nonenow turns off both paths.Task 3 in #2067 does this change.
atecontroller: the :8080 metrics listener follows OTEL_METRICS_EXPORT…