Repository navigation
docs/observability: fix stale and wrong information - #2886
Conversation
Da Huang (git286)
commented
Oct 19, 2026
Da Huang (git286)
commented
- kubectl ate logs uses the singular 'actor' subcommand; the not-running error names atespace/name; quote the crash log message exactly.
- GKE telemetry goes to the GKE managed OTel collector, not straight to GMP.
- Workers normally export through the atelet relay, so an endpoint change reaches them when atelet restarts; the controller copy is the fallback.
- List why atelet may start no relay (https, conflicting per-signal endpoints or compression, unparsable headers), per-signal endpoints as a fallback destination, and that ate-controller injects the endpoint only when it has one. A socket left by an earlier atelet keeps ateworker on the relay, failing every export instead of falling back.
- The probes moved to :9091 (Health probes move to their own server on :9091, and :9090 serves only /metrics #2424); Support default egress policies in ActorTemplates #1558 was closed unmerged.
- Drop the removed pause from the lifecycle and snapshot-kind text.
- The registry table now lists events.yaml.
- Kind Prometheus does not show the metrics that ateworker and atecontroller only push over OTLP; the credential provider does not read ate-otel-config, so its OTLP push never reaches the collector.
- Tracing is head-sampled, and the roadmap metrics already exist.
Jeff Luo (JeffLuoo)
left a comment
Jeff Luo (JeffLuoo)
left a comment
There was a problem hiding this comment.
Thanks for these fixes. I checked the new claims against the code, and most of them are correct. I have two comments to fix before merge. The first one is about the meaning of the local snapshot kind. The second one is about a sentence that contradicts a new note in this PR.
|
|
||
| The three snapshot labels are orthogonal and mean the same thing on every histogram that carries them: | ||
| * `ate.snapshot.kind`: which snapshot the operation reads or writes. `local` (node-local, written by a pause), `latest` (a durable snapshot, including a template's golden snapshot), or `boot` (from scratch, so it never appears on the atelet histograms). | ||
| * `ate.snapshot.kind`: which snapshot the operation reads or writes. `local` (node-local, written by a suspend whose upload to durable storage has not finished), `latest` (a durable snapshot, including a template's golden snapshot), or `boot` (from scratch, so it never appears on the atelet histograms). |
There was a problem hiding this comment.
This definition of local does not match the code. The label does not depend on whether the upload has finished:
- On a checkpoint,
checkpointSnapshotKind(cmd/atelet/metrics.go:171) returnslocalforUPLOAD_MODE_DEFERRED. That is every asynchronous suspend, and asynchronous is the defaultDurabilityMode. A synchronous suspend islatest. - On a restore, ateapi (
workflow_resume.go:832) and atelet label the restorelocalwhen it reads the copy on the node. The copy keeps itsLocalityafter the upload completes, until atelet detaches it (applyDetach). Thus a resume after a finished upload is alsolocal.
An operator who reads ate.actor.checkpoint.duration{ate.snapshot.kind="local"} as "uploads in progress" gets the wrong answer. Suggested text: "local (the copy on the node: an asynchronous suspend writes it, and a resume reads it while the node still has it)".
The registry has the same stale text. docs/metrics/registry/metrics.yaml:301 still says "A pause writes it". This doc calls the registry the source of truth, so please fix the registry brief in this PR too. The comments on checkpointSnapshotKind and latestSnapshotIsLocal also still mention the pause.
There was a problem hiding this comment.
Thanks, you're right. local means the copy on the node, not "upload in progress." I used your wording here. I also fixed the local, suspending, and suspended briefs in metrics.yaml, and the comments on checkpointSnapshotKind and latestSnapshotIsLocal. Fixed in ab15b1e.
| | Dashboards | Not supported | Google Cloud Monitoring (see [Dashboards](#5-dashboards)) | | ||
|
|
||
| > In Kind, `ateapi`, `atelet`, `ate-controller`, and `atenet-router` are pointed at the in-cluster collector, and the controller propagates the endpoint to the worker pods it creates, so all component telemetry lands locally. | ||
| > In Kind, `ateapi`, `atelet`, `ate-controller`, `atenet-router`, and `atenet-egress` are pointed at the in-cluster collector, and the controller propagates the endpoint to the worker pods it creates, so all component telemetry lands locally. |
There was a problem hiding this comment.
This sentence still ends with "so all component telemetry lands locally". But the new note at line 414 says that the credential provider's OTLP push goes to localhost:4317 and fails. Both statements cannot be true on a Kind install that runs the Kubernetes credential provider. Please limit the sentence to the components it names, or name the credential provider as the exception.
There was a problem hiding this comment.
Good catch. The sentence now covers only the components it names and the workers, and it calls out the credential provider as the exception. The next paragraph said "Every component reads that endpoint", which had the same problem, so I changed it to "Each of these components". Fixed in ab15b1e.
|
Please rebase |
- kubectl ate logs uses the singular 'actor' subcommand; the not-running error names atespace/name; quote the crash log message exactly. - GKE telemetry goes to the GKE managed OTel collector, not straight to GMP. - Workers normally export through the atelet relay, so an endpoint change reaches them when atelet restarts; the controller copy is the fallback. - List why atelet may start no relay (https, conflicting per-signal endpoints or compression, unparsable headers), per-signal endpoints as a fallback destination, and that ate-controller injects the endpoint only when it has one. A socket left by an earlier atelet keeps ateworker on the relay, failing every export instead of falling back. - The probes moved to :9091 (agent-substrate#2424); agent-substrate#1558 was closed unmerged. - Drop the removed pause from the lifecycle and snapshot-kind text. - The registry table now lists events.yaml. - Kind Prometheus does not show the metrics that ateworker and atecontroller only push over OTLP; the credential provider does not read ate-otel-config, so its OTLP push never reaches the collector. - Tracing is head-sampled, and the roadmap metrics already exist. Checked on a Kind cluster built from main and on a GKE install.
…ential provider is outside Kind's local telemetry - ate.snapshot.kind=local marks a checkpoint by an asynchronous suspend and a restore from the copy on the node, whether or not its upload has finished. Fix the doc, the registry briefs and the code comments that still described the removed pause. - The Kind note no longer says all component telemetry lands locally: the credential provider does not read ate-otel-config.
…local Since agent-substrate#2883 a synchronous suspend keeps a copy on the node after its upload, so the resume after it reads that copy and is labeled local. The checkpoint itself stays latest.
ab15b1e to
e463199
Compare
