Skip to content

feat(go): add gated OTLP metrics, reconcile events, and log bridge (#2148) - #2675

Open
MartinForReal wants to merge 1 commit into
kagent-dev:mainfrom
MartinForReal:issue/2148-go-observability-gaps
Open

MartinForReal wants to merge 1 commit into
kagent-dev:mainfrom
MartinForReal:issue/2148-go-observability-gaps

Conversation

@MartinForReal

@MartinForReal MartinForReal commented Sep 3, 2026 •

Copy link
Copy Markdown

Implements the Go-side observability gaps from #2148 as three independent, default-OFF additions so the Go runtime/controller expose the same OpenTelemetry signals as the Python runtime.

1. Go ADK — GenAI token-usage metrics (go/adk)

Configures a MeterProvider with a periodic OTLP metric exporter and emits gen_ai.client.token.usage (Int64 histogram, unit {token}) per LLM call.

  • Attribute set matches the Python runtime so one dashboard covers both runtimes: gen_ai.token.type (input/output), gen_ai.request.model, gen_ai.response.model, gen_ai.provider.name, gen_ai.agent.name.
  • The A2A executor records one input + one output observation per LLM call (prompt tokens; output = candidate + reasoning tokens), skipping streamed Partial events so chunked streams are not double-counted.
  • Provider/model labels resolved from agent config (incl. config_usage.ModelName); response model falls back to the request model.
  • Gated behind OTEL_METRICS_ENABLED (default OFF) and the standard OTEL_EXPORTER_OTLP_* endpoint/protocol resolution shared with traces.
  • Tests: adk/pkg/telemetry/metrics_test.go (recording via ManualReader, provider-name mapping) and adk/pkg/a2a/executor_metrics_test.go (token accounting + partial/nil guard).

2. Kubernetes Events on reconcile (go/core)

The discovery reconcilers surface lifecycle events via an optional EventRecorder, mirroring the agentharness-substrate-controller pattern:

  • RemoteMCPServer (internal/controller/remotemcpserver): Normal ToolsDiscovered, Warning ValidationFailed, Warning ReconcileFailed.
  • MCPServer (MCPServerTool role, internal/controller/mcpserver): Normal ToolsDiscovered, Warning ValidationFailed, Warning ReconcileFailed.

Note on Agent/ModelConfig: in current main those objects reconcile through the KRT-based status collections in go/core/v2/controller, not classic controller-runtime Reconcile installers, so they have no EventRecorder path to attach events to. The events gap is therefore implemented on the two remaining classic discovery controllers — the direct successors of the MCPServerTool/RemoteMCPServer roles named in the issue. Emission is a no-op when no recorder is wired (the default).

3. Controller log → OTLP bridge (go/core)

  • telemetry.InitLoggerProvider builds an OTLP LoggerProvider via autoexport (same resource attributes as traces) and sets it global.
  • telemetry.ControllerZapOpts additively tees the controller zap logger with an otelzap bridge core, preserving stdout.
  • Wired into cmd/controller-v2/main.go before the controller logger is built.
  • Gated behind OTEL_LOGGING_ENABLED (already plumbed via the Helm chart otel.logging.enabled); when unset both functions are no-ops and logs are byte-identical.

Validation

  • go build ./..., go vet on all changed packages, gofmt/goimports clean.
  • golangci-lint on changed packages: 0 issues.
  • go test passes for adk/pkg/a2a, adk/pkg/telemetry, adk/pkg/config, core/internal/telemetry, and core/internal/controller/{mcpserver,remotemcpserver} (incl. the new event tests).
    • pkg/session has two pre-existing Windows-only TempDir-cleanup failures (SQLite handle held open during cleanup); unrelated to this change.

Dependencies

Promotes otlplog, otlpmetric(grpc/http), otel/log, otel/metric, otel/sdk/metric to direct requires at the same versions already in the graph, and adds go.opentelemetry.io/contrib/bridges/otelzap v0.19.0. No version bumps.

Refs: #2148

@MartinForReal
MartinForReal requested review from a team and supreme-gg-gg as code owners September 3, 2026 06:54
@github-actions github-actions Bot added enhancement New feature or request and removed enhancement New feature or request labels Sep 3, 2026
@MartinForReal
MartinForReal force-pushed the issue/2148-go-observability-gaps branch 2 times, most recently from fcb707f to 2ff253f Compare September 10, 2026 08:21
@MartinForReal
MartinForReal force-pushed the issue/2148-go-observability-gaps branch from 2ff253f to 8fa258e Compare September 20, 2026 08:58
@MartinForReal

Copy link
Copy Markdown
Author

Hi @supreme-gg-gg, friendly ping for review when you have a moment. This is the gated Go observability work for #2148 (OTLP metrics, reconcile Events, controller log to OTLP bridge). I have rebased it on latest main and resolved the conflicts, so it is ready for another look. Thanks!

@MartinForReal
MartinForReal force-pushed the issue/2148-go-observability-gaps branch from 8fa258e to 457af0a Compare September 21, 2026 05:29
Comment thread go/adk/pkg/telemetry/metrics.go Outdated
@MartinForReal
MartinForReal force-pushed the issue/2148-go-observability-gaps branch from 457af0a to f2a65d5 Compare September 23, 2026 09:32
@MartinForReal

Copy link
Copy Markdown
Author

Addressed: the adk-side OTLP token metric that duplicated #2673 was removed entirely (go/adk/pkg/telemetry/metrics.go and its hooks/tests are gone from this branch). The PR now stays focused on the controller-scoped OTLP metrics, reconcile events, and the controller log bridge; the GenAI token metric lives only in #2673.

@github-actions

github-actions Bot commented Oct 8, 2026

Copy link
Copy Markdown

This pull request has been marked as stale because of no activity in the last 15 days. It will be closed in the next 5 days unless it is tagged "no stalebot" or other activity occurs.

@github-actions github-actions Bot added the stale This issue or PR has become stale label Oct 8, 2026
@MartinForReal
MartinForReal force-pushed the issue/2148-go-observability-gaps branch from eb2faad to 672bfbf Compare October 12, 2026 02:00
@MartinForReal
MartinForReal force-pushed the issue/2148-go-observability-gaps branch from 672bfbf to ea259fa Compare October 12, 2026 02:01
@github-actions github-actions Bot removed the stale This issue or PR has become stale label Oct 12, 2026
Comment thread go/core/internal/controller/mcpserver/reconciler.go Outdated
Comment thread go/core/internal/controller/remotemcpserver/reconciler.go Outdated
Comment thread go/go.mod Outdated
Comment thread go/core/pkg/app/app.go Outdated
Refs: kagent-dev#2148
Signed-off-by: Shangxiang Fan (from Dev Box) <shafan@microsoft.com>
@MartinForReal
MartinForReal force-pushed the issue/2148-go-observability-gaps branch from ea259fa to 969629e Compare October 16, 2026 06:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants