Refresh agent, logs, synthetic, incident, workflow, and AI Observability screenshots

src/content/docs/docs/ai-observability/ai-observability-interface.mdx

@@ -104,7 +104,7 @@ The same tool name can appear on more than one row. For example, a `get_weather`
 
 ### Tool detail
 
-![The tool detail drawer for fetch_page, with call and latency charts, the agents that used it, error types, and the latest failing calls with their arguments and results](./images/ai-observability-tool-detail.png)
+![The tool detail drawer for create_ticket, with call and latency charts, the agents that used it, error types, and the latest failing calls with their arguments and results](./images/ai-observability-tool-detail.png)
 
 Select a tool to see its calls and errors over time, along with:
 

src/content/docs/docs/apm-and-tracing/apm-views/api-monitoring.mdx

@@ -13,7 +13,7 @@ Where the [Service Map](../service-map/) shows caller-to-callee dependencies and
 
 Open **API Monitoring**.
 
-![API Monitoring page, showing the filter bar and the endpoints table with protocol badges, the Status column, and throughput trend sparklines.](./images/api-monitoring-1.png)
+![API Monitoring page, showing the filter bar and the endpoints table with protocol badges, the Status column, and trend sparklines.](./images/api-monitoring-1.png)
 
 Use the **time range** picker to set the observation window. The view defaults to the last **24 hours** so the per-endpoint trend sparklines have enough history to draw.
 
@@ -25,11 +25,11 @@ Each row is one endpoint, identified by its service, protocol, and (for HTTP) me
 
 - **Endpoint:** a protocol badge (`HTTP`, `GRPC`, or `CONNECT_RPC`), a method chip for HTTP rows (`GET`, `POST`, …), and the endpoint label — a templated route such as `GET /users/{id}` for HTTP, or the full method name such as `pkg.Svc/Method` for RPC.
 - **Service:** the service that handles the endpoint. It links to that service's [APM dashboard](../apm-and-services/).
-- **Throughput (req/min):** request rate over the selected window.
+- **Req/min:** request rate over the selected window.
 - **Error rate:** the percentage of requests counted as errors (see [How Errors Are Counted](#how-errors-are-counted)). Shown in red when above zero.
 - **p50 / p95 / p99:** latency percentiles, in milliseconds.
 - **Status:** an error-focused summary of the HTTP status mix. It reads `healthy` when there are no `4xx` or `5xx` responses, and surfaces `4xx` and `5xx` counts when they occur. Hover it to see the full `2xx` / `3xx` / `4xx` / `5xx` breakdown, plus any uncategorized requests.
-- **Throughput trend:** a sparkline of request volume across the window.
+- **Trend:** a sparkline of request volume across the window.
 
 ### HTTP vs. RPC rows
 
@@ -59,7 +59,7 @@ Use the filter bar above the table to narrow the list:
 - **Method:** one or more HTTP methods (`GET`, `POST`, `PUT`, `PATCH`, `DELETE`, `HEAD`, `OPTIONS`).
 - **Status:** restrict to endpoints that returned a given HTTP status class — `4xx` or `5xx`.
 
-Click the **Throughput**, **Error rate**, or **p95** column header to sort by that metric. Sorting is always highest-first and is applied across all matching endpoints, so the table always shows the true top results rather than re-ordering only the current page.
+Click the **Req/min**, **Error rate**, or **p95** column header to sort by that metric. Sorting is always highest-first and is applied across all matching endpoints, so the table always shows the true top results rather than re-ordering only the current page.
 
 :::tip
 To find failing endpoints quickly, sort by **Error rate**, or set the **Status** filter to `5xx`.

src/content/docs/docs/apm-and-tracing/apm-views/apm-and-services.mdx

@@ -24,24 +24,25 @@ Each row represents one service and includes:
 
 ## Service APM Dashboard
 
-Clicking any service in the **List** view opens its APM dashboard, which is a preconfigured view focused on that specific service.
+Clicking any service in the **List** view opens its APM dashboard, which is a preconfigured view focused on that specific service. The dashboard has tabs for **Overview**, [Runtime](../runtime/), [Infrastructure](../infrastructure/), **Traces**, and **Logs**.
 
-![A service's APM dashboard, with the request and latency charts, the Service dependencies panel, and the Issues, Alerts, and Dashboards panels](./images/apm-and-services-2.png)
+![The Overview tab of a service's APM dashboard, with the Throughput, Error rate, and latency cards, the Requests & Errors and Latency charts, and the Internal operations table](./images/apm-and-services-2.png)
 
-The dashboard includes:
+The **Overview** tab includes:
 
 - **Requests, errors, and latency over time:** timeline charts that help identify traffic spikes, error bursts, and latency regressions
 - **Service dependencies:** tables of downstream dependencies, outbound calls, and upstream callers, including request rate, error percentage, and latency
+- **Internal operations:** work with no traced caller, such as jobs, scheduled tasks, and startup work, with its runs, errors, and latency
 - **Related issues, alerts, and dashboards:** links to broader operational context for that service
 
 ## Traces and Logs
 
-From the APM dashboard, you can inspect related traces and logs for a selected service.
+Use the **Traces** and **Logs** tabs to inspect the traces and logs for a selected service.
 
-The **Traces** table shows recent traces with details such as start time, span name, kind, duration, and a link to the full trace for end-to-end request analysis.
+The **Traces** tab shows recent traces with details such as start time, span name, kind, and duration. Click **View Trace** to open the full trace for end-to-end request analysis, or **Explore traces** to search across them.
 
-![image](./images/apm-and-services-3.png)
+![The Traces tab of a service's APM dashboard, with a table of start time, span name, kind, duration, and a View Trace action on each row](./images/apm-and-services-3.png)
 
-The **Logs** table lists recent logs with time, severity, and message, along with actions to explore logs in detail and correlate them with traces and issues for faster root-cause investigation.
+The **Logs** tab lists recent logs with time, severity, and message. Click **View Trace** on a log line to open the trace it belongs to, or **Explore logs** to investigate further.
 
-![image](./images/apm-and-services-4.png)
+![The Logs tab of a service's APM dashboard, with a table of time, severity, and message, and a View Trace action on log lines that carry a trace](./images/apm-and-services-4.png)

src/content/docs/docs/apm-and-tracing/apm-views/index.mdx

@@ -21,6 +21,8 @@ It is built from tracing data and RED-style operational metrics. Use APM when yo
 
 - [Services](./apm-and-services/) for service-level request, latency, and error trends
 - [Service Map](./service-map/) for dependency topology and service-to-service flow analysis
+- [Runtime](./runtime/) for a service's memory, GC, threads, leak signals, and release-to-release cost
+- [Infrastructure](./infrastructure/) for the Kubernetes pods a service runs in, its resource findings, and capacity planning
 
 ## Setup Paths
 

src/content/docs/docs/apm-and-tracing/apm-views/infrastructure.mdx

@@ -0,0 +1,73 @@
+---
+title: "Infrastructure"
+description: "Use a service's Infrastructure tab to see the Kubernetes pods and containers it runs in: CPU and memory against requests and limits, restarts, leak signals, and capacity planning."
+sidebar:
+  order: 6
+---
+The **Infrastructure** tab on a service's APM page shows the Kubernetes pods and containers the service runs in. It answers questions the traces can't: is the pod near its memory limit, is it being throttled, are its requests sized right, and how many instances it needs at higher traffic.
+
+Open **APM & Services**, click a service, then select **Infrastructure**. The tab follows the page's time range.
+
+![The Infrastructure tab, with health cards for CPU and memory of limit, restarts, and pods, findings that the memory request is below real usage and the CPU request is oversized, and a stable container working set floor](./images/infrastructure-1.png)
+
+The tab needs the service to run on Kubernetes with the [KloudMate Agent for Kubernetes](../../../kloudmate-agent/installation/kubernetes-agent/) collecting container metrics. KloudMate matches the service to its pods through the traces, so the tab is empty for a range in which no pods ran the service.
+
+## Filters
+
+When the service runs in more than one pod, a **Pod** filter appears next to the tabs. When it reports `service.version`, a **Version** filter appears too. Both narrow every section of the tab. These filters are shared with the [Runtime](../runtime/) tab, so switching tabs keeps them.
+
+## Health
+
+The health cards show the pods' key values over the selected range:
+
+- **CPU of limit (peak):** the busiest pod against its CPU limit. At the limit, the kernel throttles the pod and latency rises.
+- **Memory of limit (peak):** the busiest pod's working set against its memory limit. The working set is what the OOM killer measures, so at 100% the pod is killed.
+- **Restarts:** container restarts in the range.
+- **Pods:** pods that reported in the range.
+
+Green, amber, and red mark healthy, warning, and critical values.
+
+## Findings
+
+**Findings** checks the pods' limits and requests against what they actually used. Each finding names what to change:
+
+- **Containers were OOMKilled:** a container hit its memory limit and was killed. Raise the limit, or find what grows under **Memory leak signals**.
+- **Container restarts:** each restart drops in-flight requests and resets runtime caches and JIT warm-up.
+- **No memory limit:** one leaking pod can starve every other pod on the node.
+- **Peak memory near the limit:** at 80% or more of the limit, a traffic spike or a slow leak will OOMKill the pod.
+- **Memory request below real usage:** the scheduler overcommits the node. The finding suggests a request about 20% above the peak.
+- **Memory or CPU request oversized:** the request is far above what the pods used. The finding suggests a smaller request that still covers the peak, which frees capacity on every node.
+- **No CPU limit:** fine for latency-sensitive services, but **Capacity planning** then sizes from observed CPU.
+- **CPU peaks near the limit:** at 90% or more of the limit, the kernel throttles the pod, which shows up as tail latency.
+
+## Memory leak signals
+
+The **Container working set floor** card tracks the lowest container memory in each 15 minutes, with a sparkline over the range. A steady climb is a leak. The card is marked **Growing** or **Stable**, shows the current value and how fast it's rising per hour, and estimates when it reaches the memory limit if it keeps growing.
+
+## Kubernetes resources
+
+These charts cover the containers of the pods the service ran on.
+
+- **CPU vs request and limit**
+- **Memory vs request and limit**, using the working set
+- **Memory by pod**, one line per pod, so an outlier stands out
+- **Restarts and terminations**, per pod, with containers currently waiting or terminated and the reason
+- **Major page faults**, pages read back from disk. Sustained faults mean memory pressure.
+
+![CPU vs request and limit, and Memory vs request and limit charts for the service's pods](./images/infrastructure-2.png)
+
+## Capacity planning
+
+**Capacity planning** estimates how many instances the service needs at higher traffic. It starts from what a request costs in CPU today, then sizes each traffic level so that every instance runs at 70% CPU.
+
+The summary line gives the CPU per request, the peak request rate, the instance count, and the cores per instance. Cores come from the CPU limit, or the CPU request when there's no limit. The table then shows the current peak and 1.5×, 2×, 3×, and 5× that peak, each with the requests per second, the CPU needed, the instances needed, and the memory those instances use. An instance count above today's is highlighted with how many more you need.
+
+![Capacity planning for a service: CPU per request, peak traffic, and the requests per second, CPU, instances, and memory needed at the current peak and at 1.5x, 2x, 3x, and 5x that peak](./images/infrastructure-3.png)
+
+The estimate assumes CPU grows in line with traffic. It needs request metrics and CPU usage for the service, and traffic in the selected range.
+
+## Related
+
+- [Runtime](../runtime/) for the language runtime inside these containers
+- [Services](../apm-and-services/) for the service list and its APM dashboard
+- [Kubernetes](../../../infrastructure/) for cluster-wide views

src/content/docs/docs/apm-and-tracing/apm-views/runtime.mdx

@@ -0,0 +1,83 @@
+---
+title: "Runtime"
+description: "Use a service's Runtime tab to read its language runtime health: memory, garbage collection, threads, leak signals, runtime settings, and how a release changed what an instance costs to run."
+sidebar:
+  order: 5
+---
+The **Runtime** tab on a service's APM page shows how the service's language runtime behaves: memory, garbage collection (GC), threads, and the event loop or scheduler. Use it when a service is slow or restarting and the traces don't explain why, to tell whether the runtime is the cause and not the code or a dependency.
+
+Open **APM & Services**, click a service, then select **Runtime**. The tab follows the page's time range.
+
+![The Runtime tab of a Node.js service, with health cards for event loop delay, event loop utilization, heap of V8 limit, and time in GC, a finding that the V8 heap limit is above the container limit, and a stable old_space floor leak card](./images/runtime-1.png)
+
+## Supported runtimes
+
+The tab reads the runtime metrics that the OpenTelemetry SDK reports.
+
+| Runtime | How to get the metrics |
+| --- | --- |
+| Java (JVM) | The OpenTelemetry Java agent reports them by default. For heap buffers and CPU load, add `-Dotel.instrumentation.runtime-telemetry.emit-experimental-metrics=true`. |
+| Go | Call `runtime.Start()` from `go.opentelemetry.io/contrib/instrumentation/runtime`. Scheduler latency also needs `runtime.NewProducer()` on the metric reader. |
+| Node.js | `@opentelemetry/auto-instrumentations-node` includes them by default. With a manual setup, register `@opentelemetry/instrumentation-runtime-node`. |
+| .NET | On .NET 9 and later, add the built-in meter with `AddMeter("System.Runtime")`. On .NET 6 to 8, add `OpenTelemetry.Instrumentation.Runtime` with `AddRuntimeInstrumentation()`. |
+| Python | Install `opentelemetry-instrumentation-system-metrics`. The OpenTelemetry Operator's Python image and `opentelemetry-bootstrap` include it. |
+
+Services that the [KloudMate Agent instruments with the SDK](../../auto-instrumentation/) report these metrics with no extra setup. A service traced with eBPF only doesn't report runtime metrics. If it runs on Kubernetes, its container metrics are still on the [Infrastructure](../infrastructure/) tab.
+
+## Filters
+
+When the service reports `service.version`, a **Version** filter appears next to the tabs. When it runs in more than one pod, a **Pod** filter appears too. Both narrow every section of the tab. Picking a version limits the **Pod** list to the pods that ran that version.
+
+## Health
+
+The health cards show the runtime's key values over the selected range, as a peak or an average. Green, amber, and red mark healthy, warning, and critical values.
+
+| Runtime | Cards |
+| --- | --- |
+| Java | Heap after GC (of max), Time in GC, GC pause p99, Blocked threads (peak) |
+| Go | Goroutines (peak / instance), Heap of GOMEMLIMIT, Allocation rate, Scheduler latency p99 |
+| Node.js | Event loop delay p99 (peak), Event loop utilization, Heap of V8 limit, Time in GC |
+| .NET | GC pause time, Gen2 collections, Thread pool queue (peak), Exceptions |
+| Python | RSS (peak / instance), CPU, Gen2 collections, Uncollectable objects |
+
+## Findings
+
+**Findings** checks the runtime's memory and CPU settings against the container's limits, for services that run on Kubernetes. Each finding names the setting to change. For example:
+
+- **Java:** the max heap is above 80% of the container's memory limit. Metaspace, thread stacks, and direct buffers live outside the heap, so the pod is OOMKilled before the JVM throws `OutOfMemoryError`. Keep `-Xmx` or `MaxRAMPercentage` at 75% or less.
+- **Go:** `GOMEMLIMIT` isn't set, so the GC doesn't know about the container's limit. Or `GOMAXPROCS` is above the CPU limit, so the pod gets throttled.
+- **Node.js:** the V8 heap limit is above the container's limit, or so far below it that Node runs out of heap while the container still has room. Set `--max-old-space-size` to about 75% of the container's limit.
+
+## Memory leak signals
+
+Each card tracks the level memory returns to after each collection, such as the Java heap live set or the Node.js old space floor, with a sparkline over the range. A steady climb is a leak. The card is marked **Growing** or **Stable**, shows the current value and how fast it's rising per hour, and estimates when it reaches the limit if it keeps growing.
+
+## Runtime impact and efficiency
+
+These charts compare the runtime with the service's traffic.
+
+- **Request latency vs** the runtime's pause signal (GC pause p99 for Java, event loop delay p99 for Node.js, scheduler latency p99 for Go). Spikes that line up mean the runtime is slowing requests.
+- **Throughput vs CPU.** CPU should track traffic. CPU rising while traffic is flat means each request costs more work.
+- **CPU time per request.** The unit cost of a request. Compare it before and after a release to catch efficiency regressions.
+
+![The Runtime impact and efficiency charts: request latency against event loop delay p99, and throughput against CPU cores](./images/runtime-2.png)
+
+## Runtime charts
+
+The section named after the runtime, such as **JVM** or **Node.js runtime**, charts memory by pool or generation, GC rate and pauses, threads, CPU, and runtime-specific signals such as goroutines, the event loop, the .NET thread pool, or Python's uncollectable objects. Each chart's description says what to look for.
+
+## Version comparison
+
+**Version comparison** shows whether a release changed what an instance costs to run. Pick a **Baseline** and a **Compared** version to see memory, CPU, and GC per instance side by side, with the change between them. A rise is the regression, since every row is a cost. Both versions must have run in the selected range, so widen the range to compare a release with the one before it.
+
+**Deployments in range** lists each version with when it was first and last seen and its peak instance count. The service needs to report `service.version` for this section.
+
+## Instances
+
+The table lists each instance (pod, host, or service instance) with its memory, CPU, a key runtime value, and restarts. A cell well above the fleet median is highlighted, as is any restart. Use it to find the one pod that behaves differently from the rest.
+
+## Related
+
+- [Infrastructure](../infrastructure/) for the pods and containers the service runs in
+- [Services](../apm-and-services/) for the service list and its APM dashboard
+- [Auto-Instrumentation](../../auto-instrumentation/) to add the SDK with no code change

src/content/docs/docs/apm-and-tracing/index.mdx

@@ -6,7 +6,7 @@ sidebar:
   order: 1
 ---
 
-![image](./images/traces-1.png)
+![The Traces page in Trace summary view, with filters for duration, service, severity, HTTP method, and HTTP status beside a list of traces showing timestamp, entry, services, spans, and duration.](./images/traces-1.png)
 
 The **APM & Services**, **API Monitoring**, and **Traces** pages in KloudMate give you end-to-end visibility into how requests travel across your services, along with service-level health and dependency analysis.