Cloud Native Observability: Prometheus, Grafana, OpenTelemetry, and Tracing

A Kubernetes cluster running hundreds of Pods across dozens of nodes can fail in ways that are invisible unless you're watching for them.

Observability is the practice of making a system's internal state inferable from its external outputs. In distributed systems there are three primary signal types — called the three pillars:

flowchart LR

Metrics["📊 Metrics\nWhat is happening?\n(numbers over time)"]

Logs["📄 Logs\nWhat happened?\n(events with context)"]

Traces["🔗 Traces\nWhere did it happen?\n(request flow across services)"]

Each pillar answers a different question. Metrics alert you that something is wrong. Logs tell you what the error was. Traces show you which service in a chain caused it.

---

Metrics — Prometheus

Prometheus is the de-facto standard for metrics in Kubernetes. It is a pull-based time-series database: Prometheus scrapes HTTP endpoints that expose metrics, stores them, and makes them queryable via PromQL.

flowchart LR

App["Application\n/metrics endpoint\n(Prometheus format)"]

-->|"HTTP scrape\nevery 15s"| Prometheus

Prometheus

-->|"PromQL query"| Grafana

Prometheus

-->|"alert rule"| Alertmanager

Alertmanager

-->|"notification"| PagerDuty["PagerDuty\nSlack\nEmail"]

Metric Types

Type	What it represents	Example
Counter	Monotonically increasing value (never decreases)	Total HTTP requests, total errors
Gauge	Value that can go up or down	Current memory usage, queue depth
Histogram	Distribution of values in configurable buckets	Request latency p50/p90/p99
Summary	Pre-computed quantiles (client-side)	Similar to Histogram but less flexible

# Counter
http_requests_total{method="GET", status="200"} 12483

# Gauge
kube_pod_status_ready{pod="nim-llm-abc", namespace="inference"} 1

# Histogram (multiple lines — one per bucket)
apiserver_request_duration_seconds_bucket{verb="GET", le="0.1"}  8432
apiserver_request_duration_seconds_bucket{verb="GET", le="0.5"}  9211
apiserver_request_duration_seconds_bucket{verb="GET", le="1.0"}  9398
apiserver_request_duration_seconds_count{verb="GET"}             9401
apiserver_request_duration_seconds_sum{verb="GET"}               523.4

PromQL — Prometheus Query Language

PromQL is how you ask questions of your metrics data:

# Rate of HTTP requests per second over last 5 minutes
rate(http_requests_total[5m])

# p99 API Server latency
histogram_quantile(0.99,
  rate(apiserver_request_duration_seconds_bucket[5m])
)

# GPU utilization averaged per node
avg by (node) (DCGM_FI_DEV_GPU_UTIL)

# Pods not in Running state
kube_pod_status_phase{phase!="Running"} == 1

# Free GPU memory per node
DCGM_FI_DEV_FB_FREE / DCGM_FI_DEV_FB_TOTAL * 100

kube-prometheus-stack

The standard way to deploy Prometheus on Kubernetes is the kube-prometheus-stack Helm chart — it bundles Prometheus, Alertmanager, Grafana, and a set of pre-built dashboards and alert rules:

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm upgrade --install kube-prometheus-stack \
  prometheus-community/kube-prometheus-stack \
  --namespace monitoring --create-namespace \
  -f monitoring-values.yaml

It deploys the Prometheus Operator, which introduces ServiceMonitor and PodMonitor CRDs — instead of editing Prometheus config files, you declare what to scrape with Kubernetes objects:

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: dcgm-exporter
  namespace: monitoring
spec:
  selector:
    matchLabels:
      app: dcgm-exporter        # matches the DCGM Exporter Service
  namespaceSelector:
    matchNames: [gpu-operator]
  endpoints:
  - port: metrics
    interval: 15s
    path: /metrics

Prometheus automatically discovers and scrapes any Service that this ServiceMonitor matches — no manual config reload needed.

Alertmanager

Alertmanager handles routing and deduplication of alerts fired by Prometheus alert rules:

# PrometheusRule CRD — defines alert conditions
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: gpu-alerts
  namespace: monitoring
spec:
  groups:
  - name: gpu
    rules:
    - alert: GPUHighTemperature
      expr: DCGM_FI_DEV_GPU_TEMP > 83
      for: 2m
      labels:
        severity: warning
      annotations:
        summary: "GPU temperature above throttle threshold on {{ $labels.pod }}"

    - alert: GPUECCDoublebitError
      expr: increase(DCGM_FI_DEV_ECC_DBE_VOL_TOTAL[10m]) > 0
      labels:
        severity: critical
      annotations:
        summary: "GPU hardware ECC double-bit error — node may need replacement"

---

Visualization — Grafana

Grafana is the visualization layer. It connects to multiple data sources (Prometheus, Loki, Tempo) and renders dashboards.

flowchart LR

Prometheus

-->|"PromQL"| Grafana

Loki["Loki\n(logs)"]

-->|"LogQL"| Grafana

Tempo["Tempo\n(traces)"]

-->|"TraceQL"| Grafana

Grafana

-->|"render"| Dashboard["Dashboards\nPanels\nAlerts"]

Key Grafana concepts:

Concept	What it is
Data source	Connection to a backend (Prometheus, Loki, Tempo, Elasticsearch)
Dashboard	Collection of panels arranged on a grid
Panel	A single visualization (graph, gauge, table, heatmap) with a query
Variable	Dropdown filter that parameterizes queries (e.g., $namespace, $node)
Alert	Grafana can fire alerts directly from dashboard panels

GPU Dashboard on DGX

A typical GPU monitoring dashboard has panels for:

• GPU utilization % (per GPU, heatmap across all nodes)
• HBM memory used / free
• GPU temperature with 83°C threshold line
• NVLink bandwidth (training throughput indicator)
• DCGM ECC error count (hardware health)
• Power draw vs TDP

The NVIDIA DCGM Exporter ships with pre-built Grafana dashboard JSON that imports directly into Grafana.

---

Logs — Structured Logging and Loki

Why Structured Logging

Unstructured logs are for humans. Structured logs are for machines.

# Unstructured — hard to filter, parse, or aggregate
[2026-07-07 14:32:01] ERROR: connection to database failed after 3 retries

# Structured JSON — queryable, indexable, aggregatable
{"timestamp":"2026-07-07T14:32:01Z","level":"error","msg":"connection failed",
 "retries":3,"service":"api","pod":"api-7d9f8b-xkp2m","namespace":"prod"}

Structured logs let you filter across thousands of Pods with a single query.

Log Collection — Fluent Bit

Fluent Bit runs as a DaemonSet on every node. It tails container log files from /var/log/containers/, parses them, and ships them to a log backend.

flowchart LR

Containers["Container logs\n/var/log/containers/*.log"]

-->FluentBit["Fluent Bit\n(DaemonSet — one per node)"]

FluentBit

-->|"forward"| Loki["Loki\n(log aggregator)"]

Loki

-->|"LogQL query"| Grafana

helm upgrade --install fluent-bit fluent/fluent-bit \
  --namespace logging --create-namespace \
  --set config.outputs="[OUTPUT]\n    Name loki\n    Host loki.monitoring.svc.cluster.local"

Loki — Logs Like Prometheus

Loki is a log aggregation system designed by Grafana Labs. It indexes only labels (like Prometheus), not the full log content — this makes it dramatically cheaper than Elasticsearch for Kubernetes log volumes.

# LogQL — query logs like PromQL queries metrics

# All error logs from the inference namespace
{namespace="inference"} |= "error"

# NIM Pod logs filtered to HTTP 5xx
{app="llama3-70b-nim"} | json | status >= 500

# Rate of error logs per Pod over 5 minutes
rate({namespace="training"} |= "ERROR" [5m])

# Training loss from structured log field
{app="pytorch-worker"} | json | __error__="" | unwrap loss [1m]

Loki stores log streams as compressed chunks on object storage (S3, GCS) — far cheaper than Elasticsearch for the same retention period.

---

Distributed Tracing

A single user request in a microservices system touches many services. If the request is slow or fails, which service is responsible?

Distributed tracing answers this by following a request across service boundaries.

flowchart LR

Request["User request\ntrace_id=abc123"]

-->GW["API Gateway\nspan: 5ms"]

GW

-->Auth["Auth Service\nspan: 12ms"]

GW

-->Model["Model Service\nspan: 340ms"]

Model

-->Cache["Cache\nspan: 2ms"]

Model

-->GPU["GPU Inference\nspan: 320ms"]

The full trace shows the request took 357 ms total, with 320 ms spent in GPU inference — the bottleneck is identified immediately.

Key Concepts

Concept	What it is
Trace	The complete journey of one request across all services
Span	A single operation within a trace (one service call, one DB query)
Trace ID	Unique ID propagated through all HTTP headers for a request
Context propagation	Passing the trace ID in HTTP headers (traceparent, X-B3-TraceId) so each service can attach its span to the same trace
Parent span	The span that initiated a child operation

---

OpenTelemetry — The Standard

OpenTelemetry (OTel) is the CNCF standard for instrumentation — a vendor-neutral SDK and wire protocol that works with any tracing backend (Jaeger, Tempo, Zipkin, Datadog, Honeycomb).

flowchart LR

App["Application\n(OTel SDK)"]

-->|"OTLP gRPC/HTTP"| Collector["OTel Collector\n(receive → process → export)"]

Collector

-->Tempo["Grafana Tempo\n(trace storage)"]

Collector

-->Prom["Prometheus\n(metrics)"]

Collector

-->Loki["Loki\n(logs)"]

OTel Collector

The OTel Collector is the central pipeline component — it receives telemetry from applications, processes it (sampling, enrichment, batching), and exports to one or more backends.

# otel-collector-config.yaml
receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318

processors:
  batch:
    timeout: 5s
  resource:
    attributes:
    - action: insert
      key: cluster
      value: dgx-prod

exporters:
  otlp/tempo:
    endpoint: tempo.monitoring.svc.cluster.local:4317
    tls:
      insecure: true
  prometheusremotewrite:
    endpoint: http://prometheus.monitoring.svc.cluster.local:9090/api/v1/write
  loki:
    endpoint: http://loki.monitoring.svc.cluster.local:3100/loki/api/v1/push

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch, resource]
      exporters: [otlp/tempo]
    metrics:
      receivers: [otlp]
      processors: [batch]
      exporters: [prometheusremotewrite]
    logs:
      receivers: [otlp]
      processors: [batch]
      exporters: [loki]

OTel Operator — Auto-Instrumentation

The OpenTelemetry Operator can inject instrumentation into Pods automatically — no code changes:

apiVersion: opentelemetry.io/v1alpha1
kind: Instrumentation
metadata:
  name: python-instrumentation
  namespace: inference
spec:
  exporter:
    endpoint: http://otel-collector.monitoring.svc.cluster.local:4318
  propagators:
    - tracecontext
    - baggage
  python:
    image: ghcr.io/open-telemetry/opentelemetry-operator/autoinstrumentation-python:latest

# Add annotation to Pod — OTel Operator injects the agent
metadata:
  annotations:
    instrumentation.opentelemetry.io/inject-python: "true"

The operator mounts the OTel SDK into the Pod as an init container and sets PYTHONPATH — the application emits traces without any SDK calls in application code.

---

Tracing Backends

Grafana Tempo

Tempo is Grafana's trace backend — stores traces on object storage (S3/GCS), queryable via TraceQL:

helm upgrade --install tempo grafana/tempo-distributed \
  --namespace monitoring \
  --set storage.trace.backend=s3 \
  --set storage.trace.s3.bucket=my-traces-bucket

# TraceQL — find slow GPU inference spans
{ span.service.name = "nim-llm" && duration > 500ms }

Jaeger

Jaeger (CNCF graduated) is the original open-source distributed tracing platform:

helm upgrade --install jaeger jaegertracing/jaeger \
  --namespace monitoring \
  --set storage.type=elasticsearch \
  --set elasticsearch.host=elasticsearch.logging.svc.cluster.local

Jaeger and Tempo both accept the OTLP protocol from OTel — which backend you use is an operational choice, not a code change.

---

The Grafana Observability Stack (LGTM)

flowchart TB

subgraph Backends
  Loki["Loki\n(logs)"]
  Mimir["Mimir / Prometheus\n(metrics)"]
  Tempo["Tempo\n(traces)"]
end

subgraph Collection
  FluentBit["Fluent Bit\n(log collector)"]
  OTelCol["OTel Collector\n(metrics + traces)"]
  DCGM["DCGM Exporter\n(GPU metrics)"]
end

FluentBit --> Loki
OTelCol --> Mimir
OTelCol --> Tempo
DCGM --> Mimir

Loki & Mimir & Tempo --> Grafana["Grafana\n(unified dashboard)"]

The power of running all three backends behind Grafana: correlated signals.

A GPU temperature spike alert fires → you click the Grafana panel → jump to logs from that node at that timestamp → jump from a logged error to the trace showing which inference request caused the GPU to spike → see the full call chain. All without leaving Grafana.

---

Kubernetes-Native Metrics

Kubernetes itself exposes two metrics APIs:

API	Source	What it provides
metrics.k8s.io	Metrics Server	Current CPU/memory per Pod and Node (used by kubectl top and HPA)
custom.metrics.k8s.io	Prometheus Adapter	Custom metrics from Prometheus (used by HPA for GPU utilization)
external.metrics.k8s.io	KEDA	External system metrics (used by KEDA ScaledObjects)

# Uses Metrics Server
kubectl top nodes
kubectl top pods --all-namespaces --sort-by=cpu

# Both require Prometheus Adapter or KEDA
kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta1"

Metrics Server is lightweight — it only holds the current value, not history. Prometheus stores the full time-series history and is the right choice for dashboards and alerting.

---

GPU Observability on DGX

The GPU Operator deploys DCGM Exporter as a DaemonSet. Combined with kube-prometheus-stack, every GPU metric is automatically available in Prometheus and Grafana.

Key dashboards to build for a DGX cluster:

Dashboard	Key panels
GPU Fleet Health	Per-GPU utilization heatmap, ECC error count, temperature, power draw
Training Job Tracker	NVLink bandwidth (AllReduce throughput), GPU memory per worker, loss curve
Inference SLO	Request latency p50/p99, token throughput, GPU utilization per NIM Pod
Cluster Capacity	GPU allocated vs available by node group, Kueue queue depth

# NIM inference throughput (tokens per second)
rate(nim_token_output_total{namespace="inference"}[1m])

# Training AllReduce bottleneck — NVLink bandwidth approaching saturation
DCGM_FI_DEV_NVLINK_BANDWIDTH_TOTAL / 900e9 * 100  # % of 900 GB/s NVSwitch bandwidth

# Kueue queue depth for alert
kueue_pending_workloads{cluster_queue="training-queue"} > 10

---

Key Takeaways

Pillar	Tool	Stores	Queries with
Metrics	Prometheus	Numeric time-series	PromQL
Logs	Loki	Log streams (labels only indexed)	LogQL
Traces	Tempo / Jaeger	Span trees	TraceQL / Jaeger UI
Visualization	Grafana	Nothing (reads from above)	Dashboard panels
Collection	Fluent Bit	Nothing (ships logs to Loki)	—
Collection	OTel Collector	Nothing (pipelines to backends)	—
Instrumentation	OTel SDK / Operator	Nothing (emits signals)	OTLP protocol
Alerting	Alertmanager	Nothing (routes Prometheus alerts)	Routing rules

Observability is not monitoring. Monitoring tells you when something is broken. Observability tells you why. The difference is whether you can ask a new question — one you didn't think to instrument for — and get an answer from the data you already have. For a GPU cluster, that means correlating an AllReduce slowdown (NVLink bandwidth metric) with a noisy-neighbor Pod (logs) to a specific training run (trace) without guessing which component to look at first.