GPU-Lens — Drop-in GPU Observability for Kubernetes & SLURM

Project Overview

GPU-Lens delivers production-grade observability for existing GPU clusters through two core layers:

1. GPU Health Monitoring — utilization, memory, temperature, ECC errors, and XID errors via NVIDIA DCGM
2. Scheduler State Visibility — queue depth and node status for both Kubernetes and SLURM environments

The focus is practical deployment over theoretical examples. GPU-Lens ships pre-configured alert rules and uses validated Grafana dashboards (fetched by ID from grafana.com) rather than hand-written configurations — so it works out of the box on any cluster that already runs Prometheus.

---

Key Features

DCGM Exporter — Kubernetes or Bare Metal

Deploys the NVIDIA DCGM Exporter as a Kubernetes DaemonSet (via Helm) or as a Docker Compose service for bare-metal and SLURM nodes — the same metrics pipeline either way.

Pre-Wired Grafana Dashboards

Ships with validated community dashboards imported by ID — no YAML hand-editing:

Dashboard	Grafana ID	What it shows
DCGM GPU metrics	12239	Utilization, memory, temp, ECC, XID, NVLink
Node Exporter	1860	CPU, memory, disk, network per host

Alert Rules Out of the Box

Pre-configured PrometheusRule / Alertmanager alerts with tunable thresholds:

Alert	Condition
GPUExporterDown	DCGM exporter unreachable
GPUHighTemperature	Die temp > 80°C for 5m
GPUMemoryPressure	VRAM > 90% for 10m
GPUECCError	Double-bit ECC error detected
GPUIdleReservation	GPU reserved but < 10% util for 30m

SLURM Scheduler Metrics

Optional SLURM metrics collection exposes queue depth, pending jobs, and node states to Prometheus — closing the gap between GPU health and job scheduler visibility.

---

Architecture

GPU Nodes
    │
    ├── DCGM Exporter (DaemonSet / Docker Compose)
    │       └── /metrics on :9400
    │
    └── Node Exporter
            └── /metrics on :9100

Prometheus ──scrapes──▶ Alertmanager ──routes──▶ PagerDuty / Slack
     │
     └──▶ Grafana
              ├── Dashboard 12239 (GPU / DCGM)
              └── Dashboard 1860  (Node Exporter)

---

Why It Matters

Without GPU-Lens	With GPU-Lens
Setting up GPU observability from scratch takes days	Single helm install or docker compose up
Grafana dashboards require manual JSON editing	Dashboards fetched by ID — always up to date
Alert thresholds are guesswork	Pre-tuned rules based on NVIDIA recommended limits
Kubernetes and SLURM need separate tooling	Same Prometheus/Grafana stack covers both

---

Technology Stack

GPU Metrics: NVIDIA DCGM, dcgm-exporter

Metrics Collection: Prometheus, Node Exporter

Visualisation: Grafana (dashboards 12239, 1860)

Alerting: Alertmanager

Kubernetes Deployment: Helm 3

Bare-Metal / SLURM Deployment: Docker Compose

Infrastructure: Terraform (optional)

Language: Shell, YAML

License: Apache 2.0