Tech Stack
Summary
Drop-in GPU and scheduler observability for clusters you already have — DCGM metrics, pre-wired Grafana dashboards, and alert rules for Kubernetes and SLURM without replacing existing infrastructure.
What I Built
Project Overview
GPU-Lens delivers production-grade observability for existing GPU clusters through two core layers:
- GPU Health Monitoring — utilization, memory, temperature, ECC errors, and XID errors via NVIDIA DCGM
- Scheduler State Visibility — queue depth and node status for both Kubernetes and SLURM environments
The focus is practical deployment over theoretical examples. GPU-Lens ships pre-configured alert rules and uses validated Grafana dashboards (fetched by ID from grafana.com) rather than hand-written configurations — so it works out of the box on any cluster that already runs Prometheus.
Key Features
DCGM Exporter — Kubernetes or Bare Metal
Deploys the NVIDIA DCGM Exporter as a Kubernetes DaemonSet (via Helm) or as a Docker Compose service for bare-metal and SLURM nodes — the same metrics pipeline either way.
Pre-Wired Grafana Dashboards
Ships with validated community dashboards imported by ID — no YAML hand-editing:
| Dashboard | Grafana ID | What it shows |
|---|---|---|
| DCGM GPU metrics | 12239 | Utilization, memory, temp, ECC, XID, NVLink |
| Node Exporter | 1860 | CPU, memory, disk, network per host |
Alert Rules Out of the Box
Pre-configured PrometheusRule / Alertmanager alerts with tunable thresholds:
| Alert | Condition |
|---|---|
GPUExporterDown | DCGM exporter unreachable |
GPUHighTemperature | Die temp > 80°C for 5m |
GPUMemoryPressure | VRAM > 90% for 10m |
GPUECCError | Double-bit ECC error detected |
GPUIdleReservation | GPU reserved but < 10% util for 30m |
SLURM Scheduler Metrics
Optional SLURM metrics collection exposes queue depth, pending jobs, and node states to Prometheus — closing the gap between GPU health and job scheduler visibility.
Architecture
GPU Nodes
│
├── DCGM Exporter (DaemonSet / Docker Compose)
│ └── /metrics on :9400
│
└── Node Exporter
└── /metrics on :9100
Prometheus ──scrapes──▶ Alertmanager ──routes──▶ PagerDuty / Slack
│
└──▶ Grafana
├── Dashboard 12239 (GPU / DCGM)
└── Dashboard 1860 (Node Exporter)
Why It Matters
| Without GPU-Lens | With GPU-Lens |
|---|---|
| Setting up GPU observability from scratch takes days | Single helm install or docker compose up |
| Grafana dashboards require manual JSON editing | Dashboards fetched by ID — always up to date |
| Alert thresholds are guesswork | Pre-tuned rules based on NVIDIA recommended limits |
| Kubernetes and SLURM need separate tooling | Same Prometheus/Grafana stack covers both |
Technology Stack
GPU Metrics: NVIDIA DCGM, dcgm-exporter
Metrics Collection: Prometheus, Node Exporter
Visualisation: Grafana (dashboards 12239, 1860)
Alerting: Alertmanager
Kubernetes Deployment: Helm 3
Bare-Metal / SLURM Deployment: Docker Compose
Infrastructure: Terraform (optional)
Language: Shell, YAML
License: Apache 2.0