Monitoring & Operations AI Infrastructure

AI clusters are:

• GPU-dense
• Power-hungry
• Network-intensive
• Storage-dependent

DevOPs Goals is to:

• Maximize GPU utilization
• Detect failures early
• Prevent downtime
• Optimize performance
• Ensure thermal and power stability

Observability:

Why is it happening?

• Observability includes:
  • Metrics
  • Logs
  • Traces

Monitoring:

What is happening?

AI data centers monitor multiple layers:

1. Hardware Layer

• GPU temperature
• GPU utilization
• Power draw
• CPU usage
• Memory usage
• Disk I/O
• NIC throughput

2. Network Layer

• Latency
• Packet loss
• RDMA errors
• Congestion
• Throughput

3. Storage Layer

• IOPS
• Throughput
• Latency
• File system saturation

4. Application Layer

• Training job status
• Job queue depth
• Container health
• Pod failures (Kubernetes)

---

🧮 GPU Monitoring Tools

GPU Utilization Optimization

Low GPU utilization may indicate:

• Storage bottleneck
• Network congestion
• Poor job scheduling
• Insufficient batch size
• CPU bottleneck

Operations teams should investigate the root cause before adding more GPUs.

Capacity Planning

Operations teams must:

• Track GPU utilization trends
• Forecast storage growth
• Monitor network saturation
• Plan rack power expansion

Goal: Avoid resource shortages while maximizing GPU utilization.

Multi GPU Systems: Scale Up vs Scale Out

1. Scale Up/ Vertical Scaling ⬆️

Increase the number of GPUs within a single server Node

Technologies

• NVLink for GPU-to-GPU communication
• NVSwitch for large multi-GPU systems

Best For

• Single-node training
• Large model inference
• Small to medium AI clusters

Key Challenge

• Load Balance between GPUs is critical
• Avoiding idle GPUs due to imbalance

Example: 8 × NVIDIA H100 GPUs connected through NVSwitch in a single server.

2. Scale Out/ Horizontal Scaling ➡️

Add more GPU servers to the cluster.

Technologies

• InfiniBand or RoCE for inter-node communication
• GPUDirect RDMA for GPU-to-GPU across nodes

Best for

• large clusters and distributed training
• Enterprise AI platforms
• Multi-rack GPU clusters

Key Challenge

• Network communication overhead
• Load balancing across nodes
• Load Balance across nodes is critical

Example

128 GPU nodes connected through InfiniBand.

Tools Used for GPU Monitoring

📟 1 System Management Interface (SMI)

Monitoring GPUs on a single server.

Use Case: Quick troubleshooting and health checks on individual systems.

Characteristics

• Command-line utility
• Fast diagnostics
• Real-time GPU statistics

nvidia-smi

nvidia-smi -l 1

nvidia-smi topo -m

📶 2 NVIDIA Data Center GPU Manager (DCGM)

Monitoring 10+ GPU nodes inside the operating system, at the GPU layer.

Purpose

Provides enterprise-grade monitoring, diagnostics, and health management for NVIDIA GPUs.

Kubernetes Integration

• Kubernetes Cluster level GPU Management
• Exposes metrics through DCGM Exporter
• Integrates with Prometheus and Grafana

Used by:

• Prometheus (via DCGM exporter)
• Cluster monitoring systems

flowchart
    GPU1[GPU 1 🧮] --> DCGM["Data Center GPU Manger(DGCM) 📶"]
    GPU2[GPU 2 🧮] --> DCGM
    GPU3[GPU 3 🧮] --> DCGM
    GPU4[GPU 4 🧮] --> DCGM

  DCGM -->DCGMExporter["DCGM Exporter ⬆️"]
  DCGMExporter-->Prometheus["Prometheus 🔥"]
  Prometheus--> Grafana["Grafana Dashboard 📊"]

DGCM Dashboard

GPU-level monitoring and management:

• GPU health
• Temperature
• Power usage
• Utilization
• ECC errors
• GPU diagnostics

📊 3 NVIDIA Base Command Manager (BCM)

Cluster-level infrastructure and workload management platform.

Purpose Manage and operate large GPU clusters used for AI and HPC workloads.

Capabilities

• Cluster provisioning: Mange entire cluster of GPU nodes in AI Data Center
• Job Scheduling and Monitoring
• Multi Team/ User/ environment management
• Ensure Scale optimal resource allocation
• Used REST API and CLI for management

Operates At

Cluster and platform layer

While DCGM answers:

"How healthy is the GPU?"

BCM answers:

"How is the entire GPU cluster being managed and utilized?"

SMI vs BCM vs DCGM

flowchart TD

  BCM[BCM<br/>Cluster Management <br/>Scheduling & Operations]
  BCM-->DCGM[DCGM <br/>GPU Monitoring <br/> Health & Diagnostics]
  DCGM-->SMI[NVIDIA SMI <br/>Per-Node GPU Metrics]      

Tool	Scope	Primary Purpose
SMI	Single GPU Server	Diagnostics and real-time status
DCGM	GPU Fleet / Cluster	Health monitoring, metrics, alerting
BCM	Entire AI Cluster	Cluster operations and resource management

---

👁️ Infrastructure Monitoring Tools

🔥 Prometheus

Prometheus is an open-source monitoring and alerting system built for collecting time-series metrics.

• It scrapes metrics from targets at regular intervals and stores them as time-series data.
• metric_name + labels + timestamp + value eg gpu_utilization{node="node1", gpu="0"} 92%

Key Components:

1. Exporters

Exporters expose metrics.

Common ones:

• Node Exporter → CPU, memory, disk
• DCGM Exporter → GPU metrics
• Kubernetes Exporter → Pod/node stats

2. PromQL (Query Language)

Querying time-series data for insights.

Used to:

• Calculate averages
• Detect spikes
• Aggregate across nodes -Identify trends

Example:

• Average GPU utilization across cluster
• Network errors per minute

3. Alert manager

Triggers alerts when some threshold is breached.

Example alerts

• GPU temp exceeds threshold
• Node becomes unreachable
• Disk space low
• Packet drops increase

Alerts should be actionable, not noisy.

---

📈 Grafana

Visualize and analyze metrics collected by Prometheus and other data sources.

• Visualization dashboards
• Real-time monitoring
• Alerting integration

Prometheus vs Grafana (Common Confusion)

Prometheus = Collect & store metrics Grafana = Visualize metrics

Prometheus is the data engine. Grafana is the dashboard.

---

Cluster Orchestration Monitoring

1. ☸️ Kubernetes

Kubernetes for inference clusters

• Deploy → Scale → Run continuously.

Use case:

• Model serving
• AI APIs
• Microservices
• Continuous workloads
• Auto-scaling systems

Monitor:

• Pod status
• Node health
• Resource usage
• Scheduling issues

If question mentions:

• Pods
• Replica scaling
• Microservices
• Model serving endpoint
• YAML deployment

🎛️ Simple Linux Utility for Resource Mngt(Slurm))

Open-source workload manager and job scheduler for HPC and AI training clusters.

• Large distributed training jobs
• Submit → Wait → Run → Finish.

Use case:

• HPC simulations
• Multi-node batch workloads
• Deterministic scheduling
• Queue-based execution

Monitors:

• Job queue
• Resource allocation
• Failed jobs
• Node states

If question mentions:

• Queue priority
• sbatch or srun
• HPC cluster
• Large multi-node training

sbatch train.sh

srun python train.py

squeue

sinfo

Slurm vs Kubernetes Comparison

Feature	Slurm	Kubernetes
Primary Purpose	Job Scheduling	Container Orchestration
Typical AI Phase	Training	Inference
Workload Pattern	Batch Jobs	Long-running Services
Execution Model	Run-to-completion	Always-on or auto-scaled services
Scheduling	Queue-based	Pod-based
Resource Allocation	Nodes, CPUs, GPUs	Containers, Pods
Scaling Method	More allocated nodes	More replicas/pods
Common Users	Researchers, HPC Teams	DevOps, MLOps Teams
Interface	sbatch, srun, squeue	kubectl, Helm, APIs
AI Data Center Role	Training Cluster Scheduler	Serving Platform

Based on Usage Slurm vs K8

Feature	Slurm	Kubernetes
Primary Focus	Resource allocation & batch job management	Container lifecycle management
Workload Type	HPC, AI training, data processing	AI inference, microservices, data pipelines
Architecture Style	Static jobs, queued execution	Dynamic pods, continuous service
Execution Model	Run-to-completion batch jobs	Always-on or auto-scaled services
Scheduling Logic	Priority queues, resource quotas	Load balancing, replica scaling
GPU Integration	CUDA-aware, multi-GPU aware (GPU plugin)	GPU Operator, MIG management, DCGM metrics
Scalability	Scales to thousands of compute nodes	Scales container workloads across clusters
User Interface	CLI tools (sbatch, srun)	API-driven (kubectl, Helm, YAML)
Typical Users	Researchers, HPC admins	DevOps, MLOps, platform engineers
Best Suited For	Training phase	Inference / Serving phase

---

⚡️ Power & Cooling Monitoring

AI clusters consume massive power.

Monitor:

• Rack power draw
• PSU health
• Cooling system efficiency
• Data center temperature
• Airflow

Failure to monitor → thermal shutdown.

Power Usage Effectiveness (PUE)

standard metric for measuring data center energy efficiency, calculated as the ratio of total facility power to IT equipment power

Lower PUE means better energy efficiency.

$$Formula = Total \ Facility \ Power / IT \ Equipment \ Power$$

That mean PUE is always

$PUE > 1.0$

The higher the number, the more energy is used for overhead (cooling, power losses, etc.).

$PUE = 1.0$

100% of energy supports computing

• Impossible in real world

$PUE = 1.2$

Highly efficient

• Close to ideal eg AWS/Google data centers

$PUE = 2$

Inefficient Data Center

For every 1 watt used by IT, another 1 watt is used for infrastructure.

• Typical older data centers have PUEs between 1.5 and 2.0

---

Cooling Options

1. Air Colling

• Max at 30 kW per rack
• Less efficient at high densities
• Lower infrastructure cost

2. Liquid Cooling

• Better for high density racks (30–80 kW+)
• More efficient heat removal
• Expensive infrastructure

---

High Availability (HA)

AI infrastructure should support:

• Redundant power supplies
• Redundant networking paths
• Failover nodes
• Backup storage

Single point of failure = unacceptable.

Failure Scenarios to Understand

Common failures:

• GPU overheating
• Node crash
• Network congestion
• Storage saturation
• Job scheduler deadlock

Monitoring enables:

• Rapid detection
• Root cause analysis
• Faster recovery

Security Monitoring

Includes:

• Unauthorized access attempts
• Configuration changes
• Network anomalies
• DPU isolation policies
• Role-based access control

---

Exam Scenarios to Recognize

If question mentions:

• GPU temperature spikes → Thermal monitoring
• ECC memory errors → DCGM
• Dashboard visualization → Grafana
• Metric scraping → Prometheus
• HPC job queue management → Slurm
• Container orchestration → Kubernetes
• Rack power issue → Data center monitoring
• Underutilized GPUs → Operational inefficiency

---

Lifecycle Management

Operations includes:

• Firmware updates
• Driver updates
• CUDA updates
• Security patches
• Hardware replacement

Change management must:

• Minimize downtime
• Be documented
• Be tested

Logging Systems

Logs provide:

• Error tracing
• Job debugging
• Security auditing
• System failure analysis

Centralized logging:

• Aggregated logs
• Searchable
• Long-term retention

Alerting Strategy

Monitoring without alerting = useless.

Effective alerts:

• Temperature threshold exceeded
• GPU ECC errors
• Node unreachable
• Disk nearly full
• Network congestion

Alerts should be:

• Actionable
• Prioritized
• Not noisy

---

Quick Memory Anchors

• DCGM = GPU health monitoring
• Prometheus = Metrics collection
• Grafana = Visualization
• Slurm = HPC job scheduler
• Kubernetes = Container orchestration
• Monitoring prevents GPU idle time
• Alerting must be actionable