GPU Scheduling in Kubernetes: Device Plugins, GPU Operator & MIG

Kubernetes knows how to schedule CPU and memory. It has no built-in concept of a GPU.

A standard node description looks like this:

cpu: 64
memory: 256Gi

The 8 H100 GPUs on that same node are completely invisible to the scheduler — unless something tells it they exist.

This post covers the full stack that makes GPU scheduling work: how GPUs become visible, how they get inside containers, how the GPU Operator automates the whole setup, and how MIG and time-slicing let you share expensive hardware efficiently.

---

The Device Plugin Framework

Kubernetes uses the Device Plugin Framework to support hardware accelerators — GPUs, FPGAs, network adapters, or any other specialized device.

A Device Plugin is a gRPC server that runs as a DaemonSet on every node that has the hardware. It registers with the Kubelet and handles two operations:

ListAndWatch — Advertising Devices

The plugin calls ListAndWatch to stream the list of available devices to Kubelet continuously. Kubelet then updates the node's capacity in the API Server.

sequenceDiagram

participant Plugin as Device Plugin\n(DaemonSet)
participant Kubelet
participant API as API Server

Plugin->>Kubelet: Register (resourceName: nvidia.com/gpu)
Kubelet-->>Plugin: OK
Plugin->>Kubelet: ListAndWatch → [GPU-0, GPU-1, ..., GPU-7] (Healthy)
Kubelet->>API: Update Node capacity\nnvidia.com/gpu: 8

After this, kubectl describe node shows:

Capacity:
  nvidia.com/gpu:  8
Allocatable:
  nvidia.com/gpu:  8

Allocate — Assigning a Device to a Container

When the Kubelet starts a container that requested a GPU, it calls Allocate on the Device Plugin, which returns the environment variables and device mounts needed.

sequenceDiagram

participant Kubelet
participant Plugin as Device Plugin
participant Runtime as Container Runtime

Kubelet->>Plugin: Allocate(deviceIDs: [GPU-3])
Plugin-->>Kubelet: AllocateResponse\n(envs: CUDA_VISIBLE_DEVICES=3\n devices: /dev/nvidia3, /dev/nvidiactl)
Kubelet->>Runtime: Start container with\nenv + device mounts

The container sees only its assigned GPU. Other GPUs on the node are invisible inside the container.

Requesting GPUs in a Pod

resources:
  limits:
    nvidia.com/gpu: "1"   # request exactly 1 GPU

Two rules that differ from CPU and memory:

1. No requests field — only limits. You cannot request 0.5 of a GPU.
2. Non-overcommittable — the scheduler never assigns more GPUs than physically available. There is no concept of GPU bursting.

For multi-GPU pods:

resources:
  limits:
    nvidia.com/gpu: "8"   # request all 8 GPUs on a DGX node

---

nvidia-container-toolkit

The Device Plugin tells Kubernetes which GPU to assign. But how does a GPU actually get inside a container?

That is the job of nvidia-container-toolkit (formerly nvidia-docker).

It is a container runtime hook that intercepts container creation and injects GPU access:

flowchart LR

Kubelet

-->ContainerD["containerd"]

ContainerD

-->NVHook["nvidia-container-hook\n(OCI hook)"]

NVHook

-->Inject["/dev/nvidia3\n/dev/nvidiactl\n/dev/nvidia-uvm\nCUDA libraries\nCUDA_VISIBLE_DEVICES=3"]

Inject

-->Container["Container\n(sees GPU 3 only)"]

What gets injected:

Item	Purpose
/dev/nvidia3	The actual GPU device file
/dev/nvidiactl	NVIDIA control device
/dev/nvidia-uvm	Unified Memory driver
CUDA shared libraries	Mounted from host into container
CUDA_VISIBLE_DEVICES=3	Tells CUDA which GPU index is visible

This is why GPU containers work without bundling CUDA drivers — they use the host's drivers via device file injection.

---

NVIDIA GPU Operator

Installing GPU drivers, container toolkit, device plugin, and monitoring agents across hundreds of nodes manually is error-prone and hard to keep consistent.

The GPU Operator is a Kubernetes Operator that manages the entire GPU software stack as a single reconciliation loop.

flowchart TB

GPUOperator["GPU Operator\n(reconciles ClusterPolicy)"]

-->Driver["NVIDIA Driver\n(DaemonSet — compiles kernel module)"]

GPUOperator

-->Toolkit["nvidia-container-toolkit\n(DaemonSet — injects GPU into containers)"]

GPUOperator

-->DevicePlugin["Device Plugin\n(DaemonSet — ListAndWatch + Allocate)"]

GPUOperator

-->DCGM["DCGM Exporter\n(DaemonSet — Prometheus metrics)"]

GPUOperator

-->MIGManager["MIG Manager\n(DaemonSet — configures MIG partitions)"]

GPUOperator

-->NodeFeatureDiscovery["Node Feature Discovery\n(labels GPU nodes)"]

ClusterPolicy

Everything the GPU Operator manages is declared in a single CRD: ClusterPolicy.

apiVersion: nvidia.com/v1
kind: ClusterPolicy
metadata:
  name: gpu-cluster-policy
spec:
  driver:
    enabled: true
    version: "535.104.12"
    repository: nvcr.io/nvidia

  toolkit:
    enabled: true
    version: "v1.14.3-centos7"

  devicePlugin:
    enabled: true
    version: "v0.14.5"
    config:
      name: device-plugin-config   # ConfigMap with sharing config

  dcgmExporter:
    enabled: true
    version: "3.3.0-3.2.0-ubuntu22.04"
    config:
      name: dcgm-metrics-config

  mig:
    strategy: single   # single: all GPUs use same MIG profile
                       # mixed: GPUs can have different profiles

  migManager:
    enabled: true

The GPU Operator watches ClusterPolicy and reconciles every component across every GPU node automatically — if the driver crashes, it restarts it; if a new node joins, it installs everything on that node.

---

GPU Monitoring with DCGM

The GPU Operator deploys DCGM Exporter (Data Center GPU Manager) as a DaemonSet. It exposes per-GPU Prometheus metrics.

flowchart LR

GPU

-->DCGM["DCGM\n(direct driver access)"]

DCGM

-->Exporter["DCGM Exporter\n(/metrics endpoint)"]

Exporter

-->Prometheus

Prometheus

-->Grafana

Key metrics to monitor in production:

Metric	What it measures	Alert when
DCGM_FI_DEV_GPU_UTIL	SM utilization %	< 80% sustained (underutilization)
DCGM_FI_DEV_MEM_COPY_UTIL	Memory bandwidth utilization %	Sustained 100% (memory bound)
DCGM_FI_DEV_FB_USED	Framebuffer (HBM) memory used	> 90% of capacity
DCGM_FI_DEV_POWER_USAGE	GPU power draw (watts)	> TDP (thermal throttling)
DCGM_FI_DEV_GPU_TEMP	GPU die temperature (°C)	> 83°C (H100 throttle threshold)
DCGM_FI_DEV_ECC_DBE_VOL_TOTAL	Double-bit ECC errors	Any non-zero (hardware fault)
DCGM_FI_DEV_NVLINK_BANDWIDTH_TOTAL	NVLink total bandwidth	Drop from baseline (link degraded)
DCGM_FI_DEV_PCIE_TX_THROUGHPUT	PCIe TX bytes/s	Sustained max (PCIe bottleneck)

Double-bit ECC errors (DBE) indicate a hardware fault. The GPU Operator automatically taints the node when they occur (see below).

---

Multi-Instance GPU (MIG)

An H100 SXM5 has 80 GB of HBM3 memory and 132 streaming multiprocessors (SMs).

Most inference workloads don't need all of it.

Without MIG, one Pod monopolizes the entire GPU regardless of how little of it it uses.

MIG (Multi-Instance GPU) partitions a physical GPU into isolated slices at the hardware level — each slice has its own SMs, HBM memory, and PCIe bandwidth. They cannot interfere with each other.

MIG Profiles on H100

The H100 supports the following partitions:

Profile	SMs	HBM	Max per GPU	nvidia.com resource name
1g.10gb	1/7	10 GB	7	nvidia.com/mig-1g.10gb
2g.20gb	2/7	20 GB	3	nvidia.com/mig-2g.20gb
3g.40gb	3/7	40 GB	2	nvidia.com/mig-3g.40gb
4g.40gb	4/7	40 GB	1	nvidia.com/mig-4g.40gb
7g.80gb	7/7	80 GB	1	nvidia.com/mig-7g.80gb

A common production layout for an inference node — 7 small partitions, 7 Pods per GPU:

H100 (80 GB, 132 SMs)
├── MIG 1g.10gb  → Pod: 10 GB model
├── MIG 1g.10gb  → Pod: 10 GB model
├── MIG 1g.10gb  → Pod: 10 GB model
├── MIG 1g.10gb  → Pod: 10 GB model
├── MIG 1g.10gb  → Pod: 10 GB model
├── MIG 1g.10gb  → Pod: 10 GB model
└── MIG 1g.10gb  → Pod: 10 GB model

A mixed layout for a node running both training and inference:

H100 (80 GB)
├── MIG 3g.40gb  → Training Pod (40 GB)
├── MIG 2g.20gb  → Inference Pod (20 GB)
└── MIG 2g.20gb  → Inference Pod (20 GB)

Kubernetes View of MIG

With MIG configured, the Device Plugin advertises partitions instead of whole GPUs:

Capacity:
  nvidia.com/gpu:        0           # full GPUs no longer schedulable
  nvidia.com/mig-1g.10gb: 7
  nvidia.com/mig-2g.20gb: 0
  nvidia.com/mig-3g.40gb: 1

Pods request partitions the same way they request full GPUs:

resources:
  limits:
    nvidia.com/mig-1g.10gb: "1"

The MIG Manager (deployed by GPU Operator) creates and destroys MIG partitions based on the desired configuration in a ConfigMap — no manual nvidia-smi mig commands needed.

---

GPU Sharing Strategies

Strategy	Isolation	Use case	Notes
Full GPU	Complete	Large LLM training, full-GPU inference	nvidia.com/gpu: 1
MIG	Hardware-level	Production multi-tenant inference	H100/A100 only; requires MIG Manager
Time-Slicing	None (shared context)	Development, small batch jobs	Multiple Pods share one GPU in time
Multi-Process Service (MPS)	Partial	Inference with many small requests	CUDA context shared; memory not isolated

Time-Slicing Configuration

Time-slicing is configured via a ConfigMap consumed by the Device Plugin:

apiVersion: v1
kind: ConfigMap
metadata:
  name: device-plugin-config
  namespace: gpu-operator
data:
  config.json: |
    {
      "version": "v1",
      "sharing": {
        "timeSlicing": {
          "resources": [
            {
              "name": "nvidia.com/gpu",
              "replicas": 4     # advertise each GPU as 4 schedulable units
            }
          ]
        }
      }
    }

After this, a node with 8 physical GPUs advertises nvidia.com/gpu: 32. Four Pods can share one GPU — each gets a time slice of the GPU's compute.

Important: time-slicing provides no memory isolation. A Pod that allocates all GPU memory will OOM-kill other Pods sharing the same GPU. Use MIG for production multi-tenant workloads.

---

GPU Health and Automatic Node Tainting

When a GPU develops a hardware fault, the GPU Operator automatically takes action:

flowchart TD

DCGM["DCGM detects\nDBE ECC error on GPU 3"]

-->MIGManager["GPU Operator\nhealth monitor"]

MIGManager

-->Taint["Taint node:\nnvidia.com/gpu-ecc-error=:NoSchedule"]

Taint

-->NewPods["New GPU Pods\nno longer scheduled here"]

Taint

-->Alert["Prometheus alert fired\n→ PagerDuty / Slack"]

The taint prevents new GPU Pods from landing on a degraded node. Existing Pods are not evicted — they continue running until they complete or the operator decides a reboot/replacement is needed.

DCGM also performs active health checks — it runs a short GPU compute test periodically and marks GPUs as unhealthy if the test fails, even before an ECC error appears.

Relevant taints the GPU Operator manages:

Taint	Cause
nvidia.com/gpu-driver-upgrade	Driver upgrade in progress
nvidia.com/gpu-ecc-error	Double-bit ECC hardware fault
nvidia.com/mig-config	MIG reconfiguration in progress

---

Complete Scheduling Flow

sequenceDiagram

participant User
participant API as API Server
participant Sched as Scheduler
participant Kubelet
participant Plugin as Device Plugin
participant Toolkit as nvidia-container-toolkit
participant Container

User->>API: Create Pod (nvidia.com/gpu: 1)
API->>Sched: Unscheduled Pod event
Sched->>Sched: Filter nodes (NodeResourcesFit: gpu >= 1)
Sched->>Sched: Score nodes
Sched->>API: Bind Pod → node-dgx-01
API->>Kubelet: Pod assigned (Watch event)
Kubelet->>Plugin: Allocate(GPU-3)
Plugin-->>Kubelet: /dev/nvidia3, CUDA_VISIBLE_DEVICES=3
Kubelet->>Toolkit: Create container with GPU devices
Toolkit->>Container: Inject GPU devices + CUDA libs
Container->>Container: nvidia-smi → sees GPU 3 only

---

Key Takeaways

Component	Responsibility
Device Plugin	Advertises GPU resources via ListAndWatch; allocates specific devices via Allocate
nvidia-container-toolkit	Injects /dev/nvidia* device files and CUDA libraries into the container at runtime
GPU Operator	Reconciles the full GPU software stack (driver, toolkit, plugin, monitoring) via ClusterPolicy
DCGM Exporter	Exposes per-GPU Prometheus metrics; fires alerts for ECC errors and utilization drops
MIG	Partitions one GPU into isolated hardware slices — each with its own SMs and HBM
Time-Slicing	Multiplexes one GPU across multiple Pods in time — no memory isolation
MIG Manager	Creates/destroys MIG partitions automatically based on ClusterPolicy
Health Monitor	Taints nodes on GPU fault — prevents scheduling on degraded hardware

The Kubernetes scheduler sees GPUs as opaque integer resources. Everything that makes them useful — driver compatibility, isolation, monitoring, partitioning, and health — is handled by the NVIDIA stack sitting between the scheduler's decision and the container's first CUDA call.