Dynamic Resource Allocation: The Future of GPU Scheduling in Kubernetes

The Device Plugin framework has been how Kubernetes schedules GPUs since 2018.

It works. But it has limits that become painful at DGX scale:

• GPUs are exposed as opaque integer counts — no attributes, no topology
• The scheduler cannot place a Pod on a node where GPUs are NVLink-connected
• Sharing a GPU between two Pods requires workarounds (MIG, time-slicing)
• A Pod requesting nvidia.com/gpu: 8 cannot express "give me 8 GPUs on the same NVSwitch domain"

Dynamic Resource Allocation (DRA) — graduated to beta in Kubernetes 1.31 — is the replacement. It replaces opaque integer counting with a rich, structured model: devices have attributes, users write selectors, and the scheduler understands topology.

---

Device Plugin Limitations

The Device Plugin model works like a counter: the plugin advertises nvidia.com/gpu: 8 and the scheduler subtracts integers.

flowchart LR

DevicePlugin["Device Plugin\n(ListAndWatch)"]

-->|"nvidia.com/gpu: 8\n(opaque count)"| Kubelet

Kubelet

-->|"8 integer slots"| Scheduler

Scheduler

-->|"subtract 1\nper Pod"| Allocation["nvidia.com/gpu: 1\nallocated to Pod"]

What the scheduler cannot see:

Information	Device Plugin	Consequence
GPU model (H100 vs A100)	❌	Can't select by GPU type without node labels
GPU memory (80 GB vs 40 GB)	❌	Can't select by memory requirement
NVLink topology	❌	Can't guarantee 8 GPUs are NVSwitch-connected
MIG partition attributes	❌	MIG profiles are separate resource names, not attributes
Shared access	❌	One Pod per GPU slot (no DRA-level sharing)

---

DRA Architecture

DRA introduces four new concepts: DeviceClass, ResourceSlice, ResourceClaim, and ResourceClaimTemplate.

flowchart TB

Driver["DRA Driver\n(NVIDIA GPU Driver)"]

-->|"publishes"| RS["ResourceSlice\n(per node — device inventory\nwith attributes)"]

User["User / Operator"]

-->|"creates"| RC["ResourceClaim\n(what devices I need\n+ CEL selectors)"]

Scheduler

-->|"matches claims\nto ResourceSlices"| Alloc["Allocation\n(specific device IDs\nassigned to claim)"]

Pod

-->|"references"| RC

RC

-->|"fulfilled by"| Alloc

DeviceClass

A DeviceClass is a cluster-wide template that defines a category of device and default constraints. Cluster admins create them; users reference them in claims.

apiVersion: resource.k8s.io/v1beta1
kind: DeviceClass
metadata:
  name: gpu.nvidia.com
spec:
  selectors:
  - cel:
      expression: device.driver == "gpu.resource.nvidia.com"

For specialized access (e.g., only H100 SXM5 nodes):

apiVersion: resource.k8s.io/v1beta1
kind: DeviceClass
metadata:
  name: h100-sxm5.nvidia.com
spec:
  selectors:
  - cel:
      expression: >
        device.driver == "gpu.resource.nvidia.com" &&
        device.attributes["model"].string == "H100 SXM5 80GB"

ResourceSlice

A ResourceSlice is published per-node by the DRA driver. It advertises the actual devices on that node with their full attribute set.

apiVersion: resource.k8s.io/v1beta1
kind: ResourceSlice
metadata:
  name: node-dgx-01-gpus
spec:
  driver: gpu.resource.nvidia.com
  nodeName: node-dgx-01
  pool:
    name: node-dgx-01
    generation: 1
    resourceSliceCount: 1
  devices:
  - name: gpu-0
    attributes:
      model:
        string: "H100 SXM5 80GB"
      memory:
        quantity: "80Gi"
      nvlinkDomain:
        string: "nvswitch-0"
      computeCapability:
        version: "9.0"
      uuid:
        string: "GPU-a1b2c3d4-..."
  - name: gpu-1
    attributes:
      model:
        string: "H100 SXM5 80GB"
      memory:
        quantity: "80Gi"
      nvlinkDomain:
        string: "nvswitch-0"   # same NVSwitch domain as gpu-0
      uuid:
        string: "GPU-e5f6a7b8-..."
  # ... gpu-2 through gpu-7

The scheduler reads all ResourceSlice objects — it has a complete, structured view of every device in the cluster.

ResourceClaim

A ResourceClaim is what a user (or operator) creates to request devices. It uses CEL expressions to express requirements.

apiVersion: resource.k8s.io/v1beta1
kind: ResourceClaim
metadata:
  name: training-gpus
  namespace: team-research
spec:
  devices:
    requests:
    - name: gpus
      deviceClassName: gpu.nvidia.com
      count: 8
      selectors:
      - cel:
          expression: >
            device.attributes["memory"].quantity >= "80Gi" &&
            device.attributes["model"].string == "H100 SXM5 80GB"
      allocationMode: ExactCount

The scheduler finds a node where 8 GPUs matching all selectors are available and unallocated, then writes the allocation back into the claim.

---

Pod Integration

A Pod references a ResourceClaim by name. When the Pod is scheduled, Kubernetes ensures the claim's allocated devices are on the same node.

apiVersion: v1
kind: Pod
metadata:
  name: llm-training
spec:
  resourceClaims:
  - name: training-gpus
    resourceClaimName: training-gpus   # references the claim above
  containers:
  - name: trainer
    image: nvcr.io/nvidia/pytorch:24.10-py3
    resources:
      claims:
      - name: training-gpus           # which claim this container uses
    command: ["torchrun", "--nproc-per-node=8", "train.py"]

The kubelet and DRA driver together ensure the allocated GPUs are injected into the container — analogous to the Device Plugin's Allocate gRPC call, but driven by the structured allocation in the claim rather than by opaque device IDs.

---

Topology-Aware Allocation

The most important DRA capability for DGX workloads is topology constraints on allocation.

An 8-GPU training job should always get 8 GPUs on the same NVSwitch domain. With Device Plugins, this requires manual scheduling heuristics (node affinity labels, custom scheduler plugins). With DRA:

spec:
  devices:
    requests:
    - name: gpus
      deviceClassName: gpu.nvidia.com
      count: 8
      selectors:
      - cel:
          expression: >
            device.attributes["nvlinkDomain"].string == "nvswitch-0"

The scheduler only allocates this claim on a node where 8 available GPUs all have nvlinkDomain == "nvswitch-0". If no such set exists on any node, the Pod stays Pending — correctly, rather than silently running with non-NVLink-connected GPUs and wondering why AllReduce is slow.

---

ResourceClaimTemplate — Per-Pod Claims

For workloads like Deployments where each replica needs its own devices, use ResourceClaimTemplate:

apiVersion: resource.k8s.io/v1beta1
kind: ResourceClaimTemplate
metadata:
  name: inference-gpu-template
  namespace: inference
spec:
  spec:
    devices:
      requests:
      - name: gpu
        deviceClassName: gpu.nvidia.com
        count: 1
        selectors:
        - cel:
            expression: device.attributes["memory"].quantity >= "40Gi"

# In the Pod template of a Deployment:
spec:
  resourceClaims:
  - name: inference-gpu
    resourceClaimTemplateName: inference-gpu-template

Each replica Pod gets its own independent ResourceClaim generated from the template. When the Pod is deleted, its claim is deleted too.

---

NVIDIA GPU DRA Driver

NVIDIA ships a DRA driver (gpu.resource.nvidia.com) that:

1. Publishes ResourceSlice objects for every GPU on the node (with model, memory, NVLink topology, MIG capability, etc.)
2. Handles NodePrepareResources and NodeUnprepareResources gRPC calls from the kubelet to set up and tear down GPU access
3. Supports MIG partitions as first-class devices with their own attributes

flowchart LR

NVIDIADriver["NVIDIA DRA Driver\n(DaemonSet)"]

-->|"publishes"| RS["ResourceSlice\n(all GPU attributes)"]

NVIDIADriver

-->|"NodePrepareResources"| Setup["Creates /dev/nvidia*\nconfigures environment"]

Setup

-->Container["Container\n(CUDA_VISIBLE_DEVICES set\nfrom claim allocation)"]

The GPU Operator (ClusterPolicy) deploys the DRA driver alongside the legacy Device Plugin during the transition period — both can run simultaneously.

---

DRA vs Device Plugin

Capability	Device Plugin	DRA
Device advertisement	Integer count per resource name	Structured attributes (model, memory, topology)
Selection criteria	None (only count)	CEL expressions on any attribute
Topology awareness	❌	✅ (same NVSwitch domain, same NUMA node)
MIG support	Separate resource names per profile	First-class devices with attributes
Sharing	Via time-slicing or MPS workarounds	Native via allocationMode: All or admin access
Dynamic partitioning	❌	✅ (driver can create MIG partitions on demand)
Scheduler visibility	Opaque	Full — scheduler reads ResourceSlice
Migration effort	None (existing)	New driver, new Pod spec fields

---

Coexistence During Migration

Device Plugins and DRA can run on the same cluster simultaneously. A node can advertise nvidia.com/gpu (Device Plugin path) and also publish ResourceSlice (DRA path).

flowchart LR

Node["DGX Node"]

-->DP["Device Plugin\nnvidia.com/gpu: 8\n(legacy Pods)"]

Node

-->DRA["DRA Driver\nResourceSlice\n(new Pods)"]

DP

-->OldPod["Pod using\nnvidia.com/gpu: 1"]

DRA

-->NewPod["Pod using\nResourceClaim"]

Migration path:

1. Deploy the NVIDIA DRA driver alongside the existing GPU Operator
2. New workloads opt in by using resourceClaims in their Pod spec
3. Legacy workloads continue using nvidia.com/gpu limits unchanged
4. Once all workloads are migrated, Device Plugin can be disabled

---

Kubernetes Version Timeline

K8s Version	DRA Status	Key change
1.26	Alpha	Initial DRA API
1.30	Alpha	Structured Parameters introduced
1.31	Beta	Structured Parameters stable; production-ready
1.32	Beta	ResourceSlice partition API (for MIG)
~1.33	Expected GA	Full stability

DRA beta in 1.31 means the API is stable enough to deploy in production, though the feature gate (DynamicResourceAllocation) must still be enabled explicitly.

---

Key Takeaways

Concept	Purpose
DeviceClass	Cluster-admin-defined template for a device category
ResourceSlice	Per-node device inventory published by the DRA driver
ResourceClaim	User request for specific devices with CEL selectors
ResourceClaimTemplate	Per-Pod claim template for Deployments/StatefulSets
CEL selectors	Express constraints: memory, model, NVLink domain, MIG capability
Topology awareness	Guarantee 8 GPUs are NVSwitch-connected — impossible with Device Plugins
Coexistence	Device Plugins and DRA work simultaneously during migration

DRA is not a drop-in replacement for Device Plugins yet — it requires Pod spec changes, a new DRA driver, and cluster feature gates. But it is the only path to topology-aware GPU allocation and per-device attribute selection at DGX scale. Any cluster running K8s 1.31+ should be evaluating migration.