GPU Autoscaling on Kubernetes: KEDA, HPA, and Cluster Autoscaler

GPU autoscaling operates at three independent layers.

flowchart TB

L1["Layer 1: Pod count\n(HPA / KEDA)\nScale how many replica Pods run"]

L2["Layer 2: Node count\n(Cluster Autoscaler)\nScale how many GPU nodes exist"]

L3["Layer 3: Resource per Pod\n(VPA)\nResize CPU/memory around GPU Pods"]

L1

-->|"Pending Pods trigger"| L2

Each layer has different triggers, different timescales, and different constraints when GPU workloads are involved.

This post covers all three, with GPU-specific concerns: DCGM metrics, training job protection, and queue-depth-driven scaling with Kueue.

---

Layer 1A: HPA with DCGM Custom Metrics

The built-in HPA scales on CPU and memory. For GPU inference replicas, you want to scale on GPU utilization — which means feeding DCGM metrics into the Kubernetes metrics API.

The Metrics Pipeline

flowchart LR

DCGM["DCGM Exporter\n(DaemonSet)\nper-GPU Prometheus metrics"]

-->Prom["Prometheus\n(scrapes DCGM)"]

Prom

-->Adapter["Prometheus Adapter\n(bridges Prometheus → K8s metrics API)"]

Adapter

-->|"/apis/custom.metrics.k8s.io"| K8sAPI["Kubernetes\nMetrics API"]

K8sAPI

-->HPA["HPA\n(reads custom metrics\nand scales Deployment)"]

Four components must all be running:

Component	Role
DCGM Exporter	Exposes per-GPU Prometheus metrics (deployed by GPU Operator)
Prometheus	Scrapes DCGM Exporter and stores time-series
Prometheus Adapter	Converts Prometheus queries into Kubernetes custom metrics API
HPA	Reads the custom metric, scales the Deployment

Prometheus Adapter Configuration

The adapter needs a rule that maps a Prometheus query to a Kubernetes metric name:

# prometheus-adapter ConfigMap
rules:
- seriesQuery: 'DCGM_FI_DEV_GPU_UTIL{namespace!="",pod!=""}'
  resources:
    overrides:
      namespace:
        resource: namespace
      pod:
        resource: pod
  name:
    matches: "DCGM_FI_DEV_GPU_UTIL"
    as: "gpu_utilization"
  metricsQuery: 'avg(DCGM_FI_DEV_GPU_UTIL{<<.LabelMatchers>>}) by (<<.GroupBy>>)'

HPA Targeting GPU Utilization

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: inference-hpa
  namespace: inference
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: llama3-8b-nim
  minReplicas: 1
  maxReplicas: 8
  metrics:
  - type: Pods
    pods:
      metric:
        name: gpu_utilization         # registered by Prometheus Adapter
      target:
        type: AverageValue
        averageValue: "70"            # target 70% GPU utilization per Pod
  behavior:
    scaleUp:
      stabilizationWindowSeconds: 60  # wait 60s before scaling up
      policies:
      - type: Pods
        value: 2
        periodSeconds: 60             # add max 2 replicas per minute
    scaleDown:
      stabilizationWindowSeconds: 300 # wait 5 min before scaling down

Important: GPU utilization (DCGM_FI_DEV_GPU_UTIL) is a per-GPU SM utilization percentage. For a multi-GPU Pod (e.g., NIM on 8 GPUs), the adapter averages across all GPUs in the Pod. Target 60–80% — below that is wasteful, above is latency-degrading.

---

Layer 1B: KEDA — Simpler GPU Autoscaling

The Prometheus Adapter is powerful but operationally complex (a separate Deployment, ConfigMap rule syntax, API aggregation registration). KEDA (Kubernetes Event-Driven Autoscaling) is the simpler alternative.

flowchart LR

KEDA["KEDA\n(operator)"]

-->ScaledObject["ScaledObject\n(your config)"]

ScaledObject

-->|"Prometheus trigger"| Prom["Prometheus\n(DCGM metrics)"]

KEDA

-->|"drives"| HPA["Kubernetes HPA\n(KEDA creates + manages)"]

HPA

-->Deployment["Deployment\n(inference replicas)"]

KEDA creates and manages the HPA on your behalf. You write a ScaledObject instead of wiring up the Prometheus Adapter.

KEDA ScaledObject for GPU Utilization

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: nim-gpu-scaler
  namespace: inference
spec:
  scaleTargetRef:
    name: llama3-8b-nim
  minReplicaCount: 1
  maxReplicaCount: 8
  cooldownPeriod: 300           # seconds before scale-down after last trigger

  triggers:
  - type: prometheus
    metadata:
      serverAddress: http://prometheus.monitoring.svc.cluster.local:9090
      metricName: gpu_utilization
      query: |
        avg(DCGM_FI_DEV_GPU_UTIL{
          namespace="inference",
          pod=~"llama3-8b-nim-.*"
        })
      threshold: "70"           # scale up when avg GPU util > 70%
      activationThreshold: "10" # don't scale below 10% (avoid idle scaling)

KEDA polls Prometheus on its own schedule (default every 30 s) and adjusts the HPA desiredReplicas accordingly. No Prometheus Adapter needed.

KEDA for Queue-Depth Scaling

For batch inference or training queues, scaling on GPU utilization is wrong — if the queue is long but GPUs are full, you want more Pods, not fewer. Scale on queue depth instead:

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: batch-inference-scaler
  namespace: inference
spec:
  scaleTargetRef:
    name: batch-inference-worker
  minReplicaCount: 0           # scale to zero when queue empty
  maxReplicaCount: 16
  cooldownPeriod: 600

  triggers:
  - type: prometheus
    metadata:
      serverAddress: http://prometheus.monitoring.svc.cluster.local:9090
      metricName: kueue_pending_workloads
      query: |
        kueue_pending_workloads{
          cluster_queue="inference-queue",
          status="active"
        }
      threshold: "1"            # one pending workload → spin up a worker

Scale-to-zero is especially useful for batch workloads: when the queue drains, all workers terminate, freeing GPUs for other workloads.

KEDA + Kueue Integration

KEDA and Kueue complement each other:

flowchart LR

Job["New batch Job\nsubmitted"]

-->Kueue["Kueue\n(admits when quota available)"]

Kueue

-->|"pending count +1"| Prom["kueue_pending_workloads\nmetric in Prometheus"]

Prom

-->KEDA["KEDA\n(reads metric)"]

KEDA

-->|"scale up"| Workers["Batch inference\nworker Pods"]

Workers

-->|"dequeue work"| Kueue

Workers

-->|"pending count → 0"| ScaleDown["KEDA scales\nback to 0"]

Kueue manages quota and admission; KEDA manages how many worker Pods are running. The two are decoupled — Kueue doesn't know about KEDA, and KEDA doesn't know about Kueue quotas.

---

Layer 2: Cluster Autoscaler for GPU Nodes

The HPA/KEDA layer scales the number of Pods. When new Pods are Pending because no node has enough GPU capacity, the Cluster Autoscaler provisions new nodes.

flowchart LR

Pending["Pending Pod\n(no node with free GPUs)"]

-->CA["Cluster Autoscaler\n(polls for unschedulable Pods)"]

CA

-->|"chooses node group\nwith cheapest fit"| Cloud["Cloud Provider\n(AWS, GCP, Azure)"]

Cloud

-->|"provision + join"| NewNode["New GPU node\n(H100 × 8)"]

NewNode

-->Schedule["Pod scheduled\non new node"]

GPU Node Groups

GPU nodes are expensive and heterogeneous. Cluster Autoscaler manages separate node groups per GPU type:

# Example: two node groups in GKE/AWS
node-group-h100-8x:
  minSize: 0
  maxSize: 10
  gpuType: nvidia-h100
  gpuCount: 8

node-group-l40s-4x:
  minSize: 0
  maxSize: 20
  gpuType: nvidia-l40s
  gpuCount: 4

The Cluster Autoscaler chooses which node group to expand based on the expander strategy:

Expander	Selection strategy
least-waste	Picks node group that wastes fewest resources (best bin-packing)
priority	User-defined priority order — use to prefer cheaper node types first
random	Random (default)
price	Cheapest node group that satisfies the Pod (cloud-provider-specific)

For GPU clusters, least-waste or priority is almost always the right choice — GPU nodes are too expensive to provision one size larger than needed.

Scale-Down Behavior

Cluster Autoscaler removes nodes that have been underutilized for a configurable window:

# Cluster Autoscaler flags
--scale-down-enabled=true
--scale-down-delay-after-add=10m       # don't scale down for 10 min after scale-up
--scale-down-unneeded-time=10m         # node must be underutilized for 10 min
--scale-down-utilization-threshold=0.5 # "underutilized" = below 50% requested

For GPU nodes, 10 minutes is often too short — node provisioning takes 3–8 minutes, and a quick scale-down followed by immediate scale-up wastes that time. Use 30–60 minutes for GPU node groups.

Protecting Training Jobs from Scale-Down

Training jobs cannot be interrupted. A mid-training PyTorchJob evicted by scale-down loses hours of checkpoint progress.

Prevent eviction with the safe-to-evict annotation:

apiVersion: v1
kind: Pod
metadata:
  name: pytorch-worker-0
  annotations:
    cluster-autoscaler.kubernetes.io/safe-to-evict: "false"

With this annotation, Cluster Autoscaler will not evict the Pod during scale-down — even if the node is otherwise underutilized. The node will not be removed until the training job completes and the Pod terminates naturally.

For PyTorchJobs managed by the Kubeflow Training Operator, add this annotation in the Pod template:

apiVersion: kubeflow.org/v1
kind: PyTorchJob
spec:
  pytorchReplicaSpecs:
    Worker:
      template:
        metadata:
          annotations:
            cluster-autoscaler.kubernetes.io/safe-to-evict: "false"

Scale-Down Delay Strategy for Training

The recommended approach for clusters running both training and inference:

flowchart LR

TrainingNode["GPU node\nrunning PyTorchJob"]

-->|"safe-to-evict=false"| NoEvict["CA skips eviction\n(node stays up)"]

InferenceNode["GPU node\nidle (no Pods)"]

-->|"underutilized > 30 min"| ScaleDown["CA terminates node\n(saves GPU cost)"]

Inference nodes (stateless, evictable) scale down quickly. Training nodes stay up until the job finishes.

---

Autoscaling Limits with Gang-Scheduled Jobs

Gang scheduling and autoscaling have a fundamental conflict:

flowchart TD

Job["PyTorchJob\n8 workers × 8 GPUs = 64 GPUs"]

-->Kueue["Kueue admits the job\n(all 64 GPUs reserved)"]

Kueue

-->Run["Workers running\nAllReduce in progress"]

Run

-->CA["Cluster Autoscaler sees\nsome nodes at 30% CPU\n(GPU is busy but CPU is idle)"]

CA

-->|"tries to scale down"| Evict["BLOCKED: safe-to-evict=false\nor PDB prevents eviction"]

Key rules for gang-scheduled training:

1. Always set safe-to-evict: "false" on training Pods
2. Use a PodDisruptionBudget with minAvailable: N matching worker count
3. Training does not benefit from scale-out — you cannot add workers mid-training to a fixed-topology AllReduce job (unless using elastic training)

Elastic Training — The Exception

Elastic training with PyTorch's torchrun (--max-restarts, --rdzv-backend=etcd) can tolerate workers joining and leaving:

torchrun \
  --nnodes=4:8 \              # min 4 nodes, max 8 nodes
  --nproc-per-node=8 \
  --rdzv-backend=etcd \
  --rdzv-endpoint=etcd:2379 \
  train.py

With elastic training, the Cluster Autoscaler CAN add nodes to an in-progress training job — torchrun will rendezvous with the new workers and resume. This is how elastic distributed training integrates with Cluster Autoscaler for cost-efficient scale-out.

---

VPA — Right-Sizing CPU/Memory Around GPU Pods

GPU requests are static (nvidia.com/gpu: 8 is not VPA-adjustable — GPU allocation is discrete, not continuous). But the CPU and memory alongside a GPU Pod can be right-sized by the Vertical Pod Autoscaler:

apiVersion: autoscaling.k8s.io/v1
kind: VerticalPodAutoscaler
metadata:
  name: nim-vpa
  namespace: inference
spec:
  targetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: llama3-8b-nim
  updatePolicy:
    updateMode: "Off"          # recommendation-only — don't auto-restart Pods
  resourcePolicy:
    containerPolicies:
    - containerName: nim
      controlledResources: ["cpu", "memory"]
      minAllowed:
        cpu: "4"
        memory: "16Gi"
      maxAllowed:
        cpu: "32"
        memory: "128Gi"

VPA in Off mode only generates recommendations without restarting Pods. Check the VPA status to tune your Pod resource requests:

kubectl describe vpa nim-vpa -n inference
# Recommendation:
#   Container Recommendations:
#     Container Name: nim
#     Target: cpu: 12, memory: 48Gi
#     Lower Bound: cpu: 8, memory: 32Gi
#     Upper Bound: cpu: 24, memory: 80Gi

GPU Pods frequently over-request CPU and memory. VPA recommendations help right-size these, freeing headroom for more GPU Pods on the same node.

---

Complete Autoscaling Architecture

flowchart TB

subgraph Inference["Inference workloads (stateless)"]
  DCGM["DCGM metrics\n(GPU utilization)"]
  -->KEDA["KEDA\n(ScaledObject)"]
  -->|"scale replicas"| NIM["NIM Pods\n(inference replicas)"]
end

subgraph Training["Training workloads (stateful)"]
  Queue["Kueue queue depth\n(pending workloads)"]
  -->KEDA2["KEDA\n(queue trigger)"]
  -->|"scale workers"| Workers["Training worker Pods\n(safe-to-evict=false)"]
end

NIM

-->|"Pending Pods\n(no GPU capacity)"| CA["Cluster Autoscaler"]

Workers

-->|"scale-down blocked\n(safe-to-evict=false)"| CA

CA

-->NodeGroups["GPU Node Groups\n(H100 × 8, L40S × 4)"]

---

Key Takeaways

Layer	Tool	Trigger	GPU-specific concern
Pod replicas	KEDA + Prometheus	DCGM GPU utilization	Target 60–80%; don't scale training jobs
Pod replicas	KEDA + Prometheus	Kueue queue depth	Scale-to-zero for batch workers
Pod replicas	HPA + Prometheus Adapter	Same as KEDA	More complex setup; same effect
Node count	Cluster Autoscaler	Pending Pods	least-waste expander for GPU types
Node protection	safe-to-evict: "false"	Training job annotation	Prevents mid-training eviction
Pod resources	VPA (recommendation mode)	CPU/memory over-provisioning	GPU requests not VPA-adjustable
Elastic scaling	torchrun --nnodes min:max	Worker count during training	Only works with elastic training setup

GPU autoscaling is not simply "add HPA." Training jobs are fundamentally different from inference replicas: they form gangs, use NVLink for AllReduce, and cannot lose a worker mid-run. The right architecture separates inference scaling (KEDA on GPU utilization → fast scale-up/down) from training scaling (Kueue admission + Cluster Autoscaler + safe-to-evict annotations) and treats each workload class with the different guarantees it needs.