Kueue: Kubernetes-Native Job Queuing and Quota Management

Imagine a company with 64 H100 GPUs shared across three teams.

• Team A (research) submits a 32-GPU training job.
• Team B (production) submits a 32-GPU fine-tuning job.
• Team C (experiments) submits 10 separate 8-GPU jobs.

How should Kubernetes decide what runs first?

Without Kueue, there is no answer.

Pods are scheduled on a first-come, first-served basis.

There are no quotas.

There is no fairness.

There is no concept of a "job" at all — just individual Pods.

Kueue is the CNCF project that solves this.

---

The Problem with Vanilla Kubernetes

Kubernetes is excellent at scheduling individual Pods.

It is not designed for batch workloads.

flowchart LR

Team1["Team A\nsubmits 100 pods"]

-->Scheduler["Kubernetes Scheduler"]

Team2["Team B\nsubmits 50 pods"]

-->Scheduler

Team3["Team C\nsubmits 20 pods"]

-->Scheduler

Scheduler

-->Nodes["First-come\nfirst-served"]

Problems this creates:

Problem	What Happens
No quotas	One team can consume all cluster resources
No queuing	New jobs grab resources immediately, existing jobs starve
No gang scheduling	Jobs start partially — 4 of 8 workers run, the job hangs
No priorities	Production jobs compete equally with experiment pods
No fairness	A team that submits 1000 pods at once wins

---

What Kueue Adds

flowchart LR

Jobs["Jobs / PyTorchJobs\n/ RayJobs"]

-->LocalQueue

LocalQueue

-->ClusterQueue

ClusterQueue

-->|"Admission check\n(quotas, gang, priority)"| Nodes

ClusterQueue

-->Cohort["Cohort\n(shared quota pool)"]

Kueue adds a queuing layer between job submission and pod scheduling.

No pod starts until Kueue admits the workload.

---

Core Concepts

Kueue introduces five core objects.

flowchart TB

ResourceFlavor

-->ClusterQueue

ClusterQueue

-->LocalQueue

LocalQueue

-->Workload

ClusterQueue

-->Cohort

Object	Scope	Purpose
ResourceFlavor	Cluster	Labels a type of resource (e.g., H100, spot)
ClusterQueue	Cluster	Quota pool — how much of each flavor a team can use
LocalQueue	Namespace	Team's entry point into a ClusterQueue
Workload	Namespace	Kueue's internal representation of a job
Cohort	Cluster	Group of ClusterQueues that can share unused quota

---

ResourceFlavor

A ResourceFlavor describes a type of hardware.

Think of it as a label that says:

"These are the H100 nodes."

apiVersion: kueue.x-k8s.io/v1beta1
kind: ResourceFlavor
metadata:
  name: nvidia-h100
spec:
  nodeLabels:
    nvidia.com/gpu.product: "NVIDIA-H100-SXM5-80GB"

A cluster might have multiple flavors:

# GPU nodes
nvidia-h100: 64 GPUs across 8 DGX nodes

# CPU-only nodes
cpu-only: 500 vCPUs

ResourceFlavors allow Kueue to enforce quotas per hardware type.

---

ClusterQueue

A ClusterQueue is the quota pool.

It defines how many resources are available for a group of teams.

apiVersion: kueue.x-k8s.io/v1beta1
kind: ClusterQueue
metadata:
  name: research-team-cq
spec:
  namespaceSelector:
    matchLabels:
      team: research
  resourceGroups:
  - coveredResources: ["nvidia.com/gpu", "cpu", "memory"]
    flavors:
    - name: nvidia-h100
      resources:
      - name: nvidia.com/gpu
        nominalQuota: 32       # Team gets 32 H100s
        borrowingLimit: 16     # Can borrow up to 16 more from cohort
        lendingLimit: 8        # Lends up to 8 unused GPUs to cohort

Key quota fields:

Field	Meaning
nominalQuota	Team's guaranteed allocation
borrowingLimit	Max extra resources borrowable from cohort
lendingLimit	Max of own quota lendable to others

---

LocalQueue

A LocalQueue is the namespace-level entry point.

Teams submit jobs to their LocalQueue, not directly to a ClusterQueue.

apiVersion: kueue.x-k8s.io/v1beta1
kind: LocalQueue
metadata:
  name: my-queue
  namespace: research
spec:
  clusterQueue: research-team-cq

flowchart LR

subgraph NamespaceA["Namespace: research"]
LQA["LocalQueue\nmy-queue"]
end

subgraph NamespaceB["Namespace: production"]
LQB["LocalQueue\nprod-queue"]
end

LQA & LQB

-->CQ["ClusterQueue\nresearch-team-cq"]

Multiple namespaces can share one ClusterQueue.

---

Workload

A Workload is Kueue's internal representation of a job.

When you submit a PyTorchJob or batch/Job, Kueue automatically creates a Workload object.

flowchart LR

PyTorchJob["PyTorchJob\n(user submits)"]

-->Kueue["Kueue Controller"]

Kueue

-->Workload["Workload object\n(internal)"]

Workload

-->|"Admitted?"| Pods["Pods created"]

The Workload describes:

• How many pods it needs
• What resources each pod requires
• Which LocalQueue it belongs to
• Its priority

No pods are created until the Workload is admitted.

---

The Admission Flow

This is the heart of Kueue.

flowchart TD

Submit["Job submitted\n(PyTorchJob / Job / RayJob)"]

-->WL["Kueue creates Workload object"]

WL

-->Queue["Workload waits in LocalQueue"]

Queue

-->Check["ClusterQueue admission check"]

Check

-->|"Quota available\n+ gang satisfied"| Admit["Workload admitted"]

Check

-->|"Not enough quota"| Wait["Workload stays in queue\n(pending)"]

Admit

-->Pods["Pods created and scheduled"]

Kueue checks two things before admitting:

1. Does the ClusterQueue have enough quota?
2. Can all pods start together? (gang scheduling)

If both pass — the workload is admitted, pods are created, and the Kubernetes scheduler places them on nodes.

If either fails — the workload waits in the queue.

---

Gang Scheduling

Gang scheduling means:

All pods start together, or none start.

This is essential for distributed training.

Without gang scheduling:

flowchart TD

Job["8-worker PyTorchJob"]

-->4Workers["4 workers start\n(4 nodes available)"]

4Workers

-->Hang["Job hangs waiting\nfor 4 more workers"]

Kueue prevents this.

It holds the workload in the queue until all requested resources are simultaneously available.

flowchart TD

Job["8-worker PyTorchJob"]

-->Queue["Held in Kueue queue"]

Queue

-->Check["8 GPUs available simultaneously?"]

Check

-->|"Yes"| AllStart["All 8 pods created\ntogether"]

Check

-->|"No"| Wait["Wait"]

Gang scheduling is the single most important feature for multi-node AI training.

---

Priority

Workloads can have priorities.

apiVersion: kueue.x-k8s.io/v1beta1
kind: WorkloadPriorityClass
metadata:
  name: production
value: 1000
description: "Production fine-tuning jobs"

---
apiVersion: kueue.x-k8s.io/v1beta1
kind: WorkloadPriorityClass
metadata:
  name: research
value: 500
description: "Research experiments"

Higher priority workloads are admitted first from the queue.

A queue with 10 research jobs and 1 production job:

flowchart LR

Queue["Queue"]

-->Prod["Production job\npriority: 1000"]

Queue

-->R1["Research job 1\npriority: 500"]

Queue

-->R2["Research job 2\npriority: 500"]

Queue

-->R3["Research job 3\npriority: 500"]

Prod

-->|"Admitted first"| Run

---

Preemption

Preemption allows higher-priority workloads to evict lower-priority ones.

flowchart TD

Research["Research job\nusing all 32 GPUs"]

-->Running

ProdArrives["Production job\narrives (higher priority)"]

-->Kueue["Kueue triggers preemption"]

Kueue

-->Evict["Research job evicted\nback to queue"]

Evict

-->ProdRuns["Production job\nadmitted, runs"]

Evict

-->ResearchWait["Research job waits\nfor resources to free up"]

Preemption can be configured:

spec:
  preemption:
    reclaimWithinCohort: LowerPriority   # Reclaim from lower-priority jobs in cohort
    withinClusterQueue: LowerPriority    # Preempt lower-priority jobs in same queue

Options:

Policy	Meaning
Never	No preemption
LowerPriority	Evict lower-priority workloads first
LowerOrNewerEqualPriority	Also evict newer jobs of equal priority

---

Cohorts and Borrowing

A Cohort is a group of ClusterQueues that can share unused quota.

flowchart TB

subgraph Cohort["Cohort: ai-teams"]

CQ1["ClusterQueue\nresearch\nnominal: 32 GPUs"]

CQ2["ClusterQueue\nproduction\nnominal: 32 GPUs"]

CQ3["ClusterQueue\nexperiments\nnominal: 16 GPUs"]

end

Each queue is assigned to the same cohort:

spec:
  cohort: ai-teams

Now, if research is using only 16 of its 32 GPUs:

flowchart LR

Research["Research\n16/32 GPUs used\n16 unused"]

-->Cohort["Available to borrow\n(up to lendingLimit)"]

Production["Production\nneeds 40 GPUs\nborrowing 8 from research"]

-->Cohort

Production can borrow the unused quota (up to borrowingLimit).

When Research submits a new job and needs the GPUs back, Production's borrowed workloads can be preempted.

This is called quota reclamation.

---

Fair Sharing

Without fair sharing, a team that submits first gets resources first.

With fair sharing, Kueue tracks how much each queue has used historically and gives preference to under-utilized queues.

flowchart LR

QueueA["Queue A\nused 80% of quota"]

QueueB["Queue B\nused 20% of quota"]

Cohort["Cohort pool\n(unused quota)"]

Cohort

-->|"Fair share: prefer Queue B"| QueueB

This prevents one team from monopolizing shared resources over time.

Configure it on the ClusterQueue:

spec:
  fairSharing:
    weight: "1"   # Equal weight with other queues in cohort

---

Integrations

Kueue works with multiple job types out of the box.

Job Type	Framework
batch/Job	Standard Kubernetes Job
PyTorchJob	Kubeflow Training Operator
TFJob	Kubeflow TensorFlow
MPIJob	MPI (Horovod)
RayJob	KubeRay
RayCluster	KubeRay
JobSet	Kubernetes JobSet
StatefulSet	Kubernetes StatefulSet

To enable Kueue for a job, add a label:

apiVersion: batch/v1
kind: Job
metadata:
  name: training-job
  namespace: research
  labels:
    kueue.x-k8s.io/queue-name: my-queue   # ← tells Kueue which LocalQueue to use
spec:
  parallelism: 8
  template:
    spec:
      containers:
      - name: trainer
        resources:
          limits:
            nvidia.com/gpu: "1"

For PyTorchJob:

apiVersion: kubeflow.org/v1
kind: PyTorchJob
metadata:
  name: llama-training
  labels:
    kueue.x-k8s.io/queue-name: my-queue   # ← same label

---

Complete Multi-Team Setup

A real DGX cluster serving three teams:

# ResourceFlavor — H100 nodes
apiVersion: kueue.x-k8s.io/v1beta1
kind: ResourceFlavor
metadata:
  name: h100
spec:
  nodeLabels:
    nvidia.com/gpu.product: "NVIDIA-H100-SXM5-80GB"

---
# ClusterQueues — one per team
apiVersion: kueue.x-k8s.io/v1beta1
kind: ClusterQueue
metadata:
  name: research-cq
spec:
  cohort: ai-teams
  resourceGroups:
  - coveredResources: ["nvidia.com/gpu"]
    flavors:
    - name: h100
      resources:
      - name: nvidia.com/gpu
        nominalQuota: 32
        borrowingLimit: 16
        lendingLimit: 8
  preemption:
    reclaimWithinCohort: LowerPriority

---
apiVersion: kueue.x-k8s.io/v1beta1
kind: ClusterQueue
metadata:
  name: production-cq
spec:
  cohort: ai-teams
  resourceGroups:
  - coveredResources: ["nvidia.com/gpu"]
    flavors:
    - name: h100
      resources:
      - name: nvidia.com/gpu
        nominalQuota: 24
        borrowingLimit: 24
        lendingLimit: 4
  preemption:
    reclaimWithinCohort: LowerPriority
    withinClusterQueue: LowerPriority

---
apiVersion: kueue.x-k8s.io/v1beta1
kind: ClusterQueue
metadata:
  name: experiments-cq
spec:
  cohort: ai-teams
  resourceGroups:
  - coveredResources: ["nvidia.com/gpu"]
    flavors:
    - name: h100
      resources:
      - name: nvidia.com/gpu
        nominalQuota: 8
        borrowingLimit: 40
        lendingLimit: 8

The cohort layout:

flowchart TB

subgraph Cohort["Cohort: ai-teams (64 H100 GPUs total)"]

Research["Research\nnominal: 32 GPUs"]

Production["Production\nnominal: 24 GPUs"]

Experiments["Experiments\nnominal: 8 GPUs"]

end

Research & Production & Experiments

<-->|"Borrow unused quota"| Cohort

---

Observing Queue State

# List all workloads and their status
kubectl get workloads -n research

# NAME                 QUEUE    ADMITTED   AGE
# llama-training-wl    my-queue True       2m
# gpt-finetune-wl      my-queue False      5s   ← waiting in queue

# Check ClusterQueue usage
kubectl get clusterqueue research-cq -o yaml

# status:
#   admittedWorkloads: 1
#   pendingWorkloads: 1
#   reservingWorkloads: 0
#   flavorsUsage:
#   - name: h100
#     resources:
#     - name: nvidia.com/gpu
#       borrowed: 0
#       total: 32

# Check LocalQueue
kubectl get localqueue my-queue -n research

# NAME       CLUSTERQUEUE   PENDING   ADMITTED
# my-queue   research-cq    1         1

---

Kueue vs Plain Kubernetes Scheduler

Feature	Kubernetes Scheduler	Kueue
Schedules Pods	Yes	No (delegates to scheduler)
Job-level awareness	No	Yes
Resource quotas	No (ResourceQuota is per-namespace, not gang-aware)	Yes
Gang scheduling	No	Yes
Queuing / backlog	No	Yes
Priority and preemption	Pod-level only	Workload-level
Borrowing across teams	No	Yes (Cohorts)
Fair sharing	No	Yes
Multi-framework (Job + PyTorchJob + Ray)	N/A	Yes

Kueue does not replace the Kubernetes scheduler.

It sits above the scheduler.

flowchart TB

Jobs["Jobs / PyTorchJobs / RayJobs"]

-->Kueue["Kueue\n(admission control, queuing, quotas)"]

Kueue

-->|"Creates Pods after admission"| Scheduler["Kubernetes Scheduler\n(places Pods on nodes)"]

Scheduler

-->Nodes

The scheduler still handles node selection.

Kueue handles job ordering, admission, and quota enforcement.

---

Key Takeaways

Concept	Purpose
ResourceFlavor	Labels hardware types (H100, CPU, spot) for quota tracking
ClusterQueue	Quota pool — guaranteed allocation per team, borrow/lend limits
LocalQueue	Team's namespace-level entry point into a ClusterQueue
Workload	Internal representation of a job; no pods until admitted
Gang scheduling	All pods start together — essential for distributed training
Cohort	Cluster of ClusterQueues that share unused quota
Borrowing	Temporary use of another queue's unused quota
Preemption	Higher-priority jobs can evict lower-priority ones to reclaim quota
Fair sharing	Distributes idle resources proportionally based on past usage

Kueue turns a GPU cluster into a multi-tenant platform — teams get guaranteed quotas, can burst into unused capacity, and production jobs always beat experiments without any manual intervention.