Kueue: Kubernetes-Native Job Queuing and Quota Management

Kueue: Kubernetes-Native Job Queuing and Quota Management

Imagine a company with 64 H100 GPUs shared across three teams.

• Team A (research) submits a 32-GPU training job.
• Team B (production) submits a 32-GPU fine-tuning job.
• Team C (experiments) submits 10 separate 8-GPU jobs.

How should Kubernetes decide what runs first?

Without Kueue, there is no answer.

Pods are scheduled on a first-come, first-served basis.

• There are no quotas.
• There is no fairness.
• There is no concept of a "job" at all — just individual Pods.

Kueue is the CNCF project that solves this.

---

The Problem with Vanilla Kubernetes

Kubernetes is excellent at scheduling individual Pods.

It is not designed for batch workloads.

flowchart TD

    Scheduler["Kubernetes Scheduler"]

    Team1["Team A <br/> submits 100 pods"]-->Scheduler
    Team2["Team B <br/> submits 50 pods"]-->Scheduler
    Team3["Team C <br/> submits 20 pods"]-->Scheduler
    
    Scheduler-->Nodes["First-come <br/> first-served"]

Problems this creates:

Problem	What Happens
No quotas	One team can consume all cluster resources
No queuing	New jobs grab resources immediately, existing jobs starve
No gang scheduling	Jobs start partially — 4 of 8 workers run, the job hangs
No priorities	Production jobs compete equally with experiment pods
No fairness	A team that submits 1000 pods at once wins

---

What Kueue Adds

flowchart TD

Jobs["Jobs / PyTorchJobs <br/> / RayJobs"]

    Jobs-->LocalQueue
    LocalQueue-->ClusterQueue
    
    ClusterQueue-->|"Admission check <br/> (quotas, gang, priority)"| Nodes
    ClusterQueue-->Cohort["Cohort <br/> (shared quota pool)"]

Kueue adds a queuing layer between job submission and pod scheduling.

No pod starts until Kueue admits the workload.

---

Core Concepts

Kueue introduces five core objects.

flowchart TB

ResourceFlavor -->ClusterQueue

ClusterQueue-->LocalQueue
LocalQueue-->Workload

ClusterQueue-->Cohort

Object	Scope	Purpose
ResourceFlavor	Cluster	Labels a type of resource (e.g., H100, spot)
ClusterQueue	Cluster	Quota pool — how much of each flavor a team can use
LocalQueue	Namespace	Team's entry point into a ClusterQueue
Workload	Namespace	Kueue's internal representation of a job
Cohort	Cluster	Group of ClusterQueues that can share unused quota

---

ResourceFlavor

A ResourceFlavor describes a type of hardware.

Think of it as a label that says:

"These are the H100 nodes."

    apiVersion: kueue.x-k8s.io/v1beta1
    kind: ResourceFlavor
    metadata:
      name: nvidia-h100
    spec:
      nodeLabels:
        nvidia.com/gpu.product: "NVIDIA-H100-SXM5-80GB"

A cluster might have multiple flavors:

    # GPU nodes
    nvidia-h100: 64 GPUs across 8 DGX nodes
    
    # CPU-only nodes
    cpu-only: 500 vCPUs

ResourceFlavors allow Kueue to enforce quotas per hardware type.

---

ClusterQueue

A ClusterQueue is the quota pool.

It defines how many resources are available for a group of teams.

    apiVersion: kueue.x-k8s.io/v1beta1
    kind: ClusterQueue
    metadata:
      name: research-team-cq
    spec:
      namespaceSelector:
        matchLabels:
          team: research
      resourceGroups:
      - coveredResources: ["nvidia.com/gpu", "cpu", "memory"]
        flavors:
        - name: nvidia-h100
          resources:
          - name: nvidia.com/gpu
            nominalQuota: 32       # Team gets 32 H100s
            borrowingLimit: 16     # Can borrow up to 16 more from cohort
            lendingLimit: 8        # Lends up to 8 unused GPUs to cohort

Key quota fields:

Field	Meaning
nominalQuota	Team's guaranteed allocation
borrowingLimit	Max extra resources borrowable from cohort
lendingLimit	Max of own quota lendable to others

---

LocalQueue

A LocalQueue is the namespace-level entry point.

Teams submit jobs to their LocalQueue, not directly to a ClusterQueue.

    apiVersion: kueue.x-k8s.io/v1beta1
    kind: LocalQueue
    metadata:
      name: my-queue
      namespace: research
    spec:
      clusterQueue: research-team-cq

flowchart TD

subgraph NamespaceA["Namespace: research"]
LQA["LocalQueue <br/> my-queue"]
end

subgraph NamespaceB["Namespace: production"]
LQB["LocalQueue <br/> prod-queue"]
end

LQA & LQB-->CQ["ClusterQueue <br/> research-team-cq"]

Multiple namespaces can share one ClusterQueue.

---

Workload

A Workload is Kueue's internal representation of a job.

When you submit a PyTorchJob or batch/Job, Kueue automatically creates a Workload object.

flowchart TD

PyTorchJob["PyTorchJob <br/> (user submits)"]
PyTorchJob-->Kueue["Kueue Controller"]
Kueue-->Workload["Workload object <br/> (internal)"]
Workload-->|"Admitted?"| Pods["Pods created"]

The Workload describes:

• How many pods it needs
• What resources each pod requires
• Which LocalQueue it belongs to
• Its priority

No pods are created until the Workload is admitted.

---

The Admission Flow

This is the heart of Kueue.

flowchart TD

Submit["Job submitted <br/> (PyTorchJob / Job / RayJob)"]
Submit-->WL["Kueue creates Workload object"]

WL-->Queue["Workload waits in LocalQueue"]
Queue-->Check["ClusterQueue admission check"]
Check-->|"Quota available <br/> + gang satisfied"| Admit["Workload admitted"]
Check-->|"Not enough quota"| Wait["Workload stays in queue <br/> (pending)"]

Admit-->Pods["Pods created and scheduled"]

Kueue checks two things before admitting:

1. Does the ClusterQueue have enough quota?
2. Can all pods start together? (gang scheduling)

If both pass — the workload is admitted, pods are created, and the Kubernetes scheduler places them on nodes.

If either fails — the workload waits in the queue.

---

Gang Scheduling

Gang scheduling means:

All pods start together, or none start.

This is essential for distributed training.

Without gang scheduling:

flowchart TD

    Job["8-worker PyTorchJob"]
    Job-->4Workers["4 workers start <br/> (4 nodes available)"]
    4Workers-->Hang["Job hangs waiting <br/> for 4 more workers"]

Kueue prevents this.

It holds the workload in the queue until all requested resources are simultaneously available.

flowchart TD

    Job["8-worker PyTorchJob"]
    Job-->Queue["Held in Kueue queue"]

    Queue-->Check["8 GPUs available simultaneously?"]
    Check-->|"Yes"| AllStart["All 8 pods created <br/> together"]
    Check-->|"No"| Wait["Wait"]

Gang scheduling is the single most important feature for multi-node AI training.

---

Priority

Workloads can have priorities.

    apiVersion: kueue.x-k8s.io/v1beta1
    kind: WorkloadPriorityClass
    metadata:
      name: production
    value: 1000
    description: "Production fine-tuning jobs"
    
    ---
    apiVersion: kueue.x-k8s.io/v1beta1
    kind: WorkloadPriorityClass
    metadata:
      name: research
    value: 500
    description: "Research experiments"

Higher priority workloads are admitted first from the queue.

A queue with 10 research jobs and 1 production job:

flowchart LR

    Queue["Queue"]
    Queue-->Prod["Production job <br/> priority: 1000"]
    
    Queue-->R1["Research job 1 <br/> priority: 500"]
    Queue-->R2["Research job 2 <br/> priority: 500"]
    Queue-->R3["Research job 3 <br/> priority: 500"]
    
    Prod-->|"Admitted first"| Run

---

Preemption

Preemption allows higher-priority workloads to evict lower-priority ones.

flowchart TD

    Research["Research job <br/> using all 32 GPUs"]
    Research-->Running

    ProdArrives["Production job <br/> arrives (higher priority)"]
    ProdArrives-->Kueue["Kueue triggers preemption"]

   
    Kueue-->Evict["Research job evicted <br/> back to queue"]
    Running--> Evict
    
    Evict-->ProdRuns["Production job <br/> admitted, runs"]
    Evict-->ResearchWait["Research job waits <br/> for resources to free up"]

Preemption can be configured:

    spec:
      preemption:
        reclaimWithinCohort: LowerPriority   # Reclaim from lower-priority jobs in cohort
        withinClusterQueue: LowerPriority    # Preempt lower-priority jobs in same queue

Options:

Policy	Meaning
Never	No preemption
LowerPriority	Evict lower-priority workloads first
LowerOrNewerEqualPriority	Also evict newer jobs of equal priority

---

Cohorts and Borrowing

A Cohort is a group of ClusterQueues that can share unused quota.

flowchart LR

    subgraph Cohort["Cohort: ai-teams"]
    
    CQ1["ClusterQueue <br/> research <br/> nominal: 32 GPUs"]
    CQ2["ClusterQueue <br/> production <br/> nominal: 32 GPUs"]
    CQ3["ClusterQueue <br/> experiments <br/> nominal: 16 GPUs"]
    
    end

Each queue is assigned to the same cohort:

    spec:
      cohort: ai-teams

Now, if research is using only 16 of its 32 GPUs:

flowchart LR

    Research["Research <br/> 16/32 GPUs used <br/> 16 unused"]
    Production["Production <br/> needs 40 GPUs <br/> borrowing 8 from research"]

    Research-->Cohort["Available to borrow <br/> (up to lendingLimit)"]
    Production-->Cohort

Production can borrow the unused quota (up to borrowingLimit).

When Research submits a new job and needs the GPUs back, Production's borrowed workloads can be preempted.

This is called quota reclamation.

---

Fair Sharing

Without fair sharing, a team that submits first gets resources first.

With fair sharing, Kueue tracks how much each queue has used historically and gives preference to under-utilized queues.

    flowchart LR
    
    QueueA["Queue A <br/> used 80% of quota"]
    QueueB["Queue B <br/> used 20% of quota"]
    Cohort["Cohort pool <br/> (unused quota)"]
    Cohort-->|"Fair share: prefer Queue B"| QueueB

This prevents one team from monopolizing shared resources over time.

Configure it on the ClusterQueue:

    spec:
      fairSharing:
        weight: "1"   # Equal weight with other queues in cohort

---

Integrations

Kueue works with multiple job types out of the box.

Job Type	Framework
batch/Job	Standard Kubernetes Job
PyTorchJob	Kubeflow Training Operator
TFJob	Kubeflow TensorFlow
MPIJob	MPI (Horovod)
RayJob	KubeRay
RayCluster	KubeRay
JobSet	Kubernetes JobSet
StatefulSet	Kubernetes StatefulSet

To enable Kueue for a job, add a label:

    apiVersion: batch/v1
    kind: Job
    metadata:
      name: training-job
      namespace: research
      labels:
        kueue.x-k8s.io/queue-name: my-queue   # ← tells Kueue which LocalQueue to use
    spec:
      parallelism: 8
      template:
        spec:
          containers:
          - name: trainer
            resources:
              limits:
                nvidia.com/gpu: "1"

For PyTorchJob:

    apiVersion: kubeflow.org/v1
    kind: PyTorchJob
    metadata:
      name: llama-training
      labels:
        kueue.x-k8s.io/queue-name: my-queue   # ← same label

---

Complete Multi-Team Setup

A real DGX cluster serving three teams:

    # ResourceFlavor — H100 nodes
    apiVersion: kueue.x-k8s.io/v1beta1
    kind: ResourceFlavor
    metadata:
      name: h100
    spec:
      nodeLabels:
        nvidia.com/gpu.product: "NVIDIA-H100-SXM5-80GB"
    
    ---
    # ClusterQueues — one per team
    apiVersion: kueue.x-k8s.io/v1beta1
    kind: ClusterQueue
    metadata:
      name: research-cq
    spec:
      cohort: ai-teams
      resourceGroups:
      - coveredResources: ["nvidia.com/gpu"]
        flavors:
        - name: h100
          resources:
          - name: nvidia.com/gpu
            nominalQuota: 32
            borrowingLimit: 16
            lendingLimit: 8
      preemption:
        reclaimWithinCohort: LowerPriority
    
    ---
    apiVersion: kueue.x-k8s.io/v1beta1
    kind: ClusterQueue
    metadata:
      name: production-cq
    spec:
      cohort: ai-teams
      resourceGroups:
      - coveredResources: ["nvidia.com/gpu"]
        flavors:
        - name: h100
          resources:
          - name: nvidia.com/gpu
            nominalQuota: 24
            borrowingLimit: 24
            lendingLimit: 4
      preemption:
        reclaimWithinCohort: LowerPriority
        withinClusterQueue: LowerPriority
    
    ---
    apiVersion: kueue.x-k8s.io/v1beta1
    kind: ClusterQueue
    metadata:
      name: experiments-cq
    spec:
      cohort: ai-teams
      resourceGroups:
      - coveredResources: ["nvidia.com/gpu"]
        flavors:
        - name: h100
          resources:
          - name: nvidia.com/gpu
            nominalQuota: 8
            borrowingLimit: 40
            lendingLimit: 8

The cohort layout:

flowchart LR

subgraph Cohort["Cohort: ai-teams (64 H100 GPUs total)"]

Research["Research <br/> nominal: 32 GPUs"]
Production["Production <br/> nominal: 24 GPUs"]
Experiments["Experiments <br/> nominal: 8 GPUs"]
end

Research & Production & Experiments<-->|"Borrow unused quota"| Cohort

---

Observing Queue State

    # List all workloads and their status
    kubectl get workloads -n research
    
    # NAME                 QUEUE    ADMITTED   AGE
    # llama-training-wl    my-queue True       2m
    # gpt-finetune-wl      my-queue False      5s   ← waiting in queue
    
    # Check ClusterQueue usage
    kubectl get clusterqueue research-cq -o yaml
    
    # status:
    #   admittedWorkloads: 1
    #   pendingWorkloads: 1
    #   reservingWorkloads: 0
    #   flavorsUsage:
    #   - name: h100
    #     resources:
    #     - name: nvidia.com/gpu
    #       borrowed: 0
    #       total: 32
    
    # Check LocalQueue
    kubectl get localqueue my-queue -n research
    
    # NAME       CLUSTERQUEUE   PENDING   ADMITTED
    # my-queue   research-cq    1         1

---

Kueue vs Plain Kubernetes Scheduler

Feature	Kubernetes Scheduler	Kueue
Schedules Pods	Yes	No (delegates to scheduler)
Job-level awareness	No	Yes
Resource quotas	No (ResourceQuota is per-namespace, not gang-aware)	Yes
Gang scheduling	No	Yes
Queuing / backlog	No	Yes
Priority and preemption	Pod-level only	Workload-level
Borrowing across teams	No	Yes (Cohorts)
Fair sharing	No	Yes
Multi-framework (Job + PyTorchJob + Ray)	N/A	Yes

Kueue does not replace the Kubernetes scheduler.

It sits above the scheduler.

flowchart LR

    Jobs["Jobs / PyTorchJobs / RayJobs"]
    Jobs-->Kueue["Kueue <br/> (admission control, queuing, quotas)"]
    Kueue-->|"Creates Pods after admission"| Scheduler["Kubernetes Scheduler <br/> (places Pods on nodes)"]
    Scheduler-->Nodes

The scheduler still handles node selection.

Kueue handles job ordering, admission, and quota enforcement.

---

Key Takeaways

Concept	Purpose
ResourceFlavor	Labels hardware types (H100, CPU, spot) for quota tracking
ClusterQueue	Quota pool — guaranteed allocation per team, borrow/lend limits
LocalQueue	Team's namespace-level entry point into a ClusterQueue
Workload	Internal representation of a job; no pods until admitted
Gang scheduling	All pods start together — essential for distributed training
Cohort	Cluster of ClusterQueues that share unused quota
Borrowing	Temporary use of another queue's unused quota
Preemption	Higher-priority jobs can evict lower-priority ones to reclaim quota
Fair sharing	Distributes idle resources proportionally based on past usage

Kueue turns a GPU cluster into a multi-tenant platform — teams get guaranteed quotas, can burst into unused capacity, and production jobs always beat experiments without any manual intervention.

---

Related Posts

• Kubernetes Scheduler Internals — how Kueue hooks into the PreEnqueue extension point to hold Pods before they enter the scheduling queue
• Multi-Node Distributed Training on Kubernetes — the primary consumer of Kueue's gang scheduling; PyTorchJob gang admission explained end-to-end
• GPU Autoscaling: KEDA, HPA, Cluster Autoscaler — how KEDA reads Kueue's pending workload count as a scaling trigger for worker Pods
• Megatron-LM and Distributed LLM Training — the training workloads that Kueue queues and admits at scale