Kubernetes Scheduler Internals

Most people think the Kubernetes Scheduler simply "finds a node."

It does much more than that.

The scheduler goes through five stages before a Pod starts running, handling priority, resource constraints, topology, and preemption along the way.

flowchart LR

Pending["Pending Pod"]

-->Queue["Scheduling\nQueue"]

Queue

-->Filter["Filter\n(feasibility)"]

Filter

-->Score["Score\n(rank)"]

Score

-->Preempt["Preempt\n(if needed)"]

Preempt

-->Bind["Bind\n(assign node)"]

Bind

-->Kubelet["Kubelet\nstarts Pod"]

Think of it like hiring: collect applications, eliminate unsuitable candidates, rank the rest, then hire the best one. If budget is frozen and no one fits, you can let go of a lower-priority contractor to free a slot.

---

Step 1: Scheduling Queue

Every newly created Pod without a nodeName is Pending and enters the scheduling queue.

spec:
  nodeName: ""   # ← unscheduled

The scheduler maintains three internal queues:

Queue	Contents	When a Pod moves here
ActiveQ	Pods ready for immediate scheduling	New Pods, Pods that became schedulable
BackoffQ	Pods that recently failed scheduling	After a failed attempt; retried with exponential backoff (1s → 2s → 4s … up to 10s)
UnschedulableQ	Pods with no viable node	Node doesn't exist yet, resource shortage; moved back to ActiveQ when cluster state changes

Pod Priority and PriorityClass

Pods in the ActiveQ are sorted by PriorityClass — higher priority Pods are dequeued first.

apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
  name: gpu-training-high
value: 1000
globalDefault: false
description: "High-priority GPU training jobs"

spec:
  priorityClassName: gpu-training-high

Built-in priority classes:

Class	Value	Used by
system-cluster-critical	2,000,000,000	CoreDNS, kube-proxy
system-node-critical	2,000,001,000	kube-apiserver (static pods)
User-defined	0 – 1,000,000,000	Workloads

---

Step 2: Filtering

The scheduler asks: which nodes can run this Pod?

This is a hard constraint check. Any node that fails a single filter is eliminated.

flowchart LR

AllNodes["All Nodes\n(e.g. 1,000)"]

-->Filter["Filter plugins\n(run in parallel)"]

Filter

-->FeasibleNodes["Feasible Nodes\n(e.g. 12)"]

Common Filter Plugins

Plugin	What it checks
NodeResourcesFit	Node has enough CPU, memory, GPU to satisfy requests
TaintToleration	Pod tolerates every taint on the node
NodeAffinity	nodeSelector and nodeAffinity rules match the node
PodAffinity	podAffinity / podAntiAffinity hard rules are satisfied
NodeUnschedulable	Node is not cordoned (kubectl cordon)
VolumeBinding	Required PersistentVolumes can be bound on this node
TopologySpreadConstraints	Pod placement respects spread constraints (see below)

Node Allocatable vs Requests

The scheduler does not use raw node capacity. It uses allocatable — capacity minus what the OS and Kubernetes components reserve.

Node capacity:       16 CPU, 64 Gi memory
System reserved:      0.5 CPU, 2 Gi
Kube reserved:        0.5 CPU, 2 Gi
─────────────────────────────────────
Allocatable:          15 CPU, 60 Gi   ← what the scheduler sees
Already requested:    10 CPU, 40 Gi   ← sum of all running Pod requests
─────────────────────────────────────
Free:                  5 CPU, 20 Gi   ← used for feasibility check

A Pod requesting more than the free allocatable fails the NodeResourcesFit filter.

---

Step 3: Scoring

From the feasible nodes, the scheduler asks: which one is best?

Every scoring plugin gives each node a score from 0 to 100. Scores are weighted and summed. The highest total wins.

flowchart LR

FeasibleNodes["Feasible Nodes"]

-->Score["Score plugins\n(weighted sum)"]

Score

-->Ranked["Ranked Nodes\nNode C: 96 ✅\nNode A: 87\nNode F: 71"]

Common Score Plugins

Plugin	Prefers	Why
LeastAllocated	Nodes with more free resources	Spreads load evenly
BalancedAllocation	Nodes where CPU and memory usage ratios are similar	Avoids CPU-heavy / memory-idle nodes
ImageLocality	Nodes that already have the container image	Avoids pull time
NodeAffinity	Nodes matching preferred affinity labels	Soft placement preferences
InterPodAffinity	Nodes where preferred co-located Pods already run	Keeps communicating Pods close

Scoring at Scale

Scoring every node in a 10,000-node cluster for every Pod would be prohibitively slow.

Kubernetes uses percentageOfNodesToScore to limit this:

# kube-scheduler config
percentageOfNodesToScore: 5   # score only 5% of feasible nodes (min 100)

The scheduler samples a random subset of feasible nodes and scores only those. The larger the cluster, the smaller the percentage needed — statistical sampling is sufficient to find a good node.

---

Step 4: Preemption

If no node passes the filter for a high-priority Pod, the scheduler doesn't give up.

It asks: can I evict lower-priority Pods from a node to make room?

flowchart TD

HighPriPod["High-priority GPU Pod\nrequests 8 GPUs"]

-->NoFeasible["No node has 8 free GPUs"]

NoFeasible

-->Preempt["Scheduler finds a node\nrunning low-priority Pods"]

Preempt

-->Evict["Evicts low-priority Pods\n(they enter Pending again)"]

Evict

-->Bind["High-priority Pod\nbinds to freed node"]

Preemption Rules

• Only Pods with lower priority than the incoming Pod can be evicted
• The scheduler chooses the victim node that minimizes the number of evictions
• Evicted Pods are not deleted immediately — they receive a grace period (based on terminationGracePeriodSeconds)
• Pod Disruption Budgets (PDB) can block preemption: if evicting a Pod would violate a PDB, that node is skipped

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: web-pdb
spec:
  minAvailable: 2    # always keep at least 2 replicas — preemption cannot reduce below this
  selector:
    matchLabels:
      app: web

Preemption can be disabled per PriorityClass:

preemptionPolicy: Never   # this class cannot preempt others

---

Step 5: Binding

The scheduler writes the decision to the API Server:

sequenceDiagram

participant Scheduler
participant API as API Server
participant etcd
participant Kubelet

Scheduler->>API: Create Binding (pod=nginx, node=node-c)
API->>etcd: Update pod.spec.nodeName = "node-c"
etcd-->>API: Committed
API-->>Kubelet: Pod assigned (Watch event)
Kubelet->>Kubelet: Pull image, create container, start pod
Kubelet->>API: Update pod.status = Running

The scheduler's work ends at the Binding step. It does not start containers — that is entirely the Kubelet's responsibility.

---

Topology Spread Constraints

For fault tolerance and performance, Pods often need to be spread across failure domains — availability zones, racks, or individual nodes.

TopologySpreadConstraints is a filter (hard) and score (soft) mechanism for this.

spec:
  topologySpreadConstraints:
  - maxSkew: 1                           # max difference in Pod count between any two zones
    topologyKey: topology.kubernetes.io/zone
    whenUnsatisfiable: DoNotSchedule     # hard constraint
    labelSelector:
      matchLabels:
        app: web
  - maxSkew: 2
    topologyKey: kubernetes.io/hostname
    whenUnsatisfiable: ScheduleAnyway   # soft — prefer but don't enforce
    labelSelector:
      matchLabels:
        app: web

Example — 3 zones, 6 Pods:

flowchart LR

subgraph ZoneA["Zone A"]
P1 & P2
end

subgraph ZoneB["Zone B"]
P3 & P4
end

subgraph ZoneC["Zone C"]
P5 & P6
end

maxSkew: 1 means no zone can have more than 1 extra Pod compared to the least-loaded zone. Scheduling a 7th Pod to Zone A would create skew of 2 — blocked if whenUnsatisfiable: DoNotSchedule.

For DGX clusters this is used to spread training workers across DGX nodes or racks, ensuring a node failure doesn't kill the entire gang.

---

The Scheduler Plugin Framework

The scheduler is built entirely as plugins. Each stage of the pipeline is an extension point that plugins hook into.

flowchart TB

QueueSort

-->PreEnqueue

PreEnqueue

-->PreFilter

PreFilter

-->Filter

Filter

-->PostFilter["PostFilter\n(preemption)"]

PostFilter

-->PreScore

PreScore

-->Score

Score

-->Reserve

Reserve

-->Permit

Permit

-->PreBind

PreBind

-->Bind

Bind

-->PostBind

Extension Point	Purpose
QueueSort	Defines how Pods are ordered in the ActiveQ (default: by priority + timestamp)
PreEnqueue	Gates whether a Pod enters the queue at all (Kueue hooks here)
PreFilter	Precomputes state used by Filter plugins (e.g., aggregate resource requests)
Filter	Hard feasibility checks per node
PostFilter	Runs if Filter found zero feasible nodes — triggers preemption
PreScore	Precomputes state shared by Score plugins
Score	Ranks feasible nodes 0–100
Reserve	Speculatively reserves resources (prevents races when binding is async)
Permit	Can approve, deny, or wait (used by gang scheduling — waits until all Pods of a group are schedulable)
Bind	Writes the nodeName assignment to the API Server

Custom plugins are registered via the KubeSchedulerConfiguration API, allowing cluster operators to replace or augment any extension point without forking the scheduler.

---

Where Kueue Fits

Kueue does not replace the scheduler. It hooks into PreEnqueue.

flowchart LR

Job["PyTorchJob / batch/Job"]

-->Kueue["Kueue\n(PreEnqueue plugin)\nquota check + gang check"]

Kueue

-->|"Admitted"| ActiveQ["Scheduler\nActiveQ"]

Kueue

-->|"Not admitted"| Hold["Held in\nKueue queue"]

ActiveQ

-->Filter & Score & Bind

Kueue's PreEnqueue plugin blocks Pods from entering the ActiveQ until:

1. The ClusterQueue has sufficient quota
2. All Pods in the gang (e.g., all workers of a PyTorchJob) can start simultaneously

Only when both conditions are met does Kueue release the Pods into the scheduler's ActiveQ.

The Kubernetes Scheduler then handles node selection as normal — it has no knowledge of Kueue quotas or gangs.

---

Key Takeaways

Stage	Question answered	Failure outcome
Scheduling Queue	In what order do Pods get scheduled?	Lower-priority Pods wait
Filter	Which nodes can run this Pod?	Pod stays Pending if zero nodes pass
Score	Which feasible node is best?	Ties broken arbitrarily
Preemption	Can we evict lower-priority Pods to make room?	Pod stays Pending if preemption is blocked (PDB)
Bind	Assign the Pod to the winning node	Rare conflicts retried by scheduler
Kubelet	Start the container on the assigned node	Pod shows ContainerCreating

The scheduler is a recommendation engine, not a gatekeeper. The API Server and Kubelet enforce the final state. Understanding the pipeline — especially preemption, topology constraints, and the Permit extension point — is what separates cluster operators from cluster architects.