Kubernetes Scheduler Internals

Kubernetes Scheduler Internals 🕣

The Kubernetes Scheduler (kube-scheduler) is the control plane component responsible for selecting the best node for every newly created Pod.

Its primary responsibility is:

Watch for Pods without a node assignment and bind them to the most suitable node.

Where the Scheduler Fits

The Scheduler is responsible for:

• Watching Pending Pods
• Filtering unsuitable nodes
• Scoring candidate nodes
• Selecting the best node
• Binding the Pod to the selected node

The scheduler does not

• Create Pods
• Start containers
• communicate directly with worker nodes.

It simply decides where a Pod should run.

The Scheduler only communicates with the API Server.

flowchart LR
    User[kubectl]

    User --> API[API Server]

    API <--> ETCD[(etcd)]
    API <--> Scheduler[Scheduler]
    API --> Kubelet
    API -->  Controller

Scheduler vs Kubelet

Scheduler	Kubelet
Selects node	Runs Pod
Watches Pending Pods	Watches Assigned Pods
One per cluster	One per node
Makes placement decisions	Manages containers

Scheduler vs Controller Manager

Scheduler	Controller Manager
Chooses node	Creates Pods
One scheduling cycle	Continuous reconciliation
Doesn't scale workloads	Ensures desired replicas exist

--

How Scheduler Works

Most people think the Kubernetes Scheduler simply "finds a node."

It does much more than that.

When we start a pod:

kubectl apply -f nginx.yaml

Pod Scheduling Flow

sequenceDiagram
    participant User
    participant API
    participant Scheduler
    participant ETCD
    participant Kubelet

    User->>API: Create Pod

    API->>ETCD: Store Pod

    Scheduler->>API: Watch Pending Pods

    API-->>Scheduler: New Pending Pod

    Scheduler->>Scheduler: Select Best Node

    Scheduler->>API: Bind Pod to Node-2

    API->>ETCD: Update Pod

    Kubelet->>API: Watch Assigned Pods

    API-->>Kubelet: Pod Spec

The scheduler goes through five stages before a Pod starts running, handling priority, resource constraints, topology, and preemption along the way.

flowchart TD

    Pending["Pending Pod ⏳"]
    Pending-->Queue["Scheduling\nQueue 🚧"]
    Queue-->Filter["Filter <br/> (feasibility) 🔎"]
    Filter-->Score["Score <br/> (rank) 📋 "]

    Score-->Preempt["Preempt <br/> (if needed) ⛔ "]
    Preempt-->Bind["Bind <br/> (assign node) 📌"]
    Bind-->Kubelet["Kubelet\nstarts Pod ✨"]

Think of it like hiring:

• collect applications
• eliminate unsuitable candidates
• rank the rest
• then hire the best one.

If budget is frozen and no one fits, you can let go of a lower-priority contractor to free a slot.

---

Step 1: Scheduling Queue 🚧

Every newly created Pod without a nodeName is Pending and enters the scheduling queue.

spec:
  nodeName: ""   # ← unscheduled

The scheduler maintains three internal queues:

Queue	Contents	When a Pod moves here
🟢 ActiveQ	Pods ready for immediate scheduling	New Pods, Pods that became schedulable
🟡 UnschedulableQ	Pods with no viable node	Node doesn't exist yet, resource shortage; moved back to ActiveQ when cluster state changes
🔴 BackoffQ	Pods that recently failed scheduling	After a failed attempt; retried with exponential backoff (1s → 2s → 4s … up to 10s)

Pod Priority and PriorityClass

Pods in the ActiveQ are sorted by PriorityClass — higher priority Pods are dequeued first.

apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
  name: gpu-training-high
value: 1000
globalDefault: false
description: "High-priority GPU training jobs"

spec:
  priorityClassName: gpu-training-high

Built-in priority classes:

Class	Value	Used by
system-cluster-critical	2,000,000,000	CoreDNS, kube-proxy
system-node-critical	2,000,001,000	kube-apiserver (static pods)
User-defined	0 – 1,000,000,000	Workloads

---

Step 2: Filtering 🔎

The scheduler asks: which nodes can run this Pod?

This is a hard constraint check. Any node that fails a single filter is eliminated.

flowchart LR

    AllNodes["All Nodes <br/> (e.g. 1,000)"]
    AllNodes-->Filter["Filter plugins <br/> (run in parallel)"]
    Filter-->FeasibleNodes["Feasible Nodes <br/> (e.g. 12)"]

Common Filter Plugins

Plugin	What it checks
NodeResourcesFit	Node has enough CPU, memory, GPU to satisfy requests
TaintToleration	Pod tolerates every taint on the node
NodeAffinity	nodeSelector and nodeAffinity rules match the node
PodAffinity	podAffinity / podAntiAffinity hard rules are satisfied
NodeUnschedulable	Node is not cordoned (kubectl cordon)
VolumeBinding	Required PersistentVolumes can be bound on this node
TopologySpreadConstraints	Pod placement respects spread constraints (see below)

Node Allocatable vs Requests

The scheduler does not use raw node capacity. It uses allocatable — capacity minus what the OS and Kubernetes components reserve.

Node capacity:       16 CPU, 64 Gi memory
System reserved:      0.5 CPU, 2 Gi
Kube reserved:        0.5 CPU, 2 Gi
─────────────────────────────────────
Allocatable:          15 CPU, 60 Gi   ← what the scheduler sees
Already requested:    10 CPU, 40 Gi   ← sum of all running Pod requests
─────────────────────────────────────
Free:                  5 CPU, 20 Gi   ← used for feasibility check

A Pod requesting more than the free allocatable fails the NodeResourcesFit filter.

---

Step 3: Scoring 📋

From the feasible nodes, the scheduler asks: which one is best?

Every scoring plugin gives each node a score from 0 to 100. Scores are weighted and summed. The highest total wins.

flowchart LR

    FeasibleNodes["Feasible Nodes"]
    FeasibleNodes-->Score["Score plugins <br/> (weighted sum)"]
    Score-->Ranked["Ranked Nodes\nNode C: 96 ✅\nNode A: 87\nNode F: 71"]

Common Score Plugins

Plugin	Prefers	Why
LeastAllocated	Nodes with more free resources	Spreads load evenly
BalancedAllocation	Nodes where CPU and memory usage ratios are similar	Avoids CPU-heavy / memory-idle nodes
ImageLocality	Nodes that already have the container image	Avoids pull time
NodeAffinity	Nodes matching preferred affinity labels	Soft placement preferences
InterPodAffinity	Nodes where preferred co-located Pods already run	Keeps communicating Pods close

Scoring at Scale

Scoring every node in a 10,000-node cluster for every Pod would be prohibitively slow.

Kubernetes uses percentageOfNodesToScore to limit this:

# kube-scheduler config
percentageOfNodesToScore: 5   # score only 5% of feasible nodes (min 100)

The scheduler samples a random subset of feasible nodes and scores only those. The larger the cluster, the smaller the percentage needed — statistical sampling is sufficient to find a good node.

---

Step 4: Preemption ⛔

If no node passes the filter for a high-priority Pod, the scheduler doesn't give up.

It asks: can I evict lower-priority Pods from a node to make room?

flowchart TD

    HighPriPod["High-priority GPU Pod\nrequests 8 GPUs"]
    HighPriPod-->NoFeasible["No node has 8 free GPUs"]
    NoFeasible-->Preempt["Scheduler finds a node\nrunning low-priority Pods"]
    Preempt-->Evict["Evicts low-priority Pods <br/> (they enter Pending again)"]
    Evict-->Bind["High-priority Pod <br/> binds to freed node"]

Preemption Rules

• Only Pods with lower priority than the incoming Pod can be evicted
• The scheduler chooses the victim node that minimizes the number of evictions
• Evicted Pods are not deleted immediately — they receive a grace period (based on terminationGracePeriodSeconds)
• Pod Disruption Budgets (PDB) can block preemption: if evicting a Pod would violate a PDB, that node is skipped

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: web-pdb
spec:
  minAvailable: 2    # always keep at least 2 replicas — preemption cannot reduce below this
  selector:
    matchLabels:
      app: web

Preemption can be disabled per PriorityClass:

preemptionPolicy: Never   # this class cannot preempt others

---

Step 5: Binding 📌

The scheduler writes the decision to the API Server:

sequenceDiagram

participant Scheduler
participant API as API Server
participant etcd
participant Kubelet

Scheduler->>API: Create Binding (pod=nginx, node=node-c)
API->>etcd: Update pod.spec.nodeName = "node-c"
etcd-->>API: Committed
API-->>Kubelet: Pod assigned (Watch event)
Kubelet->>Kubelet: Pull image, create container, start pod
Kubelet->>API: Update pod.status = Running

The scheduler's work ends at the Binding step. It does not start containers — that is entirely the Kubelet's responsibility.

---

Topology Spread Constraints

For fault tolerance and performance, Pods often need to be spread across failure domains — availability zones, racks, or individual nodes.

TopologySpreadConstraints is a filter (hard) and score (soft) mechanism for this.

spec:
  topologySpreadConstraints:
  - maxSkew: 1                           # max difference in Pod count between any two zones
    topologyKey: topology.kubernetes.io/zone
    whenUnsatisfiable: DoNotSchedule     # hard constraint
    labelSelector:
      matchLabels:
        app: web
  - maxSkew: 2
    topologyKey: kubernetes.io/hostname
    whenUnsatisfiable: ScheduleAnyway   # soft — prefer but don't enforce
    labelSelector:
      matchLabels:
        app: web

Example — 3 zones, 6 Pods:

flowchart LR

subgraph ZoneA["Zone A"]
P1 & P2
end

subgraph ZoneB["Zone B"]
P3 & P4
end

subgraph ZoneC["Zone C"]
P5 & P6
end

maxSkew: 1 means no zone can have more than 1 extra Pod compared to the least-loaded zone. Scheduling a 7th Pod to Zone A would create skew of 2 — blocked if whenUnsatisfiable: DoNotSchedule.

For DGX clusters this is used to spread training workers across DGX nodes or racks, ensuring a node failure doesn't kill the entire gang.

---

The Scheduler Plugin Framework

The scheduler is built entirely as plugins. Each stage of the pipeline is an extension point that plugins hook into.

flowchart TB

QueueSort-->PreEnqueue-->PreFilter-->Filter
Filter-->PostFilter["PostFilter <br/> (preemption)"]
PostFilter-->PreScore-->Score-->Reserve
Reserve-->Permit-->PreBind-->Bind
Bind-->PostBind

Extension Point	Purpose
QueueSort	Defines how Pods are ordered in the ActiveQ (default: by priority + timestamp)
PreEnqueue	Gates whether a Pod enters the queue at all (Kueue hooks here)
PreFilter	Precomputes state used by Filter plugins (e.g., aggregate resource requests)
Filter	Hard feasibility checks per node
PostFilter	Runs if Filter found zero feasible nodes — triggers preemption
PreScore	Precomputes state shared by Score plugins
Score	Ranks feasible nodes 0–100
Reserve	Speculatively reserves resources (prevents races when binding is async)
Permit	Can approve, deny, or wait (used by gang scheduling — waits until all Pods of a group are schedulable)
Bind	Writes the nodeName assignment to the API Server

Custom plugins are registered via the KubeSchedulerConfiguration API, allowing cluster operators to replace or augment any extension point without forking the scheduler.

---

Where Kueue Fits

Kueue does not replace the scheduler. It hooks into PreEnqueue.

flowchart LR

    Job["PyTorchJob / batch/Job"]
    Job-->Kueue["Kueue <br/> (PreEnqueue plugin) <br/> quota check + gang check"]
    Kueue-->|"Admitted"| ActiveQ["Scheduler <br/> ActiveQ"]
    Kueue-->|"Not admitted"| Hold["Held in <br/> Kueue queue"]
    ActiveQ-->Filter & Score & Bind

Kueue's PreEnqueue plugin blocks Pods from entering the ActiveQ until:

1. The ClusterQueue has sufficient quota
2. All Pods in the gang (e.g., all workers of a PyTorchJob) can start simultaneously

Only when both conditions are met does Kueue release the Pods into the scheduler's ActiveQ.

The Kubernetes Scheduler then handles node selection as normal — it has no knowledge of Kueue quotas or gangs.

---

Key Takeaways

Stage	Question answered	Failure outcome
Scheduling Queue	In what order do Pods get scheduled?	Lower-priority Pods wait
Filter	Which nodes can run this Pod?	Pod stays Pending if zero nodes pass
Score	Which feasible node is best?	Ties broken arbitrarily
Preemption	Can we evict lower-priority Pods to make room?	Pod stays Pending if preemption is blocked (PDB)
Bind	Assign the Pod to the winning node	Rare conflicts retried by scheduler
Kubelet	Start the container on the assigned node	Pod shows ContainerCreating

The scheduler is a recommendation engine, not a gatekeeper.

The Kubernetes Scheduler is the cluster's decision maker.

• It continuously watches for Pods that have no assigned node.
• It filters nodes that cannot satisfy the Pod's requirements.
• It scores the remaining nodes using multiple scheduling plugins.
• It binds the Pod to the best node through the API Server.
• The Kubelet on that node then creates and runs the Pod.

Think of the Scheduler as the placement engine of Kubernetes:

• API Server → Stores cluster state.
• Scheduler → Decides where Pods run.
• Kubelet → Makes Pods run.
• etcd → Remembers everything.

---

Related Posts

• GPU Scheduling in Kubernetes — how this generic scheduler handles the GPU-specific case via device plugins
• Kubernetes Topology Manager — the NUMA-aware layer that refines the scheduler's node-level placement decisions