Kubernetes Performance at Scale

Kubernetes Performance at Scale 🏗️

Running Kubernetes with 100 nodes is very different from running it with 10,000 nodes.

The failures that emerge at hyperscale are not crashes — they are latency cliffs: operations that take 10 ms at 100 nodes take 10 seconds at 10,000 nodes, and the cluster appears healthy right until it doesn't.

This post covers where those cliffs are, how to measure them, and how the Kubernetes ecosystem addresses each one.

---

Kubernetes Scalability SLIs and SLOs

The Kubernetes project defines official Service Level Indicators (SLIs) and Service Level Objectives (SLOs) — measurable guarantees about control plane behavior at the certified scale ceiling.

Certified Scale Ceiling (as of Kubernetes 1.30)

Resource	Certified Limit	Notes
🖥️ Nodes	5,000	Maximum supported cluster size
📦 Pods per Node	110	Default kubelet maximum (--max-pods)
📦 Total Pods	150,000	Cluster-wide scalability target
🐳 Total Containers	300,000	Approximately 2 containers per Pod on average
🏠 Namespaces	10,000+	No fixed certified limit; practical limits depend on API server and etcd performance
🌐 Services	10,000+	Practical limit varies by networking implementation (iptables, IPVS, eBPF) and cluster scale

These are not hard limits — clusters can exceed them — but performance guarantees only apply within these bounds.

Official SLOs

SLI	SLO
Mutating API call latency (POST/PUT/PATCH/DELETE)	p99 < 1 s, excluding watch
Non-streaming read API call latency (GET/LIST)	p99 < 30 s for large resources, < 1 s for small
Pod startup latency (scheduled + running, no image pull)	p99 < 5 s
In-cluster DNS lookup latency	p99 < 5 s
Scheduling throughput	≥ 100 Pods/s average

The pod startup latency SLO is the most operationally visible — it is what users feel when they scale a Deployment.

---

Pod Startup Latency — The Key SLI

Pod startup latency is not one number. It is a pipeline of stages, each of which can become the bottleneck independently.

flowchart TD
    Submit["Submit<br/><br/>kubectl apply <br/> (t=0)"]
    Submit-->Admitted["Admitted<br/><br/>API Server <br/> admitted <br/> (t=50ms)"]
    Admitted--> Scheduled["Scheduled<br/><br/>Scheduler <br/> binds to node <br/> (t=200ms)"]
    Scheduled--> ImagePull["Image pull <br/> (t=1–30s)"]
    ImagePull-->ContainerCreate["Container <br/> created <br/> (t+100ms)"]
    ContainerCreate--> Running["Pod Running <br/> (t=?)"]

Stage	Typical duration	What slows it down
API Server admission	10–100 ms	Slow admission webhooks
Scheduler decision	50–500 ms	Queue depth, filter/score at scale
Image pull	0–60 s	Cold node, large image, registry rate limits
Container create	50–200 ms	CNI plugin latency
Probe warm-up	0–30 s	initialDelaySeconds on readiness probe

The SLO of p99 < 5 s excludes image pull time — it assumes the image is already cached. When benchmarking, always separate the two.

Key Prometheus metrics:

# Scheduling queue time
scheduler_e2e_scheduling_duration_seconds

# API Server admission latency
apiserver_admission_webhook_admission_duration_seconds

# Full pod startup (from creation to Running)
kubelet_pod_start_duration_seconds

---

Gigawatt Scaling Challenges

Kubernetes scaling has three dimensions

1. Cluster size
2. Object count
3. API request rate

🚨 1: API Server Watch Storms

With 10,000 nodes and 50 controllers, there can be over 50,000 active watch connections to the API Server simultaneously.

A single busy Deployment — say, a rolling update touching 1,000 Pods — generates one event per Pod per status change. That is potentially tens of thousands of events that must be serialized and sent to every interested watcher.

flowchart TD
    OneUpdate["1 Deployment <br/> rolling update"]
    OneUpdate-->Events["~5,000 Pod events <br/> (Pending → Running → Ready)"]
    Events--> FanOut["Fan-out to <br/> 50,000 watchers"]
    FanOut--> CPUSpike["API Server <br/> CPU spike"]

The Watch Cache

The Watch Cache is the primary defense. It holds a ring buffer of recent events per resource type, served entirely from memory.

flowchart LR
    etcd-->|" 1 watch connection "|WatchCache["Watch Cache <br/> (in-memory ring buffer)"]
    WatchCache-->|" 50,000 watch streams "|Clients["Controllers <br/> Kubelets <br/> Operators"]

The API Server watches etcd once per resource type. All clients watch the API Server. This multiplexing is what keeps etcd alive at scale.

The Watch Cache also serves LIST requests from memory if the client passes resourceVersion=0 — skipping etcd entirely.

LIST Scalability and resourceVersion

LIST requests are a hidden scalability trap. A large LIST with no resourceVersion set forces a consistent read from etcd — serializing all objects, holding an etcd read transaction open, and loading everything into API Server memory.

# Consistent read — hits etcd, expensive
kubectl get pods --all-namespaces
# Internally: GET /api/v1/pods  (no resourceVersion → etcd read)

# Served from cache — cheap, slightly stale
kubectl get pods --all-namespaces -l app=web
# With resourceVersion=0: GET /api/v1/pods?resourceVersion=0  (cache)

Controllers using the SharedInformer always pass resourceVersion=0 for their initial LIST — this is why informer initialization does not hammer etcd even during a mass restart.

At large object counts (> 100,000 Pods), even cache-served LISTs can spike API Server memory because all objects must be serialized into the response. Pagination (the limit parameter) keeps any single response bounded:

# Pages of 500 objects instead of one massive response
kubectl get pods --chunk-size=500 --all-namespaces

Horizontal API Server Scaling

The API Server is stateless — it stores nothing locally. Multiple replicas can run behind a load balancer.

flowchart TD
    Clients["Clients<br/><br/> kubectl, <br/> Controllers, <br/> Kubelets"]
    Clients-->LB["Load Balancer <br/> (e.g. HAProxy / kube-vip)"]
    
    LB--> AS1["kube-apiserver <br/> replica 1"]
    LB--> AS2["kube-apiserver <br/> replica 2"]
    LB--> AS3["kube-apiserver <br/> replica 3"]
    
    AS1 & AS2 & AS3 --> etcd["etcd cluster"]

Each replica maintains its own Watch Cache — they are independent. A client connecting to replica 2 gets the same data as one on replica 1 because both read from the same etcd cluster (with slight propagation delay).

At DGX Cloud scale, API Server replicas are sized by

• memory (Watch Cache grows with object count) and
• CPU (fan-out serialization).

💡A cluster with 300,000 Pods may need API Server replicas with 64–128 GB RAM each.

---

🚨 2: etcd Bottlenecks

Every write to the API Server eventually becomes a Raft log entry in etcd.

At scale, write throughput and commit latency become the gating factor.

flowchart TD
    APIServer["API Server <br/> (writes)"]
    APIServer-->Leader["etcd Leader <br/> (Raft log)"]
    
    Leader--> F1["Follower 1"]
    Leader--> F2["Follower 2"]
    Leader-->|" fsync to WAL "|Disk["NVMe SSD <br/> (WAL fsync < 1ms p99)"]

The bottleneck is almost always disk fsync latency on the leader. Each committed log entry requires a WAL fsync.

Network-attached storage (NFS, EBS gp2) regularly exceeds 10 ms per fsync — etcd expects < 1 ms.

At hyperscale, additional pressures:

Pressure	Effect	Mitigation
High write rate	Longer Raft replication latency	Batch more writes (API Server request coalescing)
Large object count	etcd DB grows, compaction takes longer	Regular compaction + defrag
DB size approaching 8 GB	etcd refuses new writes	Emergency compaction; increase --quota-backend-bytes
Leader election under load	Brief write freeze (~2s)	NVMe-backed etcd; dedicated etcd nodes

💡 Dedicated etcd nodes with NVMe storage and no other workloads is the standard recommendation for clusters > 1,000 nodes.

---

🚨 3. Object Count

Every Kubernetes object consumes memory in three places simultaneously:

flowchart TD
    Object["Kubernetes Object"]

Object-->etcd["etcd <br/> (on disk + BoltDB page cache)"]
Object--> WatchCache["API Server Watch Cache <br/> (in-memory)"]
Object--> InformerCache["Controller Informer Cache <br/> (in-memory, per controller)"]

A cluster with

• 300,000 Pods
• 300,000 ReplicaSets, and
• 500,000 Events

might hold 5–10 million objects in etcd.

Why Events Are Dangerous

Events are the most proliferative object type. Every Pod lifecycle step generates events:

Scheduled → Pulling → Pulled → Created → Started → Readiness probe passed

Six events per Pod.

• 300,000 Pods
• 1.8 million events — just for one rollout.

Events are also watched by monitoring tools, log aggregators, and alerting systems. One event write fans out to potentially thousands of watchers.

💡 Mitigations:

• Events have a default TTL of 1 hour (--event-ttl on kube-apiserver, default 1h)
• Use events.k8s.io/v1 (the new Events API) which coalesces repeated events into a count field
• Deploy a separate event sink (e.g., Event Exporter to ElasticSearch) and disable in-cluster event retention for large clusters

Object Lifecycle Management

For long-lived clusters, zombie objects accumulate: completed Jobs, finished Pods, stale ReplicaSets from old Deployments.

# Completed Jobs older than 1 hour (TTL controller)
kubectl get jobs --field-selector=status.completionTime  # prune manually

# Or use TTL controller on Job:
spec:
  ttlSecondsAfterFinished: 3600

The kube-controller-manager runs a garbage collector that follows owner references and deletes orphaned objects, but it has throughput limits — at very high object counts it can lag behind.

---

🚨 4: API Priority and Fairness (APF)

At scale, a flood of kubectl get pods from a CI system can starve the scheduler and kubelets of API Server capacity.

APF classifies every request into a FlowSchema (who is making it, what operation) and routes it to a * PriorityLevelConfiguration* (how much concurrency it gets).

Priority Levels and Concurrency

flowchart TD
    Request["Incoming Request"]
    Request-->FS["FlowSchema <br/> match rules"]
    FS--> PL["PriorityLevel <br/> (concurrency share)"]
    PL-->|" Admitted "|Workers["API Server <br/> worker goroutines"]
    PL-->|" Queue full "|Drop["429 Too Many Requests"]

Built-in priority levels with their concurrency shares (percentage of total server concurrency):

Level	Default share	Users
exempt	Unlimited	system:masters, health probes
node-high	40%	Kubelet node status, lease updates
system	30%	kube-system controllers, scheduler
leader-election	8%	Controller leader election
workload-high	30%	Authenticated operators
workload-low	10%	Regular user kubectl
global-default	10%	Unauthenticated, catch-all

Concurrency shares are relative — if node-high is using only 20% of its share, the remainder is lent to other levels. APF is not a strict partition.

Diagnosing APF Throttling

  # See if requests are being rejected (429)
  kubectl get --raw /metrics | grep apiserver_flowcontrol_rejected_requests_total
  
  # See queue depth per priority level
  kubectl get --raw /metrics | grep apiserver_flowcontrol_current_inqueue_requests
  
  # See which FlowSchemas exist and their priority levels
  kubectl get flowschemas
  kubectl get prioritylevelconfigurations

If workload-low is saturated and rejecting requests, tune its nominalConcurrencyShares up or reduce the flood source.

---

🚨 5: Scheduler Throughput

At 100,000 pending Pods, the scheduler becomes the bottleneck even if the API Server and etcd are healthy.

What Limits Throughput

The scheduler runs a scheduling cycle for one Pod at a time, but runs multiple goroutines in parallel ( controlled by --parallelism, default 16).

flowchart LR
    ActiveQ["Active Queue <br/> 100,000 Pods"]
    ActiveQ-->G1["Goroutine 1 <br/> Filter → Score → Bind"]
    ActiveQ--> G2["Goroutine 2 <br/> Filter → Score → Bind"]
    ActiveQ--> G3["Goroutine 3 <br/> ..."]
    ActiveQ--> G16["Goroutine 16 <br/> ..."]

Throughput ceiling: ~100–200 Pods/second for a default scheduler with 5,000 nodes.

Key Tuning Knobs

Parameter	Default	Effect
--parallelism	16	Number of concurrent scheduling goroutines
percentageOfNodesToScore	0 (auto)	Fraction of feasible nodes to score; auto-scales with cluster size
Filter plugin order	Built-in	Cheapest filters first eliminates most nodes early
--kube-api-qps	50	QPS to API Server for binding writes
--kube-api-burst	100	Burst above QPS for binding

The percentageOfNodesToScore auto-mode scores 50% of nodes at cluster size 100, dropping to ~5% at 5,000 nodes — balancing placement quality against throughput.

---

Benchmarking: ClusterLoader2 and KWOK

KWOK — Fake Clusters at Laptop Scale

Testing scheduler or controller behavior at 10,000 nodes normally requires 10,000 VMs. KWOK (Kubernetes WithOut Kubelet) simulates nodes and Pods without running any real workloads.

flowchart TD
    KWOK["KWOK <br/> (fake node controller)"]
    KWOK-->|" Creates "|FakeNodes["10,000 fake nodes <br/> (API objects only)"]
    
    FakeNodes--> API["API Server <br/> (sees real node objects)"]
    API--> Scheduler["Scheduler <br/> (schedules real Pods <br/> onto fake nodes)"]
    API--> Controllers["Controllers <br/> (reconcile normally)"]

KWOK nodes respond to kubelet heartbeat protocols — controllers and the scheduler behave identically to a real cluster. Only actual container execution is absent.

  # Install KWOK
  kubectl apply -f https://github.com/kubernetes-sigs/kwok/releases/latest/download/kwok.yaml
  
  # Create 1,000 fake nodes
  kwokctl create cluster --name=perf-test
  kubectl apply -f - <<EOF
  apiVersion: v1
  kind: Node
  metadata:
    name: fake-node-0001
    labels:
      type: kwok
  spec:
    taints:
    - key: kwok.x-k8s.io/node
      effect: NoSchedule
  EOF

ClusterLoader2 — Workload Benchmark

ClusterLoader2 is the Kubernetes project's official performance benchmark. It creates configurable workloads and measures SLI timings against the SLOs.

flowchart TD
    CL2["ClusterLoader2 <br/> (test definition YAML)"]
    CL2-->|" Creates "|Workloads["Pods / Deployments <br/> / Services / Jobs"]
  
    Workloads--> Cluster["Cluster <br/> (KWOK or real)"]
    Cluster--> Metrics["Prometheus <br/> metrics"]
    Metrics--> Report["SLI Report <br/> (p50/p90/p99 latencies)"]

A typical test definition:

  name: load
  namespace:
    number: 100
  tuningSets:
    - name: Uniformly1qps
      qpsLoad:
        qps: 1
  steps:
    - name: Create Deployments
      phases:
        - namespaceRange:
            min: 1
            max: 100
          replicasPerNamespace: 10
          tuningSet: Uniformly1qps
          objectBundle:
            - basename: test-deployment
              objectTemplatePath: deployment.yaml
    - name: Wait for Pods to be running
      phases:
        - namespaceRange:
            min: 1
            max: 100
          replicasPerNamespace: 10
          tuningSet: Uniformly1qps
          objectBundle:
            - basename: test-deployment
              objectTemplatePath: deployment.yaml
              currentLoad:
                type: Running

ClusterLoader2 automatically collects and reports SLI measurements after each test:

API call latency:
  Metric: scheduler_e2e_scheduling_duration_seconds
  Percentile: 99
  Value: 487ms
  SLO: 1000ms ✅

Pod startup latency:
  Metric: kubelet_pod_start_duration_seconds
  Percentile: 99
  Value: 3.2s
  SLO: 5s ✅

---

Performance Metrics Reference

Metrics to monitor continuously in a production cluster at scale:

Metric	Source	Alert threshold
apiserver_request_duration_seconds	API Server	p99 > 1 s for mutating
apiserver_current_inflight_requests	API Server	Sustained > 80% of max
apiserver_flowcontrol_rejected_requests_total	API Server	Any increase
scheduler_e2e_scheduling_duration_seconds	Scheduler	p99 > 1 s
scheduler_pending_pods	Scheduler	Sustained > 1,000
etcd_disk_wal_fsync_duration_seconds	etcd	p99 > 10 ms
etcd_server_leader_changes_seen_total	etcd	> 3 per hour
etcd_mvcc_db_total_size_in_bytes	etcd	> 6 GB
kubelet_pod_start_duration_seconds	Kubelet (per node)	p99 > 5 s
apiserver_watch_events_total	API Server	Sudden spike

---

Key Takeaways

Scaling Kubernetes is not about adding more CPU or memory — it is about eliminating pressure on the control plane.

Bottleneck	Root cause	Solution
Watch storm	Too many watchers receiving too many events	Watch Cache fan-out; SharedInformers
LIST memory spike	Full etcd read into API Server memory	resourceVersion=0 cache reads; pagination
API Server saturation	One request class starving others	API Priority & Fairness
etcd write latency	WAL fsync on slow disk	NVMe-backed dedicated etcd nodes
Object count growth	Events and completed jobs accumulating	TTL controller; Event TTL; garbage collection
Scheduler throughput	Single scheduling pipeline	Increase parallelism; tune percentageOfNodesToScore
Pod startup regression	New admission webhook or CNI change	ClusterLoader2 SLI regression test in CI

At hyperscale, the control plane is a distributed system in its own right. Every component — API Server, etcd, scheduler, kubelet — has its own throughput ceiling, and they interact.

The difference between a cluster that scales and one that doesn't is almost always instrumentation: if you can measure each SLI independently, you can find the bottleneck.

---

Related Posts

• NVIDIA DCGM — the GPU-level metrics that complement node/pod-level performance data here
• Kubernetes Topology Manager — NUMA misalignment is one of the most common causes of the performance cliffs this post covers