Kubernetes Performance at Scale

Running Kubernetes with 100 nodes is very different from running it with 10,000 nodes.

The failures that emerge at hyperscale are not crashes — they are latency cliffs: operations that take 10 ms at 100 nodes take 10 seconds at 10,000 nodes, and the cluster appears healthy right until it doesn't.

This post covers where those cliffs are, how to measure them, and how the Kubernetes ecosystem addresses each one.

---

Kubernetes Scalability SLIs and SLOs

The Kubernetes project defines official Service Level Indicators (SLIs) and Service Level Objectives (SLOs) — measurable guarantees about control plane behavior at the certified scale ceiling.

Certified Scale Ceiling (as of Kubernetes 1.30)

Resource	Limit
Nodes	5,000
Total Pods	150,000
Pods per node	110
Total Services	10,000
Total Namespaces	10,000

These are not hard limits — clusters can exceed them — but performance guarantees only apply within these bounds.

Official SLOs

SLI	SLO
Mutating API call latency (POST/PUT/PATCH/DELETE)	p99 < 1 s, excluding watch
Non-streaming read API call latency (GET/LIST)	p99 < 30 s for large resources, < 1 s for small
Pod startup latency (scheduled + running, no image pull)	p99 < 5 s
In-cluster DNS lookup latency	p99 < 5 s
Scheduling throughput	≥ 100 Pods/s average

The pod startup latency SLO is the most operationally visible — it is what users feel when they scale a Deployment.

---

Pod Startup Latency — The Key SLI

Pod startup latency is not one number. It is a pipeline of stages, each of which can become the bottleneck independently.

flowchart LR

Submit["kubectl apply\n(t=0)"]

-->Admitted["API Server\nadmitted\n(t=50ms)"]

Admitted

-->Scheduled["Scheduler\nbinds to node\n(t=200ms)"]

Scheduled

-->ImagePull["Image pull\n(t=1–30s)"]

ImagePull

-->ContainerCreate["Container\ncreated\n(t+100ms)"]

ContainerCreate

-->Running["Pod Running\n(t=?)"]

Stage	Typical duration	What slows it down
API Server admission	10–100 ms	Slow admission webhooks
Scheduler decision	50–500 ms	Queue depth, filter/score at scale
Image pull	0–60 s	Cold node, large image, registry rate limits
Container create	50–200 ms	CNI plugin latency
Probe warm-up	0–30 s	initialDelaySeconds on readiness probe

The SLO of p99 < 5 s excludes image pull time — it assumes the image is already cached. When benchmarking, always separate the two.

Key Prometheus metrics:

# Scheduling queue time
scheduler_e2e_scheduling_duration_seconds

# API Server admission latency
apiserver_admission_webhook_admission_duration_seconds

# Full pod startup (from creation to Running)
kubelet_pod_start_duration_seconds

---

Challenge 1: API Server Watch Storms

With 10,000 nodes and 50 controllers, there can be over 50,000 active watch connections to the API Server simultaneously.

A single busy Deployment — say, a rolling update touching 1,000 Pods — generates one event per Pod per status change. That is potentially tens of thousands of events that must be serialized and sent to every interested watcher.

flowchart LR

OneUpdate["1 Deployment\nrolling update"]

-->Events["~5,000 Pod events\n(Pending → Running → Ready)"]

Events

-->FanOut["Fan-out to\n50,000 watchers"]

FanOut

-->CPUSpike["API Server\nCPU spike"]

The Watch Cache

The Watch Cache is the primary defense. It holds a ring buffer of recent events per resource type, served entirely from memory.

flowchart LR

etcd

-->|"1 watch connection"| WatchCache["Watch Cache\n(in-memory ring buffer)"]

WatchCache

-->|"50,000 watch streams"| Clients["Controllers\nKubelets\nOperators"]

The API Server watches etcd once per resource type. All clients watch the API Server. This multiplexing is what keeps etcd alive at scale.

The Watch Cache also serves LIST requests from memory if the client passes resourceVersion=0 — skipping etcd entirely.

LIST Scalability and resourceVersion

LIST requests are a hidden scalability trap. A large LIST with no resourceVersion set forces a consistent read from etcd — serializing all objects, holding an etcd read transaction open, and loading everything into API Server memory.

# Consistent read — hits etcd, expensive
kubectl get pods --all-namespaces
# Internally: GET /api/v1/pods  (no resourceVersion → etcd read)

# Served from cache — cheap, slightly stale
kubectl get pods --all-namespaces -l app=web
# With resourceVersion=0: GET /api/v1/pods?resourceVersion=0  (cache)

Controllers using the SharedInformer always pass resourceVersion=0 for their initial LIST — this is why informer initialization does not hammer etcd even during a mass restart.

At large object counts (> 100,000 Pods), even cache-served LISTs can spike API Server memory because all objects must be serialized into the response. Pagination (the limit parameter) keeps any single response bounded:

# Pages of 500 objects instead of one massive response
kubectl get pods --chunk-size=500 --all-namespaces

Horizontal API Server Scaling

The API Server is stateless — it stores nothing locally. Multiple replicas can run behind a load balancer.

flowchart LR

Clients["kubectl\nControllers\nKubelets"]

-->LB["Load Balancer\n(e.g. HAProxy / kube-vip)"]

LB

-->AS1["kube-apiserver\nreplica 1"]

LB

-->AS2["kube-apiserver\nreplica 2"]

LB

-->AS3["kube-apiserver\nreplica 3"]

AS1 & AS2 & AS3

-->etcd["etcd cluster"]

Each replica maintains its own Watch Cache — they are independent. A client connecting to replica 2 gets the same data as one on replica 1 because both read from the same etcd cluster (with slight propagation delay).

At DGX Cloud scale, API Server replicas are sized by memory (Watch Cache grows with object count) and CPU (fan-out serialization). A cluster with 300,000 Pods may need API Server replicas with 64–128 GB RAM each.

---

Challenge 2: etcd Bottlenecks

Every write to the API Server eventually becomes a Raft log entry in etcd. At scale, write throughput and commit latency become the gating factor.

flowchart LR

APIServer["API Server\n(writes)"]

-->Leader["etcd Leader\n(Raft log)"]

Leader

-->F1["Follower 1"]

Leader

-->F2["Follower 2"]

Leader

-->|"fsync to WAL"| Disk["NVMe SSD\n(WAL fsync < 1ms p99)"]

The bottleneck is almost always disk fsync latency on the leader. Each committed log entry requires a WAL fsync. Network-attached storage (NFS, EBS gp2) regularly exceeds 10 ms per fsync — etcd expects < 1 ms.

At hyperscale, additional pressures:

Pressure	Effect	Mitigation
High write rate	Longer Raft replication latency	Batch more writes (API Server request coalescing)
Large object count	etcd DB grows, compaction takes longer	Regular compaction + defrag
DB size approaching 8 GB	etcd refuses new writes	Emergency compaction; increase --quota-backend-bytes
Leader election under load	Brief write freeze (~2s)	NVMe-backed etcd; dedicated etcd nodes

Dedicated etcd nodes with NVMe storage and no other workloads is the standard recommendation for clusters > 1,000 nodes.

---

Challenge 3: Object Count

Every Kubernetes object consumes memory in three places simultaneously:

flowchart LR

Object["Kubernetes Object"]

-->etcd["etcd\n(on disk + BoltDB page cache)"]

Object

-->WatchCache["API Server Watch Cache\n(in-memory)"]

Object

-->InformerCache["Controller Informer Cache\n(in-memory, per controller)"]

A cluster with 300,000 Pods, 300,000 ReplicaSets, and 500,000 Events might hold 5–10 million objects in etcd.

Why Events Are Dangerous

Events are the most proliferative object type. Every Pod lifecycle step generates events:

Scheduled → Pulling → Pulled → Created → Started → Readiness probe passed

Six events per Pod. 300,000 Pods. 1.8 million events — just for one rollout.

Events are also watched by monitoring tools, log aggregators, and alerting systems. One event write fans out to potentially thousands of watchers.

Mitigations:

• Events have a default TTL of 1 hour (--event-ttl on kube-apiserver, default 1h)
• Use events.k8s.io/v1 (the new Events API) which coalesces repeated events into a count field
• Deploy a separate event sink (e.g., Event Exporter to ElasticSearch) and disable in-cluster event retention for large clusters

Object Lifecycle Management

For long-lived clusters, zombie objects accumulate: completed Jobs, finished Pods, stale ReplicaSets from old Deployments.

# Completed Jobs older than 1 hour (TTL controller)
kubectl get jobs --field-selector=status.completionTime  # prune manually

# Or use TTL controller on Job:
spec:
  ttlSecondsAfterFinished: 3600

The kube-controller-manager runs a garbage collector that follows owner references and deletes orphaned objects, but it has throughput limits — at very high object counts it can lag behind.

---

Challenge 4: API Priority and Fairness (APF)

At scale, a flood of kubectl get pods from a CI system can starve the scheduler and kubelets of API Server capacity.

APF classifies every request into a FlowSchema (who is making it, what operation) and routes it to a PriorityLevelConfiguration (how much concurrency it gets).

Priority Levels and Concurrency

flowchart LR

Request["Incoming Request"]

-->FS["FlowSchema\nmatch rules"]

FS

-->PL["PriorityLevel\n(concurrency share)"]

PL

-->|"Admitted"| Workers["API Server\nworker goroutines"]

PL

-->|"Queue full"| Drop["429 Too Many Requests"]

Built-in priority levels with their concurrency shares (percentage of total server concurrency):

Level	Default share	Users
exempt	Unlimited	system:masters, health probes
node-high	40%	Kubelet node status, lease updates
system	30%	kube-system controllers, scheduler
leader-election	8%	Controller leader election
workload-high	30%	Authenticated operators
workload-low	10%	Regular user kubectl
global-default	10%	Unauthenticated, catch-all

Concurrency shares are relative — if node-high is using only 20% of its share, the remainder is lent to other levels. APF is not a strict partition.

Diagnosing APF Throttling

# See if requests are being rejected (429)
kubectl get --raw /metrics | grep apiserver_flowcontrol_rejected_requests_total

# See queue depth per priority level
kubectl get --raw /metrics | grep apiserver_flowcontrol_current_inqueue_requests

# See which FlowSchemas exist and their priority levels
kubectl get flowschemas
kubectl get prioritylevelconfigurations

If workload-low is saturated and rejecting requests, tune its nominalConcurrencyShares up or reduce the flood source.

---

Challenge 5: Scheduler Throughput

At 100,000 pending Pods, the scheduler becomes the bottleneck even if the API Server and etcd are healthy.

What Limits Throughput

The scheduler runs a scheduling cycle for one Pod at a time, but runs multiple goroutines in parallel (controlled by --parallelism, default 16).

flowchart LR

ActiveQ["Active Queue\n100,000 Pods"]

-->G1["Goroutine 1\nFilter → Score → Bind"]

ActiveQ

-->G2["Goroutine 2\nFilter → Score → Bind"]

ActiveQ

-->G3["Goroutine 3\n..."]

ActiveQ

-->G16["Goroutine 16\n..."]

Throughput ceiling: ~100–200 Pods/second for a default scheduler with 5,000 nodes.

Key Tuning Knobs

Parameter	Default	Effect
--parallelism	16	Number of concurrent scheduling goroutines
percentageOfNodesToScore	0 (auto)	Fraction of feasible nodes to score; auto-scales with cluster size
Filter plugin order	Built-in	Cheapest filters first eliminates most nodes early
--kube-api-qps	50	QPS to API Server for binding writes
--kube-api-burst	100	Burst above QPS for binding

The percentageOfNodesToScore auto-mode scores 50% of nodes at cluster size 100, dropping to ~5% at 5,000 nodes — balancing placement quality against throughput.

---

Benchmarking: ClusterLoader2 and KWOK

KWOK — Fake Clusters at Laptop Scale

Testing scheduler or controller behavior at 10,000 nodes normally requires 10,000 VMs. KWOK (Kubernetes WithOut Kubelet) simulates nodes and Pods without running any real workloads.

flowchart LR

KWOK["KWOK\n(fake node controller)"]

-->|"Creates"| FakeNodes["10,000 fake nodes\n(API objects only)"]

FakeNodes

-->API["API Server\n(sees real node objects)"]

API

-->Scheduler["Scheduler\n(schedules real Pods\nonto fake nodes)"]

API

-->Controllers["Controllers\n(reconcile normally)"]

KWOK nodes respond to kubelet heartbeat protocols — controllers and the scheduler behave identically to a real cluster. Only actual container execution is absent.

# Install KWOK
kubectl apply -f https://github.com/kubernetes-sigs/kwok/releases/latest/download/kwok.yaml

# Create 1,000 fake nodes
kwokctl create cluster --name=perf-test
kubectl apply -f - <<EOF
apiVersion: v1
kind: Node
metadata:
  name: fake-node-0001
  labels:
    type: kwok
spec:
  taints:
  - key: kwok.x-k8s.io/node
    effect: NoSchedule
EOF

ClusterLoader2 — Workload Benchmark

ClusterLoader2 is the Kubernetes project's official performance benchmark. It creates configurable workloads and measures SLI timings against the SLOs.

flowchart LR

CL2["ClusterLoader2\n(test definition YAML)"]

-->|"Creates"| Workloads["Pods / Deployments\n/ Services / Jobs"]

Workloads

-->Cluster["Cluster\n(KWOK or real)"]

Cluster

-->Metrics["Prometheus\nmetrics"]

Metrics

-->Report["SLI Report\n(p50/p90/p99 latencies)"]

A typical test definition:

name: load
namespace:
  number: 100
tuningSets:
- name: Uniformly1qps
  qpsLoad:
    qps: 1
steps:
- name: Create Deployments
  phases:
  - namespaceRange:
      min: 1
      max: 100
    replicasPerNamespace: 10
    tuningSet: Uniformly1qps
    objectBundle:
    - basename: test-deployment
      objectTemplatePath: deployment.yaml
- name: Wait for Pods to be running
  phases:
  - namespaceRange:
      min: 1
      max: 100
    replicasPerNamespace: 10
    tuningSet: Uniformly1qps
    objectBundle:
    - basename: test-deployment
      objectTemplatePath: deployment.yaml
      currentLoad:
        type: Running

ClusterLoader2 automatically collects and reports SLI measurements after each test:

API call latency:
  Metric: scheduler_e2e_scheduling_duration_seconds
  Percentile: 99
  Value: 487ms
  SLO: 1000ms ✅

Pod startup latency:
  Metric: kubelet_pod_start_duration_seconds
  Percentile: 99
  Value: 3.2s
  SLO: 5s ✅

---

Performance Metrics Reference

Metrics to monitor continuously in a production cluster at scale:

Metric	Source	Alert threshold
apiserver_request_duration_seconds	API Server	p99 > 1 s for mutating
apiserver_current_inflight_requests	API Server	Sustained > 80% of max
apiserver_flowcontrol_rejected_requests_total	API Server	Any increase
scheduler_e2e_scheduling_duration_seconds	Scheduler	p99 > 1 s
scheduler_pending_pods	Scheduler	Sustained > 1,000
etcd_disk_wal_fsync_duration_seconds	etcd	p99 > 10 ms
etcd_server_leader_changes_seen_total	etcd	> 3 per hour
etcd_mvcc_db_total_size_in_bytes	etcd	> 6 GB
kubelet_pod_start_duration_seconds	Kubelet (per node)	p99 > 5 s
apiserver_watch_events_total	API Server	Sudden spike

---

Key Takeaways

Scaling Kubernetes is not about adding more CPU or memory — it is about eliminating pressure on the control plane.

Bottleneck	Root cause	Solution
Watch storm	Too many watchers receiving too many events	Watch Cache fan-out; SharedInformers
LIST memory spike	Full etcd read into API Server memory	resourceVersion=0 cache reads; pagination
API Server saturation	One request class starving others	API Priority & Fairness
etcd write latency	WAL fsync on slow disk	NVMe-backed dedicated etcd nodes
Object count growth	Events and completed jobs accumulating	TTL controller; Event TTL; garbage collection
Scheduler throughput	Single scheduling pipeline	Increase parallelism; tune percentageOfNodesToScore
Pod startup regression	New admission webhook or CNI change	ClusterLoader2 SLI regression test in CI

At hyperscale, the control plane is a distributed system in its own right. Every component — API Server, etcd, scheduler, kubelet — has its own throughput ceiling, and they interact. The difference between a cluster that scales and one that doesn't is almost always instrumentation: if you can measure each SLI independently, you can find the bottleneck.