Kubernetes API Server Internals

The API Server (kube-apiserver) is the heart of Kubernetes.

Every operation goes through it — without exception.

Whether you're using kubectl, Helm, ArgoCD, Terraform, or you're a Controller, Scheduler, or Kubelet — you talk to the API Server.

flowchart LR

User["kubectl / Client"]

-->APIServer["API Server"]

APIServer

-->etcd

APIServer

-->Scheduler

APIServer

-->Controller

APIServer

-->Kubelet

Notice: nothing talks directly to etcd. Everything goes through the API Server.

---

Why Everything Goes Through the API Server

Imagine a bank.

Customers don't walk into the vault.

Customer → Bank Teller → Vault

The API Server is Kubernetes' bank teller. It:

• Authenticates every caller
• Authorizes every action
• Validates every object
• Persists state to etcd
• Notifies every watcher of changes

Direct etcd access would bypass all of this — no auth, no validation, no audit trail.

---

The Request Lifecycle

Every request follows the same six-stage pipeline.

flowchart LR

Request

-->Authentication

Authentication

-->Authorization

Authorization

-->AdmissionControllers["Admission\nControllers"]

AdmissionControllers

-->Validation

Validation

-->Storage

Storage

-->WatchNotification["Watch\nNotification"]

WatchNotification

-->Response

---

Step 1: Authentication

The API Server first asks: Who are you?

Method	How it works
Client Certificates	TLS cert signed by cluster CA — used by kubelets, controllers
Bearer Tokens	Service Account JWT tokens — used by pods
OIDC	Federated identity (Google, Dex, Keycloak) — used for human users
Webhook	External auth service called per request

Every kubectl command sends your kubeconfig certificate. The API Server verifies it against the cluster CA.

If authentication fails → 401 Unauthorized. The request stops here.

---

Step 2: Authorization

Authentication answers who you are.

Authorization answers: what are you allowed to do?

flowchart LR

Request["alice\nPOST /api/v1/pods"]

-->RBAC["RBAC check\nCan alice create pods\nin namespace prod?"]

RBAC

-->|"Role binding found"| Allow["Allow → continue"]

RBAC

-->|"No binding"| Deny["403 Forbidden"]

RBAC

Role-Based Access Control is the default and most common authorization mode.

It has four objects:

Object	Scope	Purpose
Role	Namespace	Grants permissions within one namespace
ClusterRole	Cluster	Grants permissions cluster-wide
RoleBinding	Namespace	Binds a Role to a user/group/service account
ClusterRoleBinding	Cluster	Binds a ClusterRole cluster-wide

Example — a developer who can only view pods in production:

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: pod-reader
  namespace: production
rules:
- apiGroups: [""]
  resources: ["pods"]
  verbs: ["get", "list", "watch"]

---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: alice-pod-reader
  namespace: production
subjects:
- kind: User
  name: alice
roleRef:
  kind: Role
  name: pod-reader
  apiGroup: rbac.authorization.k8s.io

Now alice can kubectl get pods -n production but cannot delete, create, or access other namespaces.

Other authorization modes: Node (kubelets authorizing node-scoped requests), Webhook (external policy engine), ABAC (attribute-based, rarely used).

---

Step 3: Admission Controllers

After authorization, admission controllers run.

Authentication says: You're Alice.

Authorization says: Alice may create Pods.

Admission controllers ask: should this specific Pod be modified or rejected?

They act as middleware — running after auth but before persistence.

flowchart LR

Request

-->MutatingAdmission["Mutating\nAdmission"]

MutatingAdmission

-->ValidatingAdmission["Validating\nAdmission"]

ValidatingAdmission

-->Store

Mutating Admission

Mutating webhooks can modify the incoming object before it is stored.

User submits:

containers:
- name: app
  image: nginx

A mutating controller automatically injects:

containers:
- name: app
  image: nginx
  resources:
    requests:
      cpu: "100m"
      memory: "128Mi"
  securityContext:
    runAsNonRoot: true

No user action required. Common examples: Istio sidecar injection, default resource requests, image pull policy enforcement.

Validating Admission

Validating webhooks can only accept or reject — they cannot modify.

securityContext:
  privileged: true

A policy (Kyverno, OPA Gatekeeper, Pod Security Admission) checks this and rejects the request.

403 Forbidden: pods "my-pod" is forbidden: privileged containers are not allowed

Mutating always runs before validating — so a mutating webhook cannot sneak past a validating one by injecting a disallowed field.

---

Step 4: Schema Validation

After admission, Kubernetes validates the object against its OpenAPI schema.

This is distinct from admission — it checks structural correctness, not policy.

replicas: -5       # rejected — must be >= 0

apiVersion: apps/v10  # rejected — unknown API version

containerPort: "80"   # rejected — must be integer, not string

If schema validation fails → 422 Unprocessable Entity.

---

Step 5: Persist to etcd

Only now is the object written.

sequenceDiagram

participant Client
participant APIServer as "API Server"
participant Admission
participant etcd

Client->>APIServer: Create Pod

APIServer->>Admission: Mutate + Validate

Admission-->>APIServer: OK

APIServer->>etcd: Store Object

etcd-->>APIServer: Success (resourceVersion: 12345)

APIServer-->>Client: 201 Created

ResourceVersion and Optimistic Concurrency

Every object returned by the API Server has a resourceVersion field:

metadata:
  name: my-pod
  resourceVersion: "12345"

This is a monotonically increasing revision number stored in etcd.

If two clients try to update the same object simultaneously, only the first one wins:

sequenceDiagram

participant ClientA
participant ClientB
participant APIServer as "API Server"

ClientA->>APIServer: GET pod (resourceVersion=12345)
ClientB->>APIServer: GET pod (resourceVersion=12345)

ClientA->>APIServer: PUT pod (resourceVersion=12345) ← arrives first
APIServer-->>ClientA: 200 OK (new resourceVersion=12346)

ClientB->>APIServer: PUT pod (resourceVersion=12345) ← stale
APIServer-->>ClientB: 409 Conflict — resourceVersion mismatch

Client B must re-fetch and retry. This is optimistic concurrency — no locks held, conflicts detected at write time.

Controllers rely on this to safely reconcile without distributed locks.

---

Step 6: Watch Cache and Fan-Out

After writing to etcd, the API Server notifies every interested watcher.

What is the Watch Cache?

The API Server maintains an in-memory ring buffer of recent events, keyed by resource type and namespace. Clients open long-lived HTTP/2 watch connections and receive events as they happen.

flowchart LR

etcd

-->WatchCache["Watch Cache\n(in-memory ring buffer)"]

WatchCache

-->Scheduler

WatchCache

-->Controller

WatchCache

-->Kubelet

WatchCache

-->Operators

The API Server watches etcd once. Everyone else watches the API Server.

Why Not Watch etcd Directly?

With 10,000 nodes:

10,000 Kubelets + 50 Controllers + 20 Operators + Schedulers = ~10,100 watchers

If all of them opened watch connections to etcd, the load would be crushing.

The Watch Cache acts as a fan-out multiplexer — one etcd watch, thousands of API Server watches.

List + Watch

The standard pattern every controller and kubelet uses:

1. LIST — fetch all current objects and store them in local cache
2. WATCH — subscribe to future events from the Watch Cache
3. From that point on, the local cache stays current via events — no more full lists

This is why a controller restart is fast: it lists once, then tails events. It doesn't re-read etcd from scratch on every decision.

---

API Priority & Fairness (APF)

At scale, the API Server can be overwhelmed by a flood of requests from one source.

APF classifies every incoming request into a PriorityLevelConfiguration and routes it to a flow queue.

flowchart LR

Requests

-->Classify["FlowSchema\nclassification"]

Classify

-->High["exempt\n(system:masters)"]

Classify

-->Med["workload-high\n(system controllers)"]

Classify

-->Low["global-default\n(user requests)"]

High & Med & Low

-->Workers["API Server\nworker goroutines"]

Built-in priority levels (ordered highest to lowest):

Level	Who uses it	Can be queued?
exempt	system:masters, health checks	No — always passes through
node-high	Kubelet node status updates	Yes
system	System controllers (deployment, replicaset)	Yes
leader-election	Controller leader election	Yes
workload-high	Authenticated service accounts	Yes
workload-low	kubectl user requests	Yes
global-default	Everything else	Yes

This ensures a kubectl get pods flood from a CI system cannot starve the Deployment Controller or kubelet heartbeats.

APF is configured via two CRDs: FlowSchema (matches requests to a priority level) and PriorityLevelConfiguration (sets concurrency limits and queue depth per level).

---

Complete Request Flow

flowchart TB

Request

-->Authentication

Authentication

-->Authorization

Authorization

-->MutatingAdmission["Mutating Admission\n(inject defaults, sidecars)"]

MutatingAdmission

-->ValidatingAdmission["Validating Admission\n(policy check)"]

ValidatingAdmission

-->SchemaValidation["Schema Validation\n(OpenAPI)"]

SchemaValidation

-->etcd["etcd\n(persist + resourceVersion)"]

etcd

-->WatchCache["Watch Cache"]

WatchCache

-->Controllers & Scheduler & Kubelet

---

Key Takeaways

Stage	What it does	Failure code
Authentication	Verifies caller identity	401
Authorization (RBAC)	Checks verb/resource/namespace permissions	403
Mutating Admission	Injects defaults, sidecars, labels	400 / webhook error
Validating Admission	Enforces policy — cannot modify	403
Schema Validation	Enforces OpenAPI object structure	422
etcd persistence	Writes object, assigns resourceVersion	500 on failure
Watch Cache fan-out	Notifies all watchers with no etcd round-trip	—
API Priority & Fairness	Protects critical traffic under load	429 Too Many Requests

The API Server is not just a REST gateway. It is the policy engine, the validator, the state store, and the event bus for the entire cluster — all in one binary.