Kubernetes API Server Internals

Kubernetes API Server Internals ⚡

The API Server is the central communication hub of Kubernetes.

The API Server (kube-apiserver) is the heart of Kubernetes.

• It is the only component that communicates directly with etcd.
• Every operation goes through it — without exception.

Whether you're using kubectl, Helm, ArgoCD, Terraform, or you're a Controller, Scheduler, or Kubelet — you talk to the API Server.

flowchart LR

User["kubectl / Client"] -->APIServer["API Server  ⚡"]

APIServer-->Storage[("etcd")]
APIServer-->Scheduler
APIServer-->Controller
APIServer-->Kubelet
APIServer --> Proxy[kube-proxy]

Notice: nothing talks directly to etcd. Everything goes through the API Server.

---

Why Everything Goes Through the API Server

Imagine a bank.

Customers don't walk into the vault.

Customer → Bank Teller → Vault

The API Server is Kubernetes' bank teller. It:

• Authenticates every caller
• Authorizes every action
• Validates every object
• Persists state to etcd
• Notifies every watcher of changes

Direct etcd access would bypass all of this — no auth, no validation, no audit trail.

---

Complete Request Flow

Suppose a user runs

kubectl apply -f nginx.yaml

The API Server is responsible for:

• Authenticating requests
• Authorizing requests
• Admission control
• Validating API objects
• Persisting objects in etcd
• Watching and notifying cluster components
• Serving the Kubernetes REST API

API server request flow

sequenceDiagram
    participant User
    participant API as API Server
    participant Auth
    participant Admission
    participant ETCD
    participant Scheduler
    participant Kubelet

    User->>API: POST /api/v1/pods

    API->>Auth: Authenticate

    Auth-->>API: Success

    API->>API: Authorize Request

    API->>Admission: Admission Controllers

    Admission-->>API: Approved

    API->>ETCD: Store Pod

    API-->>User: 201 Created

    Scheduler->>API: Watch Pending Pods

    API-->>Scheduler: Pod Created

    Scheduler->>API: Bind Pod to Node

    API->>ETCD: Update Pod

    Kubelet->>API: Watch Assigned Pods

    API-->>Kubelet: Pod Spec

Simplified API Server flow

flowchart LR

    Request-->Authentication

    Authentication-->Authorization
    Authorization-->AdmissionControllers["Admission <br/> Controllers"]
    AdmissionControllers-->Validation
    Validation-->Storage[("Storage")]

    Storage-->WatchNotification["Watch <br/> Notification"]
    WatchNotification-->Response

---

Step 1: Authentication

The API Server first asks: Who are you?

Method	How it works
Client Certificates	TLS cert signed by cluster CA — used by kubelets, controllers
Bearer Tokens	Service Account JWT tokens — used by pods
OIDC	Federated identity (Google, Dex, Keycloak) — used for human users
Webhook	External auth service called per request

Every kubectl command sends your kubeconfig certificate. The API Server verifies it against the cluster CA.

If authentication fails → 401 Unauthorized. The request stops here.

---

Step 2: Authorization 🔐

👤 Authentication

answers who you are.

• Client Certificates
• Bearer Tokens
• Service Accounts
• OpenID Connect (OIDC)
• Webhook Authentication

🪪 Authorization

answers: what are you allowed to do?

• RBAC
• ABAC
• Node Authorization
• Webhook

flowchart LR

    Request["alice <br/> POST /api/v1/pods"]
    Request-->RBAC["RBAC check <br/> Can alice create pods <br/> in namespace prod?"]

    RBAC-->|"Role binding found"| Allow["Allow → continue"]
    RBAC-->|"No binding"| Deny["403 Forbidden"]

RBAC

Role-Based Access Control is the default and most common authorization mode.

It has four objects:

Object	Scope	Purpose
Role	Namespace	Grants permissions within one namespace
ClusterRole	Cluster	Grants permissions cluster-wide
RoleBinding	Namespace	Binds a Role to a user/group/service account
ClusterRoleBinding	Cluster	Binds a ClusterRole cluster-wide

Example — a developer who can only view pods in production:

    apiVersion: rbac.authorization.k8s.io/v1
    kind: Role
    metadata:
      name: pod-reader
      namespace: production
    rules:
    - apiGroups: [""]
      resources: ["pods"]
      verbs: ["get", "list", "watch"]
    ---
    apiVersion: rbac.authorization.k8s.io/v1
    kind: RoleBinding
    metadata:
      name: alice-pod-reader
      namespace: production
    subjects:
    - kind: User
      name: alice
    roleRef:
      kind: Role
      name: pod-reader
      apiGroup: rbac.authorization.k8s.io

Now alice can kubectl get pods -n production but cannot delete, create, or access other namespaces.

Other authorization modes: Node (kubelets authorizing node-scoped requests), Webhook (external policy engine), ABAC (attribute-based, rarely used).

---

Step 3: Admission Controllers 🛡️

Admission controllers can modify or reject objects before they are stored.

After authorization, admission controllers run.

• Authentication says: You're Alice.
• Authorization says: Alice may create Pods.
• Admission controllers ask: should this specific Pod be modified or rejected?

They act as middleware — running after auth but before persistence.

flowchart TD

Request-->MutatingAdmission["🧬 Mutating <br/> Admission"]

MutatingAdmission-->ValidatingAdmission["📋 Validating <br/> Admission"]

ValidatingAdmission-->Store[("store")]

1. Mutating Admission 🧬

Mutating webhooks can modify the incoming object before it is stored.

User submits:

containers:
- name: app
  image: nginx

A mutating controller automatically injects:

containers:
- name: app
  image: nginx
  resources:
    requests:
      cpu: "100m"
      memory: "128Mi"
  securityContext:
    runAsNonRoot: true

No user action required. Common examples: Istio sidecar injection, default resource requests, image pull policy enforcement.

2. Validating Admission 📋

Validating webhooks can only accept or reject — they cannot modify.

securityContext:
  privileged: true

A policy (Kyverno, OPA Gatekeeper, Pod Security Admission) checks this and rejects the request.

403 Forbidden: pods "my-pod" is forbidden: privileged containers are not allowed

Mutating always runs before validating — so a mutating webhook cannot sneak past a validating one by injecting a disallowed field.

---

Step 4: Schema Validation 🔎

After admission, Kubernetes validates the object against its OpenAPI schema.

This is distinct from admission — it checks structural correctness, not policy.

replicas: -5       # rejected — must be >= 0

apiVersion: apps/v10  # rejected — unknown API version

containerPort: "80"   # rejected — must be integer, not string

If schema validation fails → 422 Unprocessable Entity.

---

Step 5: Persist to etcd 💾

Only now is the object written.

sequenceDiagram

participant Client
participant APIServer as "API Server"
participant Admission
participant etcd

Client->>APIServer: Create Pod
APIServer->>Admission: Mutate + Validate
Admission-->>APIServer: OK

APIServer->>etcd: Store Object
etcd-->>APIServer: Success (resourceVersion: 12345)

APIServer-->>Client: 201 Created

ResourceVersion and Optimistic Concurrency

Every object returned by the API Server has a resourceVersion field:

metadata:
  name: my-pod
  resourceVersion: "12345"

This is a monotonically increasing revision number stored in etcd.

If two clients try to update the same object simultaneously, only the first one wins:

sequenceDiagram

participant ClientA
participant ClientB
participant APIServer as "API Server"

ClientA->>APIServer: GET pod (resourceVersion=12345)
ClientB->>APIServer: GET pod (resourceVersion=12345)

ClientA->>APIServer: PUT pod (resourceVersion=12345) ← arrives first
APIServer-->>ClientA: 200 OK (new resourceVersion=12346)

ClientB->>APIServer: PUT pod (resourceVersion=12345) ← stale
APIServer-->>ClientB: 409 Conflict — resourceVersion mismatch

Client B must re-fetch and retry. This is optimistic concurrency — no locks held, conflicts detected at write time.

Controllers rely on this to safely reconcile without distributed locks.

---

Step 6: Watch Cache and Fan-Out 👀

API Server is the only component that reads from and writes to etcd.

After writing to etcd, the API Server notifies every interested watcher.

What is the Watch Cache?

The API Server maintains an in-memory ring buffer of recent events, keyed by resource type and namespace.

Clients open long-lived HTTP/2 watch connections and receive events as they happen.

flowchart LR


    Storage[("etcd")]

    Storage-->WatchCache["Watch Cache <br/> (in-memory ring buffer)"]

    WatchCache-->Scheduler
    WatchCache-->Controller
    WatchCache-->Kubelet
    WatchCache-->kubeproxy[kube-proxy]
    WatchCache-->Operators

The API Server watches etcd once. Everyone else watches the API Server.

Why Not Watch etcd Directly?

With 10,000 nodes:

10,000 Kubelets + 
50 Controllers + 
20 Operators + 
Schedulers = ~10,100 watchers

If all of them opened watch connections to etcd, the load would be crushing.

The Watch Cache acts as a fan-out multiplexer — one etcd watch, thousands of API Server watches.

List + Watch

The standard pattern every controller and kubelet uses:

1. LIST — fetch all current objects and store them in local cache
2. WATCH — subscribe to future events from the Watch Cache
3. From that point on, the local cache stays current via events — no more full lists

This is why a controller restart is fast: it lists once, then tails events. It doesn't re-read etcd from scratch on every decision.

---

API Priority & Fairness (APF)

At scale, the API Server can be overwhelmed by a flood of requests from one source.

APF classifies every incoming request into a PriorityLevelConfiguration and routes it to a flow queue.

flowchart LR

Requests-->Classify["FlowSchema <br/> classification"]

Classify-->High["exempt <br/> (system:masters)"]
Classify-->Med["workload-high <br/> (system controllers)"]
Classify-->Low["global-default <br/> (user requests)"]

High & Med & Low-->Workers["API Server <br/> worker goroutines"]

Built-in priority levels (ordered highest to lowest):

Level	Who uses it	Can be queued?
exempt	system:masters, health checks	No — always passes through
node-high	Kubelet node status updates	Yes
system	System controllers (deployment, replicaset)	Yes
leader-election	Controller leader election	Yes
workload-high	Authenticated service accounts	Yes
workload-low	kubectl user requests	Yes
global-default	Everything else	Yes

This ensures a kubectl get pods flood from a CI system cannot starve the Deployment Controller or kubelet heartbeats.

APF is configured via two CRDs: FlowSchema (matches requests to a priority level) and PriorityLevelConfiguration (sets concurrency limits and queue depth per level).

---

Complete Request Flow

flowchart TB

Request-->Authentication
Authentication-->Authorization

Authorization-->MutatingAdmission["Mutating Admission <br/> (inject defaults, sidecars)"]
MutatingAdmission-->ValidatingAdmission["Validating Admission <br/> (policy check)"]

ValidatingAdmission-->SchemaValidation["Schema Validation <br/> (OpenAPI)"]

SchemaValidation-->etcd[("etcd <br/> (persist + resourceVersion)")]

etcd-->WatchCache["Watch Cache"]

WatchCache-->Controllers & Scheduler & Kubelet

---

Key Takeaways

Stage	What it does	Failure code
Authentication	Verifies caller identity	401
Authorization (RBAC)	Checks verb/resource/namespace permissions	403
Mutating Admission	Injects defaults, sidecars, labels	400 / webhook error
Validating Admission	Enforces policy — cannot modify	403
Schema Validation	Enforces OpenAPI object structure	422
etcd persistence	Writes object, assigns resourceVersion	500 on failure
Watch Cache fan-out	Notifies all watchers with no etcd round-trip	—
API Priority & Fairness	Protects critical traffic under load	429 Too Many Requests

The API Server is not just a REST gateway. It is the policy engine, the validator, the state store, and the event bus for the entire cluster — all in one binary.

---

Related Posts

• etcd Architecture Explained — the API server's only persistent backing store
• Kubernetes Scheduler Internals — the other core control-plane component, watching the API server for unscheduled pods