Kubernetes Informers & Controllers Explained

One of Kubernetes' greatest strengths is that it is self-healing.

Pods crash. Nodes disappear. Deployments scale. Yet the cluster automatically returns to the desired state.

How?

Through Controllers — control loops that continuously watch the cluster and drive it toward what you declared.

flowchart LR

User["kubectl apply"]

-->APIServer["API Server"]

APIServer

-->Informer

Informer

-->Controller

Controller

-->APIServer

This loop is called the Reconciliation Loop. It is the heart of Kubernetes.

---

Desired State vs Current State

Kubernetes is a state machine, not a task runner.

You don't tell Kubernetes what to do. You tell it what you want.

replicas: 3

Kubernetes figures out how to get there and keeps it there forever.

	Traditional automation	Kubernetes controller
Model	Imperative — "run this command"	Declarative — "make it so"
After failure	Stops, requires human	Automatically retries
After drift	Does nothing	Detects and corrects
Lifecycle	Runs once	Runs forever

---

The Reconciliation Loop

Every controller runs the same cycle indefinitely.

flowchart LR

Observe["Observe\ncurrent state"]

-->Compare["Compare to\ndesired state"]

Compare

-->Act["Take corrective\naction (if needed)"]

Act

-->Observe

Observe → Compare → Act

The key insight: controllers never "finish". A controller that ran successfully a moment ago runs again when the next event arrives — forever.

This means reconciliation is idempotent. Running it ten times in a row has the same result as running it once. This property is what makes the system resilient: any component can restart at any time and simply pick up where it left off.

The Worker Loop

Inside every controller, a worker goroutine processes items from the work queue:

flowchart LR

Queue

-->Worker

Worker

-->ReadCache["Read desired state\nfrom local cache"]

ReadCache

-->ReadActual["Read current state\nfrom API / cache"]

ReadActual

-->Diff["Diff"]

Diff

-->|"Gap exists"| Act["Call API to\ncorrect gap"]

Diff

-->|"In sync"| Done["Done —\npop from queue"]

Act

-->Done

Reading desired state from the local cache (not the API Server) is critical for performance — it avoids an API round-trip on every reconciliation.

---

Built-in Controllers

Every Kubernetes resource type has a controller responsible for it. They all live inside a single binary: kube-controller-manager.

Resource	Controller	What it reconciles
Deployment	Deployment Controller	Creates/updates ReplicaSets to match spec
ReplicaSet	ReplicaSet Controller	Creates/deletes Pods to match replicas
StatefulSet	StatefulSet Controller	Ordered Pod creation and persistent volume binding
DaemonSet	DaemonSet Controller	Ensures one Pod per matching node
Job	Job Controller	Tracks completions, retries failures
Node	Node Controller	Marks nodes unschedulable and evicts pods on failure
Endpoints	Endpoints Controller	Keeps Service endpoint slices updated
Namespace	Namespace Controller	Cleans up all resources when namespace is deleted

kube-controller-manager and Leader Election

In an HA control plane, multiple kube-controller-manager replicas run — but only one is active at a time.

They use a leader election lease stored in a ConfigMap or Lease object in etcd. Only the leader runs reconciliation loops. If the leader crashes, a standby acquires the lease within seconds and takes over.

flowchart LR

KCM1["kube-controller-manager\n(leader — active)"]

KCM2["kube-controller-manager\n(standby — watches lease)"]

KCM3["kube-controller-manager\n(standby — watches lease)"]

KCM1

-->Lease["Lease object\nrenewed every 2s"]

KCM1

-->Reconciliation["Running\nreconciliation loops"]

This is the same leader election pattern available to custom controllers via the leaderelection package.

---

Informers

A controller needs to know when something changes. But polling the API Server continuously would destroy it at scale.

The solution is Informers — a client-side abstraction over the API Server's List+Watch mechanism.

flowchart LR

APIServer["API Server"]

-->|"Watch stream"| Informer

Informer

-->Cache["Thread-safe\nlocal cache"]

Informer

-->EventHandlers["Event handlers\n(OnAdd / OnUpdate / OnDelete)"]

EventHandlers

-->WorkQueue

List + Watch

An Informer starts by listing all current objects, then opens a long-lived watch stream for future changes.

1. LIST /api/v1/pods — fetch all pods, populate the local cache
2. WATCH /api/v1/pods?resourceVersion=12345 — receive events from that revision onward

If the watch stream drops (network blip, API Server restart), the Informer reconnects automatically and resumes from its last-seen resourceVersion. No events are lost.

Local Cache (Thread-Safe Store)

After the initial list, the Informer maintains an in-memory store that stays current via watch events.

Controllers always read from this local cache — never directly from the API Server:

// Fast — reads from in-memory cache
pod, err := podLister.Pods("default").Get("my-pod")

// Slow — network round-trip to API Server every time
pod, err := clientset.CoreV1().Pods("default").Get(ctx, "my-pod", metav1.GetOptions{})

The cache is thread-safe. Multiple goroutines can read it simultaneously. This is what makes high-throughput controllers possible — thousands of reconciliations per second without any API Server pressure.

Event Handlers and the Work Queue

Controllers register callbacks on the Informer:

informer.AddEventHandler(cache.ResourceEventHandlerFuncs{
    AddFunc:    func(obj interface{}) { enqueue(obj) },
    UpdateFunc: func(old, new interface{}) { enqueue(new) },
    DeleteFunc: func(obj interface{}) { enqueue(obj) },
})

Critically: handlers enqueue the object key, not the object itself. The actual reconciliation happens later in a worker goroutine — not inside the event handler. This keeps the handler fast and non-blocking.

---

Shared Informers

Suppose the Deployment Controller and HPA Controller both need to watch Pods.

Without sharing:

flowchart LR

API

-->DeploymentCtrl["Deployment Controller\n(own watch + cache)"]

API

-->HPACtrl["HPA Controller\n(own watch + cache)"]

API

-->RSCtrl["ReplicaSet Controller\n(own watch + cache)"]

Three watches, three caches, three copies of every Pod in memory.

With a SharedInformerFactory:

flowchart LR

APIServer["API Server"]

-->|"one watch"| SharedInformer["Shared Informer\n(one cache)"]

SharedInformer

-->DeploymentCtrl

SharedInformer

-->HPACtrl

SharedInformer

-->RSCtrl

One watch stream. One cache. All controllers share the same data.

In kube-controller-manager, every built-in controller is initialized with the same SharedInformerFactory. This is one of the most important scalability decisions in the Kubernetes codebase.

---

Work Queue — Reliability Guarantees

The work queue is not a simple FIFO. It provides four guarantees that make controllers robust:

Guarantee	What it means
Deduplication	If the same key is enqueued 100 times before a worker processes it, it is processed once
Rate limiting	Workers are throttled to avoid overwhelming the API Server
Exponential backoff	On failure, the key is re-enqueued after 5 ms → 10 ms → 20 ms → ... up to 1000 s
Concurrency	Multiple workers drain the queue in parallel (configurable)

Exponential Backoff in Practice

flowchart TD

Reconcile["Reconcile pod/nginx"]

-->|"API Server error"| Fail["Failure"]

Fail

-->Backoff1["Re-enqueue after 5 ms"]

Backoff1

-->|"Still failing"| Backoff2["Re-enqueue after 10 ms"]

Backoff2

-->|"Still failing"| Backoff3["Re-enqueue after 20 ms"]

Backoff3

-->|"Eventually"| Backoff4["Re-enqueue after ~16 min (max)"]

This means a controller that encounters a transient error (API Server blip, etcd timeout) will automatically retry with increasing delays — no manual intervention required.

---

Owner References and Garbage Collection

Kubernetes objects form a parent-child ownership tree tracked via ownerReferences.

# Pod owned by a ReplicaSet
metadata:
  ownerReferences:
  - apiVersion: apps/v1
    kind: ReplicaSet
    name: web-d5b9f7c44
    uid: a1b2c3d4-...
    controller: true
    blockOwnerDeletion: true

The ownership chain for a Deployment looks like:

flowchart LR

Deployment

-->|"owns"| ReplicaSet

ReplicaSet

-->|"owns"| Pod1

ReplicaSet

-->|"owns"| Pod2

ReplicaSet

-->|"owns"| Pod3

When you delete a Deployment, the Garbage Collector controller follows owner references and deletes all owned objects — ReplicaSets and their Pods.

Two deletion modes:

Mode	Behavior
Foreground	Parent waits for all children to be deleted before disappearing
Background (default)	Parent deleted immediately, children cleaned up asynchronously

---

Generation and ObservedGeneration

How does a controller know whether it has processed the current spec, or is still acting on a stale one?

Every Kubernetes object has a generation counter in its spec. Every time the spec changes, generation increments. The controller writes observedGeneration into the status once it has fully reconciled that spec version.

# After kubectl set image deployment/web ...
status:
  observedGeneration: 4   # controller has reconciled spec v4
  replicas: 3
  updatedReplicas: 3
  readyReplicas: 3

# If a controller crashes mid-rollout:
status:
  observedGeneration: 3   # controller only processed spec v3
  updatedReplicas: 1      # rollout is incomplete

This lets kubectl rollout status and other tooling know whether a rollout is truly complete or still in progress — without guessing.

---

Finalizers

When Kubernetes deletes an object, it normally removes it from etcd immediately.

But some resources require cleanup before deletion — releasing a cloud load balancer, removing DNS records, draining a queue.

Finalizers block deletion until a controller performs the cleanup.

flowchart TD

Delete["kubectl delete service lb-svc"]

-->SetTimestamp["API Server sets\ndeletionTimestamp\n(object still exists)"]

SetTimestamp

-->Controller["Controller notices\ndeletionTimestamp"]

Controller

-->Cleanup["Deregisters cloud\nload balancer"]

Cleanup

-->RemoveFinalizer["Controller removes\nfinalizer from object"]

RemoveFinalizer

-->APIServer2["API Server deletes\nobject from etcd"]

If a controller is deleted before removing its finalizer, the object gets stuck with a deletionTimestamp but never disappears. This is the source of the classic "namespace stuck in Terminating" problem.

---

Reconciliation in Action

A complete walk-through of kubectl apply -f deployment.yaml:

sequenceDiagram

participant User
participant API as API Server
participant DC as Deployment Controller
participant RC as ReplicaSet Controller
participant Sched as Scheduler
participant K as Kubelet

User->>API: Apply Deployment (replicas=3)
API->>DC: ADDED event (via Shared Informer)
DC->>API: Create ReplicaSet (replicas=3)
API->>RC: ADDED event
RC->>API: Create Pod A, Pod B, Pod C
API->>Sched: ADDED events (3 unscheduled Pods)
Sched->>API: Bind Pod A → Node 1
Sched->>API: Bind Pod B → Node 2
Sched->>API: Bind Pod C → Node 3
API->>K: Pod A assigned (Node 1 kubelet)
K->>K: Pull image, start container
K->>API: Update Pod A status → Running

Every step is a reconciliation triggered by an event from the Shared Informer. No component polls. No component directly calls the next one. They are fully decoupled through the API Server.

---

The Operator Pattern

The same Informer + WorkQueue + Reconcile loop that powers built-in controllers is available to anyone building a custom controller.

This is the Operator pattern — a custom controller that manages a domain-specific resource.

flowchart LR

CRD["Custom Resource\nMyDatabase"]

-->Informer

Informer

-->OperatorController["Operator Controller"]

OperatorController

-->|"Creates"| StatefulSet & Service & ConfigMap

OperatorController

-->|"Reconciles"| MyDatabase["MyDatabase\nstatus"]

Popular examples: Cert-Manager, Strimzi (Kafka), CloudNativePG, Argo Workflows, Kubeflow Training Operator.

The operator extends Kubernetes' self-healing model to your own application lifecycle — backups, failover, schema migrations, scaling — all expressed as a reconciliation loop.

---

Key Takeaways

Component	Responsibility
Controller	Runs reconciliation: observes current state, compares to desired, takes corrective action
Informer	Wraps List+Watch — populates local cache and fires event handlers
Shared Informer	One watch + one cache shared across all controllers — essential for scalability
Local Cache	In-memory thread-safe store — controllers read from here, never from the API Server directly
Work Queue	Deduplication + rate limiting + exponential backoff + concurrent workers
Owner References	Tracks parent-child ownership; drives garbage collection on deletion
Generation / ObservedGeneration	Lets a controller signal "I have fully processed spec version N"
Finalizers	Block object deletion until a controller completes cleanup
Operator Pattern	Extends the same loop to custom resources — the foundation of the cloud-native ecosystem

The controller pattern is why Kubernetes heals itself. No orchestration script, no runbook, no human in the loop — just a loop that runs forever, comparing what is to what should be, and closing the gap.