Kubernetes Informers & Controllers Explained

One of Kubernetes' greatest strengths is that it is self-healing.

Pods crash. Nodes disappear. Deployments scale. Yet the cluster automatically returns to the desired state.

How?

Through Controllers — control loops that continuously watch the cluster and drive it toward what you declared.

flowchart LR

    User["kubectl apply"]
    User-->APIServer["API Server"]

    APIServer-->Informer["Informer ℹ️ "]
    Informer-->Controller["Controller 🔄"]
    Controller-->APIServer["API Server ⚡"]

This loop is called the Reconciliation Loop. It is the heart of Kubernetes.

Desired State vs Current State

Kubernetes is a state machine, not a task runner.

You don't tell Kubernetes what to do. You tell it what you want.

replicas: 3

Informers detect changes. Controllers react to those changes.

Kubernetes figures out how to get there and keeps it there forever.

	Traditional automation	Kubernetes controller
Model	Imperative — "run this command"	Declarative — "make it so"
After failure	Stops, requires human	Automatically retries
After drift	Does nothing	Detects and corrects
Lifecycle	Runs once	Runs forever

---

The Reconciliation Loop

Observe → Compare → Act

The key insight: controllers never "finish". A controller that ran successfully a moment ago runs again when the next event arrives — forever.

Every controller runs the same cycle indefinitely.

flowchart TD
    Desired[Desired State]--> Observe[Observe <br/> current state]
    Observe --> Compare[Compare to <br/> desired state]
    Compare --> Act{Match?}

    Act -- Yes --> Sleep
    Act -- No --> Reconcile
    Reconcile --> Desired

This means reconciliation is idempotent. Running it ten times in a row has the same result as running it once.

This property is what makes the system resilient: any component can restart at any time and simply pick up where it left off.

The Worker Loop

Inside every controller, a worker goroutine processes items from the work queue:

flowchart TD

Queue-->Worker

Worker-->ReadCache["Read desired state <br/> from local cache"]
ReadCache-->ReadActual["Read current state <br/> from API / cache"]

ReadActual-->Diff["Diff"]

Diff-->|"Gap exists"| Act["Call API to <br/> correct gap"]

Diff-->|"In sync"| Done["Done — <br/> pop from queue"]

Act-->Done

Reading desired state from the local cache (not the API Server) is critical for performance — it avoids an API round-trip on every reconciliation.

Built-in Controllers

Every Kubernetes resource type has a controller responsible for it. They all live inside a single binary: kube-controller-manager.

Resource	Controller	What it reconciles
Deployment	Deployment Controller	Creates/updates ReplicaSets to match spec
ReplicaSet	ReplicaSet Controller	Creates/deletes Pods to match replicas
StatefulSet	StatefulSet Controller	Ordered Pod creation and persistent volume binding
DaemonSet	DaemonSet Controller	Ensures one Pod per matching node
Job	Job Controller	Tracks completions, retries failures
Node	Node Controller	Marks nodes unschedulable and evicts pods on failure
Endpoints	Endpoints Controller	Keeps Service endpoint slices updated
Namespace	Namespace Controller	Cleans up all resources when namespace is deleted

Reconciliation in Action

A complete walk-through of kubectl apply -f deployment.yaml:

sequenceDiagram

participant User
participant API as API Server
participant DC as Deployment Controller
participant RC as ReplicaSet Controller
participant Sched as Scheduler
participant K as Kubelet

User->>API: Apply Deployment (replicas=3)
API->>DC: ADDED event (via Shared Informer)
DC->>API: Create ReplicaSet (replicas=3)
API->>RC: ADDED event
RC->>API: Create Pod A, Pod B, Pod C
API->>Sched: ADDED events (3 unscheduled Pods)
Sched->>API: Bind Pod A → Node 1
Sched->>API: Bind Pod B → Node 2
Sched->>API: Bind Pod C → Node 3
API->>K: Pod A assigned (Node 1 kubelet)
K->>K: Pull image, start container
K->>API: Update Pod A status → Running

Every step is a reconciliation triggered by an event from the Shared Informer.

No component polls. No component directly calls the next one. They are fully decoupled through the API Server.

---

kube-controller-manager and Leader Election

In an HA control plane, multiple kube-controller-manager replicas run — but only one is active at a time.

They use a leader election lease stored in a ConfigMap or Lease object in etcd. Only the leader runs reconciliation loops. If the leader crashes, a standby acquires the lease within seconds and takes over.

flowchart LR

KCM1["kube-controller-manager <br/> (leader — active)"]
KCM2["kube-controller-manager <br/> (standby — watches lease)"]
KCM3["kube-controller-manager <br/> (standby — watches lease)"]

KCM1-->Lease["Lease object <br/> renewed every 2s"]
KCM1-->Reconciliation["Running <br/> reconciliation loops"]

This is the same leader election pattern available to custom controllers via the leaderelection package.

---

Informer ℹ️

An Informer watches Kubernetes resources and maintains a local cache.

Why we need Informers

A controller needs to know when something changes. But polling the API Server continuously would destroy it at scale.

The solution is Informers — a client-side abstraction over the API Server's List+Watch mechanism.

flowchart LR

    APIServer["API Server"]

    APIServer-->|"Watch stream"| Informer
    Informer-->Cache["Thread-safe <br/> local cache"]
    
    Informer-->EventHandlers["Event handlers <br/> (OnAdd / OnUpdate / OnDelete)"]
    EventHandlers -->WorkQueue

Controllers don't constantly poll the API Server.

They use Informers, which maintain a local cache and receive events efficiently.

Instead of repeatedly calling the API Server, Controllers simply query the cache.

flowchart TD
    API[API Server]

    API --> Watch["Watch API 👀"]

    Watch --> Informer["Informer ℹ️"]

    Informer --> Cache["Cache 💾"]

    Cache --> Controller["Controller 🔄"]

List + Watch

An Informer starts by listing all current objects, then opens a long-lived watch stream for future changes.

1. LIST /api/v1/pods — fetch all pods, populate the local cache
2. WATCH /api/v1/pods?resourceVersion=12345 — receive events from that revision onward

If the watch stream drops (network blip, API Server restart), the Informer reconnects automatically and resumes from its last-seen resourceVersion. No events are lost.

Local Cache (Thread-Safe Store)

After the initial list, the Informer maintains an in-memory store that stays current via watch events.

Controllers always read from this local cache — never directly from the API Server:

// Fast — reads from in-memory cache
pod, err := podLister.Pods("default").Get("my-pod")

// Slow — network round-trip to API Server every time
pod, err := clientset.CoreV1().Pods("default").Get(ctx, "my-pod", metav1.GetOptions{})

The cache is thread-safe. Multiple goroutines can read it simultaneously. This is what makes high-throughput controllers possible — thousands of reconciliations per second without any API Server pressure.

Event Handlers and the Work Queue

Controllers register callbacks on the Informer:

informer.AddEventHandler(cache.ResourceEventHandlerFuncs{
    AddFunc:    func(obj interface{}) { enqueue(obj) },
    UpdateFunc: func(old, new interface{}) { enqueue(new) },
    DeleteFunc: func(obj interface{}) { enqueue(obj) },
})

Critically: handlers enqueue the object key, not the object itself. The actual reconciliation happens later in a worker goroutine — not inside the event handler. This keeps the handler fast and non-blocking.

Informer Flow

sequenceDiagram
    participant API
    participant Informer
    participant Cache
    participant Controller

    Informer->>API: LIST Pods

    API-->>Informer: All Pods

    Informer->>Cache: Store Objects

    Informer->>API: WATCH Pods

    API-->>Informer: Pod Added

    Informer->>Cache: Update Cache

    Informer-->>Controller: OnAdd Event

    Controller->>Cache: Read Current State

    Controller->>API: Create/Delete Resources

Controller vs Informer

Informer	Controller
Watches resources	Reconciles state
Maintains cache	Takes action
Generates events	Creates, updates or deletes resources
Read-only	Performs writes through the API Server

---

Shared Informers

Suppose the Deployment Controller and HPA Controller both need to watch Pods.

Without sharing:

flowchart LR

    APIServer["API Server"]

    APIServer-->DeploymentCtrl["Deployment Controller <br/> (own watch + cache)"]
    APIServer-->HPACtrl["HPA Controller <br/> (own watch + cache)"]
    APIServer-->RSCtrl["ReplicaSet Controller <br/> (own watch + cache)"]

Three watches, three caches, three copies of every Pod in memory.

With a SharedInformerFactory:

flowchart LR

APIServer["API Server"]

    APIServer-->|"one watch"| SharedInformer["Shared Informer <br/> (one cache)"]

    SharedInformer-->DeploymentCtrl
    SharedInformer-->HPACtrl
    SharedInformer-->RSCtrl

One watch stream. One cache. All controllers share the same data.

In kube-controller-manager, every built-in controller is initialized with the same SharedInformerFactory. This is one of the most important scalability decisions in the Kubernetes codebase.

---

Work Queue — Reliability Guarantees

Controllers don't reconcile immediately.

Events go into a work queue.

Each controller has its own Informers and work queues.

flowchart TD
    User[kubectl apply]

    --> API[API Server]

    --> ETCD[(etcd)]

    API --> Informer

    Informer --> Cache

    Cache --> WorkQueue

    WorkQueue --> Controller

    Controller --> API

    API --> ETCD

    Scheduler --> API

    Kubelet --> API

The work queue is not a simple FIFO. It provides four guarantees that make controllers robust:

Guarantee	What it means
Deduplication	If the same key is enqueued 100 times before a worker processes it, it is processed once
Rate limiting	Workers are throttled to avoid overwhelming the API Server
Exponential backoff	On failure, the key is re-enqueued after 5 ms → 10 ms → 20 ms → ... up to 1000 s
Concurrency	Multiple workers drain the queue in parallel (configurable)

Exponential Backoff in Practice

flowchart TD

    Reconcile["Reconcile pod/nginx"]
    Reconcile-->|"API Server error"| Fail["Failure"]
    
    Fail-->Backoff1["Re-enqueue after 5 ms"]
    Backoff1-->|"Still failing"| Backoff2["Re-enqueue after 10 ms"]
    Backoff2-->|"Still failing"| Backoff3["Re-enqueue after 20 ms"]
    Backoff3-->|"Eventually"| Backoff4["Re-enqueue after ~16 min (max)"]

This means a controller that encounters a transient error (API Server blip, etcd timeout) will automatically retry with increasing delays — no manual intervention required.

---

Owner References and Garbage Collection

Kubernetes objects form a parent-child ownership tree tracked via ownerReferences.

# Pod owned by a ReplicaSet
metadata:
  ownerReferences:
  - apiVersion: apps/v1
    kind: ReplicaSet
    name: web-d5b9f7c44
    uid: a1b2c3d4-...
    controller: true
    blockOwnerDeletion: true

The ownership chain for a Deployment looks like:

flowchart LR

    Deployment
    Deployment-->|"owns"| ReplicaSet

    ReplicaSet-->|"owns"| Pod1
    ReplicaSet-->|"owns"| Pod2
    ReplicaSet-->|"owns"| Pod3

When you delete a Deployment, the Garbage Collector controller follows owner references and deletes all owned objects — ReplicaSets and their Pods.

Two deletion modes:

Mode	Behavior
Foreground	Parent waits for all children to be deleted before disappearing
Background (default)	Parent deleted immediately, children cleaned up asynchronously

---

Generation and ObservedGeneration

How does a controller know whether it has processed the current spec, or is still acting on a stale one?

Every Kubernetes object has a generation counter in its spec.

Every time the spec changes, generation increments. The controller writes observedGeneration into the status once it has fully reconciled that spec version.

# After kubectl set image deployment/web ...
status:
  observedGeneration: 4   # controller has reconciled spec v4
  replicas: 3
  updatedReplicas: 3
  readyReplicas: 3

# If a controller crashes mid-rollout:
status:
  observedGeneration: 3   # controller only processed spec v3
  updatedReplicas: 1      # rollout is incomplete

This lets kubectl rollout status and other tooling know whether a rollout is truly complete or still in progress — without guessing.

---

Finalizers

When Kubernetes deletes an object, it normally removes it from etcd immediately.

But some resources require cleanup before deletion — releasing a cloud load balancer, removing DNS records, draining a queue.

Finalizers block deletion until a controller performs the cleanup.

flowchart TD

    Delete["kubectl delete service lb-svc"]
    Delete-->SetTimestamp["API Server sets <br/> deletionTimestamp <br/> (object still exists)"]
    
    SetTimestamp-->Controller["Controller notices <br/> deletionTimestamp"]
    Controller-->Cleanup["Deregisters cloud <br/> load balancer"]
    Cleanup-->RemoveFinalizer["Controller removes <br/> finalizer from object"]
    RemoveFinalizer-->APIServer2["API Server deletes <br/> object from etcd"]

If a controller is deleted before removing its finalizer, the object gets stuck with a deletionTimestamp but never disappears. This is the source of the classic "namespace stuck in Terminating" problem.

---

The Operator Pattern

The same Informer + WorkQueue + Reconcile loop that powers built-in controllers is available to anyone building a custom controller.

This is the Operator pattern — a custom controller that manages a domain-specific resource.

flowchart LR

    CRD["Custom Resource <br/> MyDatabase"]

    CRD-->Informer
    Informer-->OperatorController["Operator Controller"]
    OperatorController-->|"Creates"| StatefulSet & Service & ConfigMap
    OperatorController-->|"Reconciles"| MyDatabase["MyDatabase <br/> status"]

Popular examples: Cert-Manager, Strimzi (Kafka), CloudNativePG, Argo Workflows, Kubeflow Training Operator.

The operator extends Kubernetes' self-healing model to your own application lifecycle — backups, failover, schema migrations, scaling — all expressed as a reconciliation loop.

---

Key Takeaways

Component	Responsibility
Controller	Runs reconciliation: observes current state, compares to desired, takes corrective action
Informer	Wraps List+Watch — populates local cache and fires event handlers
Shared Informer	One watch + one cache shared across all controllers — essential for scalability
Local Cache	In-memory thread-safe store — controllers read from here, never from the API Server directly
Work Queue	Deduplication + rate limiting + exponential backoff + concurrent workers
Owner References	Tracks parent-child ownership; drives garbage collection on deletion
Generation / ObservedGeneration	Lets a controller signal "I have fully processed spec version N"
Finalizers	Block object deletion until a controller completes cleanup
Operator Pattern	Extends the same loop to custom resources — the foundation of the cloud-native ecosystem

The controller pattern is why Kubernetes heals itself. No orchestration script, no runbook, no human in the loop — just a loop that runs forever, comparing what is to what should be, and closing the gap.