Kubelet Internals: The Node Agent That Runs Everything

The API Server, etcd, the scheduler, and controllers all run as processes that talk to each other over HTTP. None of them ever touches a container.

The kubelet is the one component that does. It runs on every node — including control plane nodes — and it is the only piece of Kubernetes that turns a PodSpec into an actual running process.

flowchart LR

    APIServer["API Server"]
    APIServer-->|"watch: Pods <br/> bound to this node"| Kubelet["Kubelet"]
    Kubelet-->|"CRI gRPC"| Runtime["Container Runtime <br/> (containerd / CRI-O)"]
    Runtime-->Container["Container Process"]

Every other control plane component reasons about desired state. The kubelet is where desired state becomes reality.

---

What the Kubelet Owns

Responsibility	Detail
Pod lifecycle	Create, start, restart, and tear down containers per PodSpec
Node registration	Reports the node object into the API Server on startup
Node health	Periodic heartbeats (NodeStatus / Lease) so the cluster knows it's alive
Resource enforcement	Turns requests/limits into cgroup values
Probes	Runs liveness, readiness, and startup probes
Volume mounting	Calls the CSI driver to attach and mount volumes into the pod
Image and container GC	Reclaims disk space by pruning unused images and dead containers
Device allocation	Runs Device Manager and Topology Manager for GPUs and other extended resources

It does all of this without ever contacting etcd directly — everything the kubelet knows about the cluster comes through the API Server's watch mechanism (see Kubernetes API Server Internals).

---

Pod Sources and the Sync Loop

The kubelet doesn't only run pods that come from the API Server. It merges three pod sources into one internal view:

flowchart TD

    APIWatch["API Server watch <br/> (Pods bound to this node)"]
    StaticManifest["Static manifest directory <br/> (/etc/kubernetes/manifests)"]
    HTTPSource["HTTP endpoint <br/> (rarely used)"]

    APIWatch-->PodConfig["PodConfig <br/> (merged pod state)"]
    StaticManifest-->PodConfig
    HTTPSource-->PodConfig

    PodConfig-->SyncLoop["SyncLoop"]
    SyncLoop-->PLEG["PLEG events"]
    SyncLoop-->Probes["Probe results"]
    SyncLoop-->Housekeeping["Periodic housekeeping"]

    SyncLoop-->CRI["CRI calls to runtime"]

PodConfig fans in updates from all three sources and de-duplicates them by pod UID. The SyncLoop is the kubelet's main loop — conceptually the same reconciliation idea as a controller (see Kubernetes Informers & Controllers Explained), but instead of reconciling against etcd-backed objects, it reconciles the desired set of pods against the actual state of the container runtime.

The loop wakes up on four kinds of events:

1. Config change — a pod was added, updated, or removed from PodConfig
2. PLEG update — the runtime reported a container state change
3. Probe result — a liveness/readiness/startup check completed
4. Periodic sync — a housekeeping tick (roughly every second) that catches anything missed

---

The Container Runtime Interface (CRI)

The kubelet never calls Docker or containerd APIs directly. Since Kubernetes 1.5, all runtime communication goes through CRI, a gRPC contract with two services:

Service	Purpose
RuntimeService	Manage pod sandboxes and containers (create, start, stop, remove)
ImageService	Pull, list, and remove images

sequenceDiagram
    participant Kubelet
    participant Runtime as containerd (CRI plugin)
    participant Shim as containerd-shim
    participant OCI as runc

    Kubelet->>Runtime: RunPodSandbox()
    Runtime->>Shim: create sandbox (pause container)
    Kubelet->>Runtime: CreateContainer()
    Runtime->>Shim: create container
    Shim->>OCI: runc create/start
    OCI-->>Shim: container running

This is the same pull-and-unpack path described in Image Internals and the same namespace/cgroup setup described in Container Internals — CRI is simply the interface the kubelet uses to trigger it. Because the contract is standardized, the kubelet doesn't care whether the runtime is containerd, CRI-O, or anything else that implements the two services above.

---

PLEG — the Pod Lifecycle Event Generator

Polling every container's state on every sync loop tick would mean thousands of CRI calls per second on a busy node. PLEG exists to avoid that.

flowchart LR

    PLEG["PLEG"]
    PLEG-->|"ListPodSandbox + ListContainers <br/> every ~1s"| Runtime["Container Runtime"]
    Runtime-->|"container states"| PLEG
    PLEG-->|"diff vs last snapshot"| Events["Lifecycle events <br/> (started, died, removed)"]
    Events-->SyncLoop["SyncLoop"]

PLEG polls the runtime on its own schedule, keeps the previous snapshot in memory, diffs the two, and only emits an event when a container's state actually changed. The sync loop then reacts to the event instead of re-deriving pod state on every tick.

PLEG is not healthy in kubelet logs is one of the most common node-scale failure signatures — it means the relist call to the runtime is taking too long, usually because the runtime itself is overloaded (too many containers, a slow storage driver, or CPU starvation on the node). Once PLEG falls behind, the kubelet stops noticing container deaths promptly, which cascades into stale pod statuses and missed restarts.

---

Node Registration and Heartbeats

On startup, the kubelet is what creates the Node object — the API Server doesn't know a node exists until its kubelet registers it.

flowchart TD

    Kubelet["Kubelet starts"]
    Kubelet-->|"1. Register"| NodeObj["Node object created <br/> (capacity, allocatable, labels)"]
    Kubelet-->|"2. Heartbeat"| Lease["Lease object <br/> renewed every 10s"]
    Kubelet-->|"3. Full status"| NodeStatus["NodeStatus updated <br/> every ~60s (or on change)"]

    Lease-->NodeController["Node Controller"]
    NodeController-->|"no renewal for 40s"| MarkUnknown["Node condition → Unknown"]
    MarkUnknown-->|"5 min grace"| Evict["Pods evicted from node"]

Two separate signals keep a node "alive" from the cluster's point of view:

• Lease renewal (every ~10s) — a lightweight, cheap heartbeat used purely for liveness. This is what the Node Controller watches to decide whether a node has gone unreachable.
• Full NodeStatus update (every ~60s, or immediately on a meaningful change) — capacity, allocatable resources, conditions (Ready, MemoryPressure, DiskPressure, PIDPressure), and images present on the node.

Splitting these apart was a deliberate scalability fix: at a few thousand nodes, having every kubelet write a full NodeStatus object every 10 seconds hammered etcd and the API Server's watch cache (see Kubernetes Performance at Scale). The Lease object is tiny and cheap to write at high frequency; NodeStatus is expensive and only needs to be current, not instantaneous.

---

Static Pods and Mirror Pods

Static pods solve a bootstrapping problem: how does the API Server itself get scheduled, if the scheduler needs the API Server to be running?

The kubelet watches a manifest directory (typically /etc/kubernetes/manifests) directly on disk, completely independent of the API Server:

flowchart LR

    Manifests["/etc/kubernetes/manifests <br/> etcd.yaml, kube-apiserver.yaml, <br/> kube-scheduler.yaml"]
    Manifests-->|"kubelet watches directory"| Kubelet["Kubelet"]
    Kubelet-->|"runs pod directly"| StaticPod["Static Pod <br/> (no scheduler involved)"]
    Kubelet-->|"creates read-only mirror"| APIServer["API Server <br/> (mirror pod object)"]

This is how kubeadm-based clusters bring up etcd, the API Server, the scheduler, and the controller manager — they're all static pods, started by kubelet reading YAML off local disk with no API Server required. Once the API Server is up, the kubelet creates a mirror pod — a read-only representation in the API so kubectl get pods -n kube-system shows them like any other pod, even though kubectl delete on a mirror pod only deletes the mirror, not the actual static pod (you have to remove or edit the manifest file to do that).

---

cgroup Enforcement — Where Requests and Limits Become Real

Kubernetes Resource Allocation covers requests, limits, and QoS classes at the API level. The kubelet is what actually enforces them, by translating a PodSpec into cgroup values at container creation time.

flowchart TD

    PodSpec["PodSpec <br/> requests: 500m/512Mi <br/> limits: 1000m/512Mi"]
    PodSpec-->Kubelet["Kubelet"]
    Kubelet-->|"cpu.shares = 512 <br/> cpu.cfs_quota_us = 100000 <br/> memory.limit_in_bytes = 512Mi"| Cgroup["cgroup <br/> (cgroupfs or systemd driver)"]

Two mechanical details matter here:

• cgroup driver must match between the kubelet and the container runtime (cgroupfs or systemd). A mismatch is a classic cause of nodes that appear to schedule pods fine but silently fail to enforce limits correctly.
• QoS class (Guaranteed, Burstable, BestEffort) isn't a label the kubelet reads from the pod — it computes it from the requests/limits combination and uses it later, during eviction, to decide kill order.

The kubelet also nests cgroups hierarchically: a top-level kubepods cgroup, split into per-QoS-class cgroups, split into per-pod cgroups, split into per-container cgroups — so resource accounting at every level (pod, QoS class, node) falls out of the hierarchy rather than being computed separately.

---

The Eviction Manager

Requests and limits handle steady-state resource sharing. The eviction manager handles the failure mode where a node is actually running out of a resource that can't be throttled — mainly memory and disk.

flowchart LR

    Signals["Node signals <br/> memory.available, nodefs.available, <br/> imagefs.available, pid.available"]
    Signals-->Eviction["Eviction Manager"]
    Eviction-->|"below soft threshold + grace period"| SoftEvict["Soft eviction <br/> (grace period, then kill)"]
    Eviction-->|"below hard threshold"| HardEvict["Hard eviction <br/> (kill immediately)"]
    HardEvict-->Rank["Rank pods by QoS <br/> BestEffort → Burstable → Guaranteed"]

Eviction ranks pods for termination in a fixed order: BestEffort pods go first, then Burstable pods exceeding their requests, and Guaranteed pods only as a last resort. Within a tier, pods using the most of the starved resource relative to their request go first.

This is a different mechanism from the OOM killer: the eviction manager acts proactively at the node level based on periodic signal checks (roughly every 10s), while the kernel OOM killer acts reactively, instantly, when an actual allocation fails — and it uses oom_score_adj (which the kubelet also sets per QoS class) rather than the eviction manager's ranking logic. Both exist because neither alone is fast enough or holistic enough on its own: OOM killer is instant but per-cgroup and blunt; eviction manager is node-aware but has a polling delay.

---

Probe Execution

Liveness, readiness, and startup probes are declared in the PodSpec, but nothing in the API Server ever runs them — the kubelet does, locally, on the node where the pod lives.

Probe type	Failure action
Liveness	Kubelet restarts the container
Readiness	Container stays running, but is pulled out of Service endpoints
Startup	Blocks liveness/readiness checks until it passes — protects slow-starting apps from being killed prematurely

Each probe (exec, httpGet, tcpSocket, or grpc) runs as a worker goroutine per container, on its own ticker, independent of the sync loop. A failing readiness probe never touches the container process — it only changes what the kubelet reports back to the API Server, which is what actually removes the pod from Service endpoints (see Kubernetes Networking: Pods, Services, Ingress, and CNI).

---

Image and Container Garbage Collection

Left alone, a node accumulates old images and exited containers until it runs out of disk. Two independent GC loops handle this:

• Image GC — triggered by imagefs disk usage crossing a threshold (default 85% high / 80% low). Removes unused images, least-recently-used first, until usage drops below the low threshold.
• Container GC — removes dead (exited) containers, keeping a bounded number per pod so kubectl logs on a recently crashed container still works, while preventing unbounded accumulation.

Both run independently of the sync loop, on their own periodic timers.

---

Device Manager and Topology Hints

For extended resources like GPUs, the kubelet doesn't know how to allocate a device on its own — it delegates to whatever Device Plugin has registered for that resource name (see GPU Scheduling in Kubernetes).

flowchart LR

    DevicePlugin["Device Plugin <br/> (ListAndWatch)"]
    DevicePlugin-->|"register nvidia.com/gpu"| DeviceManager["Kubelet Device Manager"]
    DeviceManager-->|"hint: which NUMA node"| TopologyManager["Kubelet Topology Manager"]
    TopologyManager-->|"aligned allocation"| Container["Container <br/> (CPU + GPU + NIC same NUMA node)"]

The Device Manager is the kubelet-side half of the story told from the scheduler's side in GPU Scheduling — the scheduler only ever sees an integer count; the actual Allocate() gRPC call that maps a specific GPU device node into the container happens inside the kubelet, mediated by the Device Manager. The Topology Manager sits one layer above it, collecting NUMA hints from the Device Manager, CPU Manager, and Memory Manager so that a Pod's GPU, CPU cores, and NIC all land on the same NUMA node — the full mechanics of that alignment are covered in Kubernetes Topology Manager: NUMA-Aware GPU Scheduling.

---

Key Takeaways

Component	Responsibility
SyncLoop	Main reconciliation loop; reacts to config changes, PLEG events, probe results, and periodic housekeeping
PodConfig	Merges pod sources — API Server watch, static manifests, HTTP — into one view
CRI	gRPC contract to the container runtime; decouples kubelet from containerd/CRI-O internals
PLEG	Polls runtime for container state changes so the sync loop doesn't have to
Lease / NodeStatus	Split cheap heartbeat (10s) from expensive full status (60s) to protect etcd at scale
Static / Mirror Pods	Bootstraps the control plane itself before the API Server exists
cgroup enforcement	Turns requests/limits into cgroup hierarchy values; drives QoS class computation
Eviction Manager	Proactively kills pods (lowest QoS first) before a node fully exhausts memory/disk
Probes	Run locally by kubelet; liveness restarts, readiness gates Service endpoints
Device Manager	Executes the actual device Allocate() call for GPUs and other extended resources

Every other control plane component operates on objects in etcd. The kubelet is the only one that operates on an actual Linux machine — cgroups, namespaces, disk, and devices — which is why almost every hard-to-debug node issue (PLEG not healthy, OOM kills, GPU allocation failures, stuck evictions) traces back to something happening inside it.

---

Related Posts

• Kubernetes Pod Internals: What a Pod Really Is — the pod creation flow that starts with the kubelet's SyncLoop picking up a new pod
• Kubernetes API Server Internals — the watch mechanism the kubelet uses as its primary pod source
• Kubernetes Informers & Controllers Explained — the same observe-compare-act loop pattern, applied inside the kubelet itself
• Kubernetes Resource Allocation — the requests/limits/QoS model the kubelet's cgroup enforcement and eviction manager implement
• Kubernetes Topology Manager: NUMA-Aware GPU Scheduling — how the kubelet's Device, CPU, and Memory Managers coordinate NUMA-aligned allocation