Kubernetes: Control Loops, Scheduling, and GPUs

Kubernetes ☸️

Most Kubernetes write-ups start with a glossary — Pod, Service, Deployment — and stop there.

The glossary is the easy part. The thing worth internalizing is the one idea the whole system is built on, because every object below is just a different face of it.

Who this guide is for

This article assumes you already know basic container concepts and want to understand how Kubernetes behaves in production.

The focus is on the mechanics behind scheduling, networking, storage, and GPU workloads rather than memorizing Kubernetes objects.

Posts in this section progressively goes deeper and difficult as you progress

The one idea: reconciliation

Kubernetes is not a script runner. It is a set of control loops. You declare a desired state ("I want 3 replicas of this image, fronted by a load balancer"), Kubernetes records that intent, and a controller continuously compares * desired state* against observed state and takes action to close the gap. Pod died? The gap reappears, the controller acts. Node vanished? Same loop, same response.

This is why Kubernetes is declarative rather than imperative, and it's the reason it self-heals without anyone writing recovery scripts. Once this clicks, the object zoo stops being a list to memorize and becomes obvious: a Deployment is a loop that keeps N Pods alive; a Service is a loop that keeps an IP pointed at healthy endpoints; the scheduler is a loop that keeps unscheduled Pods moving onto nodes. Everything is reconciliation.

flowchart LR
    User[Desired State]
    API[API Server]
    ETCD[(etcd)]
    Controller[Controller]
    Cluster[Cluster State]
    User --> API
    API --> ETCD
    Controller --> API
    API --> Controller
    Cluster --> Controller
    Controller --> Cluster

---

Pod 📦

Pods are the smallest deployable units of computing that you can create and manage in Kubernetes.

An abstraction over one or more co-located containers that share a network namespace (same IP, same localhost) and can share volumes.

Most Pods hold a single app container plus the occasional sidecar.

Pods are cattle, not pets: each gets an internal IP, but on respawn it's a new Pod with a new IP.

You almost never create a bare Pod — you let a higher-level controller manage them.

☸️ Kubernetes Cluster
├── 🖥️ Node 1
│   ├── 📦 Pod A
│   │   ├── 🐳 Container 1
│   │   └── 🐳 Container 2
│   └── 📦 Pod B
└── 🖥️ Node 2
└── 📦 Pod C

Deployment 🪂

The control loop for stateless apps.

You give it a Pod template and a replica count; it creates a ReplicaSet, keeps that many Pods alive, and handles rolling updates and rollbacks.

flowchart TB
    Deployment["Deployment 🪂"]
    ReplicaSet["ReplicaSet 💠 "]
    Deployment --> ReplicaSet
    ReplicaSet --> Pod1["Pod 📦"]
    ReplicaSet --> Pod2["Pod 📦"]
    ReplicaSet --> Pod3["Pod 📦"]
    Service["Service 🌐"]
    Service --> Pod1
    Service --> Pod2
    Service --> Pod3

ReplicaSet 💠

Maintain a stable set of replica Pods running at any given time. Usually, you define a Deployment and let that Deployment manage ReplicaSets automatically.

If a Pod or node dies, the loop notices the gap and replaces it. Scaling is one field.

Example Config:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: web
spec:
  replicas: 3
  selector:
    matchLabels: { app: web }
  template:
    metadata:
      labels: { app: web }
    spec:
      containers:
        - name: web
          image: registry.example.com/web:1.4.2
          ports:
            - containerPort: 8080
          resources: # requests == limits -> Guaranteed QoS
            requests: { cpu: "500m", memory: "256Mi" }
            limits: { cpu: "500m", memory: "256Mi" }
          readinessProbe: # gate traffic until the app is actually ready
            httpGet: { path: /healthz, port: 8080 }
            initialDelaySeconds: 5
          livenessProbe: # restart if it wedges
            httpGet: { path: /healthz, port: 8080 }
            periodSeconds: 10

Two details that matter: setting requests == limits puts this Pod in the Guaranteed QoS class (evicted last under pressure), and the readiness probe is what keeps the fronting Service from sending traffic to a Pod that's still warming up a common cause of deploy-time 502s when it's missing.

StatefulSet 💾

Runs a group of Pods, and maintains a sticky identity for each of those Pods.

This is useful for managing applications that need persistent storage or a stable, unique network identity.

For workloads that need stable identity and storage

• databases, message brokers,
• anything where replica-0 is meaningfully different from replica-1.

Unlike a Deployment, it gives each Pod a sticky network name and its own persistent volume that survives rescheduling, and it manages Pods in order.

The honest caveat: a StatefulSet handles the orchestration of stateful Pods, but it does not solve data consistency or replication for you — that's the database's job.

People reach for StatefulSets expecting magic clustering; they get stable plumbing, nothing more.

---

Kubernetes Architecture

flowchart TD
subgraph Master Node["Master Node 1 🖥️"]
    subgraph ControlPlane["Control Plane 🎛<br/><br/>"]
        API["API Server ⚡"]
        Scheduler["Scheduler 🕣"]
        Controller["Controller Manager 🔄"]
        ETCD[("etcd 🛢"️)]
    end
end

    subgraph Worker1["Worker Node 1 🖥️"]
        Kubelet1["Kubelet 👨🏻‍🔧🔧"]
        Runtime1["Container Runtime 📟"]
        Pod1["Pods 💠 "]
    end

    subgraph Worker2["Worker Node 2 🖥"️]
        Kubelet2["Kubelet 👨🏻‍🔧"]
        Runtime2["Container Runtime 📟"]
        Pod2["Pods 💠"]
    end

    API <--> Scheduler
    API <--> Controller
    API <--> ETCD
    API <--> Kubelet1
    API <--> Kubelet2
    Kubelet1 --> Runtime1
    Runtime1 --> Pod1
    Kubelet2 --> Runtime2
    Runtime2 --> Pod2

🖥 NODE

A Node is a physical or virtual machine that runs part of the cluster.

Nodes split into the

1. Master Node/ Control plane
2. workers Node/ Data Plane

1. CONTROL PLANE 🎛️

• The Control Plane is responsible for managing the cluster.

🖥 Master Node

Node responsible for managing the cluster

Single-Master vs. High-Availability Setups

1 Master Node:

• Used for local learning, testing, and dev environments.
• If this node fails, you cannot manage the cluster or deploy changes.

3 Master Nodes

• The standard production choice.Tolerates the failure of 1 master node without losing cluster control.

5 Master Nodes

• Used for larger or more critical environments.
• Tolerates the simultaneous failure of up to 2 master nodes.Why Odd Numbers?

Master Node component

Four core components, each a piece of the reconciliation machinery:

1.1 API Server ⚡

The only component that talks to etcd, and the single front door for everything else (kubectl, controllers, kubelets).

• It's a stateless REST layer doing auth, admission control, and validation.
• Everything in the cluster is a Write to, or a watch on, the API server.
• API Servers are stateless and generally easy to run in a self-healing instance group or scaleset.

Bottlenecks and scaling

1. API Servers can take up a fair bit of memory, and that tends to scale linearly with the number of nodes in the cluster.

• Cluster with 7,500 nodes can take 70GB of heap being used per API Server
• Best to run 3 or 5 API Servers in a dedicated Nods outside kube to avoid single point of failure.

2. Autoscaling too much at once.

• There are many requests generated when a new node joins a cluster, and adding hundreds of nodes at once can overload API server capacity.
• Smoothing this out, even just by a few seconds, has helped avoid outages.

kubectl

Command line tool for communicating with a Kubernetes cluster's control plane, using the Kubernetes API.

---

1.2. Scheduler 🕣

Watches for newly created Pods with no node assigned and binds them to a node.

• It does not start containers; it only decides where.

request

• kubelet reserves at least the request amount of that system resource specifically for that container to use.

limit

• running container is not allowed to use more of that resource than the limit.

   apiVersion: v1
   kind: Pod
   metadata:
     name: pod-resources-demo
     namespace: pod-resources-example

   spec:
     resources:
       requests:
         cpu: "2"
         memory: "2Gi"
       limits:
         cpu: "4"
         memory: "4Gi"

     containers:
       - name: pod-resources-demo-ctr-1
         image: nginx
         resources:
           requests:
             cpu: "1"
             memory: "1Gi"
           limits:
             cpu: "2"
             memory: "2Gi"

       - name: pod-resources-demo-ctr-2
         image: fedora
         command:
           - sleep
           - infinity



             Pod Limit = 4 CPU
       ┌─────────────────────────────┐
       │                             │
       │ nginx      <= 2 CPU         │
       │                             │
       │ fedora     uses remainder   │
       │                             │
       └─────────────────────────────┘

Scheduler allocation scenario

How Pod-level and Container-level resources interact in Kubernetes (v1.34+ with the Pod-Level Resources feature).

Scenario	Pod Resources	Container Resources	Scheduling	Runtime Behavior	QoS Class
1. Only Container Requests/Limits (Traditional)	❌ None	✅ All containers specify requests/limits	Sum of all container requests	Each container is individually limited	Guaranteed / Burstable / BestEffort
2. Only Pod Requests/Limits	✅ Yes	❌ None	Uses Pod request	Containers share the Pod resource pool	Based on Pod resources
3. Pod + Some Containers	✅ Yes	✅ Only selected containers	Scheduler uses Pod request	Containers with limits are capped; others share remaining Pod budget	Based on Pod + container settings
4. Pod + All Containers	✅ Yes	✅ Every container	Scheduler uses Pod request	Pod cannot exceed Pod limit; containers cannot exceed their own limits	Guaranteed if requests=limits everywhere
5. No Resources Anywhere	❌ None	❌ None	Scheduler assumes zero request	Containers compete freely until node pressure	BestEffort
6. Requests Only	Pod or Containers	Requests only	Scheduler reserves requested resources	Containers may burst if node has spare capacity	Burstable
7. Limits Only	Pod or Containers	Limits only	No reservation during scheduling	CPU throttling, Memory OOM at limits	Burstable
8. Requests = Limits	Pod or Containers	Requests equal limits	Exact reservation	No bursting beyond limit	Guaranteed

OOM condition

Assume: Pod Limit = 4 GiB

• nginx Container A = 2 GiB Limit
• fedora Container B = No Limit

Container A	Container B	Total	Result
1 Gi	1 Gi	2 Gi	✅ Allowed
2 Gi	1 Gi	3 Gi	✅ Allowed
2 Gi	2 Gi	4 Gi	✅ Allowed
2 Gi	3 Gi	5 Gi	❌ Pod exceeds memory limit; OOM kill likely
3 Gi	1 Gi	4 Gi	❌ Container A exceeds its own limit and is likely OOM killed

Request exceed available resource

If a node has 4 CPUs and a Pod requests 8 CPUs,

Node CPU	Pod Request	Pod Limit	Scheduled?	Reason
4	8	8	❌ No	Request exceeds node capacity
4	4	8	✅ Yes	Request fits exactly
4	2	8	✅ Yes	Scheduler considers only the request
4 (1 CPU free)	2	4	❌ No	Not enough allocatable CPU remaining
4	0	8	✅ Yes	No reservation required, but the Pod competes for CPU at runtime

Scheduling internals

Pod must fit entirely on a single node based on its resource requests.

• Kubernetes never splits one Pod across multiple nodes.

To find the node where a pod can be allocated the scheduler runs two phases per Pod:

1. Filter (predicates) ⛔

Eliminate nodes that can't run the Pod

• Insufficient resources
• Failed taint match
• Unsatisfied node selector.

2. Score (priorities) 🥇

Rank the surviving nodes and bind to the winner

• Spread
• Least-loaded
• Affinity

The levers you actually use to control placement:

Taints & Tolerations

a node repels Pods unless they explicitly tolerate its taint.

Taints: "Don't schedule Pods here."

# Don't schedule Pods here unless they tolerate this taint.
 kubectl taint nodes gpu-node \ 
         gpu=true:NoSchedule
 
# Avoid if possible. If no better node exists schedule it
 kubectl taint nodes gpu-node \
         gpu=true:PreferNoSchedule
 
# Strongest rule: kill the pod and evict unless they tolerate it
 kubectl taint nodes gpu-node \
         gpu=true:NoExecute

• Key = gpu

• Value = true

• Effect = NoSchedule

• This is how you keep general workloads off specialized nodes (the standard pattern for GPU nodes).

Tolerations: "This Pod is allowed to ignore that restriction."

apiVersion: v1
kind: Pod
metadata:
  name: gpu-app

spec:
  tolerations:
    - key: "gpu"
      operator: "Equal"
      value: "true"
      effect: "NoSchedule" # ✅ Scheduler may place Pod there

  containers:
    - name: app
      image: nginx

flowchart TD
    A[Scheduler] --> B{Is node <br/> tainted?}

    B -- No --> C[Schedule]
    B -- Yes --> D{Pod has <br/>matching toleration?}

    D -- No --> E[Cannot schedule]
    D -- Yes --> F[Schedule allowed]

Node affinity

A property of Pods that attracts them to a set of nodes

• pull Pods toward nodes with certain labels.

apiVersion: v1
kind: Pod
metadata:
  name: example-vector-add
spec:
  restartPolicy: OnFailure
  # You can use Kubernetes node affinity to schedule this Pod onto a node
  # that provides the kind of GPU that its container needs in order to work
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
          - matchExpressions:
              - key: "gpu.gpu-vendor.example/installed-memory"
                operator: Gt # (greater than)
                values: [ "40535" ]
              - key: "feature.node.kubernetes.io/pci-10.present" # NFD Feature label
                values: [ "true" ] # (optional) only schedule on nodes with PCI device 10
  containers:
    - name: example-vector-add
      image: "registry.example/example-vector-add:v42"
      resources:
        limits:
          gpu-vendor.example/example-gpu: 1 # requesting 1 GPU

Pod affinity / anti-affinity

Co-locate or separate Pods relative to each other

"I want to run Backend where Redis already exists."

    Node A
    ├── Redis Pod
    └── Metrics Pod
    
    Node B
    ├── Frontend Pod
    └── Logging Pod
    
    Schedule Backend → Node A

Config

    
    apiVersion: v1
    kind: Pod
    metadata:
      name: backend
    
    spec:
      affinity:
        podAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
          - labelSelector:
              matchLabels:
                app: redis
            topologyKey: kubernetes.io/hostname
    
      containers:
      - name: backend
        image: nginx

Node vs Pod affinity

Feature	Node Affinity	Pod Affinity
Looks at	Node labels	Existing Pod labels
Example	gpu=true	app=redis
Used for	Hardware or node characteristics	Application placement
Scope	Node properties	Running workloads

Topology spread constraints

finer control over even distribution across failure domains.

Control how Pods are spread across your cluster among failure-domains such as

Topology Key	Spread Across
kubernetes.io/hostname	Nodes
topology.kubernetes.io/zone	Availability Zones
topology.kubernetes.io/region	Regions
Custom label	Any logical grouping

maxSkew

The maximum allowed difference in the number of matching Pods between topology domains.

Example:

• Node A :: 2 Pod = App-1 , App-2
• Node B :: 1 Pod = App-3
• Node C :: 1 Pod = App-4

Deploy another Pod with maxSkew = 1

Node	Before	New Count	Max Skew	Result
A	2	3	2	❌ Not allowed (maxSkew=1)
B	1	2	1	✅ Allowed
C	1	2	1	✅ Allowed

Example Config

    apiVersion: apps/v1
    kind: Deployment
    metadata:
      name: web
    
    spec:
      replicas: 6
    
      selector:
        matchLabels:
          app: web
    
      template:
        metadata:
          labels:
            app: web
    
        spec:
          topologySpreadConstraints:
          - maxSkew: 1
            topologyKey: kubernetes.io/hostname
            whenUnsatisfiable: DoNotSchedule
            labelSelector:
              matchLabels:
                app: web
    
          containers:
          - name: nginx
            image: nginx

Extended resources

beyond CPU and memory, nodes can advertise custom countable resources.

• This is the hook the entire GPU story hangs on.

GPUs are only supposed to be specified in the limits section, which means:

• You can specify GPU limits without specifying requests, because Kubernetes will use the limit as the request value by default.
• You can specify GPU in both limits and requests but these two values must be equal.
• You cannot specify GPU requests without specifying limits.

Resource model & QoS

Every container can declare requests (what the scheduler reserves) and limits (the hard ceiling).

This pair drives two things people learn the hard way:

1. Scheduling

the scheduler places Pods based on requests, not actual usage.

• Under-request and you overcommit a node into instability;
• over-request and you waste capacity.

3. QoS class (Quolity of service)

Pods are bucketed into

• Guaranteed (requests == limits)
• Burstable: pod has some requests or limits, but they are not equal, or only one is specified.
• BestEffort: no request, no limit specified

kubectl get pod mypod -o yaml

kubectl describe pod mypod

Category decision

flowchart TD
A[Pod Created] --> B{Requests and Limits <br/>set for all containers?}

    B -- No --> C{Any requests or <br/>limits defined?}

    B -- Yes --> D{Requests == Limits <br/>for every container?}

    D -- Yes --> E[Guaranteed]
    D -- No --> F[Burstable]

    C -- Yes --> F
    C -- No --> G[BestEffort]

QoS Class	Requests	Limits	Requests = Limits	Eviction Priority	Example
Guaranteed	Required	Required	Yes	Last	Triton Inference Server
Burstable	Some	Optional	No	Middle	Prometheus Exporter
BestEffort	None	None	N/A	First	Debug Shell

Evacuation Order

Under memory pressure, the kubelet evicts BestEffort first and Guaranteed last.

A Pod exceeding its memory limit is OOMKilled — one of the most common "why did my Pod restart" answers.

flowchart LR
    A[Memory Pressure] --> B[BestEffort]
    B --> C[Burstable]
    C --> D[Guaranteed]

---

1.3. Controller Manager 🔄

Controllers are control loops that watch the state of your cluster, then make or request changes where needed.

Its core responsibility is to detect differences and take corrective action until the cluster reaches the desired state.

• It does not schedule Pods. That is the Scheduler's job.
• It does not run containers. That is the Kubelet's job.

It consists of multiple controllers, each responsible for a different Kubernetes resource.

• Logically, each controller is a separate process, but to reduce complexity, they are all compiled into a single binary and run in a single process.

Each controller tries to move the current cluster state closer to the desired state.

• Runs the built-in control loops Deployment, ReplicaSet, Node, Job, and dozens more), each reconciling its slice of desired vs observed state.

There are many different types of controllers. Some examples of them are:

Controller	Responsibility
Deployment Controller	Creates and updates ReplicaSets
ReplicaSet Controller	Maintains the desired number of Pods
StatefulSet Controller	Manages stateful applications
DaemonSet Controller	Runs one Pod per node
Job Controller	Executes batch jobs
CronJob Controller	Schedules Jobs periodically
Node Controller	Detects failed nodes
Namespace Controller	Deletes all resources in a namespace during namespace deletion
ServiceAccount Controller	Creates default ServiceAccounts
EndpointSlice Controller	Updates Service endpoints
PersistentVolume Controller	Binds PVs and PVCs

---

4. etcd 🛢️

The cluster's source of truth.

A distributed key-value store using the Raft consensus protocol with odd-numbered quorum

• 3 nodes
• 5 nodes
• 7 nodes

Backing up etcd is one of the most important operational tasks.

# Backup
etcdctl snapshot save backup.db

# Recovery
etcdctl snapshot restore backup.db

Internally it look like a long json

    Key
    ------------------------------------------------------------
    /registry/pods/default/nginx
    
    Value
    ------------------------------------------------------------
    {
      "kind":"Pod",
      "metadata":{
          "name":"nginx"
      },
      ...
    }

Raft Consensus in etcd

Allows multiple etcd servers to behave like a single, consistent database.

• Leader: Raft guarantees there is exactly one leader that accepts writes.
• Followers: Rest etcd are read only replicas (they participate in consensus but do not accept client writes directly).

flowchart TD

    L[(Leader <br/>Read/Write Node)]

    F1[(Follower 1<br/> Read Only)]
    F2[(Follower 2<br/>  Read Only)]

    APIServer <--read/write--> L
    L --write--> F1
    L --write--> F2

Database is only commited if both Follower node updates its database & majority ACK.

$Majority = floor(N/2) + 1$

Cluster Size	Majority Needed	Failures Tolerated
1	1	0
3	2	1
5	3	2
7	4	3

FLow of controls before commit a change in etcd DB

sequenceDiagram
participant User
participant API
participant Leader
participant F1
participant F2

    User->>API: Create Pod
    API->>Leader: Write Request

    Leader->>F1: Replicate Log Entry
    Leader->>F2: Replicate Log Entry

    F1-->>Leader: ACK
    F2-->>Leader: ACK

    Leader->>API: Commit Success

Limitation

When a large cluster mysteriously gets slow, etcd disk latency is the first place to look.

Bottlenecks and scaling

etcd is the thing that actually caps your cluster's scale.

1. It's sensitive to disk fsync latency

• put it on fast local SSD (200us), never network storage (2ms),

2. etcd’s hard storage limit

• default DB size quota is 2 GB (8 GB is the practical ceiling),
• a write-heavy cluster needs periodic compaction and defrag or it wedges with a NOSPACE alarm.

3. Kubernetes Events in a separate etcd cluster

• The default etcd cluster is shared between the API server and the event store.
• In a large cluster, events can overwhelm the main etcd cluster and cause API server timeouts.
• The fix is to run a separate etcd cluster for events, which is supported in Kubernetes 1.24+.

---

2. 📟 WORKER NODE / NODE

A node is a VM or a physical computer that serves as a worker machine in a Kubernetes cluster.

• Workers do the actual work, so they're sized for it. Each can run many Pods.
• Two Nodes cannot have the same name at the same time.

Each worker runs three processes:

2.1. Kubelet 👨🏻‍🔧

An agent that runs on each node in the cluster.

It makes sure that containers are running in a Pod.

It watches the API server for Pods bound to its node and drives the runtime to make them real, then reports status back.

sequenceDiagram
participant User
participant API
participant Scheduler
participant Kubelet
participant Runtime

    User->>API: Create Pod
    API->>Scheduler: Pending Pod

    Scheduler->>API: Bind Pod to Node-1

    Kubelet->>API: Watch Assigned Pods

    API-->>Kubelet: Pod Spec

    Kubelet->>Runtime: Create Pod

    Runtime->>Runtime: Pull Image
    Runtime->>Runtime: Start Container

    Kubelet->>API: Running

HeartBeat

Kubelet periodically reports

sequenceDiagram
participant Kubelet
participant API

    loop Every few seconds
        Kubelet->>API: Node Heartbeat
    end

2.2 Container Runtime Interface (CRI) 📟

Standard gRPC API that allows the Kubelet to communicate with any compatible container runtime.

• plugin interface which enables the kubelet to use a wide variety of container runtimes, without having a need to recompile the cluster components.
• Example: containerd or CRI-O, spoken to over the Container Runtime Interface (CRI).

Note: Docker-as-a-runtime was removed in Kubernetes v1.24 (the dockershim deprecation). Docker-built images still run everywhere — they're OCI-compliant — but Docker the daemon is no longer the runtime.

flowchart TD

    Kubelet --> CRI["CRI (gRPC API)"]
    CRI --> containerd
    CRI --> CRI-O

CRI have 2 services

1. Runtime Service

Manages pod sandboxes

service RuntimeService {

    // Sandbox operations.

    rpc RunPodSandbox(RunPodSandboxRequest) returns (RunPodSandboxResponse) {}  
    rpc StopPodSandbox(StopPodSandboxRequest) returns (StopPodSandboxResponse) {}  
    rpc RemovePodSandbox(RemovePodSandboxRequest) returns (RemovePodSandboxResponse) {}  
    rpc PodSandboxStatus(PodSandboxStatusRequest) returns (PodSandboxStatusResponse) {}  
    rpc ListPodSandbox(ListPodSandboxRequest) returns (ListPodSandboxResponse) {}  

    // Container operations.  
    rpc CreateContainer(CreateContainerRequest) returns (CreateContainerResponse) {}  
    rpc StartContainer(StartContainerRequest) returns (StartContainerResponse) {}  
    rpc StopContainer(StopContainerRequest) returns (StopContainerResponse) {}  
    rpc RemoveContainer(RemoveContainerRequest) returns (RemoveContainerResponse) {}  
    rpc ListContainers(ListContainersRequest) returns (ListContainersResponse) {}  
    rpc ContainerStatus(ContainerStatusRequest) returns (ContainerStatusResponse) {}

    ...  
}

2. Image Services

Pulls container images from a registry.

• PullImage()
• ListImages()
• ImageStatus()
• RemoveImage()

It is responsible for managing the execution and lifecycle of containers within the Kubernetes environment.

sequenceDiagram
    participant API as API Server
    participant K as Kubelet
    participant CRI as containerd (CRI)
    participant OCI as runc
    participant Kernel

    API->>K: Pod assigned to node

    K->>CRI: RunPodSandbox()

    CRI->>OCI: Create Sandbox

    K->>CRI: PullImage()

    K->>CRI: CreateContainer()

    K->>CRI: StartContainer()

    CRI->>OCI: Run OCI Container

    OCI->>Kernel: Create namespaces & cgroups

---

Pod Sandbox

It is the shared execution environment that Kubernetes creates before any application containers start.

A Pod Sandbox is not your application it's the environment

Suppose we have 2 container

    spec:
        containers:
        - name: nginx      # Container 1
          image: nginx
        - name: log-agent  # Container 1
          image: fluent-bit

They share

• Same IP address
• Same localhost
• Same network interfaces
• Same IPC namespace (optional)
• Same mounted volumes

Pod Sandbox give them an env:

Pod Sandbox
│
├── Shared Network Namespace
├── Shared IPC Namespace
├── Shared UTS Namespace (Hostname)
└── Shared Volumes

Pause Container

Keeps Namespace occupied by staying alive

Why Pause Container

Every Linux namespace must have at least one running process.

If no process exists, the namespace disappears.

Therefore, Kubernetes creates a tiny container called pause

The pause container owns

• Network namespace
• IPC namespace
• Hostname

# Pod View

containers:
    - nginx
    - fluent-bit

# Runtime View

   Pod
    ├── pause
    ├── nginx
    └── fluent-bit

Network View

Application containers do not own networking. The sandbox does.

flowchart TD
    PS[Pod Sandbox]

    PS --> NIC[eth0]
    PS --> IP[IP 10.244.1.25]
    PS --> HOST[localhost]

    NIC --> Container[Container<br>nginx]
    NIC --> SideCarAgent[SideCar<br/>log-agent]
    NIC --> SideCar[metrics]

Pod Creation flow

flowchart TD
A[Kubelet receives Pod] --> B[RunPodSandbox]
B --> C[Create Pause Container]
C --> D[Create Network Namespace]
D --> E[Assign Pod IP]
E --> F[Mount Shared Volumes]
F --> G[Create Application Containers]
G --> H[Join Sandbox]

Who Own the Storage Mount?

The Pod sandbox owns the Pod-level mount namespace, but the Kubelet performs the actual volume mounting.

Containers then see those mounts because they join the Pod's mount namespace.

1. Kubelet prepares the volume. Volume exist on the node, not inside any container.
2. Kubelet mounts the volume. This happens before your containers start.
3. The pause container creates the Pod's namespaces.

Pod Sandbox
    Network Namespace
    IPC Namespace
    Mount Namespace

4. Containers join the mount namespace

    flowchart LR
    A[Kubelet] --> B[Create Volume]
    B --> C[Mount on Host]
    C --> D[Pod Sandbox]
    D --> E[Container 1]
    D --> F[Container 2]

The Kubelet is responsible for creating and mounting volumes on the node, often with help from a CSI driver for persistent storage.

Those mounts are then made available inside the Pod's shared mount namespace, which is established by the Pod sandbox (pause container).

All containers in the Pod join that namespace, allowing them to access the same mounted storage, even if they mount it at different paths.

Host OS
────────────────────────────────────────────

Kubelet
   │
   ├── Creates emptyDir
   │
   ├── Mounts volume
   │
   ▼
/var/lib/kubelet/pods/<uid>/volumes/

────────────────────────────────────────────

Pod Sandbox (Pause Container)

Mount Namespace
      │
      ├── /data
      ├── /cache
      └── /config
      │
      │
      ├─────────────┐
      ▼             ▼

Container A     Container B

---

Storage

Pods are ephemeral, so anything written to a container's filesystem dies with it. Volumes outlive the container; * PersistentVolumes (PV)* outlive the Pod. The model:

Persistent Volume Claim (PVC) 🧾

A Pod's request for storage ("10Gi, fast").

Persistent Volume (PV) 🛢️

The actual backing storage that satisfies a claim.

Storage Class 🗃️

Defines how PVs get provisioned dynamically (e.g. an AWS gp3 EBS class), so you don't pre-create volumes by hand.

Kubernetes orchestrates the binding; it does not itself guarantee durability — that's the storage backend's job.

  apiVersion: v1
  kind: Service
  metadata:
    name: nginx
    labels:
      app: nginx
  spec:
    ports:
      - port: 80
        name: web
    clusterIP: None
    selector:
      app: nginx
  ---
  apiVersion: apps/v1
  kind: StatefulSet
  metadata:
    name: web
  spec:
    selector:
      matchLabels:
        app: nginx # has to match .spec.template.metadata.labels
    serviceName: "nginx"
    replicas: 3 # by default is 1
    minReadySeconds: 10 # by default is 0
    template:
      metadata:
        labels:
          app: nginx # has to match .spec.selector.matchLabels
      spec:
        terminationGracePeriodSeconds: 10
        containers:
          - name: nginx
            image: registry.k8s.io/nginx-slim:0.24
            ports:
              - containerPort: 80
                name: web
            volumeMounts:
              - name: www
                mountPath: /usr/share/nginx/html
    volumeClaimTemplates:
      - metadata:
          name: www
        spec:
          accessModes: [ "ReadWriteOnce" ]
          storageClassName: "my-storage-class" # 🗃️ Storage Class
          resources:
            requests:
              storage: 1Gi

Open Container Initiative (OCI) runtime

is a low-level software component that directly interfaces with the host operating system kernel to create, run, and manage containers.

Actually performs

• Create namespaces
• Configure cgroups
• Mount OverlayFS
• Start Linux process

It interacts directly with the Linux kernel.

flowchart TD
A[Kubelet]

    A -->|CRI| B[containerd]

    B -->|OCI| C[runc]

    C --> D[Linux Kernel]

---

3. Kube Proxy 🌐

kube-proxy is the Kubernetes component that enables Service networking.

Why We need Kube Proxy

Pods are ephemeral ie. their IP does not survive crash & restart

Pods Ip changes when they are recreated so Applications cannot rely on Pod IPs.

Kubernetes mandates a flat network:

• Every Pod gets a unique, routable IP
• Any Pod can reach any other Pod without NAT.

Kubernetes itself doesn't implement this — it delegates to a CNI plugin (Calico, Cilium, AWS VPC CNI, etc.), which is why your network behavior depends heavily on which CNI you picked.

Choosing a CNI

Scenario	Recommended Plugin
Learning Kubernetes	Flannel
Small production cluster	Calico
Enterprise Kubernetes	Calico
High-performance networking	Cilium
AI/ML GPU clusters	Cilium
Amazon EKS	AWS VPC CNI
Azure AKS	Azure CNI
Google GKE	Dataplane V2

CNI comes before Kube Proxy & answers:

"How does this Pod get an IP address?"

It performs tasks like:

• Create network namespace
• Create veth pair
• Assign Pod IP
• Configure routing

CNI (Container Network Interface)

Give every Pod an IP address and make Pods able to communicate with each other.

• CRI manages containers.
• CNI manages networking.

flowchart LR
    A[Kubelet]
    A -->|CRI| B[containerd]
    B -->|OCI| C[runc]
    C --> D[Linux Kernel]

    A -->|CNI| E[CNI Plugin]
    E --> D

The network is configured before application containers start.

sequenceDiagram
participant API
participant Scheduler
participant Kubelet
participant Runtime
participant CNI

    API->>Scheduler: Schedule Pod

    Scheduler->>Kubelet: Assign Pod

    Kubelet->>Runtime: Create Pod Sandbox

    Runtime->>CNI: ADD Network

    CNI->>Runtime: Pod IP

    Runtime->>Kubelet: Sandbox Ready

    Kubelet->>Runtime: Start Containers

veth pair

A virtual cable connecting Pod network namespace to host

flowchart LR
    subgraph Pod1
        ETH0[eth0<br/>10.244.1.2]
    end

    subgraph Pod2
        ETH1[eth0<br/>10.244.1.3]
    end

    subgraph Host
        VETH0[veth0]
        VETH1[veth1]
        BRIDGE[cni0 Bridge]
    end

    ETH0 --- VP0[veth-peer0]
    VP0 --- VETH0

    ETH1 --- VP1[veth-peer1]
    VP1 --- VETH1

    VETH0 --- BRIDGE
    VETH1 --- BRIDGE

Once Pods already have IPs, kube-proxy answers:

"How does traffic sent to a Service reach one of these Pod IPs?"

Kube Proxy 📡

kube-proxy runs on every node as a DaemonSet.

sequenceDiagram
participant Scheduler
participant Kubelet
participant Runtime
participant CNI
participant kubeproxy as kube-proxy

    Scheduler->>Kubelet: Assign Pod
    Kubelet->>Runtime: RunPodSandbox()
    Runtime->>CNI: ADD network
    CNI-->>Runtime: Pod IP assigned
    Runtime->>Runtime: Start Containers

    Note over kubeproxy: Already watching Services
    kubeproxy->>kubeproxy: Update routing rules if needed

Each kube-proxy watches Services and EndpointSlices from the API Server.

flowchart LR
subgraph Node A
KP1[kube-proxy]
P1[Pods]
end

    subgraph Node B
        KP2[kube-proxy]
        P2[Pods]
    end

    subgraph Node C
        KP3[kube-proxy]
        P3[Pods]
    end

    API[API Server]

    API --> KP1
    API --> KP2
    API --> KP3

How Client talk to a POD

sequenceDiagram
participant Client
participant kubeproxy as kube-proxy
participant Pod

    Client->>kubeproxy: Service IP
    kubeproxy->>Pod: Select Backend Pod
    Pod-->>Client: Response

Kube-proxy modes (a real-world gotcha)

How a Service IP actually routes to a Pod depends on kube-proxy's mode:

1. iptables (default)

Fine for small clusters, but rule evaluation is O(n) in the number of Services, so it degrades badly at thousands of Services.

2. IPVS(IP Virtual Server)

Hash-based, scales far better for large Service counts.

3. nftables

Newer, addresses the iptables scaling problem natively.

4. eBPF (Cilium)

Replaces kube-proxy entirely and routes in-kernel increasingly the choice for large or performance-sensitive clusters.

Kube-proxy modes Comparison

Feature	iptables	IPVS	nftables	eBPF (Cilium)
Packet forwarding	Kernel iptables	Linux IPVS	Linux nftables	Kernel eBPF
kube-proxy required	✅	✅	✅	❌ (replacement mode)
Scalability	Good	Very Good	Very Good	Excellent
Service update speed	Slower with many rules	Fast	Fast	Very Fast
Load balancing	Basic	Multiple algorithms	Basic	Advanced
Kernel dependency	Standard	IPVS modules	Modern kernel	Modern kernel + eBPF
Best for	Small/medium clusters	Large clusters	Modern Linux	Very large, high-performance clusters

CNI vs Kube Proxy

CNI	kube-proxy
Pod networking	Service networking
Gives Pods IPs	Routes Service traffic
Creates veth pairs	Creates iptables/IPVS/eBPF rules
Pod-to-Pod communication	Service-to-Pod communication
Works during Pod creation	Watches Services continuously

Service 🌐

Provide a stable IP address and DNS name, while automatically routing traffic to healthy Pods.

A Service is a stable virtual IP (and DNS name) that load-balances across a healthy set of Pods, selected by label.

The Service's identity is independent of any Pod's lifecycle — that's the whole point.

• ClusterIP (default): reachable only inside the cluster — e.g. a database other Pods talk to.
• NodePort / LoadBalancer: exposed externally — e.g. an API hit from a browser.

flowchart LR
    User[Browser]
    Ingress["Ingress / Gateway 🚪"]
    Service["Service 🌐"]
    Pod1["Pod 📦"]
    Pod2["Pod 📦"]
    Pod3["Pod 📦"]
    User --> Ingress
    Ingress --> Service
    Service --> Pod1
    Service --> Pod2
    Service --> Pod3

Ingress / Gateway API 🚪

Ingress maps external HTTP(S) URLs to Services — host/path routing, TLS termination — so clients hit a meaningful hostname instead of a raw IP and port. The Gateway API is the newer, more expressive successor that's gradually replacing Ingress for serious setups.

kube-proxy	Ingress Controller
Layer 4 (TCP/UDP)	Layer 7 (HTTP/HTTPS)
Routes Service traffic	Routes HTTP requests
Uses iptables/IPVS/nftables	NGINX, Envoy, HAProxy, etc.
Built into Kubernetes	Optional add-on

---

Configuration & secrets

1. ConfigMap 📜

external configuration injected as env vars or files, so the same image runs across environments without a rebuild.

2. Secret 🔐

Same mechanism for credentials and certs, stored base64-encoded. Critical caveat that trips people up: base64 is encoding, not encryption. A raw Secret is plaintext to anyone with API or etcd access. For real protection you enable encryption at rest for etcd and/or use an external manager (Vault, AWS/GCP secret stores via the Secrets Store CSI driver).

---

DaemonSet 👺

Defines Pods that provide node-local facilities. These might be fundamental to the operation of your cluster, such as a networking helper tool, or be part of an add-on.

Runs exactly one Pod per node (or per matching node). This is how node-level agents ship: log collectors, CNI plugins, monitoring exporters — and, relevant later, the NVIDIA device plugin and DCGM exporter, which must run on every GPU node.

Job 🏋️‍♀️

Run-to-completion workloads rather than long-running services.

• A Job runs a Pod until it succeeds; CronJob schedules them.

  apiVersion: batch/v1
  kind: Job
  metadata:
    name: my-job
  spec:
    template:
      spec:
        containers:
          - name: my-job
            image: my-image
        restartPolicy: Never
    backoffLimit: 4

CronJob 📅

A CronJob starts one-time Jobs on a repeating schedule.

Batch ML training fits here — though multi-Pod distributed training needs scheduling help the default Job controller doesn't provide (see GPUs, below).

  apiVersion: batch/v1
  kind: CronJob
  metadata:
    name: my-cronjob
  spec:
    schedule: "0 0 * * *"
    jobTemplate:
      spec:
        template:
          spec:
            containers:
              - name: my-job
                image: my-image
            restartPolicy: OnFailure

---

GPUs on Kubernetes

Kubernetes has no native concept of a GPU. Out of the box it understands CPU and memory, full stop. GPUs are made schedulable through the device plugin framework plus a stack of NVIDIA-specific components — and getting this right is the difference between a cluster that has GPUs and one that can actually run training and inference on them.

flowchart LR
    GPU[NVIDIA GPU]
    DevicePlugin[Device Plugin]
    Kubelet[Kubelet]
    Scheduler[Scheduler]
    Pod[GPU Job]
    GPU --> DevicePlugin
    DevicePlugin --> Kubelet
    Kubelet --> Scheduler
    Scheduler --> Pod
    Pod --> Resource["nvidia.com/gpu"]

How a GPU becomes schedulable

The device plugin runs as a DaemonSet on every GPU node. It discovers the GPUs, then registers them with the kubelet over gRPC as an extended resource named nvidia.com/gpu. From that point the scheduler can treat GPUs like any other countable resource — but a real GPU workload also has to land on a GPU node, which means tolerating the node's taint and selecting it by label:

  apiVersion: batch/v1
  kind: Job
  metadata:
    name: train-resnet
  spec:
    backoffLimit: 2
    template:
      spec:
        restartPolicy: Never
        nodeSelector:
          nvidia.com/gpu.present: "true"     # NFD label from the GPU Operator
        tolerations: # GPU nodes are tainted to repel CPU work
          - key: nvidia.com/gpu
            operator: Exists
            effect: NoSchedule
        containers:
          - name: trainer
            image: nvcr.io/example/trainer:24.10
            command: [ "python", "train.py" ]
            resources:
              limits:
                nvidia.com/gpu: 2             # requests == limits is enforced for GPUs

One important quirk visible here: GPUs are not overcommittable by default. You can only request whole units, and you set them under limits only (requests is auto-set to match) — there's no fractional or burstable GPU in the base model. That constraint drives a lot of GPU cluster economics, and it's why MIG and time-slicing (below) exist.

---

The NVIDIA GPU Operator

Standing up the GPU stack by hand — matching driver versions, the container toolkit, the device plugin, node labeling — is fragile. The GPU Operator automates all of it as a set of controllers and DaemonSets:

• Driver + container toolkit (so containers can reach the GPU)
• Device plugin (advertises nvidia.com/gpu)
• Node Feature Discovery (labels nodes with GPU model, memory, MIG capability)
• DCGM Exporter (streams GPU telemetry — utilization, memory, temperature, ECC errors — into Prometheus/Grafana)
• MIG Manager (partitioning, below)

DCGM metrics are what turn "the GPUs are busy" into an actual SLO dashboard, and they're the basis for GPU-aware autoscaling.

Topology-aware placement

At single-node multi-GPU scale, which GPUs a Pod gets matters. GPUs connected by NVLink talk far faster than GPUs forced across PCIe or sockets, and a GPU should sit on the same NUMA node as its CPU and its NIC. The **Topology Manager ** aligns CPU, device, and NUMA assignments so a Pod doesn't get GPUs that can't talk to each other efficiently. Ignore this and your collective-communication bandwidth quietly tanks.

Multi-node distributed training

This is where the default scheduler actively fails you. A data-parallel training job is N Pods that must run * simultaneously* — but the default scheduler places Pods one at a time. Schedule 6 of 8, run out of GPUs, and you've got 6 Pods burning allocated GPUs while deadlocked waiting for 2 that will never come. The fix is gang scheduling ( all-or-nothing): Volcano, Kueue, or the coscheduling plugin.

Once the Pods are co-scheduled, the network is the bottleneck. Collective operations run over NCCL, and to hit real bandwidth you want GPUDirect RDMA so GPUs DMA across the network without staging through host memory — over InfiniBand on-prem, or EFA on AWS. (For reference, a well-tuned RDMA fabric gets you into the tens of GB/s of NCCL bus bandwidth per node-pair; a misconfigured one falls back to TCP and a fraction of that — which is exactly the kind of regression DCGM + NCCL tests catch.)

flowchart LR
    subgraph Node1
        GPU1[GPU]
        Trainer1[Trainer Pod]
    end

    subgraph Node2
        GPU2[GPU]
        Trainer2[Trainer Pod]
    end

    subgraph Node3
        GPU3[GPU]
        Trainer3[Trainer Pod]
    end

    Trainer1 <-- NCCL --> Trainer2
    Trainer2 <-- NCCL --> Trainer3
    Trainer3 <-- NCCL --> Trainer1

---

Failure modes worth knowing

The glossary doesn't prepare you for the 3am pages. A starter set:

• Pod stuck Terminating — usually a finalizer that never completed or a stuck volume detach.
• OOMKilled — container exceeded its memory limit; raise the limit or fix the leak.
• ImagePullBackOff — bad image ref or missing registry credentials.
• etcd NOSPACE alarm — DB hit its quota; needs compaction + defrag.
• CoreDNS latency — DNS is a shockingly common cause of "random" app slowness at scale.
• Node drain stalls — a PodDisruptionBudget correctly refusing to let the last healthy replica be evicted.
• GPU "lost" — driver/toolkit version mismatch or an Xid error; DCGM surfaces these before your job does.

The triage commands you'll actually reach for:

  # What's not Running, across all namespaces
  kubectl get pods -A --field-selector status.phase!=Running
  
  # Events at the bottom tell you WHY: scheduling failure, image pull, OOM
  kubectl describe pod <pod> -n <ns>
  
  # Logs from the *previous* (crashed) container, not the restarted one
  kubectl logs <pod> -n <ns> --previous
  
  # Actual usage vs. what was requested (needs metrics-server)
  kubectl top pods -n <ns>
  kubectl top nodes
  
  # Cluster-wide event stream, newest last
  kubectl get events -A --sort-by=.lastTimestamp

---

Test setup

For local learning, Minikube spins up a single-node cluster (control plane + worker in one VM) with a runtime preinstalled.

Requirements: 2+ CPUs · 2 GB free memory · 20 GB free disk · a VM/container driver (Docker, Podman, KVM, Hyper-V, VirtualBox, VMware, etc.).

kubectl is the CLI that talks to the API server to create and delete objects — and it works against any conformant cluster, not just Minikube, so the muscle memory transfers straight to production.

One honest limitation: Minikube is great for the fundamentals but useless for the GPU material above — you need real NVIDIA hardware (or a cloud GPU instance) plus the GPU Operator to exercise device plugins, MIG, and multi-node NCCL. That gap is exactly why a built-from-scratch GPU lab is worth more on a portfolio than another Minikube tutorial.

---

Putting it together

A typical AI inference platform might look like:

• Ingress for external traffic
• Service for load balancing
• Deployment for API servers
• GPU-enabled Deployment for model serving
• ConfigMaps for configuration
• Secrets for credentials
• Persistent Volumes for model storage
• Prometheus + Grafana for monitoring
• NVIDIA GPU Operator for GPU lifecycle management

---

Related Posts

• NVIDIA AI Infrastructure and Operations Fundamentals — the hardware and platform layer this control-plane overview sits on top of
• AI Infra Computing: GPU, DPU, Virtualization, DGX Systems — the GPU/DPU hardware this post explains how Kubernetes schedules