Kubernetes ☸️

Most Kubernetes write-ups start with a glossary — Pod, Service, Deployment — and stop there. The glossary is the easy part. The thing worth internalizing is the one idea the whole system is built on, because every object below is just a different face of it.

Who this guide is for

This article assumes you already know basic container concepts and want to understand how Kubernetes behaves in production. The focus is on the mechanics behind scheduling, networking, storage, and GPU workloads rather than memorizing Kubernetes objects.

The one idea: reconciliation

Kubernetes is not a script runner. It is a set of control loops. You declare a desired state ("I want 3 replicas of this image, fronted by a load balancer"), Kubernetes records that intent, and a controller continuously compares desired state against observed state and takes action to close the gap. Pod died? The gap reappears, the controller acts. Node vanished? Same loop, same response.

This is why Kubernetes is declarative rather than imperative, and it's the reason it self-heals without anyone writing recovery scripts. Once this clicks, the object zoo stops being a list to memorize and becomes obvious: a Deployment is a loop that keeps N Pods alive; a Service is a loop that keeps an IP pointed at healthy endpoints; the scheduler is a loop that keeps unscheduled Pods moving onto nodes. Everything is reconciliation.

flowchart LR
    User[Desired State]
    API[API Server]
    ETCD[(etcd)]
    Controller[Controller]
    Cluster[Cluster State]

    User --> API
    API --> ETCD

    Controller --> API
    API --> Controller

    Cluster --> Controller
    Controller --> Cluster

---

Architecture

A Node is a physical or virtual machine that runs part of the cluster. Nodes split into the control plane and workers.

flowchart TB

    subgraph ControlPlane[Control Plane 🎛️]
        API[API Server ⚡]
        Scheduler[Scheduler 🕣]
        Controller[Controller Manager 🔄]
        ETCD[(etcd 🛢️)]
    end

    subgraph Worker1[Worker Node 🖥️]
        Kubelet1[Kubelet 👨🏻‍🔧]
        Runtime1[Container Runtime 📟]
        Pod1[Pods 💠 ]
    end

    subgraph Worker2[Worker Node 🖥️]
        Kubelet2[Kubelet 👨🏻‍🔧]
        Runtime2[Container Runtime 📟]
        Pod2[Pods 💠 ]
    end

    API <--> Scheduler
    API <--> Controller
    API <--> ETCD

    API <--> Kubelet1
    API <--> Kubelet2

    Kubelet1 --> Runtime1
    Runtime1 --> Pod1

    Kubelet2 --> Runtime2
    Runtime2 --> Pod2

Control plane 🎛️

The Control Plane is responsible for managing the cluster.

Four core components, each a piece of the reconciliation machinery:

1. API Server ⚡

The only component that talks to etcd, and the single front door for everything else (kubectl, controllers, kubelets).

• It's a stateless REST layer doing auth, admission control, and validation. Everything in the cluster is a write to, or a watch on, the API server.

Bottlenecks and scaling

1. API Servers can take up a fair bit of memory, and that tends to scale linearly with the number of nodes in the cluster.

• Cluster with 7,500 nodes can take 70GB of heap being used per API Server
• Best to run 3 or 5 API Servers in a dedicated Nods outside kube to avoid single point of failure.
• API Servers are stateless and generally easy to run in a self-healing instance group or scaleset.

2. Autoscaling too much at once.

• There are many requests generated when a new node joins a cluster, and adding hundreds of nodes at once can overload API server capacity.
• Smoothing this out, even just by a few seconds, has helped avoid outages.

kubectl

Command line tool for communicating with a Kubernetes cluster's control plane, using the Kubernetes API.

2. Scheduler 🕣

watches for Pods with no node assigned and binds them to a node.

• It does not start containers; it only decides where.

Scheduling internals

The scheduler runs two phases per Pod:

1. Filter (predicates): eliminate nodes that can't run the Pod — insufficient resources, failed taint match, unsatisfied node selector.
2. Score (priorities): rank the surviving nodes — spread, least-loaded, affinity — and bind to the winner.

The levers you actually use to control placement:

• Taints & Tolerations — a node repels Pods unless they explicitly tolerate its taint. This is how you keep general workloads off specialized nodes (the standard pattern for GPU nodes).
• Node affinity — pull Pods toward nodes with certain labels.
• Pod affinity / anti-affinity — co-locate or separate Pods relative to each other (e.g. spread replicas across zones).
• Topology spread constraints — finer control over even distribution across failure domains.
• Extended resources — beyond CPU and memory, nodes can advertise custom countable resources. This is the hook the entire GPU story hangs on.

Resource model & QoS

Every container can declare requests (what the scheduler reserves) and limits (the hard ceiling). This pair drives two things people learn the hard way:

• Scheduling: the scheduler places Pods based on requests, not actual usage. Under-request and you overcommit a node into instability; over-request and you waste capacity.
• QoS class: Pods are bucketed into Guaranteed (requests == limits), Burstable, or BestEffort. Under memory pressure, the kubelet evicts BestEffort first and Guaranteed last. A Pod exceeding its memory limit is OOMKilled — one of the most common "why did my Pod restart" answers.

3. Controller Manager 🔄

Controllers are control loops that watch the state of your cluster, then make or request changes where needed. Each controller tries to move the current cluster state closer to the desired state.

• Runs the built-in control loops Deployment, ReplicaSet, Node, Job, and dozens more), each reconciling its slice of desired vs observed state.

4. etcd 🛢️

The cluster's source of truth.

A distributed key-value store using the Raft consensus protocol, so it needs an odd-numbered quorum (3 or 5 members) to tolerate failures.

When a large cluster mysteriously gets slow, etcd disk latency is the first place to look.

Bottlenecks and scaling

etcd is the thing that actually caps your cluster's scale.

1. It's sensitive to disk fsync latency

• put it on fast local SSD (200us), never network storage (2ms),

2. etcd’s hard storage limit

• default DB size quota is 2 GB (8 GB is the practical ceiling),
• a write-heavy cluster needs periodic compaction and defrag or it wedges with a NOSPACE alarm.

3. Kubernetes Events in a separate etcd cluster

• The default etcd cluster is shared between the API server and the event store.
• In a large cluster, events can overwhelm the main etcd cluster and cause API server timeouts.
• The fix is to run a separate etcd cluster for events, which is supported in Kubernetes 1.24+.

---

Nodes / Worker node 🖥️

A node is a VM or a physical computer that serves as a worker machine in a Kubernetes cluster.

• Workers do the actual work, so they're sized for it. Each can run many Pods.

Each worker runs three processes:

1. Container Runtime Interface (CRI) 📟

The CRI is a plugin interface which enables the kubelet to use a wide variety of container runtimes, without having a need to recompile the cluster components.

• Example: containerd or CRI-O, spoken to over the Container Runtime Interface (CRI).

Note: Docker-as-a-runtime was removed in Kubernetes v1.24 (the dockershim deprecation). Docker-built images still run everywhere — they're OCI-compliant — but Docker the daemon is no longer the runtime.

2. Kubelet 👨🏻‍🔧

The node agent.

It watches the API server for Pods bound to its node and drives the runtime to make them real, then reports status back.

3. Kube Proxy 🌐

Programs the node's networking so traffic to a Service IP reaches a healthy Pod (more on its modes below).

---

Workload objects

Pod 📦

The smallest schedulable unit — an abstraction over one or more co-located containers that share a network namespace (same IP, same localhost) and can share volumes.

Most Pods hold a single app container plus the occasional sidecar.

Pods are cattle, not pets: each gets an internal IP, but on respawn it's a new Pod with a new IP. You almost never create a bare Pod — you let a higher-level controller manage them.

☸️ Kubernetes Cluster
├── 🖥️ Node 1
│   ├── 📦 Pod A
│   │   ├── 🐳 Container 1
│   │   └── 🐳 Container 2
│   └── 📦 Pod B
└── 🖥️ Node 2
└── 📦 Pod C

Deployment 🪂

The control loop for stateless apps.

You give it a Pod template and a replica count; it creates a ReplicaSet, keeps that many Pods alive, and handles rolling updates and rollbacks.

flowchart TB

    Deployment["Deployment 🪂"]
    ReplicaSet["ReplicaSet 💠 "]

    Deployment --> ReplicaSet

    ReplicaSet --> Pod1["Pod 📦"]
    ReplicaSet --> Pod2["Pod 📦"]
    ReplicaSet --> Pod3["Pod 📦"]

    Service["Service 🌐"]
    Service --> Pod1
    Service --> Pod2
    Service --> Pod3

ReplicaSet 💠

Maintain a stable set of replica Pods running at any given time. Usually, you define a Deployment and let that Deployment manage ReplicaSets automatically.

If a Pod or node dies, the loop notices the gap and replaces it. Scaling is one field.

Example Config:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: web
spec:
  replicas: 3
  selector:
    matchLabels: { app: web }
  template:
    metadata:
      labels: { app: web }
    spec:
      containers:
        - name: web
          image: registry.example.com/web:1.4.2
          ports:
            - containerPort: 8080
          resources:             # requests == limits -> Guaranteed QoS
            requests: { cpu: "500m", memory: "256Mi" }
            limits:   { cpu: "500m", memory: "256Mi" }
          readinessProbe:        # gate traffic until the app is actually ready
            httpGet: { path: /healthz, port: 8080 }
            initialDelaySeconds: 5
          livenessProbe:         # restart if it wedges
            httpGet: { path: /healthz, port: 8080 }
            periodSeconds: 10

Two details that matter: setting requests == limits puts this Pod in the Guaranteed QoS class (evicted last under pressure), and the readiness probe is what keeps the fronting Service from sending traffic to a Pod that's still warming up — a common cause of deploy-time 502s when it's missing.

StatefulSet 💾

R uns a group of Pods, and maintains a sticky identity for each of those Pods. This is useful for managing applications that need persistent storage or a stable, unique network identity.

• For workloads that need stable identity and storage — databases, message brokers, anything where replica-0 is meaningfully different from replica-1.
• Unlike a Deployment, it gives each Pod a sticky network name and its own persistent volume that survives rescheduling, and it manages Pods in order.

The honest caveat: a StatefulSet handles the orchestration of stateful Pods, but it does not solve data consistency or replication for you — that's the database's job.

People reach for StatefulSets expecting magic clustering; they get stable plumbing, nothing more.

---

Storage

Pods are ephemeral, so anything written to a container's filesystem dies with it. Volumes outlive the container; PersistentVolumes (PV) outlive the Pod. The model:

Persistent Volume Claim (PVC) 🧾

A Pod's request for storage ("10Gi, fast").

Persistent Volume (PV) 🛢️

The actual backing storage that satisfies a claim.

Storage Class 🗃️

Defines how PVs get provisioned dynamically (e.g. an AWS gp3 EBS class), so you don't pre-create volumes by hand.

Kubernetes orchestrates the binding; it does not itself guarantee durability — that's the storage backend's job.

apiVersion: v1
kind: Service
metadata:
  name: nginx
  labels:
    app: nginx
spec:
  ports:
    - port: 80
      name: web
  clusterIP: None
  selector:
    app: nginx
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: web
spec:
  selector:
    matchLabels:
      app: nginx # has to match .spec.template.metadata.labels
  serviceName: "nginx"
  replicas: 3 # by default is 1
  minReadySeconds: 10 # by default is 0
  template:
    metadata:
      labels:
        app: nginx # has to match .spec.selector.matchLabels
    spec:
      terminationGracePeriodSeconds: 10
      containers:
        - name: nginx
          image: registry.k8s.io/nginx-slim:0.24
          ports:
            - containerPort: 80
              name: web
          volumeMounts:
            - name: www
              mountPath: /usr/share/nginx/html
  volumeClaimTemplates:
    - metadata:
        name: www
      spec:
        accessModes: [ "ReadWriteOnce" ]
        storageClassName: "my-storage-class" # 🗃️ Storage Class
        resources:
          requests:
            storage: 1Gi

DaemonSet 👺

Defines Pods that provide node-local facilities. These might be fundamental to the operation of your cluster, such as a networking helper tool, or be part of an add-on.

Runs exactly one Pod per node (or per matching node). This is how node-level agents ship: log collectors, CNI plugins, monitoring exporters — and, relevant later, the NVIDIA device plugin and DCGM exporter, which must run on every GPU node.

Job 🏋️‍♀️

Run-to-completion workloads rather than long-running services.

• A Job runs a Pod until it succeeds; CronJob schedules them.

apiVersion: batch/v1
kind: Job
metadata:
  name: my-job
spec:
  template:
    spec:
      containers:
      - name: my-job
        image: my-image
      restartPolicy: Never
  backoffLimit: 4

CronJob 📅

A CronJob starts one-time Jobs on a repeating schedule.

Batch ML training fits here — though multi-Pod distributed training needs scheduling help the default Job controller doesn't provide (see GPUs, below).

apiVersion: batch/v1
kind: CronJob
metadata:
  name: my-cronjob
spec:
  schedule: "0 0 * * *"
  jobTemplate:
    spec:
      template:
        spec:
          containers:
          - name: my-job
            image: my-image
          restartPolicy: OnFailure

---

Networking

Kubernetes mandates a flat network: every Pod gets a unique, routable IP, and any Pod can reach any other Pod without NAT.

Kubernetes itself doesn't implement this — it delegates to a CNI plugin (Calico, Cilium, AWS VPC CNI, etc.), which is why your network behavior depends heavily on which CNI you picked.

flowchart LR

    User[Browser]

    Ingress["Ingress / Gateway 🚪"]
    Service["Service 🌐"]
    Pod1["Pod 📦"]
    Pod2["Pod 📦"]
    Pod3["Pod 📦"]

    User --> Ingress
    Ingress --> Service

    Service --> Pod1
    Service --> Pod2
    Service --> Pod3

Service 🌐

Pods are ephemeral; their IPs churn. A Service is a stable virtual IP (and DNS name) that load-balances across a healthy set of Pods, selected by label. The Service's identity is independent of any Pod's lifecycle — that's the whole point.

• ClusterIP (default): reachable only inside the cluster — e.g. a database other Pods talk to.
• NodePort / LoadBalancer: exposed externally — e.g. an API hit from a browser.

Kube-proxy modes (a real-world gotcha)

How a Service IP actually routes to a Pod depends on kube-proxy's mode:

• iptables (default): fine for small clusters, but rule evaluation is O(n) in the number of Services, so it degrades badly at thousands of Services.
• IPVS: hash-based, scales far better for large Service counts.
• nftables: newer, addresses the iptables scaling problem natively.
• eBPF (Cilium): replaces kube-proxy entirely and routes in-kernel — increasingly the choice for large or performance-sensitive clusters.

Knowing this distinction is a small thing that signals you've run Kubernetes past toy scale.

Ingress / Gateway API 🚪

Ingress maps external HTTP(S) URLs to Services — host/path routing, TLS termination — so clients hit a meaningful hostname instead of a raw IP and port. The Gateway API is the newer, more expressive successor that's gradually replacing Ingress for serious setups.

---

Configuration & secrets

1. ConfigMap 📜

external configuration injected as env vars or files, so the same image runs across environments without a rebuild.

2. Secret 🔐

Same mechanism for credentials and certs, stored base64-encoded. Critical caveat that trips people up: base64 is encoding, not encryption. A raw Secret is plaintext to anyone with API or etcd access. For real protection you enable encryption at rest for etcd and/or use an external manager (Vault, AWS/GCP secret stores via the Secrets Store CSI driver).

---

GPUs on Kubernetes

Kubernetes has no native concept of a GPU. Out of the box it understands CPU and memory, full stop. GPUs are made schedulable through the device plugin framework plus a stack of NVIDIA-specific components — and getting this right is the difference between a cluster that has GPUs and one that can actually run training and inference on them.

flowchart LR

    GPU[NVIDIA GPU]

    DevicePlugin[Device Plugin]
    Kubelet[Kubelet]
    Scheduler[Scheduler]

    Pod[GPU Job]

    GPU --> DevicePlugin
    DevicePlugin --> Kubelet
    Kubelet --> Scheduler

    Scheduler --> Pod

    Pod --> Resource["nvidia.com/gpu"]

How a GPU becomes schedulable

The device plugin runs as a DaemonSet on every GPU node. It discovers the GPUs, then registers them with the kubelet over gRPC as an extended resource named nvidia.com/gpu. From that point the scheduler can treat GPUs like any other countable resource — but a real GPU workload also has to land on a GPU node, which means tolerating the node's taint and selecting it by label:

apiVersion: batch/v1
kind: Job
metadata:
  name: train-resnet
spec:
  backoffLimit: 2
  template:
    spec:
      restartPolicy: Never
      nodeSelector:
        nvidia.com/gpu.present: "true"     # NFD label from the GPU Operator
      tolerations:                          # GPU nodes are tainted to repel CPU work
        - key: nvidia.com/gpu
          operator: Exists
          effect: NoSchedule
      containers:
        - name: trainer
          image: nvcr.io/example/trainer:24.10
          command: ["python", "train.py"]
          resources:
            limits:
              nvidia.com/gpu: 2             # requests == limits is enforced for GPUs

One important quirk visible here: GPUs are not overcommittable by default. You can only request whole units, and you set them under limits only (requests is auto-set to match) — there's no fractional or burstable GPU in the base model. That constraint drives a lot of GPU cluster economics, and it's why MIG and time-slicing (below) exist.

---

The NVIDIA GPU Operator

Standing up the GPU stack by hand — matching driver versions, the container toolkit, the device plugin, node labeling — is fragile. The GPU Operator automates all of it as a set of controllers and DaemonSets:

• Driver + container toolkit (so containers can reach the GPU)
• Device plugin (advertises nvidia.com/gpu)
• Node Feature Discovery (labels nodes with GPU model, memory, MIG capability)
• DCGM Exporter (streams GPU telemetry — utilization, memory, temperature, ECC errors — into Prometheus/Grafana)
• MIG Manager (partitioning, below)

DCGM metrics are what turn "the GPUs are busy" into an actual SLO dashboard, and they're the basis for GPU-aware autoscaling.

Topology-aware placement

At single-node multi-GPU scale, which GPUs a Pod gets matters. GPUs connected by NVLink talk far faster than GPUs forced across PCIe or sockets, and a GPU should sit on the same NUMA node as its CPU and its NIC. The Topology Manager aligns CPU, device, and NUMA assignments so a Pod doesn't get GPUs that can't talk to each other efficiently. Ignore this and your collective-communication bandwidth quietly tanks.

Multi-node distributed training

This is where the default scheduler actively fails you. A data-parallel training job is N Pods that must run simultaneously — but the default scheduler places Pods one at a time. Schedule 6 of 8, run out of GPUs, and you've got 6 Pods burning allocated GPUs while deadlocked waiting for 2 that will never come. The fix is gang scheduling (all-or-nothing): Volcano, Kueue, or the coscheduling plugin.

Once the Pods are co-scheduled, the network is the bottleneck. Collective operations run over NCCL, and to hit real bandwidth you want GPUDirect RDMA so GPUs DMA across the network without staging through host memory — over InfiniBand on-prem, or EFA on AWS. (For reference, a well-tuned RDMA fabric gets you into the tens of GB/s of NCCL bus bandwidth per node-pair; a misconfigured one falls back to TCP and a fraction of that — which is exactly the kind of regression DCGM + NCCL tests catch.)

flowchart LR

    subgraph Node1
        GPU1[GPU]
        Trainer1[Trainer Pod]
    end

    subgraph Node2
        GPU2[GPU]
        Trainer2[Trainer Pod]
    end

    subgraph Node3
        GPU3[GPU]
        Trainer3[Trainer Pod]
    end

    Trainer1 <-- NCCL --> Trainer2
    Trainer2 <-- NCCL --> Trainer3
    Trainer3 <-- NCCL --> Trainer1

---

Failure modes worth knowing

The glossary doesn't prepare you for the 3am pages. A starter set:

• Pod stuck Terminating — usually a finalizer that never completed or a stuck volume detach.
• OOMKilled — container exceeded its memory limit; raise the limit or fix the leak.
• ImagePullBackOff — bad image ref or missing registry credentials.
• etcd NOSPACE alarm — DB hit its quota; needs compaction + defrag.
• CoreDNS latency — DNS is a shockingly common cause of "random" app slowness at scale.
• Node drain stalls — a PodDisruptionBudget correctly refusing to let the last healthy replica be evicted.
• GPU "lost" — driver/toolkit version mismatch or an Xid error; DCGM surfaces these before your job does.

The triage commands you'll actually reach for:

# What's not Running, across all namespaces
kubectl get pods -A --field-selector status.phase!=Running

# Events at the bottom tell you WHY: scheduling failure, image pull, OOM
kubectl describe pod <pod> -n <ns>

# Logs from the *previous* (crashed) container, not the restarted one
kubectl logs <pod> -n <ns> --previous

# Actual usage vs. what was requested (needs metrics-server)
kubectl top pods -n <ns>
kubectl top nodes

# Cluster-wide event stream, newest last
kubectl get events -A --sort-by=.lastTimestamp

---

Test setup

For local learning, Minikube spins up a single-node cluster (control plane + worker in one VM) with a runtime preinstalled.

Requirements: 2+ CPUs · 2 GB free memory · 20 GB free disk · a VM/container driver (Docker, Podman, KVM, Hyper-V, VirtualBox, VMware, etc.).

kubectl is the CLI that talks to the API server to create and delete objects — and it works against any conformant cluster, not just Minikube, so the muscle memory transfers straight to production.

One honest limitation: Minikube is great for the fundamentals but useless for the GPU material above — you need real NVIDIA hardware (or a cloud GPU instance) plus the GPU Operator to exercise device plugins, MIG, and multi-node NCCL. That gap is exactly why a built-from-scratch GPU lab is worth more on a portfolio than another Minikube tutorial.

---

Putting it together

A typical AI inference platform might look like:

• Ingress for external traffic
• Service for load balancing
• Deployment for API servers
• GPU-enabled Deployment for model serving
• ConfigMaps for configuration
• Secrets for credentials
• Persistent Volumes for model storage
• Prometheus + Grafana for monitoring
• NVIDIA GPU Operator for GPU lifecycle management