Kubernetes Networking 🛜

Kubernetes networking is built on one rule that everything else follows from:

The Kubernetes Networking Model

Every Pod gets its own IP address. Pods can reach any other Pod directly, without NAT.

Three requirements the model must satisfy:

Requirement	Meaning
Pod-to-Pod	Any Pod can reach any other Pod using its IP, even on a different node
No NAT	The IP a Pod sees as its own is the same IP other Pods use to reach it
Node-to-Pod	Nodes can reach any Pod directly

This is called the flat network model — from any Pod's perspective, every other Pod is reachable at its IP with no address translation.

flowchart LR

subgraph Node1["Node 1"]
  PodA["Pod A <br/> 10.244.1.5"]
  PodB["Pod B <br/> 10.244.1.6"]
end

subgraph Node2["Node 2"]
  PodC["Pod C <br/> 10.244.2.3"]
end

PodA-->|"direct IP <br/> no NAT"| PodC

PodA-->|"same node <br/> loopback bridge"| PodB

Pod IPs come from a cluster-wide Pod CIDR (e.g., 10.244.0.0/16).

Each node gets a slice of that range (e.g., 10.244.1.0/24 for Node 1, 10.244.2.0/24 for Node 2).

This sounds simple. Implementing it across hundreds of nodes, cloud VPCs, and heterogeneous hardware is what the networking stack is for.

---

Linux Networking Fundamentals

Before understanding Kubernetes networking, you need to understand Linux networking.

Kubernetes doesn't invent a new networking stack. It builds on top of the Linux kernel's networking features such as network namespaces, virtual Ethernet (veth) devices, bridges, routing tables, and iptables.

Understanding these concepts makes Kubernetes networking much easier to understand.

Network namespace 📦

A network namespace provides an isolated networking environment.

Each namespace has its own:

• Network interfaces
• Routing table
• Firewall rules
• ARP table
• Port space

Think of it as a completely independent network stack.

flowchart LR
    subgraph Host Namespace
        ETH0[eth0]
        LO0[lo]
    end

    subgraph Namespace A
        ETH1[eth0]
        LO1[lo]
    end

    subgraph Namespace B
        ETH2[eth0]
        LO2[lo]
    end

Processes inside one namespace cannot see the interfaces of another namespace.

Kubernetes creates a separate network namespace for every Pod.

Network Interfaces 🔌

A network interface connects a system to a network.

Common interfaces include:

Interface	Purpose
eth0	Physical Ethernet interface
lo	Loopback interface
veth	Virtual Ethernet device
docker0	Docker bridge
cni0	Kubernetes bridge (bridge-based CNIs)
vxlan.calico	VXLAN tunnel interface

Loopback Interface

Every Linux namespace has a loopback interface.

Traffic sent to localhost never leaves the namespace.

127.0.0.1 --> localhost

This is why containers inside the same Pod can communicate using:

localhost:8080

Virtual Ethernet (veth) Pair 🪢

A veth pair acts like a virtual network cable.

Packets entering one end immediately appear on the other.

If one end belongs to a Pod, the other remains on the host.

flowchart LR
    A["veth0 📶"] <--> B["veth1 🔌"]

When Kubernetes creates a Pod, CNI creates a veth pair.

flowchart LR
    subgraph Pod
        ETH["eth0 📶<br/>10.244.1.5"]
    end

    VP[veth-peer]
    VH[veth-host]
    BR[cni0 Bridge]

    ETH --- VP
    VP --- VH
    VH --- BR

The Pod thinks it owns eth0.

Linux Bridge 🔗

A Linux bridge behaves like a virtual Layer-2 switch.

Multiple interfaces connect to it.

flowchart LR
    subgraph Pod1
        ETH0["eth0 📶<br/>10.244.1.2"]
    end

    subgraph Pod2
        ETH1["eth0 📶<br/>10.244.1.3"]
    end

    subgraph Host
        VETH0["veth0 🔌"]
        VETH1["veth1 🔌"]
        BRIDGE["cni0 Bridge 🔗"]
    end

    ETH0 --- VP0[veth-peer0]
    VP0 --- VETH0

    ETH1 --- VP1[veth-peer1]
    VP1 --- VETH1

    VETH0 --- BRIDGE
    VETH1 --- BRIDGE

The bridge forwards Ethernet frames between connected interfaces.

Same-Node vs Cross-Node Communication

flowchart TD

subgraph "Same node"
    PodA["Pod A <br/> 10.244.1.5"]
    PodA-->|"veth → bridge"| Bridge["cbr0 bridge <br/> Linux bridge 🔗"]
    Bridge-->|"veth → Pod"| PodB["Pod B <br/> 10.244.1.6"]
end

subgraph "Cross-node (Flannel VXLAN)"
    PodC["Pod C <br/> 10.244.1.7"]
    PodC-->|"veth → bridge → VXLAN"| Tunnel["UDP tunnel <br/> encapsulated 🔌"]
    Tunnel-->|"decapsulate"| PodD["Pod D <br/> 10.244.2.3 <br/> on Node 2"]
end

Same-node: packets go through the Linux bridge

• no overhead, effectively loopback speed.

VXLAN

VXLAN creates an overlay network cross Node.

• Cross-node with VXLAN: packets are encapsulated in UDP, sent to the destination node's IP, then decapsulated.

• This is the "overlay" — it adds ~10–15% CPU overhead and latency, which is why GPU training workloads replace it with InfiniBand (see Network Operator post).

Why This Matters for GPU Workloads

The CNI overlay network (VXLAN) is fine for API traffic — a few MB/s per request.

For distributed training, AllReduce between 8 DGX nodes transfers hundreds of GB/s of gradient data. The VXLAN overhead and CPU involvement make overlay networking a hard bottleneck.

The solution (covered in the Network Operator post):

• A second NIC on each GPU node — an InfiniBand HCA
• RDMA — bypasses the kernel and CPU entirely
• Multus CNI — attaches both the cluster CNI (for API traffic) and the RDMA NIC (for training traffic) to the same Pod

The cluster CNI (Calico/Cilium) handles all Kubernetes control plane and application traffic. InfiniBand handles all GPU-to-GPU training traffic. They are completely separate data paths.

---

How Networking work in Pods

The Full Request Flow

A browser request hitting an application running in Kubernetes:

sequenceDiagram

participant Browser
participant LB as Cloud Load Balancer
participant NGINX as Ingress Controller Pod
participant SvcIP as kube-proxy (iptables)
participant AppPod as App Pod

Browser->>LB: GET https://api.example.com/v1/users
LB->>NGINX: TCP to NodePort 443
NGINX->>NGINX: TLS termination <br/> route by hostname + path
NGINX->>SvcIP: HTTP to api-svc ClusterIP:80
SvcIP->>AppPod: DNAT → 10.244.2.3:8080
AppPod-->>Browser: 200 OK (response path)

CNI (Container Network Interface) 📟

Give every Pod an IP address and make Pods able to communicate with each other.

• CRI manages containers.
• CNI manages networking.

Kubernetes does not implement networking itself. It delegates to a CNI plugin via the Container Network Interface spec.

flowchart TD

    Kubelet
    Kubelet-->|"calls CNI"| CNI["CNI Plugin <br/> (Flannel / Calico / Cilium)"]
    CNI-->|"sets up"| VethPair["veth pair <br/> (Pod ↔ host bridge)"]
    CNI-->|"routes packets"| CrossNode["Cross-node routing <br/> (overlay or BGP)"]

When a Pod is created, the Kubelet calls the CNI plugin to:

1. Create a network namespace for the Pod
2. Create a veth pair — one end inside the Pod (eth0), one end on the host bridge
3. Assign the Pod's IP from the node's slice of Pod CIDR
4. Set up routing so packets to other nodes' Pod CIDRs reach the right node

flowchart LR
    A[Kubelet]
    A -->|CRI| B[containerd]
    B -->|OCI| C[runc]
    C --> D[Linux Kernel]

    A -->|CNI| E[CNI Plugin]
    E --> D

The network is configured before application containers start.

sequenceDiagram
participant API
participant Scheduler
participant Kubelet
participant Runtime
participant CNI

    API->>Scheduler: Schedule Pod

    Scheduler->>Kubelet: Assign Pod

    Kubelet->>Runtime: Create Pod Sandbox

    Runtime->>CNI: ADD Network

    CNI->>Runtime: Pod IP

    Runtime->>Kubelet: Sandbox Ready

    Kubelet->>Runtime: Start Containers

Common CNI Plugins

Plugin	Mechanism	Key strength
Flannel	VXLAN overlay (UDP encapsulation)	Simple, works anywhere
Calico	BGP routing (no overlay)	High performance, NetworkPolicy support
Cilium	eBPF (kernel-level packet processing)	Observability, L7 policy, high performance
Weave	Encrypted overlay	Easy multi-cluster

---

Services 🌐

Stable Endpoints for Ephemeral Pods

Pod IPs are ephemeral — when a Pod is replaced (crash, rolling update), it gets a new IP.

A Service provides a stable virtual IP (ClusterIP) that load-balances across all healthy Pods matching a label selector.

flowchart LR

    Client["Client Pod"]
    Client-->ClusterIP["Service ClusterIP <br/> 10.96.45.12 <br/> (stable forever)"]
    
    ClusterIP-->|"kube-proxy <br/> iptables rules"| Pod1["Pod 10.244.1.5"]
    ClusterIP-->Pod2["Pod 10.244.1.7"]
    ClusterIP-->Pod3["Pod 10.244.2.3"]

Service Types

Type	Use case	How it works
ClusterIP	In-cluster communication	Virtual IP only reachable inside the cluster
NodePort	Basic external access	Opens a port (30000–32767) on every node; traffic forwarded to Service
LoadBalancer	Production external access	Cloud provider provisions a load balancer with a public IP
ExternalName	DNS alias to external service	Returns a CNAME — no proxying
Headless	StatefulSets, direct Pod access	No ClusterIP; DNS returns individual Pod IPs

# ClusterIP (default)
apiVersion: v1
kind: Service
metadata:
  name: web-svc
spec:
  selector:
    app: web          # routes to all Pods with this label
  ports:
  - port: 80
    targetPort: 8080

---
# LoadBalancer — gets a cloud public IP
apiVersion: v1
kind: Service
metadata:
  name: web-lb
spec:
  type: LoadBalancer
  selector:
    app: web
  ports:
  - port: 80
    targetPort: 8080

---
# Headless — for StatefulSets (stable DNS per Pod)
apiVersion: v1
kind: Service
metadata:
  name: postgres-headless
spec:
  clusterIP: None     # ← headless
  selector:
    app: postgres
  ports:
  - port: 5432

---

kube-proxy 📡

kube-proxy runs on every node as a DaemonSet.

Services have no process listening on the ClusterIP.

The IP is virtual — kube-proxy programs iptables rules on every node that intercept packets to the ClusterIP and DNAT them to a real Pod IP.

sequenceDiagram

participant Pod as Client Pod
participant IPTables as iptables (on node)
participant Backend as Backend Pod

Pod->>IPTables: packet to 10.96.45.12:80
IPTables->>IPTables: match Service rule <br/> DNAT → 10.244.1.5:8080 <br/> (randomly selected backend)
IPTables->>Backend: packet to 10.244.1.5:8080
Backend-->>Pod: response (src IP = Pod IP)

Each kube-proxy watches Services and EndpointSlices from the API Server.

flowchart LR
subgraph Node A
KP1[kube-proxy]
P1[Pods]
end

    subgraph Node B
        KP2[kube-proxy]
        P2[Pods]
    end

    subgraph Node C
        KP3[kube-proxy]
        P3[Pods]
    end

    API[API Server]

    API --> KP1
    API --> KP2
    API --> KP3

How Client talk to a POD

sequenceDiagram
participant Client
participant kubeproxy as kube-proxy
participant Pod

    Client->>kubeproxy: Service IP
    kubeproxy->>Pod: Select Backend Pod
    Pod-->>Client: Response

Kube-proxy modes (a real-world gotcha)

At very high connection rates (> 10,000 Services), iptables rule chaining becomes slow.

IPVS mode (Layer-4 load balancing in the kernel) or Cilium eBPF replaces iptables for high-scale clusters.

How a Service IP actually routes to a Pod depends on kube-proxy's mode:

1. iptables (default)

Fine for small clusters, but rule evaluation is O(n) in the number of Services, so it degrades badly at thousands of Services.

2. IPVS(IP Virtual Server)

Hash-based, scales far better for large Service counts.

3. nftables

Newer, addresses the iptables scaling problem natively.

4. eBPF (Cilium)

Replaces kube-proxy entirely and routes in-kernel increasingly the choice for large or performance-sensitive clusters.

Kube-proxy modes Comparison

Feature	iptables	IPVS	nftables	eBPF (Cilium)
Packet forwarding	Kernel iptables	Linux IPVS	Linux nftables	Kernel eBPF
kube-proxy required	✅	✅	✅	❌ (replacement mode)
Scalability	Good	Very Good	Very Good	Excellent
Service update speed	Slower with many rules	Fast	Fast	Very Fast
Load balancing	Basic	Multiple algorithms	Basic	Advanced
Kernel dependency	Standard	IPVS modules	Modern kernel	Modern kernel + eBPF
Best for	Small/medium clusters	Large clusters	Modern Linux	Very large, high-performance clusters

CNI vs Kube Proxy

CNI	kube-proxy
Pod networking	Service networking
Gives Pods IPs	Routes Service traffic
Creates veth pairs	Creates iptables/IPVS/eBPF rules
Pod-to-Pod communication	Service-to-Pod communication
Works during Pod creation	Watches Services continuously

---

Ingress / Gateway API 🚪

HTTP Routing into the Cluster

Ingress maps external HTTP(S) URLs to Services — host/path routing, TLS termination — so clients hit a meaningful hostname instead of a raw IP and port.

The Gateway API is the newer, more expressive successor that's gradually replacing Ingress for serious setups.

Services of type LoadBalancer give each Service its own cloud load balancer (expensive for dozens of Services). Ingress puts one load balancer in front and routes to many Services by hostname or URL path.

flowchart TD

    Internet-->LB["Cloud Load Balancer <br/> (one public IP)"]
    LB-->IngressCtrl["Ingress Controller <br/> (NGINX / Traefik / AWS ALB)"]
    
    IngressCtrl-->|"api.example.com"| APISvc["api Service"]
    IngressCtrl-->|"example.com/dashboard"| DashSvc["dashboard Service"]
    IngressCtrl-->|"TLS termination"| TLS["TLS cert <br/> (cert-manager)"]

Ingress Config Example:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: app-ingress
  annotations:
    nginx.ingress.kubernetes.io/rewrite-target: /
spec:
  ingressClassName: nginx
  tls:
  - hosts:
    - api.example.com
    secretName: api-tls-cert
  rules:
  - host: api.example.com
    http:
      paths:
      - path: /v1
        pathType: Prefix
        backend:
          service:
            name: api-svc
            port:
              number: 80
  - host: example.com
    http:
      paths:
      - path: /dashboard
        pathType: Prefix
        backend:
          service:
            name: dashboard-svc
            port:
              number: 3000

The Ingress Controller (a Pod running NGINX, Traefik, or a cloud-native controller) watches Ingress objects and configures its load balancer rules dynamically. Kubernetes defines the Ingress API but does not ship a controller — you install one separately.

kube-proxy vs Ingress

kube-proxy	Ingress Controller
Layer 4 (TCP/UDP)	Layer 7 (HTTP/HTTPS)
Routes Service traffic	Routes HTTP requests
Uses iptables/IPVS/nftables	NGINX, Envoy, HAProxy, etc.
Built into Kubernetes	Optional add-on

---

NetworkPolicies

NetworkPolicy adds firewall rules at the Pod level:

By default, all Pods can reach all other Pods (the flat model has no built-in isolation).

Firewall Rules for Pods

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-only-frontend
  namespace: backend
spec:
  podSelector:
    matchLabels:
      app: api             # applies to "api" Pods
  policyTypes:
  - Ingress
  - Egress
  ingress:
  - from:
    - podSelector:
        matchLabels:
          app: frontend    # only frontend Pods can reach api
    ports:
    - protocol: TCP
      port: 8080
  egress:
  - to:
    - podSelector:
        matchLabels:
          app: postgres    # api can only talk to postgres
    ports:
    - protocol: TCP
      port: 5432

NetworkPolicies are additive — you can only allow, not explicitly deny.

A common pattern is a default-deny policy that blocks all ingress/egress, then explicit allow policies for each allowed connection:

# Default deny all ingress in namespace
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny-ingress
spec:
  podSelector: {}      # matches all Pods in namespace
  policyTypes:
  - Ingress
  # no ingress rules → deny all

NetworkPolicy enforcement requires a CNI plugin that supports it — Calico and Cilium do; Flannel alone does not.

---

CoreDNS — Service Discovery

Pods don't hardcode ClusterIPs. They use DNS names that resolve to Service ClusterIPs.

Every Service gets a DNS name: <service-name>.<namespace>.svc.cluster.local

flowchart TD

    Pod["Pod <br/> nslookup web-svc"]
    Pod-->CoreDNS["CoreDNS <br/> (kube-dns Service)"]
    CoreDNS-->|"A record → ClusterIP"| IP["10.96.45.12"]
    IP-->|"routed by kube-proxy"| Backend["Backend Pod"]

# Inside any Pod:
curl http://web-svc                          # same namespace
curl http://web-svc.default                 # explicit namespace
curl http://web-svc.default.svc.cluster.local  # fully qualified

# Headless Service — returns individual Pod IPs:
# postgres-0.postgres-headless.default.svc.cluster.local → 10.244.1.5
# postgres-1.postgres-headless.default.svc.cluster.local → 10.244.2.3

CoreDNS is a Deployment (typically 2 replicas) that watches the API Server for Service and Pod changes and updates DNS records automatically.

---

Key Takeaways

Concept	Purpose
Flat network model	Every Pod has a routable IP; no NAT between Pods
CNI plugin	Implements pod networking (veth pairs, IP assignment, cross-node routing)
Service	Stable virtual IP + DNS name for a set of Pods; survives Pod restarts
kube-proxy	Programs iptables/IPVS rules that implement Service virtual IPs
CoreDNS	Resolves <svc>.<ns>.svc.cluster.local → ClusterIP
Ingress	HTTP routing: one load balancer, many Services, by hostname/path
NetworkPolicy	Pod-level firewall; additive allow rules; requires CNI support
For GPU training	Replace overlay CNI with RDMA over InfiniBand — different data path, not a replacement for the cluster CNI