Kubernetes Networking: Pods, Services, Ingress, and CNI

Kubernetes networking is built on one rule that everything else follows from:

Every Pod gets its own IP address. Pods can reach any other Pod directly, without NAT.

This sounds simple. Implementing it across hundreds of nodes, cloud VPCs, and heterogeneous hardware is what the networking stack is for.

---

The Kubernetes Networking Model

Three requirements the model must satisfy:

Requirement	Meaning
Pod-to-Pod	Any Pod can reach any other Pod using its IP, even on a different node
No NAT	The IP a Pod sees as its own is the same IP other Pods use to reach it
Node-to-Pod	Nodes can reach any Pod directly

This is called the flat network model — from any Pod's perspective, every other Pod is reachable at its IP with no address translation.

flowchart LR

subgraph Node1["Node 1"]
  PodA["Pod A\n10.244.1.5"]
  PodB["Pod B\n10.244.1.6"]
end

subgraph Node2["Node 2"]
  PodC["Pod C\n10.244.2.3"]
end

PodA

-->|"direct IP\nno NAT"| PodC

PodA

-->|"same node\nloopback bridge"| PodB

Pod IPs come from a cluster-wide Pod CIDR (e.g., 10.244.0.0/16). Each node gets a slice of that range (e.g., 10.244.1.0/24 for Node 1, 10.244.2.0/24 for Node 2).

---

CNI — Container Network Interface

Kubernetes does not implement networking itself. It delegates to a CNI plugin via the Container Network Interface spec.

flowchart LR

Kubelet

-->|"calls CNI"| CNI["CNI Plugin\n(Flannel / Calico / Cilium)"]

CNI

-->|"sets up"| VethPair["veth pair\n(Pod ↔ host bridge)"]

CNI

-->|"routes packets"| CrossNode["Cross-node routing\n(overlay or BGP)"]

When a Pod is created, the Kubelet calls the CNI plugin to:

1. Create a network namespace for the Pod
2. Create a veth pair — one end inside the Pod (eth0), one end on the host bridge
3. Assign the Pod's IP from the node's slice of Pod CIDR
4. Set up routing so packets to other nodes' Pod CIDRs reach the right node

Common CNI Plugins

Plugin	Mechanism	Key strength
Flannel	VXLAN overlay (UDP encapsulation)	Simple, works anywhere
Calico	BGP routing (no overlay)	High performance, NetworkPolicy support
Cilium	eBPF (kernel-level packet processing)	Observability, L7 policy, high performance
Weave	Encrypted overlay	Easy multi-cluster

Same-Node vs Cross-Node Communication

flowchart TB

subgraph "Same node"
  PodA["Pod A\n10.244.1.5"]
  -->|"veth → bridge"| Bridge["cbr0 bridge\n(Linux bridge)"]
  Bridge
  -->|"veth → Pod"| PodB["Pod B\n10.244.1.6"]
end

subgraph "Cross-node (Flannel VXLAN)"
  PodC["Pod C\n10.244.1.7"]
  -->|"veth → bridge → VXLAN"| Tunnel["UDP tunnel\n(encapsulated)"]
  Tunnel
  -->|"decapsulate"| PodD["Pod D\n10.244.2.3\non Node 2"]
end

Same-node: packets go through the Linux bridge — no overhead, effectively loopback speed.

Cross-node with VXLAN: packets are encapsulated in UDP, sent to the destination node's IP, then decapsulated. This is the "overlay" — it adds ~10–15% CPU overhead and latency, which is why GPU training workloads replace it with InfiniBand (see Network Operator post).

---

Services — Stable Endpoints for Ephemeral Pods

Pod IPs are ephemeral — when a Pod is replaced (crash, rolling update), it gets a new IP.

A Service provides a stable virtual IP (ClusterIP) that load-balances across all healthy Pods matching a label selector.

flowchart LR

Client["Client Pod"]

-->ClusterIP["Service ClusterIP\n10.96.45.12\n(stable forever)"]

ClusterIP

-->|"kube-proxy\niptables rules"| Pod1["Pod 10.244.1.5"]

ClusterIP

-->Pod2["Pod 10.244.1.7"]

ClusterIP

-->Pod3["Pod 10.244.2.3"]

Service Types

Type	Use case	How it works
ClusterIP	In-cluster communication	Virtual IP only reachable inside the cluster
NodePort	Basic external access	Opens a port (30000–32767) on every node; traffic forwarded to Service
LoadBalancer	Production external access	Cloud provider provisions a load balancer with a public IP
ExternalName	DNS alias to external service	Returns a CNAME — no proxying
Headless	StatefulSets, direct Pod access	No ClusterIP; DNS returns individual Pod IPs

# ClusterIP (default)
apiVersion: v1
kind: Service
metadata:
  name: web-svc
spec:
  selector:
    app: web          # routes to all Pods with this label
  ports:
  - port: 80
    targetPort: 8080

---
# LoadBalancer — gets a cloud public IP
apiVersion: v1
kind: Service
metadata:
  name: web-lb
spec:
  type: LoadBalancer
  selector:
    app: web
  ports:
  - port: 80
    targetPort: 8080

---
# Headless — for StatefulSets (stable DNS per Pod)
apiVersion: v1
kind: Service
metadata:
  name: postgres-headless
spec:
  clusterIP: None     # ← headless
  selector:
    app: postgres
  ports:
  - port: 5432

kube-proxy — How Service IPs Work

Services have no process listening on the ClusterIP. The IP is virtual — kube-proxy programs iptables rules on every node that intercept packets to the ClusterIP and DNAT them to a real Pod IP.

sequenceDiagram

participant Pod as Client Pod
participant IPTables as iptables (on node)
participant Backend as Backend Pod

Pod->>IPTables: packet to 10.96.45.12:80
IPTables->>IPTables: match Service rule\nDNAT → 10.244.1.5:8080\n(randomly selected backend)
IPTables->>Backend: packet to 10.244.1.5:8080
Backend-->>Pod: response (src IP = Pod IP)

At very high connection rates (> 10,000 Services), iptables rule chaining becomes slow. IPVS mode (Layer-4 load balancing in the kernel) or Cilium eBPF replaces iptables for high-scale clusters.

---

CoreDNS — Service Discovery

Pods don't hardcode ClusterIPs. They use DNS names that resolve to Service ClusterIPs.

Every Service gets a DNS name: <service-name>.<namespace>.svc.cluster.local

flowchart LR

Pod["Pod\nnslookup web-svc"]

-->CoreDNS["CoreDNS\n(kube-dns Service)"]

CoreDNS

-->|"A record → ClusterIP"| IP["10.96.45.12"]

IP

-->|"routed by kube-proxy"| Backend["Backend Pod"]

# Inside any Pod:
curl http://web-svc                          # same namespace
curl http://web-svc.default                 # explicit namespace
curl http://web-svc.default.svc.cluster.local  # fully qualified

# Headless Service — returns individual Pod IPs:
# postgres-0.postgres-headless.default.svc.cluster.local → 10.244.1.5
# postgres-1.postgres-headless.default.svc.cluster.local → 10.244.2.3

CoreDNS is a Deployment (typically 2 replicas) that watches the API Server for Service and Pod changes and updates DNS records automatically.

---

Ingress — HTTP Routing into the Cluster

Services of type LoadBalancer give each Service its own cloud load balancer (expensive for dozens of Services). Ingress puts one load balancer in front and routes to many Services by hostname or URL path.

flowchart LR

Internet

-->LB["Cloud Load Balancer\n(one public IP)"]

LB

-->IngressCtrl["Ingress Controller\n(NGINX / Traefik / AWS ALB)"]

IngressCtrl

-->|"api.example.com"| APISvc["api Service"]

IngressCtrl

-->|"example.com/dashboard"| DashSvc["dashboard Service"]

IngressCtrl

-->|"TLS termination"| TLS["TLS cert\n(cert-manager)"]

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: app-ingress
  annotations:
    nginx.ingress.kubernetes.io/rewrite-target: /
spec:
  ingressClassName: nginx
  tls:
  - hosts:
    - api.example.com
    secretName: api-tls-cert
  rules:
  - host: api.example.com
    http:
      paths:
      - path: /v1
        pathType: Prefix
        backend:
          service:
            name: api-svc
            port:
              number: 80
  - host: example.com
    http:
      paths:
      - path: /dashboard
        pathType: Prefix
        backend:
          service:
            name: dashboard-svc
            port:
              number: 3000

The Ingress Controller (a Pod running NGINX, Traefik, or a cloud-native controller) watches Ingress objects and configures its load balancer rules dynamically. Kubernetes defines the Ingress API but does not ship a controller — you install one separately.

---

NetworkPolicies — Firewall Rules for Pods

By default, all Pods can reach all other Pods (the flat model has no built-in isolation).

NetworkPolicy adds firewall rules at the Pod level:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-only-frontend
  namespace: backend
spec:
  podSelector:
    matchLabels:
      app: api             # applies to "api" Pods
  policyTypes:
  - Ingress
  - Egress
  ingress:
  - from:
    - podSelector:
        matchLabels:
          app: frontend    # only frontend Pods can reach api
    ports:
    - protocol: TCP
      port: 8080
  egress:
  - to:
    - podSelector:
        matchLabels:
          app: postgres    # api can only talk to postgres
    ports:
    - protocol: TCP
      port: 5432

NetworkPolicies are additive — you can only allow, not explicitly deny. A common pattern is a default-deny policy that blocks all ingress/egress, then explicit allow policies for each allowed connection:

# Default deny all ingress in namespace
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny-ingress
spec:
  podSelector: {}      # matches all Pods in namespace
  policyTypes:
  - Ingress
  # no ingress rules → deny all

NetworkPolicy enforcement requires a CNI plugin that supports it — Calico and Cilium do; Flannel alone does not.

---

The Full Request Flow

A browser request hitting an application running in Kubernetes:

sequenceDiagram

participant Browser
participant LB as Cloud Load Balancer
participant NGINX as Ingress Controller Pod
participant SvcIP as kube-proxy (iptables)
participant AppPod as App Pod

Browser->>LB: GET https://api.example.com/v1/users
LB->>NGINX: TCP to NodePort 443
NGINX->>NGINX: TLS termination\nroute by hostname + path
NGINX->>SvcIP: HTTP to api-svc ClusterIP:80
SvcIP->>AppPod: DNAT → 10.244.2.3:8080
AppPod-->>Browser: 200 OK (response path)

---

Why This Matters for GPU Workloads

The CNI overlay network (VXLAN) is fine for API traffic — a few MB/s per request.

For distributed training, AllReduce between 8 DGX nodes transfers hundreds of GB/s of gradient data. The VXLAN overhead and CPU involvement make overlay networking a hard bottleneck.

The solution (covered in the Network Operator post):

• A second NIC on each GPU node — an InfiniBand HCA
• RDMA — bypasses the kernel and CPU entirely
• Multus CNI — attaches both the cluster CNI (for API traffic) and the RDMA NIC (for training traffic) to the same Pod

The cluster CNI (Calico/Cilium) handles all Kubernetes control plane and application traffic. InfiniBand handles all GPU-to-GPU training traffic. They are completely separate data paths.

---

Key Takeaways

Concept	Purpose
Flat network model	Every Pod has a routable IP; no NAT between Pods
CNI plugin	Implements pod networking (veth pairs, IP assignment, cross-node routing)
Service	Stable virtual IP + DNS name for a set of Pods; survives Pod restarts
kube-proxy	Programs iptables/IPVS rules that implement Service virtual IPs
CoreDNS	Resolves <svc>.<ns>.svc.cluster.local → ClusterIP
Ingress	HTTP routing: one load balancer, many Services, by hostname/path
NetworkPolicy	Pod-level firewall; additive allow rules; requires CNI support
For GPU training	Replace overlay CNI with RDMA over InfiniBand — different data path, not a replacement for the cluster CNI