Kubernetes Topology Manager: NUMA-Aware GPU Scheduling

Two pods.

Same node.

Same GPU.

Same code.

Pod A: GPUDirect RDMA latency ~1 μs.

Pod B: GPUDirect RDMA latency ~3 μs.

Same hardware. Three times slower.

The difference: Pod B's CPU threads are on a different NUMA node than its GPU and NIC.

Every memory access, every kernel launch, every RDMA transfer crosses a slow inter-processor link instead of staying local.

This is NUMA misalignment — and Kubernetes enables it by default.

The Topology Manager is the kubelet component that prevents it.

---

What Is NUMA?

NUMA stands for Non-Uniform Memory Access.

Modern servers have multiple CPU sockets.

Each socket has its own memory channels.

flowchart LR

subgraph NUMA0["NUMA Node 0"]
CPU0["CPU Socket 0\n(56 cores)"]
MEM0["Memory\nChannels 0–7\n512 GB"]
CPU0 <--> MEM0
end

subgraph NUMA1["NUMA Node 1"]
CPU1["CPU Socket 1\n(56 cores)"]
MEM1["Memory\nChannels 8–15\n512 GB"]
CPU1 <--> MEM1
end

CPU0 <-->|"QPI / UPI\n~100 ns penalty"| CPU1

A CPU can access memory on its own NUMA node fast.

It can also access memory on the other NUMA node — but it must cross the inter-processor link (Intel UPI or AMD Infinity Fabric), adding ~100 ns latency per access.

For most workloads, this is negligible.

For GPU training, where hundreds of thousands of small operations happen per second, it accumulates.

---

DGX NUMA Topology

A DGX H100 has two NUMA nodes — one per CPU socket.

Each NUMA node owns:

• 4 H100 GPUs (attached via PCIe lanes from that socket)
• 4 ConnectX-7 InfiniBand NICs (attached via PCIe lanes from that socket)
• 512 GB of system memory

flowchart TB

subgraph NUMA0["NUMA Node 0 (Socket 0)"]
CPU0["CPU\n56 cores"]
MEM0["512 GB RAM"]
GPU0["GPU 0\nH100"]
GPU1["GPU 1\nH100"]
GPU2["GPU 2\nH100"]
GPU3["GPU 3\nH100"]
NIC0["NIC 0\nConnectX-7"]
NIC1["NIC 1\nConnectX-7"]
NIC2["NIC 2\nConnectX-7"]
NIC3["NIC 3\nConnectX-7"]
end

subgraph NUMA1["NUMA Node 1 (Socket 1)"]
CPU1["CPU\n56 cores"]
MEM1["512 GB RAM"]
GPU4["GPU 4\nH100"]
GPU5["GPU 5\nH100"]
GPU6["GPU 6\nH100"]
GPU7["GPU 7\nH100"]
NIC4["NIC 4\nConnectX-7"]
NIC5["NIC 5\nConnectX-7"]
NIC6["NIC 6\nConnectX-7"]
NIC7["NIC 7\nConnectX-7"]
end

NUMA0 <-->|"UPI"| NUMA1

For peak performance, a pod's CPU threads, GPU, and NIC must all come from the same NUMA node.

---

The Problem: Kubernetes Doesn't Know About NUMA

Without the Topology Manager, each kubelet resource manager works independently.

flowchart LR

DevicePlugin["Device Plugin\nassigns GPU 0 (NUMA 0)"]

CPUManager["CPU Manager\nassigns CPUs from NUMA 1"]

MemManager["Memory Manager\nallocates from NUMA 1"]

GPU 0 is attached to NUMA 0.

CPU threads are pinned to NUMA 1.

Every GPU kernel launch — CPU writes command to GPU — crosses UPI.

Every RDMA receive — NIC on NUMA 0 wants to write to system memory — but memory is on NUMA 1.

The resources are scattered across the topology.

---

The Three Managers That Must Coordinate

Kubelet has three resource managers that each know their piece of the topology:

Manager	Controls	NUMA-aware?
CPU Manager	Which CPU cores a container gets	Yes
Memory Manager	Which NUMA node memory pages come from	Yes
Device Manager	Which GPU / NIC device gets allocated	Via Device Plugin hints

Without the Topology Manager, they don't talk to each other.

With the Topology Manager, they all register hints — then the Topology Manager finds the allocation that satisfies all three simultaneously.

---

How the Topology Manager Works

The Topology Manager acts as a hint broker.

Before allocating any resource to a pod, it collects hints from every registered provider.

sequenceDiagram

participant KubeletAdmit as Kubelet Admit
participant TM as Topology Manager
participant CPU as CPU Manager
participant Mem as Memory Manager
participant Dev as Device Manager

KubeletAdmit->>TM: New pod needs GPU + CPU + memory
TM->>CPU: What NUMA nodes can satisfy this CPU request?
CPU-->>TM: [NUMA 0, NUMA 1] or [NUMA 0 only] or [NUMA 1 only]
TM->>Mem: What NUMA nodes can satisfy this memory request?
Mem-->>TM: [NUMA 0]
TM->>Dev: What NUMA nodes can satisfy this GPU request?
Dev-->>TM: [NUMA 0] (GPU 0 is on NUMA 0)
TM->>TM: Intersect hints → NUMA 0
TM-->>KubeletAdmit: Admit pod, all resources on NUMA 0

Each hint is a bitmask of NUMA nodes that can satisfy the request.

The Topology Manager intersects all hints to find the preferred (or required) NUMA node set.

---

The Four Policies

The Topology Manager has four policies that determine what happens when hints don't align.

---

None (default)

flowchart LR

Pod["Pod requests\nGPU + CPU"]

-->TM["Topology Manager\n(None policy)"]

TM

-->|"No coordination"| Assign["Assign resources\nfrom anywhere"]

No hint collection.

No alignment.

Resources assigned independently by each manager.

This is the Kubernetes default and the worst option for DGX performance.

---

BestEffort

flowchart TD

Pod["Pod requests\nGPU + CPU"]

-->Collect["Collect hints\nfrom all managers"]

Collect

-->Intersect["Find preferred\nNUMA node"]

Intersect

-->|"Alignment found"| AlignedAlloc["Allocate all\nresources from NUMA 0"]

Intersect

-->|"No alignment possible"| AnyAlloc["Allocate anyway\n(misaligned)"]

Tries to align.

If it can't — admits the pod anyway with a misaligned allocation.

Pod runs, but performance may be degraded.

Useful for development clusters where you don't want pods to fail.

---

Restricted

flowchart TD

Pod["Pod requests\nGPU + CPU"]

-->Collect["Collect hints"]

Collect

-->Intersect["Find preferred\nNUMA node"]

Intersect

-->|"Alignment found"| AlignedAlloc["Allocate aligned"]

Intersect

-->|"No alignment possible"| Reject["Pod rejected\nTopologyAffinityError"]

If alignment is impossible — the pod fails to start.

You get a clear error, not a silently degraded pod.

Use this in production DGX clusters.

---

SingleNUMANode

The strictest policy.

All resources must come from exactly one NUMA node.

Not NUMA 0 and NUMA 1 together — just one.

flowchart TD

Pod["Pod requests\nGPU + 8 CPUs + memory"]

-->Check["Can one NUMA node\nsatisfy ALL requests?"]

Check

-->|"Yes — NUMA 0 has\nGPU 0, enough CPUs,\nenough memory"| Admit["Admit pod\nfully local"]

Check

-->|"No — would need\nboth NUMA nodes"| Reject["Reject pod\nTopologyAffinityError"]

This is the policy to use for DGX training pods.

It guarantees:

• CPU → NUMA 0
• GPU → NUMA 0
• Memory → NUMA 0
• NIC (via SR-IOV hint) → NUMA 0

Every memory access, kernel launch, and RDMA transfer stays local.

---

The CPU Manager

The CPU Manager is the prerequisite for Topology Manager to work for CPU resources.

By default, Kubernetes assigns CPU shares (cgroups) — not specific cores.

Your process can run on any core at any time, migrating between NUMA nodes.

flowchart LR

Process["Container Process"]

-->|"time slice 0"| Core0["Core 0\nNUMA 0"]

Process

-->|"time slice 1"| Core28["Core 28\nNUMA 1"]

Process

-->|"time slice 2"| Core7["Core 7\nNUMA 0"]

The CPU Manager static policy pins specific cores to a container:

flowchart LR

Process["Container Process"]

-->|"always"| Cores["Cores 0–7\nNUMA 0 only"]

No migration. No cross-NUMA cache invalidation.

To use Topology Manager for CPUs, the container must request integer CPU limits (Guaranteed QoS):

resources:
  requests:
    cpu: "8"
  limits:
    cpu: "8"    # integer → CPU Manager pins to specific cores

Non-integer CPU limits (e.g., cpu: "0.5") are handled by cgroups only — the Topology Manager cannot pin them.

---

The Memory Manager

Similarly, memory pages normally come from wherever the kernel decides.

The Memory Manager (with Static policy) allocates hugepages from a specific NUMA node.

resources:
  requests:
    memory: "16Gi"
    hugepages-1Gi: "4Gi"    # ← enables Memory Manager coordination
  limits:
    memory: "16Gi"
    hugepages-1Gi: "4Gi"

With hugepages requested, the Memory Manager provides a NUMA hint and the Topology Manager includes it in the intersection.

---

Configuring the Topology Manager

All three components are configured via kubelet flags.

# /etc/kubernetes/kubelet-config.yaml

apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration

# CPU Manager
cpuManagerPolicy: static
cpuManagerPolicyOptions:
  full-pcpus-only: "true"    # allocate whole physical cores, not hyperthreads

# Memory Manager
memoryManagerPolicy: Static

# Topology Manager
topologyManagerPolicy: single-numa-node
topologyManagerScope: pod    # coordinate all containers in a pod together

Scope options:

Scope	Meaning
container	Each container in a pod is aligned independently
pod	All containers in a pod must share the same NUMA topology

For training pods, use pod scope — the master and worker containers in a pod must all be on the same NUMA node.

---

Verifying NUMA Alignment

After a pod starts, verify it got an aligned allocation.

Check which cores were assigned:

# Find the container's cgroup
CONTAINER_ID=$(kubectl get pod llama-trainer -o jsonpath='{.status.containerStatuses[0].containerID}' | cut -d/ -f3)

# Read the CPU set
cat /sys/fs/cgroup/cpuset/kubepods/pod<uid>/<container-id>/cpuset.cpus
# 0-7,56-63     ← cores 0–7 and HT pairs 56–63 — all on NUMA 0 ✅
# 0-7,28-35    ← cores from both sockets — NUMA misaligned ❌

Check GPU NUMA affinity:

nvidia-smi topo -m
# Output:
#         GPU0  GPU1  GPU2  GPU3  GPU4  GPU5  GPU6  GPU7  NIC0  NIC1  NIC2  NIC3  NIC4  NIC5
# GPU0     X    NV18  NV18  NV18  SYS   SYS   SYS   SYS   PIX   PIX   PIX   PIX   SYS   SYS
# GPU4    SYS   SYS   SYS   SYS   X     NV18  NV18  NV18  SYS   SYS   SYS   SYS   PIX   PIX
#
# PIX = same PCIe switch (same NUMA) ← fast path ✅
# SYS = cross NUMA ← slow path ❌

Check kubelet logs for topology decisions:

journalctl -u kubelet | grep -i topology
# kubelet[1234]: topology_manager.go:307] "Topology Admit Handler"
# kubelet[1234]: topology_manager.go:332] "Topology affinity for pod"
#   affinityAssignment="{NUMANodeAffinity:{Bits:0x01}}"
#   ← 0x01 = NUMA node 0 only ✅

Check if a pod failed due to topology:

kubectl describe pod llama-trainer
# Events:
#   Warning  TopologyAffinityError  kubelet  Resources cannot be allocated
#             with Topology locality for pod

This means the node doesn't have enough aligned resources — try a different node or reduce the request.

---

What Alignment Actually Changes

A container that requested GPU 0 (NUMA 0), 8 CPUs, and a ConnectX-7 NIC:

Without Topology Manager (None policy):

flowchart TB

subgraph NUMA0["NUMA Node 0"]
GPU0["GPU 0 ✅"]
NIC0["NIC 0 ✅"]
end

subgraph NUMA1["NUMA Node 1"]
CPU["CPU threads ❌\n(wrong NUMA)"]
MEM["Memory pages ❌\n(wrong NUMA)"]
end

GPU0 <-->|"Cross-NUMA UPI\n+100 ns per access"| CPU
NIC0 <-->|"Cross-NUMA UPI\n+100 ns per DMA"| MEM

With Topology Manager (SingleNUMANode policy):

flowchart TB

subgraph NUMA0["NUMA Node 0 — everything local"]
GPU0["GPU 0"]
NIC0["NIC 0"]
CPU["CPU threads"]
MEM["Memory pages"]

GPU0 <-->|"local PCIe\n<1 μs"| NIC0
GPU0 <-->|"local bus"| CPU
NIC0 <-->|"local memory\ncontroller"| MEM
end

The performance difference compounds across an entire training run:

Operation	Misaligned	Aligned
GPU kernel launch	+100 ns cross-NUMA	Local — baseline
RDMA receive DMA	Cross-NUMA write	Local write
GPU memory copy	Possible cross-NUMA	Local
Effective RDMA BW	~35 GB/s	~50 GB/s

For a 70B model training across 8 nodes, that bandwidth gap translates directly to AllReduce time.

---

NUMA Topology Tools

Inspect the node's NUMA topology before setting policy:

# Show NUMA node layout
numactl --hardware
# available: 2 nodes (0-1)
# node 0 cpus: 0 1 2 3 ... 55 112 113 ... 167
# node 0 size: 524288 MB
# node 1 cpus: 56 57 58 ... 111 168 169 ... 223
# node 1 size: 524288 MB
# node distances:
# node   0   1
#   0:  10  21    ← local=10, remote=21 (2.1× penalty)

# Show GPU-NIC affinity matrix
nvidia-smi topo -m

# Show which NUMA node each GPU is on
for i in 0 1 2 3 4 5 6 7; do
  echo -n "GPU $i → NUMA "
  cat /sys/bus/pci/devices/$(nvidia-smi --query-gpu=pci.bus_id --format=noheader -i $i | awk '{print tolower($0)}')/numa_node
done
# GPU 0 → NUMA 0
# GPU 1 → NUMA 0
# GPU 2 → NUMA 0
# GPU 3 → NUMA 0
# GPU 4 → NUMA 1
# GPU 5 → NUMA 1
# GPU 6 → NUMA 1
# GPU 7 → NUMA 1

# Show which NUMA node each NIC is on
cat /sys/class/infiniband/mlx5_0/device/numa_node
# 0
cat /sys/class/infiniband/mlx5_4/device/numa_node
# 1

---

Policy Selection Guide

Cluster Type	Recommended Policy	Reason
Development / shared	BestEffort	Pods never fail; alignment attempted but not enforced
Production training (DGX)	Restricted or SingleNUMANode	Fail fast on misalignment rather than run degraded
Inference serving (mixed sizes)	BestEffort	Smaller pods may not fill a single NUMA node
Benchmark / perf testing	SingleNUMANode	Guarantees reproducible, fully local allocation

---

Complete Kubelet Configuration for DGX

apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration

# Pin CPUs to containers (prerequisite for Topology Manager CPU hints)
cpuManagerPolicy: static
cpuManagerPolicyOptions:
  full-pcpus-only: "true"

# Reserve system CPUs (don't give all cores to pods)
systemReserved:
  cpu: "4"
  memory: "8Gi"
kubeReserved:
  cpu: "2"
  memory: "4Gi"

# Hugepages NUMA-aware allocation
memoryManagerPolicy: Static
reservedMemory:
- numaNode: 0
  limits:
    hugepages-1Gi: "8Gi"
- numaNode: 1
  limits:
    hugepages-1Gi: "8Gi"

# Topology Manager — all resources on one NUMA node
topologyManagerPolicy: single-numa-node
topologyManagerScope: pod

---

Key Takeaways

Component	What It Does	Why It Matters
NUMA	Memory access speed depends on proximity to CPU socket	Cross-NUMA adds ~100 ns per access — multiplies across millions of ops
CPU Manager (static)	Pins container to specific cores on one NUMA node	Eliminates CPU migration and cross-NUMA cache misses
Memory Manager (static)	Allocates hugepages from a specific NUMA node	Ensures memory-controller locality for GPU DMA
Device Plugin hints	Reports which NUMA node each GPU/NIC is attached to	Feeds topology information into the hint broker
Topology Manager	Intersects all hints, finds the NUMA node that satisfies everyone	The coordination layer — nothing works without this
SingleNUMANode policy	Requires all resources from one NUMA node or pod fails	Preferred for DGX — guarantees fully local allocation
Scope: pod	Aligns all containers in a pod together	Training pods need master + worker on the same NUMA node

The Topology Manager is the difference between a GPU cluster that happens to use InfiniBand and a DGX cluster that achieves the bandwidth InfiniBand was designed to deliver. Without it, Kubernetes quietly scatters your resources across NUMA nodes and no one notices until someone runs nvidia-smi topo -m.