NVIDIA Network Operator: InfiniBand, SR-IOV, RDMA, and Multus

Two clusters.

Both have 64 H100 GPUs.

Both run the same PyTorchJob.

Cluster A finishes training in 2 days.

Cluster B finishes in 20 days.

Same GPUs. Same code. Same model.

The only difference is networking.

This post explains why.

---

Training Is Not Compute-Bound

Most people assume more GPUs means faster training.

That's true — up to a point.

Beyond that point, the bottleneck shifts.

flowchart LR

Compute["GPU Compute\n(FLOPS)"]

-->|"Fast enough"| Comm["GPU Communication\n(gradient sync)"]

Comm

-->|"Becomes bottleneck"| Network["Network Bandwidth\nand Latency"]

In every distributed training step:

1. Each GPU computes gradients — takes milliseconds
2. Gradients must be synchronized across all GPUs — AllReduce
3. Training cannot continue until sync completes

If the network is slow, GPUs sit idle waiting for AllReduce.

The faster the network, the higher GPU utilization.

---

The Standard Kubernetes Network

By default, Kubernetes uses an overlay network (Flannel, Calico, Cilium, etc.).

Every packet between pods travels this path:

flowchart LR

GPU["GPU\n(source)"]

-->SystemMem["Copy to\nCPU memory"]

SystemMem

-->Kernel["Linux kernel\nnetwork stack"]

Kernel

-->veth["veth pair\n(virtual NIC)"]

veth

-->Bridge["Linux bridge\nor eBPF"]

Bridge

-->Encap["Encapsulation\n(VXLAN / GENEVE)"]

Encap

-->pNIC["Physical NIC"]

pNIC

-->Wire["Network"]

Every packet:

• Touches CPU and RAM multiple times
• Goes through the Linux kernel network stack
• Gets encapsulated (adding overhead)
• Competes with other pods for the physical NIC

For web services — perfectly fine.

For gradient AllReduce across 64 GPUs — devastating.

---

What DGX Networking Looks Like

A DGX H100 node has two separate networks:

flowchart TB

subgraph DGXNode["DGX H100 Node"]

subgraph GPUs["8× H100 GPUs"]
G1 & G2 & G3 & G4 & G5 & G6 & G7 & G8
end

NVSwitch["NVSwitch\n(intra-node, 900 GB/s)"]

subgraph NICs["8× ConnectX-7 NICs"]
N1 & N2 & N3 & N4 & N5 & N6 & N7 & N8
end

GPUs <-->|"NVLink"| NVSwitch

GPUs <-->|"GPUDirect RDMA\n(PCIe peer-to-peer)"| NICs

end

NICs

-->|"InfiniBand NDR\n400 Gb/s per port"| Fabric["InfiniBand Fabric"]

Link	Bandwidth	Protocol
NVSwitch (intra-node)	900 GB/s	NVLink
InfiniBand NDR (inter-node)	400 Gb/s per port × 8	RDMA

Each GPU has its own dedicated InfiniBand port.

No sharing. No contention.

---

The Four Technologies That Make DGX Fast

1. RDMA — Remote Direct Memory Access

Standard TCP/IP requires the CPU to copy data:

Application buffer → kernel socket buffer → NIC → wire

RDMA bypasses the kernel entirely.

sequenceDiagram

participant App as Application
participant CPU as CPU / Kernel
participant NIC as RDMA NIC
participant Remote as Remote NIC

App->>NIC: Post send (DMA descriptor)
Note over CPU: CPU not involved
NIC->>Remote: Send packet directly
Remote->>App: Data lands in app buffer
Note over CPU: CPU not notified until complete

What RDMA eliminates:

• CPU copy from user space to kernel space
• CPU copy from kernel space to NIC
• Interrupt processing for every packet
• Socket system calls

Results:

Metric	TCP/IP	RDMA
Latency	~50 μs	~1 μs
CPU overhead	High	Near zero
Bandwidth (100G link)	~8 GB/s effective	~12 GB/s effective

---

2. GPUDirect RDMA

Regular RDMA moves data between CPU memory on different hosts.

GPUDirect RDMA extends this to GPU memory.

Without GPUDirect RDMA:

flowchart LR

GPUA["GPU A\n(memory)"]

-->|"PCIe copy"| SysMemA["System RAM\n(Node A)"]

SysMemA

-->|"RDMA"| SysMemB["System RAM\n(Node B)"]

SysMemB

-->|"PCIe copy"| GPUB["GPU B\n(memory)"]

Two PCIe copies. CPU involved. Slow.

With GPUDirect RDMA:

flowchart LR

GPUA["GPU A\n(memory)"]

-->|"PCIe P2P\n(direct)"| NICA["ConnectX NIC\n(Node A)"]

NICA

-->|"InfiniBand RDMA"| NICB["ConnectX NIC\n(Node B)"]

NICB

-->|"PCIe P2P\n(direct)"| GPUB["GPU B\n(memory)"]

No CPU. No system RAM. Just GPU → wire → GPU.

Latency drops by 3–5×.

This is enabled by PCIe peer-to-peer (P2P) access between the GPU and the NIC, which NVIDIA ConnectX NICs support natively.

---

3. SR-IOV — Single Root I/O Virtualization

Standard Kubernetes: every pod shares the physical NIC through a virtual switch.

flowchart TB

Pod1 & Pod2 & Pod3

-->Bridge["Linux Bridge /\nOVS / eBPF"]

Bridge

-->pNIC["Physical NIC\n(shared)"]

Every packet goes through software switching — overhead, latency, CPU.

SR-IOV splits one physical NIC (PF — Physical Function) into many virtual NICs (VFs — Virtual Functions).

flowchart TB

pNIC["Physical NIC\n(Physical Function)"]

-->VF1["Virtual Function 1"]

pNIC

-->VF2["Virtual Function 2"]

pNIC

-->VF3["Virtual Function 3"]

VF1

-->Pod1["Pod 1\n(direct hardware access)"]

VF2

-->Pod2["Pod 2\n(direct hardware access)"]

VF3

-->Pod3["Pod 3\n(direct hardware access)"]

Each pod gets a real NIC from the OS perspective.

No bridge. No veth pair. No iptables.

The pod talks to hardware directly — same as a bare-metal process.

Combined with RDMA, this means the pod's GPU can do GPUDirect RDMA through its own dedicated VF.

---

4. Multus CNI — Multiple Network Interfaces per Pod

Kubernetes normally gives each pod one network interface (eth0).

This is fine for web traffic.

But training pods need two:

Interface	Traffic	Network
eth0	Cluster management (kubectl, services, DNS)	Overlay network
net1	Training gradient traffic	InfiniBand / SR-IOV

Multus is a CNI (Container Network Interface) plugin that chains multiple CNI plugins together, giving each pod multiple interfaces.

flowchart LR

Pod["Training Pod"]

-->eth0["eth0\n(Calico / Flannel)\nCluster traffic"]

Pod

-->net1["net1\n(SR-IOV VF)\nRDMA gradient traffic"]

eth0

-->Overlay["Overlay Network"]

net1

-->IB["InfiniBand Fabric\n(GPUDirect RDMA)"]

Separation of concerns:

• Management traffic stays on the slow path (no problem — it's small)
• Gradient traffic takes the fast path (RDMA, no kernel involvement)

---

The NVIDIA Network Operator

Installing all of the above manually across hundreds of DGX nodes is error-prone and painful.

The NVIDIA Network Operator is a Kubernetes Operator that automates the entire stack.

flowchart TB

NetworkOperator["NVIDIA Network Operator"]

-->OFED["Mellanox OFED\n(InfiniBand drivers)"]

NetworkOperator

-->SRIOV["SR-IOV Network Operator\n+ Device Plugin"]

NetworkOperator

-->RDMA["RDMA Shared\nDevice Plugin"]

NetworkOperator

-->Multus["Multus CNI"]

NetworkOperator

-->IPAM["Whereabouts IPAM\n(IP management)"]

NetworkOperator

-->NFD["Node Feature\nDiscovery"]

It introduces one top-level CRD: NicClusterPolicy.

You declare what you want. The operator reconciles the cluster to match.

---

NicClusterPolicy

The single configuration object that drives the entire operator.

apiVersion: mellanox.com/v1alpha1
kind: NicClusterPolicy
metadata:
  name: nic-cluster-policy
spec:
  ofedDriver:
    image: nvcr.io/nvidia/mellanox/mofed
    version: "23.10-0.5.5.0"
    startupProbe:
      initialDelaySeconds: 10
      periodSeconds: 20

  rdmaSharedDevicePlugin:
    image: nvcr.io/nvidia/k8s/k8s-rdma-shared-dev-plugin
    version: "sha-fe7f371c7e1b8315bf900f71cd25cfc1251dc775"
    config: |
      {
        "configList": [{
          "resourceName": "hca_shared_devices_a",
          "rdmaHcaMax": 63,
          "selectors": {
            "vendors": ["15b3"],
            "deviceIDs": ["101b"]
          }
        }]
      }

  sriovDevicePlugin:
    image: ghcr.io/k8snetworkplumbingwg/sriov-network-device-plugin
    version: "v3.7.0"
    config: |
      {
        "resourceList": [{
          "resourceName": "mlnx_sriov_rdma",
          "selectors": {
            "vendors": ["15b3"],
            "pfNames": ["ens2f0#0-11"]
          }
        }]
      }

  secondaryNetwork:
    cniPlugins:
      image: ghcr.io/k8snetworkplumbingwg/plugins
      version: "v1.4.0"
    multus:
      image: ghcr.io/k8snetworkplumbingwg/multus-cni
      version: "v4.0.2"
    ipamPlugin:
      image: ghcr.io/k8snetworkplumbingwg/whereabouts
      version: "v0.7.0"

The operator reads this and deploys the full software stack as DaemonSets on every node.

---

SR-IOV Configuration

After the operator runs, configure which physical NICs get VFs:

apiVersion: sriovnetwork.openshift.io/v1
kind: SriovNetworkNodePolicy
metadata:
  name: sriov-policy-ib
  namespace: sriov-network-operator
spec:
  resourceName: mlnx_sriov_rdma
  nodeSelector:
    feature.node.kubernetes.io/network-sriov.capable: "true"
  numVfs: 8             # Create 8 virtual functions per physical NIC
  nicSelector:
    vendor: "15b3"      # Mellanox vendor ID
    pfNames:
    - "ens2f0"          # Physical function name
  deviceType: netdevice
  isRdma: true          # Enable RDMA on VFs

The operator creates 8 VFs on each InfiniBand NIC across every matching node.

Kubernetes then sees:

Node capacity:
  mellanox.com/mlnx_sriov_rdma: 8

---

Multus — NetworkAttachmentDefinition

Define what the secondary network looks like:

apiVersion: k8s.cni.cncf.io/v1
kind: NetworkAttachmentDefinition
metadata:
  name: sriov-net-ib
  namespace: default
  annotations:
    k8s.v1.cni.cncf.io/resourceName: mellanox.com/mlnx_sriov_rdma
spec:
  config: |
    {
      "cniVersion": "0.3.1",
      "name": "sriov-net-ib",
      "type": "sriov",
      "ipam": {
        "type": "whereabouts",
        "range": "192.168.2.0/24"
      }
    }

---

Requesting RDMA in a Pod

A training pod requests both resources:

apiVersion: v1
kind: Pod
metadata:
  name: nccl-training
  annotations:
    k8s.v1.cni.cncf.io/networks: sriov-net-ib   # ← request secondary interface
spec:
  containers:
  - name: trainer
    image: nvcr.io/nvidia/pytorch:24.01-py3
    resources:
      limits:
        nvidia.com/gpu: "1"
        mellanox.com/mlnx_sriov_rdma: "1"        # ← request SR-IOV VF
    env:
    - name: NCCL_IB_HCA                           # ← tell NCCL which NIC to use
      value: "mlx5_0"
    - name: NCCL_IB_GID_INDEX
      value: "3"
    - name: UCX_NET_DEVICES
      value: "mlx5_0:1"

Inside this pod:

# eth0 — cluster network (overlay)
ip addr show eth0
# 10.244.1.5/24

# net1 — InfiniBand SR-IOV (direct hardware)
ip addr show net1
# 192.168.2.10/24

# RDMA device visible
ibv_devices
# device   node GUID
# mlx5_0   946dae0300a12345

NCCL automatically uses net1 for gradient traffic.

---

OFED — The InfiniBand Driver Layer

Before any of the above works, the InfiniBand kernel drivers must be installed on every node.

The standard Linux kernel includes basic RDMA support, but it is often outdated.

MLNX_OFED (Mellanox OpenFabrics Enterprise Distribution) provides:

• Up-to-date ConnectX drivers (mlx5_core, mlx5_ib)
• RDMA userspace libraries (libibverbs, librdmacm)
• InfiniBand management tools (ibstat, ibv_devinfo)
• NCCL-compatible transport layer

The Network Operator deploys OFED as a DaemonSet that runs on every node with Mellanox NICs.

It compiles and installs drivers automatically — no manual apt install across nodes.

---

The Complete Stack Comparison

Regular Kubernetes GPU Cluster

flowchart LR

GPU["GPU"]

-->|"PCIe copy\n(slow)"| SysMem["System RAM"]

SysMem

-->|"memcpy"| KernelBuf["Kernel Buffer"]

KernelBuf

-->|"veth"| Bridge["Linux Bridge"]

Bridge

-->|"VXLAN encap"| pNIC["Physical NIC\n(100 GbE shared)"]

pNIC

-->Wire["Network"]

Hops: 6+
CPU involvement: Every packet
Latency: ~50 μs
Effective bandwidth: ~8 GB/s

---

DGX with Network Operator

flowchart LR

GPU["GPU"]

-->|"PCIe P2P\n(GPUDirect)"| NIC["ConnectX-7\n(SR-IOV VF)"]

NIC

-->|"InfiniBand NDR\nRDMA"| Wire["Network\n400 Gb/s"]

Hops: 2
CPU involvement: None (RDMA)
Latency: ~1 μs
Effective bandwidth: ~50 GB/s per port

---

Impact on Training Throughput

For a 70B parameter model training across 8 DGX nodes (64 GPUs):

Each AllReduce step must synchronize ~140 GB of gradients (BF16).

Network	AllReduce Time	GPU idle time per step
100 GbE overlay	~18 seconds	~18 seconds
InfiniBand + RDMA	~0.5 seconds	~0.5 seconds

Training step time (compute + communication):

Configuration	Step time	GPUs busy
Overlay network	~20 seconds	~10% of time
InfiniBand + RDMA	~2.5 seconds	~80% of time

The 8× difference in training time between the two clusters from the opening example comes almost entirely from this.

---

Node Feature Discovery

How does Kubernetes know which nodes have InfiniBand NICs?

Node Feature Discovery (NFD) runs as a DaemonSet and labels nodes with their hardware capabilities:

# Labels added automatically by NFD
feature.node.kubernetes.io/network-sriov.capable: "true"
feature.node.kubernetes.io/pci-15b3.present: "true"   # Mellanox NIC present
feature.node.kubernetes.io/kernel-config.IB_CORE: "true"

The Network Operator and SR-IOV operator use these labels to target only InfiniBand-capable nodes.

---

NCCL Topology Awareness

NCCL is not just network-agnostic — it is topology-aware.

It reads the system topology to decide which communication algorithm to use:

# NCCL reads the topology at startup
NCCL INFO Trees [0] : 3->1->0->5->7->6->4->2
NCCL INFO Channel 00/04 : 0 1 2 3 4 5 6 7
NCCL INFO Ring 00 : 3 -> 0 -> 1 -> 2 -> 7 -> 4 -> 5 -> 6 -> 3

For DGX SuperPOD topology:

flowchart TB

subgraph DGX1["DGX Node 1"]
GPU0 & GPU1 & GPU2 & GPU3 & GPU4 & GPU5 & GPU6 & GPU7
NVSwitch1["NVSwitch"]
GPU0 & GPU1 & GPU2 & GPU3 & GPU4 & GPU5 & GPU6 & GPU7 <--> NVSwitch1
end

subgraph DGX2["DGX Node 2"]
GPU8 & GPU9 & GPU10 & GPU11 & GPU12 & GPU13 & GPU14 & GPU15
NVSwitch2["NVSwitch"]
GPU8 & GPU9 & GPU10 & GPU11 & GPU12 & GPU13 & GPU14 & GPU15 <--> NVSwitch2
end

NVSwitch1 <-->|"InfiniBand RDMA\n(inter-node)"| NVSwitch2

NCCL prefers:

• Ring AllReduce when all links have similar bandwidth
• Tree AllReduce for larger clusters with hierarchical topology
• Collnet for InfiniBand switch-based collective operations (offloads AllReduce to the switch ASIC)

---

Key Environment Variables

When running NCCL with InfiniBand:

# Tell NCCL which InfiniBand devices to use
NCCL_IB_HCA=mlx5_0,mlx5_1,mlx5_2,mlx5_3,mlx5_4,mlx5_5,mlx5_6,mlx5_7

# Enable GPUDirect RDMA
NCCL_IB_CUDA_SUPPORT=1

# Disable fallback to slower transports
NCCL_NET_GDR_LEVEL=5

# Log what transport NCCL chose (for debugging)
NCCL_DEBUG=INFO
NCCL_DEBUG_SUBSYS=NET

A correctly configured DGX job will show in logs:

NCCL INFO NET/IB : Using [0]mlx5_0:1/IB [1]mlx5_1:1/IB ... [7]mlx5_7:1/IB
NCCL INFO Using RDMA protocol (GPU Direct RDMA)

If you see TCP instead of IB or RDMA — the Network Operator is misconfigured or the SR-IOV interface was not requested in the pod spec.

---

Key Takeaways

Component	What It Does	Why It Matters
InfiniBand NDR	400 Gb/s RDMA fabric, 8 ports per DGX node	Raw bandwidth — the pipe between nodes
RDMA	Kernel bypass — NIC moves data without CPU	Eliminates CPU overhead and latency for every packet
GPUDirect RDMA	GPU memory → NIC → wire → NIC → GPU memory	Removes PCIe copies through system RAM
SR-IOV	Physical NIC split into VFs — each pod gets a real NIC	Eliminates software switching overhead
Multus	Multiple NICs per pod	Separates training traffic (fast path) from cluster traffic (slow path)
MLNX_OFED	InfiniBand drivers and RDMA userspace libraries	Foundation for all RDMA communication
Network Operator	Kubernetes Operator that deploys and manages all of the above	Automates the full stack — no manual driver installs
NFD	Labels nodes with hardware capabilities	Targets SR-IOV/RDMA config only at capable nodes

A DGX cluster without the Network Operator is just an expensive Kubernetes cluster. With it, GPUs spend their time computing — not waiting for gradients to arrive.