NVIDIA NIM: Optimized Inference Microservices on Kubernetes

Deploying a large language model in production is not just downloading weights and running them.

You need:

• TensorRT compilation optimized for the exact GPU (H100 vs A100 vs L40S)
• The right batching configuration for your latency/throughput target
• An OpenAI-compatible API so existing applications work without changes
• Health checks, metrics, and a container that restarts cleanly
• The right model parallelism for models too large for a single GPU

NIM (NVIDIA Inference Microservices) packages all of this into a single container image per model, ready to deploy with one Helm chart.

NIM is NVIDIA's answer to "how do you make production inference on DGX Cloud as easy as docker run?"

---

What NIM Is

NIM is a pre-built, optimized, containerized inference microservice.

flowchart TB

NGC["NGC Catalog\n(model artifacts + NIM profiles)"]

NGC

-->NIMContainer["NIM Container\n= Triton backend\n+ TensorRT-LLM engine\n+ OpenAI API server\n+ health/metrics endpoints"]

NIMContainer

-->|"exposes"| API["POST /v1/chat/completions\nPOST /v1/completions\nPOST /v1/embeddings\nGET /v1/models\nGET /health/ready"]

NIMContainer

-->|"runs on"| GPU["NVIDIA GPU\n(H100, A100, L40S...)"]

A single NIM container image serves a model through an OpenAI-compatible HTTP API. Any application that talks to OpenAI can point at NIM with one URL change.

---

NIM Profiles

The same model (e.g., Llama-3.1-70B) behaves very differently on an H100 SXM5 vs an A100 PCIe. Different memory bandwidth, different NVLink topology, different precision support.

A NIM profile is a pre-compiled, hardware-specific optimization bundle:

llama-3.1-70b-instruct/
├── profiles/
│   ├── h100x8-fp8-tp8/          # 8× H100, FP8, tensor parallel=8
│   ├── h100x4-bf16-tp4/         # 4× H100, BF16, tensor parallel=4
│   ├── a100x8-bf16-tp8/         # 8× A100, BF16, tensor parallel=8
│   ├── l40sx2-bf16-tp2/         # 2× L40S, BF16, tensor parallel=2
│   └── h100x1-int4-awq-tp1/     # 1× H100, INT4 AWQ quantized

When a NIM container starts, it detects the GPU type (via nvidia-smi) and automatically selects the matching profile. The TensorRT-LLM engine for that profile is either:

• Already baked into the container image, or
• Downloaded from NGC and cached locally on first startup

flowchart LR

NIMStart["NIM container starts"]

-->Detect["Detect GPU:\nnvidia-smi → H100 SXM5"]

Detect

-->Select["Select profile:\nh100x8-fp8-tp8"]

Select

-->|"cache hit"| Serve["Serve immediately\n(< 30s startup)"]

Select

-->|"cache miss"| Download["Download TRT-LLM engine\nfrom NGC\n(10-60 min first time)"]

Download

-->Cache["Cache to PVC\n/model-store"]

Cache

-->Serve

---

NIM Architecture

flowchart TB

Client["Client application\n(OpenAI SDK)"]

-->OpenAI_API["NIM API Server\n(FastAPI — OpenAI-compatible)"]

OpenAI_API

-->|"request routing"| Triton["Triton Inference Server\n(backend)"]

Triton

-->|"TensorRT-LLM backend"| TRT["TRT-LLM Engine\n(GPU-optimized, pre-compiled)"]

TRT

-->GPU["NVIDIA GPU(s)"]

subgraph NIMContainer["NIM Container"]
  OpenAI_API
  Triton
  TRT
end

Key components inside a NIM container:

Component	Role
NIM API Server	FastAPI app that implements the OpenAI chat/completions/embeddings API
Triton Inference Server	Handles batching, model parallelism, request scheduling
TensorRT-LLM Engine	The compiled, GPU-optimized model execution engine
Health endpoints	/health/ready, /health/live for Kubernetes probes
Prometheus metrics	/metrics — request rate, latency, token throughput, GPU utilization

---

NIM on Kubernetes

Direct Deployment (Helm)

The fastest path is the NVIDIA NIM Helm chart:

helm install llama3-70b \
  nvidia/nim-llm \
  --set model.name=meta/llama-3.1-70b-instruct \
  --set persistence.enabled=true \
  --set persistence.size=200Gi \
  --set resources.limits."nvidia.com/gpu"=8 \
  --namespace inference

The resulting Deployment:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: llama3-70b-nim
  namespace: inference
spec:
  replicas: 1
  template:
    spec:
      containers:
      - name: nim
        image: nvcr.io/nim/meta/llama-3.1-70b-instruct:1.3.0
        env:
        - name: NGC_API_KEY
          valueFrom:
            secretKeyRef:
              name: ngc-secret
              key: api-key
        - name: NIM_MODEL_NAME
          value: meta/llama-3.1-70b-instruct
        - name: NIM_SERVER_PORT
          value: "8000"
        resources:
          limits:
            nvidia.com/gpu: "8"
        volumeMounts:
        - name: model-store
          mountPath: /model-store   # NIM profile cache
        ports:
        - containerPort: 8000
        readinessProbe:
          httpGet:
            path: /v1/health/ready
            port: 8000
          initialDelaySeconds: 30
          periodSeconds: 10
          failureThreshold: 60     # TRT engine download can take time
        livenessProbe:
          httpGet:
            path: /v1/health/live
            port: 8000
          periodSeconds: 30
      volumes:
      - name: model-store
        persistentVolumeClaim:
          claimName: nim-model-store   # survives Pod restarts

NGC API Key

NIM containers pull model artifacts from NGC. An NGC API key must be present:

kubectl create secret generic ngc-secret \
  --from-literal=api-key=$NGC_API_KEY \
  --namespace inference

---

NIM Operator

For multi-model deployments and automated lifecycle management, NVIDIA provides the NIM Operator — a Kubernetes Operator that manages NIM instances via a NIMService CRD.

flowchart LR

NIMService["NIMService\n(CRD — desired state)"]

-->NIMOperator["NIM Operator\n(reconciles)"]

NIMOperator

-->|"creates"| Deployment["NIM Deployment\n+ Service + PVC"]

NIMOperator

-->|"watches"| GPU["GPU inventory\n(selects correct profile)"]

NIMOperator

-->|"manages"| Cache["Model cache PVC\n(per model)"]

NIMService CRD

apiVersion: apps.nvidia.com/v1alpha1
kind: NIMService
metadata:
  name: llama3-70b
  namespace: inference
spec:
  image:
    repository: nvcr.io/nim/meta/llama-3.1-70b-instruct
    tag: "1.3.0"
    pullSecrets:
    - name: ngc-secret

  model:
    name: meta/llama-3.1-70b-instruct
    ngcAPISecret: ngc-secret

  resources:
    limits:
      nvidia.com/gpu: "8"

  storage:
    nimCache:
      name: llama3-70b-cache
      profile: h100x8-fp8-tp8   # pin to a specific profile, or leave auto

  replicas: 1

  expose:
    service:
      type: ClusterIP
      port: 8000

The NIM Operator automatically:

• Detects the GPU type on available nodes and selects the matching NIM profile
• Pre-warms the model cache PVC before the first Pod starts
• Updates the Deployment when a new NIM image tag is available
• Scales replicas for multi-instance deployments

---

NIM API — OpenAI Compatible

NIM exposes an OpenAI-compatible API. Zero application changes required:

from openai import OpenAI

# Point at NIM instead of OpenAI
client = OpenAI(
    base_url="http://llama3-70b-nim.inference.svc.cluster.local:8000/v1",
    api_key="not-used-for-nim"  # NIM uses NGC auth at startup, not per-request
)

response = client.chat.completions.create(
    model="meta/llama-3.1-70b-instruct",
    messages=[{"role": "user", "content": "Explain RDMA in 2 sentences."}],
    max_tokens=200,
    temperature=0.7
)
print(response.choices[0].message.content)

Available endpoints:

Endpoint	What it does
POST /v1/chat/completions	Chat-style completion (messages array)
POST /v1/completions	Raw text completion (legacy)
POST /v1/embeddings	Text → embedding vector
GET /v1/models	List available models on this NIM
GET /v1/health/ready	Kubernetes readiness probe
GET /v1/health/live	Kubernetes liveness probe
GET /metrics	Prometheus metrics (latency, throughput, GPU util)

---

NIM Model Cache

The most operationally critical aspect of NIM on Kubernetes is the model cache — the compiled TRT-LLM engines stored on a PVC.

flowchart LR

subgraph First["First startup (cold)"]
  NGC -->|"download\n10-60 min"| PVC["PVC\n/model-store"]
  PVC -->|"compile TRT engine"| Engine["TRT-LLM Engine"]
  Engine -->|"save to PVC"| PVC
end

subgraph After["Subsequent startups (warm)"]
  PVC2["PVC\n(cached engines)"] -->|"load < 60s"| Serve["Ready to serve"]
end

PVC sizing guide:

Model	Cache size
Llama 3.1 8B	~20 GB
Llama 3.1 70B	~150 GB
Llama 3.1 405B	~800 GB
Mixtral 8×7B	~100 GB

Pre-warming the cache before routing traffic is essential for production — a cold NIM Pod behind a Service will fail readiness probes for 30–60 minutes on first launch.

---

NIM vs Triton vs vLLM

	NIM	Raw Triton	vLLM
Setup effort	Low — one Helm chart	High — manual TRT compilation, model config	Low — pip install
GPU optimization	Automatic (profile selection)	Manual — user compiles TRT engine	Automatic (PagedAttention)
API compatibility	OpenAI ✅	Triton HTTP/gRPC only	OpenAI ✅
Multi-GPU support	✅ Tensor parallel	✅ Tensor parallel	✅ Tensor/pipeline parallel
NGC integration	✅ Native	❌	❌
License	NVIDIA EULA	Apache 2.0	Apache 2.0
Best for	Production DGX Cloud inference	Custom model backends	Open-source, non-NVIDIA
Performance ceiling	Highest (TRT-LLM, hardware-tuned)	High (TRT-LLM but manual)	High (PagedAttention, custom kernels)

NIM wraps Triton — it uses Triton as its backend. NIM can also use vLLM as a backend for certain models. The value of NIM is the profile system (automatic hardware optimization selection) and the operator-managed lifecycle, not a fundamentally different inference engine.

---

Multi-Model Deployments on DGX Cloud

A typical DGX Cloud inference deployment runs multiple NIM services for different models:

flowchart LR

Router["Inference Gateway\n(NGINX / Envoy)"]

Router

-->|"/v1/chat\nmodel=llama-3.1-70b"| NIM1["NIM: Llama 3.1 70B\n(8× H100, FP8)"]

Router

-->|"/v1/embeddings\nmodel=nv-embedqa"| NIM2["NIM: NV-EmbedQA\n(1× H100)"]

Router

-->|"/v1/chat\nmodel=mixtral-8x7b"| NIM3["NIM: Mixtral 8×7B\n(4× H100, BF16)"]

Each NIM Service runs as a separate Kubernetes Deployment. GPU allocation is via nvidia.com/gpu limits (or DRA ResourceClaims for newer clusters).

Namespace isolation ensures team A's inference quota doesn't impact team B:

# Kueue ClusterQueue per team
apiVersion: kueue.x-k8s.io/v1beta1
kind: ClusterQueue
metadata:
  name: team-a-inference
spec:
  resourceGroups:
  - coveredResources: ["nvidia.com/gpu"]
    flavors:
    - name: h100-sxm5
      resources:
      - name: nvidia.com/gpu
        nominalQuota: 8     # team A gets 8 GPUs for inference

---

Key Takeaways

Concept	Purpose
NIM container	Pre-built inference microservice: Triton + TRT-LLM + OpenAI API
NIM profile	Hardware-specific TRT-LLM engine (H100 FP8, A100 BF16, etc.) — selected automatically
NGC model cache	PVC-backed engine cache — first download is slow, subsequent startups are fast
NIM Operator	Manages NIM deployments via NIMService CRD — lifecycle, profile selection, cache
OpenAI API	Drop-in replacement for OpenAI SDK — no application code changes
vs Triton	NIM wraps Triton — adds profile automation and operator management
vs vLLM	Both are valid; NIM is the NVIDIA-native path with TRT-LLM optimizations

NIM removes the gap between "I have an H100" and "I have a production inference endpoint." The profile system is the key insight: hardware-specific TRT compilation is what makes DGX hardware faster than a generic GPU server, and NIM automates that optimization decision entirely.