Helm: Kubernetes Package Manager

Deploying a real application to Kubernetes isn't one YAML file — it's a Deployment, a Service, a ConfigMap, a Secret, an Ingress, a ServiceAccount, RBAC rules, and a HorizontalPodAutoscaler. That's seven files before you've thought about environment-specific configuration (staging vs production).

Helm is the package manager for Kubernetes. It bundles related manifests into a chart, lets you configure them with values, and manages the full lifecycle of a deployed application as a release.

Think of it like apt or brew, but for Kubernetes workloads.

---

Core Concepts

flowchart LR

Repo["Helm Repository\n(OCI registry or\nHTTPS chart server)"]

-->|"helm pull"| Chart["Chart\n(templates + values.yaml\n+ Chart.yaml)"]

Chart

-->|"helm install\n--set key=value"| Release["Release\n(named instance\nin a namespace)"]

Release

-->|"creates"| K8s["Kubernetes resources\n(Deployment, Service,\nConfigMap, RBAC...)"]

Concept	What it is
Chart	A packaged collection of Kubernetes manifest templates + default values
Release	One deployed instance of a chart in a namespace — you can install the same chart multiple times under different release names
Repository	Where charts are stored — HTTPS servers or OCI registries
Values	Configuration that customizes a chart — overrides the chart's defaults
Revision	Every helm upgrade creates a new revision of a release — helm rollback reverts to any previous one

---

Chart Structure

gpu-operator/
├── Chart.yaml            # chart metadata (name, version, description)
├── values.yaml           # default values
├── charts/               # sub-chart dependencies
│   └── node-feature-discovery/
└── templates/            # Go-templated Kubernetes manifests
    ├── _helpers.tpl      # reusable template snippets
    ├── daemonset.yaml
    ├── clusterrole.yaml
    ├── clusterrolebinding.yaml
    └── serviceaccount.yaml

Chart.yaml

apiVersion: v2
name: gpu-operator
description: NVIDIA GPU Operator for Kubernetes
type: application
version: 24.9.0          # chart version (semver)
appVersion: 24.9.0       # version of the app this chart deploys
dependencies:
- name: node-feature-discovery
  version: "0.15.4"
  repository: https://kubernetes-sigs.github.io/node-feature-discovery/charts

values.yaml

Default configuration for every parameter the chart exposes:

# gpu-operator/values.yaml (simplified)
operator:
  defaultRuntime: containerd

driver:
  enabled: true
  version: "535.104.12"
  repository: nvcr.io/nvidia

toolkit:
  enabled: true

devicePlugin:
  enabled: true

dcgmExporter:
  enabled: true

mig:
  strategy: single

templates/daemonset.yaml

Templates use Go template syntax to inject values and release metadata:

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: {{ include "gpu-operator.fullname" . }}-driver
  namespace: {{ .Release.Namespace }}
  labels:
    app.kubernetes.io/name: {{ .Chart.Name }}
    app.kubernetes.io/version: {{ .Chart.AppVersion }}
    helm.sh/chart: {{ .Chart.Name }}-{{ .Chart.Version }}
spec:
  selector:
    matchLabels:
      app: nvidia-driver-daemonset
  template:
    spec:
      containers:
      - name: nvidia-driver
        image: {{ .Values.driver.repository }}/driver:{{ .Values.driver.version }}
        {{- if .Values.driver.enabled }}
        # driver config
        {{- end }}

Key template variables:

Variable	Value
{{ .Values.* }}	Values from values.yaml (or overridden by user)
{{ .Release.Name }}	The release name given at helm install
{{ .Release.Namespace }}	Namespace the release is installed into
{{ .Chart.Name }}	Chart name from Chart.yaml
{{ .Chart.Version }}	Chart version from Chart.yaml

---

Essential Commands

Working with Repositories

# Add a repository
helm repo add nvidia https://helm.ngc.nvidia.com/nvidia
helm repo add kueue-charts https://charts.kueue.x-k8s.io

# Update cached index from all repos
helm repo update

# Search for charts
helm search repo nvidia
helm search repo nvidia/gpu-operator --versions   # show all versions

# Show default values for a chart
helm show values nvidia/gpu-operator
helm show values nvidia/gpu-operator --version 24.6.0

Installing and Managing Releases

# Install (release name = gpu-operator, namespace = gpu-operator)
helm install gpu-operator nvidia/gpu-operator \
  --namespace gpu-operator \
  --create-namespace \
  --version 24.9.0

# Install with value overrides
helm install gpu-operator nvidia/gpu-operator \
  --namespace gpu-operator \
  --create-namespace \
  --set driver.version="550.54.15" \
  --set mig.strategy=mixed

# Install with a custom values file (recommended for multiple overrides)
helm install gpu-operator nvidia/gpu-operator \
  --namespace gpu-operator \
  --create-namespace \
  -f my-values.yaml

# Upgrade an existing release
helm upgrade gpu-operator nvidia/gpu-operator \
  --namespace gpu-operator \
  --set driver.version="550.90.07"

# Install or upgrade in one command
helm upgrade --install gpu-operator nvidia/gpu-operator \
  --namespace gpu-operator \
  --create-namespace \
  -f my-values.yaml

# List all releases in all namespaces
helm list --all-namespaces

# Check release status
helm status gpu-operator -n gpu-operator

# Show computed values for a running release
helm get values gpu-operator -n gpu-operator

# Show all Kubernetes resources a release created
helm get manifest gpu-operator -n gpu-operator

Rolling Back and Uninstalling

# View revision history
helm history gpu-operator -n gpu-operator
# REVISION  STATUS     CHART                    DESCRIPTION
# 1         superseded gpu-operator-24.6.0       Install complete
# 2         deployed   gpu-operator-24.9.0       Upgrade complete

# Roll back to previous revision
helm rollback gpu-operator 1 -n gpu-operator

# Uninstall (deletes all resources the release created)
helm uninstall gpu-operator -n gpu-operator

Dry Run and Debugging

# Render templates locally — see what would be applied (no cluster needed)
helm template gpu-operator nvidia/gpu-operator \
  -f my-values.yaml \
  --namespace gpu-operator

# Dry run against the cluster — validates against the API Server
helm install gpu-operator nvidia/gpu-operator \
  --dry-run \
  --debug \
  -f my-values.yaml \
  --namespace gpu-operator

helm template is invaluable for reviewing exactly what a chart will create before running it, especially for complex charts like the GPU Operator.

---

Values Overrides

Values can be overridden three ways, applied in this priority order (highest wins):

flowchart LR

Default["values.yaml\n(chart defaults)\nlowest priority"]

-->File["-f custom.yaml\n(your overrides)"]

-->Set["--set key=value\n(CLI flags)\nhighest priority"]

Using a values file (recommended)

A values.yaml you maintain alongside your GitOps repo:

# my-gpu-operator-values.yaml
driver:
  version: "550.90.07"
  repository: nvcr.io/nvidia

mig:
  strategy: mixed

dcgmExporter:
  enabled: true
  config:
    name: dcgm-metrics-config

toolkit:
  enabled: true
  version: "v1.16.0-ubuntu22.04"

helm upgrade --install gpu-operator nvidia/gpu-operator \
  -f my-gpu-operator-values.yaml \
  --namespace gpu-operator

Using --set for quick overrides

# Simple key
--set driver.enabled=true

# Nested key
--set dcgmExporter.config.name=my-config

# List value
--set tolerations[0].key=nvidia.com/gpu,tolerations[0].operator=Exists

# Multiple values
--set driver.version="550.90.07" --set mig.strategy=mixed

For anything beyond two or three overrides, use -f values.yaml — --set chains become hard to read and maintain.

---

Real Examples: GPU Stack on DGX

Every major component of the DGX Kubernetes stack is Helm-deployed:

# 1. GPU Operator (manages driver, toolkit, device plugin, DCGM)
helm upgrade --install gpu-operator nvidia/gpu-operator \
  --namespace gpu-operator --create-namespace \
  -f gpu-operator-values.yaml

# 2. Network Operator (InfiniBand, SR-IOV, RDMA, Multus)
helm upgrade --install network-operator nvidia/network-operator \
  --namespace network-operator --create-namespace \
  -f network-operator-values.yaml

# 3. Kueue (job queuing and quota management)
helm upgrade --install kueue kueue-charts/kueue \
  --namespace kueue-system --create-namespace \
  --version 0.9.0

# 4. NIM (inference microservice — one chart per model)
helm upgrade --install llama3-70b nvidia/nim-llm \
  --namespace inference --create-namespace \
  --set model.name=meta/llama-3.1-70b-instruct \
  --set resources.limits."nvidia\.com/gpu"=8 \
  --set persistence.size=200Gi

The order matters: GPU Operator must be running (driver + device plugin ready) before scheduling GPU Pods for NIM or Kueue workloads.

---

Helm Hooks

Hooks run Jobs at specific points in the release lifecycle:

# templates/pre-install-job.yaml
apiVersion: batch/v1
kind: Job
metadata:
  name: {{ .Release.Name }}-pre-install
  annotations:
    "helm.sh/hook": pre-install         # runs before any other resources
    "helm.sh/hook-weight": "-5"         # lower = runs first
    "helm.sh/hook-delete-policy": hook-succeeded
spec:
  template:
    spec:
      restartPolicy: Never
      containers:
      - name: pre-install
        image: alpine
        command: ["sh", "-c", "echo 'Running pre-install checks...'"]

Hook	When it runs
pre-install	Before any chart resources are created
post-install	After all resources are created
pre-upgrade	Before upgrade begins
post-upgrade	After upgrade completes
pre-rollback	Before rollback
post-rollback	After rollback
pre-delete	Before uninstall
post-delete	After uninstall
test	Only when helm test is run

Common uses: database migrations before upgrade, validation jobs after install, cleanup jobs after uninstall.

---

OCI Registries

Modern Helm charts are stored as OCI artifacts (like Docker images) in container registries — no separate chart server needed.

# Pull from NGC (NVIDIA's OCI registry)
helm pull oci://nvcr.io/nvidia/gpu-operator-chart --version 24.9.0

# Install directly from OCI
helm upgrade --install gpu-operator \
  oci://nvcr.io/nvidia/gpu-operator-chart \
  --version 24.9.0 \
  --namespace gpu-operator \
  --create-namespace

OCI charts work with any container registry (ECR, GCR, ACR, Harbor, GHCR). No helm repo add needed — just helm install oci://....

---

Helm vs Kustomize

	Helm	Kustomize
Model	Templating (Go templates)	Patching (overlays on top of base YAMLs)
Packaging	Charts — self-contained package with versioning	Base + overlays — plain YAML directories
Variables	values.yaml — typed, documented	configMapGenerator, patches
Versioning	Chart versions in a repo	Git-based (no built-in versioning)
Distribution	Helm repo / OCI registry	Git repo
Best for	Third-party software (GPU Operator, NIM)	Your own application configs
Built into kubectl	❌ (separate CLI)	✅ (kubectl apply -k)

In practice, many teams use both:

• Helm for upstream software (GPU Operator, Kueue, Cert-Manager)
• Kustomize for their own application manifests

# Kustomize usage
kubectl apply -k ./overlays/production

# or render to stdout
kubectl kustomize ./overlays/production

---

Helm Release Lifecycle

flowchart TD

Install["helm install\n(revision 1)"]

-->Running["Release: DEPLOYED\nRevision 1"]

Running

-->Upgrade["helm upgrade\n(revision 2)"]

Upgrade

-->Running2["Release: DEPLOYED\nRevision 2"]

Running2

-->|"something wrong"| Rollback["helm rollback 1\n(restore revision 1)"]

Rollback

-->Running3["Release: DEPLOYED\nRevision 1 restored"]

Running2

-->Uninstall["helm uninstall\n(all resources deleted)"]

Helm stores release state as Secrets in the release's namespace — one Secret per revision. This is how helm history and helm rollback work: the previous revision's manifest is replayed against the cluster.

# Helm stores state here:
kubectl get secrets -n gpu-operator -l owner=helm
# NAME                             TYPE                 DATA
# sh.helm.release.v1.gpu-operator.v1   helm.sh/release.v1   1
# sh.helm.release.v1.gpu-operator.v2   helm.sh/release.v1   1

---

Key Takeaways

Concept	Purpose
Chart	Versioned package of Kubernetes templates + default values
Release	Named deployed instance of a chart; one chart can have many releases
values.yaml	Default config; override with -f files or --set flags
Go templates	{{ .Values.* }}, {{ .Release.Name }} inject config into manifests
helm upgrade --install	Idempotent: installs if not present, upgrades if present
helm rollback	Restore any previous revision; Helm keeps full history as cluster Secrets
helm template	Render manifests locally — essential for reviewing before applying
OCI registry	Modern chart storage — same registry as container images
Hooks	Run Jobs at lifecycle points (pre-install, post-upgrade, pre-delete)
vs Kustomize	Helm for third-party packages; Kustomize for your own app overlays

Every tool in the DGX stack — GPU Operator, Network Operator, Kueue, NIM — ships as a Helm chart. Understanding Helm means you can inspect exactly what a chart deploys (helm template), customize it without forking (-f values.yaml), and recover from a bad upgrade (helm rollback) without touching the cluster manually.