Helm: Kubernetes Package Manager 🪖

Helm is the package manager for Kubernetes.

Fun fact

Helm gets its name from a nautical theme that plays directly on Kubernetes, which means helmsman or pilot in Greek.

It bundles related manifests into a chart, lets you configure them with values, and manages the full lifecycle of a deployed application as a release.

Think of it like apt or brew, but for Kubernetes workloads.

---

Why we need Helm

Deploying a real application to Kubernetes isn't one YAML file

Imagine deploying PostgreSQL manually.

You need

    deployment.yaml
    service.yaml
    secret.yaml
    configmap.yaml
    ingress.yaml
    pvc.yaml
    networkpolicy.yaml

You might end up maintaining 20+ YAML files.

With Helm

  helm install postgres bitnami/postgresql

Everything is deployed automatically.

---

What is a Helm Chart 📰?

A Chart is a packaged Kubernetes application.

Just Like

• Image 📄 --> Container 🐳
• Npm 📦 --> JS Application 🌐
• Helm Chart 📰 --> Kubernetes Application ☸️

Core Concepts

Concept	What it is
Chart 📰	A packaged collection of Kubernetes manifest templates + default values
Values 📝	Configuration that customizes a chart — overrides the chart's defaults
Repository 🗃️	Where charts are stored — HTTPS servers or OCI registries
Revision 📈	Every helm upgrade creates a new revision of a release — helm rollback reverts to any previous one
Release 🚀	One deployed instance of a chart in a namespace — you can install the same chart multiple times under different release names

Helm Deployment workflow

flowchart TD
    Repo["Helm Repository  🗃️ <br/><br/> (OCI registry or <br/> HTTPS chart server)"]
    Repo -->|" helm pull "| Chart["Chart 📰 <br/><br/> (templates + values.yaml <br/> + Chart.yaml)"]
    Chart -->|" helm install <br/> --set key=value "| Release["Release 🚀<br/><br/> (named instance <br/> in a namespace)"]
    Release -->|" creates "| K8s["Kubernetes resources <br/> (Deployment, Service, ☸️ <br/><br/> ConfigMap, RBAC...)"]

---

Chart Structure 📰

    gpu-operator/
    ├── Chart.yaml            # chart metadata (name, version, description)
    ├── values.yaml           # default values
    ├── charts/               # sub-chart dependencies
    │   └── node-feature-discovery/
    └── templates/            # Go-templated Kubernetes manifests
        ├── _helpers.tpl      # reusable template snippets
        ├── daemonset.yaml
        ├── clusterrole.yaml
        ├── clusterrolebinding.yaml
        └── serviceaccount.yaml

Chart.yaml

Contains metadata.

Example:

    apiVersion: v2
    name: gpu-operator
    description: NVIDIA GPU Operator for Kubernetes
    type: application
    version: 24.9.0          # chart version (semver)
    appVersion: 24.9.0       # version of the app this chart deploys
    dependencies:
      - name: node-feature-discovery
        version: "0.15.4"
        repository: https://kubernetes-sigs.github.io/node-feature-discovery/charts

values.yaml

Stores configurable values.

Default configuration for every parameter the chart exposes:

Instead of editing templates, users edit only values.yaml

Example:

    # gpu-operator/values.yaml (simplified)
    operator:
      defaultRuntime: containerd
    
    driver:
      enabled: true
      version: "535.104.12"
      repository: nvcr.io/nvidia
    
    toolkit:
      enabled: true
    
    devicePlugin:
      enabled: true
    
    dcgmExporter:
      enabled: true
    
    mig:
      strategy: single

templates/daemonset.yaml

Templates are ordinary Kubernetes YAML files with Go template expressions.

Templates use Go template syntax to inject values and release metadata:

During installation

    Template + values.yaml --> Rendered YAML

Example:

    apiVersion: apps/v1
    kind: DaemonSet
    metadata:
      name: { { include "gpu-operator.fullname" . } }-driver
      namespace: { { .Release.Namespace } }
      labels:
        app.kubernetes.io/name: { { .Chart.Name } }
        app.kubernetes.io/version: { { .Chart.AppVersion } }
        helm.sh/chart: { { .Chart.Name } }-{{ .Chart.Version }}
    spec:
      selector:
        matchLabels:
          app: nvidia-driver-daemonset
      template:
        spec:
          containers:
            - name: nvidia-driver
              image: { { .Values.driver.repository } }/driver:{{ .Values.driver.version }}
              {{- if .Values.driver.enabled }}
              # driver config
              {{- end }}

Key template variables:

Variable	Value
{{ .Values.* }}	Values from values.yaml (or overridden by user)
{{ .Release.Name }}	The release name given at helm install
{{ .Release.Namespace }}	Namespace the release is installed into
{{ .Chart.Name }}	Chart name from Chart.yaml
{{ .Chart.Version }}	Chart version from Chart.yaml

---

Values Overrides

Values can be overridden three ways, applied in this priority order (highest wins):

flowchart TD
    Default["values.yaml <br/> (chart defaults) <br/> lowest priority"]

    Default-->File["-f custom.yaml <br/> (your overrides)"]

    File --> Set["--set key=value <br/> (CLI flags) <br/> highest priority"]

1. Using a values file (recommended)

A values.yaml you maintain alongside your GitOps repo:

    # my-gpu-operator-values.yaml
    driver:
      version: "550.90.07"
      repository: nvcr.io/nvidia
    
    mig:
      strategy: mixed
    
    dcgmExporter:
      enabled: true
      config:
        name: dcgm-metrics-config
    
    toolkit:
      enabled: true
      version: "v1.16.0-ubuntu22.04"

    helm upgrade --install gpu-operator nvidia/gpu-operator \
      -f my-gpu-operator-values.yaml \
      --namespace gpu-operator

2. Using --set for quick overrides

    # Simple key
    --set driver.enabled=true
    
    # Nested key
    --set dcgmExporter.config.name=my-config
    
    # List value
    --set tolerations[0].key=nvidia.com/gpu,tolerations[0].operator=Exists
    
    # Multiple values
    --set driver.version="550.90.07" --set mig.strategy=mixed

For anything beyond two or three overrides, use -f values.yaml — --set chains become hard to read and maintain.

---

Helm Hooks

Hooks run Jobs at specific points in the release lifecycle:

    # templates/pre-install-job.yaml
    apiVersion: batch/v1
    kind: Job
    metadata:
      name: { { .Release.Name } }-pre-install
      annotations:
        "helm.sh/hook": pre-install         # runs before any other resources
        "helm.sh/hook-weight": "-5"         # lower = runs first
        "helm.sh/hook-delete-policy": hook-succeeded
    spec:
      template:
        spec:
          restartPolicy: Never
          containers:
            - name: pre-install
              image: alpine
              command: [ "sh", "-c", "echo 'Running pre-install checks...'" ]

Hook	When it runs
pre-install	Before any chart resources are created
post-install	After all resources are created
pre-upgrade	Before upgrade begins
post-upgrade	After upgrade completes
pre-rollback	Before rollback
post-rollback	After rollback
pre-delete	Before uninstall
post-delete	After uninstall
test	Only when helm test is run

Common uses: database migrations before upgrade, validation jobs after install, cleanup jobs after uninstall.

---

Helm Release Lifecycle

flowchart TD
    Install["helm install <br/> (revision 1)"]
    Install --> Running["Release: DEPLOYED <br/> Revision 1"]
    Running --> Upgrade["helm upgrade <br/> (revision 2)"]
    Upgrade --> Running2["Release: DEPLOYED <br/> Revision 2"]
    Running2 -->|" something wrong "| Rollback["helm rollback 1 <br/> (restore revision 1)"]
    Rollback --> Running3["Release: DEPLOYED <br/> Revision 1 restored"]
    Running2 --> Uninstall["helm uninstall <br/> (all resources deleted)"]

Helm stores release state as Secrets in the release's namespace — one Secret per revision. This is how helm history and helm rollback work: the previous revision's manifest is replayed against the cluster.

    # Helm stores state here:
    kubectl get secrets -n gpu-operator -l owner=helm
    # NAME                             TYPE                 DATA
    # sh.helm.release.v1.gpu-operator.v1   helm.sh/release.v1   1
    # sh.helm.release.v1.gpu-operator.v2   helm.sh/release.v1   1

Essential Commands

Working with Repositories

OCI Registries

Modern Helm charts are stored as OCI artifacts (like Docker images) in container registries — no separate chart server needed.

    # Pull from NGC (NVIDIA's OCI registry)
    helm pull oci://nvcr.io/nvidia/gpu-operator-chart --version 24.9.0
    
    # Install directly from OCI
    helm upgrade --install gpu-operator \
      oci://nvcr.io/nvidia/gpu-operator-chart \
      --version 24.9.0 \
      --namespace gpu-operator \
      --create-namespace

OCI charts work with any container registry (ECR, GCR, ACR, Harbor, GHCR). No helm repo add needed — just helm install oci://....

    # Add a repository
    helm repo add nvidia https://helm.ngc.nvidia.com/nvidia
    helm repo add kueue-charts https://charts.kueue.x-k8s.io
    
    # Update cached index from all repos
    helm repo update
    
    # Search for charts
    helm search repo nvidia
    helm search repo nvidia/gpu-operator --versions   # show all versions
    
    # Show default values for a chart
    helm show values nvidia/gpu-operator
    helm show values nvidia/gpu-operator --version 24.6.0

---

📥 Installing and Managing Releases

When you run helm install

• Helm retrieves the chart from a repository (or a local directory).

• Helm combines configuration values from multiple sources.

• Helm processes all Go templates & Render Templates to generated plain Kubernetes YAML.

• Helm sends the rendered manifests to the Kubernetes API Server.

• API Server create/upgrade cluster

• Helm stores metadata about the installation inside the cluster.

  Namespace
  │
  ├── Deployment
  ├── Service
  ├── ConfigMap
  └── Secret
      └── sh.helm.release.v1.my-nginx.v3
  

  Helm simply reads these Secrets to answer commands like:

  helm list
  
  helm history
  
  helm rollback
  
  helm status
  

Example:

    # Install (release name = gpu-operator, namespace = gpu-operator)
    helm install gpu-operator nvidia/gpu-operator \
      --namespace gpu-operator \
      --create-namespace \
      --version 24.9.0
    
    # Install with value overrides
    helm install gpu-operator nvidia/gpu-operator \
      --namespace gpu-operator \
      --create-namespace \
      --set driver.version="550.54.15" \
      --set mig.strategy=mixed
    
    # Install with a custom values file (recommended for multiple overrides)
    helm install gpu-operator nvidia/gpu-operator \
      --namespace gpu-operator \
      --create-namespace \
      -f my-values.yaml

Managing Installed charts

    # List all releases in all namespaces
    helm list --all-namespaces
    
    # Check release status
    helm status gpu-operator -n gpu-operator
    
    # Show computed values for a running release
    helm get values gpu-operator -n gpu-operator
    
    # Show all Kubernetes resources a release created
    helm get manifest gpu-operator -n gpu-operator

---

▶️ Dry Run and Debugging

Render templates locally — see what would be applied (no cluster needed)

helm template gpu-operator nvidia/gpu-operator \
  -f my-values.yaml \
  --namespace gpu-operator

Dry run against the cluster — validates against the API Server

helm install gpu-operator nvidia/gpu-operator \
  --dry-run \
  --debug \
  -f my-values.yaml \
  --namespace gpu-operator

helm template is invaluable for reviewing exactly what a chart will create before running it, especially for complex charts like the GPU Operator.

Lint Helm files

helm lint .

---

⬆️ Upgrading a release

flowchart LR
    Install--> Revision1--> Upgrade--> Revision2--> Upgrade--> Revision3--> Rollback--> Revision2

Every upgrade creates a new revision.

    # Upgrade an existing release
    helm upgrade gpu-operator nvidia/gpu-operator \
      --namespace gpu-operator \
      --set driver.version="550.90.07"
    
    # Install or upgrade in one command
    helm upgrade --install gpu-operator nvidia/gpu-operator \
      --namespace gpu-operator \
      --create-namespace \
      -f my-values.yaml

---

🔄 Rolling Back

If a deployment fails

    # View revision history
    helm history gpu-operator -n gpu-operator
    # REVISION  STATUS     CHART                    DESCRIPTION
    # 1         superseded gpu-operator-24.6.0       Install complete
    # 2         deployed   gpu-operator-24.9.0       Upgrade complete

Roll back to previous revision

  helm rollback gpu-operator 1 -n gpu-operator

🗑️ Uninstalling

Uninstall (deletes all resources the release created)

    helm uninstall gpu-operator -n gpu-operator

Helm removes every resource created by the release.

---

Real Examples: GPU Stack on DGX

Every major component of the DGX Kubernetes stack is Helm-deployed:

    # 1. GPU Operator (manages driver, toolkit, device plugin, DCGM)
    helm upgrade --install gpu-operator nvidia/gpu-operator \
      --namespace gpu-operator --create-namespace \
      -f gpu-operator-values.yaml
    
    # 2. Network Operator (InfiniBand, SR-IOV, RDMA, Multus)
    helm upgrade --install network-operator nvidia/network-operator \
      --namespace network-operator --create-namespace \
      -f network-operator-values.yaml
    
    # 3. Kueue (job queuing and quota management)
    helm upgrade --install kueue kueue-charts/kueue \
      --namespace kueue-system --create-namespace \
      --version 0.9.0
    
    # 4. NIM (inference microservice — one chart per model)
    helm upgrade --install llama3-70b nvidia/nim-llm \
      --namespace inference --create-namespace \
      --set model.name=meta/llama-3.1-70b-instruct \
      --set resources.limits."nvidia\.com/gpu"=8 \
      --set persistence.size=200Gi

The order matters: GPU Operator must be running (driver + device plugin ready) before scheduling GPU Pods for NIM or Kueue workloads.

---

Helm vs Kustomize

	Helm	Kustomize
Model	Templating (Go templates)	Patching (overlays on top of base YAMLs)
Packaging	Charts — self-contained package with versioning	Base + overlays — plain YAML directories
Variables	values.yaml — typed, documented	configMapGenerator, patches
Versioning	Chart versions in a repo	Git-based (no built-in versioning)
Distribution	Helm repo / OCI registry	Git repo
Best for	Third-party software (GPU Operator, NIM)	Your own application configs
Built into kubectl	❌ (separate CLI)	✅ (kubectl apply -k)

In practice, many teams use both:

• Helm for upstream software (GPU Operator, Kueue, Cert-Manager)
• Kustomize for their own application manifests

# Kustomize usage
kubectl apply -k ./overlays/production

# or render to stdout
kubectl kustomize ./overlays/production

---

Key Takeaways

Every tool in the DGX stack — GPU Operator, Network Operator, Kueue, NIM — ships as a Helm chart.

Understanding Helm means you can inspect exactly what a chart deploys (helm template), customize it without forking (-f values.yaml), and recover from a bad upgrade (helm rollback) without touching the cluster manually.

Concept	Purpose
Chart	Versioned package of Kubernetes templates + default values
Release	Named deployed instance of a chart; one chart can have many releases
values.yaml	Default config; override with -f files or --set flags
Go templates	{{ .Values.* }}, {{ .Release.Name }} inject config into manifests
helm upgrade --install	Idempotent: installs if not present, upgrades if present
helm rollback	Restore any previous revision; Helm keeps full history as cluster Secrets
helm template	Render manifests locally — essential for reviewing before applying
OCI registry	Modern chart storage — same registry as container images
Hooks	Run Jobs at lifecycle points (pre-install, post-upgrade, pre-delete)
vs Kustomize	Helm for third-party packages; Kustomize for your own app overlays