Kubernetes Storage: PV, PVC, StorageClass, and CSI

Containers are ephemeral. When a Pod is deleted or restarted, everything written to the container's filesystem is gone.

Most workloads need data to survive:

• A database must keep its data across Pod restarts
• A training job must save checkpoints before a node fails
• Multiple inference Pods must read the same model weights

Kubernetes solves this with a layered storage model: Volumes (Pod-scoped), PersistentVolumes (cluster-scoped), and StorageClass (dynamic provisioning).

---

Volumes — Pod-Scoped Storage

A Volume is a directory accessible to containers in a Pod. Unlike a container's filesystem, a volume's lifetime is tied to the Pod, not the container — it survives container restarts within the Pod.

apiVersion: v1
kind: Pod
spec:
  containers:
  - name: app
    image: nginx
    volumeMounts:
    - name: data
      mountPath: /data
  volumes:
  - name: data
    emptyDir: {}     # created when Pod starts, deleted when Pod dies

Common volume types:

Type	Lifetime	Use case
emptyDir	Pod	Scratch space, cache, shared between containers in Pod
hostPath	Node	Access node files (logs, device files) — avoid in production
configMap	Pod	Mount ConfigMap as files
secret	Pod	Mount Secret as files (credentials, TLS certs)
projected	Pod	Combine multiple sources into one mount

Volumes disappear when the Pod is deleted. For data that must outlive any Pod, use PersistentVolumes.

---

PersistentVolume and PersistentVolumeClaim

The Kubernetes storage model separates the provision of storage from the consumption of storage:

flowchart LR

Admin["Cluster Admin\n(or provisioner)"]

-->|"creates"| PV["PersistentVolume\n(cluster resource)\n50Gi NFS"]

User["Developer"]

-->|"creates"| PVC["PersistentVolumeClaim\n(namespace resource)\nrequests 20Gi"]

PVC

-->|"bound to"| PV

Pod

-->|"mounts"| PVC

PersistentVolume — Cluster-Level Storage Resource

A PersistentVolume (PV) is a piece of storage provisioned by an admin (or dynamically by a StorageClass). It is a cluster-level resource — not namespaced.

apiVersion: v1
kind: PersistentVolume
metadata:
  name: nfs-pv-01
spec:
  capacity:
    storage: 100Gi
  accessModes:
  - ReadWriteMany              # multiple Pods can read and write
  persistentVolumeReclaimPolicy: Retain
  nfs:
    server: 10.0.0.5
    path: /exports/data

PersistentVolumeClaim — Namespace-Level Request

A PersistentVolumeClaim (PVC) is a request for storage from a namespace. Kubernetes finds a PV that satisfies the request and binds them together.

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: model-weights
  namespace: training
spec:
  accessModes:
  - ReadWriteMany
  resources:
    requests:
      storage: 50Gi

Once bound, a Pod uses the PVC:

spec:
  volumes:
  - name: weights
    persistentVolumeClaim:
      claimName: model-weights
  containers:
  - name: trainer
    volumeMounts:
    - name: weights
      mountPath: /model

Binding Rules

Kubernetes binds a PVC to a PV when:

• The PV's capacity ≥ PVC's request
• The PV's access modes include all modes the PVC requests
• The PV is not already bound to another PVC
• StorageClass matches (if specified)

---

Access Modes

Mode	Short	Meaning
ReadWriteOnce	RWO	One node can mount read-write
ReadOnlyMany	ROX	Many nodes can mount read-only
ReadWriteMany	RWX	Many nodes can mount read-write
ReadWriteOncePod	RWOP	One Pod (not node) can mount read-write

Access mode availability depends on the storage backend:

Backend	RWO	ROX	RWX
AWS EBS	✅	❌	❌
GCP Persistent Disk	✅	✅	❌
Azure Disk	✅	❌	❌
NFS	✅	✅	✅
Lustre / WekaIO	✅	✅	✅
Local SSD	✅	❌	❌

RWX is essential for training jobs where multiple worker Pods on different nodes write checkpoints to the same shared filesystem.

---

Reclaim Policies

When a PVC is deleted, what happens to the PV?

Policy	Behavior
Retain	PV remains, data preserved; admin must manually reclaim
Delete	PV and its underlying storage are deleted automatically
Recycle (deprecated)	Data wiped (rm -rf), PV made available again

For production data, use Retain. For dynamically provisioned scratch space, Delete is appropriate.

---

StorageClass — Dynamic Provisioning

Manually creating PVs for every claim doesn't scale. StorageClass enables dynamic provisioning — a PV is created automatically when a PVC is submitted.

flowchart LR

PVC["PersistentVolumeClaim\n(requests 50Gi, class=fast)"]

-->SC["StorageClass\n(fast — CSI driver: ebs.csi.aws.com)"]

SC

-->|"calls CSI driver"| CSI["AWS EBS CSI Driver"]

CSI

-->|"creates"| EBSPV["EBS Volume\n(50Gi)"]

EBSPV

-->|"bound to"| PVC

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: fast
provisioner: ebs.csi.aws.com        # CSI driver that creates the volume
parameters:
  type: gp3
  iops: "16000"
  throughput: "1000"
reclaimPolicy: Delete
allowVolumeExpansion: true
volumeBindingMode: WaitForFirstConsumer  # wait until a Pod claims it (zone-aware)

A PVC referencing this StorageClass:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: training-checkpoint
spec:
  storageClassName: fast
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 200Gi
# → AWS EBS gp3 volume created automatically, bound to this PVC

volumeBindingMode: WaitForFirstConsumer is important for GPU clusters: it delays PV creation until a Pod is scheduled, so the PV is created in the same availability zone as the node — avoiding cross-AZ volume attachment failures.

---

CSI — Container Storage Interface

Like CNI for networking, CSI (Container Storage Interface) is the plugin spec for storage drivers. It decouples Kubernetes from specific storage systems.

flowchart LR

K8s["Kubernetes\n(kubelet + controller)"]

-->|"CSI gRPC calls"| CSIDriver["CSI Driver\n(AWS EBS / GCP PD / Portworx / Lustre)"]

CSIDriver

-->|"creates/mounts"| Storage["Underlying Storage\n(block device, NFS, Lustre)"]

CSI driver operations:

Operation	When called	What it does
CreateVolume	PVC + StorageClass	Provisions the underlying storage
DeleteVolume	PVC deleted	Removes the storage
ControllerPublishVolume	Pod scheduled	Attaches volume to the node
NodeStageVolume	Node receiving Pod	Formats and mounts to a staging path
NodePublishVolume	Pod starting	Bind-mounts into the Pod's namespace

CSI drivers run as DaemonSets (node component) + Deployments (controller component). The GPU Operator deploys CSI drivers automatically for the recommended storage backends on DGX nodes.

---

StatefulSets and Stable Storage

For stateful applications (databases, distributed key-value stores, Kafka), Pods need:

1. A stable network identity (same DNS name across restarts)
2. Their own dedicated PVC that follows them

StatefulSet provides both via volumeClaimTemplates:

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: postgres
spec:
  serviceName: postgres-headless     # headless Service for stable DNS
  replicas: 3
  template:
    spec:
      containers:
      - name: postgres
        image: postgres:16
        volumeMounts:
        - name: data
          mountPath: /var/lib/postgresql/data
  volumeClaimTemplates:              # one PVC per Pod, automatically created
  - metadata:
      name: data
    spec:
      storageClassName: fast
      accessModes: [ReadWriteOnce]
      resources:
        requests:
          storage: 100Gi

This creates:

• postgres-0 → PVC data-postgres-0 → dedicated 100Gi PV
• postgres-1 → PVC data-postgres-1 → dedicated 100Gi PV
• postgres-2 → PVC data-postgres-2 → dedicated 100Gi PV

When postgres-1 is restarted, it remounts data-postgres-1 — its data is preserved. Pods are created and deleted in order (0 → 1 → 2), which is important for primary/replica election.

---

Storage for GPU Training

Training workloads have extreme storage requirements that standard cloud block storage cannot meet.

Checkpoint Write Requirements

A 70B parameter model in BF16 = 140 GB. Saving a checkpoint writes all weights + optimizer state + gradient buffers:

Model weights (BF16):  70B × 2 bytes = 140 GB
Optimizer state (Adam): 70B × 8 bytes = 560 GB
Total checkpoint:       ~700 GB

At DGX scale (8 nodes, 64 GPUs), saving a checkpoint from all workers simultaneously requires:

Storage backend	Typical throughput	Suitable?
AWS EBS gp3 (per volume)	1 GB/s	❌ Too slow
NFS (single server)	5–10 GB/s	❌ Saturates quickly
AWS EFS	3–10 GB/s	⚠️ Marginal
Lustre	100–500 GB/s	✅
WekaIO	100–500 GB/s	✅
GPFS / IBM Spectrum Scale	100+ GB/s	✅

Lustre and WekaIO expose a POSIX filesystem that multiple DGX nodes can write to simultaneously at full InfiniBand bandwidth. The checkpoint directory is mounted as an RWX PVC backed by a Lustre StorageClass.

Model Loading for Inference

NIM containers cache compiled TRT-LLM engines on a PVC. A 70B model cache is ~150 GB. At inference startup, the NIM Pod reads the cache into GPU HBM:

• H100 SXM5 HBM bandwidth: 3.35 TB/s (on-chip)
• PCIe 5.0 bandwidth (storage to GPU): ~64 GB/s
• Model load time from NVMe: ~150 GB ÷ 10 GB/s (storage read) ≈ 15 seconds

An NVMe-backed StorageClass on local SSD minimizes NIM cold-start time significantly vs network-attached storage.

---

Storage Object Relationships

flowchart LR

StorageClass

-->|"creates on demand"| PV["PersistentVolume\n(cluster-scoped)"]

PV

-->|"bound to"| PVC["PersistentVolumeClaim\n(namespace-scoped)"]

PVC

-->|"mounted by"| Pod

Pod

-->|"reads/writes"| Data["Persistent Data\n(survives Pod death)"]

---

Key Takeaways

Concept	Purpose
Volume	Pod-scoped storage; survives container restarts but not Pod deletion
PersistentVolume (PV)	Cluster-level storage resource — provisioned by admin or dynamically
PersistentVolumeClaim (PVC)	Namespace-level request; binds to a matching PV
StorageClass	Dynamic provisioning policy — requests a PV creates the storage automatically
Access modes	RWO (one node), ROX (many nodes read), RWX (many nodes read-write)
Reclaim policy	What happens to data when PVC is deleted (Retain vs Delete)
CSI	Plugin spec for storage drivers — same role as CNI for networking
StatefulSet volumeClaimTemplates	One dedicated PVC per replica, persists across Pod restarts
Lustre / WekaIO	High-throughput parallel filesystems for training checkpoints at DGX scale

The key insight: storage in Kubernetes is a two-sided contract. Admins (or StorageClass provisioners) supply PersistentVolumes; developers claim them with PVCs. The Pod never cares what the underlying storage is — it just mounts the PVC. This separation is what lets the same Pod spec work on a laptop (local path) and a DGX cluster (Lustre) without code changes.