Kubernetes Storage

Containers are ephemeral. When a Pod is deleted or restarted, everything written to the container's filesystem is gone.

Most workloads need data to survive:

• A database must keep its data across Pod restarts
• A training job must save checkpoints before a node fails
• Multiple inference Pods must read the same model weights

Ephemeral vs Persistent Containers

Ephemeral	Persistent
emptyDir	PersistentVolume
Lost when Pod dies	Survives Pod restart
Temporary	Long-term
Fast	Durable

Kubernetes solves this with a layered storage model:

• Volumes (Pod-scoped)
• PersistentVolumes (PV) (cluster-scoped)
• StorageClass (dynamic provisioning).
• Container Storage Interface(CSI)

flowchart TD
    Pod["Pod 📦"] --> Volume[("Volume 📒")]
    Volume --> PVC["PersistentVolumeClaims 📜<br/><br/> Namespace-Level Request"]
    PVC --> PV[("PersistentVolume 🛢️ <br/><br/>cluster-level resource")]
    PV --> CSI["CSI Driver 🔌"]
    CSI --> Storage[("Storage Backend 🧩")]

---

Storage Types

Kubernetes supports several storage options.

Type	Persistent	Scope
emptyDir	❌	Pod
hostPath	Node only	Node
PersistentVolume	✅	Cluster
CSI Volume	✅	Cluster
ConfigMap	Configuration	Pod
Secret	Sensitive Data	Pod

Kubernetes separates compute from storage.

• Volumes expose storage to Pods.
• PersistentVolumeClaims allow applications to request storage without knowing the underlying implementation.
• PersistentVolumes represent the actual storage resources.
• StorageClasses enable automatic provisioning.
• CSI provides a standard interface for integrating with cloud, on-premises, and enterprise storage systems.

This abstraction allows Pods to be rescheduled or replaced without losing application data, making Kubernetes suitable for both stateless and stateful workloads.

1. Volumes 📒

A Volume is a Pod-Scoped Storage directory accessible to containers in a Pod.

Unlike a container's filesystem, a volume's lifetime is tied to the Pod, not the container

it survives container restarts within the Pod.

Think of it like a Notebook to keep notes

Volume Lifecycle

The behavior depends on the volume type.

flowchart LR
    Pod --> Mount
    Mount --> ReadWrite
    ReadWrite --> PodDeleted
    PodDeleted --> VolumeLifecycle

The behavior depends on the volume type.

Example

    apiVersion: v1
    kind: Pod
    spec:
      containers:
        - name: app
          image: nginx
          volumeMounts:
            - name: data
              mountPath: /data
      volumes:
        - name: data
          emptyDir: { }     # created when Pod starts, deleted when Pod dies

Common volume types:

Type	Lifetime	Use case
emptyDir	Pod	Scratch space, cache, shared between containers in Pod
hostPath	Node	Access node files (logs, device files) — avoid in production
configMap	Pod	Mount ConfigMap as files
secret	Pod	Mount Secret as files (credentials, TLS certs)
projected	Pod	Combine multiple sources into one mount

Volumes disappear when the Pod is deleted.

For data that must outlive any Pod, use PersistentVolumes.

---

2. PersistentVolume (PV) 🛢️

A Piece of storage provisioned by an admin (or dynamically by a StorageClass).

It is a cluster-level resource — not namespaced.

Example creating a 100GB PV

    apiVersion: v1
    kind: PersistentVolume
    metadata:
      name: nfs-pv-01
    spec:
      capacity:
        storage: 100Gi
      accessModes:
        - ReadWriteMany              # multiple Pods can read and write
      persistentVolumeReclaimPolicy: Retain
      nfs:
        server: 10.0.0.5
        path: /exports/data

PersistentVolumeClaim (PVC) 📜

A request for storage from a namespace.

• Namespace-Level Request
• Kubernetes finds a PV that satisfies the request and binds them together.

The Kubernetes storage model separates the provision of storage from the consumption of storage:

The Pod never uses the PV directly.

flowchart LR
    Admin["Cluster Admin <br/> (or provisioner)"]
    Admin -->|" creates "| PV["PersistentVolume <br/> (cluster resource) <br/> 50Gi NFS"]
    User["Developer"]
    User -->|" creates "| PVC["PersistentVolumeClaim <br/> (namespace resource) <br/> requests 20Gi"]
    PVC -->|" bound to "| PV
    Pod -->|" mounts "| PVC

    apiVersion: v1
    kind: PersistentVolumeClaim
    metadata:
      name: model-weights
      namespace: training
    spec:
      accessModes:
        - ReadWriteMany
      resources:
        requests:
          storage: 50Gi

Once bound, a Pod uses the PVC:

    spec:
      volumes:
        - name: weights
          persistentVolumeClaim:
            claimName: model-weights
      containers:
        - name: trainer
          volumeMounts:
            - name: weights
              mountPath: /model

Binding Rules

Kubernetes binds a PVC to a PV when:

• The PV's capacity ≥ PVC's request
• The PV's access modes include all modes the PVC requests
• The PV is not already bound to another PVC
• StorageClass matches (if specified)

Access Modes

Mode	Short	Meaning
ReadWriteOnce	RWO	One node can mount read-write
ReadOnlyMany	ROX	Many nodes can mount read-only
ReadWriteMany	RWX	Many nodes can mount read-write
ReadWriteOncePod	RWOP	One Pod (not node) can mount read-write

Access mode availability depends on the storage backend:

Backend	RWO	ROX	RWX
AWS EBS	✅	❌	❌
GCP Persistent Disk	✅	✅	❌
Azure Disk	✅	❌	❌
NFS	✅	✅	✅
Lustre / WekaIO	✅	✅	✅
Local SSD	✅	❌	❌

RWX is essential for training jobs where multiple worker Pods on different nodes write checkpoints to the same shared filesystem.

Reclaim Policies

When a PVC is deleted, what happens to the PV?

Policy	Behavior
Retain	PV remains, data preserved; admin must manually reclaim
Delete	PV and its underlying storage are deleted automatically
Recycle (deprecated)	Data wiped (rm -rf), PV made available again

For production data, use Retain. For dynamically provisioned scratch space, Delete is appropriate.

---

##3. StorageClass 🧩

Dynamic Provisioning a PV automatically when a PVC is submitted.

Manually creating PVs for every claim doesn't scale. StorageClass enables dynamic provisioning PV for PVC

flowchart TD
    PVC["PersistentVolumeClaim <br/> (requests 50Gi, class=fast)"]
    PVC --> SC["StorageClass <br/> (fast — CSI driver: ebs.csi.aws.com)"]
    SC -->|" calls CSI driver "| CSI["AWS EBS CSI Driver"]
    CSI -->|" creates "| EBSPV["EBS Volume <br/> (50Gi)"]
    EBSPV -->|" bound to "| PVC

    apiVersion: storage.k8s.io/v1
    kind: StorageClass
    metadata:
      name: fast
    provisioner: ebs.csi.aws.com        # CSI driver that creates the volume
    parameters:
      type: gp3
      iops: "16000"
      throughput: "1000"
    reclaimPolicy: Delete
    allowVolumeExpansion: true
    volumeBindingMode: WaitForFirstConsumer  # wait until a Pod claims it (zone-aware)

A PVC referencing this StorageClass:

    apiVersion: v1
    kind: PersistentVolumeClaim
    metadata:
      name: training-checkpoint
    spec:
      storageClassName: fast
      accessModes:
        - ReadWriteOnce
      resources:
        requests:
          storage: 200Gi
    # → AWS EBS gp3 volume created automatically, bound to this PVC

volumeBindingMode: WaitForFirstConsumer is important for GPU clusters: it delays PV creation until a Pod is scheduled, so the PV is created in the same availability zone as the node — avoiding cross-AZ volume attachment failures.

---

Container Storage Interface (CSI) 🔌

Like CNI for networking, CSI (Container Storage Interface) is the plugin spec for storage drivers.

flowchart LR
    Pod --> PVC
    PVC --> PV
    PV --> CSI
    CSI --> AWS
    CSI --> Azure
    CSI --> GCP
    CSI --> Ceph
    CSI --> NFS

It decouples Kubernetes from specific storage systems.

flowchart TD
    K8s["Kubernetes <br/> (kubelet + controller)"]
    K8s -->|" CSI gRPC calls "| CSIDriver["CSI Driver <br/> (AWS EBS / GCP PD / Portworx / Lustre)"]
    CSIDriver -->|" creates/mounts "| Storage["Underlying Storage <br/> (block device, NFS, Lustre)"]

CSI driver operations:

Operation	When called	What it does
CreateVolume	PVC + StorageClass	Provisions the underlying storage
DeleteVolume	PVC deleted	Removes the storage
ControllerPublishVolume	Pod scheduled	Attaches volume to the node
NodeStageVolume	Node receiving Pod	Formats and mounts to a staging path
NodePublishVolume	Pod starting	Bind-mounts into the Pod's namespace

CSI drivers run as DaemonSets (node component) + Deployments (controller component).

The GPU Operator deploys CSI drivers automatically for the recommended storage backends on DGX nodes.

Local vs Network Storage

Local Storage	Network Storage
Fast	Shared
Node-specific	Multi-node
Low latency	Highly available
Cannot move with Pod	Accessible across nodes

---

StatefulSets and Stable Storage

For stateful applications (databases, distributed key-value stores, Kafka), Pods need:

1. A stable network identity (same DNS name across restarts)
2. Their own dedicated PVC that follows them

StatefulSet provides both via volumeClaimTemplates:

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: postgres
spec:
  serviceName: postgres-headless     # headless Service for stable DNS
  replicas: 3
  template:
    spec:
      containers:
        - name: postgres
          image: postgres:16
          volumeMounts:
            - name: data
              mountPath: /var/lib/postgresql/data
  volumeClaimTemplates: # one PVC per Pod, automatically created
    - metadata:
        name: data
      spec:
        storageClassName: fast
        accessModes: [ ReadWriteOnce ]
        resources:
          requests:
            storage: 100Gi

This creates:

• postgres-0 → PVC data-postgres-0 → dedicated 100Gi PV
• postgres-1 → PVC data-postgres-1 → dedicated 100Gi PV
• postgres-2 → PVC data-postgres-2 → dedicated 100Gi PV

When postgres-1 is restarted, it remounts data-postgres-1 — its data is preserved. Pods are created and deleted in order (0 → 1 → 2), which is important for primary/replica election.

---

Storage for GPU Training

Training workloads have extreme storage requirements that standard cloud block storage cannot meet.

Checkpoint Write Requirements

A 70B parameter model in BF16 = 140 GB. Saving a checkpoint writes all weights + optimizer state + gradient buffers:

Model weights (BF16):  70B × 2 bytes = 140 GB
Optimizer state (Adam): 70B × 8 bytes = 560 GB
Total checkpoint:       ~700 GB

At DGX scale (8 nodes, 64 GPUs), saving a checkpoint from all workers simultaneously requires:

Storage backend	Typical throughput	Suitable?
AWS EBS gp3 (per volume)	1 GB/s	❌ Too slow
NFS (single server)	5–10 GB/s	❌ Saturates quickly
AWS EFS	3–10 GB/s	⚠️ Marginal
Lustre	100–500 GB/s	✅
WekaIO	100–500 GB/s	✅
GPFS / IBM Spectrum Scale	100+ GB/s	✅

Lustre and WekaIO expose a POSIX filesystem that multiple DGX nodes can write to simultaneously at full InfiniBand bandwidth. The checkpoint directory is mounted as an RWX PVC backed by a Lustre StorageClass.

Model Loading for Inference

NIM containers cache compiled TRT-LLM engines on a PVC. A 70B model cache is ~150 GB. At inference startup, the NIM Pod reads the cache into GPU HBM:

• H100 SXM5 HBM bandwidth: 3.35 TB/s (on-chip)
• PCIe 5.0 bandwidth (storage to GPU): ~64 GB/s
• Model load time from NVMe: ~150 GB ÷ 10 GB/s (storage read) ≈ 15 seconds

An NVMe-backed StorageClass on local SSD minimizes NIM cold-start time significantly vs network-attached storage.

---

Storage Object Relationships

flowchart LR
    StorageClass
    StorageClass -->|" creates on demand "| PV["PersistentVolume <br/> (cluster-scoped)"]
    PV -->|" bound to "| PVC["PersistentVolumeClaim <br/> (namespace-scoped)"]
    PVC -->|" mounted by "| Pod
    Pod -->|" reads/writes "| Data["Persistent Data <br/> (survives Pod death)"]

---

Key Takeaways

Concept	Purpose
Volume	Pod-scoped storage; survives container restarts but not Pod deletion
PersistentVolume (PV)	Cluster-level storage resource — provisioned by admin or dynamically
PersistentVolumeClaim (PVC)	Namespace-level request; binds to a matching PV
StorageClass	Dynamic provisioning policy — requests a PV creates the storage automatically
Access modes	RWO (one node), ROX (many nodes read), RWX (many nodes read-write)
Reclaim policy	What happens to data when PVC is deleted (Retain vs Delete)
CSI	Plugin spec for storage drivers — same role as CNI for networking
StatefulSet volumeClaimTemplates	One dedicated PVC per replica, persists across Pod restarts
Lustre / WekaIO	High-throughput parallel filesystems for training checkpoints at DGX scale

The key insight: storage in Kubernetes is a two-sided contract.

• Admins (or StorageClass provisioners) supply
• PersistentVolumes; developers claim them with PVCs.

The Pod never cares what the underlying storage is — it just mounts the PVC.

This separation is what lets the same Pod spec work on a laptop (local path) and a DGX cluster (Lustre) without code changes.

---

Related Posts

• Multi-Node Distributed Training on Kubernetes — the primary consumer of high-throughput shared storage; explains checkpoint size math and why Lustre/WekaIO are needed
• NVIDIA NIM: Optimized Inference Microservices — NIM model cache stored on PVCs; explains the cold vs warm startup difference and PVC sizing per model
• AI Infra Networking: GPU Clusters and Storage — the storage network layer ( GPUDirect Storage, NVMe-oF) that backs the high-bandwidth PVCs used for training