AI Infra Storage: NVMe, Parallel File Systems, Object Storage, and GPUDirect Storage

AI Infra Storage

A $2M cluster of H100s idling while it waits on a dataset read is one of the most expensive failure modes in AI infrastructure — GPUs that cost hundreds of dollars an hour to run are only earning that cost back while they're actually computing.

Storage in an AI data center exists entirely to keep that from happening: extremely high throughput, parallel access from hundreds of GPUs simultaneously, low latency during training, and enough scalable capacity for datasets and checkpoints that keep growing.

Bottlenecks in AI Storage

Key Principle: GPUs must never sit idle waiting for data.

Bottleneck	Cause	Impact
Insufficient I/O throughput	Storage can't feed data as fast as GPUs consume it	GPU utilization drops during data loading
Network congestion	Storage traffic competing with other traffic on the same fabric	Unpredictable stalls, not just slow ones
Poor file system scaling	Single metadata server becomes a hotspot at scale	Small-file access (dataset shards) collapses under concurrent load
CPU bottlenecks during data movement	Data staged through host memory and CPU before reaching the GPU	The CPU becomes the ceiling on throughput, not the storage device

Every mitigation below exists to attack one of these four causes directly.

---

Tiered Storage Architecture

AI data centers use a hybrid, tiered model — not because one tier is "better," but because no single tier is simultaneously fast, cheap, and infinitely scalable:

flowchart TD

    subgraph Hot["🔥 Hot Tier"]
        direction LR
        NVMe["Local NVMe SSD <br/> ~1M+ IOPS, µs latency"]
        ParallelFS["Parallel FS <br/> (Lustre, WekaIO, BeeGFS) <br/> 100s of GB/s aggregate"]
    end

    subgraph Cold["❄️ Cold Tier"]
        direction LR
        Object["Object Storage <br/> (S3-compatible) <br/> ms latency, lowest cost/TB"]
    end

    NVMe-->|"active dataset shards, checkpoints"| ParallelFS
    ParallelFS-->|"cold after training completes"| Object

Hot Tier

Technology	Scope	Characteristics	Used for
Local NVMe SSD	Single node	Highest IOPS, lowest latency, directly attached	Active training shards, temporary datasets, local checkpoint staging
Network File Systems (NFS)	Shared, moderate scale	Moderate latency, simplest to operate	Shared datasets and checkpoints where extreme scale isn't the priority
Parallel & Distributed File Systems (Lustre, WekaIO, BeeGFS)	Whole cluster	Scales horizontally, hundreds of GB/s aggregate, metadata distributed across servers	Distributed training reading the same dataset from hundreds of GPUs concurrently

Cold Tier

Technology	Characteristics	Used for
Object Storage (S3-compatible)	Massive scalability, lowest cost per TB, millisecond-scale latency	Raw dataset archives, historical checkpoints, logs — anything not on the active training path

Data Locality

Local NVMe  >  Parallel FS  >  Object Storage
(fastest,      (shared,         (cheapest,
 smallest)      scalable)        highest latency)

Performance improves as data moves physically closer to compute and crosses fewer network hops — the entire point of tiering is deliberately placing only the data currently needed by a running job on the fastest, most expensive tier, and letting everything else live cheaply on object storage until it's needed again.

---

Storage Access Patterns in AI

	During Training	During Inference
Read pattern	Large sequential reads, multi-node concurrent access	Smaller model loads, mostly one-time
Write pattern	Frequent large checkpoint writes	Rare — logs only
Bandwidth need	High, sustained	Lower, bursty
Latency sensitivity	Tolerant for reads, sensitive for checkpoint writes (don't want to stall the training step)	High — cold model load directly delays first response
Typical backing store	Parallel file system, RDMA-backed	Local NVMe, optimized loading pipelines

---

RDMA & Storage: GPUDirect Storage

flowchart LR

    subgraph Traditional["Traditional Path"]
    direction LR
    S1["Storage"]-->C1["CPU"]-->M1["System Memory"]-->G1["GPU"]
    end

    subgraph GDS["GPUDirect Storage"]
    direction LR
    S2["Storage <br/> (NVMe / parallel FS)"]-->|"direct DMA"|G2["GPU Memory"]
    end

GPUDirect Storage (GDS) removes the CPU and system memory from the storage read path entirely — data moves via DMA straight from an NVMe drive or a parallel file system directly into GPU memory.

# Check GPUDirect Storage (cuFile) status and whether it's actually engaging the fast path
gdscheck -p

# cuFile config controls whether GDS is enabled and its fallback behavior
cat /etc/cufile.json | grep -A2 '"properties"'

If gdscheck -p reports the storage path as unsupported for GDS (common on some network file systems or misconfigured NVMe-oF targets), cuFile silently falls back to the traditional CPU-staged path — same as the SYS GPUDirect RDMA fallback covered in AI Infra Networking, the acceleration only works when every hop in the path actually supports it.

Best for: large dataset ingestion at the start of every epoch, and high-performance training clusters where CPU cycles spent staging I/O are cycles not available for anything else on the host.

NVMe over Fabrics (NVMe-oF)

NVMe-oF extends the NVMe protocol across the network, so a remote NVMe drive behaves — from the application's perspective — almost identically to a local one:

flowchart LR
    Initiator["NVMe-oF Initiator <br/> (compute node)"]
    Initiator-->|"NVMe commands over RDMA"| Target["NVMe-oF Target <br/> (storage node)"]
    Target-->Drive["Physical NVMe drives"]

Combined with RDMA as the transport (rather than TCP), NVMe-oF gets remote storage access close to local-NVMe latency — which is exactly what lets a parallel file system built from NVMe-oF targets deliver hundreds of GB/s aggregate to a training job spread across many nodes.

---

Storage Networking Considerations

Storage traffic needs to be isolated, high-bandwidth, low-contention, and predictable — the same Storage Network plane covered in AI Infra Networking, kept separate from compute (GPU-to-GPU) and management traffic so a burst of checkpoint writes never competes with an AllReduce for the same wire.

---

Storage Scalability

AI datasets grow faster than almost any other part of the stack. Parallel file systems scale horizontally by adding storage nodes and distributing metadata across multiple metadata servers rather than one — the second part matters as much as the first, since a single metadata server handling "does this file exist, what are its attributes" for millions of small dataset shards becomes the actual bottleneck long before raw throughput does, even on a system with plenty of spare disk bandwidth.

---

Storage and Checkpointing

Checkpoints exist so a multi-week training run surviving a single GPU failure means resuming from the last saved step, not restarting from scratch. What actually gets saved is larger than most people expect:

flowchart LR
    Checkpoint["Checkpoint"]
    Checkpoint-->Weights["Model Weights"]
    Checkpoint-->OptState["Optimizer State <br/> (Adam momentum + variance)"]
    Checkpoint-->Step["Current Step"]
    Checkpoint-->RNG["RNG State <br/> (reproducibility)"]

Adam's optimizer state (first and second moment estimates) is typically 3× the size of the weights themselves — for a 70B-parameter model in BF16:

Weights:         140 GB
Optimizer state: 420 GB
Total:           560 GB per checkpoint

At a sustained 20 GB/s write throughput (a realistic number for a well-tuned parallel file system under concurrent load from many ranks), writing one 560 GB checkpoint takes roughly 28 seconds — long enough that it has to happen asynchronously (overlapped with the next training step) rather than blocking, or checkpoint frequency itself becomes a throughput tax on the whole job. This is exactly the storage requirement referenced in Multi-Node Distributed Training on Kubernetes's fault-tolerance section.

Storage for checkpointing must handle frequent large writes, many GPUs writing simultaneously (sharded checkpoint formats write one shard per rank in parallel rather than funneling through a single writer), and fast recovery reads after a restart.

---

RAID & Data Protection

Mechanism	Where it's used	Tradeoff
RAID	Local NVMe / block storage	Redundancy and read performance, at a fixed capacity overhead
Erasure coding	Large-scale parallel and object storage	Better storage efficiency than RAID mirroring at similar durability, at the cost of more compute on rebuild

At the scale AI datasets and checkpoint archives reach, erasure coding (splitting data into fragments with parity across many drives/nodes) is what most large parallel and object storage systems use instead of traditional RAID — it tolerates multiple simultaneous drive failures across a much larger pool without paying RAID-1's full mirroring capacity cost.

---

Storage in Cloud vs On-Prem

	Cloud	On-Prem
Dominant tier	Object storage	Parallel file systems
Scaling model	Elastic, pay-as-you-go	Capacity-planned, added in discrete node increments
Control	Limited to provider's offerings	Full control over hardware and tuning
Cost at scale	Higher long-term for sustained high-throughput workloads	Lower long-term cost once utilization is high
Typical fit	Bursty, variable workloads; cold archival	Sustained, predictable, large-scale training

---

Exam Scenarios to Recognize

If a question mentions:

• GPUs starving for data → Storage bottleneck
• Massive shared dataset across nodes → Parallel file system
• Long-term archive → Object storage
• Direct storage-to-GPU transfer → GPUDirect Storage
• Ultra-fast local I/O → NVMe SSD

Quick Memory Anchors

• NVMe = Fastest local storage
• Parallel FS = Shared high-speed cluster storage
• Object storage = Massive, cheap, long-term
• GPUDirect Storage = Bypass CPU
• Training = High throughput demand
• Inference = Lower bandwidth, latency focus

---

Key Takeaways

Concept	Summary
Tiering	Hot (NVMe, parallel FS) for active data; cold (object storage) for archives — placement, not one tier being universally "better"
Parallel file systems	Scale by adding nodes and distributing metadata — metadata hotspots bottleneck small-file access before raw throughput does
GPUDirect Storage	DMA straight from NVMe/parallel FS into GPU memory, bypassing CPU and system memory — falls back silently if any hop doesn't support it
NVMe-oF	Extends NVMe across the network, usually over RDMA, to get remote storage close to local-NVMe latency
Checkpoint size	Optimizer state is ~3× weight size; a 70B model checkpoint is ~560 GB, written asynchronously so it doesn't stall training
Erasure coding	The durability mechanism of choice at scale — better storage efficiency than RAID mirroring across much larger failure domains
Cloud vs on-prem	Object storage + elastic scaling vs parallel FS + full control — a cost/control tradeoff, not a technical one

Every storage decision in an AI cluster traces back to one constraint: GPUs are the most expensive thing in the building, and every microsecond one spends waiting on data is a microsecond of idle silicon — which is why the entire storage stack, from tiering to GPUDirect Storage to asynchronous checkpointing, exists purely to keep data arriving faster than the GPUs can consume it.

---

Related Posts

• AI Infra Networking — the storage network plane and RDMA fundamentals GPUDirect Storage and NVMe-oF build on
• Kubernetes Storage: PV, PVC, StorageClass, and CSI — how these storage tiers are provisioned and consumed as Kubernetes volumes
• Multi-Node Distributed Training on Kubernetes — the fault-tolerant checkpointing this post's checkpoint math directly supports
• NVIDIA DCGM: GPU Health, Diagnostics, and Prometheus Metrics — monitoring GPU idle time caused by storage-side bottlenecks