Slurm

Slurm is a HPC workload management system behind AI Training Clusters

Kubernetes schedules long-running services.

Large-scale AI training jobs and HPC simulations are the opposite shape — a job runs to completion across hundreds of nodes and then exits.

Slurm (Simple Linux Utility for Resource Management) is the scheduler purpose-built for that shape, and it's still the default on most supercomputers and dedicated AI training clusters, DGX SuperPODs included.

• Slurm Official Docs

• Developed originally at the Lawrence Livermore National Laboratory.

---

Why Not Just Use Kubernetes?

Kubernetes' scheduling unit is a Pod that's expected to run indefinitely, restarted on failure, load-balanced by a Service.

A Slurm job is the opposite: submitted once, allocated a fixed set of nodes for its duration, and expected to finish — there's no restart policy because there's no "desired state" to reconcile back to, just a batch of work that either completes or fails.

	Slurm	Kubernetes
Primary purpose	Batch job scheduling	Container orchestration
Typical AI phase	Training	Inference / serving
Execution model	Run-to-completion	Always-on, self-healing
Scheduling unit	Job (nodes, CPUs, GPUs)	Pod
Resource allocation	Whole nodes / GPUs per job	Containers / requests-limits
Multi-node coordination	Native (MPI-aware launch)	Requires an operator (PyTorchJob, MPIJob)
Interface	sbatch, srun, squeue	kubectl, Helm, APIs
Typical users	HPC/research teams	Platform/DevOps/MLOps teams

Neither replaces the other on a real DGX cluster — it's common to run Slurm for the training partition and Kubernetes for the inference partition of the same physical hardware, with NVIDIA Base Command Manager provisioning both from one head node.

---

Architecture

flowchart TD

    User["Researcher <br/> sbatch train.sh"]
    User-->slurmctld["slurmctld <br/> (controller — scheduling decisions)"]
    slurmctld-->slurmdbd["slurmdbd <br/> (accounting database)"]

    slurmctld-->Node1["slurmd <br/> Compute Node 1"]
    slurmctld-->Node2["slurmd <br/> Compute Node 2"]
    slurmctld-->NodeN["slurmd <br/> Compute Node N"]

Component	Role
slurmctld	The controller — makes all scheduling decisions, tracks node/partition state
slurmd	Runs on every compute node — launches and monitors the job's processes locally
slurmdbd	Accounting database — job history, usage, fair-share tracking
Partition	A named subset of nodes (e.g. gpu, cpu, debug) jobs are submitted into

A job never talks to slurmd directly — it's submitted to slurmctld, which decides which nodes' slurmd processes actually launch it.

---

Submitting Jobs

Slurm has both a graphical interface and command line tools for submitting, monitoring, modifying and deleting jobs

It is normally used with job scripts to submit and execute jobs.

Various settings can be put in the job script, such as

• number of processors
• resource usage
• application specific variables.

The steps for running a job through Slurm are to:

• Create the script or executable that will be handled as a job
• Create a job script that sets the resources for the script/executable
• Submit the job script to the workload management system

Three commands cover almost everything:

Command	Use
sbatch	Submit a batch script, runs asynchronously, returns immediately with a job ID
srun	Run a command interactively/synchronously, optionally inside an sbatch script to launch the actual parallel job step
salloc	Allocate resources for an interactive session without running a script

A minimal GPU training job:

#!/bin/bash
#SBATCH --job-name=llama3-train
#SBATCH --partition=gpu
#SBATCH --nodes=4
#SBATCH --ntasks-per-node=8
#SBATCH --gres=gpu:8
#SBATCH --cpus-per-task=16
#SBATCH --time=24:00:00
#SBATCH --output=train-%j.log

srun python train.py --model llama3-70b --nodes $SLURM_NNODES

sbatch train.sh          # submit — returns a job ID immediately
squeue                   # see queued/running jobs
sinfo                    # see node/partition state
scancel 12345            # cancel a job
sacct -j 12345 --format=JobID,Elapsed,State,MaxRSS   # accounting history

--gres=gpu:8 is a Generic Resource (GRES) request — Slurm's mechanism for allocating non-CPU resources (GPUs, but also things like Infiniband adapters) per job, the same underlying concept as nvidia.com/gpu resource requests in Kubernetes, just at the node-allocation level rather than the container level.

---

Multi-Node MPI Jobs

MPI (Message Passing Interface) is the standard API for processes running on different nodes to exchange data — point-to-point sends/receives and collective operations like broadcast and all-reduce. Distributed training across many nodes needs exactly this: every rank starting at the same time, knowing its rank and world size, and able to talk to every other rank. Slurm handles the launch and rank-discovery side of that natively.

Two Ways to Launch an MPI Job Under Slurm

Approach	How it works
Native srun launch	Slurm itself launches one process per task directly via srun, and exports rank/size information through the PMI (Process Management Interface) so the MPI library initializes against Slurm's own launch mechanism
mpirun inside an allocation	salloc/sbatch only reserves the nodes; Open MPI's own mpirun launcher (running inside that allocation) does the actual process launch and rank assignment, using Slurm only as the resource allocator underneath

# Approach 1: native srun launch — Slurm launches every rank directly
#SBATCH --nodes=4
#SBATCH --ntasks-per-node=8
srun --mpi=pmix ./mpi_app

# Approach 2: mpirun inside a Slurm allocation — Open MPI does its own launch
#SBATCH --nodes=4
#SBATCH --ntasks-per-node=8
mpirun -np 32 ./mpi_app

Native srun launch is generally preferred at scale — it avoids mpirun having to SSH into every node itself, since Slurm already has a process on every allocated node ready to launch the job.

PMI, PMI2, and PMIx

The Process Management Interface is the actual protocol Slurm and the MPI runtime use to agree on ranks and exchange the connection information each process needs to find every other process:

Version	Notes
pmi (v1)	Original protocol, limited scalability
pmi2	Improved scalability, still widely supported
pmix	Current standard — faster startup at large rank counts, and what NCCL's Slurm integration also relies on

srun --mpi=list shows exactly which PMI versions a given Slurm installation was built with support for — mismatching this against what the MPI library expects is a common source of "job hangs at startup with zero output" failures.

A Real MPI Example

Before the PyTorch-specific launcher, the same rank/size pattern in raw MPI, using mpi4py:

#!/bin/bash
#SBATCH --nodes=4
#SBATCH --ntasks-per-node=8
#SBATCH --gres=gpu:8

srun --mpi=pmix python -c "
from mpi4py import MPI
comm = MPI.COMM_WORLD
print(f'Rank {comm.Get_rank()} of {comm.Get_size()} on {MPI.Get_processor_name()}')
"

Every task prints its own rank and the total world size — 32 lines of output, one per task across the 4 nodes, with no explicit hostfile: Slurm already told each mpi4py process its rank and how to reach the others via PMIx.

The GPU training case is the same launch mechanism, just with a deep learning framework instead of raw MPI collectives:

#!/bin/bash
#SBATCH --nodes=8
#SBATCH --ntasks-per-node=8      # 8 GPUs per node = 8 tasks per node
#SBATCH --gres=gpu:8

# srun launches one task per GPU across all 8 nodes simultaneously,
# and sets up the process ranks MPI/NCCL need to find each other.
srun --mpi=pmix python -m torch.distributed.run \
    --nnodes=$SLURM_NNODES \
    --nproc_per_node=8 \
    train.py

Slurm's --mpi=pmix flag wires up PMIx so an MPI runtime — or NCCL, which is not MPI but bootstraps its own rank/host discovery through the same PMIx handshake — can establish collective communication across all 64 GPUs without a separate discovery step. This is the same rank-0/AllReduce coordination problem covered in Multi-Node Distributed Training on Kubernetes, just solved at the scheduler level instead of via a Kubernetes operator.

Slurm vs MPI Environment Variables

Both Slurm and the MPI runtime expose rank/size information, under different variable names — worth knowing which is which when debugging a launch:

Concept	Slurm variable	Typical Open MPI variable
This process's rank	SLURM_PROCID	OMPI_COMM_WORLD_RANK
Total number of ranks	SLURM_NTASKS	OMPI_COMM_WORLD_SIZE
Rank within this node	SLURM_LOCALID	OMPI_COMM_WORLD_LOCAL_RANK
Node count	SLURM_NNODES	—

Frameworks like PyTorch's torch.distributed.run read whichever set is present to figure out RANK/WORLD_SIZE/LOCAL_RANK — which is why the same launch script works whether Slurm or mpirun did the actual process spawning underneath it.

---

Running Containers Under Slurm: enroot and pyxis

HPC clusters didn't traditionally run Docker — no root daemon, no arbitrary privilege escalation on shared supercomputers.

NVIDIA's answer is two pieces that plug directly into Slurm:

flowchart TD

    Image["Container image <br/> (Docker/OCI)"]
    Image-->|"enroot import"| Squashfs["Unprivileged squashfs <br/> root filesystem"]
    Squashfs-->|"enroot start"| Sandbox["Unprivileged container <br/> (no root daemon)"]

    Pyxis["pyxis <br/> (Slurm SPANK plugin)"]
    Pyxis-->|"wires enroot into srun"| Sandbox

1. enroot

Converts a standard container image into an unprivileged sandbox a regular user can run

• with no root daemon
• no elevated privileges required, unlike Docker.

2. pyxis

A Slurm SPANK plugin that adds container-aware flags directly to srun, so launching a containerized job looks like any other Slurm job:

srun --container-image=nvcr.io/nvidia/pytorch:24.01-py3 \
     --container-mounts=/data:/data \
     python train.py

This is the direct HPC-world equivalent of a Kubernetes Pod's container spec — same idea (run this image, mount this volume), different enforcement model (unprivileged squashfs instead of a container runtime daemon)

Because the constraint is "thousands of untrusted users share this supercomputer" rather than "one team owns this cluster."

---

Scheduling: Partitions, QOS, and Fair-Share

Slurm's scheduler has to decide, across potentially thousands of queued jobs, whose job runs next:

Concept	What it controls
Partition	Which pool of nodes a job can run on (gpu, cpu, debug)
QOS (Quality of Service)	Priority tier, max job time, max resources per user/QOS
Fair-share	Users/groups who've consumed less cluster time recently get scheduled ahead of heavy recent users
Backfill scheduling	Smaller/shorter jobs get slotted into gaps ahead of a large job still waiting for enough free nodes, without delaying that large job's eventual start

Backfill is what keeps utilization high on a cluster with a mix of quick debug jobs and week-long training runs — without it, a single queued 512-GPU job would leave every smaller job waiting behind it even while most of the cluster sits idle.

---

Job Priority — The Multifactor Plugin

"Fair-share" is one input into a larger formula. Slurm's default scheduler plugin computes a single priority number per job from several weighted factors:

Job Priority = (Age Weight × Age Factor)
             + (Fair-share Weight × Fair-share Factor)
             + (QOS Weight × QOS Factor)
             + (Partition Weight × Partition Factor)
             + (Job Size Weight × Size Factor)

Factor	Grows as…
Age	A job sits in the queue longer — prevents starvation
Fair-share	A user/group has consumed less recent cluster time than their allocated share
QOS	The job's Quality of Service tier is weighted higher (e.g. qos=urgent outranks qos=normal)
Partition	The partition itself carries a configured weight (a debug partition might be weighted low)
Job size	Configurable either way — some clusters favor large jobs to keep utilization high, others favor small jobs to reduce turnaround time

sprio -l shows the actual computed breakdown per queued job — the single most useful command for answering "why is my job still waiting" when several hundred jobs are queued ahead of it.

---

Preemption

On a shared cluster, a lower-priority job already running can be told to make way for a higher-priority one, rather than the higher-priority job simply waiting:

Preemption Mode	What happens to the lower-priority job
SUSPEND	Frozen in place (process suspended), resumed later when resources free up
REQUEUE	Killed and re-submitted from the start — only safe for jobs that checkpoint or are cheap to restart
CANCEL	Killed outright, not resubmitted

Preemption is configured per-QOS (PreemptMode, GraceTime in slurm.conf) — a common pattern is a low qos=preemptible tier for exploratory/best-effort jobs that get suspended the moment a qos=production training job needs those nodes, similar in spirit to how Kubernetes Pod priority/preemption evicts lower-priority Pods, just enforced against whole-node batch jobs instead of Pods.

---

Job Arrays and Dependencies

Two patterns come up constantly in ML workflows: sweeping many similar jobs, and chaining stages that depend on each other.

Job arrays — one sbatch submission, many independent tasks (a hyperparameter sweep is the canonical example):

#!/bin/bash
#SBATCH --job-name=hparam-sweep
#SBATCH --array=0-9                # 10 tasks, indices 0..9
#SBATCH --gres=gpu:1

LR_VALUES=(0.001 0.003 0.01 0.03 0.1 0.0003 0.0001 0.03 0.3 1.0)
srun python train.py --lr ${LR_VALUES[$SLURM_ARRAY_TASK_ID]}

Each array task gets its own $SLURM_ARRAY_TASK_ID, its own log file, and is scheduled independently — a failure in task 7 doesn't affect the other nine.

Job dependencies — chain a pipeline so evaluation only starts after training actually succeeds:

TRAIN_JOB=$(sbatch --parsable train.sh)
sbatch --dependency=afterok:$TRAIN_JOB evaluate.sh

afterok only releases the dependent job if the upstream job exited with status 0 — afterany runs regardless of exit status, afternotok runs only if it failed (useful for an alerting/cleanup job).

---

Node States and Health

Every node Slurm manages is in exactly one state, and the scheduler only places jobs on nodes in a schedulable state:

State	Meaning
IDLE	No jobs running, available
ALLOCATED	Fully allocated to running job(s)
MIXED	Partially allocated — some CPUs/GPUs free, some in use
DOWN	Unreachable or failed — slurmd isn't responding
DRAINING	Currently running jobs finish normally, but no new jobs will be scheduled here
DRAIN	Manually marked unschedulable, whether or not anything is running

# Drain a node before hardware service — running jobs finish, no new ones start
scontrol update NodeName=dgx-node-04 State=DRAIN Reason="GPU RMA pending"

# See why a node is drained
sinfo -R

This is the same node-health concept covered in NVIDIA DCGM — in practice, a Node Health Check (NHC) script runs between jobs (or on a timer) and can automatically drain a node the moment it sees a DCGM XID error or a failed dcgmi diag, so a flaky GPU pulls itself out of the schedulable pool before it silently corrupts the next job placed on it.

---

Resource Enforcement with cgroups

Requesting --gres=gpu:2 --cpus-per-task=8 doesn't just influence scheduling — Slurm actually enforces it. The task/cgroup plugin creates a cgroup per job step and constrains it to exactly the CPUs, memory, and GPU devices (via /dev/nvidia* cgroup device rules) that were allocated, so one job can't quietly consume a GPU a neighboring job on the same node was granted.

slurmd (task/cgroup plugin)
        │
        ▼
  cgroup created per job step
        │
   ┌────┴────┬──────────┐
   ▼         ▼          ▼
 cpuset   memory    devices (GPU)
 (pin to  (limit    (only allocated
 alloc'd  to req)   /dev/nvidia* visible)
 cores)

This is the exact same enforcement mechanism described in Kubelet Internals — cgroups turning a resource request into a kernel-enforced guarantee — just applied by slurmd against a whole job step instead of by the kubelet against a container.

---

Topology-Aware Scheduling

On a DGX SuperPod, which specific nodes a multi-node job lands on matters — nodes on the same NVLink/InfiniBand leaf switch communicate far faster than nodes across the fabric. Slurm's optional topology.conf describes the physical network tree, and the topology/tree plugin uses it to prefer placing a job's nodes as close together in that tree as possible:

switch.top:
  SwitchName=leaf1 Nodes=dgx[01-08]
  SwitchName=leaf2 Nodes=dgx[09-16]
  SwitchName=spine1 Switches=leaf1,leaf2

Given a choice between eight free nodes scattered across two leaf switches versus eight free nodes all under one leaf switch, topology-aware scheduling picks the latter — minimizing the inter-switch hops an AllReduce has to cross, the same NVLink-vs-InfiniBand bandwidth cliff covered in NVIDIA Network Operator.

---

The Slurm REST API

slurmrestd exposes the same job submission and query operations as sbatch/squeue over HTTP, which is what CI/CD pipelines and higher-level orchestration tools use instead of shelling out to CLI commands:

# Submit a job via the REST API instead of sbatch
curl -X POST http://slurm-head:6820/slurm/v0.0.40/job/submit \
  -H "X-SLURM-USER-NAME: ci-bot" \
  -H "X-SLURM-USER-TOKEN: $SLURM_JWT" \
  -H "Content-Type: application/json" \
  -d '{"job": {"partition": "gpu", "name": "ci-train", "script": "#!/bin/bash\nsrun python train.py"}}'

This is the same pattern as BCM's REST API and cmsh — every interactive CLI operation on the cluster has a scriptable HTTP equivalent, which is what actually makes "trigger a training run from a CI/CD pipeline" (see CI/CD Pipelines) possible without a human running sbatch by hand.

---

Key Takeaways

Concept	Summary
slurmctld / slurmd / slurmdbd	Controller makes scheduling decisions; per-node daemon launches/monitors; accounting DB tracks usage
sbatch / srun / salloc	Submit async, run a parallel step, or allocate interactively
GRES	Generic Resource — how Slurm allocates GPUs (and other non-CPU resources) per job
MPI / PMIx	srun --mpi=pmix bootstraps rank discovery so NCCL/MPI collectives can start across nodes
enroot + pyxis	Unprivileged container execution under Slurm — no root daemon, wired into srun via a SPANK plugin
Partition / QOS / fair-share / backfill	The mechanisms that decide whose job runs next on a shared, heavily-queued cluster
Multifactor priority	Age + fair-share + QOS + partition + job-size weights combine into one priority number per job
Preemption	SUSPEND / REQUEUE / CANCEL — how a higher-priority job takes resources from a running lower-priority one
Job arrays / dependencies	--array for parameter sweeps, --dependency=afterok: for pipeline chaining
Node states + NHC	IDLE/MIXED/DRAIN/DOWN; a Node Health Check script can auto-drain a node on a DCGM XID error
cgroups enforcement	task/cgroup constrains each job step to exactly its allocated CPUs, memory, and GPU devices
Topology-aware scheduling	topology.conf places multi-node jobs on physically nearby nodes to minimize AllReduce hops
slurmrestd	HTTP equivalent of sbatch/squeue, for CI/CD and orchestration tools to trigger jobs programmatically
vs Kubernetes	Run-to-completion batch jobs vs always-on self-healing services — different execution models for training vs inference

Slurm's entire design center is a queue of jobs that will eventually finish, on hardware shared fairly across many users — which is exactly the opposite assumption Kubernetes makes, and exactly why supercomputers and DGX training clusters still run Slurm even in a Kubernetes-everywhere world.

---

Interview Questions

Q: Why would an HPC/AI training cluster use Slurm instead of Kubernetes? A: Kubernetes' scheduling unit is a Pod expected to run indefinitely and be restarted on failure — it reconciles toward a desired state that persists. A Slurm job is the opposite shape: submitted once, allocated a fixed set of nodes, and expected to run to completion and exit, with no restart policy because there's no ongoing desired state to reconcile toward. Slurm also natively coordinates multi-node MPI rank discovery at launch time, which on Kubernetes requires an additional operator (PyTorchJob, MPIJob). Large training clusters often run both — Slurm for the training partition, Kubernetes for the inference partition — provisioned from the same head node by something like NVIDIA Base Command Manager.

Q: What's the difference between sbatch, srun, and salloc? A: sbatch submits a batch script asynchronously and returns immediately with a job ID — the normal way to submit a job that doesn't need a human watching it run. srun runs a command as a job step, either standalone (interactive/synchronous) or inside an sbatch script to launch the actual parallel process across allocated nodes. salloc allocates resources for an interactive session without running any script — useful for interactive debugging on allocated GPUs.

Q: How does Slurm decide which of several thousand queued jobs runs next? A: Through the multifactor priority plugin — a weighted sum of an age factor (prevents starvation of long-queued jobs), a fair-share factor (users who've consumed less recent cluster time rank higher), a QOS factor (priority tier), a partition factor, and a job-size factor. sprio -l shows the actual computed breakdown per job. Backfill scheduling additionally lets smaller/shorter jobs slot into gaps ahead of a large job still waiting for enough free nodes, without delaying that large job's eventual start.

Q: How would you take a GPU node out of service for maintenance without killing what's currently running on it? A: scontrol update NodeName=<node> State=DRAIN Reason="..." — this puts the node into DRAINING if a job is currently running on it (letting it finish normally) or straight to DRAIN if idle, and the scheduler simply stops placing any new jobs there. This is also what an automated Node Health Check script does when it detects a DCGM XID error or a failed diagnostic — draining the node before a bad GPU can silently corrupt the next job placed on it.

Q: How does Slurm actually enforce that a job only uses the CPUs, memory, and GPUs it was allocated? A: The task/cgroup plugin creates a Linux cgroup per job step and constrains it to exactly the allocated CPU set, memory limit, and GPU devices (via cgroup device rules restricting which /dev/nvidia* nodes are visible) — the same cgroup-based enforcement mechanism the kubelet uses for container resource limits, just applied by slurmd against a batch job step instead of a container.

Q: Why would you configure topology.conf, and what does it actually change? A: It describes the physical network tree (which nodes sit under which leaf switch, up to the spine) so the topology/tree scheduling plugin can prefer placing a multi-node job's nodes as physically close together as possible — e.g. all under one leaf switch rather than scattered across two. This matters because inter-switch hops are slower than intra-switch ones; a distributed training job's AllReduce step is only as fast as the slowest link between any two of its ranks, so topology-aware placement directly affects training throughput on a fabric like NVLink/InfiniBand.

Q: What are enroot and pyxis, and why does HPC need them instead of just using Docker? A: Traditional HPC clusters don't run a Docker daemon — that would mean a root-privileged service on shared supercomputers with thousands of untrusted users, which is an unacceptable security posture. enroot converts a standard OCI/Docker image into an unprivileged squashfs-based sandbox a regular user can run without any daemon or elevated privileges. pyxis is a Slurm SPANK plugin that adds container-aware flags directly onto srun (--container-image, --container-mounts), so launching a containerized job looks and schedules exactly like any other Slurm job.

Q: How would you chain a training job and an evaluation job so evaluation only runs if training actually succeeds? A: Submit training first with --parsable to capture its job ID cleanly, then submit the evaluation job with --dependency=afterok:<train_job_id>. afterok only releases the dependent job if the upstream job exited with status 0; afternotok is the inverse (useful for a cleanup/alert job), and afterany runs regardless of exit status.

Q: How would a CI/CD pipeline trigger a Slurm training job without a human running sbatch manually? A: Through slurmrestd, which exposes job submission, status queries, and cancellation over HTTP — the same operations sbatch/squeue/scancel perform from the CLI, just callable from a pipeline step with a JWT auth token instead of an interactive shell session.

---

Related Posts

• NVIDIA Base Command Manager — provisions and manages the same cluster Slurm schedules jobs onto
• Multi-Node Distributed Training on Kubernetes — the same rank/AllReduce coordination problem, solved via a Kubernetes operator instead of srun
• Kubernetes Scheduler Internals — where Kueue fits as the closest Kubernetes-native equivalent to Slurm's queueing model
• Kueue: Batch Job Scheduling — Kubernetes-native gang scheduling and quotas, for clusters that run training on Kubernetes instead of Slurm
• NVIDIA DCGM: GPU Health, Diagnostics, and Prometheus Metrics — the GPU health signal Slurm jobs depend on before and during a multi-node run