Fine-Tuning LLMs: LoRA, QLoRA, PEFT, and NeMo on Kubernetes

A pre-trained LLM knows a lot about language. It does not know your company's policies, your proprietary data format, or how to respond in your product's voice.

Fine-tuning adapts a pre-trained model to a specific task or domain by continuing to train on a curated dataset. The question is always: how much of the model do you actually need to change?

---

Why Not Train From Scratch?

Pre-training a 70B model on 1–2 trillion tokens costs ~$10–50M in GPU compute and takes weeks on a DGX cluster. Fine-tuning reuses those learned representations and only adjusts the weights needed for your task.

Approach	Cost	When to use
Pre-training	$10M–$50M, weeks	Building a foundation model from scratch
Full fine-tuning	$10K–$500K, days	Large-scale domain adaptation with ample GPU budget
LoRA / PEFT	$100–$5K, hours	Task-specific adaptation on limited hardware
Prompt engineering	$0, minutes	Behavior you can describe; no new knowledge required
RAG	Low (inference cost only)	Injecting current or private knowledge at query time

---

Full Fine-Tuning Memory Problem

Full fine-tuning updates every parameter. For a 70B parameter model in BF16:

Model weights (BF16):         70B × 2 bytes   = 140 GB
Gradient buffer (BF16):       70B × 2 bytes   = 140 GB
Optimizer state — Adam m:     70B × 4 bytes   = 280 GB  (FP32)
Optimizer state — Adam v:     70B × 4 bytes   = 280 GB  (FP32)
─────────────────────────────────────────────────────────
Total:                                         ~840 GB

A DGX H100 node has 640 GB of HBM across 8× H100s. Full fine-tuning of a 70B model requires multi-node even with ZeRO-3 optimizer state sharding. LoRA eliminates the gradient and optimizer state costs for 99%+ of parameters.

---

Parameter-Efficient Fine-Tuning (PEFT)

PEFT is the umbrella term for techniques that adapt a model by training only a small fraction of its parameters. The base model weights are frozen; only the adapter parameters are trained.

flowchart LR

Base["Frozen Base Model <br/> (70B params — no gradients)"]

-->|"forward pass"| Adapter["Trainable Adapter <br/> (~10M params — LoRA)"]

Adapter

-->|"add to hidden state"| Output["Model Output"]

Main PEFT families:

Method	Where adapters go	Parameters trained
LoRA	Weight matrices (Q, V attention + FFN)	~0.1–1% of base
Prompt tuning	Input embedding space	Soft prompt tokens
Prefix tuning	K, V in each attention layer	Prefix vectors
Adapter layers	Between transformer sub-layers	Small bottleneck MLPs

LoRA is the dominant approach in 2024–2026 because it adds zero inference latency after merging and generalizes well.

---

LoRA: Low-Rank Adaptation

The key insight: weight updates during fine-tuning have low intrinsic rank — you don't need to update a full d × k matrix; a low-rank factorization captures the task-specific change.

The Math

For a pre-trained weight matrix W₀ ∈ R^(d×k), instead of learning a full ΔW:

W = W₀ + ΔW

LoRA constrains ΔW to be a low-rank product:

ΔW = B × A

where:
  B ∈ R^(d×r)   — randomly initialized to zero
  A ∈ R^(r×k)   — randomly initialized with Gaussian
  r << min(d, k)  — the rank hyperparameter

During training, W₀ is frozen. Only B and A are updated.

Parameter Savings

For a 70B model's attention layer (d=4096, k=4096):

Full ΔW:     4096 × 4096 = 16,777,216 parameters
LoRA r=8:    (4096×8) + (8×4096) = 65,536 parameters — 256× fewer
LoRA r=16:   (4096×16) + (16×4096) = 131,072 parameters — 128× fewer

flowchart LR

x["Input x <br/> (d-dim)"]

-->|"frozen W₀"| h1["W₀ · x <br/> (d-dim)"]

x

-->|"A (r×k)"| z["A · x <br/> (r-dim, r≪k)"]

z

-->|"B (d×r)"| h2["B · A · x <br/> (d-dim)"]

h1

-->|"+"| out["Output h <br/> = W₀·x + B·A·x"]

h2

-->out

Rank Selection

Rank (r)	Parameters	Use case
4	Minimal	Simple style / format adaptation
8	Low	Task-specific fine-tuning
16	Medium	Domain adaptation with larger dataset
64	High	Near-full-fine-tune quality; more memory

After training, LoRA adapters can be merged back into W₀:

W_merged = W₀ + B × A

The merged model has identical inference speed to the base model — zero adapter overhead.

Which Layers to Target

Standard practice targets the attention projection matrices:

target_modules = ["q_proj", "v_proj"]          # minimal — attention only
target_modules = ["q_proj", "k_proj", "v_proj", "o_proj"]  # full attention
target_modules = ["q_proj", "k_proj", "v_proj", "o_proj",
                  "gate_proj", "up_proj", "down_proj"]       # attention + FFN

Including FFN layers (gate/up/down projections) significantly improves adaptation quality at the cost of more trainable parameters.

---

QLoRA: LoRA on Quantized Models

QLoRA (2023) makes fine-tuning 70B models feasible on a single A100 or two H100s by quantizing the base model to 4-bit NF4 (NormalFloat4) precision before applying LoRA.

flowchart TB

Base["Base Model (BF16) <br/> 70B × 2 bytes = 140 GB"]

-->|"NF4 quantization"| Quant["4-bit Base Model <br/> 70B × 0.5 bytes = 35 GB"]

Quant

-->|"freeze"| FrozenBase["Frozen in 4-bit <br/> (dequantized to BF16 during forward)"]

LoRA["LoRA Adapters (BF16) <br/> ~65M params = ~0.1 GB"]

-->|"train in BF16"| LoRA

FrozenBase

-->Output["Training output"]

LoRA

-->Output

Memory breakdown for QLoRA on Llama-3.1-70B:

4-bit base model:          70B × 0.5 bytes ≈  35 GB
LoRA adapter weights:       ~65M × 2 bytes ≈   0.1 GB
LoRA gradients (BF16):      ~65M × 2 bytes ≈   0.1 GB
LoRA optimizer (Adam FP32): ~65M × 8 bytes ≈   0.5 GB
Activations (gradient ckpt): varies          ~  8 GB
────────────────────────────────────────────────────
Total:                                        ~44 GB

44 GB fits on a single H100 SXM5 (80 GB HBM). Without QLoRA, the same fine-tuning requires 840 GB.

---

Instruction Tuning (Supervised Fine-Tuning / SFT)

Instruction tuning trains the model to follow human-written instructions using (instruction, response) pairs. It does not change the model's knowledge — it changes how the model uses that knowledge.

Data Format

{"instruction": "Summarize this Kubernetes incident in 2 sentences.", "input": "...", "output": "..."}
{"instruction": "Write a Helm chart for a NIM deployment.", "input": "", "output": "..."}
{"instruction": "What does this CUDA kernel do?", "input": "...", "output": "..."}

Or in chat format (preferred for modern models):

{
  "messages": [
    {"role": "system", "content": "You are a Kubernetes expert."},
    {"role": "user", "content": "Why is my Pod stuck in Pending?"},
    {"role": "assistant", "content": "Check the events: kubectl describe pod ..."}
  ]
}

SFT with LoRA is the most common fine-tuning workflow on DGX Cloud.

---

Alignment: RLHF and DPO

SFT teaches what to say. Alignment techniques teach how good the output is relative to human preferences.

RLHF (Reinforcement Learning from Human Feedback)

flowchart LR

SFT["SFT Model"]

-->|"generate pairs"| Pairs["(response A, response B) <br/> for same prompt"]

Pairs

-->|"human raters prefer A"| RM["Reward Model <br/> (trained on preferences)"]

RM

-->|"PPO optimization"| Policy["Aligned Policy <br/> (rewarded for RM score)"]

RLHF is expensive: requires a trained reward model, PPO training loop, and careful hyperparameter tuning. GPT-4 and Claude use RLHF-based alignment.

DPO (Direct Preference Optimization)

DPO achieves similar alignment to RLHF without a separate reward model or RL loop. It directly optimizes the policy using a paired dataset of (preferred, rejected) responses:

# DPO loss (simplified)
loss = -log(sigmoid(β * (log_prob_chosen - log_prob_rejected)))

DPO is now preferred over RLHF for most use cases: simpler pipeline, comparable results, no reward model to train and maintain.

---

NeMo Fine-Tuning on Kubernetes

NVIDIA NeMo provides end-to-end fine-tuning pipelines that run as PyTorchJob workloads on Kubernetes.

LoRA Config in NeMo

# nemo_lora_config.yaml
model:
  restore_from_path: /models/llama-3.1-70b.nemo

  peft:
    peft_scheme: lora
    lora_tuning:
      target_modules: ["attention_qkv", "attention_dense", "mlp_fc1", "mlp_fc2"]
      adapter_dim: 16          # rank r
      alpha: 32                # scaling: ΔW = (α/r) × B × A
      dropout: 0.1

  data:
    train_ds:
      file_names: [/data/train.jsonl]
      micro_batch_size: 1
      global_batch_size: 128
    
  trainer:
    num_nodes: 1
    devices: 8                 # 8× H100
    max_steps: 1000
    val_check_interval: 100
    precision: bf16-mixed

  optim:
    name: distributed_fused_adam
    lr: 1e-4
    weight_decay: 0.01

Kubernetes PyTorchJob for Fine-Tuning

apiVersion: kubeflow.org/v1
kind: PyTorchJob
metadata:
  name: llama3-lora-finetune
  namespace: training
spec:
  pytorchReplicaSpecs:
    Master:
      replicas: 1
      template:
        spec:
          containers:
          - name: trainer
            image: nvcr.io/nvidia/nemo:24.09
            command:
            - torchrun
            - --nproc-per-node=8
            - /opt/NeMo/examples/nlp/language_modeling/tuning/megatron_gpt_finetuning.py
            - --config-path=/config
            - --config-name=nemo_lora_config.yaml
            resources:
              limits:
                nvidia.com/gpu: "8"
                memory: "1000Gi"
            volumeMounts:
            - name: models
              mountPath: /models
            - name: data
              mountPath: /data
            - name: config
              mountPath: /config
          volumes:
          - name: models
            persistentVolumeClaim:
              claimName: base-model-pvc
          - name: data
            persistentVolumeClaim:
              claimName: training-data-pvc
          - name: config
            configMap:
              name: lora-config

Multi-Node LoRA with Tensor Parallelism

For 70B models, even QLoRA may require multiple nodes. NeMo handles tensor parallelism transparently:

trainer:
  num_nodes: 2
  devices: 8              # 2 nodes × 8 GPUs = 16 GPUs total

model:
  tensor_model_parallel_size: 8    # model sharded across 8 GPUs per node
  pipeline_model_parallel_size: 2  # pipeline across 2 nodes

Kueue manages the quota so fine-tuning jobs don't starve inference workloads:

apiVersion: kueue.x-k8s.io/v1beta1
kind: LocalQueue
metadata:
  name: fine-tuning-queue
  namespace: training
spec:
  clusterQueueName: research-gpu-queue

---

Serving the Fine-Tuned Model

Option 1: Merge LoRA and Export to NIM

# Merge LoRA adapters into base weights
from peft import PeftModel
import torch

base_model = AutoModelForCausalLM.from_pretrained("meta-llama/Llama-3.1-70B")
model = PeftModel.from_pretrained(base_model, "/checkpoints/lora-adapter")
merged = model.merge_and_unload()      # fuses B×A into W₀
merged.save_pretrained("/models/llama3-70b-finetuned")

The merged model can then be converted to a NIM-compatible format and served with TRT-LLM optimization.

Option 2: Serve Adapter Separately (vLLM / Triton)

vLLM supports serving multiple LoRA adapters on a single base model:

vllm serve meta/llama-3.1-70b \
  --enable-lora \
  --lora-modules customer-a=/adapters/customer-a customer-b=/adapters/customer-b \
  --max-loras 4

This enables multi-tenant fine-tuning: one base model, many adapters, selected per-request via --lora-id.

---

Decision Guide

Scenario	Recommended approach
Adapt response style / format	Prompt tuning or small LoRA (r=4)
Domain-specific knowledge (legal, medical)	SFT + LoRA (r=16–64) on curated corpus
Task-specific accuracy (NER, classification)	Full SFT with LoRA on labeled examples
Better instruction following	SFT + DPO alignment
Budget < 1 GPU	QLoRA (4-bit base + LoRA adapters)
Budget 8 GPUs	Full LoRA SFT on 7B–13B; QLoRA on 70B
Budget 16–32 GPUs	Full LoRA SFT on 70B; full fine-tune on 7B–13B
Latest knowledge injection	RAG — fine-tuning doesn't add new facts reliably

---

Key Takeaways

Concept	Purpose
LoRA	Low-rank ΔW = B×A — trains ~0.1–1% of parameters; zero inference overhead after merging
QLoRA	LoRA on 4-bit NF4 quantized base — 70B fine-tuning on a single H100
PEFT	Umbrella term for adapter-based methods that freeze the base model
SFT	Supervised fine-tuning on (instruction, response) pairs — the first step
DPO	Alignment without a reward model — train directly on (preferred, rejected) pairs
NeMo + PyTorchJob	NVIDIA's production fine-tuning stack on Kubernetes
Rank r	Higher rank → more parameters → better quality, more memory
Merge vs serve separately	Merge for maximum inference speed; serve separately for multi-tenant adapters

The critical insight: fine-tuning changes how a model uses what it knows, not what it knows. For injecting new facts (company data, recent events), RAG is the right tool. For changing the model's behavior, style, or task specialization, LoRA is the most efficient path.

---

Related Posts

• NVIDIA NeMo and Enterprise AI Platforms — NeMo is the framework that runs the fine-tuning jobs described here; covers the full NeMo training stack and deployment pipeline
• NVIDIA NIM: Optimized Inference Microservices — the production inference path for merged fine-tuned models; NIM profiles compile the fine-tuned weights to TRT-LLM
• Multi-Node Distributed Training on Kubernetes — PyTorchJob orchestration, gang scheduling, and NCCL communication that multi-node fine-tuning jobs use
• Kueue: Job Queuing and Quota Management — queue fine-tuning jobs alongside inference workloads so GPU quotas are respected
• RAG: Retrieval-Augmented Generation — the alternative to fine-tuning for injecting current or proprietary knowledge at query time
• Megatron-LM and Distributed LLM Training — the distributed training engine NeMo uses under the hood for tensor and pipeline parallelism during fine-tuning