Optimizing AI Inference at Scale

Modern AI infrastructure is a stack of optimizations, with each layer solving a different bottleneck.

Think of it like building a Formula 1 car.

A better engine alone won't win the race.

You also need better aerodynamics, suspension, tires, strategy, and pit stops.

The same is true for AI infrastructure.

There is no single technique to optimize GPU utilization for AI inference.

---

The AI Inference Optimization Stack

flowchart TB

A["Model Optimization 🧠 "]-->
B["Inference Engine ⚡ "] -->
C["KV Cache 💾 "] -->
D["Continuous Batching 📦 "] -->
E["GPU Sharing 🧮 "] -->
F["Job Scheduling 🕘 "] -->
G["Intelligent Request Routing 🔀 "]-->
H["Autoscaling 📈 "] -->
I["Hardware & Networking 🖥️"]

classDef step fill:#E8F5E9,stroke:#2E7D32,stroke-width:2px,color:#000;

class A,B,C,D,E,F,G,H,I step;

Each layer contributes to one goal:

$$\textbf{Maximize Throughput}$$

while minimizing

$$\textbf{Latency} \qquad \textbf{Cost} \qquad \textbf{GPU Idle Time}$$

---

1. Model Optimization 🧠

The cheapest GPU cycle is the one you never have to execute.

Instead of optimizing infrastructure, optimize the model itself.

Techniques include:

1. Quantization

Instead of using 16-bit floating point numbers (FP16), use:

• INT8
• FP8
• INT4

Memory consumption changes from

$$16\text{-bit} \rightarrow 8\text{-bit}$$

$8/16 = 50\%$

which immediately halves memory usage.

Or even

$$16\text{-bit} \rightarrow 4\text{-bit}$$

$4/16 = 25\%$

reducing memory requirements by 75%.

Popular tools include:

• TensorRT-LLM
• GPTQ
• AWQ
• SmoothQuant

2. Smaller Models

Use a smaller model if it meets your accuracy requirements.

Instead of 70B model

run 8B model

or 3B model

Latency and memory drop dramatically.

3. Model Distillation

Train a smaller model that imitates a larger one.

Distilled models require less memory and fewer computing operations per request, delivering faster response times for end users.

flowchart LR
    Teacher[Large Model]<--> Student[Small Model]

4. Structured Pruning

Structured pruning is a neural network compression technique that removes entire groups of parameters—such as neurons, channels, filters, attention heads, or layers—rather than individual weights.

• arxiv: https://arxiv.org/abs/2305.17180

This produces a smaller, dense sub-network that runs faster on standard hardware without needing specialized sparse libraries

---

2. Faster Inference Engines ⚡

Even with the same model, different inference engines produce dramatically different performance.

Popular runtimes include:

• vLLM
• TensorRT-LLM
• SGLang
• LMDeploy
• DeepSpeed-Inference

These optimize:

• CUDA kernels
• Memory allocation
• Continuous batching
• KV Cache management

---

3. KV Cache Optimization 💾

The LLM spends time understanding the concept. It stores that understanding inside memory (the KV Cache).

Transformer models spend most of their time processing the prompt.

Conceptually,

$$Inference = Prefill + Decode$$

The expensive part is Prefill.

Instead of recomputing previous tokens,

modern systems reuse a KV Cache.

flowchart LR

Prompt
-->Prefill

Prefill
-->KVCache

KVCache
-->Decode

The larger the cache hit rate, the lower the latency.

Techniques include

• Prefix caching
• Shared KV cache
• PagedAttention (vLLM)
• Cache-aware scheduling (llm-d)

Example

Suppose someone asks:

Explain Kubernetes.

Now another user asks:

Explain Kubernetes scheduling.

Much of the work has already been done.

Instead of recomputing everything, the model reuses its cached information. That saves GPU computation.

---

4. Continuous Batching 📦

Instead of serving one request at a time, modern inference engines continuously merge incoming requests into a single batch.

Traditional inference serves one request at a time.

flowchart LR

Request1["Request 1 📁"]-->GPU["GPU 🧮"]
Request2["Request 2 📁"]-->GPU
Request3["Request 3 📁"]-->GPU

Modern inference engines continuously merge incoming requests.

flowchart LR

Request1["Request 1 📁"]-->Batch["Batch 🗃️"]
Request2["Request 2 📁"]-->Batch
Request3["Request 3 📁"]-->Batch
Request4["Request 4 📁"]-->Batch

Batch-->GPU["GPU 🧮"]

The GPU stays busy instead of waiting.

Higher utilization means better throughput.

---

5. GPU Sharing 🧮

Allow multiple workloads to safely share the same GPU.

Imagine GPU as Bus with 40 seats. Giving entire bus to one user is wasteful.

Similarly, One inference request rarely consumes an entire GPU.

Without sharing, large portions of the GPU remain idle while jobs wait in a queue.

flowchart TD
   GPU["GPU 🧮"] -->Workload1["Workload 1 🏃"]
   Waiting["Waiting ⏳"]--> Workload2["Workload 2 🏃🏻‍♂️‍➡️"]
   Waiting["Waiting ⏳"]--> Workload3["Workload 3 🏃‍♀️‍➡️"]
   Waiting["Waiting ⏳"]--> Workload4["Workload 4 🏃🏼‍♀️"]

We can share the bus with other users.

flowchart TD
GPU["GPU 🧮"] 
GPU --> Slice1["Slice 1 🧩"] --> Workload1["Workload 1 🏃"]
GPU --> Slice2["Slice 2 🧩"] --> Workload2["Workload 2 🏃🏻‍♂️‍➡️"]
GPU --> Slice3["Slice 3 🧩"] --> Workload3["Workload 3 🏃‍♀️‍➡️"]
GPU --> Slice4["Slice 4 🧩"] --> Workload4["Workload 4 🏃🏼‍♀️"]

Technologies like

• NVIDIA MIG: Hardware partitioning.
• Time Slicing: Multiple processes share the GPU.
• Dynamic Accelerator Slicer (DAS): Automatically creates GPU slices when needed.

Sharing a GPU: MIG vs. time-slicing vs MPS

A whole A100/H100 per Pod is wasteful for small inference jobs. Three ways to share:

MIG (Multi-Instance GPU)

Hardware partitioning, up to 7 isolated instances per A100/H100, each with dedicated memory and compute.

• Real Isolation: The isolation is real hardware isolation: one instance can't touch another's memory, can't steal its bandwidth, and can't crash it
• Quantum: You pick from fixed profiles (1g.5gb, 2g.10gb, 3g.20gb, 7g.40gb…), you can't get arbitrary fractions, and reconfiguring means draining the instance.
• Hardware bound: it only exists on datacenter silicon — A100, A30, H100, H200, Blackwell.

Time-slicing

. The scheduler round-robins: each process gets the whole GPU for a quantum, then context-switches to the next.

• Virtual Isolation: No memory isolation and no fault isolation
• Flexible: It runs on any GPU including consumer cards
• fine for dev or bursty low-priority work, dangerous for anything that needs guarantees.

MPS (Multi-Process Service)

MPS is also spatial like MIG, but soft: multiple processes submit into a shared CUDA context so their kernels run concurrently on the SMs — no round-robin context-switch overhead.

• "Spatial soft": Isolation is weaker than MIG (memory isn't hard-fenced, fault isolation is limited, though newer CUDA versions improved both)
• Flexible: It runs on any GPU including consumer cards
• Useful for many concurrent small inference requests

---

6. Better Scheduling 🕘

Instead of improving the GPU itself, improve how jobs enter the cluster.

• Think of it as a traffic controller for GPUs.

Imagine a busy restaurant. There are only eight tables.

Without a receptionist:

• Everyone rushes inside
• People compete for tables
• Some customers wait randomly

flowchart LR
    Customer1["Customer 1 🙍‍♂️"] --> Table1["Table 1 🍽️"]
    Customer2["Customer 2 🙍🏻‍♂️"] --> Table2["Table 2 🍽️"]
    Customer3["Customer 3 👩🏻‍💼"] --> Table1["Table 1 🍽️"]
    Customer4["Customer 4 👭👫"] --> Waiting["Waiting ⏳"]

Similar to having 8 GPUs, 1000 AI Jobs. Many jobs simply sit around waiting. Scheduling becomes unpredictable.

With a receptionist:

flowchart LR
    Receptionist["Receptionist 🧑‍💼"]
    Customer1["Customer 1 🙍‍♂️"] --> Receptionist
    Customer2["Customer 2 🙍🏻‍♂️"] --> Receptionist
    Customer3["Customer 3 👩🏻‍💼"] --> Receptionist
    Customer4["Customer 4 👭👫"] --> Receptionist
    Receptionist --> Queue["Queue ⏳"]
    Queue--> Table1["Table 1 🍽️"]
    Queue--> Table2["Table 2 🍽️"]

This is exactly what Kueue does.

Instead of immediately creating every Kubernetes Job, it decides:

• Which jobs should wait
• Which jobs should run first
• Which jobs are more important

Scheduling answers a completely different question.

Not

How fast should a job run?

Instead,

Which job should run first?

Examples include:

• Kubernetes Scheduler
• Kueue
• Volcano
• YuniKorn
• Run:ai
• Slurm

The paper focuses on Kueue, which introduces admission control, fairness, priorities, and resource borrowing for AI workloads.

---

7. Intelligent Request Routing 🔀

The problem with random routing is that similar prompts may end up on different servers. Both servers repeat the same expensive work.

Traditional Kubernetes Services distribute requests randomly. That works well for stateless applications.

flowchart LR
User["User 👤"] -->Pod1["Pod1 🐋"]
User["User 👤"] -->Pod2["Pod2 🐋"]
User["User 👤"] -->Pod3["Pod3 🐋"]

LLMs are not stateless. Similar prompts often share the same prefix. Modern inference platforms use cache-aware routing.

Instead of acting like a normal load balancer & asking

Which server is free?

they ask

Which server already knows most of this prompt?

Then send similar requests to the same server whenever possible.

It considers:

• Current queue length
• GPU load
• Cache availability
• Response time

flowchart LR

User["User 👤"] -->
InferenceGateway["Inference Gateway 🔀"]-->
InferencePool["Inference Pool ⏳"] -->
BestReplica["Best Replica 🏆"]

This dramatically improves efficiency.

Popular approaches include:

• Gateway API Inference Extension (GAIE)
• llm-d

---

8. Parallelism

Large models often don't fit on one GPU.

Some models simply don't fit on one GPU.

They must be distributed.

flowchart LR

LLM["Large Model 🔢"]
LLM --> Layer1["Layer 1 🔣"] --> GPU1["GPU1 🧮"]
LLM --> Layer2["Layer 2 🔣"] --> GPU2["GPU2 🧮"]
LLM --> Layer3["Layer 3 🔣"] --> GPU3["GPU3 🧮"]
LLM --> Layer4["Layer 4 🔣"] --> GPU4["GPU4 🧮"]

Popular approaches include:

• Tensor Parallelism
• Pipeline Parallelism
• Expert Parallelism (MoE)
• Sequence Parallelism

Libraries:

• Megatron-LM
• DeepSpeed
• TensorRT-LLM

---

9. Autoscaling ↔️

Running 100 GPUs all day is expensive.

Instead,

Morning = 10 GPUs
Noon    = 50 GPUs
Evening = 20 GPUs
Night   = 5 GPUs

Technologies include:

• KEDA
• KServe
• HPA
• Cluster Autoscaler

Resources scale with demand.

---

10. Hardware & Networking 🖥️

Distributed inference spends significant time communicating.

Sometimes software isn't the bottleneck.

Hardware matters.

GPU	Typical Use
L4	Small inference
L40S	Medium inference
A100	Enterprise inference and training
H100	High-performance LLM inference
B200	Next-generation large-scale inference

For distributed inference, networking becomes equally important.

Technologies include:

• NVLink
• NVSwitch
• RDMA
• GPUDirect RDMA
• AWS Elastic Fabric Adapter (EFA)

---

The Bigger Picture

AI infrastructure is evolving in the same way cloud infrastructure evolved a decade ago.

It's no longer enough to build better models.

We also need:

• Better schedulers
• Better inference engines
• Better GPU sharing
• Better request routing
• Better networking
• Better orchestration

The future isn't about one breakthrough.

It's about combining dozens of optimizations across the stack.

That's exactly why Kubernetes is becoming more than a container orchestrator.

It's rapidly evolving into the operating system for AI infrastructure.