What is TensorRT? 🖲

NVIDIA’s high-performance deep learning inference SDK designed to optimize and accelerate trained AI models on NVIDIA GPUs.

It takes trained models from frameworks such as PyTorch, TensorFlow, and ONNX, and optimizes them for high-performance deployment with support for mixed precision (FP32/FP16/BF16/FP8/INT8), dynamic shapes, and specialized optimizations for transformers and large language models (LLMs).

It is mainly used for:

• Low-latency inference
• High-throughput AI serving
• Real-time AI applications
• LLM inference optimization
• Edge AI deployments

TensorRT takes trained models from frameworks like PyTorch or TensorFlow and converts them into highly optimized GPU execution engines.

TensorRT Architecture

flowchart TD

    A["Trained Model 🎛<br/>PyTorch / TensorFlow"]
        --> B["ONNX Export"]

    B --> C["TensorRT Optimizer<br/><br/>• Layer Fusion<br/>• Quantization<br/>• Kernel Tuning"]

    C --> D["TensorRT Engine 🖲" ]

    D --> E["CUDA Runtime 📟"]

    E --> F["NVIDIA GPU 🧮" ]

---

How TensorRT Works Under the Hood

1. Model Import

TensorRT typically imports models using ONNX.

torch.onnx.export(model, sample_input, "model.onnx")

Supported sources:

• PyTorch
• TensorFlow
• ONNX
• Hugging Face Transformers
• TensorFlow-TRT integration

2. Graph Optimization

TensorRT analyzes the neural network computation graph and applies optimizations such as:

• Layer fusion
• Kernel auto-tuning
• Precision calibration
• Memory optimization
• Tensor layout optimization

Example:

    Conv + BatchNorm + ReLU
            ↓
    Single fused GPU kernel

This reduces:

• GPU memory reads/writes
• Kernel launch overhead
• Latency

Precision Optimization

TensorRT supports multiple precision modes:

Precision	Description
FP32	Standard floating point
FP16	Half precision for faster inference
INT8	Quantized inference for maximum speed
FP8	Newer ultra-efficient precision on modern GPUs

Lower precision:

• reduces memory usage
• increases throughput
• improves latency

FP32 vs FP16 vs INT8

Mode	Speed	Accuracy	Memory Usage
FP32	Slowest	Highest	Highest
FP16	Faster	Very close	Lower
INT8	Fastest	Slight drop possible	Lowest

Example:

config.set_flag(trt.BuilderFlag.FP16)

3. CUDA Kernel Selection

TensorRT benchmarks multiple CUDA kernels internally and selects the fastest implementation for the target GPU.

This is called:

Kernel Auto-Tuning

Different GPUs may produce different optimized engines.

4. Engine Generation

TensorRT builds a serialized inference engine.

serialized_engine = engine.serialize()

This engine contains:

• optimized kernels
• memory plans
• execution graphs
• scheduling strategies

The engine is GPU-specific.

5. Runtime Execution

Inference executes directly on the GPU with minimal CPU overhead.

context.execute_v2(bindings)

TensorRT optimizes:

• memory reuse
• asynchronous execution
• CUDA stream utilization
• batching

---

TensorRT + LLM Inference

TensorRT is heavily used for LLM acceleration.

NVIDIA provides:

• TensorRT-LLM
• FasterTransformer
• Triton Inference Server integration

Optimizations for LLMs include:

• KV cache optimization
• Attention kernel fusion
• Paged attention
• Tensor parallelism
• Continuous batching

TensorRT LLM Example

1. Convert ONNX model to TensorRT engine

trtexec --onnx=model.onnx --fp16 --saveEngine=model.engine

2. Python inference example

import tensorrt as trt

logger = trt.Logger(trt.Logger.WARNING)
runtime = trt.Runtime(logger)

with open("model.engine", "rb") as f:
    engine = runtime.deserialize_cuda_engine(f.read())

context = engine.create_execution_context()

TensorRT vs PyTorch Inference

Feature	PyTorch	TensorRT
Ease of use	Easier	More optimization setup
Training support	Yes	No
Inference speed	Good	Excellent
GPU optimization	General	Highly optimized
Production deployment	Moderate	Excellent
Latency	Higher	Lower

Common TensorRT Use Cases

• LLM serving
• Real-time computer vision
• Autonomous driving
• Recommendation systems
• Speech AI
• Video analytics
• Edge AI devices
• Robotics
• Medical imaging

TensorRT Ecosystem

Component	Purpose
CUDA	GPU compute platform
cuDNN	Deep learning kernels
TensorRT	Inference optimization
Triton Server	Model serving
TensorRT-LLM	LLM optimization
NCCL	Multi-GPU communication

Why TensorRT Is Fast

TensorRT improves performance through:

• Kernel fusion
• Reduced precision inference
• GPU-specific tuning
• Optimized memory reuse
• Parallel CUDA execution
• Reduced data movement
• Efficient batching

This can often produce:

• 2x–10x faster inference
• lower latency
• lower GPU memory usage

compared to standard framework inference.