LLM inference
LLM Inference Engines Explained: vLLM, SGLang, TensorRT-LLM
A technical whitepaper on how LLM inference engines actually work: prefill and decode, the KV cache, continuous batching, parallelism, quantization and kernels, followed by an engine-by-engine analysis of vLLM, SGLang, TensorRT-LLM, TGI, llama.cpp, Ollama, DeepSpeed and MLX, a reproducible benchmark method, Kubernetes reference architecture, cost engineering and a decision framework for choosing an engine.
108 min read
Read the whitepaper