LLM Inference Engines Explained: vLLM, SGLang, TensorRT-LLM

We wrote this as the reference we wished we'd had when choosing and running inference engines: how they work inside, why they're built the way they are, and where the major engines make different bets. It's aimed at the engineers who will run these systems and at whoever has to pick one.

A note on sources. Engines change fast, so treat this as a snapshot of official docs and repositories as of September 2026. Where we're giving our own opinion, we say so. There are no benchmark results in here; the numbers are back-of-envelope arithmetic from published specs.

1. Executive summary

LLM inference has two phases with opposite hardware profiles. Prefill processes the whole prompt in parallel and is limited by arithmetic throughput. Decode produces one token per sequence per step and, at the batch sizes most services run, is limited by how fast the GPU can read weights and cached attention state from memory. Almost every design decision in a modern inference engine follows from that asymmetry.

The central data structure is the KV cache: the attention keys and values for every token already processed, kept so that decode does not recompute them. It grows linearly with context length and with the number of concurrent sequences, and on a long-context workload it can exceed the size of the model itself. How an engine allocates, shares, evicts, compresses and moves the KV cache largely determines how many requests it can serve at once.

Modern engines combine a small set of techniques to keep the GPU busy:

• Continuous batching admits and retires requests at every decode step instead of waiting for a whole batch to finish.
• Paged KV cache allocates cache in fixed-size blocks, eliminating most fragmentation.
• Prefix caching reuses KV blocks across requests that share a prompt prefix, which is the dominant pattern in chat and agentic workloads.
• Chunked prefill splits long prompts so they can share a step with decode work without stalling other users.
• Disaggregated serving runs prefill and decode on separate GPU pools so each can be sized and tuned independently.
• Quantization reduces bytes per weight and per cache entry, which directly raises decode throughput on bandwidth-bound hardware.
• Speculative decoding uses a cheap draft to propose several tokens that the target model verifies in one pass.
• Fused, attention-aware kernels such as FlashAttention and CUDA Graphs cut memory traffic and launch overhead.

There is no single best engine. vLLM and SGLang are the general-purpose open-source standards for GPU serving, with SGLang particularly strong where prompts share long prefixes, as in agents. TensorRT-LLM extracts the most from NVIDIA hardware at the cost of being NVIDIA-only and more involved to operate. llama.cpp is the portable engine for CPUs, Apple Silicon, consumer GPUs and edge devices, and Ollama packages it for developers. MLX-LM is the native path on Apple Silicon. Hugging Face TGI is now in maintenance mode, and DeepSpeed-MII/FastGen has seen little activity since early 2024, so neither is a default choice for new deployments, though both introduced ideas the others adopted.

The right engine follows from the workload: its prompt and output length distributions, its concurrency, its prefix-sharing pattern, its latency targets, the hardware available, and the team that will operate it. The decision framework in section 21 is organised around those inputs rather than a ranking.

2. What is LLM inference?

Autoregressive generation

A decoder-only transformer maps a sequence of tokens to a probability distribution over the next token. Generation is a loop: run the model, pick a token from that distribution, append it, run again. Each iteration depends on the previous one, so output tokens for a single sequence cannot be produced in parallel. This sequential dependency is the root of nearly every performance problem in inference.

Prefill vs decode

Reading the question and writing the answer are different jobs. The model reads the whole prompt at once, which is prefill, then writes the answer one token at a time, which is decode.

Mechanically, prefill pushes all N prompt tokens through every layer in one forward pass. The weight matrices multiply an N-row activation matrix, which is a large matrix-matrix multiply (GEMM) that keeps the Tensor Cores busy. Decode pushes one new token per sequence through every layer. With a batch of B sequences, the weights multiply a B-row matrix, and when B is small that is close to a matrix-vector product: the GPU must read every weight from memory to do very little arithmetic with each one.

To put rough numbers on it: for a dense model with P parameters, the forward pass costs roughly 2P floating-point operations per token (one multiply and one add per weight), plus attention. Prefill amortises one read of the weights over N tokens. Decode amortises it over B tokens. The ratio of arithmetic to memory traffic, the arithmetic intensity, is therefore roughly proportional to N for prefill and to B for decode. Section 6 turns this into numbers.

Latency and throughput vocabulary

Some tools define TPOT to include the first token and some do not. Section 14 addresses the definitional traps.

Batch size, concurrency and context length

• Batch size is how many sequences are processed in one forward pass. In a continuously batched engine it changes every step.
• Concurrency is how many requests are in flight in the system, including those queued or waiting on the network. Concurrency bounds the achievable batch size but is not the same thing.
• Context length is prompt plus generated tokens for one sequence. It sets the KV cache footprint of that sequence and the cost of its attention.

These three interact through memory. A GPU holds the weights once, and every remaining byte is shared among the KV caches of all active sequences. Longer contexts mean fewer concurrent sequences, which means smaller decode batches, which means lower throughput per GPU.

GPU compute vs memory bandwidth

A GPU has two ceilings: how many operations per second its cores can execute, and how many bytes per second its memory can deliver. An NVIDIA H100 SXM is specified at roughly 989 dense BF16 TFLOPS and 3.35 TB/s of HBM3 bandwidth. Dividing one by the other gives about 295 FLOPs per byte: a kernel must do at least that much arithmetic for each byte it reads before compute, rather than memory, becomes the limit. This ratio is the ridge point of the roofline model. Prefill sits far to the right of it, and small-batch decode sits far to the left.

The KV cache in one paragraph

Attention lets each new token look at every previous token through their keys and values, which are projections computed per layer. Those projections for past tokens never change, so engines keep them in GPU memory rather than recompute them. That store is the KV cache. It turns decode from quadratic recomputation into a linear read, and in exchange it becomes the largest dynamic consumer of GPU memory. Section 7 covers it in depth.

3. Why inference engines exist

A few lines of Hugging Face Transformers will load a model and call generate(). That is the right tool for experiments, evaluation and fine-tuning. It is the wrong tool for serving many users, for reasons that compound.

GPU utilisation. A naive loop serves one request, or one static batch, at a time. During decode the GPU performs a matrix-vector product per layer and spends most of each step waiting on memory. Utilisation, measured as useful FLOPs against the hardware peak, is typically in the low single digits at batch size 1 (the arithmetic in section 6 shows why). Serving one user per GPU is the most expensive way to run a model.

Memory management. A straightforward implementation reserves a contiguous KV buffer per sequence sized for the maximum possible length. Most sequences never reach it. The PagedAttention paper measured that existing systems of the time used only 20.4% to 38.2% of KV cache memory for actual token state, with the rest lost to reservation and fragmentation (Kwon et al., SOSP 2023). Wasted cache is directly lost concurrency.

Scheduling. Requests arrive at random times with random lengths. Something must decide which to admit, which to pause when memory runs out, how to mix long prompts with in-progress decodes, and how to keep one long request from starving everyone else. A library call has no scheduler.

Batching. Batching is the only way to raise decode arithmetic intensity. Static batching, where a batch starts together and ends when its longest member ends, leaves GPU slots idle as shorter sequences finish. The Orca paper introduced iteration-level scheduling to fix this (Yu et al., OSDI 2022), and every serious engine now does some form of it.

KV-cache management. Beyond allocation, a server needs to share cache across requests with common prefixes, evict cold entries, optionally offload to CPU or storage, and sometimes transfer cache between machines. These are systems problems, not modelling problems.

Parallelism. A 70-billion-parameter model in BF16 needs about 140 GB for weights alone, more than any single 80 GB GPU holds. Splitting a model across GPUs, and across machines, requires coordinated communication inside every layer. That coordination lives in the engine.

Kernels. The operations in a transformer layer, written naively as separate framework calls, each read inputs from and write outputs to GPU memory. Fusing them, and writing attention to avoid materialising the full attention matrix, cuts memory traffic by large factors. Engines ship specialised kernels for exactly this.

An inference engine is the component that owns all of these at once: a scheduler, a memory manager, a parallel execution runtime and a kernel library behind an API.

4. Anatomy of an LLM inference request

Follow one chat completion request through a typical GPU inference engine.

Put simply, the API layer turns chat messages into token IDs, the scheduler finds how much of the prompt is already cached, the GPU computes the rest of the prompt in one pass and produces the first token, and then the request joins the running batch, producing one token per step until it hits a stop condition. Tokens are converted back to text and streamed as they arrive.

The stages in more detail:

1. API layer. Validates the request, applies authentication and rate limits, and parses sampling parameters: temperature, top-p, top-k, maximum tokens, stop sequences, and structured output constraints such as a JSON schema. Most engines expose an OpenAI-compatible HTTP API.
2. Chat template and tokenisation. Messages are rendered through the model's chat template (a Jinja template shipped with the tokenizer) into a single string, then encoded to token IDs. Tokenisation is CPU work and, for long prompts under high concurrency, can become a measurable part of TTFT. Several engines run it in a separate process for that reason.
3. Admission and scheduling. The request enters a waiting queue. On each engine step the scheduler decides which waiting requests to admit, based on available KV blocks, a per-step token budget, and a maximum number of concurrent sequences.
4. Prefix lookup. If prefix caching is on, the engine checks whether the leading blocks of the prompt already exist in the cache from an earlier request. Only uncached tokens need prefill.
5. Prefill. The uncached prompt tokens run through the model in one pass, or several chunks. Keys and values for every layer are written into the allocated KV blocks. The final position's logits yield the first output token.
6. Decode. The request joins the running batch. Each step, the engine runs one forward pass over one new token for each running sequence, reads the full KV cache of each sequence in attention, appends the new K and V, and samples the next token.
7. Sampling and constraints. Logits are adjusted by penalties and constraints (for example a grammar mask that forbids tokens that would break a JSON schema), then sampled.
8. Detokenisation and streaming. Token IDs are converted back to text incrementally. This is subtler than it sounds, because one token may be half a UTF-8 character or a word piece that only makes sense next to its neighbour. Text is streamed to the client as server-sent events.
9. Completion. On a stop token, stop string, or length limit, the sequence leaves the batch. Its KV blocks are freed or, with prefix caching, kept as reusable cache until evicted.

5. Inference engine architecture

Although the engines differ in language and emphasis, most GPU serving engines converge on the same components.

The short version: a frontend handles HTTP and text, an engine core decides what runs on each step and where its memory lives, and an executor drives one worker process per GPU that runs the actual kernels. The sampler turns logits into tokens and hands them back to the scheduler, which streams them out.

API layer. Terminates HTTP, validates input and streams output. Separating it from the engine core into its own process, as vLLM's V1 architecture does, keeps Python work such as JSON parsing and tokenisation from stealing time from the loop that feeds the GPU (vLLM architecture docs).

Request scheduler. Runs once per engine step. Its inputs are the waiting and running queues, the free KV block count and a token budget per step. It outputs the set of sequences, and the number of tokens for each, that the next forward pass will process. vLLM's V1 scheduler, for example, does not distinguish a prefill phase from a decode phase: every request simply has some number of tokens to compute, and the scheduler packs them into the budget.

Continuous batching. Implemented by the scheduler admitting new work and retiring finished sequences at every step, rather than per batch. Section 8 covers it.

Model executor. Prepares input tensors (token IDs, positions, block tables, slot mappings for new KV entries), launches the forward pass on all GPU workers, and collects logits. In multi-GPU setups it also coordinates the collective communication that tensor parallelism requires.

GPU workers. Typically one process per GPU, each holding its shard of the weights and its share of the KV cache, running the same step in lockstep.

KV-cache manager. Owns a pool of fixed-size blocks allocated once at startup from the memory left after weights and activation workspace. It maps each sequence to a list of blocks (a block table), keeps an index of cached prefixes, reference-counts shared blocks, and evicts when the pool is exhausted.

CUDA kernels. GEMMs for linear layers, attention kernels that understand paged block tables, fused kernels for normalisation, rotary embeddings, activation functions and quantised matrix multiplies, and collective communication through NCCL.

Sampling. Applies repetition and presence penalties, logit bias, grammar or JSON-schema masks, temperature, top-k and top-p, then draws a token. Constrained decoding libraries such as XGrammar precompute token masks so this step does not dominate the loop (Dong et al., 2024).

Token streaming. Incremental detokenisation and server-sent events. Streaming does not change the compute, but it changes perceived latency, and it forces an engine to handle client disconnects by aborting the sequence and freeing its cache.

One pattern worth pointing out: the engines that scale best treat the CPU side as a first-class performance problem. At high batch sizes, a decode step for a mid-sized model can take a few milliseconds of GPU time, so a few milliseconds of Python scheduling overhead per step halves throughput. SGLang's overlapped scheduler and vLLM's multiprocess engine core are both answers to that problem.

6. Prefill and decode in depth

Mathematical intuition

Consider a dense decoder with P parameters stored at b bytes each, L layers, hidden size d, and a KV cache of c bytes per token (section 7 derives c).

FLOPs. A forward pass over T tokens costs approximately:

FLOPs(T) ≈ 2 · P · T  +  attention term

The attention term for a prompt of length N grows as roughly 2 · L · N² · d for causal attention (score and value products). It is small next to 2PN for short prompts and dominates for very long ones.

Bytes moved. Every forward pass must read the weights at least once:

Bytes(step) ≈ P · b  +  (KV bytes read by attention)  +  activations

Arithmetic intensity is FLOPs divided by bytes. For a prefill of N tokens, ignoring attention:

I_prefill ≈ (2 · P · N) / (P · b) = 2N / b

For a decode step over a batch of B sequences, where each sequence also reads its own KV cache of length S:

I_decode ≈ (2 · P · B) / (P · b  +  B · S · c)

Why prefill is compute-bound

With BF16 weights (b = 2), a 2,000-token prompt gives an intensity near 2,000 FLOPs per byte, well above the H100 ridge point of about 295. The Tensor Cores, not memory, are the limit.

A quick estimate. Llama 3.1 8B has about 8.0 × 10⁹ parameters. Prefill of 2,000 tokens costs about 2 × 8.0 × 10⁹ × 2,000 = 3.2 × 10¹³ FLOPs. At an assumed 50% of the H100's 989 TFLOPS dense BF16 peak, that is 3.2 × 10¹³ / 4.9 × 10¹⁴ ≈ 65 ms. Real TTFT adds queueing, tokenisation, attention and kernel efficiency losses.

Why decode is memory-bandwidth-bound

At batch size 1, I_decode ≈ 2 / b, which is 1 FLOP per byte in BF16, about 300 times below the H100 ridge point. The GPU spends its time streaming weights.

Back of the envelope. Reading 16 GB of BF16 weights for Llama 3.1 8B at 3.35 TB/s takes about 4.8 ms, so a single sequence cannot exceed roughly 210 tokens per second on one H100, however fast the arithmetic units are. That is a ceiling from specifications, not a measurement, and real systems land below it.

Increasing the batch amortises the weight read: 64 sequences still read the weights once per step, so aggregate throughput rises almost linearly with B while weight traffic dominates. Two effects stop that growth:

1. KV traffic grows with the batch. Each sequence reads its whole KV cache every step. The B · S · c term in the denominator eventually dwarfs the weight term, and attention becomes the memory bottleneck even at large batch. Longer contexts reach this point sooner.
2. KV capacity caps the batch. The batch can only grow until the KV pool is full.

This is why decode throughput depends so heavily on KV cache efficiency and on bytes per weight, and why quantization and cache compression help decode far more than they help prefill.

So prefill is one big, efficient pass that fills the cache and produces the first token. Decode is a loop of small passes, each of which rereads the weights and the entire cache to produce a single token per sequence.

Chunked prefill

Problem. A 32,000-token prompt admitted into a running batch produces one enormous step. Every sequence sharing that step waits for it, so their inter-token latency spikes: a visible stall in streamed output.

Mechanism. Chunked prefill splits a long prompt into chunks that fit a per-step token budget, and schedules those chunks alongside the decode tokens of running sequences. Each step then carries a roughly constant amount of work: a mix of cheap decode tokens and a slice of prefill. The decode tokens get a smoother ITL, and the prefill chunk fills otherwise idle compute in a memory-bound step. Sarathi-Serve formalised this as stall-free batching (Agrawal et al., OSDI 2024); DeepSpeed-FastGen introduced a similar idea as Dynamic SplitFuse (Holmes et al., 2024).

Tradeoff. Chunking a prompt slightly increases its own TTFT, because it is spread over several steps and re-reads earlier chunks' KV. The chunk size, or token budget, is the knob between TTFT for new requests and ITL for running ones. vLLM V1 enables chunked prefill by default; SGLang, TensorRT-LLM ("chunked context") and llama.cpp all expose it.

Disaggregated prefill and decode

Even with chunking, prefill and decode compete for the same GPU and have opposite ideal configurations: prefill wants high compute and benefits from modest parallelism, while decode wants large batches, lots of memory for KV, and memory bandwidth. Disaggregated serving runs them on separate pools of GPUs.

One set of GPUs reads prompts and builds their caches, ships the cache to a second set of GPUs over a fast interconnect, and the second set generates the answer. Each set can then be scaled and tuned for its one job.

DistServe (Zhong et al., OSDI 2024) and Splitwise (Patel et al., ISCA 2024) established the approach, and Mooncake (Qin et al., 2024) described a KV-cache-centric production design at Moonshot AI. Today vLLM, SGLang and TensorRT-LLM all support prefill/decode disaggregation, typically moving KV through a transfer library such as NVIDIA NIXL or through LMCache, and orchestration layers such as NVIDIA Dynamo and llm-d coordinate the pools.

Is it worth it? Disaggregation adds a KV transfer on every request and an extra pool to operate. It pays when the workload has long prompts and strict ITL targets at scale, when the two phases would otherwise interfere badly, or when the model is large enough that prefill and decode benefit from different parallelism layouts, as with large mixture-of-experts models. For a single-node service with moderate prompts, chunked prefill usually captures most of the benefit at far lower complexity.

7. KV cache

What it stores

Think of it as the model's notes on everything it has read so far, kept so that it does not have to reread the conversation for every new word.

More precisely, it holds the key and value vectors produced by each layer's attention projections, for every layer and every token in the sequence. With multi-head attention each layer has H heads, each with its own K and V of dimension d_head. Grouped-query attention (GQA) shares each K/V head across several query heads, so the cache stores only H_kv heads (Ainslie et al., 2023). Multi-query attention (MQA) is the extreme case H_kv = 1 (Shazeer, 2019).

Some newer architectures change what gets stored. In DeepSeek's multi-head latent attention (MLA), the cache stores a low-rank compressed latent vector per token per layer, plus a small decoupled rotary component, instead of full per-head K and V. The model up-projects that latent inside attention (DeepSeek-AI, 2024). This changes both the memory formula and which attention kernels an engine needs, which is why MLA models initially ran well on only a few engines.

Memory calculation

For standard or grouped-query attention:

KV bytes per token = 2 × L × H_kv × d_head × bytes_per_element
KV bytes total     = KV bytes per token × Σ (context length of each active sequence)

The factor 2 is one K and one V.

Worked example 1: Llama 3.1 8B (L = 32, H_kv = 8, d_head = 128, from the model's published configuration), BF16 cache:

2 × 32 × 8 × 128 × 2 bytes = 131,072 bytes = 128 KiB per token
8,192-token context   →   1 GiB per sequence
128K-token context    →  16 GiB per sequence

Worked example 2: Llama 3.1 70B (L = 80, H_kv = 8, d_head = 128), BF16 cache:

2 × 80 × 8 × 128 × 2 bytes = 327,680 bytes = 320 KiB per token
8,192-token context   →  2.5 GiB per sequence
128K-token context    →   40 GiB per sequence

Worked example 3: DeepSeek-V3 with MLA (61 layers, a 512-dimension compressed KV latent plus a 64-dimension rotary key per token per layer), BF16:

(512 + 64) × 61 × 2 bytes = 70,272 bytes ≈ 68.6 KiB per token

Despite being a 671B-parameter model, DeepSeek-V3's per-token cache is about a fifth of Llama 3.1 70B's. Architecture choices in the model set the serving economics before any engine is involved.

Worked example 4: concurrency on one H100 80 GB, Llama 3.1 8B, BF16.

Usable memory at 90% utilisation target     ≈ 72 GiB
Weights (8.0B × 2 bytes)                    ≈ 15 GiB
Activations, CUDA graphs, workspace         ≈  4 GiB   (rough guess)
KV cache pool                               ≈ 53 GiB
Tokens that fit: 53 GiB / 128 KiB           ≈ 434,000 tokens
Sequences at 8K tokens each                 ≈ 53 concurrent
Sequences at 32K tokens each                ≈ 13 concurrent

This is the arithmetic an engine does at startup. vLLM, for example, profiles peak activation memory, subtracts weights and activations from the configured --gpu-memory-utilization fraction, and reports how many KV blocks, and so how many tokens, it can hold.

Why the KV cache becomes the bottleneck

1. Capacity. It caps concurrency, and so decode batch size, and so throughput per GPU.
2. Bandwidth. Each decode step reads every active sequence's entire cache. At long contexts, cache reads exceed weight reads.
3. Allocation. Sequences grow unpredictably. Reserving for the maximum wastes memory; growing contiguously fragments it.
4. Lifetime. Agentic and multi-turn workloads resend a growing conversation every turn. Without reuse, the same prefix is recomputed over and over.

Paged KV cache

The idea is simple: instead of reserving one long shelf for each conversation, cut memory into small equal boxes and hand a conversation a new box whenever it needs one.

In practice, PagedAttention divides the KV pool into fixed-size blocks, 16 tokens by default in vLLM, and gives each sequence a block table that maps logical token positions to physical blocks, analogous to virtual memory pages (Kwon et al., 2023). Blocks are allocated on demand as the sequence grows. Waste is limited to the unfilled part of each sequence's last block. The attention kernel reads K and V through the block table instead of from a contiguous tensor.

The indirection buys more than tidiness. Because physical blocks are referenced indirectly, two sequences can point at the same physical block. That makes copy-on-write sharing possible for parallel sampling and beam search, and it is the foundation for prefix caching. The cost is an extra indirection in the attention kernel and less contiguous memory access. Kernels such as FlashAttention and FlashInfer support paged layouts natively to keep that cost small.

Each sequence keeps a short list of which memory blocks hold its cache. Two sequences that start with the same prompt both point at the same physical block, so that prefix is stored once, and new blocks come from a shared free pool as sequences grow.

SGLang uses a token-granular memory pool with a configurable page size rather than fixed 16-token blocks, and TensorRT-LLM, TGI and DeepSpeed-FastGen ("blocked KV cache") all use paged or blocked designs. llama.cpp manages a unified KV buffer divided into cells shared among parallel slots, which serves the same purpose at the scale of a single machine.

Prefix caching

When requests share a prompt prefix, such as a system prompt, a tool list, few-shot examples, a long document, or the earlier turns of a conversation, the KV entries for that prefix are identical. Prefix caching keeps them and lets new requests skip their prefill.

• vLLM hashes each full block together with the hash of its parent block, so a block's identity depends on every token before it, and looks up a new prompt's blocks by hash. It is enabled by default in V1.
• SGLang RadixAttention stores cached prefixes in a radix tree keyed by token sequence, sharing any common prefix at token granularity and evicting leaves in LRU order (Zheng et al., 2024).
• TensorRT-LLM calls the feature KV cache reuse.
• llama.cpp reuses a slot's cache for the matching prefix of the next prompt in that slot (cache_prompt).

Prefix caching is correctness-neutral: results are the same as recomputing, up to floating-point non-determinism. It is not free in memory, because cached blocks occupy the pool until evicted, but engines evict unused cached blocks before refusing new work.

Why this matters for agents. For agentic workloads prefix caching is often the single largest lever. An agent loop resends the system prompt, tool definitions and the full growing trajectory on every step, so each call is mostly a prefix of the next. A cache hit converts that prefill into a lookup. This only works if requests from the same session reach the same replica, which is why prefix-aware routing (section 19) matters as much as the cache itself.

KV cache quantization

Storing K and V in FP8 instead of BF16 halves the cache, doubling the number of tokens that fit and halving the bytes attention reads per step. vLLM, SGLang and TensorRT-LLM support FP8 KV caches, TensorRT-LLM also supports INT8, llama.cpp offers quantised cache types such as q8_0 and q4_0 through --cache-type-k and --cache-type-v, and MLX-LM offers --kv-bits. Research methods such as KIVI push to 2 bits with per-channel key and per-token value quantization (Liu et al., 2024). Quality impact depends on the model and on whether scales are calibrated. Keys are typically more sensitive than values because of outlier channels.

Cache eviction and offload

When the pool is full, an engine must choose:

• Evict cached-but-unused prefix blocks, usually LRU. This is the cheap first step.
• Preempt a running sequence. Recompute its cache later, or swap it to CPU memory. vLLM V1 prefers recomputation, since recomputing via prefill is often cheaper than PCIe transfer.
• Offload to a slower tier. CPU DRAM, local NVMe or a distributed cache. LMCache and Mooncake-style stores extend the prefix cache beyond GPU memory so that a cache hit from DRAM can still beat recomputation for long prefixes.

Token-level eviction inside a sequence, keeping only important or recent tokens as in H2O or StreamingLLM, changes model outputs and belongs to approximation rather than caching (Zhang et al., 2023; Xiao et al., 2023). It is used for very long streams where exact attention over everything is not affordable.

8. Batching and scheduling

Static, dynamic and continuous batching

Static batching. Collect B requests, run them together from first token to last, return all results, start the next batch. Simple and predictable, and used by offline batch jobs. The batch runs as long as its longest member, so slots held by finished sequences sit idle, and a new request waits for the whole batch to end.

Dynamic batching. A server-side window, for example "up to 32 requests or 10 ms, whichever comes first", forms batches from whatever has arrived. This is how general-purpose model servers such as NVIDIA Triton batch classical models. It fixes the arrival problem but not the length problem: once formed, the batch still runs to its longest member.

Continuous batching, also called in-flight or iteration-level batching, makes the batching decision at every decode step (Yu et al., OSDI 2022). When a sequence finishes, its slot is filled on the next step by a waiting request. The batch is a rolling set of sequences at different positions, not a fixed group.

Look at what happens when B and D finish early. Under static batching their slots stay empty until A, the longest request, is done, and new arrivals E to H wait outside. With continuous batching, E and F take the free slots on the very next step, so the GPU always carries as many sequences as memory allows.

Continuous batching requires that the engine handle batches in which each sequence has a different length and position. That is exactly what the paged KV cache and variable-length attention kernels provide, which is why the two ideas arrived together.

Request scheduling

On each step the scheduler has to answer:

1. What to admit. Admit a waiting request only if there are KV blocks for at least its first chunk. Admitting too eagerly causes preemption later; too conservatively leaves memory idle.
2. How much prefill per step. The per-step token budget, max_num_batched_tokens in vLLM or chunked_prefill_size in SGLang, divides each step between prefill chunks and decode tokens.
3. What to preempt when memory runs out. Usually the most recently admitted sequence, so the oldest requests finish. Its cache is either dropped and recomputed later or swapped out.
4. In what order. First come, first served by default. vLLM supports a priority scheduling policy, and SGLang offers a longest-prefix-match policy that orders waiting requests to maximise prefix cache hits.

Fairness vs throughput

Throughput is maximised by keeping the batch as large as possible and preferring requests whose prefixes are cached, which may starve requests that do not share them. Latency for any one request is minimised by running it alone. Every scheduler sits somewhere between these, and the right point depends on the SLO:

• Interactive chat wants tight ITL and bounded TTFT, so smaller token budgets and a cap on concurrent sequences.
• Batch and offline work wants maximum tokens per GPU-hour and tolerates long latency, so large budgets and maximum concurrency.
• Mixed tenants need isolation: separate replicas, priority classes, or per-tenant rate limits at the gateway. Research systems such as the Virtual Token Counter scheduler apply fair queueing to token consumption (Sheng et al., OSDI 2024), but in production, fairness is more often enforced by the gateway than by the engine.

Short vs long requests

Mixing a few very long prompts with many short chats is the classic hard case. Without chunked prefill, a long prompt stalls every decode stream in the batch. With it, the long prompt's TTFT stretches while it trickles through in chunks. Common responses:

• Keep chunked prefill on and tune the token budget to the ITL target.
• Route long-context requests to a dedicated pool with different limits, or to a disaggregated prefill tier.
• Cap maximum model length per deployment so a single request cannot consume the cache.
• Use priority scheduling so interactive requests are admitted ahead of batch ones.

A practical note. In our view, many latency problems in LLM serving turn out to be scheduling problems rather than model problems: the queue grew because admission was limited by KV capacity, and TTFT rose for everyone. Watch waiting-queue depth and KV utilisation, not just GPU utilisation.

9. GPU parallelism

A model is split across GPUs for one of two reasons: it does not fit on one, or one is not fast enough. Each form of parallelism splits a different dimension and pays a different communication cost.

Tensor parallelism (TP)

Here every layer is cut into slices, and each GPU computes its slice of every layer at the same time.

The standard recipe, Megatron-style tensor parallelism, splits the first linear layer of each MLP by columns and the second by rows, and splits attention by heads, so that each layer needs only one all-reduce in the forward pass for the MLP and one for attention (Shoeybi et al., 2019). With TP = 4, each GPU holds a quarter of the weights and a quarter of the KV heads, and reads a quarter of the bytes per step, so decode latency falls.

The catch is communication. Two all-reduces per layer per step, each moving the activations for every token in the step, put TP on the critical path of every forward pass. Over NVLink (900 GB/s per H100) that is affordable; over PCIe or between nodes it usually is not. TP degree is also limited by the number of KV heads: with 8 KV heads and TP = 16, heads must be replicated. In practice TP stays inside one NVLink domain, typically 2, 4 or 8 GPUs.

All four GPUs work on the same layer at the same moment, each on its own share of heads and columns, then combine their partial results with an all-reduce before the next layer. It is fast only because NVLink makes that combine cheap.

Pipeline parallelism (PP)

Assign contiguous groups of layers to different GPUs or nodes. GPU 0 runs layers 1 to 20 and passes activations to GPU 1 for layers 21 to 40, and so on. Communication is point-to-point and small, only activations at stage boundaries, so PP tolerates slower links and is the usual way to span nodes. Its cost is the pipeline bubble: with one batch, only one stage works at a time. Engines keep multiple micro-batches in flight to fill the pipeline. PP increases throughput and capacity but not single-request latency.

Here the model is split by depth across two machines. Each machine uses tensor parallelism internally over its fast NVLink, and only the activations crossing the midpoint travel over the slower network.

Data parallelism (DP)

Run independent full replicas and spread requests across them. No communication during inference, linear scaling of throughput, and the simplest fault isolation. It needs the model to fit on each replica's GPUs. Most production scaling is data parallelism behind a load balancer. In mixture-of-experts serving, attention data parallelism, where attention runs data-parallel while experts run expert-parallel, is used to avoid replicating the KV cache for small numbers of KV heads (SGLang and vLLM docs).

Expert parallelism (EP)

A mixture-of-experts (MoE) layer replaces one large MLP with many expert MLPs and a router that sends each token to a few of them (Fedus et al., 2021; DeepSeek-AI, 2024). Expert parallelism places different experts on different GPUs. Each MoE layer then needs two all-to-all exchanges, dispatching tokens to the GPUs holding their chosen experts and combining the results. Load imbalance between popular and unpopular experts is the central problem; engines replicate hot experts (the Expert Parallelism Load Balancer, EPLB, pattern) to smooth it. Wide EP across many GPUs is how the largest open MoE models, such as DeepSeek-V3, are served efficiently, with communication libraries such as DeepEP optimising the all-to-all.

Context parallelism (CP)

For very long sequences, split the sequence itself across GPUs. Each GPU holds the KV cache for a slice of the context and attention is computed by exchanging partial results, as in Ring Attention (Liu et al., 2023). CP reduces per-GPU KV memory and prefill time for a single million-token request, at the cost of communication inside attention. It is most useful for prefill of extremely long inputs.

Comparison

Multi-GPU and multi-node inference

Within a node, use TP across the NVLink domain for models that do not fit on one GPU, then add DP replicas. Compare TP = 8 with two replicas at TP = 4: the second often gives more total throughput when the model fits in four GPUs, because it pays half the all-reduce cost and doubles the number of independent schedulers. Measure both.

Across nodes, combine TP inside each node with PP or EP across nodes, over InfiniBand or RoCE with GPUDirect RDMA. Multi-node serving adds failure modes: one GPU fault takes down a replica spanning 16 GPUs, startup must coordinate all ranks, and rolling updates must replace the whole group. On Kubernetes, the LeaderWorkerSet API exists to manage exactly such multi-pod replicas as one unit.

10. Quantization

The basic idea is to store each number with fewer bits. The model shrinks, more of it fits, and the GPU spends less time reading it.

In practice, weights, and optionally activations and the KV cache, are mapped from 16-bit floating point to a lower-precision format with a scale factor per tensor, per channel or per group. Weight-only schemes (W4A16, W8A16) dequantise weights on the fly inside the matmul kernel and mainly help memory-bound decode. Weight-and-activation schemes (W8A8 in INT8 or FP8, W4A4 in FP4) run the matmul itself in low precision on Tensor Cores and also speed up compute-bound prefill, but only on hardware with native support for that format.

The hard part is outliers. LLM activations contain a few channels with magnitudes far larger than the rest, and naive per-tensor INT8 quantization of activations clips them or wastes resolution (Dettmers et al., 2022). The methods below differ mainly in how they handle outliers and how much calibration they need.

Formats

FP8 formats are specified in Micikevicius et al. (2022). E4M3 is used for weights and activations in inference; E5M2 has more range and less precision. NVFP4 and the OCP MX formats (MXFP8, MXFP4) attach a shared scale to small blocks of values, which is what makes 4-bit floating point usable (NVIDIA and OCP specifications).

Methods

GPTQ quantises weights one layer at a time, using approximate second-order information from a small calibration set to adjust the remaining weights so that each layer's output error stays small. It produces 3- and 4-bit weight-only models (Frantar et al., 2022).

AWQ (activation-aware weight quantization) observes that a small fraction of weight channels matter disproportionately, identified by the size of the activations they multiply, and scales those channels before quantization to protect them. It needs no backpropagation and generalises well across domains (Lin et al., 2023).

SmoothQuant makes activations easier to quantise by mathematically migrating their outlier magnitude into the weights through a per-channel scale, enabling W8A8 INT8 with both matmul operands in INT8 (Xiao et al., 2022).

GGUF is not a quantization algorithm. It is llama.cpp's single-file model format, which stores weights, tokenizer and metadata together and supports a family of block quantization types: legacy Q4_0 and Q8_0, "K-quants" such as Q4_K_M and Q5_K_M, and "I-quants" such as IQ2_XS down to about 1.5 bits per weight, optionally guided by an importance matrix computed on calibration text (llama.cpp). GGUF is designed for fast memory-mapped loading and for CPU, Metal and consumer GPU inference. vLLM can load some GGUF files, but GGUF is not the natural format for datacenter GPU engines.

Engines also consume pre-quantised checkpoints in vendor formats such as those produced by NVIDIA Model Optimizer, the compressed-tensors format produced by LLM Compressor, and TorchAO (vLLM docs).

Tradeoffs

"Bytes saved" is not a measured speed-up. It is the upper bound implied by memory traffic, and real gains depend on kernel quality and batch size. At large batch, decode moves toward compute-bound and weight-only quantization gains shrink.

Our advice. Measure quality on your own tasks, not only on perplexity. Quantization errors are uneven: long-context retrieval, code, maths and tool-call formatting can degrade before general chat does. For agentic systems, run the agent evaluation suite against the quantised model before swapping it in. FP8 on Hopper-class or newer hardware is usually the first quantization step to take; INT4 is where the memory-constrained cases start.

11. CUDA and kernel optimization

CUDA kernels. A kernel is a function that runs on the GPU across a grid of thread blocks. Each block runs on one streaming multiprocessor (SM), threads execute in groups of 32 called warps, and each SM has fast on-chip shared memory and registers. The central discipline of GPU programming is keeping data on-chip and reusing it as many times as possible before going back to HBM.

GEMM. General matrix multiply is where most transformer FLOPs live: QKV projections, attention output, and the MLP. Libraries such as cuBLAS and CUTLASS tile the matrices so that each tile is loaded into shared memory once and reused across many multiply-accumulates on Tensor Cores. For small decode batches the "matrix" on one side has only B rows, so engines use specialised skinny-GEMM and GEMV kernels, and for quantised weights, mixed-precision kernels such as Marlin that dequantise INT4 weights in registers and feed Tensor Cores (Frantar et al., 2024).

Attention kernels. Naive attention materialises an N × N score matrix in HBM, applies softmax, and multiplies by V: three trips through memory, quadratic in length. Serving adds complications: variable lengths within a batch, paged block tables, prefix sharing, grouped-query and latent attention, and FP8 caches. Libraries such as FlashInfer provide attention kernels designed specifically for these serving patterns (Ye et al., 2025).

FlashAttention.

The trick is to compute attention in small tiles that fit on-chip, so the giant score matrix is never written to memory.

In detail, FlashAttention tiles Q, K and V, computes softmax incrementally using a running maximum and running sum (online softmax), and fuses the whole computation into one kernel. It is exact, not an approximation, and it turns attention from memory-bound into much closer to compute-bound (Dao et al., 2022).

Later versions pushed further. FlashAttention-2 improved parallelism across sequence length and work partitioning between warps (Dao, 2023). FlashAttention-3 targets Hopper, using asynchronous Tensor Core instructions (WGMMA) and the Tensor Memory Accelerator (TMA) to overlap data movement with compute, and supports FP8 (Shah et al., 2024). For decode, where one query attends to a long cache, split-KV variants (for example Flash-Decoding) parallelise across the cache length so that a single sequence can occupy the whole GPU.

Fused kernels. Every separate kernel reads its input from HBM and writes its output back. Fusing adjacent element-wise and reduction operations, such as residual add with RMSNorm, rotary embedding with the QKV write into the paged cache, SiLU with the gate multiply, or dequantise with GEMM, removes intermediate round-trips. Compiler approaches such as torch.compile with Triton generate some fusions automatically; hand-written kernels cover the rest.

Memory coalescing. When the 32 threads of a warp access consecutive addresses, the hardware combines them into a few wide transactions. Scattered access, such as naive reads through a paged block table, wastes bandwidth. Paged attention kernels lay out blocks so that each warp still reads contiguous runs.

GPU occupancy. Occupancy is the fraction of an SM's maximum resident warps that are actually active. Higher occupancy lets the SM hide memory latency by switching warps. Register and shared-memory use per thread limits it. Decode kernels at small batch often have too little parallelism to fill all SMs, which is one reason split-KV attention and persistent kernels exist.

Tensor Cores. Dedicated matrix-multiply-accumulate units. Each generation added formats: FP16 on Volta, BF16 and TF32 on Ampere, FP8 on Hopper and Ada, FP4 on Blackwell. Peak throughput is available only to operations shaped as matrix multiplies in supported formats, which is why quantised formats without Tensor Core support speed up memory traffic but not arithmetic.

Kernel launch overhead and CUDA Graphs. Each kernel launch costs a few microseconds of CPU and driver time. A decode step for a 32-layer model may launch hundreds of kernels, so at small batch the GPU can finish each kernel faster than the CPU can launch the next, and sits idle. CUDA Graphs record the whole sequence of launches once and replay it with a single call. Because a captured graph has fixed shapes, engines capture graphs for a set of batch sizes and pad each step up to the nearest one. vLLM uses piecewise graphs, capturing everything except attention, and full graphs where the attention backend allows. Graph capture is a large part of engine startup time and of the memory reserved outside the KV cache.

12. Major inference engines

We cover each engine under the same headings so they're easy to compare side by side. Capability details come from each project's own docs and repositories as of September 2026; the strengths, limitations and use cases are our opinion.

vLLM

Architecture. An open-source engine that began at UC Berkeley with the PagedAttention paper and is now a PyTorch Foundation project. Its V1 architecture, the default since 2025, separates an API server process from an EngineCore process that runs a busy loop of schedule, execute and update, communicating over ZeroMQ so that HTTP handling, tokenisation and detokenisation never block the GPU loop. The executor drives one worker per GPU, each with a model runner that prepares inputs, runs the model compiled through torch.compile, and replays CUDA graphs (vLLM docs).

Core optimizations. PagedAttention; automatic prefix caching; chunked prefill; piecewise and full CUDA/HIP graphs; pluggable attention backends including FlashAttention, FlashInfer, TRTLLM-GEN, FlashMLA and Triton; speculative decoding with n-gram, suffix, EAGLE-family and other drafters; a large library of quantised kernels.

Scheduling. A unified token-budget scheduler with no separate prefill and decode phases. Each step packs decode tokens and prefill chunks up to max_num_batched_tokens and max_num_seqs. FCFS by default, with a priority policy available. Preemption by recomputation.

KV cache. Block-based paged cache with hash-chained blocks for prefix caching and LRU eviction of unreferenced cached blocks. The KV connector interface supports offload and cross-instance transfer through integrations such as LMCache and NIXL, which is how disaggregated prefill and decode is built.

Hardware. NVIDIA GPUs, AMD GPUs through ROCm, and x86, ARM and PowerPC CPUs in the main project; Google TPU, Intel Gaudi, IBM Spyre, Huawei Ascend, Apple Silicon and others through hardware plugins.

Quantization. FP8, MXFP8 and MXFP4, NVFP4, INT8, INT4, GPTQ and AWQ, GGUF, compressed-tensors, ModelOpt and TorchAO checkpoints, plus FP8 KV cache.

Distributed inference. Tensor, pipeline, data, expert and context parallelism; multi-node with Ray or native multiprocessing; disaggregated prefill, decode and encode.

Strengths. The broadest model coverage of any GPU engine, usually with day-zero support for major open releases; a large contributor base; the widest hardware reach among datacenter engines; an OpenAI-compatible server that is straightforward to run; and a clear integration point for the Kubernetes ecosystem, including llm-d, KServe and the Gateway API Inference Extension.

Limitations. A large surface area that moves quickly: flags, defaults and metrics names change between minor releases, so upgrades need testing. Performance on a specific model and GPU is not always the best available, and a hand-tuned TensorRT-LLM or SGLang configuration can beat it. Startup time with compilation and graph capture can be long for large models, which matters for autoscaling.

Ideal use cases. The default for general-purpose GPU serving of open models, on NVIDIA or AMD, from one GPU to multi-node, when model breadth and ecosystem integration matter.

SGLang

Architecture. Developed from LMSYS research and now an independent open-source project, SGLang pairs a frontend language for programming multi-call LLM workflows with a high-performance serving runtime. The runtime has a tokenizer manager, a scheduler process per data-parallel rank that drives tensor-parallel workers, and a detokenizer manager, connected by inter-process messaging (Zheng et al., 2024; SGLang docs).

Core optimizations. RadixAttention prefix caching; a zero-overhead batch scheduler that overlaps CPU scheduling of the next batch with GPU execution of the current one; FlashInfer and custom attention kernels; speculative decoding with EAGLE-2 and EAGLE-3 and multi-token prediction; structured output through grammar backends such as XGrammar; extensive optimisation for large MoE models, including DeepEP-based expert parallelism and DP attention.

Scheduling. Continuous batching with chunked prefill and a cache-aware scheduling policy: waiting requests can be ordered by longest prefix match to raise the cache hit rate. A separate router, the SGLang model gateway, performs cache-aware load balancing across replicas.

KV cache. A token-granular memory pool, with configurable page size, indexed by a radix tree over token sequences. Any shared prefix at token granularity is reused automatically, and eviction is LRU over tree leaves. Hierarchical caching extends it to host memory.

Hardware. NVIDIA from A100 through Blackwell, including selected RTX cards; AMD Instinct MI300X, MI325X, MI350X and MI355X; Google TPU; Intel GPUs and Xeon CPUs; Apple Silicon through Metal and MLX; Huawei Ascend; and others (SGLang repository).

Quantization. FP8, FP4, INT4 AWQ and GPTQ, INT8, and FP8 KV cache, loading common pre-quantised formats.

Distributed inference. Tensor, pipeline, data, expert and DP-attention parallelism; multi-node; prefill/decode disaggregation.

Strengths. Excellent performance where requests share prefixes: multi-turn chat, agents, few-shot pipelines, tree search and RL rollouts. Strong MoE and DeepSeek-class serving at large scale. A compact, readable Python codebase that is relatively easy to profile and extend.

Limitations. Model coverage, while broad, generally trails vLLM for less popular architectures. Its benefits depend on the prefix-sharing pattern of the workload; on independent single-turn prompts the advantage over other engines narrows. Operational tooling and defaults evolve quickly, as with vLLM.

Ideal use cases. Agentic and multi-turn workloads, structured output-heavy pipelines, RL rollout generation, and large MoE deployments.

NVIDIA TensorRT-LLM

Architecture. NVIDIA's open-source inference library. Since the 1.0 release its primary architecture is built on PyTorch, with a high-level Python LLM API, a runtime executor with in-flight batching, and trtllm-serve as an OpenAI-compatible server. The earlier workflow of compiling a model into a TensorRT engine remains available for some cases. It integrates with Triton Inference Server and NVIDIA Dynamo (TensorRT-LLM repository and docs).

Core optimizations. Hand-tuned CUDA kernels for each NVIDIA architecture, including FP8 and FP4 GEMMs and attention on Hopper and Blackwell; fused MoE kernels; multiblock attention for long sequences; CUDA graphs; overlap scheduling; speculative decoding including draft-target, n-gram, Medusa, EAGLE, ReDrafter and multi-token prediction.

Scheduling. In-flight batching with chunked context (NVIDIA's term for chunked prefill) and configurable capacity policies that either guarantee no eviction or maximise utilisation.

KV cache. Paged KV cache with KV cache reuse across requests, priority-based eviction, host-memory offload, and INT8 and FP8 cache formats.

Hardware. NVIDIA GPUs only: Ampere, Ada, Hopper and Blackwell datacenter parts, plus supported RTX cards.

Quantization. FP8, NVFP4, INT4 AWQ, INT8 SmoothQuant, W4A8 and others, typically produced by NVIDIA Model Optimizer.

Distributed inference. Tensor, pipeline, expert (including wide EP), context and attention-data parallelism; multi-node; disaggregated serving.

Strengths. Typically the highest attainable performance on NVIDIA hardware, especially for new architectures and low-precision formats, because the kernels are written by the hardware vendor alongside the silicon. First access to Blackwell-specific features such as NVFP4.

Limitations. NVIDIA-only by design. More configuration surface and a steeper learning curve than vLLM or SGLang, and the fastest paths often require specific container versions, checkpoint conversion or quantization steps. Model coverage is narrower and new architectures can arrive later than in community engines.

Ideal use cases. Large, stable NVIDIA fleets where a few percent of efficiency is worth engineering time; latency-critical production on Hopper and Blackwell; deployments standardised on NVIDIA's stack, including Dynamo and Triton.

Hugging Face Text Generation Inference (TGI)

Architecture. A Rust router that handles HTTP, validation, queueing and continuous batching, talking over gRPC to Python model shards that run the model with tensor parallelism. It later added multiple backends, including TensorRT-LLM and llama.cpp.

Status. The TGI repository states that the project is now in maintenance mode, accepting only minor bug fixes, documentation and lightweight maintenance, and recommends vLLM, SGLang and local engines such as llama.cpp or MLX going forward (TGI README).

Core optimizations, scheduling, KV cache. Continuous batching, paged attention, FlashAttention, prefix caching, and speculative decoding through Medusa and n-gram methods.

Hardware. NVIDIA GPUs, AMD Instinct MI210 and MI250 and later, AWS Inferentia, Intel GPUs, Intel Gaudi and Google TPU.

Quantization. bitsandbytes, GPTQ, EETQ, AWQ, Marlin and FP8.

Distributed inference. Tensor parallelism within a node.

Strengths. A mature, well-instrumented server with Prometheus metrics and OpenTelemetry tracing, easy Docker deployment, and many existing production deployments.

Limitations. Maintenance mode means new models, hardware and optimisations will not arrive. No multi-node serving.

Ideal use cases. Existing deployments that are stable and meet their targets. Plan migration for anything that will need new models. Not recommended for new deployments.

llama.cpp

Architecture. A C/C++ implementation of LLM inference on the GGML tensor library, with no required dependencies. It ships a CLI, llama-server (an HTTP server with an OpenAI-compatible API and a web UI), benchmarking tools and quantization tools. Models use the GGUF format. Compute runs through pluggable GGML backends (llama.cpp repository).

Core optimizations. Hand-written SIMD kernels for x86 (AVX2, AVX-512, AMX) and ARM (NEON, SVE); Metal, CUDA, HIP, Vulkan, SYCL and other GPU backends; a wide range of block-quantised types; memory-mapped model loading; partial GPU offload by layer (-ngl); FlashAttention (-fa); speculative decoding with a draft model.

Scheduling. llama-server runs a fixed number of parallel slots (-np) with continuous batching across them. It is designed for one machine and modest concurrency rather than hundreds of concurrent users.

KV cache. A unified KV buffer of cells shared across slots, with per-slot prompt caching that reuses the matching prefix of a new prompt, saving and restoring slot state, and quantised cache types (--cache-type-k, --cache-type-v).

Hardware. Practically everything: x86 and ARM CPUs, Apple Silicon through Metal, NVIDIA through CUDA, AMD through HIP and Vulkan, Intel GPUs through SYCL, and other accelerators through community backends.

Quantization. GGUF types from roughly 1.5 to 8 bits per weight, including K-quants and I-quants with importance-matrix calibration.

Distributed inference. Multi-GPU on one host by splitting layers or rows across devices (--split-mode); an RPC backend to spread work across machines, documented as experimental and not for untrusted networks.

Strengths. Runs where nothing else does: laptops, desktops, CPU-only servers, edge devices and phones. Fast to start, a single binary, easy to embed in applications, and has excellent low-bit quantization. Offloading lets models larger than VRAM run at all.

Limitations. Not built for high-concurrency multi-tenant GPU serving; throughput on datacenter GPUs at high concurrency is well below the engines built for it. Limited distributed support. GGUF conversions of new models depend on community work.

Ideal use cases. Local and edge inference, CPU-only servers, Apple Silicon, consumer GPUs, embedded use inside applications, and air-gapped deployments of small models where simplicity matters more than peak throughput.

Ollama

Architecture. A Go daemon and CLI that manages model download, storage and lifecycle, and serves a REST API plus OpenAI-compatible endpoints. It runs models through GGML-based runners, historically built on llama.cpp and now including Ollama's own engine for newer architectures, and on Apple Silicon it began previewing MLX as the inference backend in March 2026 with version 0.19, for Macs with 32 GB or more of unified memory (Ollama repository and release announcement).

Core optimizations. Those of its underlying runners, plus automatic memory estimation to decide how many layers to offload to the GPU, and model keep-alive so a warm model is reused across requests.

Scheduling. A per-model parallel request limit (OLLAMA_NUM_PARALLEL), a queue, and a limit on simultaneously loaded models (OLLAMA_MAX_LOADED_MODELS), with models loaded on demand and unloaded after an idle timeout.

KV cache. Inherited from the runner, with optional KV cache quantization settings.

Hardware. macOS on Apple Silicon, Linux and Windows on NVIDIA and AMD GPUs, and CPUs.

Quantization. GGUF quantised models from its library or imported from Hugging Face; a Modelfile describes model, template, parameters and adapters.

Distributed inference. None across machines.

Strengths. The easiest way to run an open model locally. Excellent developer experience: ollama run <model> and an API on localhost.

Limitations. Designed for a single user or small team, not multi-tenant production. Less control over scheduling, batching and memory than the underlying engines. Behaviour changes when the runner changes, so pin versions.

Ideal use cases. Developer laptops, prototypes, local agents and internal tools with light concurrency.

DeepSpeed-Inference and DeepSpeed-MII

Architecture. DeepSpeed-Inference (2022) added inference to Microsoft's DeepSpeed training library, with kernel injection that swaps transformer layers for optimised fused kernels, tensor parallelism, and ZeRO-Inference for offloading weights of very large models to CPU or NVMe (Aminabadi et al., 2022). DeepSpeed-MII is the serving layer, and DeepSpeed-FastGen (2024) its continuous-batching engine (Holmes et al., 2024).

Core optimizations. Blocked KV cache, continuous batching, Dynamic SplitFuse (splitting long prompts and fusing them with decode tokens to keep forward-pass sizes consistent, essentially chunked prefill) and custom CUDA kernels.

Hardware. NVIDIA GPUs with compute capability 8.0 or higher (Ampere and later) for FastGen's precompiled kernels.

Quantization. Primarily FP16 and BF16 for FastGen, with separate DeepSpeed work on low-bit formats.

Distributed inference. Tensor parallelism.

Status. DeepSpeed-MII's most recent major release announcement dates from January 2024, and its supported model list centres on 2023-era architectures. DeepSpeed remains active for training.

Strengths. ZeRO-Inference remains a documented way to run a model far larger than GPU memory at low throughput, for offline work. Dynamic SplitFuse influenced the chunked-prefill designs now standard elsewhere.

Limitations. Little recent development for serving, limited coverage of current model families, and no modern features such as prefix caching or disaggregation.

Ideal use cases. Existing DeepSpeed-based pipelines, and research. Not recommended for new serving deployments.

MLX and MLX-LM

Architecture. MLX is Apple's array framework for Apple Silicon, with lazy evaluation, a NumPy-like API, and operations that run on the GPU through Metal without copying, because CPU and GPU share unified memory. MLX-LM is the companion package for LLM generation, quantization, fine-tuning and serving (MLX and MLX-LM repositories).

Core optimizations. Metal kernels tuned for Apple GPUs, including quantised matmuls; zero-copy unified memory; a rotating fixed-size KV cache for long generations; prompt caching.

Scheduling. mlx_lm.server exposes an HTTP API modelled on the OpenAI chat API and can batch requests, except that with a quantised KV cache (--kv-bits) it processes requests one at a time.

KV cache. Standard and rotating caches, prompt cache files that can be saved and reused, and optional KV quantization.

Hardware. Apple Silicon Macs only.

Quantization. Group-wise low-bit quantization, typically 4-bit or 8-bit, with conversion and upload tooling for Hugging Face.

Distributed inference. mx.distributed supports distributed inference and fine-tuning across multiple Macs.

Strengths. The native, best-integrated path on Apple Silicon, and one that others are adopting: Ollama's MLX backend and SGLang's Apple Silicon support both build on it. Large unified memory, up to 512 GB on an M3 Ultra, lets a single Mac hold models that would need several discrete GPUs.

Limitations. Apple-only. The MLX-LM documentation states that its server "is not recommended for production as it only implements basic security checks." Apple GPUs have lower memory bandwidth and much lower compute than datacenter GPUs, so prefill of long prompts is slow.

Ideal use cases. Local development and experimentation on Macs, on-device applications, and private single-user inference of large models on high-memory Macs.

Related systems worth knowing

Several systems sit around or above these engines rather than replacing them (each project's documentation):

• NVIDIA Dynamo, a datacenter-scale orchestration layer for disaggregated serving, KV-aware routing and KV transfer across vLLM, SGLang and TensorRT-LLM workers.
• llm-d, a Kubernetes-native distributed inference stack built on vLLM, with prefix-cache-aware scheduling and disaggregation. Covaratech's earlier piece on llm-d and HAMi covers its architecture.
• KServe, a Kubernetes model serving control plane that can run these engines as runtimes.
• NVIDIA Triton Inference Server, a general model server that hosts TensorRT-LLM and vLLM backends among others.
• LMCache and Mooncake, KV cache stores that extend prefix caching to CPU memory and storage and move KV between instances.

13. Detailed comparison

Legend: Yes means documented and supported in the main project. Partial means supported with significant limits noted in the cell. No means not supported or not documented as of late September 2026. These tables summarise; the sections above carry the nuance.

Architecture and core serving features

Parallelism and hardware

Maturity and operations

Reading the tables

Rows of "Yes" hide large differences in how well a feature performs. Three tradeoffs matter more than the checkmarks:

1. Breadth vs depth. vLLM optimises for supporting everything reasonably well. TensorRT-LLM optimises for NVIDIA hardware as far as it can go. SGLang optimises for a workload shape, shared prefixes and structured, multi-call programs, and is very fast on it.
2. Throughput vs portability. The GPU engines assume datacenter GPUs and many concurrent users. llama.cpp, Ollama and MLX-LM assume one machine, few users and whatever hardware is present, and win on every device the GPU engines do not support.
3. Velocity vs stability. The fastest-moving projects deliver new models and optimisations first and also change defaults and flags most often. Pin versions, keep a benchmark suite, and upgrade deliberately.

14. Performance metrics

Three definitional traps:

• "GPU utilisation" from NVML is not efficiency. It reports the fraction of sample periods in which any kernel was running. A GPU running memory-bound decode at batch 1 can show close to 100% while using a small fraction of its compute. Use SM activity, Tensor Core activity and memory bandwidth from DCGM profiling metrics instead.
• Memory utilisation is nearly meaningless for engines that pre-allocate. vLLM, SGLang and TensorRT-LLM claim most of HBM for the KV pool at startup. KV-cache utilisation is the metric that moves.
• TPOT and ITL are averaged differently. TPOT averages per request and then across requests; ITL distributions pool every gap. A few long stalls from prefill interference can hide in mean TPOT but show plainly in p99 ITL.

Why optimising one metric hurts another

The useful framing is goodput: the throughput achieved while meeting the SLO, for example "p99 TTFT under 1 s and p99 ITL under 50 ms". A configuration with higher raw throughput and worse goodput is worse. The DistServe paper uses per-GPU goodput as its central metric for this reason (Zhong et al., 2024).

15. Benchmarking inference engines

Published engine comparisons disagree because they measure different things: different hardware, versions, workloads, concurrency and definitions. A benchmark worth acting on is one that someone else could reproduce and that resembles your traffic. The methodology below is designed for that.

1. Fix and record the environment

2. Define the workload

• Input and output length distributions drawn from production logs where possible, not fixed lengths. At minimum run three shapes: short-in/short-out (chat, for example 512 in, 256 out), long-in/short-out (RAG and summarisation, for example 8,000 in, 256 out), and short-in/long-out (generation and reasoning, for example 512 in, 2,000 out).
• Output length control. Either force exact output lengths (ignore EOS) for comparability, or use natural stopping for realism, and state which.
• Prefix sharing. Real chat and agent traffic shares prefixes. A random-token benchmark gives prefix caching nothing to find, and a benchmark that repeats one prompt gives it everything. Use a dataset with a realistic sharing pattern, or a synthetic shared-prefix generator with a stated share ratio.
• Arrival process. Closed loop at fixed concurrency, which measures capacity, and open loop with Poisson arrivals at fixed request rates, which measures latency under load. Report both.

3. Sweep concurrency or request rate

Run a ladder, for example concurrency 1, 4, 16, 32, 64, 128 and 256, or request rates from low to past saturation. The result is a curve of latency against throughput, and the knee of that curve is the operating point. A single number at one concurrency hides the only thing that matters.

4. Warm up

Discard the first run after startup. Graph capture, compilation, lazy kernel selection and cold caches distort early requests. Send a warm-up set of at least a few dozen requests at the target shape. If prefix caching is on, decide deliberately whether the measured run should start with a warm or cold cache, and reset it between configurations.

5. Measure

Measure latency at the client, with streaming enabled, so TTFT and ITL reflect what users see. Collect server metrics (queue depth, KV utilisation, preemptions) and DCGM metrics in parallel. Run each point long enough to reach steady state: several minutes, or at least several hundred completed requests, beyond warm-up.

6. Report

For each point: request rate or concurrency, completed requests, failures, output tokens/sec, total tokens/sec, and p50, p90 and p99 for TTFT, TPOT, ITL and end-to-end latency. Plot throughput against p99 TTFT and p99 ITL. State the SLO and report goodput against it.

7. Statistical considerations

• Repeat each configuration at least three times on fresh server processes and report the spread. Run-to-run variance of several percent is common.
• Tail percentiles need many samples. A p99 from 200 requests rests on two data points.
• Randomise the order of configurations to avoid drift from temperature or neighbouring workloads being attributed to one engine.
• Compare engines at equal quality: same weights and precision, or quality measured alongside speed.
• Do not compare numbers from different benchmark tools, since their definitions of TPOT and ITL differ.

Example commands

These examples use each project's own tooling. Flags change between releases; check --help for the version under test.

Serve a model with vLLM, then benchmark it with vLLM's serving benchmark:

vllm serve meta-llama/Llama-3.1-8B-Instruct \
  --max-model-len 16384 \
  --gpu-memory-utilization 0.90 \
  --enable-prefix-caching
 
vllm bench serve \
  --backend vllm \
  --model meta-llama/Llama-3.1-8B-Instruct \
  --dataset-name random \
  --random-input-len 1024 --random-output-len 256 \
  --num-prompts 2000 \
  --max-concurrency 64 \
  --percentile-metrics ttft,tpot,itl,e2el \
  --metric-percentiles 50,90,99 \
  --save-result

Serve with SGLang and use its benchmark script, which can also target other OpenAI-compatible servers:

python -m sglang.launch_server \
  --model-path meta-llama/Llama-3.1-8B-Instruct \
  --port 30000
 
python -m sglang.bench_serving \
  --backend sglang \
  --dataset-name random \
  --random-input-len 1024 --random-output-len 256 \
  --num-prompts 2000 \
  --max-concurrency 64

Serve with TensorRT-LLM's OpenAI-compatible server:

trtllm-serve meta-llama/Llama-3.1-8B-Instruct --tp_size 1 --port 8000

Measure raw prefill and decode speed of a GGUF model with llama.cpp, independent of any server:

llama-bench -m Llama-3.1-8B-Instruct-Q4_K_M.gguf -p 512,2048 -n 128 -ngl 99 -fa 1

For engine-neutral comparisons, point one load generator, such as NVIDIA's AIPerf (the successor to GenAI-Perf) or the SGLang script above, at each engine's OpenAI-compatible endpoint, so that every engine is measured with identical definitions.

A concurrency sweep is a loop around the same command:

for c in 1 4 16 32 64 128 256; do
  vllm bench serve --backend vllm \
    --model meta-llama/Llama-3.1-8B-Instruct \
    --dataset-name random --random-input-len 1024 --random-output-len 256 \
    --num-prompts $((c * 20)) --max-concurrency "$c" \
    --save-result --result-filename "c${c}.json"
done

16. Hardware considerations

The figures below are vendor-published peak specifications, collected from official datasheets. They are not measured performance. The last column is an arithmetic ceiling: bandwidth divided by the bytes of an 8B-parameter model at 4 bits per weight (about 4.7 GB for a typical Q4_K_M or INT4 file), which bounds single-sequence decode speed before KV reads and overheads. Real results are lower.

Nobody serves an 8B model at batch 1 on a B200. The column is there to show how directly decode speed tracks memory bandwidth across very different devices. Achieved bandwidth is typically well below peak, especially on CPUs.

What each property means for inference

VRAM capacity decides what fits: weights plus KV cache for the concurrency you need. Section 7's arithmetic applies directly. It is usually the first constraint.

Memory bandwidth decides decode speed. At equal capacity, the device with more bandwidth decodes faster at every batch size where decode is memory-bound. This is why an H200 decodes faster than an H100 despite having the same compute.

Tensor Cores and supported formats decide prefill speed and whether low-precision formats deliver compute gains as well as memory gains. FP8 needs Ada, Hopper or newer on NVIDIA; FP4 needs Blackwell.

PCIe is the path between host and GPU, and between GPUs without NVLink: roughly 32 GB/s per direction for Gen4 x16 and 64 GB/s for Gen5 x16. It matters for model loading time, CPU offload of weights or KV cache, and tensor parallelism across consumer cards, where it makes TP communication expensive. Pipeline parallelism or single-GPU serving suits PCIe-only boxes better.

NVLink and NVSwitch provide hundreds of GB/s to TB/s between GPUs in a node, and in rack-scale systems such as GB200 NVL72 across a whole rack. They are what make tensor and expert parallelism efficient.

CPU RAM matters for loading, for offloading KV cache (a CPU-memory tier for prefix caching), and for CPU-only inference, where capacity is cheap but bandwidth is low.

Unified memory on Apple Silicon gives the GPU access to the entire system memory pool at a bandwidth between desktop DDR and datacenter HBM. It allows very large models on one machine, but with far less compute than a datacenter GPU, so long-prompt prefill is slow.

Characteristics by class

17. Scaling an inference service

A model running in one process on one GPU is a demo. A service needs to survive traffic spikes, GPU failures, deployments, noisy tenants and a finance review. The shape that emerges is consistent across organisations.

Reading it top to bottom: clients call a gateway that knows who they are and how much they may use. A router chooses a replica based on which model is requested, which replica already holds the relevant cache, and how loaded each one is. Replicas run the inference engine on GPUs, pull weights from shared storage, report metrics, and are added or removed by an autoscaler.

API gateway. Authentication, per-tenant quotas measured in tokens as well as requests, request size limits, usage metering for chargeback, and a single OpenAI-compatible surface across models.

Load balancer and inference router. Round-robin is a poor fit for LLMs because requests differ in cost by orders of magnitude and because cache locality matters. An inference-aware router picks a replica using engine metrics, such as queue depth and KV utilisation, and prefix affinity, sending a request to the replica most likely to hold its prefix. The Kubernetes Gateway API Inference Extension standardises this pattern with an endpoint picker that reads model-server metrics.

Queueing. Queue at the router or gateway rather than letting engine queues grow without bound. Apply deadlines: a request that cannot start within its TTFT budget should be rejected quickly with a retryable error rather than served late. Separate queues per priority class.

Autoscaling. CPU utilisation is useless as a signal, and GPU utilisation is misleading (section 14). Scale on engine queue depth (for example vLLM's vllm:num_requests_waiting), KV-cache utilisation, or TTFT against target. Account for scale-up time: pulling a container with CUDA libraries, loading tens to hundreds of gigabytes of weights, compiling and capturing graphs can take minutes. Keep headroom, pre-pull images, cache weights on local NVMe, and consider scheduled scaling for predictable daily peaks. Scale to zero only where cold starts are acceptable.

Health checks. Separate liveness (the process is alive) from readiness (weights loaded, graphs captured, a warm-up request succeeded). Startup probes need long timeouts. A replica that has stopped making progress, for example because of a hung NCCL collective, may still answer HTTP health checks, so also check that generation is progressing.

Model loading. Weights dominate startup. Options, from simplest: download from a model hub at start (slow, and a dependency on an external service), a shared read-only volume, a node-local cache populated ahead of time, or streaming loaders that read tensors directly from object storage in parallel. Safetensors format allows memory-mapped, zero-copy loading.

GPU scheduling. Decide which models get which GPUs. Large models get whole nodes with NVLink; small models may share GPUs through MIG or time-slicing; batch jobs run on preemptible capacity.

Multi-model serving. Many models with low traffic each waste GPUs if each has dedicated replicas. Options: multi-LoRA serving of fine-tunes on one base model (section 19), MIG partitions for small models, on-demand loading with LRU unloading for long-tail models, or routing rarely used models to a shared pool.

Rate limiting. Limit in tokens per minute as well as requests per minute, since a single request can ask for 100,000 input tokens. Reserve capacity for priority tenants.

Observability. Collect TTFT, TPOT, ITL and end-to-end latency histograms, queue depth, KV utilisation, preemption and prefix-cache hit rates from the engine; SM activity, memory bandwidth, temperature, power and XID errors from DCGM; and request traces spanning gateway, router and engine. Log prompts and outputs only under a deliberate data policy.

Fault tolerance. GPUs fail more often than CPUs, and a multi-GPU replica fails if any of its GPUs fails. Plan N+1 replicas per model, drain replicas gracefully on deployment (stop admitting, finish in-flight streams), make clients retry idempotent requests on another replica, and alert on XID errors and ECC faults to cordon bad nodes.

18. Kubernetes inference architecture

Kubernetes is the most common substrate for self-managed inference because it already solves scheduling, rollout, service discovery and health management. GPUs add a layer of components.

The pieces, briefly: GPU nodes run NVIDIA's GPU Operator, which installs drivers and advertises GPUs to Kubernetes. Inference pods request GPUs like any other resource and read weights from shared storage. A gateway with an inference-aware endpoint picker spreads requests across pods, Prometheus scrapes both engine and GPU metrics, and an autoscaler adds pods when queues grow.

GPU Operator. Installs and manages everything a node needs to run GPU workloads: the driver, the NVIDIA Container Toolkit, the device plugin, GPU Feature Discovery labels, the DCGM exporter for metrics, and the MIG manager, as containers (NVIDIA GPU Operator docs). It removes most hand-built node images. AMD provides an equivalent GPU Operator for Instinct GPUs.

NVIDIA device plugin. Advertises GPUs as the extended resource nvidia.com/gpu, so a pod requests them in resources.limits. Allocation is by whole device unless MIG or time-slicing is configured. Dynamic Resource Allocation (DRA), generally available in Kubernetes since 1.34, lets drivers expose richer device attributes and sharing policies, and NVIDIA's DRA driver builds on it.

GPU scheduling and node pools. Put GPU nodes in dedicated pools with a taint, such as nvidia.com/gpu=present:NoSchedule, so only GPU workloads land there, and use node labels from GPU Feature Discovery (for example nvidia.com/gpu.product) to pin models to the right GPU type. Keep separate pools per GPU class. Gang scheduling through Kueue, Volcano or similar is needed when a multi-pod replica must start all at once.

Topology awareness. A TP = 4 pod should get four GPUs on the same NVLink domain, and multi-node replicas should be placed close together in the network. Use whole-node allocations for large tensor-parallel jobs, and the LeaderWorkerSet API to deploy a multi-node replica as a leader pod plus workers that are created, scaled and restarted together.

MIG. Multi-Instance GPU partitions an A100, H100, H200 or B200 into up to seven isolated instances with dedicated memory and compute slices, exposed as resources such as nvidia.com/mig-1g.10gb. It is well suited to small models, embeddings and rerankers where a whole H100 would be mostly idle. Time-slicing is the alternative when isolation is not needed. Neither helps a model that needs the whole GPU's bandwidth.

Model storage and persistent volumes. Options: a ReadOnlyMany PersistentVolume on a shared file system, pre-populated; an init container or sidecar that syncs from object storage to node-local NVMe; an image volume or OCI artifact containing the weights; or KServe-style storage initialisers. Prefer node-local copies for large models to cut startup time, and never download from a public hub at pod start in production.

Autoscaling. The Horizontal Pod Autoscaler with custom metrics, or KEDA with a Prometheus scaler, on queue depth or KV utilisation. Pair it with a cluster autoscaler or node auto-provisioning for GPU nodes, and set scale-down stabilisation windows long enough to ride out load dips given multi-minute scale-up.

Prometheus and Grafana. Scrape the engine's /metrics endpoint and the DCGM exporter. Dashboards: request rate, TTFT, ITL and E2E percentiles, queue depth, KV utilisation, prefix hit rate, preemptions, tokens/sec, SM activity, memory bandwidth, power, and XID errors.

Logging. Structured request logs with request ID, tenant, model, input and output token counts, latency breakdown and finish reason. Treat prompt and completion content as sensitive data.

Tracing. OpenTelemetry spans from gateway to router to engine; vLLM and TGI can emit traces. For agents, connect inference spans to the agent's own trace so each model call is attributable to a step.

A minimal Deployment for one vLLM replica on one GPU:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: llama-8b
spec:
  replicas: 2
  selector:
    matchLabels: { app: llama-8b }
  template:
    metadata:
      labels: { app: llama-8b }
    spec:
      tolerations:
        - key: nvidia.com/gpu
          operator: Exists
          effect: NoSchedule
      nodeSelector:
        nvidia.com/gpu.product: NVIDIA-H100-80GB-HBM3
      containers:
        - name: vllm
          image: vllm/vllm-openai:<pinned-version>
          args:
            - --model=/models/Llama-3.1-8B-Instruct
            - --max-model-len=16384
            - --gpu-memory-utilization=0.90
          ports:
            - containerPort: 8000
          resources:
            limits:
              nvidia.com/gpu: "1"
          startupProbe:
            httpGet: { path: /health, port: 8000 }
            failureThreshold: 60
            periodSeconds: 10
          readinessProbe:
            httpGet: { path: /health, port: 8000 }
            periodSeconds: 5
          volumeMounts:
            - { name: models, mountPath: /models, readOnly: true }
            - { name: shm, mountPath: /dev/shm }
      volumes:
        - name: models
          persistentVolumeClaim: { claimName: models-ro }
        - name: shm
          emptyDir: { medium: Memory, sizeLimit: 16Gi }

The shared-memory volume matters: engines use /dev/shm for inter-process communication between their frontend, engine core and workers, and the container default is too small.

19. Advanced inference techniques

Speculative decoding

The idea: a small, fast helper guesses the next few words; the big model checks all the guesses at once and keeps the ones it agrees with.

Mechanically, a draft proposes k tokens. The target model runs one forward pass over all k positions, which costs about the same as decoding one token because decode is memory-bound and the extra positions are nearly free arithmetic. It accepts the longest prefix of the draft consistent with its own distribution and produces one extra token of its own. If on average a tokens are accepted, each target pass yields a + 1 tokens instead of 1.

What makes this more than a heuristic is that, with the modified rejection sampling scheme of Leviathan et al. and Chen et al., the output distribution is provably identical to sampling from the target model alone, so speculative decoding is lossless (Leviathan et al., 2023; Chen et al., 2023). Draft sources differ: a separate small model; extra decoding heads on the target (Medusa, documented, Cai et al., 2024); a lightweight head that extrapolates the target's own hidden features (EAGLE and its successors, documented, Li et al., 2024); multi-token prediction heads trained with the model, as in DeepSeek-V3; or model-free n-gram and suffix matching against the prompt and history, which is very effective when outputs copy from inputs, as in code editing and RAG.

In this example the draft proposes four tokens, the large model checks them in a single pass, keeps the first two, replaces the third with its own choice, and discards the rest. Three tokens came out of one expensive pass.

When does it actually help? Speculation trades spare compute for latency. At low batch sizes the GPU has idle compute and TPOT improves substantially. At high batch sizes the GPU is already busy, verification of rejected tokens is wasted work, and throughput can fall. Engines therefore let you limit speculation to lower loads, and the right setting depends on acceptance rate, which depends on the task. Measure acceptance rate in production traffic, not on a benchmark prompt set.

Prefix caching and prompt caching

Prefix caching (section 7) is automatic, engine-level reuse of KV blocks. Prompt caching usually refers to the same mechanism exposed as a product feature by hosted APIs, often with explicit cache breakpoints and discounted pricing for cached input tokens. For self-hosted systems the lesson is the same: structure prompts so that the stable parts come first (system prompt, tool definitions, documents) and the variable parts last. A timestamp or request ID at the top of a system prompt silently defeats every prefix cache.

Disaggregated serving

Covered in section 6. The operational additions are a KV transfer layer (NIXL, LMCache, Mooncake), a router that assigns each request to a prefill worker and a decode worker, and independent autoscaling of the two pools according to the input-to-output token ratio of the traffic.

Multi-LoRA serving

Many fine-tuned variants of one base model can share a single deployment. Each LoRA adapter adds small low-rank matrices to selected layers. S-LoRA and Punica showed that a server can hold many adapters in a paged pool and batch requests for different adapters together with custom kernels that apply each adapter to its own rows (Sheng et al., 2023; Chen et al., 2023). vLLM, SGLang, TensorRT-LLM and TGI support this. It turns hundreds of per-customer fine-tunes from hundreds of deployments into one.

Mixture-of-experts inference and expert parallelism

An MoE model activates only a few experts per token, so its compute per token is that of a much smaller dense model, but all experts must be resident in memory. DeepSeek-V3, for example, has 671B total parameters with 37B activated per token (DeepSeek-AI, 2024). Serving implications:

• Memory is sized by total parameters; compute by active parameters.
• At small batch, each expert sees few tokens, so expert GEMMs are memory-bound and inefficient. Large batches, gathered across many GPUs with expert parallelism, fix this.
• The all-to-all communication of expert parallelism needs a fast fabric and communication kernels designed for it.
• Expert load imbalance must be managed with redundant hot experts.

This is why the most efficient MoE deployments use many GPUs per replica, combining wide EP with DP attention and prefill/decode disaggregation.

Chunked prefill

Covered in section 6. In practice, leave it on and tune the per-step token budget: larger budgets favour TTFT and throughput, smaller budgets favour ITL.

Request prioritisation

Priority scheduling admits and preempts by class, such as interactive over batch or paid over free. Combine an engine-level priority, where supported, with gateway-level admission control, since the engine only sees what reaches it.

Dynamic routing

Routing decides which replica, and increasingly which model, serves a request:

• Load-aware: fewest queued requests, or lowest KV utilisation.
• Cache-aware: the replica with the longest matching cached prefix, tracked by approximate prefix trees at the router or KV events from the engines, as in SGLang's router, llm-d's scheduler and NVIDIA Dynamo's KV-aware router.
• Model routing: sending easy requests to a smaller, cheaper model and hard ones to a larger model, based on a classifier or rules. This is a quality decision as much as a performance one and must be evaluated as such.
• Session affinity: keeping an agent's or conversation's turns on one replica, the simplest form of cache-aware routing.

Model quantization

Covered in section 10. The newer operational pattern is to quantise once, in a controlled pipeline with a calibration set drawn from production traffic, evaluate against the task suite, and publish the quantised checkpoint as a versioned artifact, rather than quantising on the fly at load time.

KV-cache compression

Beyond FP8 caches: lower-bit KV quantization (KIVI and similar); architectural compression that must be trained into the model (GQA, MQA, MLA, sliding-window attention); and token eviction or selection (H2O, StreamingLLM, and attention-sparsity methods), which approximate attention and change outputs. Offload to CPU and storage tiers compresses nothing but extends effective capacity.

20. Cost engineering

The cost of inference is the cost of GPU time divided by the useful work done in it. Every number in this section is an assumption chosen to make the arithmetic concrete. Substitute measured throughput and real prices.

Core formulas

Tokens per GPU-hour            = tokens/sec/GPU × 3,600
Cost per 1M tokens             = (GPU $/hour ÷ tokens per GPU-hour) × 1,000,000
Effective cost per 1M tokens   = Cost per 1M tokens ÷ average utilisation
Fully loaded cost per 1M       = Effective cost × (1 + infrastructure overhead)
Cost per request               = GPU-seconds per request × $/GPU-second
                               ≈ (GPU $/hour ÷ 3,600) ÷ requests/sec/GPU

Tokens/sec/GPU must come from a benchmark at the operating point that meets your SLO, not from peak throughput at unlimited latency. For a replica spanning several GPUs, divide replica throughput by its GPU count.

Average utilisation here means the fraction of provisioned GPU time spent serving at the benchmarked operating point. Traffic has daily peaks and troughs, and replicas are sized for the peak, so a fleet typically averages well below its peak throughput.

Infrastructure overhead covers the rest of the bill: CPU nodes for gateways and routers, storage, networking and egress, observability, redundancy (N+1 replicas), and the engineering time to run it.

Worked example 1: cost per million output tokens

Assumptions: an 8B model on one H100 at an assumed $3.00 per GPU-hour, with a measured 2,500 output tokens/sec at the SLO for the workload's length mix (an illustrative figure; measure your own).

Tokens per GPU-hour           = 2,500 × 3,600          = 9,000,000
Cost per 1M output tokens     = 3.00 ÷ 9.0             = $0.333
At 40% average utilisation    = 0.333 ÷ 0.40           = $0.833
With 30% infrastructure overhead = 0.833 × 1.30        = $1.08 per 1M output tokens

The gap between $0.33 and $1.08 is the point: utilisation and overhead, not the raw engine number, dominate the real cost.

Worked example 2: cost per request

Assumptions: the same GPU serves 6 requests/sec at the SLO for a workload averaging 1,500 input and 300 output tokens.

$ per GPU-second              = 3.00 ÷ 3,600           = $0.000833
Cost per request at full load = 0.000833 ÷ 6           = $0.000139
Per 1,000 requests            = $0.139
At 40% utilisation, +30% overhead = 0.139 ÷ 0.40 × 1.30 = $0.45 per 1,000 requests

For an agent that makes 15 model calls per task, that is roughly $0.0068 per task from inference alone, before tools and retrieval.

Worked example 3: the effect of prefix caching on an agent

Assumptions: an agent step sends 12,000 input tokens of which 10,000 are an unchanged prefix (system prompt, tools, prior trajectory), and generates 200 tokens. Using the prefill estimate from section 6 as a proxy for GPU cost, prefill work falls from 12,000 tokens to 2,000 on a cache hit, a six-fold reduction in prefill compute for that step. How much this lowers total cost depends on how much of the GPU's time went to prefill, which is why agent workloads, dominated by long repeated inputs and short outputs, benefit most.

Worked example 4: self-hosting against an API

Break-even monthly tokens     = Fixed monthly self-hosting cost ÷ (API $/token − self-hosted marginal $/token)

With assumed fixed costs of $25,000 per month (reserved GPUs, platform, a share of engineering time), an API blended price of $2.00 per 1M tokens and a self-hosted marginal cost of $0.40 per 1M:

25,000 ÷ ((2.00 − 0.40) ÷ 1,000,000) = 15.6 billion tokens per month

Below that volume the API is cheaper on these assumptions. Data control, latency, customisation and availability often decide the question before cost does, but the break-even should still be calculated with your own numbers.

Levers, roughly in order of effort

21. Choosing an inference engine

Choose from the workload outward. The questions, in order:

1. Where must it run? Laptop, Mac, CPU-only, NVIDIA, AMD, air-gapped, Kubernetes.
2. How many concurrent users, and what latency targets?
3. What does the traffic look like? Prompt and output lengths, prefix sharing, structured output, multimodal.
4. Which models, and how quickly do new ones need to be supported?
5. Who operates it, and how much tuning can they afford?

In short: off the datacenter, pick by device. On datacenter GPUs, start with vLLM or SGLang, lean to SGLang when requests share long prefixes, and bring in TensorRT-LLM when you run NVIDIA-only at a scale where the extra efficiency repays the extra engineering. In every case, confirm with a benchmark on your own traffic.

Our take. For most teams the engine decision is less consequential than the decisions around it: model size and precision, prompt structure, routing, autoscaling signal and SLOs. A well-operated vLLM or SGLang deployment will outperform a poorly operated deployment of any engine.

22. Future of LLM inference

Inference-time scaling and reasoning models. Models that think before answering generate thousands of hidden reasoning tokens per request. Output length, not prompt length, becomes the dominant cost, decode efficiency matters more than ever, and latency targets move from TTFT to time-to-answer. Speculative decoding, low-precision decode, and scheduling that is aware of how long a request is likely to run all gain importance.

Inference-time compute as a product knob. Best-of-n sampling, self-consistency, tree search and verifier loops multiply the number of forward passes per user request. These patterns share long prefixes across branches, which favours radix-tree prefix caches and engines that treat a group of related requests as one program.

Longer contexts. Million-token contexts make KV capacity and prefill the central constraints. Expect more context parallelism, more hierarchical KV storage from HBM through DRAM to SSD and network stores, more KV compression trained into model architectures, and serving systems whose main job is moving KV cache efficiently.

Multimodal inference. Images, audio and video add encoder stages with different compute profiles from the language model. Engines are adding encoder disaggregation and encoder caches alongside prefill/decode separation, as vLLM's encode disaggregation already shows.

Agentic workloads. Agents produce many short-output calls with long, mostly shared inputs, strict structured output, and tool latency between calls. Serving them well means session-aware routing, prefix and KV retention across tool calls, fast grammar-constrained decoding, and schedulers that can pause and resume a program rather than treat each call independently.

Specialised accelerators. Beyond GPUs, TPUs, Trainium and Inferentia, Gaudi, and inference-specific designs with large on-chip SRAM trade flexibility for speed or cost on particular workloads. Engines are responding with hardware plugin architectures, as in vLLM and SGLang, so that a single serving stack targets many back ends.

Disaggregated inference. Separating prefill, decode and encode onto different pools, possibly different hardware generations, is moving from research into mainstream stacks such as Dynamo and llm-d. The KV cache is becoming a network-addressable object with its own transfer protocols and storage tiers.

Memory-centric architectures. Because decode is bandwidth-bound, hardware roadmaps are increasingly memory roadmaps: more HBM per GPU, higher bandwidth, rack-scale NVLink domains that make a whole rack look like one GPU to tensor and expert parallelism, and CXL-attached memory for KV tiers.

Emerging serving architectures. The direction is from "a model server" to "an inference platform": a routing layer that understands models, prefixes, SLOs and cost; heterogeneous pools of engines and hardware underneath; and KV cache treated as shared infrastructure. The engine remains central, but more of the performance now comes from the system around it.

23. Glossary

24. Final takeaways

1. Prefill is compute-bound, decode is bandwidth-bound. Nearly every optimisation targets one side of that asymmetry. Know which side your workload is on.
2. The KV cache is the capacity plan. Calculate it per token from the model config before choosing hardware. Concurrency, batch size and throughput all follow from what is left after the weights.
3. Batching is how inference becomes affordable. Continuous batching, paged KV and CUDA graphs are baseline expectations, not differentiators.
4. Reuse beats recomputation. Prefix caching with affinity routing is the largest single lever for chat and agents, and it depends on prompt structure as much as on the engine.
5. Quantise deliberately. FP8 on supported hardware is usually safe and valuable; lower precisions need task-level evaluation.
6. Parallelise by need. Data parallelism for throughput, tensor parallelism inside NVLink for size and latency, pipeline or expert parallelism across nodes only when the model demands it.
7. Measure goodput against an SLO, with client-side streaming latency percentiles across a concurrency sweep. A single throughput number is not a benchmark.
8. Cost is dominated by utilisation and overhead. Autoscaling on the right signal and choosing the smallest adequate model beat most kernel-level gains.
9. Choose engines by workload and hardware, then verify. vLLM and SGLang for general GPU serving, TensorRT-LLM for maximum NVIDIA efficiency, llama.cpp, Ollama and MLX-LM for local, CPU and Apple Silicon. Benchmark on your own traffic before committing.
10. The engine is one layer of the platform. Routing, autoscaling, model storage, observability and fault handling decide whether a fast engine becomes a reliable service.

Inference is one layer of the agentic systems Covaratech builds. Our AI inference infrastructure practice engineers the model serving, routing, GPU scheduling and cost optimisation that agent workloads run on, and the deployment practice takes those systems into private, self-hosted and air-gapped environments.

References

Papers

1. Kwon, W. et al. "Efficient Memory Management for Large Language Model Serving with PagedAttention." SOSP 2023. arXiv:2309.06180
2. Yu, G.-I. et al. "Orca: A Distributed Serving System for Transformer-Based Generative Models." OSDI 2022. usenix.org
3. Zheng, L. et al. "SGLang: Efficient Execution of Structured Language Model Programs." NeurIPS 2024. arXiv:2312.07104
4. Agrawal, A. et al. "Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve." OSDI 2024. arXiv:2403.02310
5. Zhong, Y. et al. "DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving." OSDI 2024. arXiv:2401.09670
6. Patel, P. et al. "Splitwise: Efficient Generative LLM Inference Using Phase Splitting." ISCA 2024. arXiv:2311.18677
7. Qin, R. et al. "Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving." 2024. arXiv:2407.00079
8. Holmes, C. et al. "DeepSpeed-FastGen: High-throughput Text Generation for LLMs via MII and DeepSpeed-Inference." 2024. arXiv:2401.08671
9. Aminabadi, R. Y. et al. "DeepSpeed-Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scale." SC 2022. arXiv:2207.00032
10. Dao, T. et al. "FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness." NeurIPS 2022. arXiv:2205.14135
11. Dao, T. "FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning." 2023. arXiv:2307.08691
12. Shah, J. et al. "FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precision." 2024. arXiv:2407.08608
13. Ye, Z. et al. "FlashInfer: Efficient and Customizable Attention Engine for LLM Inference Serving." MLSys 2025. arXiv:2501.01005
14. Leviathan, Y., Kalman, M., Matias, Y. "Fast Inference from Transformers via Speculative Decoding." ICML 2023. arXiv:2211.17192
15. Chen, C. et al. "Accelerating Large Language Model Decoding with Speculative Sampling." 2023. arXiv:2302.01318
16. Cai, T. et al. "Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads." 2024. arXiv:2401.10774
17. Li, Y. et al. "EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty." ICML 2024. arXiv:2401.15077
18. Frantar, E. et al. "GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers." ICLR 2023. arXiv:2210.17323
19. Lin, J. et al. "AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration." MLSys 2024. arXiv:2306.00978
20. Xiao, G. et al. "SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models." ICML 2023. arXiv:2211.10438
21. Dettmers, T. et al. "LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale." NeurIPS 2022. arXiv:2208.07339
22. Micikevicius, P. et al. "FP8 Formats for Deep Learning." 2022. arXiv:2209.05433
23. Frantar, E. et al. "MARLIN: Mixed-Precision Auto-Regressive Parallel Inference on Large Language Models." 2024. arXiv:2408.11743
24. Liu, Z. et al. "KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cache." ICML 2024. arXiv:2402.02750
25. Zhang, Z. et al. "H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models." NeurIPS 2023. arXiv:2306.14048
26. Xiao, G. et al. "Efficient Streaming Language Models with Attention Sinks." ICLR 2024. arXiv:2309.17453
27. Shoeybi, M. et al. "Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism." 2019. arXiv:1909.08053
28. Liu, H., Zaharia, M., Abbeel, P. "Ring Attention with Blockwise Transformers for Near-Infinite Context." 2023. arXiv:2310.01889
29. Ainslie, J. et al. "GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints." EMNLP 2023. arXiv:2305.13245
30. Shazeer, N. "Fast Transformer Decoding: One Write-Head is All You Need." 2019. arXiv:1911.02150
31. DeepSeek-AI. "DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model." 2024. arXiv:2405.04434
32. DeepSeek-AI. "DeepSeek-V3 Technical Report." 2024. arXiv:2412.19437
33. Fedus, W., Zoph, B., Shazeer, N. "Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity." JMLR 2022. arXiv:2101.03961
34. Sheng, Y. et al. "S-LoRA: Serving Thousands of Concurrent LoRA Adapters." MLSys 2024. arXiv:2311.03285
35. Chen, L. et al. "Punica: Multi-Tenant LoRA Serving." MLSys 2024. arXiv:2310.18547
36. Sheng, Y. et al. "Fairness in Serving Large Language Models." OSDI 2024. arXiv:2401.00588
37. Dong, Y. et al. "XGrammar: Flexible and Efficient Structured Generation Engine for Large Language Models." 2024. arXiv:2411.15100
38. Williams, S., Waterman, A., Patterson, D. "Roofline: An Insightful Visual Performance Model for Multicore Architectures." CACM 2009. doi:10.1145/1498765.1498785

Official documentation and repositories (checked September 2026)

• vLLM documentation, docs.vllm.ai, and repository, github.com/vllm-project/vllm
• SGLang documentation, docs.sglang.ai, and repository, github.com/sgl-project/sglang
• NVIDIA TensorRT-LLM documentation, nvidia.github.io/TensorRT-LLM, and repository, github.com/NVIDIA/TensorRT-LLM
• Hugging Face Text Generation Inference, github.com/huggingface/text-generation-inference
• llama.cpp, github.com/ggml-org/llama.cpp
• Ollama, github.com/ollama/ollama and docs.ollama.com
• DeepSpeed-MII, github.com/deepspeedai/DeepSpeed-MII
• MLX, github.com/ml-explore/mlx, and MLX-LM, github.com/ml-explore/mlx-lm
• NVIDIA Dynamo, github.com/ai-dynamo/dynamo
• llm-d, llm-d.ai
• Kubernetes Gateway API Inference Extension, gateway-api-inference-extension.sigs.k8s.io
• LeaderWorkerSet, github.com/kubernetes-sigs/lws
• NVIDIA GPU Operator, docs.nvidia.com/datacenter/cloud-native/gpu-operator
• NVIDIA Multi-Instance GPU user guide, docs.nvidia.com/datacenter/tesla/mig-user-guide
• NVIDIA H100, H200 and B200 datasheets, nvidia.com/en-us/data-center
• AMD Instinct MI300X and MI325X specifications, amd.com/en/products/accelerators/instinct
• Meta Llama 3.1 model configurations, huggingface.co/meta-llama
• Ollama MLX preview announcement, March 2026, reported at appleinsider.com