DeepInfer Deep Dive: Inside DeepSeek's Production Inference Engine

In April 2025, DeepSeek said its inference engine was too tightly coupled to internal infrastructure to open source, and contributed kernels to other projects instead. On October 20, 2026, they changed their minds and released DeepInfer, the framework that serves DeepSeek's production API, under Apache 2.0.

I spent the day after the release reading the source instead of the announcement. What struck me is how little it resembles the frameworks everyone will compare it with. There is no tensor parallelism anywhere in the codebase. The scheduler is C++. There is an n-gram memory subsystem with its own parallelism dimension. Speculative decoding decides at runtime how many draft tokens are worth verifying, per request, based on a calibrated confidence head. And Huawei Ascend 950 is a day one target alongside NVIDIA Blackwell, while NVIDIA Hopper is not supported at all.

Below I go through what ships and what does not, the model it was built for, how the engine is put together, the main features and the environment variables behind them, the published benchmarks, and how it stacks up against vLLM, SGLang, and TensorRT-LLM.

What Is and Is Not in the Release

Start with this table. It decides whether DeepInfer is any use to you.

The README is unusually direct about the gaps, listing a KV cache system, VE/prefill/decode disaggregation, and data parallel load balancing as "required for a complete inference service (not included in this release)." Reporting around the launch put the minimum efficient production unit at roughly 64 prefill GPUs plus 64 decode GPUs, which tells you who this is for.

The LLM entrypoint hardcodes offline_inference=True. There is no server. You get a batch API, and you build the service around it.

The Model Is Half the Story

DeepInfer only makes sense as one half of a codesigned pair. DeepSeek-V4.1-Flash is not a conventional dense or even conventional MoE transformer, and most of the engine's unusual choices exist to serve its specific structure. From the model card, the kernel repositories, and the engine source, the architecture includes:

Multi head Latent Attention (MLA) compresses keys and values into a shared low rank latent, which is why the KV cache per token is small enough to make very long contexts practical.

DeepSeek Sparse Attention (DSA), introduced with V3.2 and used through V4 and V4.1. A "lightning indexer" scores candidate tokens and a top-k selection picks which ones each query attends to, making attention token level sparse rather than dense. This is why DeepSelect exists as a separate kernel library: its specialized top-k is reportedly 2 to 20 times faster than torch.topk, and the engine caps its working memory with DEEPINFER_INDEX_ATTN_MEMORY_LIMIT, defaulting to 10 GiB.

A Mixture of Experts backbone with enough experts that expert parallelism rather than tensor parallelism is the natural sharding axis.

YoCo layer topology. The config carries n_prefill_layer. Layers below it are producers, layers from it upward are consumers that reuse KV computed earlier and use sliding window attention. This is cross layer KV sharing, and it is the structural precondition for the bounded replay trick described later.

Engram, which is the most unusual component and gets its own section below.

Manifold hyper connections (mHC), with Sinkhorn normalization kernels in TileKernels and a lazy evaluation path (hc_lazy) in the engine.

Weight formats are FP8 and MXFP4, context is 1M tokens, and the engine fixes compute dtype to bfloat16 (self.dtype = torch.bfloat16 in ModelConfig, not configurable).

Engram: an n-gram memory layer with its own parallelism

This is the piece I had not seen in any other inference stack. From deepinfer/model_executor/models/v4_modules/engram.py, the runtime "hash n-grams, gather the sharded tables and gate the fused projection."

Here is the flow:

The embedding table is sharded across ranks and fetched by hash, which means a lookup is a remote memory access rather than a local gather. DeepEP provides an EngramBuffer for exactly this, with "multi layer Engram storage on GPU or CPU, with one wait hook per layer." The engine exposes DEEPINFER_ENGRAM_FORCE_GPU_STORAGE, which is forced on for Ascend, and DEEPINFER_ENGRAM_PAD_TOKEN.

There is also a CompressedTokenizer as a global singleton, mapping raw token ids to a compressed id space before hashing, with out of vocabulary ids such as image sentinels mapping to IMAGE_SENTINEL. That compression exists to keep the n-gram table tractable.

Engram also gets its own parallel size. In ParallelConfig:

self.engram_parallel_size = engram_parallel_size or (self.model_parallel_size * data_parallel_size)
self.world_size = max(self.model_parallel_size * data_parallel_size, self.engram_parallel_size)

So world size is the maximum of the attention and MoE arrangement and the Engram arrangement. Engram can be spread wider than the model itself. My read is that the table is big, and that sharding it is a capacity planning problem in its own right.

Engine Architecture

The most important implementation choice is where the line between Python and C++ falls.

A few things that stand out when reading llm_engine.py.

Scheduling is one C++ call. self.scheduler.schedule(self.sequence_metadata_pool) returns the sequence metadata list and scheduler outputs, and also frees finished sequence groups in the same pass. The Python side never walks a request list to make scheduling decisions. There is even a comment in the decode path flagging remaining Python overhead as a known issue: "TODO(CPU): Eliminate O(num_reqs) python operations in decoding step."

Everything is pooled. SequenceMetadataPool, RequestOutputPool, and a DeferredRelease helper that collects sequence metadata, request outputs, and clone unpins and releases them in batches. RequestOutput objects returned from step() are owned by the pool and recycled on the next call, so the entrypoint explicitly clones them. You usually see this kind of allocation discipline in trading systems, not in Python ML serving. Given that DeepSeek grew out of a quant fund, that is probably no accident.

CPU GPU synchronization is actively forbidden. There is a forbid_cuda_sync guard around model forward and sampling launch, with DEEPINFER_DISABLE_SYNC_GUARD provided only for debugging. An accidental .item() in the hot path fails loudly rather than silently costing throughput.

Instrumentation is built in at named sections. cpu_timer.record("schedule:cpp"), "schedule:clone", "schedule:post", "multi_modal_fetch", "run_workers". You can see exactly where CPU time goes without attaching a profiler.

NUMA binding is explicit, with _bind_numa and get_idle_cpus in the worker path, plus CUDA_SCALE_LAUNCH_QUEUES=4x set at import.

The engine also supports three roles through EngineType:

class EngineType(enum.Enum):
    NORMAL = 0      # process prompts & completions
    PROMPT = ...    # only process prompts
    COMPLETION = ...# only process completions

with a note to keep it in sync with the C++ enum. Prefill and decode disaggregation is therefore implemented at the engine level, which matches the README's claim that "most required engine functionality is already implemented" even though the surrounding disaggregation infrastructure is not shipped.

The Parallelism Story: No Tensor Parallelism

This is the biggest break from other engines, and it is easy to miss. In ParallelConfig:

self.model_parallel_size = self.context_parallel_size

Model parallelism is context parallelism. There is no tensor parallel dimension in the configuration at all, and the LLM entrypoint takes expert_parallel_size where vLLM would take tensor_parallel_size.

Why this works for this model, and why it would not work for everything:

MLA makes attention cheap to replicate. Because the KV cache is compressed into a low rank latent, replicating attention weights per rank and splitting requests across ranks costs less memory than sharding attention would, and it removes the all-reduce per attention layer that tensor parallelism requires.

MoE is where the parameters are, so expert parallelism is where sharding pays off. The recommended configurations from launch coverage are EP64 to EP128 for decode within a scale up domain, and the smallest EP that fits in memory for prefill, typically EP2 to EP8. The published charts show EP64 ahead of EP8 at every batch size.

Context parallelism handles the long sequence case that data parallelism cannot: a single 1M token request has to be split somewhere, and CP splits it across ranks while keeping attention DP for separate requests. The README calls this "balanced CP," and --cp must divide ep_size.

The constraint set is enforced in _verify_args:

assert self.expert_parallel_size > 0
assert (self.model_parallel_size * self.data_parallel_size) % self.expert_parallel_size == 0
assert self.world_size % (self.model_parallel_size * self.data_parallel_size) == 0
assert self.world_size % self.engram_parallel_size == 0

If you are planning a deployment, this changes the sizing conversation. You are deciding how wide expert parallelism should be and whether you need context parallelism, not picking a tensor parallel degree. No mainstream serving stack works that way.

DSpark and Confidence Scheduled Verification

Speculative decoding normally has a fixed draft length. You propose k tokens, verify all k, accept a prefix. The waste is structural: you pay to verify tokens that were never likely to be accepted.

DSpark, which DeepSeek published separately in mid 2026 and which DeepInfer implements, attacks exactly that. It has two parts.

A semi autoregressive drafter. A DFlash style backbone runs one forward pass over the whole draft block, producing hidden states and base logits for every draft position at once. Then a lightweight head adds a prefix dependent bias so each position still conditions on tokens sampled earlier in the block, at a cost far below the parallel stage. Two head variants exist: a Markov head using a low rank factorization of a token to token transition bias (rank 256 by default), and an RNN head with a gated recurrent state over the in-block prefix. The drafter shares the target model's frozen embedding and LM head, and receives context through hidden states injected from selected target layers.

A confidence head with calibrated scheduling. For each draft position k, a sigmoid over a linear projection of the backbone hidden state and the previous token's embedding produces a score estimating the probability that position k survives verification given earlier positions were accepted. The training target is the analytical acceptance rate, one minus half the L1 distance between draft and target distributions.

The calibration is the part I liked most. Sequential Temperature Scaling moves left to right, picking a per position temperature by grid search on a held out set to minimize expected calibration error of the cumulative product. Temperature scaling preserves ranking, so the head's token ordering is unaffected while its probabilities become trustworthy.

The scheduler ranks candidate prefixes by survival probability and uses a profiled step time curve to estimate throughput at each batch size, then picks per request verification lengths. Under heavy load it drops low survival suffix tokens so the batch capacity goes to other requests. Under light load it keeps longer prefixes because verification is cheap.

The numbers. The DSpark paper reports offline macro average accepted length improvements of 26.7 to 30.9 percent over Eagle3 and 16.3 to 18.4 percent over DFlash across Qwen3 4B, 8B, and 14B targets. Online, against an MTP-1 baseline at matched aggregate throughput on DeepSeek-V4 serving, per user generation speed improves 60 to 85 percent on V4-Flash and 57 to 78 percent on V4-Pro, and under strict SLAs where the baseline degrades sharply, DSpark sustains throughput.

Launch coverage of DeepInfer specifically reported that confidence scheduling cuts verified draft tokens by roughly 30 percent while accepted length stays nearly flat, 3.29 against 3.20. The same report admitted it could not fully square that 30 percent with the published figures, so I would treat it as approximate.

Configuration:

export DEEPINFER_ENABLE_MTP_VARLEN=1          # enable the DSpark path
export DEEPINFER_USE_NEXTN_DECODE=5           # max draft length, default 5, set 0 to disable
export DEEPINFER_DFLASH_CONFIDENCE_HEAD_TEMPERATURE=1.0   # scalar, or one per draft position

That last one accepts a comma separated list with exactly DEEPINFER_USE_NEXTN_DECODE elements, which is the per position calibration temperature from the paper exposed directly as a knob. There are also separate CUDA graph controls for the drafter (DEEPINFER_DFLASH_ENFORCE_EAGER, DEEPINFER_DFLASH_CUDAGRAPH_BATCH_SIZE_INTERVAL), confirming DFlash runs as its own graph captured module.

One subtlety from config.py: draft length inflates the runtime position bound.

def _get_runtime_max_model_len(max_model_len: int) -> int:
    nextn = envs.DEEPINFER_USE_NEXTN_DECODE
    scheduler_overhead = nextn + (nextn + 1 if envs.DEEPINFER_ZERO_OVERHEAD_SCHEDULING else 0)
    draft_overhead = nextn
    return max_model_len + scheduler_overhead + draft_overhead

So with nextn=5 and zero overhead scheduling on, the engine reserves 16 extra positions per sequence beyond your max_model_len.

YoCo: SWA Bounded Replay

This is my favourite feature in the release. It is a free win that only exists because the model was designed for it.

The setup: in a YoCo model, layers from n_prefill_layer upward are consumer layers that reuse KV from earlier layers and use sliding window attention. On a prefix cache hit you would normally still have to replay the whole prompt through those consumer layers to rebuild their state.

But a sliding window layer can only see a bounded window back. So the visible domain for a truncated replay is the intersection of the sliding window and the retained suffix. If you keep enough tail tokens to cover the window plus whatever backtracking speculative decoding might need, the replay is correct while being independent of total context length.

Two modes are implemented, and the distinction matters:

# mask: keep all query tokens, only narrow the SWA visible domain
# compact: physically remove truncated queries, reducing consumer layer compute
export DEEPINFER_YOCO_PREFILL_TRUNCATION_MODE=compact
export DEEPINFER_YOCO_PREFILL_KEEP_TOKENS=140

The documented correctness constraint is KEEP_TOKENS >= window_size + DEEPINFER_MAX_BACKTRACKING_TOKENS, where max backtracking is itself floored at nextn + 1. Default is 0 and truncation must be enabled explicitly, which is the right default for a feature where getting the bound wrong is silently incorrect rather than loudly broken.

DeepSeek's stated result is that this nearly halves total prefill compute in the decoder layers. The README gives 140 keep tokens as the V4.1 default, while some launch coverage described 128 tokens of SWA state, so there is minor inconsistency in the public numbers.

There is one catch, and it sits in a comment in step() (translated):

# NOTE: when YoCo truncation is enabled, results may change due to preemption

So bounded replay and bitwise reproducibility under preemption pull against each other. If you need both, plan for that.

Asynchronous and Zero Overhead Scheduling

The claim is that CPU batch scheduling overlaps with device execution so the GPU never idles between steps, and it is enabled by default.

Beyond the C++ scheduler and the pooling already described, the interesting part is how prefill is coordinated across data parallel ranks. From SchedulerConfig (comments translated):

# Decode steps between prefill attempts on a NORMAL engine
# (ALL vote: prefill only if every rank has work)
prefill_attempt_interval: int = 70
# Fallback ANY vote: prefill together if any rank has work, to prevent starvation
prefill_any_attempt_interval: int = 100

Because prefill and decode are separate batch types in this engine rather than mixed in one batch, every rank has to switch to prefill together. So there are two voting points. Every 70 decode steps there is an ALL point: the ranks collectively prefill only if every rank has pending prompts, which maximizes prefill batch efficiency. Every 100 steps there is an ANY point: if any single rank has work, everyone prefills, which prevents an isolated rank's request from starving. An unlucky request waits at most 100 decode steps.

The engine also runs placeholder steps (is_placeholder=scheduler_outputs.is_empty()) so ranks stay in lockstep even with nothing to do, which is necessary when collective operations are on the critical path.

export DEEPINFER_ZERO_OVERHEAD_SCHEDULING=1   # on by default for NORMAL and COMPLETION engines
export DEEPINFER_PREFILL_ATTEMPT_INTERVAL=70
export DEEPINFER_PREFILL_ANY_ATTEMPT_INTERVAL=100

This is quite different from vLLM V1, where chunked prefill is on by default and prefill chunks are batched alongside decodes with decode prioritized. DeepInfer instead keeps the two phases as distinct global steps and coordinates the switch. Which is better depends on your workload: vLLM's approach smooths inter token latency, DeepInfer's maximizes prefill batch size and keeps decode steps uniform, which also happens to make batch invariance tractable.

Batch Invariance and Deterministic Sampling

Most engines produce slightly different logits for the same sequence depending on what else was in the batch, because reduction orders change with batch shape. That is usually tolerable and occasionally maddening, particularly for reinforcement learning from model outputs, for debugging, and for any compliance story that involves reproducibility.

DeepInfer ships both halves:

# identical model results regardless of batch composition
export DEEPINFER_BATCH_INVARIANT=1
 
# identical outputs for identical inputs and sampling params
export DEEPINFER_BATCH_INVARIANT=1
export DEEPINFER_ENABLE_MTP_VARLEN=0
# plus a fixed seq_seed per request

SamplingParams(temperature=0.7, max_tokens=256, seq_seed=12345)

The per sequence seed is a uint64 stored on SamplingParams and defaults to random.getrandbits(64) when unset, so determinism is a per request property rather than a global engine mode.

Two important limitations. Both features are NVIDIA only, which is the clearest feature parity gap between the two supported platforms. And deterministic sampling requires turning speculative decoding off, which is a real throughput cost. Reproducibility and the DSpark speedup are mutually exclusive today.

Structured Outputs

DeepInfer uses XGrammar, which is the same choice vLLM and SGLang made, and supports regex, JSON schema, and EBNF:

from deepinfer.sampling_params import SamplingParams
 
SamplingParams(temperature=0, max_tokens=128,
               so_args={"schema_type": "json", "schema": json.dumps(json_schema)})
SamplingParams(so_args={"schema_type": "regex", "schema": r"[A-Z]{2}-[0-9]{3}"})
SamplingParams(so_args={"schema_type": "ebnf",
                        "schema": 'root ::= "answer=" ("yes" | "no") ";level=" [1-3]'})

Three implementation details are worth more attention than the feature itself.

Grammar compilation is asynchronous on a dedicated CPU pool. There is an SOMatcherBuilder with a so_builder_has_pending property and DEEPINFER_SO_BUILDER_CPUS defaulting to 4, a 20 second timeout, and a cache with a 10 GiB default limit. Compilation never blocks the scheduling loop, which is the correct answer to the grammar compile latency problem I keep seeing teams hit.

export DEEPINFER_SO_BUILDER_CPUS=4
export DEEPINFER_XGRAMMAR_TIMEOUT=20
export DEEPINFER_XGRAMMAR_CACHE_LIMIT_BYTES=10737418240
export DEEPINFER_XGRAMMAR_DISABLE_CACHE=0

Grammar activation can be deferred by a trigger token sequence. From the SamplingParams docstring, so_args optionally takes trigger_token_ids, a token id sequence that activates the grammar once it appears in full, with free generation before that, and bound_token_id, which once in effect stops allowing free tokens and forces the trigger sequence token by token before entering the grammar. This is the clean solution to the reasoning then structured output problem: let the model think freely, then hard constrain the answer once a boundary token appears. Most stacks make you choose between constraining from token zero or not constraining at all.

A grammar bookkeeping failure kills only the affected request. The step loop checks so_status on the last prefill chunk and, on an unexpected state, logs an error and sets SequenceStatus.FINISHED_SO_FAILED for that sequence rather than failing the batch. A small thing, but the right call.

The repository's own structured output test is also a nice demonstration of constrained decoding being a hard guarantee rather than a suggestion: the prompts actively ask for the opposite of the schema, including "write a long story, no numbering" against a regex of [A-Z]{2}-[0-9]{3}, and "ramble in natural language, do not output JSON or code" against a JSON schema. The constraint wins and the test asserts an exact match.

EPLB: Runtime Expert Rebalancing

In a large MoE, expert load is uneven and shifts with the workload. A rank holding hot experts becomes the straggler for every all-to-all, and all-to-all is on the critical path for every MoE layer.

DeepInfer carries an expert parallel load balancer with a substantial tuning surface:

export DEEPINFER_OVERRIDE_NUM_EXTRA_EXPERTS=32        # redundant expert replicas, a lower bound
export DEEPINFER_EXPERT_COUNT_WINDOW_SIZE=100         # routing statistics window
export DEEPINFER_EXPERT_REARRANGEMENT_STEP_INTERVAL=0 # how often to physically rearrange
export DEEPINFER_MOE_LB_STAT_INTERVAL=128             # stats flush interval
export DEEPINFER_REARRANGE_AFTER_PREFILL_IN_NORMAL_ENGINE=0
export DEEPINFER_SHOW_EPLB_DETAIL=0
export DEEPINFER_FORCE_LOAD_BALANCE=0                 # testing
export DEEPINFER_FORCE_IMBALANCE_RATIO=0.0            # testing

The redundant expert count is explicitly a floor, not an exact value: per the examples documentation, the engine rounds num_routed_experts + num_extra_experts up to be divisible by ep_size, with the backbone and DFlash rounded separately, and the effective rounded value is what gets recorded and plotted.

The stats flush is visible enough to matter for measurement. The benchmark documentation notes that the EP all-gather for load balance statistics introduces jitter of "a dozen ms" and that intervals colliding with a flush are dropped from the measurement window. If you benchmark this engine yourself and see periodic outliers, that is probably what you are looking at.

There is also a TORCH_SYMM_MEM_ALLOW_OVERLAPPING_DEVICES=1 set at import with the comment "EPLB shuffle symmem," indicating expert shuffling uses PyTorch symmetric memory.

The Kernel Stack

DeepInfer sits on top of a stack DeepSeek has been publishing piece by piece for over a year. Most of the performance comes from these libraries.

A detail from the FlashMLA release notes that is relevant to anyone evaluating hardware: on Ascend 950, the sparse attention kernels reportedly reach 410 TFlops in prefill, described as 95 percent of hardware peak, and 360 TFlops in decoding, 83 percent of peak. Those are DeepSeek's numbers, but achieving 95 percent of peak on a non NVIDIA accelerator is a strong claim about how seriously the Ascend path is being engineered.

One caveat from launch coverage: the internal FlashMLA decoding kernel is said to be 5 to 10 percent faster than the public one, with DeepSeek putting the end to end gap at 5 percent or less. So the open release is not quite the production binary.

The pinned dependencies in requirements-nvidia.txt show how new all of this is:

torch==2.13.0+cu132
flash-attn-4==4.0.0b34

CUDA 13.2, PyTorch 2.13, FlashAttention 4 beta, plus a forced NCCL patch to 2.32.3 in build.sh. The Ascend path uses torch==2.13.0+cpu with torch-npu==2.13.0rc1.

Hardware: Two Backends, Unequal

Day one support covers NVIDIA SM100 (Blackwell data center) and Huawei Ascend 950 NPUs. Hopper is not supported, and FlashMLA explicitly dropped Hopper support on 2026.09.30 along with V3, V3.2 and V4.0 models and changed the FP8/FP4 KV cache format.

The dual backend story is handled cleanly in the build. From deepinfer/__init__.py, compiled artifacts live in per platform directories:

_so_dir = os.path.join(os.path.dirname(__file__), "_so", _so_platform_tag())
if os.path.isdir(_so_dir):
    __path__.append(_so_dir)

with the tag being <accelerator>-<arch>, so NVIDIA and Ascend builds of the same architecture coexist in one source tree. Platform selection happens through DEEPINFER_ACCELERATOR, and the platform object is initialized lazily specifically because "NPU initialization patches torch.cuda and initializes the ACL runtime, so it must happen only after ASCEND_RT_VISIBLE_DEVICES has been configured."

The two backends are not equal, though:

The politics are hard to miss, and launch coverage said so directly: giving Huawei equal billing with NVIDIA, after US export restrictions on advanced NVIDIA chips for Chinese firms and reported plans for large Ascend 950DT deployments, is a statement as much as an engineering choice.

The Published Benchmarks, Read Carefully

DeepSeek published Pareto curves rather than single numbers, which is the right way to do it: throughput against latency, so you can see the tradeoff instead of a cherry picked point.

Setup: 4 SM100 GPUs for prefill, 64 for decode, each within one scale up domain. From the figures as reported at 131,072 context:

• Decode (EP64 DP64, nextn=3, 75 percent acceptance): above 25,000 tokens per second per GPU at 256 requests per rank, around 100 tokens per second per user. At 8 requests per rank, per user speed exceeds 300 tokens per second.
• Prefill (EP4 DP4, 140 token bounded replay): about 60,000 non cached tokens per second per GPU at 4,096 tokens per rank, approaching 90,000 at 65,536 tokens per rank, at roughly 730 ms per step.
• DSpark dynamic verification: adds 4 to 7 percent throughput at around 60 sequences per rank, but is 1.5 percent slower than fixed 5 token verification at 16 sequences per rank.

That last line is the most credible thing in the set, because it reports a regime where the feature loses. Confidence scheduling helps when the batch is contended and costs a little when it is not, which is exactly what the mechanism predicts.

Things to keep in mind before quoting any of this:

The accounting is generous by construction. The benchmark documentation states decode throughput is computed as avg_SPS × decode_num_requests × (1 + nextn) / --cp "under full-acceptance accounting, 1 + nextn tokens per request per step, regardless of the true acceptance rate." So the headline decode throughput assumes every draft token is accepted. The 75 percent acceptance figure appears as a separate chart annotation, not as a divisor.

Per GPU numbers at 64 GPUs are not per GPU numbers at 8. Wide EP is where this engine wins, and the published curve shows EP64 ahead of EP8 at every batch size, so small deployments should not expect these figures.

The 155 trillion tokens per day cluster claim circulating in coverage is DeepSeek's own and is not independently verified.

That said, the benchmark harness is careful about how it measures: 5 warmup chunks skipped for prefill and 50 warmup steps plus 60 logged steps for decode, intervals colliding with EPLB stats flushes dropped, the profiler attached strictly last because "after the profiler exits there is still a residual slowdown of ~7-11%," and an explicit warning that in trace step times "up to nearly 50% inflation at bsz=1" means they must not be used to derive steps per second. The reproduction script is examples/sweep/run_sweep.sh, which sweeps batch size and plots the Pareto curve.

How It Compares

Here is how DeepInfer lines up against the engines you are more likely running.

In plain terms:

DeepInfer is not a vLLM alternative for most people. One model, two accelerator families, no server, no KV cache system, and a minimum sensible footprint of dozens of GPUs. If you serve Llama or Qwen on H100s, this release changes nothing for you operationally.

It is the reference implementation for how DeepSeek thinks inference should be built. That is where most readers will get value from it. The ideas are portable even though the code is not: dropping tensor parallelism when your attention is MLA shaped, moving the scheduler into C++ and pooling every hot path object, making draft verification length a scheduling decision rather than a constant, exploiting sliding window structure for constant cost replay, and treating batch invariance as a shipped feature rather than an aspiration.

It also owes a lot to vLLM. The README says DeepInfer "was originally built primarily on vLLM, with additional inspiration from SGLang and TensorRT-LLM," and the source carries # Derived from vLLM; modified by DeepSeek. headers with SamplingParams still documenting that it follows the OpenAI completion API. This is a fork that diverged a long way, and the divergence is the interesting part.

Expect the ideas to flow back. DSpark was already open sourced separately and reported as integrated into vLLM. The confidence scheduled verification concept in particular is model agnostic and likely to appear elsewhere.

Getting Started

If you have the hardware, the path is short.

# Prerequisites: Linux, GCC >= 13, Python 3.12, CUDA 13 (NVIDIA) or CANN 9.2 (Ascend)
# In a fresh, empty Python 3.12 environment:
export DEEPINFER_ACCELERATOR=nvidia   # or ascend; keep it exported at runtime
./build.sh

build.sh installs platform requirements, force reinstalls NCCL 2.32.3 on NVIDIA, builds vendored operators, then builds the DeepInfer extension.

# Offline inference, default checkpoint deepseek-ai/DeepSeek-V4.1-Flash-1017
python examples/offline_inference.py
 
# or from a local copy
export DEEPINFER_HF_MODEL_DIR=/path/to/DeepSeek-V4.1-Flash-1017
python examples/offline_inference.py --ep 8
python examples/offline_inference.py --ep 2 --dp 8
python examples/offline_inference.py --ep 2 --cp 8

The Python API is familiar if you know vLLM, with the parallelism argument being the tell:

from deepinfer.entrypoints.llm import LLM
from deepinfer.sampling_params import SamplingParams
 
llm = LLM(model="deepseek-ai/DeepSeek-V4.1-Flash-1017",
          expert_parallel_size=8,          # not tensor_parallel_size
          gpu_memory_utilization=0.9)
 
outputs = llm.generate(
    prompts=["Explain MLA in two sentences."],
    sampling_params=SamplingParams(temperature=0.7, max_tokens=256, seq_seed=42),
)

Benchmarking, with parallel scale derived from the environment as ep_size = WORLD_SIZE × len(CUDA_VISIBLE_DEVICES):

export WORLD_SIZE=1 RANK=0
export CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7
 
python examples/benchmark.py
python examples/benchmark.py --cp 8 --prefill_batch_size 65536
python examples/benchmark.py --decode_num_requests 1                      # low latency
python examples/benchmark.py --decode_num_requests 64 \
  --context_length 131072 --yoco_clip                                     # 128k with truncation

Evaluation covers AIME 2026, HLE, and GPQA Diamond with generation and scoring as separate stages, reporting Pass@1, with 32 samples per task for AIME, 1 for HLE across 2,500 tasks, and 8 for GPQA Diamond. Datasets are not downloaded for you.

Configuration reference

The flags I would reach for first, grouped by purpose.

Who Should Care

You should read the source if you build or operate inference infrastructure, regardless of whether you run DeepSeek models. The engineering decisions are unusually legible and several are directly portable.

You should deploy it if you are serving DeepSeek-V4.1-Flash, you have SM100 or Ascend 950 capacity at the scale of dozens of GPUs, and you have engineering capacity to build the KV cache system, disaggregation, and load balancing that the release deliberately omits.

You should ignore it operationally if you serve other models, run Hopper or older, need an HTTP server out of the box, or run fewer than roughly 64 GPUs. Nothing here replaces vLLM or SGLang for you today.

You should watch it if you care about where inference engineering is heading. The combination of confidence scheduled speculation, structural exploitation of sliding window attention, shipped determinism, and a serious non NVIDIA backend is a good hint of where mainstream stacks are heading.

Pitfalls

• Expecting a server. There is none. offline_inference=True is hardcoded in the entrypoint.
• Reaching for tensor_parallel_size. It does not exist. Think in expert parallel width and context parallel degree.
• Enabling YoCo truncation without the bound. KEEP_TOKENS must be at least window_size + DEEPINFER_MAX_BACKTRACKING_TOKENS. Default 0 means off, and getting the bound wrong is a correctness issue, not a crash.
• Expecting determinism and speculation together. Deterministic sampling needs DEEPINFER_ENABLE_MTP_VARLEN=0. You pick one.
• Expecting determinism on Ascend. Batch invariance and deterministic sampling are NVIDIA only.
• Quoting decode throughput as achieved tokens. The published figure uses full acceptance accounting at 1 + nextn tokens per request per step.
• Benchmarking into an EPLB stats flush. The all-gather adds jitter of around a dozen milliseconds. Their own harness discards those intervals.
• Reading step times from a profiler trace. Their docs warn of up to nearly 50 percent inflation at batch size 1.
• Assuming the open kernels are the production kernels. The internal FlashMLA decode kernel is reportedly 5 to 10 percent faster.
• Trying to run V3, V3.2 or V4.0. FlashMLA dropped them, along with Hopper, and changed the KV cache format.
• Setting gpu_memory_utilization too high with CUDA graphs on. The source notes that an illegal memory access error is often CUDA graph OOM, and the fix is to lower it.

Conclusion

DeepSeek open sourcing its production engine is the news. What the code says about how they think is the more interesting part.

Almost every distinctive choice in DeepInfer is a consequence of codesign. MLA makes attention cheap enough to replicate, so tensor parallelism goes away and expert parallelism becomes the only sharding axis that matters. Sliding window consumer layers make replay cost constant rather than linear in context, so prefill on a cache hit gets dramatically cheaper. A confidence head trained against analytical acceptance rate and calibrated position by position turns draft length into a scheduling variable. An n-gram memory table sharded across its own parallel dimension turns retrieval into a fetch over the interconnect. None of those are generic optimizations. They are what you build when the same organization owns the model and the serving stack and is optimizing a single number across both.

For most readers the code is not directly usable, and the release says so plainly: one model, two new accelerator families, no server, no KV cache system, and a sensible floor of around 64 GPUs per role. But the ideas travel, and some already have. If you want a preview of what inference teams will be debating in 2027, it is in this repository, along with the flags and the benchmark scripts to try it yourself.

References

1. GitHub: deepseek-ai/DeepInfer
2. Hugging Face: deepseek-ai/DeepSeek-V4.1-Flash-1017
3. DSpark paper: Confidence-Scheduled Speculative Decoding with Semi-Autoregressive Generation
4. The New Stack: DeepSeek open-sources the inference engine behind its API, and Huawei gets equal billing with Nvidia
5. MarkTechPost: DeepSeek Open-Sources DeepInfer, the Inference Framework Behind DeepSeek-V4.1-Flash
6. GitHub: FlashMLA
7. GitHub: DeepGEMM
8. GitHub: DeepEP
9. GitHub: TileKernels
10. GitHub: DeepSelect
11. VentureBeat: DeepSeek open sources DSpark
12. vLLM docs: Structured Outputs and Optimization and Tuning, for the comparison