Less Wasted GPU Work: DeepSeek Open-Sources DeepInfer, Its Core Inference Engine

DeepSeek has open-sourced DeepInfer, the core inference engine powering its V4.1 Flash online service. It orchestrates how GPUs divide workloads, reuse previously computed results, and generate responses continuously—minimizing redundant computation and idle waiting across hardware.

This technology is already deployed in production serving approximately 155 trillion tokens per day on a single cluster. The eliminated compute translates into concrete gains: when preparing the model to answer, a bounded local state reconstruction optimization nearly halves total prefill compute; during response generation, the system allocates verification compute based on confidence in draft tokens, achieving an estimated 4%–7% throughput gain under official near-production benchmarks compared to a baseline that validates a fixed 5 drafts per step. One reduces preparatory compute before generating an answer, while the other increases output per unit time during generation.

Alongside the engine, DeepSeek released deepseek-recipe: it converts API requests from external software into model inputs, and parses model outputs back into content text, thinking processes, or tool-call responses. What is delivered to developers is the core code ready for inspection, testing, and reuse; full online serving still requires separate cache management and service-level load balancing.

To understand the core challenges these two components address, consider a familiar real-world interaction: you send a long article to a large language model asking for a summary, and then ask a follow-up question about a specific point in that summary.

This interaction involves two primary computational phases, along with a caching mechanism that determines whether recomputation is required during follow-up turns:

1. Reading Inputs (Prefill): The model must first read the long text you provide. The text is split into the model's fundamental processing units—tokens(tokens do not map 1:1 to word counts; a single word may be split into multiple tokens). Because these input tokens are known upfront, GPUs can organize them into large matrix operations for batch computation, fully engaging tensor cores.
2. Step-by-step Generation (Decode): When generating the summary text, the model cannot anticipate future content; it must compute and predict the next token step by step based on preceding context. A Mixture-of-Experts(MoE) model comprises many computational modules called "experts," with each token selecting only a subset to participate in computation. While the model does not need to activate all experts on every step, each generated token still requires memory access and autoregressive progress, making it primarily constrained by GPU memory bandwidth and inter-device communication.
3. State Reuse (KV Cache): When you ask a follow-up question based on the summary, the prior context has already been computed. If the system preserves the previously computed key-value cache(KV Cache), subsequent conversational turns can reuse those precalculated states, avoiding redundant computation over the entire long article.

Targeting compute overhead across these stages, DeepInfer implements kernel optimizations, wide expert parallelism, bounded replay, and dynamic speculative verification within its offline engine; deepseek-recipe builds an efficiently aligned protocol and parsing pipeline bridging frontend interfaces and backend inference.

Reading Inputs and Generating Step by Step: Disaggregated Architecture at Scale

Processing long inputs and generating responses step by step impose distinctly different demands on hardware. Deploying them on separate infrastructure allows resources to be allocated independently, optimizing both time-to-first-token latency and token generation speed.

In its large-scale online serving infrastructure, DeepSeek opted to physically disaggregate Vision Encoding (VE), input Prefill, and step-by-step Decode across distinct server tiers (E/P/D Disaggregation):

• Prefill Phase: Faced with long documents or high-volume user prompts, compute characteristics favor large-batch matrix operations. DeepSeek deploys prefill on standard 8-GPU servers with moderate expert parallelism (such as EP2–EP8), leveraging GPU tensor cores to rapidly process uncached inputs.
• Decode Phase: Step-by-step autoregressive generation must balance batch concurrency, memory bandwidth, and inter-GPU communication. DeepSeek deploys decode on 64- to 128-GPU high-bandwidth interconnected supernodes using Wide Expert Parallelism (Wide-EP), distributing experts across more GPUs to dilute per-step memory access pressure.
• Vision Encoding (VE): Processes image inputs independently on single GPUs, balancing loads based on the number of image patches before routing visual features to Prefill nodes.

Through this division of labor, Prefill can focus on maximizing throughput and computational efficiency prior to generating the first token, while Decode concentrates on maintaining steady generation streaming rates for end users. Full online disaggregation relies on caching infrastructure and load-balancing services; what is open-sourced today is the offline inference engine core, which does not yet include the complete online serving orchestration.

Memory Access and Reuse: Wide-EP, KV Caches Stored by Type, and Bounded SWA Replay

During the Decode phase, each token activates only a fraction of the MoE model's experts. If executed on a small cluster of GPUs, each device must host many expert parameters, making GPU memory bandwidth a bottleneck under concurrent workloads.

Distributing Experts Across More GPUs, Adding Replicas for Hot Experts

DeepSeek's solution is Wide-EP, scaling expert distribution across 64 or even 128 GPUs. With fewer experts assigned per GPU, the volume of expert weights each GPU must load per step decreases. To prevent disproportionate compute loads when certain popular experts experience traffic spikes, the system incorporates Expert Parallelism Load Balancing (EPLB): GPUs maintain redundant expert slots as hot replicas, dynamically adjusting expert allocations based on recent execution statistics.

The chart below compares 8-GPU and 64-GPU configurations. Here, a DP rank denotes one data-parallel execution process, with each GPU corresponding to one rank. "64 requests per DP rank" indicates that each rank concurrently processes 64 requests, rather than 64 requests across the entire cluster. Single-user speed, per-GPU throughput, and aggregate cluster throughput therefore represent distinct measurement scopes.

Reusing Prior Computation on Follow-Up Queries

When users repeatedly ask questions about a long document, recomputing historical prompt tokens is costly. The production architecture segregates two categories of cache:

• Block KV Cache: The bulk chunked KV cache for the model body is stored in a dedicated 3FS distributed SSD cluster (~100 PiB in capacity), retained for approximately 3 days in Agent workloads and 7 days for Chatbots. Storing this cache separately allows any Prefill instance to read existing prefix caches, eliminating the need to route requests to specific machines solely for cache affinity—follow-up queries do not need to be locked to the server that initially processed the document.
• SWA State and Host Memory: Sliding-window attention (SWA) states are not written to 3FS; instead, they are held in the local host memory of the generating compute node and retrieved across machines over the local area network.
• Matching and Transfer: Prefix matching must be performed rapidly by entrypoint nodes using hash lookups prior to routing, verifying the valid hit length. Cache data is subsequently pulled asynchronously in the background, overlapping data movement with forward computation as much as possible to reduce transfer-induced stalls.

Reconstructing Only the Missing Slice When Local State Drops

This model architecture relies simultaneously on local window states and reusable block caches. Sliding Window Attention(SWA) calculates token interactions strictly within a bounded window. When long-context caches suffer partial misses or when initializing decoders, is it necessary to recompute thousands of tokens from scratch?

DeepInfer employs bounded SWA replay for approximate local state reconstruction:

• Block KV Hit, Local SWA Miss: If Block KV hits but the local SWA state is missing, the system does not need to recompute the entire document. It only recalculates the trailing 128 tokens of the matched prefix to rebuild local SWA state, directly reusing the existing Block KV.
• Decoder Prefill Phase: The model also divides labor internally: encoding layers process context first and hand results over to decoding layers. Block KV needed by decoding layers can be directly transformed from encoding-layer outputs without layer-by-layer recomputation; what remains to be restored is each layer's local SWA state. (This refers to internal computational layers, distinct from the physically separated server tiers discussed earlier.) Exact reconstruction of decoder SWA states would require passing the trailing 2,560 tokens through 20 decoder layers; bounded replay instead performs an approximate local recomputation over just the trailing 128 prompt tokens within the decoding layers. This approximation is used solely to bootstrap the current decode pass and is not persisted into prefix caches. Official estimates indicate this technique nearly halves total prefill compute.

In engine configurations, the retention parameter is typically set to 140 tokens (e.g., DEEPINFER_YOCO_PREFILL_KEEP_TOKENS=140). This is because the retained slice must cover both the 128-token local window and boundary buffers for speculative backtracking, each serving distinct mechanical roles.

Pipeline Overlapping: Asynchronous Scheduling to Minimize Device Idle Time

During autoregressive decoding in large language models, the standard execution flow typically proceeds as: GPU computes the current step (forward pass and sampling) → results synchronize back to CPU → CPU processes results and schedules the next batch → CPU dispatches new tasks to GPU. Between steps, the GPU frequently sits idle waiting for CPU scheduling and post-processing logic to complete.

Frameworks like SGLang and NanoFlow have explored pipeline overlapping extensively. DeepInfer similarly adopts an asynchronous scheduling mechanism: while the GPU executes the forward pass and sampling for step N, the CPU concurrently schedules the batch and dispatches tasks for step N+1 in the background, while interleaving post-processing for step N-1. By maximizing overlap between CPU preparation and GPU computation, the system minimizes GPU idle stalls.

DSpark: Focusing Verification Compute on High-Confidence Drafts

A common approach to accelerating autoregressive decoding is speculative decoding: a lightweight draft model predicts several candidate tokens in advance (up to 5 by default), which the target model validates simultaneously in a single forward pass.

The fixed-length baseline examined here verifies 5 draft tokens on every step. However, acceptance probabilities vary across different requests and token positions; validating all candidates unconditionally expends compute on low-confidence drafts.

DeepInfer's DSpark introduces confidence-guided dynamic verification scheduling. After the draft model proposes candidate tokens, a Confidence Head estimates acceptance probabilities at each position. Combining these estimates with hardware throughput characteristics, the system dynamically determines the optimal verification length for the current step, prioritizing verification budget on tokens with high expected yield:

• Production Benchmark Concurrency (60 Sequences per Rank): The mean verified draft length per step drops from a fixed 5.00 to 3.58; meanwhile, the mean number of accepted draft tokens dips only slightly from 3.30 to 3.21. Based on this, official benchmarks estimate per-rank throughput increases from 13,730 to 14,466 token/s, yielding an approximate +5.4% throughput gain.
• Impact of Concurrency Scale: Under high concurrency at 96 sequences per rank, the throughput gain from dynamic verification reaches +11.0%. Conversely, at a lower concurrency of 16 sequences per rank, throughput records -1.5% (dropping from 4,891 to 4,816 token/s/rank).

Serving More Users Versus Making Each Request Faster: Two Distinct Objectives

Through parameter sweep scripts across various batch configurations, official benchmarks trace Pareto curves of throughput versus latency across both Prefill and Decode phases. Interpreting these charts requires understanding their specific measurement methodologies.

1. Prefill Phase: Batch Size and Per-Step Latency

In prefill benchmarks, the horizontal axis records the physical elapsed time for a GPU to process a single batch chunk, while the vertical axis represents the number of uncached input tokens processed per GPU per second:

• At a batch size of 4,096, per-step latency is roughly 68 milliseconds with per-GPU throughput of approximately 60,000 token/s; expanding batch size to 65,536 raises per-GPU throughput to approximately 89,500 token/s, with per-step latency correspondingly increasing to roughly 732 milliseconds.
• Note that per-step chunk latency is not identical to end-user Time-to-First-Token (TTFT). In actual production serving, TTFT is also influenced by network transit, queuing delays, cache lookup, and chunked data streaming, while extremely long prompts are decomposed across multiple sequential chunks.

2. Decode Phase: Concurrency and Streaming Velocity

The decode Pareto curve illustrates the engineering trade-offs across concurrency levels:

• When a single DP rank processes only 8 concurrent requests, output throughput per GPU is approximately 2,600 token/s/GPU, translating to an effective single-user generation speed of roughly 325 token/s. At 256 concurrent requests, per-GPU output throughput climbs to approximately 26,300 token/s/GPU, while single-user effective speed scales down to roughly 103 token/s.
• This curve applies a fixed conversion parameter from the sweep script: setting a fixed 3 draft tokens (nextn=3) normalized at an assumed 75% acceptance rate, or 1 + 3 × 0.75 = 3.25 tokens per step. This figure is not a direct measurement of end-user terminal typing latency, nor does it correspond to the 5-draft setup in the earlier DSpark evaluation; it serves as a standardized baseline for benchmarking output throughput incorporating speculative decoding gains.

deepseek-recipe: Equipping Self-Hosted Deployments to Handle Real-World API Traffic

Co-released with DeepInfer, deepseek-recipe is a request translation and tokenization component written in Rust with Python bindings. It connects upper-layer standard APIs with the underlying inference engine:

Upper-Layer Requests (Messages / Chat Completions / Responses)
  │
  ▼
deepseek-recipe: Convert to unified Conversation format
  │
  ▼
Render template and encode into Token ID sequence (with structured output constraints, etc.)
  │
  ▼
Caller's inference engine executes forward pass (DeepInfer, etc.)
  │
  ▼
Parse output token stream into events (thinking process, content text, tool calls)
  │
  ▼
Assemble stream and return protocol-compliant response

The core responsibility of this component is ensuring identical processing logic to DeepSeek's official API. Callers must still provide inference execution, HTTP transport, and tool runtime environments independently. This parity manifests across three specific areas:

1. Isolating Control Tokens from Plain Text

During prompt assembly, deepseek-recipe explicitly tracks the positions of template control tokens (such as role boundary markers), mapping them directly to dedicated special Token IDs. Conversely, all plain text originating from user inputs or tool returns is strictly encoded using the base vocabulary. For example, if a user includes <｜User｜> as literal text within their prompt, the string is encoded as standard text characters rather than being misinterpreted as an actual role marker. Similarly, during output parsing, the model's genuine thought termination token </think> and tool delimiters are identified via special Token IDs; even if the model emits matching literal text inside content tokens, it is preserved as plain text, preventing interference with flow control.

2. Segmented Parallel Tokenization and Structured Constraints

Because the positions of special tokens within the template are fixed during prompt construction, plain text segments sandwiched between markers can undergo BPE encoding independently. deepseek-recipe tokenizes these segments in parallel before concatenating them, cutting encoding latency on long-context prompts. For requests requiring strictly formatted responses, deepseek-recipe organizes and passes structured output constraints, leaving enforcement to the inference backend. DeepInfer supports compiling JSON Schemas, regular expressions, or grammar rules into grammar state machines. At each decode step, the engine evaluates the state machine to identify valid candidate tokens and applies logit masks, enforcing compliance with the predefined schema—guaranteeing structural validity, though not factual accuracy.

3. Cross-Chunk Streaming Parsing and Tool Parameter Chunking

Inference engines emit outputs token by token, meaning multibyte UTF-8 characters, DSML tags structuring tool calls, or serialized tool arguments may be fragmented across distinct packets:

• Upon encountering incomplete multibyte UTF-8 sequences, the component's internal state machine buffers intermediate bytes until subsequent tokens complete the character, preventing garbled characters from being pushed to the client.
• When tool calls are detected, the component does not wait for the entire JSON payload to complete; it parses arguments along DSML grammar boundaries, escaping and streaming partial arguments to the client as they are generated so clients receive initial parameter fragments sooner.

Which Teams Can Adopt It, and What Remains for Full Online Serving?

1. Open-Source Scope and Omitted Components

• Included in the Release: DeepInfer open-sources the core offline inference engine, covering attention data parallelism (DP), MoE expert parallelism (EP), bounded SWA replay, DSpark speculative verification, and structured output support, tailored for the DeepSeek-V4.1-Flash model. The accompanying kernel suite includes DeepGEMM, DeepEP, TileKernels, and FlashMLA (with an official note indicating it lacks one hand-tuned SASS optimization present in the internal version, making the decode kernel 5%–10% slower, with end-to-end variance within 5%). deepseek-recipe provides the full request conversion, tokenization, and streaming parser implementation. The codebases are released under Apache 2.0 and MIT licenses, respectively, with official commitments to update repositories with inference code and optimizations for future DeepSeek models.
• Lineage and Excluded Elements: DeepInfer was initially developed primarily atop vLLM, drawing architectural insights from SGLang and TensorRT-LLM. Components not included in this release comprise the KV cache management system deeply integrated with production clusters, cross-DP replica load-balancing gateways, and complete online E/P/D disaggregated orchestration.
• Hardware Support: DeepInfer debuts with support for NVIDIA SM100 GPUs and Huawei Ascend 950 NPUs.

2. Deployment Footprints and Applicable Scenarios

• Cluster Footprints and Test Setups: The official production recommendation for high-efficiency operation is a 64-GPU Prefill cluster paired with a 64-GPU Decode cluster (64P+64D, maximizing Wide-EP memory access gains). However, the framework does not strictly require 128 GPUs to operate; documentation notes that deployment can run with one Prefill service instance and one Decode service instance (1P+1D)—where an instance does not imply a single GPU—and offline examples in the repository provide an 8-GPU single-node operational setup.
• Value Across Engineering Roles:
  • Teams with Compatible Hardware: Can directly evaluate long-context throughput using DeepInfer's offline engine and kernel libraries as an industrial reference for distributed inference.
  • API Service Integrators: Even without deploying the DeepInfer engine, teams can independently adopt deepseek-recipe to standardize multi-protocol request conversions, template rendering, and streaming event parsing.
  • Teams Running Regression Testing: NVIDIA platforms also offer batch-invariance and deterministic sampling configurations, facilitating consistency verification across identical inputs; deterministic sampling requires fixing the random seed and disabling dynamic drafting.