How LLM Inference Engines Actually Work: From a Single Token to Agent Caching

To most developers, calling a large language model feels like a straightforward black-box network request: we send over a prompt, a GPU somewhere in the cloud runs the model, and lines of text stream back. But step inside the server rack, and the physical reality looks completely different: an expensive GPU's memory is constantly at risk of filling up, compute cores can idle because they are starved of input, and the underlying system must continuously coordinate the handoff of work between the CPU and GPU.

At the Open Source AI Summit, Zain Hasan, an engineer at Together AI, delivered a technical deep dive into the inner workings of inference engines. Stripping away the black-box abstractions, the talk reconstructed the physical reality of LLM inference: an inference engine is not a mere mathematical calculator, but a continuous saturation machine engineered to wring every drop of compute from a $30,000 GPU.

Naive Illusions vs. Physical Truth: The Model Emits Just One Integer at a Time

Slide 3 sketches the naive mental model people have of LLM calls: a user asks a question, the request travels to a GPU or model somewhere, and the answer returns directly.

This diagram intentionally omits tokenization, scheduling, KV cache memory management, and the token-by-token autoregressive loop, setting the stage to demystify the black box. Such a simple interface easily creates the misconception that an LLM works like a code compiler—parsing an entire syntactic structure and outputting a complete piece of writing in a single pass.

Yet a GPU has never seen an English letter or Chinese character. Before a request ever touches the GPU, a tokenizer running on the CPU must first chop the text into pieces and map them to integer IDs that a computer can recognize.

The vocabulary in the original slides contains over 100,000 entries and offers a rule of thumb: one token equals roughly three-quarters of an English word. What is fed to the model is not a sentence, but a pure tensor of integers.

Once inside the GPU, the generative behavior of an LLM can be summarized in a single sentence: one forward pass predicts only one token.

Based on the current input sequence, the model calculates the single most probable next integer, appends it to the end of the sequence, feeds this expanded sequence back in as the input for the next calculation, and runs another forward pass. This single-step loop repeats continuously until the model emits a stop token or hits a maximum length limit. To produce a 500-word English response, the model must run roughly 700 full forward passes on the underlying hardware back-to-back.

The Dual Personality of Inference: The Fundamental Divide Between Prefill and Decode

Although these 700 iterations are driven by the exact same model weights, the inference lifecycle is cleaved into two phases with polar-opposite physical characteristics.

The first phase is prefill (Prefill). Here, the GPU ingests the user's entire prompt all at once, allowing attention tensors across all input positions to be computed in parallel. The matrix multiplications are massive, easily saturating tens of thousands of GPU compute cores. This phase is purely compute-bound, and the time it takes directly determines how long the user waits for the very first word to appear—the Time To First Token (TTFT).

The second phase is autoregressive decoding (Decode). Once generation begins, the model is forced to degenerate into single-step serialization: each forward pass yields only a single token. At every step, the hardware must read the context cache accumulated up to that point before calculating this one new token. With minimal parallel computation per step and massive memory reads, the bottleneck flips to being memory bandwidth-bound, which strictly bounds how many tokens can be generated per second (Tokens Per Second, TPS).

This computational duality forms the foundational cornerstone for understanding every scheduling mechanism in modern inference engines.

The Memory Monster Known as KV Cache: Trading Space to Dodge Quadratic Recomputation

During the Decode phase, the self-attention mechanism requires the new token to compute correlations against all preceding tokens. Without any caching, every single generation step would force the hardware to start from token 1 and recompute the entire sequence's Key and Value matrices from scratch. The compute requirements would explode at an unacceptable quadratic rate as text length grows.

To avoid this colossal waste of work, inference engines introduce the KV Cache: once computed, the Key and Value tensors of all historical tokens are retained in GPU memory. Each subsequent step only needs to compute a single new column of vectors for the incoming token, fetching all past columns directly from VRAM.

As illustrated in the original slides, the KV Cache transforms redundant K/V calculations at each step into computing only one column for the new token, while reading previous columns from the cache. The volume of reads still scales with context length, and this trade-off of space for time causes the KV Cache to expand alongside concurrent users and context size. Consequently, the presentation identifies VRAM capacity—rather than raw compute alone—as the typical ceiling on inference batch size.

The Three-Tier Pipeline Architecture of an Inference Engine

Slide 8 highlights four core constraints that real-world inference services must juggle simultaneously:

• Steep hardware costs: A GPU costs around $30,000 and should never sit idle;
• Divergent concurrency states: The system simultaneously serves hundreds of users, each at a different stage of their request;
• Minimal step yield: Each user produces only 1 token per forward pass;
• Memory bottlenecks strike first: GPU VRAM capacity is frequently exhausted long before compute capacity.

These four physical constraints serve as the causal hinge for understanding the subsequent system engineering: to keep costly GPUs from idling and to coordinate user requests at disparate stages while memory runs out before compute, inference engines require a CPU scheduler, Continuous Batching, and paged KV memory (PagedAttention).

Consequently, a modern inference engine is fundamentally a continuous feeding pipeline built around the GPU. The overall system is distinctly partitioned into three tiers, with model weights residing exclusively in the final tier:

API Server (running on the CPU): Responsible for handling HTTP I/O, slicing incoming prompts into integer token sequences, and on the egress path, detokenizing the integer IDs computed by the GPU back into readable characters before streaming them to the client.

Scheduler (running on the CPU): The brain of the entire engine. Before every execution step (pass), it decides which requests enter the active batch, how many tokens each request processes, and which KV memory blocks to allocate.

Execution Workers (running on the GPU): Each GPU corresponds to an independent worker dedicated to high-intensity tensor-parallel computation. It receives packed batches from the scheduler, executes a forward pass, samples from the vocabulary space, and returns the selected token IDs to the CPU.

The three tiers advance concurrently along the timeline: while the GPU executes the forward pass for step N at full throttle, the CPU-bound API process is already detokenizing the tokens produced in step N-1 in parallel and pushing them across the network.

At the exact moment the GPU produces its final output, the underlying system undergoes a probabilistic collapse from hundreds of thousands of dimensions down to a single integer.

The original talk walked through a concrete sampling example: the GPU's final linear layer produces raw scores (logits) across a vocabulary of over 100,000 entries. After reshaping the probability distribution using temperature, Top-p, Top-k, and repetition penalties, the GPU performs stochastic sampling across the candidate set (for instance, candidates in the vocabulary scoring 0.52 for "mat", 0.21 for "rug", 0.14 for "floor", and so on) to settle on discrete ID 4352, which is then handed off to the CPU to be translated into text and emitted.

The Three Classical Scheduling Pillars for Saturating Hardware

In an ideal scenario, the scheduler would like to process requests from hundreds of users concurrently. In practice, however, every user brings a different input length and generates at an unpredictable pace. Under traditional batching logic, hardware utilization collapses quickly. To keep GPUs saturated, modern inference engines rely on three core mechanisms.

Continuous Batching

Under early static batching, the system bundled several requests together into a single batch and handed it to the GPU. Because generation lengths vary wildly across requests, a short query might finish at step 10, whereas a long query might require 500 steps. Across the remaining 490 steps, the compute slots vacated by finished requests were forced to sit idle, waiting until the slowest request in the batch completed before the entire batch could be swapped out.

Continuous Batching shifts the scheduling granularity by re-evaluating and reconstructing the active batch at every single generation step. The moment a request emits a stop token and vacates its memory slot, the scheduler immediately slips a new request into the opening on the very next forward pass. All slots remain saturated, eliminating the compute bubbles caused by passive waiting.

PagedAttention

If Continuous Batching curbs the waste of compute slots, PagedAttention uses on-demand chunking to eliminate the waste born of memory pre-allocation and fragmentation.

Previously, the system had to pre-allocate a contiguous chunk of physical VRAM for each user based on the maximum possible request length (such as 4K or 8K tokens). The original presentation noted that under this static reservation model, just 3 long requests could lock up an entire GPU's VRAM—even if they had only generated a few tokens and left most of the reserved memory empty—leaving a 4th request stranded in the queue.

PagedAttention borrows the virtual memory paging principles of modern operating systems: slicing the KV Cache into fixed-size physical blocks that are allocated on demand and dynamically attached. These physical blocks do not need to be contiguous in VRAM; instead, they are addressed logically via a virtual mapping table. With a shared dynamic block pool, VRAM that previously choked on 3 requests can now comfortably run 4, while still keeping half of the pool free and available. As the original slides summarized, KV cache capacity typically constrains batch size before compute power does.

Chunked Prefill

When a system hosts both short requests actively generating text and incoming ultra-long prompts at the same time, another tension emerges.

If the scheduler allows a massive prompt to monopolize an entire forward pass, other active users in the decoding phase will be starved of new decode tokens during that step.

Chunked Prefill resolves this by enforcing a per-step compute budget (illustrated in the slides as 2K tokens per step). Faced with an 8K prompt, the engine does not compute it in one go; instead, it chops it into four consecutive 2K chunks (0–2047, 2048–4095, 4096–6143, and 6144–8191), processing them across four separate forward passes. While executing each 2K prefill chunk, the scheduler piggybacks single decode tokens from active users into the same batch. This keeps the long prompt's prefill moving steadily forward without stalling token generation for active users at any step.

The Agent Surge: The Expanding Triangle and Prefix Caching

As large language models transition from conversational chat to autonomous agents, underlying traffic patterns shift dramatically.

In a typical agent loop, a human supplies an objective, and the model first deliberates before emitting a tool call command; an external system (a search API, a Python interpreter, an internal service) executes the tool and appends lengthy execution results back into the conversation log; the model then re-reads the entire history to decide its next action. This cycle repeats dozens of times.

To the inference engine, every iteration of an agent is not an incremental update, but a brand-new request that resends the entire conversation history from scratch.

Under the hood, the engine faces an unusual "expanding triangle": the input prompt inflates dramatically with each turn (system instructions + dozens of tool definitions + an ever-longer trajectory), while the model's output per turn remains extremely brief (often just a few dozen tokens of tool-call JSON).

Under conventional uncached logic, every agent turn would treat all the text generated across prior rounds as a fresh prompt, running a full prefill pass from scratch.

To quantify this overhead, Slide 17 presents an illustrative computational model showing the benefits of prefix caching:

Readers should be explicitly cautioned that while the slide title reads "across a 100-turn loop," the horizontal axis in the diagram terminates squarely at TURN 10. The figures 55 and 10 actually correspond to the 10 turns shown in the diagram under the assumption of a model where each turn appends an equal tail of 1 illustrative unit: without caching, the 10 turns sum to 1 + 2 + … + 10 = 55 compute units, whereas with caching, each of the 10 turns processes only 1 unit of newly appended tail, totaling 10 units. Thus, 55 and 10 are merely illustrative units derived from the 10 turns shown with equal-length incremental tails, rather than measured benchmarks or totals for 100 turns. Without prefix caching, every turn reprocesses the entire history of the previous turn.

The weapon that breaks this vicious cycle is Prefix Caching.

Because the underlying memory is already paged via PagedAttention, the engine can compute hashes for the contents of each previously evaluated physical block. Whether it is a multi-turn agent loop for the same user or identical system prompts and tool schemas shared across different users, as long as the prefix text matches, the new request simply points to physical blocks 0 through 3 already residing in VRAM via the mapping table. The original talk stressed that this capability operates completely transparently under the hood, requiring no special API flags from callers.

With Prefix Caching in place, each loop iteration only needs to run prefill on the brief tail newly appended in that round, while the lengthy history hits the cache. Based on the illustrative units of 10 turns with equal-length newly appended tails shown in the diagram, the total prefill compute across 10 turns drops to 10 units, down sharply from 55 units without caching; the original slides also note that the first token arrives faster on every turn.

Mapping Traffic Characteristics to Systems Engineering

To help readers clearly connect application-level agent behaviors with the inference engine's engineering solutions, Slide 18 organizes common agent traffic patterns and system countermeasures into a structured mapping:

Highly shared prefixes across turns and users: Heavy system personas, dozens of tool prototypes, and early conversation history remain identical across repeated calls. The engine relies on Prefix Caching to achieve block-level reuse and avoid redundant computation.

Concurrent tool execution and sub-agent bursts: When an agent spawns multiple sub-tasks or triggers parallel search queries, instantaneous concurrency spikes. The engine relies on Continuous Batching to dynamically backfill vacated batch slots.

Massive input context paired with minimal generated output: To handle the extreme asymmetry of processing tens of thousands of prompt tokens only to emit a few dozen tokens of tool calls, the engine uses Chunked Prefill to slice up long inputs, and at the cluster level, disaggregates dedicated high-throughput Prefill nodes from dedicated high-bandwidth streaming Decode nodes.

Prolonged silence during tool execution: When an agent calls code execution or external retrieval, it may wait several seconds or even minutes, during which the conversation channel exhibits zero generative activity. Allowing its KV Cache to sit locked in expensive GPU VRAM would rapidly exhaust system capacity. The engine employs KV Cache Offloading to swap this data into comparatively inexpensive, abundant CPU memory during idle waiting periods, rapidly swapping it back in when the tool returns to trigger the next turn.

Conclusion: Four Principles Grounded in Systems Common Sense

From a single hardware accelerator to distributed inference clusters, complex optimization techniques ultimately converge on four fundamental physical realities:

First, a single forward pass yields only one token. The model ingests Token IDs rather than raw words; the physical hardware executes probability calculations and single-step sampling across vocabulary space.

Second, Prefill and Decode are fundamentally different computational workloads. The former is bounded by compute cores and dictates time to first token; the latter is bounded by memory bandwidth and dictates generation speed in tokens per second. Pinpointing the active physical bottleneck is the prerequisite for all throughput optimization.

Third, an inference engine is a hardware-saturation machine pushed to the absolute limit. Continuous Batching fills idle bubbles at every compute step with dynamic backfilling, while PagedAttention dismantles memory pre-allocation waste using virtual memory concepts. The singular goal of systems engineering is to keep expensive cores from ever idling.

Fourth, agent execution efficiency is deeply tethered to caching strategies. Across multi-turn loops characterized by lengthy inputs and repeated replays, keeping the prompt's prefix structure deterministic and stable is the greatest engineering kindness an application developer can offer the underlying inference engine.