What context engineering actually engineers

Models are powerful, but the system can still miss the right context

Common failures in real AI products rarely boil down to 'poorly written prompts' alone. The model may lack access to internal company documents, miss recent business changes, forget decisions already made, or pick the wrong tool from a dozen similar options. Adding more prompt text only patches the immediate call; it doesn't address gaps in data, state, or action interfaces.

In this context, engineering context means making runtime decisions about selecting and assembling intents, evidence, rules, state, and tool results, so the model has enough—and the right—information for the current step.

The handbook categorizes the risks of long context windows into four types: contamination from repeatedly reusing erroneous content; distraction caused by too much irrelevant material; confusion from similar pieces of content jumbled together; and conflicts arising from contradictory rules coexisting. Together, these make one point clear: the context window is an attention budget, and its capacity and its quality must be managed separately.

This means every round of execution requires four explicit decisions: which information stays in the current window, which state is offloaded externally and retrieved on demand, which history can be compressed into a summary, and how much space to reserve for planning and answering. Focusing only on 'how many tokens can we fit' misses the crucial budgeting question: how much attention does the model have left to decide the next step?

A complete system repeatedly cycles through six actions: first, understand the intent; then, select evidence; assemble rules and current state into the window; decide whether to act; write necessary state externally; and finally, use the results to refine the next round. The agent orchestrates this chain, with query enhancement, retrieval, prompting, memory, and tools providing the interfaces.

The handbook gives the agent concrete responsibilities: periodically compress history, verify that retrieval results are consistent and usable, delete stale or irrelevant context, rewrite queries, switch knowledge bases, or change chunking strategies when the first retrieval attempt fails, offload temporarily unused details to external storage, load only relevant tools, and synthesize answers when multiple sources conflict. The agent doesn't replace these technologies; it decides at runtime which one to use and when.

This runtime pipeline also suggests a debugging order. If the answer goes off track, start by checking whether the intent was misunderstood. For factual errors, check the retrieved evidence. If the model keeps reversing old decisions, check whether state is being persisted. If the wrong tool is used, examine the tool contract and selection logic. Switching models outright often skips the cheapest and most identifiable points of failure.

Step one: turn user questions into executable system intents

People don't talk in database fields. A user might write 'why is this still not working', cram three goals into a single sentence, or assume the system knows 'these customers' refers to a specific time period. If the entry point isn't clarified, retrieval, reranking, and generation will all be optimizing for the wrong objective.

Query rewriting, expansion, decomposition, and query agents each handle a different scenario.

Query rewriting itself involves three basic actions: rephrasing awkward or heavily elliptical questions, removing irrelevant background that interferes with retrieval, and adding keywords commonly found in target material. When necessary, complex questions are broken into sub-questions. The goal is to prepare different query forms for different stages: the version for the retriever optimizes for matching, while the version for the final answer retains the full intent, rather than simply polishing the user's original words.

For example, 'help me see why customers churned recently and find three worth re-engaging' contains at least three layers of intent: what period 'recently' refers to, what metrics define churn, and whether 'worth re-engaging' is determined by spend, activity, or probability of recovery. The system might first rewrite it into structured questions, then separately retrieve the causal analysis and customer segments, and finally synthesize an answer. If the data is scattered across a CRM, ticketing system, and call logs, a query agent is needed to dynamically decide the order of search, filtering, and aggregation based on each database's schema.

A full query agent goes through at least three stages: analysis, execution, and optional synthesis. It first determines the question type, the necessary sources and filters, then executes the search. If the results have missing fields, conflicting evidence, or are insufficient for the task, it assesses the gaps, rewrites or expands the query, and retries. Only when the evidence is complete does it move to synthesizing an answer. This loop is more expensive than 'generate right after one search', but it exposes retrieval failures as recoverable intermediate states.

Query expansion boosts recall but can also drift further from the original intent. Decomposition improves precision but adds more calls and makes synthesis harder. A query agent can handle unknown paths but brings latency, cost, and debugging complexity. The choice should hinge on the task: is the intent ambiguous, are there multiple objectives, does it span multiple databases, and is it worth continuing to explore after the first retrieval fails?

At the entry stage, aim to leave at least three inspectable artifacts: a normalized intent that retains the user's original phrasing, a set of non-overridable scope conditions, and an acceptance statement describing what constitutes a completed answer. Without these three, it's hard for the system to tell whether it's still exploring or has drifted off course.

Step two: only the knowledge needed now goes into the context window

Once the intent is clear, the system faces the next problem: with far more material than the window can hold, which small portion is both easy to find and carries enough context? This is what retrieval and chunking are actually about.

Chunk too small and individual snippets are easy to match, but you might lose the subject, conditions, or exceptions. Chunk too large and the context is complete, but the vector representation gets averaged out over multiple topics, making precise retrieval harder. The handbook frames this as a sweet spot between 'retrieval precision' and 'contextual completeness'. There's no universal number that works across the industry, as it depends on document structure, question granularity, and the downstream task.

Fixed-size chunking is fast and works well for short, regular material. Recursive chunking processes text level by level (paragraphs, sentences) and is a common default. Heading- or function-based chunking suits well-structured documents and code. Semantic, LLM-based, and agentic chunking use higher cost to detect topic boundaries. Hierarchical chunking keeps summaries, sections, and details all available. Late chunking lets the representation see the full text first, then generates segment representations from it. Beyond the method, you also decide when to chunk: pre-chunking pushes the cost into the offline stage, making queries fast but the strategy fixed; post-chunking processes dynamically based on the current question, offering flexibility but slower first rounds.

The selector below merges the handbook's strategy matrix with the pre/post trade-offs. It provides a starting point for engineering, not a performance guarantee.

After the choice, task-level evaluation is necessary. This site suggests starting with a small set of real questions, tagging the original passages that must be hit for each one. Then check whether the retrieval results cover the answer, whether they carry the necessary qualifiers, how much of the results are irrelevant, and whether the final answer can trace back to the source. If you only look at vector similarity or a single successful demo, the chunking strategy is likely to fail when the document set changes.

This also reveals the vendor perspective: Weaviate naturally puts the vector database at the center. Production systems might also combine keyword retrieval, relational databases, file indexes, event logs, and direct tool queries. The goal of the retrieval layer is to get usable evidence, not to prove that any one storage method is superior.

Step three: turn a single call into a long-running task with prompts and memory

Retrieval results don't automatically become good context. The system still has to decide how much space to allocate to task rules, evidence, recent dialogue, tool descriptions, and output space, as well as which details are only relevant for the current step and which state is worth preserving across turns.

Prompts serve as the control layer: they define goals, boundaries, trigger conditions for tools, parameter formats, return values, and completion criteria. The handbook covers few-shot, Chain of Thought, Tree of Thoughts, and ReAct. The genuinely transferable parts are using examples to fix the output contract, decomposing complex tasks into checkable stages, and feeding action results into the next round. There's no need to require the model to show its private reasoning; plans, tool parameters, execution results, citations, and state changes are sufficient for auditing.

Memory, meanwhile, extends the time horizon. The current window holds recent messages and the material needed for this step. Working memory keeps incomplete state for a multi-step task. Episodic memory records events that have happened. Semantic memory stores reusable facts. Procedural memory saves validated methods. Each has different storage locations, write gates, and cleanup gates.

'Storing everything' creates a different kind of forgetting: important information gets buried under repetitive, stale, and low-quality records. A usable memory system needs to answer at least five questions: who can write, under what conditions, how conflicts are merged, when to retrieve, and when to expire or delete. Retrieval quality is just as important as write quality. When erroneous memories are repeatedly retrieved, a single mistake spreads into a persistent bias.

Memory architecture must serve the task. A customer service agent relies more on episodic memory to reconstruct user interaction history, while a financial analysis agent depends more on stable domain facts. Simple tasks can start with a buffer of recent dialogue turns; there's no need to build a full long-term memory system from the start. The choice of layer depends on whether the task requires cross-turn state, personalized history, stable knowledge, or reusable processes.

Retrieving memories also shouldn't be a single blind search. The handbook's improvement path includes: expanding or iterating the query first, then retrieving candidates; using reranking to push the genuinely relevant records to the top; and, when necessary, letting a query agent search across multiple collections or letting personalization logic incorporate user preferences and interaction history into the reranking. Storing is just the first step; retrieving the right record at the right moment is the real test of a memory system.

The handbook frames context offloading as moving information outside the window and bringing it back when needed. In practice, the lightest implementation is often a structured task state: objectives, completed steps, key decisions, unresolved issues, and evidence locations. That's cheaper than keeping the entire chat history and easier to verify.

If you want to see this principle applied in a real product, this article on the site breaks down how Claude Code reduces its system prompts and tool descriptions.

Related readingAnthropic slashes Claude Code's system prompt by 80%: six new rules for writing contextRules gave way to judgment calls and examples gave way to interfaces — one tool description shrank from roughly 9,100 characters to a single sentence and an enum.→

Step four: let the agent choose tools and continue with the results

Tools take the model from answering questions to taking action. Function calling advances from 'outputting text that looks like a command' to producing structured function names and parameters. MCP standardizes how resources, prompts, and tools are connected. A successful connection only proves the interface works; it doesn't show that the agent will invoke it at the right time, with the right parameters, and with proper authorization.

A single tool call and a multi-step tool chain are different levels of difficulty. Checking a flight once only requires picking the tool, filling in the parameters, and interpreting the result. Planning a weekend trip might require querying flights, hotels, and local activities in sequence, where the output of each step becomes the input constraint for the next. Tool chains add plan ordering, intermediate state, partial failures, and stopping conditions.

A reliable tool loop includes at least seven checkpoints: discovering available tools, filtering out irrelevant ones, choosing or planning, constructing parameters, passing authorization, executing and validating the Observation, and finally deciding whether to continue, switch strategies, ask the user, or stop.

Parameter construction is the most underrated step. Even when the agent picks the right tool, it can miss required fields, mix up units, or generate malformed parameters. The handbook's example shows a recoverable path: the first call fails because of bad parameters, the error comes back as an Observation, and the agent fixes the parameters and tries again. So-called self-correction must land on an observable process of reading the error, changing the parameters, and re-validating—not just a line saying 'the model will reflect'.

Tool descriptions are part of the context. A good contract specifies the verb, scope of use, required parameters, units and format, return structure, error states, and limitations. Tool returns also need size control; dumping tens of thousands of lines of logs back into the window will drown the next decision in noise. Keeping a summary, key fields, the location of the original result, and an inspectable reference is usually more robust.

MCP's value is in reducing repetitive integration work. In the traditional approach, M AI applications each connect to N external systems, potentially requiring M×N custom integrations. A standard protocol lets application-side and server-side each adapt once, collapsing the integration problem to M+N. It improves modular composability, not call reliability. The core primitives in the official spec—Resources, Prompts, and Tools—go far beyond just 'hooking tools up to the model'. It still doesn't decide permissions, approvals, data boundaries, idempotency, retries, or stopping conditions. Treating a successful MCP connection as a completed agent is like assuming an appliance is safely running just because the socket is installed.

Multi-agent systems follow the same rule. Specialized sub-agents can work on narrow tasks in parallel with clean context, while the main agent only receives the compressed results. The cost is that task decomposition, evidence handback, conflict resolution, and shared state become harder. When a task shows no clear parallelization benefit, a single agent with well-defined tools and state is often easier to verify.

A context engineering system ultimately needs to pass these six checks

This handbook puts prompts, retrieval, memory, and tools into a single system view. It doesn't offer experimental results, nor does it choose a framework for you. Instead, it leaves you with six engineering gates: every step must have inspectable artifacts, clear failure signals, and a defined handoff to the next step.

To evaluate an agent, you can work backwards from failure: does the intent layer retain the user's original words and completion criteria? Can the evidence layer locate the original passage? Does the assembly layer budget for both high-signal content and output space? Does the state layer have rules for writing, updating, and forgetting? Does the action layer have contracts for authorization, parameters, and idempotency? And does the feedback layer support continuing, backing off, or stopping based on Observation?

Only when all of these pass can you say the system is starting to embody context engineering. It still needs to be validated on real task sets for accuracy, cost, latency, and recovery capabilities. The checklist in the guide is a design map; production reliability comes from continuous evaluation, logging, and post-mortem reviews.

The final page frames prompt engineering and context engineering as two different scales: the former focuses on the instructions within the current window, while the latter also manages how documents, tools, memory files, history, and tool results dynamically enter that window. That contrast can be compressed into a single sentence: model capability determines what a system might be able to do, but the context system determines what it is drawing on right now, what it can accomplish, and what it leaves behind once the work is done.