Context Engineering for AI Agents

In my previous article about few-shot techniques for text classification, I compared different ways to select examples for an LLM classifier. The problem looked narrow: we have an input, we have a set of possible labels, and we need to decide which examples to show to the model.

But that small problem is actually a useful entry point into a much larger one.

When we build agents, we still ask the same question, just with more moving parts:

What should the model see right now?

For a classifier, the answer may be: task description, label list, and 20 nearest examples. For an agent, the answer may include system instructions, user request, retrieved documents, tool schemas, MCP resources, previous tool outputs, memory, compressed history, intermediate state, and maybe a few examples as well.

This is what people now call context engineering. I do not think it is a magical new discipline. It is more like prompt engineering growing up and meeting software engineering. Prompt wording still matters, of course (however, it matters less and less with modern LLMs), but in many agent systems the harder part is not how to phrase the instruction. The harder part is deciding what information should be available to the model, when, and in which form.

In this article I will walk through how I think about context engineering for AI agents. We will start with few-shot prompting, then move to retrieval, MCP/tools, memory, compression, prefix caching, and evaluation.

Why context engineering matters

A simple LLM application can often get away with a good prompt. You write clear instructions, maybe add a couple of examples, call the model, parse the result, and move on.

Agents are different. They usually run for multiple steps. They call tools. They observe results. They may search files, inspect code, ask for clarification, update a plan, or hand off work to another agent. The input to the next model call is no longer just “the prompt”. It is a state assembled from many sources.

Anthropic describes context as the tokens included when sampling from an LLM, and frames context engineering as optimizing the usefulness of those tokens under model constraints. That definition is simple, but it is enough. The context window is finite. Even if it is large, it is not free, not equally attended to, and not always clean.

I have seen the same pattern many times:

1. Start with a small prompt.
2. Add one rule for an edge case.
3. Add examples.
4. Add output JSON.
5. Add tool instructions.
6. Add some retrieved docs.
7. Add conversation history.
8. Add another rule because the previous rule broke something.

At some point the prompt becomes a landfill. The model has all the information and still makes a bad decision because the important part is buried between stale instructions and irrelevant tool outputs.

So the goal is not to maximize context. The goal is to maximize useful context.

From few-shot prompting to context engineering

Few-shot prompting is already context engineering in a small form.

In the classification article, I compared random selection, nearest-neighbor selection, diversity-based selection, and a few other strategies. The interesting part was that the examples were not training data. They were runtime data. We changed the input to the model and changed the model behavior without updating model weights.

That is exactly the context engineering mindset.

For few-shot classification, the main question is:

Which examples should I include?

For an agent, the question becomes:

Which instructions, examples, documents, tools, memories, and previous observations should I include for this step?

Here is the rough mapping:

The bigger context surface makes everything more powerful and more fragile. With the right retrieved examples, an agent can classify a strange request correctly. With the wrong stale memory, it can confidently do the wrong thing.

This is why I like the phrase “context engineering” more than I expected. It pushes us to treat context as a designed object, not as a giant string we keep appending to.

What counts as context

When people say “context”, they often mean documents retrieved by RAG. That is only one part of it.

In an agent system, context can include:

A useful test is: if changing it can change the next model output, it is context.

This includes tool schemas. A tool description like search(query: str) is context. A more specific tool such as search_project_docs(query: str, section: Literal["api", "billing", "auth"]) is also context, but with a better contract. The model sees it and learns what actions are possible.

The Model Context Protocol makes this even more explicit. MCP is an open standard for connecting AI applications to external systems. From the model’s point of view, MCP tools and resources expand the available context surface: files, APIs, databases, tickets, browser state, and other systems can become available through a standard interface.

That is useful, but it also means context engineering includes tool engineering. Bad tools create bad context.

A useful mental model: write, select, compress, isolate

The LangChain context engineering article groups strategies into four operations: write, select, compress, and isolate. I find this taxonomy useful because it maps nicely to real engineering decisions.

Write

Some information should be saved outside the current context window.

Examples:

• a task state object;
• a note that the user prefers concise answers;
• a label correction from a human reviewer;
• a summary of a tool result;
• a decision made earlier in a workflow.

Writing context does not mean showing it to the model forever. It means storing it somewhere so it can be retrieved later if it becomes relevant.

Select

This is the heart of context engineering.

Selection decides what enters the model call. In few-shot classification, selection may mean nearest-neighbor examples. In a code agent, it may mean the three files most related to a bug. In a support agent, it may mean the current ticket, matching docs, previous messages, and the right refund policy.

Bad selection creates two opposite problems:

• missing context: the model does not see the thing it needs;
• context pollution: the model sees too much irrelevant or stale information.

Both can produce confident nonsense.

Compress

Compression keeps useful signal while reducing tokens.

This can be a summary of conversation history, extracted facts from a long API response, or a compact representation of previous agent steps.

But compression is not free. A summary is a lossy transformation. It may remove exactly the detail that mattered later. Worse, it may slightly rephrase a fact and turn it into something false. I would treat compaction summaries as useful but suspicious, especially for tasks where exact wording matters: legal text, code, financial data, medical instructions, and so on.

Isolate

Sometimes the best context is a separate context.

A research subagent can read many documents and return a short answer to the main agent. A planner can work in one context while an executor works in another. A tool-heavy step can happen in a temporary context and return only the extracted facts.

This prevents one messy context from poisoning the entire task. It also makes traces easier to read.

Just-in-time retrieval

The simplest bad idea in context engineering is to preload everything.

It is tempting. We have larger context windows now, so why not put all docs, all examples, all previous messages, all tool descriptions, and all project files into every request?

Because the model still has to process it. You pay for it. Latency increases. The important tokens compete with irrelevant tokens. And if the model sees two almost-relevant facts, one current and one stale, you may not like which one it chooses.

A better default is just-in-time retrieval:

1. Keep a small stable prefix: role, task contract, output schema, and maybe a few canonical rules.
2. Retrieve examples and documents for the current input.
3. Add only tool schemas that are useful for this step.
4. Include recent tool outputs only after extracting what matters.
5. Log what was selected so you can debug it later.

This is the same lesson as KNN few-shot selection. Nearest examples worked well because they were relevant to the current input. They were also more expensive and worse for prefix caching because the prompt changed per request. That tradeoff does not disappear in agents. It becomes the main design problem.

MCP and tools are context too

MCP is often explained as “USB-C for AI tools”. I think that analogy is fine for a quick explanation, but it hides the part that matters for context engineering.

An MCP server does not just give the agent magical abilities. It exposes a set of tools and resources. The client decides what to make available. The model sees descriptions, names, schemas, and sometimes resource metadata. All of that becomes part of the context the model uses to decide what to do.

Anthropic introduced MCP as a standard for connecting AI assistants to systems where data lives. This is useful exactly because agents need context from outside the chat: repositories, databases, project management tools, internal docs, logs, and so on.

But tool context can fail in boring ways:

For example, I would rather expose this:

from enum import StrEnum
from pydantic import BaseModel, Field


class TicketStatus(StrEnum):
    OPEN = "open"
    CLOSED = "closed"
    ANY = "any"


class SearchTicketsInput(BaseModel):
    query: str = Field(description="Semantic search query over customer tickets")
    status: TicketStatus = Field(default=TicketStatus.OPEN)
    limit: int = Field(default=5, ge=1, le=20)


class TicketSearchResult(BaseModel):
    ticket_id: str
    title: str
    status: TicketStatus
    short_summary: str
    updated_at: str

than a generic search_everything tool returning raw documents. A schema is not only validation. It is also instruction.

Memory and compaction

Memory sounds simple until you have to decide what should be remembered.

There are at least three different things people call memory:

1. Conversation history: what happened in this thread.
2. Task state: what the agent has learned or decided while solving the current task.
3. Long-term memory: facts or preferences saved across tasks.

These should not be treated the same way.

Conversation history is often too verbose. Task state should be explicit and structured. Long-term memory should be selective, because stale memories are dangerous. If a user once said “use GPT-4o for this project” and later migrated to a different stack, the agent should not keep resurrecting the old preference forever.

The Anthropic cookbook on memory, compaction, and tool clearing is useful here because it separates several strategies that are often mixed together:

• memory: save durable facts;
• compaction: summarize a long context;
• tool clearing: remove bulky tool results after extracting useful information.

I like that separation. Compaction is not memory. Clearing a tool result is not forgetting. These are different operations with different risks.

Here is a simple pattern I would use:

from datetime import datetime, timedelta
from pydantic import BaseModel, Field
from typing import Literal


class MemoryItem(BaseModel):
    kind: Literal["user_preference", "project_fact", "decision", "correction"]
    text: str
    source: str
    confidence: float = Field(ge=0.0, le=1.0)
    created_at: datetime
    expires_after_days: int | None = None

    @property
    def is_expired(self) -> bool:
        if self.expires_after_days is None:
            return False
        return datetime.now() >= self.created_at + timedelta(days=self.expires_after_days)


class CompactedState(BaseModel):
    task_goal: str
    decisions: list[str] = Field(default_factory=list)
    open_questions: list[str] = Field(default_factory=list)
    facts_needed_later: list[str] = Field(default_factory=list)
    discarded_tool_outputs: list[str] = Field(default_factory=list)

Notice the source, confidence, and optional expiration. Without those, memory becomes a pile of old text with authority it does not deserve.

Prefix caching and stable context

Prefix caching is one of those details that looks like an inference optimization but quickly becomes a context design constraint.

OpenAI prompt caching reuses exact prompt prefixes to reduce latency and cost. Claude prompt caching supports cache breakpoints for repeated prompt prefixes (fortunately, OpenAI introduced caching breakpoints as well starting with GPT-5.6). In open-source inference, vLLM automatic prefix caching describes the same core idea at the KV-cache level: if two requests share the same prefix, the system can reuse work.

This matters for context engineering because dynamic context can destroy cache reuse.

Suppose every request starts like this:

system prompt
all tool descriptions
all label definitions
all canonical examples

and only then adds the user input plus retrieved examples. That stable prefix can be cached.

But if you insert per-request retrieved examples near the top, the prefix changes early and the cache becomes much less useful.

So there is a practical layout question:

This is one of the reasons I do not like treating prompts as unstructured strings. With an explicit context builder, you can decide which blocks should be stable, which should be dynamic, and where the cache boundary should be.

A small context-building pipeline in Python

Let’s make the idea concrete. I will use a small paper/request classification assistant, because it connects nicely to the few-shot article.

The agent receives a short text and returns a structured classification. It can retrieve examples, retrieve label definitions, use memories, and expose a small set of tools.

First, define the data structures:

from enum import StrEnum
from pydantic import BaseModel, Field


class ScientificArea(StrEnum):
    SOFTWARE_ENGINEERING = "software engineering"
    MACHINE_LEARNING = "machine learning"
    BIOCHEMISTRY = "biochemistry"
    PSYCHOLOGY = "psychology"
    HYDRAULICS = "hydraulics"


class FewShotExample(BaseModel):
    input_text: str
    label: ScientificArea
    reason: str | None = None


class RetrievedDocument(BaseModel):
    doc_id: str
    title: str
    text: str
    score: float


class AgentContext(BaseModel):
    task: str
    output_schema: str
    examples: list[FewShotExample]
    documents: list[RetrievedDocument]
    memories: list[MemoryItem]
    available_tools: list[str]
    compacted_state: CompactedState | None = None

Then build retrieval as a separate step:

class RetrievedContext(BaseModel):
    examples: list[FewShotExample]
    documents: list[RetrievedDocument]
    memories: list[MemoryItem]


example_store = ... # some semantic search over labeled examples
doc_store = ... # some semantic search over reference documents
memory_store = ... # some per-user memory storage with semantic search


def retrieve_context(task_text: str, user_id: str) -> RetrievedContext:
    examples = example_store.search(task_text, k=5)
    documents = doc_store.search(
        task_text,
        filters={"type": "label_definition"},
        k=4,
    )
    memories = memory_store.search(user_id=user_id, query=task_text, k=3)

    return RetrievedContext(
        examples=examples,
        documents=documents,
        memories=memories,
    )

And assemble the model input deliberately:

def build_context(task_text: str, user_id: str) -> AgentContext:
    retrieved = retrieve_context(task_text=task_text, user_id=user_id)

    return AgentContext(
        task=task_text,
        output_schema='{"label": string, "confidence": number, "notes": string}',
        examples=retrieved.examples,
        documents=retrieved.documents,
        memories=[
            m for m in retrieved.memories
            if m.confidence >= 0.7 and not m.is_expired
        ],
        available_tools=["search_label_docs", "ask_human_reviewer"],
        compacted_state=None,
    )

Finally, render it into messages. I prefer keeping this rendering step boring and testable:

def render_messages(context: AgentContext) -> list[dict[str, str]]:
    system = """
You classify technical text into one scientific area.
Return only JSON that matches the provided schema.
If the evidence is weak, lower confidence and explain what is missing.
""".strip()

    context_block = context.model_dump_json(indent=2, exclude={"task"})

    return [
        {"role": "system", "content": system},
        {"role": "user", "content": f"Context:\n{context_block}\n\nClassify:\n{context.task}"},
    ]

This is not meant to be a universal framework. The point is simpler: context is built, not appended. Once it is structured, you can test it, log it, cache the stable parts, and compare strategies.

How to evaluate context engineering

A context strategy that sounds nice may still make the agent worse.

I would log at least this for every run:

from pydantic import BaseModel


class ContextTrace(BaseModel):
    run_id: str
    selected_example_ids: list[str]
    selected_doc_ids: list[str]
    selected_memory_ids: list[str]
    available_tools: list[str]
    input_tokens: int
    cached_input_tokens: int = 0
    output_tokens: int
    latency_ms: int
    final_label: str | None = None
    human_review_required: bool = False
    task_success: bool | None = None

This trace lets you ask useful questions:

• Did the retrieved examples actually overlap with successful runs?
• Did adding memory reduce human review, or did it introduce stale assumptions?
• Did compaction lower cost without hurting task success?
• Did dynamic retrieval improve accuracy enough to justify worse prefix-cache reuse?
• Which tool descriptions are visible when the model chooses the wrong tool?

LangChain’s State of Agent Engineering reports that observability and tracing are already common among teams building agents. That makes sense. Without traces, context engineering becomes vibes. You change a prompt, add retrieval, remove memory, and hope.

For classification-like tasks, you can use accuracy and macro F1, as I did in the few-shot article. For agents, I would also track:

The important part is to compare versions. Do not just add a memory layer because it sounds agentic. Build a small eval set, run with and without memory, and see what happens.

Practical checklist

When designing context for an agent, I would go through this checklist:

• What information must always be present?
• What can be retrieved only when needed?
• Which examples should be selected dynamically?
• Which tools should be visible for this step?
• Are tool schemas specific enough?
• Which tool outputs should be summarized or cleared?
• What should be stored as memory, and when does it expire?
• What should be compressed, and what must stay exact?
• Which blocks should remain stable for prefix caching?
• What will be logged in the trace?
• Which metric proves the context change helped?

This sounds a bit mechanical, but I think that is the point. Context engineering becomes useful when it stops being a vague phrase and turns into a set of design decisions you can inspect.

Conclusion

Few-shot prompting taught us that examples matter. Not in some abstract way, but in a very practical one: change the examples, and the model changes its behavior.

Context engineering extends the same idea to agents.

The examples are still there, but now they sit next to retrieved documents, tool definitions, MCP resources, memory, compressed state, tool outputs, and traces. Some of this context should be stable. Some should be fetched just in time. Some should be written to memory and retrieved later. Some should be aggressively removed before it pollutes the next step.

The question I keep coming back to is simple:

What should the model see right now?

If you can answer that deliberately, log the answer, and measure whether it helped, you are doing context engineering. The rest is implementation detail. Important implementation detail, sure. But still detail.

Sources

• Effective context engineering for AI agents - Anthropic
• Context Engineering - LangChain
• What is the Model Context Protocol? - MCP documentation
• Introducing the Model Context Protocol - Anthropic
• Prompt caching - OpenAI API docs
• Prompt caching - Claude docs
• Automatic Prefix Caching - vLLM
• Context engineering: memory, compaction, and tool clearing - Anthropic Cookbook
• State of Agent Engineering - LangChain