Contextual Retrieval: Fix RAG Chunks That Lose Context

Your RAG pipeline splits a document into chunks, embeds each one, and stores it. Then a chunk like "revenue grew 3% quarter over quarter" gets retrieved, and the model has no idea which company or which quarter it means. The chunk lost its home. Contextual Retrieval puts that home back before you ever embed.

What is contextual retrieval and why do plain chunks fail?#

Contextual retrieval fixes the core defect of plain RAG: a chunk embedded on its own has lost the document context that made it meaningful. Before embedding, an LLM writes a 50 to 100 token blurb explaining what the chunk is and where it sits, and you glue that blurb onto the chunk. Retrieval then matches on the whole meaning.

The failure is easy to reproduce. Split a 40 page filing into 800 chunks and one reads "the segment declined 12% year over year." Standalone, that sentence answers almost no real query, because the embedding has nothing to anchor "segment" or the fiscal period. A user asking about a named business unit in a named year never matches it. The blurb, something like "This is from Acme's Q2 2026 filing, discussing the cloud segment," restores the anchors so both vector search and keyword search can find it.

One LLM call per chunk at ingest writes the context; you prepend it, then embed and keyword-index the enriched chunk.

Two forces make this worse than it sounds. Smaller chunks improve precision but strip more context, so the tighter you chunk for relevance, the more orphaned each chunk becomes. And embeddings compress meaning into a fixed vector, so a chunk with no anchors simply lands in the wrong neighborhood of the vector space. Contextual retrieval attacks both by putting the anchors back into the text before either index sees it, rather than hoping a bigger chunk or a bigger model papers over the gap.

How does the contextual ingest pipeline actually work?#

The pipeline adds one LLM call per chunk at ingest time. For each chunk you send the model the full document plus that chunk, and ask for a short context that situates it. You prepend the returned 50 to 100 tokens to the chunk, then feed the enriched chunk to both your embedding model and your keyword index.

Anthropic calls the two halves Contextual Embeddings and Contextual BM25. The first improves semantic (vector) search: the prepended context shifts the embedding toward the chunk's real meaning. The second improves lexical search: BM25 is a keyword ranking function that rewards exact term overlap, so the added company names, dates, and entities give it far more to match on. You run both, fuse the two result lists (rank fusion), and pass the top candidates forward. The two signals catch different misses, which is why running them together beats either alone. If you have not set up lexical plus vector fusion yet, our walkthrough of hybrid search with BM25 and vectors covers that base layer.

The plain chunk lost its anchors; the contextual chunk restores the company, period, and section so both search signals can match.

The generated context does not need to be clever prose. Anthropic's own prompt asks the model for a short, plain description of what the chunk is and how it relates to the rest of the document, in a sentence or two. You are not summarizing the chunk or rewriting it; you are annotating where it lives. The chunk's original text stays intact underneath, so nothing is lost, and the context rides in front of it as extra retrieval surface.

What did Anthropic actually report, and does it hold up?#

Anthropic reported concrete reductions, and you should treat them as their benchmark, dated September 2024, not a universal guarantee. In their tests, contextual embeddings alone cut failed retrievals by about 35%. Adding contextual BM25 reached roughly 49%. Layering a reranking model on top brought it to about 67%.

Read that table as a shape, not a promise for your corpus. The gains are largest when chunks are context-poor on their own, which is common in dense reports, contracts, and long technical manuals. Numbers on your data depend on chunk size, embedding model, and how ambiguous your chunks really are.

Five ingest stages. The context call is one-time per document, and caching the document keeps it cheap.

How do you build it in n8n or any pipeline?#

Build it as five ingest stages, and cache the document so the per-chunk context call stays cheap. Chunk the document. For each chunk, call an LLM with the whole document to generate its context. Prepend that context. Embed the enriched chunk and index it for BM25. At query time, retrieve with hybrid search and optionally rerank.

The one real cost is the extra LLM call per chunk at ingest. It is a one-time cost per document, not a per-query cost, and prompt caching makes it tolerable. If you cache the full document once, every chunk's context call reuses those cached input tokens instead of re-reading the whole document 800 times. That is the difference between a workable ingest bill and an absurd one; our guide to prompt caching to cut LLM cost shows the mechanics. For a corpus under a few hundred chunks that you rarely re-index, the setup effort may outweigh the payoff, and plain hybrid search is a fine starting point.

Start from whether your chunks read fine alone; every path ends at a sensible retrieval setup for your corpus size.

A concrete cost sketch helps. Say a document is 40,000 tokens and splits into 200 chunks. Without caching, generating one context per chunk re-sends those 40,000 document tokens 200 times, roughly 8 million input tokens for a single document. With the document cached once, each of the 200 calls reads the cached copy at a fraction of the input price and only pays full rate on the small chunk and the short output. That is why Anthropic pairs the method with prompt caching in the same post: the technique is only practical because the expensive part is cacheable.

When is contextual retrieval worth the ingest cost?#

Reach for it when chunks are context-poor and the corpus is large enough that better recall pays back the one-time ingest cost. Long filings, contracts, wikis, and manuals benefit most, because their sentences lean on document-wide context. Skip it for tiny corpora or self-contained chunks, where plain hybrid search plus a reranker suffices.

If your retrieval is already returning the wrong passages, contextual retrieval is one lever among several. Diagnosing whether the miss is a chunking, retrieval, or prompting problem comes first; our post on why RAG chatbots give wrong answers walks that triage before you invest in a heavier ingest step.