Long-Context vs RAG: When a 200K-1M Token Window Beats Chunking

"Long context killed RAG" is a great headline and a bad architecture decision. So is "RAG is always cheaper." Now that 1M-token windows ship at flat per-token pricing, you can put a whole corpus in one prompt, but you pay for every token on every call. The real choice between long context and RAG is a decision boundary set by four things: corpus size, how often it changes, how often you query it, and whether you need exact citations.

This post makes that boundary concrete with reproducible cost math and cited recall research. Every dollar figure is either a published rate or a clearly labelled modeled estimate, and every recall claim carries a source.

What does "long context vs RAG" actually mean?#

Long context puts the entire corpus in the model's window every time you ask a question, so the model sees everything and you pay for everything. RAG indexes the corpus once, retrieves only the most relevant chunks per query, and answers from that small set. One trades money for completeness; the other trades completeness for a much smaller per-query bill.

The framing that "the big window made retrieval obsolete" misses how the costs scale. A 1M-token window is a capacity, not a discount: a 900K-token request bills at the same per-token rate as a 9K-token one, so stuffing the window is the most expensive way to answer a question. RAG moves far fewer tokens, but it can miss a span, split a clause across chunks, or retrieve the wrong context entirely.

Neither wins outright; the column you pick depends on corpus size, change rate, and query volume.

The rest of this post returns to four variables: corpus size (does it even fit?), update frequency (does it change between queries?), query volume (do you ask the same corpus many times?), and recall needs (do you need exact source citations?). Hold those four in mind and the decision stops being a slogan.

What does the cost math actually look like?#

Input cost per query is just (tokens in / 1,000,000) × input price. That one formula, run on real rates, settles most of the debate. The table below pairs each model's window with its per-token prices so you can plug in your own corpus size. Check the vendor pages before you commit, since rates change.

Now three modeled paths for the same 200K-token document on Sonnet 4.6, input only, so the strategy is isolated. Sending the full doc with no cache costs 0.2M × $3 = $0.60 per query, every query. Sending it with prompt caching costs one 1.25× = $3.75/MTok write up front, then 0.1× = $0.30/MTok reads, so each cached query is about 0.2M × $0.30 = $0.06, a 90% drop on the repeated portion. RAG that retrieves about 8K relevant tokens costs 0.008M × $3 = $0.024 per query plus a one-time embedding cost.

Modeled on Sonnet 4.6 input pricing; output tokens excluded so the input strategy is isolated.

Watch for a second cost wrinkle the headline rate hides: the biggest windows often cost more per token once you actually fill them. Gemini 2.5 Pro charges about $1.25/MTok up to 200K context but doubles to $2.50/MTok above it, and GPT-5.5 applies a session surcharge (2x input, 1.5x output) once a prompt crosses 272K tokens. So the move that long context is supposed to make easy, dumping everything in, is also where the per-token price quietly steps up. RAG, by sending a few thousand tokens, never reaches those tiers.

The crossover is the whole story. Caching collapses long context's cost disadvantage when you query the same fixed corpus repeatedly, because the prompt caching read is a tenth of standard input. RAG still moves the fewest tokens, so it usually keeps a raw input-cost edge, but it gives up some cross-document reasoning to do it. One honesty note: these are input-token estimates. Output tokens, embedding and vector-database hosting, and retrieval-quality misses are real costs left out of the headline numbers on purpose.

Where does long context quietly fail?#

A bigger window is not the same as perfect use of it. Two well-documented effects mean stuffing the context can lower answer quality even when every relevant token is technically present, so "just send everything" is not a free win. Long context trades retrieval bugs for attention and latency bugs.

The cons are why RAG did not die; the pros are why people keep reaching for the window.

The first effect is "lost in the middle": models use information at the start and end of a long prompt more reliably than information buried in the middle. The second is sharper. A 2025 study, "Context Length Alone Hurts LLM Performance Despite Perfect Retrieval," found that simply increasing context length degrades performance even when the needed information is perfectly retrieved and present. Vendors do report near-perfect synthetic needle-in-a-haystack recall, but finding one planted sentence is not the same as reasoning over a long, noisy, real corpus.

Latency is the other tax. Huge inputs take longer to process and cost more per call, which is a second reason caching matters: it cuts both the bill and the wait on the repeated portion. The one failure long context never has is RAG's chunk-boundary problem, where a clause split across two chunks gets retrieved incompletely. That clean, complete view of the document is the honest case for the window. For the retrieval side of that trade, why your RAG chatbot gives wrong answers covers the failure modes chunking introduces.

Four questions route the choice: fits, changes, queried often, needs citations.

So which one do you actually pick?#

Walk the four variables in order and the answer falls out. If the corpus is bigger than the window, use RAG; you have no choice. If it fits and changes between most queries, use RAG anyway, because re-sending or re-caching a moving target wastes money. If it fits and stays stable, the question becomes volume.

If you can only check three things before choosing, check these.

For a stable corpus you query often, long context plus prompt caching is hard to beat: you pay the cache write once and read it back at a tenth of the price on every later query, with full cross-document reasoning and no retrieval pipeline to maintain. For a one-shot or rare analysis of a document that fits, use long context with no cache, since there is nothing to amortize. And when you need span-level citations or the data changes constantly, reach for RAG, or a hybrid.

The hybrid pattern is the 2026 default for serious systems: use RAG to shrink a large corpus to a strong candidate set, then put that set in a long-context call to reason across it. You get retrieval's token economy and the window's complete view, and for queries that also need exact-term matches, a hybrid keyword-plus-vector retriever sharpens the candidate set before the long-context step.

What should you measure before you pick?#

Measure three numbers before you write a line of pipeline code: your corpus token count, your daily query volume, and your update cadence. A corpus that fits the window, stays stable, and gets queried hundreds of times a day points straight at long context with caching. One that is large, fast-moving, or citation-critical points at RAG.

Then run the cost formula on your own rates instead of trusting a slogan. Model the full-document path, the cached path, and the RAG path with your real token counts, and compare them against the recall you actually need. The architecture that wins is the one that matches your four variables, and it will change as your corpus and the prices do, so treat the decision as something you revisit, not a flag you set once.