LLM Context Windows: Tokens, Limits, and Lost in the Middle

You pasted a 40-page contract into a model with a huge window, asked one question, and got a confident answer that quietly missed the clause buried in the middle. The window was big enough. The recall was not. That gap is the whole story, and it is why a larger context window does not automatically make a model smarter about your document.

What is an LLM context window?#

An llm context window is the token budget for one request, and input plus output share it. A token is roughly 4 characters of English text, so about 1,000 tokens is close to 750 words. When you fill the window with a long document, you leave less room for the answer, because both live in the same pool.

That token math is the part people skip. If a model advertises a 128,000-token window and you feed it 120,000 tokens of source text, you have left roughly 8,000 tokens for the reply. Ask for a long summary and you can hit the ceiling mid-sentence. The window is not "input space plus separate output space"; it is one shared budget you spend on both.

Planning estimates for plain English, not exact conversions. Rare words, URLs, and code split into more tokens.

Tokens are also not words. Common words often map to a single token, while a rare word, a long URL, or code can split into several. The 4-characters-per-token and 750-words-per-1,000-tokens figures are rules of thumb for plain English, not exact conversions, so treat them as planning estimates rather than a guarantee for your specific text.

How big are context windows in 2026?#

As of 2026, window sizes vary widely across providers, so treat any single number as perishable and verify it on the vendor's own page. Many mainstream models sit around 128,000 tokens, Anthropic's Claude family has offered roughly 200,000, and Google's Gemini has advertised 1,000,000 tokens or more for long-context work.

Those headline numbers move often and differ by model tier, region, and API version, so do not hardcode them into a workflow. A 1,000,000-token window sounds like it removes every limit, but it changes the economics more than it changes the behavior. You still pay for every token you send, and you still depend on whether the model can actually retrieve what you buried inside those tokens.

As of 2026, verify. Sizes are perishable ballparks in thousands of tokens; confirm the exact number on the vendor's own page.

The table below sketches what each rough size class is good for and what to watch. Read the sizes as 2026-era ballparks, not fixed specs, and confirm the exact figure for your chosen model before you design around it.

Why does a bigger context window not fix recall?#

A bigger window holds more text but does not read all of it equally. Liu et al. (2023), in "Lost in the Middle" (arXiv:2307.03172), showed that models retrieve facts best when the relevant text sits near the start or the end of the input, and noticeably worse when it lands in the middle of a long context.

That "U-shaped" pattern is the catch. Adding a 1,000,000-token window does not remove it; a larger window just gives you a longer middle to lose things in. If the one clause that answers your question is at position 300 of 500 pasted pages, the model can still miss it even though the text technically fit. Capacity and attention are not the same property.

From Liu et al. (2023), arXiv:2307.03172. A longer window just gives you a longer middle to lose facts in.

There is a second cost that is easy to ignore. You pay per input token, so filling a 128,000-token window on every call is expensive and slow, whether or not the model uses all of it. Paying to stuff the window and then losing the middle is the worst of both outcomes, which is exactly why placement and retrieval still matter.

How do you place context so the model uses it?#

Put the load-bearing content where recall is strongest: the start and the end. Lead with your key instructions and the most important facts, then close by restating the question or the exact output you want. Keep the noisy, low-value filler in the middle, where a weaker recall does the least damage to your answer.

Two habits follow from this. First, do not dump 200 documents in and hope; retrieve the 5 or 10 most relevant chunks and send only those, which cuts cost and shortens the middle. Second, order what you keep so the critical material bookends the prompt. If your retrieval is returning the wrong chunks in the first place, our guide to fixing a RAG chatbot that gives wrong answers covers the retrieval side.

Start from your real constraint; if precision or cost matters, retrieval usually beats simply buying a bigger window.

This is also why retrieval-augmented generation did not die when windows grew. A 1,000,000-token window lets you skip retrieval, but you then pay for every token and inherit the middle-recall problem at full scale. Retrieval keeps the prompt short, cheap, and front-loaded with only what matters, which plays directly to the model's strengths.

Does a bigger window or retrieval win for your task?#

Neither wins by default; the right choice depends on how much text you have and how precise the recall must be. A large window is convenient for one-off, whole-document questions. Retrieval is better when accuracy matters, cost matters, or the same corpus gets queried many times, because it keeps prompts short and front-loaded.

The honest summary is that a context window is a budget, not a comprehension guarantee. Bigger windows remove the "it did not fit" problem but not the "it did not notice" problem. Spend the first tokens and the last tokens on what matters, verify the current window size before you build, and reach for retrieval when precision or cost is on the line. When you are comparing model options and per-token prices, our cheapest AI API in 2026 guide applies the same dated-pricing discipline.