Query Rewriting for RAG: Fix Retrieval at the Query Side

You have tuned chunk sizes, added a reranker, and swapped embedding models, but your RAG system still misses on short or conversational questions. The reason is often upstream of every document-side fix you tried. The user's raw query never matched your chunks in the first place, so no amount of document polishing was going to rescue that retrieval.

What is query rewriting in RAG and why does it matter?#

Query rewriting in a RAG pipeline is where you use query rewriting rag techniques to transform the raw user question into a better search query before retrieval. A model resolves pronouns, adds chat-history context, and expands vague wording. That upstream fix repairs retrieval failures that chunking, reranking, and embedding swaps on the document side simply cannot reach.

The core insight is a division of labor. Document-side fixes assume the query is already good and try to match it better. Query-side fixes assume the query is the weak link. When a user types "does it support that too?", no chunk in your index contains those pronouns, so retrieval returns noise. Rewrite it to "does the Pro plan support single sign-on?" and the same retriever suddenly lands the right passage. Our guide on why RAG chatbots give wrong answers shows how many failures trace to retrieval rather than the model itself.

The query gets cleaned and optionally fanned out before retrieval, then the ranked lists merge back into one.

How does a raw query differ from a rewritten one?#

A raw query is what the user actually typed, which is often short, context-dependent, or full of pronouns that only make sense inside the conversation. A rewritten query is a self-contained, keyword-rich version a model produced from that raw input plus recent chat history. The retriever sees only the query text, so the rewrite is what decides recall.

The raw query hides meaning in pronouns; the rewrite makes it self-contained so the retriever can match it.

The gap is easiest to feel with a conversational example. A user asks about a product, then follows up with "and how much is that on the annual plan?" The word "that" carries the entire meaning, and it is invisible to a vector search. The rewrite pulls the earlier product name forward and produces "annual plan price for the Standard tier", which now shares real vocabulary with your indexed pricing pages. Same retriever, same index, better input.

Which query-side techniques should you use?#

Four techniques sit at the query side, and they stack rather than compete. Query rewriting cleans a single query. Multi-query generates paraphrases and merges results. HyDE embeds a hypothetical answer instead of the question. Query expansion bolts on synonyms. Each buys recall at the price of at least one extra model call before retrieval even begins.

HyDE is worth understanding because it inverts the usual assumption. Instead of embedding the question, you ask a model to draft a plausible answer, embed that hypothetical document, and retrieve the real chunks nearest to it. The intuition, from Gao et al. 2022 (arXiv:2212.10496), is that an answer sits closer in embedding space to real answer passages than a bare question does. Multi-query, documented in the LangChain multi-query retriever docs, generates 3 to 5 paraphrases and unions their results, so a single awkward phrasing no longer sinks the whole retrieval.

Two or more of these and the query side is your highest-leverage fix.

How do you build a query-rewriting step end to end?#

You wire a rewrite stage between the user input and your retriever, then measure whether it actually lifts recall. Capture the raw query plus recent turns, run 1 model call to rewrite or expand, optionally fan out to multiple queries, retrieve for each, and merge the ranked lists with reciprocal rank fusion before they reach your reranker or generator.

Start from how your queries fail; each path ends at measure recall before you keep the step.

Reciprocal rank fusion is the merge step that makes multi-query practical. Each generated query returns its own ranked list, and RRF scores every document by summing 1 divided by (k + rank) across the lists, with k commonly set to 60. A chunk that appears high in several lists floats to the top, and you do not need calibrated scores from different retrievers to combine them. That property is why the same fusion trick powers hybrid search that blends BM25 and vector results.

The honest trade-off is latency and spend. Every technique here adds at least 1 model call before retrieval, and multi-query multiplies your retrieval count by N. That overhead earns its keep when queries are short, ambiguous, or conversational, and it is mostly wasted when users already type long, specific, keyword-rich questions. Start with plain rewriting, which is the cheapest at a single call, and only add multi-query or HyDE if your recall measurements still fall short.

When is query rewriting not worth the latency?#

Skip or trim the query side when your users already send long, specific, keyword-dense queries, because there is little left to rewrite and you only add latency. It is also a poor fit for hard real-time paths where a single extra round trip breaks your budget, or when a cheaper document-side fix clears the same bar.

The decision is really about where your recall is leaking. If a labeled evaluation set shows retrieval already returns the right chunks for typical questions, spend your effort on reranking or chunking instead. If retrieval whiffs on the short and conversational tail, the query side is the highest-leverage place to intervene. Pair this with the right index too, since your embedding model choice sets the ceiling that any rewrite has to work within.