Reranking for RAG: When a Reranker Is Worth It (and When It Isn't)

Your RAG bot keeps answering from a chunk that is close but not quite right. The vector search clearly found the neighborhood, because the correct passage is sitting at position seven of the results, but the model only read the top three. The advice you keep hearing is "add a reranker." A reranker re-sorts the chunks you already retrieved so the genuinely best one rises to the top, but it only ever sees what retrieval already found.

That one limit decides half of whether you should bother. A reranker is a precision fix, not a recall fix. If the right chunk never made your shortlist, a reranker has nothing to re-sort, and you have a retrieval problem to solve first. This guide gives you the decision, the real options and their cost, and the n8n wiring, including a bug worth knowing about.

What does a reranker actually do in a RAG pipeline?#

A reranker fixes ranking, not retrieval. Your vector search pulls a shortlist of roughly-right chunks but often ranks the best one low, so the model answers from a worse chunk. RAG reranking re-scores that shortlist with a sharper model and returns the genuinely best few, but it only ever sees the chunks retrieval already found.

Stage one recalls cheaply and wide; stage two re-sorts the shortlist with a slower, sharper model. The reranker never sees chunks stage one missed.

The production pattern is two stages: retrieve wide, then rerank narrow. Stage one is your bi-encoder vector search, which recalls a wide shortlist cheaply, say the top 20. Stage two is the reranker, which re-scores that shortlist and hands the best few, say the top 3, to the model. The first stage optimizes for recall, the second for precision.

Hold on to the limit, because it answers the "should I" question by itself: the reranker only sees the candidates retrieval returned. If your bot is still answering from the wrong chunk because the right one is not even in the top 20, reranking will not save you. That is why the first question below is always about recall, never about the reranker.

Why is a cross-encoder more accurate than vector search?#

Your vector search is a bi-encoder: it embeds the query and each document separately, then compares vectors, which is fast but never lets the two interact. A cross-encoder reranker reads the query and a candidate together through every layer, so it models their interaction directly and scores relevance far more accurately, at the cost of speed.

Bi-encoders make recall cheap; cross-encoders make the top of the list correct. You need both. Source: ZeroEntropy, Cohere.

That cost is why the cross-encoder cannot be your only stage. It must run once per candidate, so it is practical on a shortlist of tens, not across an index of millions. The bi-encoder makes recall cheap and wide; the cross-encoder makes the top of the list correct. You need both, in that order.

Put concretely: a bi-encoder has to commit to a single vector for your query before it has seen any document, so two passages that look similar in vector space can outrank the one that actually answers the question. The cross-encoder reads query and passage side by side and can tell "about the same topic" from "actually answers this," which is exactly the distinction that rescues a buried best chunk.

The reported quality lift is roughly +5 to +15 NDCG@10 across BEIR and MTEB-style benchmarks. Treat that as a published range on those corpora, not a promise for yours, because the gain depends entirely on your data and how badly your first stage ranks. The right embeddings still matter as much as the reranker, which is why comparing embedding models is worth doing alongside this.

What does reranking cost you in latency and money?#

Reranking adds one model call, commonly tens to a few hundred milliseconds depending on setup and candidate count. The money is usually a rounding error next to the LLM bill, and reranking can even cut total cost, because sending three clean chunks instead of ten noisy ones lowers the completion tokens you pay for.

Published latency figures vary: self-hosted on a GPU is often around 50 to 100 ms for the rescoring, while a hosted API on roughly 50 candidates is commonly cited at 100 to 400 ms p50. Measure your own, because candidate count and document length move it a lot. Here are the real options as of June 2026:

A worked example, modeled: at about $1 per 1,000 searches, reranking 10,000 user queries a month costs roughly $10 for the rerank step, assuming one search unit per query and documents under the auto-split threshold. That is typically far less than the completion cost of those same 10,000 answers. Verify the live numbers before relying on them, because rerank pricing shifted in 2026, with Cohere moving self-serve toward Model Vault tiers.

Latency is the real tax, not money. For a batch job or an internal tool, an extra 100 to 300 ms is invisible. For a live chat where the user watches the cursor blink, it is a delay you will feel, so rerank a smaller shortlist or self-host on a fast GPU when interactivity matters. Budget the latency the way you budget the tokens.

So, should you add a reranker?#

Add a reranker when precision is the problem: the right chunk is usually retrieved but ranked low, your queries are nuanced, raising k plateaus, and you pad the context with many chunks hoping one is right. Skip it when recall is the problem or your top three are already correct, because then it changes nothing.

Fix recall before reranking; add a reranker only when the right chunk is retrieved but ranked below junk.

Run this quick symptom check before you commit:

If two or more hit, reranking is likely worth it. If the right chunk isn't even retrieved, fix recall first, because a reranker can't.

If two or more of those hit, reranking is likely worth it. If the right chunk is not even being retrieved, that is a recall problem: fix your chunking, fix your embeddings, or add hybrid keyword-plus-vector search first, because a reranker cannot recover what retrieval missed. And if your top three are already correct, spend the effort on the prompt or on k instead.

A reranker is a precision correction layer, not a recall fix, and not free latency. Source: BEIR/MTEB ranges, vendor docs.

On hosting: a managed reranker such as Cohere or Voyage is the lowest-friction path and a trivial per-query cost, while the open-license bge-reranker-v2-m3 means running a GPU but pays zero per query and keeps your chunks in-house. Pick managed for speed of setup, self-host for data control or zero marginal cost.

How do you wire a reranker after retrieval in n8n?#

As of June 2026, n8n has a native Reranker Cohere node. Enable Rerank Results on a supported vector store such as PGVector, which adds a reranker leg, then attach the Cohere node with your model and credential. Set retrieval to return about 20 candidates and let the reranker narrow them.

The whole point is retrieve wide, rerank narrow: set the vector store to return a larger candidate set, around 20, and let the reranker pick the few you actually feed the model. Choosing the store matters here too, and pgvector vs Chroma vs Qdrant covers which expose the rerank toggle.

For Voyage, Jina, or a self-hosted endpoint, the Cohere node will not help, so call the rerank API from an HTTP Request node on your retrieved chunks, or use a community reranker node. Whichever path you take, verify it is doing something: run a handful of known-hard queries with reranking on and off. If the ranking and the answer do not change, you either have a recall problem or your top three were already right, and the reranker was never the fix.