How to Build a RAG System That Retrieves the Right Thing

Most RAG systems fail in the same place, and it is not the model. Retrieval is the part that breaks, and almost every "the answers are wrong" complaint traces back to the wrong chunks arriving rather than the model mishandling the right ones.

The symptom is a demo that answers beautifully on the five documents you tested with, then starts confidently citing the wrong paragraph the week you load the real corpus. Nothing errored. The model did its job on the material it was given.

What you will end up with#

A pipeline that ingests documents, retrieves the right passages, answers with citations, and a way to tell whether retrieval or generation is at fault when it does not. Twelve lessons, in the order the decisions actually bite.

What this does not cover: training your own embedding model, and multi-modal retrieval over images or audio.

Each stage caps the one after it. Retrieval quality is the ceiling on answer quality.

One framing worth carrying through the whole guide: RAG is a search problem with a language model bolted on the end, not a language model problem with search bolted on the front. Every lesson below follows from that. When people describe RAG as "giving the model your documents" they skip the part where finding the right documents is the entire difficulty.

Lesson 1Decide whether you need retrieval at all#

Retrieval is machinery. Before you build it, check that the cheaper answers do not work, because two of them often do.

1. Does it all fit in the context window? If your whole corpus is a few hundred pages and stable, putting it in the prompt is simpler, more accurate, and needs no infrastructure.
2. Is it actually one document? Answering questions about a single contract is a prompting problem, not a retrieval problem.
3. Does the answer need to be current? If the corpus never changes, a fine-tuned or cached approach may beat retrieval on both cost and latency.

Retrieval earns its complexity when the corpus is too large for context, changes often enough that baking it in is wrong, and gets queried across many different documents.

That last row is underrated. Even where long context would work, retrieval gives you a defensible answer to "where did that come from", which matters more than accuracy in some settings.

The cost and accuracy comparison against a large window is in Long Context vs RAG.

Lesson 2Chunk for the question, not for the document#

Chunking is the decision that determines your ceiling, and it is made before anything else exists, which is why so many systems inherit a bad one.

The failure it causes: a chunk that contains half the answer retrieves well and answers badly, because the model is asked to complete a thought it was only shown the beginning of.

Bad  : split every 500 characters, ignore structure
Better: split on headings, keep each section whole
Best  : split on headings, and prepend the document title and section
        path to every chunk so it carries its own context

That third form is the one that fixes most retrieval problems people blame on the embedding model. A chunk reading "the limit is 30 days" is unretrievable; the same chunk prefixed with "Refund Policy > Consumer Returns" is findable and answerable.

Overlap deserves a word, because it is the setting people tune first and it matters least. A small overlap between adjacent chunks protects against an answer landing exactly on a boundary. A large overlap inflates your index, your storage and your query cost while returning near-duplicate passages that crowd out genuinely different ones. Small and boring is right.

The prefix is what makes a chunk both findable and answerable on its own.

1. Split on structure first: headings, sections, natural boundaries.
2. Prepend the path so each chunk stands alone.
3. Only then worry about size, and overlap a little.

The strategies compared with their failure modes are in RAG Chunking Strategies, and the prefixing technique is in Contextual Retrieval for RAG.

Lesson 3Pick an embedding model and stop#

Embedding choice matters far less than chunking, and it is where people spend most of their evaluation time because it is the part with a leaderboard.

Pick a mainstream model, note its dimension count, and move on. The decision you actually need to make is dimensions, because that is what you pay for forever:

Two other properties matter more than the leaderboard position. Input length limits, because a model that truncates at 512 tokens will silently cut your carefully built chunks in half. And whether it is a hosted API or a local model, because that decides whether re-embedding your whole corpus is a bill or an afternoon of compute.

Higher dimensions cost you on every vector, forever, in storage and in query time. The quality gain flattens well before the cost does.

One more thing that is cheap and often skipped: normalise the text before embedding. Strip boilerplate headers and footers, collapse whitespace, and drop navigation chrome from scraped pages. Every one of those tokens dilutes the vector, and a chunk that is 40% cookie banner embeds as a cookie banner.

The one rule that is not negotiable: the same model must embed both your documents and your queries. Mixing them produces retrieval that looks like it works and is quietly random.

The cost and quality trade-off is in Embedding Dimensions and RAG Cost, and a direct comparison is in n8n RAG Embeddings Comparison.

Lesson 4Choose a vector store on operations, not benchmarks#

Every vector store will handle your first hundred thousand chunks. Choose on how you want to operate it.

CREATE EXTENSION vector;
CREATE INDEX ON chunks USING hnsw (embedding vector_cosine_ops);

Choose on who operates it, not on benchmark throughput you will not reach.

One operational note that decides this more often than performance does: whoever answers the pager at 2am should be the same person who chose the store. A dedicated vector database is a reasonable choice for a team and a poor one for a solo operator who now has two databases to back up, patch and monitor instead of one.

Index type is worth one paragraph rather than a project. HNSW builds slowly, queries fast, and uses more memory. IVFFlat builds fast, queries a little slower, and needs its lists tuned to the corpus size. For anything under a few million vectors the difference is not what is limiting you, and choosing between them before you have measured recall is optimising the wrong layer.

The comparison with real numbers is in pgvector vs Chroma vs Qdrant, and the managed-versus-self question is in n8n Pinecone vs Supabase Vector Store.

Index type is a second, smaller decision, and the trade is build time against query speed. The detail is in HNSW vs IVFFlat.

Lesson 5Retrieve more than you need, then narrow#

The single most effective change to a mediocre RAG system is to stop treating the first result as the answer.

Pull ten to twenty candidates, then narrow them deliberately. Vector similarity is good at finding the neighbourhood and unreliable at ranking within it, so the top result is frequently not the best one available.

1. Retrieve 10-20 candidates by vector similarity.
2. Re-rank them with a model that reads the query and the passage together.
3. Pass the top 3-5 to the generation step.

The first vector result is a candidate, not an answer.

Re-ranking is the step people skip because it adds latency and a second model call. It is also usually the difference between "mostly right" and "right", because a cross-encoder reading the query against each passage is doing a fundamentally better job than cosine distance.

What it costs, concretely: one extra model call per query against 10-20 short passages, which is a cheap-tier job rather than a flagship one. Latency goes up by roughly the time of one small call, and the quality gain is largest exactly where the corpus is dense with near-duplicates, which is where vector similarity is weakest.

The measured effect and when it stops being worth it are in RAG Reranking: When It Is Worth It.

Lesson 6Add keyword search alongside vectors#

Pure vector search has a specific, repeatable blind spot: exact strings. Product codes, error numbers, names, version identifiers. Semantic similarity has no special respect for "ERR_4021", and will happily return passages about similar-sounding errors.

Hybrid search fixes it by running both and merging:

query -> vector search  -> candidates A
      -> keyword (BM25) -> candidates B
      -> merge and dedupe -> re-rank -> top 5

If your corpus contains identifiers, part numbers, or proper nouns that users type verbatim, this is not an optimisation, it is a correctness fix.

How to tell whether you need it without guessing: take fifty real user queries and count how many contain a token a human would type verbatim and expect an exact match. Error codes, SKUs, invoice numbers, person names, file names. If that count is above about one in ten, hybrid is not optional.

The build is in n8n Hybrid Search: BM25 and Vector.

Lesson 7Filter on metadata before you search#

Most retrieval failures at scale are not ranking failures, they are scope failures: the right answer existed but so did four thousand near-identical passages from other customers, other years, or other product lines.

Attach metadata at ingest and filter on it at query time:

1. Tag every chunk with source, date, owner, and document type at ingest.
2. Filter the candidate set BEFORE the vector search where the platform allows it.
3. Make the filter explicit in the query path, so a missing filter is visible rather than silent.

A worked example of the scope failure, because it is abstract until you see it. A support bot over four years of tickets is asked about a shipping delay. Vector search returns twenty passages about shipping delays, nineteen of them from other customers in other years, all genuinely similar. The right one exists and ranks eleventh. No amount of re-ranking fixes that reliably; filtering to this customer and the last ninety days does, instantly.

Point 2 matters for both cost and quality. Filtering after retrieval means you searched the whole corpus and threw most of it away, which is slower and less accurate than searching the right subset.

Adding a field later is cheap. Adding it to two million existing chunks is not.

Decide the metadata schema at ingest, because retrofitting it means re-processing the corpus. Source, document type, owner or tenant, effective date, and a version or hash are enough for almost every filter you will later wish you had. Adding a field later is cheap; adding it to two million existing chunks is not.

The implementation is in RAG Metadata Filtering in n8n.

Lesson 8Rewrite the query before you search it#

Users do not write queries that embed well. They write "what about the other one" and "does that apply to us too", which carry almost no retrievable signal on their own.

Rewrite before retrieving:

1. Resolve pronouns and references using the conversation so far.
2. Expand the query into a standalone question that makes sense with no context.
3. Where the question is broad, split it into two or three narrower searches and merge the results.

User      : "and what about refunds?"
Rewritten : "What is the refund policy for annual enterprise subscriptions?"

That rewrite costs one cheap model call and routinely fixes more retrieval failures than any amount of index tuning. It is the highest return per line of code in the whole pipeline.

One caution: rewrite for retrieval, then answer the ORIGINAL question. If you let the rewritten question reach the generation step, you answer the machine's interpretation rather than what the user asked, and the mismatch is hard to spot because the answer is coherent.

The technique in full is in Query Rewriting for RAG.

Aside 8.5The follow-up question problem#

Multi-turn is where most RAG chatbots quietly degrade, and the reason is subtle. The first question retrieves well because it is specific. The third question is a fragment that only makes sense against the previous two, and if you embed it as written, you retrieve noise.

turn 1  "what is the refund window for enterprise plans?"
        -> embeds well, retrieves the policy section

turn 2  "does that include annual?"
        -> embeds as almost nothing. Retrieves noise.
        -> rewritten: "does the enterprise refund window include annual plans?"
        -> retrieves the same policy section, correctly

turn 3  "and for the other one?"
        -> unrewritten, this is unretrievable in any index

Worse, if you retrieve nothing useful and still pass the conversation history to the model, it will answer from the earlier context and sound confident. The failure is invisible: the answer is fluent, cites nothing, and may be wrong.

The fix is the rewrite above, applied on every turn, not just the first. Then check: if retrieval returns nothing above your similarity floor, say so rather than answering anyway. That is lesson 9.

Lesson 9Make it refuse#

An answer with no supporting passage is the most dangerous output a RAG system produces, because it is indistinguishable from a good one at a glance.

Give the model an honest exit and make refusal a first-class outcome:

Answer ONLY from the passages provided.
If they do not contain the answer, reply exactly:
  { "answer": null, "reason": "not in the provided sources" }
Never answer from prior knowledge.

Then enforce it structurally rather than trusting the instruction:

1. Set a similarity floor. Below it, do not call the model at all.
2. Require the answer to cite at least one retrieved chunk id.
3. Reject and retry any response that cites nothing.

Setting the floor is empirical, not theoretical. Take twenty questions the corpus genuinely cannot answer, record the top similarity score each one returns, and set the floor just above the highest. That number is corpus-specific and there is no universal value worth copying.

unanswerable questions -> top scores: 0.41 0.38 0.44 0.39 ... 0.46
answerable questions   -> top scores: 0.71 0.68 0.82 0.75 ... 0.66
floor -> somewhere around 0.55, then re-check monthly

Point 1 is the one that actually works. An instruction is a request; not calling the model is a guarantee.

There is a tempting shortcut here that does not work: asking the model to rate its own confidence and refusing below a threshold. Self-reported confidence correlates poorly with correctness, and it fails hardest exactly where you need it, on plausible-sounding wrong answers. The retrieval score is a measurement of something real; the model's confidence is another generation.

The failure mode in detail is in n8n RAG Chatbot Wrong Answers and n8n AI Agent Hallucinations.

One more refusal case that is easy to miss: the corpus contains a contradiction. Two documents, both retrieved, both relevant, saying different things because one superseded the other. The model will usually pick one and answer confidently. Surfacing both with their dates and letting the reader see the conflict is more useful than an arbitrary choice, and it is a one-line prompt change once you are already returning source dates.

Lesson 10Cite the source, in the answer#

Citations are not a presentation feature. They are the only mechanism that lets a reader verify an answer without re-doing the work, and the only one that lets you debug retrieval after the fact.

1. Return chunk ids with every retrieved passage.
2. Require the model to attach ids to each claim.
3. Render them as links back to the source document and position.
4. Log the ids with the answer, so a complaint is diagnosable weeks later.

Step 4 is the one that pays off. When someone says the bot was wrong, the logged ids tell you immediately whether retrieval failed or generation did, and those are entirely different repairs.

There is a presentation detail worth getting right. A citation that links to a 90-page PDF is technically a citation and practically useless. Link to the position: the page, the anchor, the section. The difference between "here is the document" and "here is the paragraph" is the difference between a citation a reader checks and one they ignore.

The implementation is in AI Agent Citations and Source Grounding.

Lesson 11Measure retrieval separately from generation#

"The answers are bad" is not a diagnosis. RAG has two failure surfaces and they need separate measurements, because the repairs share nothing.

Measure the two halves with different numbers:

Build the question set the same way as any eval: from real queries, with the correct passage identified by hand, including several questions the corpus cannot answer. Thirty is enough. The unanswerable ones are what let you measure refusal, which no purely retrieval-focused metric captures.

1. Retrieval: for a set of questions with known answers, was the correct passage in the top k? That is recall, and it caps everything downstream.
2. Generation: given the correct passage, did the model answer correctly and cite it?

If recall is 60%, no amount of prompt work fixes the system. If recall is 95% and answers are still wrong, stop touching the index.

Work on generation while recall is low and the accuracy number barely moves.

A worked read of the two numbers, because the pair is what tells you where to work:

Row four is the common starting state and the reason people thrash: work on generation while recall is 60% and the accuracy number barely moves, which reads as the model being weak.

The metric definitions and how to build the set are in RAG Evaluation Metrics.

One thing worth measuring that neither number captures: how often the system refuses when it should have answered. Over-refusal is a real failure and an invisible one, because nobody complains about a bot that says it does not know. Track the refusal rate, and if it climbs, your similarity floor is probably too high rather than your corpus too thin.

Lesson 12Keep the index fresh#

A RAG system is only as current as its last ingest, and the failure mode is the worst kind: confidently correct answers drawn from a document that was superseded last month.

1. Re-index on change, not on a schedule, wherever you can detect the change.
2. Where you cannot, schedule it and record the last-ingested timestamp per source.
3. Delete removed documents. An orphaned chunk answers questions from a document nobody can open.
4. Surface the source date in the answer, so a reader can judge staleness themselves.

Freshness has a cost shape worth knowing before you choose a cadence. Re-embedding an entire corpus is a full re-ingest bill every time; embedding only what changed is a small one. That is the whole argument for incremental updates, and it is why step 1 above is ordered first.

Deletions are the step people skip, and orphaned chunks answer from documents nobody can open.

Point 3 causes the strangest bugs in RAG. Deleting a file from the source system does not delete its vectors, so the system keeps citing a document that no longer exists.

The incremental approach is in RAG Index Freshness and Incremental Updates.

What I would do differently#

Fix the chunking before touching the embedding model. The model is the more technical-feeling problem and it has a leaderboard, which is exactly why it attracts the effort and exactly why it is the wrong place to start. Chunking sets the ceiling; the embedding model moves the last few points underneath it.

The rule worth adopting: if retrieval recall is below about 80%, do not touch the model, the index type, or the dimensions. Look at what is in the chunks.

The second thing is the refusal path, which gets skipped because the demo never needs one. "Answers confidently from nothing" is not a failure that shows up in testing, where every question asked is one the corpus can answer. It is the first thing a real user finds, and by then it has been answering that way for weeks.

Third, and this one costs the most time later: log the retrieved chunk ids alongside every answer. Without them, a complaint arriving weeks after the fact cannot be traced to retrieval or to generation, so every investigation starts from nothing. Four lines of logging turn each one into a two-minute lookup.

One note on sequencing. Lessons 5 through 8 are all improvements to retrieval and they compound awkwardly: each helps less once the others are in place, so measuring them individually overstates the first and understates the last. Add them in the order given, re-measure recall after each, and stop when the number stops moving. That order runs cheapest and most reliable first.

What I am still unsure about is re-ranking. It clearly improves quality and it clearly costs latency and a second model call, and I have not found a principled way to set the threshold beyond measuring recall with and without it on your own corpus. I default to including it for anything user-facing and skipping it for background jobs, which is a heuristic, not a rule.

Four checks that catch the failures testing never surfaces.

One last thing on sequencing. Everything in lessons 5 through 8 is an improvement to retrieval, and they compound in an awkward way: each one helps less once the others are in place, so measuring them individually overstates the first and understates the last. Add them in the order given, re-measure recall after each, and stop when the number stops moving. That order is not arbitrary, it runs cheapest and most reliable first.

Where to go next#

If nothing is built yet, lesson 1 is the honest starting point, and the most common outcome is discovering the context window would have done.

If it is built and the answers are wrong, go to lesson 11 and separate the two measurements before changing anything. If retrieval is the problem, RAG Chunking Strategies is where most of the recovery is. If you want the agent to decide when and what to retrieve rather than following a fixed pipeline, Agentic RAG vs Classic RAG covers what that buys and costs. And for a private end-to-end build, Fully Local RAG in n8n.