RAG Evaluation: How to Measure If Your Retrieval Is Actually Good

You changed the chunk size, swapped the embedder, read three answers that looked fine, and shipped. Nothing told you whether retrieval actually got better or quietly got worse. A RAG evaluation puts a number on the two things your eye cannot judge: whether the right context was retrieved, and whether the answer faithfully used it.

The trap is that a fluent, confident answer built on the wrong context reads exactly like a good one. So this gives you a small, reproducible scorecard, four metrics you can run before and after every change, and the order to read them in when an answer goes wrong. If your scores stay low no matter how you tune retrieval, it may be worth revisiting long context vs RAG as an architecture choice.

Why isn't "it feels okay" a real RAG evaluation?#

Because eyeballing three answers cannot tell you whether retrieval or generation changed. A RAG system has two jobs: retrieve the right context, then answer faithfully from it. Each fails differently, and a fluent answer built on the wrong context looks fine. Rag evaluation metrics put a number on each half so you know what moved.

The two failure modes are genuinely different. Bad retrieval with good generation gives you a confident, well-written answer to the wrong context, and only a retrieval metric catches it. Good retrieval with bad generation gives you the right context that the model then ignored or contradicted, and only a generation metric catches that.

Always diagnose retrieval first: if the right context never arrived, no prompt fixes the answer.

That is why a single number is not enough. If you only track one averaged score and it drops, you still do not know whether to fix your chunking or your prompt. The whole point of a scorecard is to separate the halves so the number points you at the actual problem instead of just confirming there is one.

What are the four metrics in a usable RAG scorecard?#

Four metrics, split by half. Context precision and context recall score retrieval: was the right context retrieved, and ranked near the top? Faithfulness and answer relevancy score generation: did the answer stick to that context, and actually address the question? Each runs 0 to 1, and you read them separately, never as one average.

Here is the scorecard, with what each measures and whether it needs a labelled reference answer:

In plain terms: context precision is a precision-at-k averaged over your ranked chunks, so it rises when relevant chunks sit near the top. Context recall breaks the reference answer into claims and checks how many the retrieved context supports. Faithfulness does the same for the answer's claims against the context, catching hallucination. Answer relevancy has a model generate a few questions from the answer and measures how close they sit to your original question, penalizing padded or incomplete replies.

A purely illustrative example makes the shape concrete. If retrieval returns five chunks and only three are relevant, a precision-style score sits near 0.6; if the reference answer makes four claims and the retrieved context supports three of them, recall sits near 0.75; if the answer states four claims and one is unsupported by the context, faithfulness sits near 0.75. Those are made-up numbers to show the arithmetic, not measured results.

The mean of these four is sometimes reported as a single "ragas score." It is a fine headline and a poor diagnosis, because averaging a precision drop against a faithfulness gain hides both. Always read the four.

No test set yet? Score faithfulness and answer relevancy now; build references to unlock recall. Source: Ragas docs.

Which of these you can run today depends on labels, so this is the practical starting line. Add references when you want the recall and reference-precision numbers; everything else you can score with no labelled data at all.

Which metric tells you what's broken?#

Diagnose in order, retrieval before generation. Low context recall means the facts never arrived, so fix chunking or embeddings first. Low context precision means they ranked too low, so add reranking. Low faithfulness means the answer ignored the context, so ground the prompt. Low answer relevancy means it dodged the question.

Walk it top to bottom: recall, then precision, then faithfulness, then answer relevancy. The first low score is your fix.

The ordering rule is the single most useful habit in RAG eval: always check retrieval before generation. If recall is low, the needed facts were never in the context, so no amount of prompt tuning will fix the answer. Builders waste days rewriting prompts for what is really a retrieval problem the bot keeps hitting.

Map each metric to its fix. Low recall points at chunking, embeddings, or adding hybrid keyword-plus-vector search. Low precision points at reranking or retrieval tuning. Low faithfulness points at grounding or tightening the generation prompt. Low answer relevancy points at a generation prompt that is incomplete or unfocused. There is also an answer-correctness metric that compares the answer to a reference, but it needs labels and blends factual with semantic overlap, so treat it as an optional add-on, not part of the day-one four.

How do you run a reproducible RAG eval?#

Freeze a small test set and a judge model, then change one thing at a time. Build 20 to 50 real questions, capture the retrieved context and the answer for each, and score the four metrics with Ragas. Change one variable, re-run the same set with the same judge, and compare the deltas.

The test set is part of the result, so keep it frozen. For context recall and reference-based precision you also write a reference answer per question; the reference-free metrics need only the question, the retrieved context, and the answer. Choosing the right embedder matters here too, which is why comparing embedding models belongs in the same loop.

Freeze the test set and the judge model so the only thing that moves between runs is your change.

Be honest about what these scores are. They are LLM-as-judge metrics, so the judge model, its temperature and seed, and your test set all shape the number, and they are non-deterministic unless you pin those settings. That is exactly why you freeze the judge and the set: so the only thing that moves between two runs is the change you made.

These metrics are honest and reproducible only if you freeze the judge and the test set. Source: Ragas.

Then read the four separately and change exactly one thing, the chunk size, the embedder, a reranker, or the prompt, and re-run the same set with the same judge. A change that lifts precision but drops faithfulness is information you would never get from one averaged number, and it is the whole reason to keep a baseline.

How does RAG eval fit an n8n pipeline?#

Ragas is a Python library, not an n8n node, so you run it offline or in CI against an exported dataset. n8n's job is to emit the data: log the retrieved chunks and the final answer per query to a store. n8n's own Evaluation feature covers lighter pass-or-fail regression checks.

Concretely, capture the retrieved_contexts from the Vector Store node and the final answer from the LLM node, write them keyed by question to Postgres, a sheet, or a file, and that export becomes your Ragas dataset. n8n also ships an Evaluation trigger that runs a workflow over a dataset and attaches metrics, which is enough for lightweight regression checks or an LLM-judge step even when you compute the formal Ragas metrics in Python.

The split is worth keeping clean: n8n produces the evidence, Ragas produces the score. Trying to compute precision-at-k or claim-level faithfulness inside a workflow is fighting the tool, while logging the inputs an evaluator needs is a few extra fields on a node you already have. Keep the dataset versioned next to the workflow so a teammate can re-run last month's baseline against this month's change and see exactly which metric moved.

Close the loop the same way every time. Pick one metric to move, usually context recall or precision, make one change in the workflow, re-export, and re-score against the same frozen set. Measure every change against the baseline, not against memory, and you turn "it feels okay" into a number you can defend.