Evaluating a RAG Pipeline: Recall@k, MRR, Faithfulness

Most teams “evaluate” their RAG pipeline by asking it a handful of questions, reading the answers, and deciding whether it feels right. That works exactly until it doesn’t — until a chunking change, an embedding model swap, or a prompt tweak silently makes retrieval worse while the two or three questions everyone always tests still happen to come back fine. Without a real eval set, every change to the pipeline is a bet you can’t actually measure, which means regressions ship quietly and improvements can’t be proven, only asserted.

The deeper problem with eyeballing answers is that it conflates two failure modes that need completely different fixes. A RAG pipeline can produce a wrong answer because retrieval failed (the right chunk was never found) or because generation failed (the right chunk was found, but the model didn’t use it correctly, or used it and still hallucinated). Reading the final answer alone can’t tell you which one happened — you need metrics that isolate each stage. This post covers building an eval set from your own corpus and measuring both halves separately: recall@k and MRR for retrieval, faithfulness and answer relevance for generation.

You’ll learn:

• Why reading final answers can’t tell you whether a RAG failure is a retrieval problem or a generation problem
• How to build a real eval set from your own corpus instead of relying on generic benchmark questions
• How to compute recall@k and MRR for the retrieval stage, with runnable Python
• How to measure faithfulness — whether the generated answer is actually supported by the retrieved context
• Where RAGAS and similar frameworks fit, and what they automate versus what still needs human judgment
• How to turn these metrics into a regression gate you actually run on every pipeline change

Table of Contents

1. Why “It Looks Right” Isn’t an Evaluation Strategy
2. Separating Retrieval Failures from Generation Failures
3. Building an Eval Set from Your Own Corpus
4. Retrieval Metrics: Recall@k and MRR
5. Context Precision: Penalizing Noisy Retrieval
6. A Worked Example: Five Queries Through the Harness
7. Generation Metrics: Faithfulness and Answer Relevance
8. Where RAGAS Fits, and What Still Needs a Human
9. Turning Metrics into a Regression Gate
10. Common Pitfalls
11. Wrapping Up

Why “It Looks Right” Isn’t an Evaluation Strategy

The core problem with manual spot-checking isn’t that it’s slow — it’s that it’s non-representative and non-repeatable. A handful of questions someone types in during a demo tend to be the easy cases: well-phrased, close to the document’s own wording, about the most obvious content in the corpus. Production traffic looks nothing like that — it’s oddly phrased, about edge-case content, and asks things no one thought to test manually. A pipeline that looks great on five demo questions can have real recall gaps on the queries actual users send.

The second problem is that manual review doesn’t compose across changes. If someone tweaks the chunking strategy this month, and the retrieval prompt next month, and swaps the embedding model the month after, there’s no way to know which change helped, which hurt, and which canceled another one out — unless the same fixed eval set was run before and after each change, with numbers to compare, not impressions.

Separating Retrieval Failures from Generation Failures

A RAG pipeline’s final answer is the product of two stages that fail independently, and the fix for each is completely different:

• Retrieval failure: the chunk containing the answer was never surfaced to the LLM at all. No prompt engineering or better model fixes this — the answer literally isn’t in the context window. This is a chunking, indexing, or embedding problem, covered in depth in why RAG retrieves the wrong chunks.
• Generation failure: the correct chunk was retrieved and present in context, but the model still produced a wrong or unsupported answer — it misread the context, over-generalized past what the context actually said, or ignored the retrieved content in favor of its own prior knowledge.

The only way to tell these apart systematically is to check, for every eval query, whether the gold-answer content was actually present in what got retrieved before judging whether the final answer is correct. A wrong answer with the right chunk retrieved is a generation bug; a wrong answer with the right chunk missing is a retrieval bug, and they should never be lumped into one “pipeline accuracy” number, because fixing one type of problem does nothing for the other.

Building an Eval Set from Your Own Corpus

Generic benchmark datasets (Natural Questions, HotpotQA, and similar) measure something real, but not the thing you need to know: how your pipeline performs on your documents, in your domain’s phrasing, against your users’ actual question patterns. An eval set built from your own corpus is the only one that answers the question you actually care about.

Sourcing questions. The two most reliable sources are real user queries pulled from logs (if you have production traffic already) and hand-written questions built by someone who knows the corpus, deliberately covering a spread of phrasing styles — some close to the document’s own wording, some paraphrased, some using synonyms the document never uses.

Recording the gold answer. For each question, record two things: the specific passage or chunk ID that contains the answer (for retrieval scoring), and a reference answer or the key facts it must contain (for generation scoring).

eval_set = [
    {
        "query": "How long does a webhook retry before giving up?",
        "gold_chunk_ids": ["doc_112#chunk_4"],
        "gold_answer": "Webhooks retry with exponential backoff for 24 hours before being marked failed.",
    },
    # ...
]

Covering failure-prone categories deliberately, not just easy cases. Include questions where the answer spans two chunks, questions phrased with synonyms rather than the document’s exact terms, and questions about content near section boundaries — these are the categories most likely to expose real chunking and retrieval weaknesses, and they’re exactly the categories a quick manual spot-check tends to skip.

A eval set in the 50-150 question range, built once and extended over time as new failure categories surface in production, is enough to catch most regressions — it doesn’t need thousands of examples to be useful, it needs to be representative and it needs to be run consistently.

Retrieval Metrics: Recall@k and MRR

Recall@k answers: across the top k retrieved chunks, was the gold chunk present at all?

def recall_at_k(retrieved_ids: list[str], gold_ids: set[str], k: int = 5) -> int:
    return int(bool(set(retrieved_ids[:k]) & gold_ids))

def mean_recall_at_k(results: list[dict], k: int = 5) -> float:
    hits = [recall_at_k(r["retrieved_ids"], set(r["gold_chunk_ids"]), k) for r in results]
    return sum(hits) / len(hits)

Mean Reciprocal Rank (MRR) answers a sharper question: not just whether the gold chunk was retrieved, but how highly it ranked. It’s the average, across all queries, of 1 / rank of the first correct chunk (0 if it never appears):

def reciprocal_rank(retrieved_ids: list[str], gold_ids: set[str]) -> float:
    for rank, doc_id in enumerate(retrieved_ids, start=1):
        if doc_id in gold_ids:
            return 1.0 / rank
    return 0.0

def mean_reciprocal_rank(results: list[dict]) -> float:
    scores = [reciprocal_rank(r["retrieved_ids"], set(r["gold_chunk_ids"])) for r in results]
    return sum(scores) / len(scores)

The two metrics catch different regressions. Recall@k can stay flat while MRR quietly drops — the correct chunk is still being retrieved, just sliding from rank 1 to rank 4, which won’t break anything if you always pass the top 5 to the LLM, but will start to matter the moment you tighten k or add a reranker with a smaller top-n cutoff. Tracking both is how you’d notice a chunking strategy change that keeps recall stable but degrades ranking quality — a change that a recall-only dashboard would report as a non-event.

Context Precision: Penalizing Noisy Retrieval

Recall@k and MRR both only ask whether the gold chunk showed up and how high it ranked — neither penalizes a retrieved set that’s technically correct but padded with irrelevant chunks. Context precision closes that gap: of the k chunks retrieved, what fraction are actually relevant to the query?

def context_precision_at_k(retrieved_ids: list[str], relevant_ids: set[str], k: int = 5) -> float:
    top_k = retrieved_ids[:k]
    hits = sum(1 for doc_id in top_k if doc_id in relevant_ids)
    return hits / k

This matters because a low-precision retrieval set has a real downstream cost even when recall is perfect: every irrelevant chunk in the context window is something the LLM has to correctly ignore, which is exactly the kind of noise that increases the odds of a generation failure even though the retrieval stage “technically” found the right chunk. A pipeline with recall@5 of 100% but context precision of 0.2 is retrieving the right chunk buried under four irrelevant ones — worth flagging separately from a pipeline with the same recall but precision of 0.8, even though a recall-only dashboard would show them as identical.

A Worked Example: Five Queries Through the Harness

To make the numbers concrete, here’s a small illustrative trace — five queries from a toy eval set run through the recall@5, MRR, and context-precision functions above, on a hypothetical retrieval pipeline:

Averaging: recall@5 = 0.80, MRR = 0.57, mean context precision@5 = 0.72. The gap between recall@5 (0.80) and MRR (0.57) is the signal worth reading closely here — three of the four successful retrievals didn’t rank the gold chunk first, meaning a smaller k (say, top-3 passed to generation instead of top-5) would start dropping queries that currently succeed. That’s a concrete, measured reason to keep k at 5 rather than tightening it purely to save context-window tokens, and it’s exactly the kind of tradeoff that’s invisible if you only ever look at recall.

Generation Metrics: Faithfulness and Answer Relevance

Retrieval metrics tell you whether the right material reached the model. Faithfulness tells you whether the model actually used it correctly — whether every claim in the generated answer is actually supported by the retrieved context, as opposed to the model filling gaps with outside knowledge or outright hallucinating.

A practical way to compute faithfulness without a specialized framework is LLM-as-judge: decompose the generated answer into individual claims, then ask a judge model whether each claim is supported by the retrieved context alone.

FAITHFULNESS_PROMPT = """Context:
{context}

Claim: {claim}

Is this claim directly supported by the context above? Answer only "yes" or "no".
"""

def faithfulness_score(claims: list[str], context: str, judge_fn) -> float:
    supported = [judge_fn(FAITHFULNESS_PROMPT.format(context=context, claim=c)) for c in claims]
    return sum(1 for s in supported if s.strip().lower() == "yes") / len(claims)

Answer relevance is a separate check in the other direction: does the generated answer actually address what was asked, independent of whether it’s faithful to the context? An answer can be perfectly faithful to the retrieved context and still fail to actually answer the question — for instance, if it faithfully summarizes a tangentially related passage instead of the one that answers the specific question. A simple version: ask a judge model to generate a plausible question the answer would be responding to, then measure its embedding similarity against the original query — low similarity suggests the answer drifted from what was actually asked.

Faithfulness and relevance catch different generation bugs: low faithfulness with high relevance means the model is answering the right question but padding with information the context didn’t actually support; high faithfulness with low relevance means the model is accurately reporting something in the context that isn’t actually what was asked.

Where RAGAS Fits, and What Still Needs a Human

RAGAS (and similar evaluation frameworks) automate the LLM-as-judge pattern above into ready-made metrics — faithfulness, answer relevance, context precision, context recall — without needing to hand-write the judge prompts yourself. That’s genuinely useful for getting a metrics pipeline running quickly, and it’s a reasonable starting point rather than something to build from scratch on day one.

What it doesn’t remove is the need for a real, hand-built eval set specific to your corpus — RAGAS scores whatever queries and gold data you feed it; it doesn’t generate a meaningful eval set out of thin air, and a generic or synthetic eval set produces numbers that are precise but not meaningful for your actual failure modes. It also doesn’t remove the need for periodic human review of the judge itself — LLM-as-judge scoring has its own failure modes (a judge model can be lenient, or can misjudge subtle unsupported claims), so spot-checking a sample of the judge’s faithfulness verdicts against actual human judgment on a recurring basis is what keeps the automated numbers trustworthy rather than just precise-looking.

It’s also worth being deliberate about which judge model you use. Using the same model for both generation and judging risks a subtle self-preference bias, where the judge is more forgiving of outputs that match its own phrasing habits. A separate, independently-chosen judge model — or at minimum, awareness of this bias when interpreting borderline scores — keeps the faithfulness numbers closer to an honest signal than a self-congratulatory one.

Turning Metrics into a Regression Gate

The eval set only pays for itself if it’s run consistently, not once. A minimal setup that catches regressions before they ship:

def run_eval(pipeline, eval_set: list[dict]) -> dict:
    retrieval_results = []
    for item in eval_set:
        retrieved_ids = pipeline.retrieve(item["query"])
        retrieval_results.append({"retrieved_ids": retrieved_ids, "gold_chunk_ids": item["gold_chunk_ids"]})

    return {
        "recall_at_5": mean_recall_at_k(retrieval_results, k=5),
        "mrr": mean_reciprocal_rank(retrieval_results),
        # faithfulness/relevance computed similarly over generated answers
    }

baseline = run_eval(pipeline, eval_set)
# after a chunking, embedding, or prompt change:
candidate = run_eval(pipeline_v2, eval_set)

assert candidate["recall_at_5"] >= baseline["recall_at_5"] - 0.02, "recall regression"

Wiring this into CI, or at minimum running it manually before merging any retrieval-affecting change, converts “does this change help?” from a guess into a number. A two-point recall tolerance (rather than requiring strict non-regression) accounts for eval-set noise on a modest-sized query set — pick a threshold that matches your eval set’s size, since a 50-query set has more measurement noise than a 500-query one.

Common Pitfalls

Building the eval set from documents the pipeline will never see in production. An eval set drawn from a stale export or a sample corpus that doesn’t match what’s actually indexed will report numbers that look fine and mean nothing — always build gold data against the live, current index.

Letting the eval set go stale as the corpus grows. New document categories and new query patterns show up in production long after the eval set was first written. Treat it as a living artifact — add a handful of cases every time a real production failure surfaces a category the eval set didn’t cover.

Trusting a single faithfulness score without spot-checking the judge. LLM-as-judge scoring is a proxy, not ground truth. Periodically sample 10-20 judged cases and have a human check whether the judge’s verdict actually matches what a careful reader would conclude — a judge that’s silently too lenient will mask real generation regressions.

Conflating recall and context precision into one “retrieval score.” A pipeline can have perfect recall and terrible precision, or the reverse, and averaging them into a single number hides which one actually needs fixing. Report them separately, always.

Only measuring after something breaks. The entire value of an eval harness is catching a regression before it ships, not diagnosing one after users complain. Wire it into the same review step as the code change that touches retrieval or generation, not as a postmortem tool.

Wrapping Up

An eval set is what turns RAG pipeline changes from bets into measurements. The specific numbers — recall@5, MRR, faithfulness — matter less than the discipline of separating retrieval quality from generation quality and measuring both on questions built from your own corpus, not a handful of easy demo queries or someone else’s benchmark dataset. Once that harness exists, every other post in this series — chunking, hybrid search, reranking — becomes something you can actually validate instead of something you have to trust.