Meaningful RAG evaluation must inspect retrieval, generation, and the interaction between them. A pipeline can fail because the relevant document was never retrieved, because generation went wrong, or because the two stages interact poorly. Checking only the final answer confirms that something failed but cannot locate the cause. Instrument all three layers.
What a RAG evaluation setup needs to cover
A complete RAG evaluation pipeline checks three separate points:
- Retriever quality: Given a query, did the retriever place relevant documents in the top-k results?
- Answer faithfulness: Given those retrieved documents, is the generated answer grounded in them?
- Answer relevance: Does the final answer address the user's actual question?
These are independent failure modes with different causes and remedies. A high-precision retriever paired with a generator that hallucinates fails faithfulness. A faithful generator paired with a low-recall retriever fails relevance because it answers what the available documents support rather than what the user asked. Combining the signals into one aggregate score can mislead you and direct investigation to the wrong component.
Evaluating retriever quality: recall@k and MRR
Retriever testing needs query and ground-truth document pairs. Creating them requires manual effort, but this set supports everything that follows. Historical material is usually the strongest source: resolved support tickets, verified Q&A pairs, and documentation with known answers.
- Recall@k: Recall@k asks whether at least one relevant document appears among the top-k results. If the generator uses the top-5 chunks, use recall@5. Low recall@5 means generation receives incorrect context or no relevant context. Prompt engineering cannot repair a retrieval gap.
- Mean Reciprocal Rank (MRR): MRR measures the position of the first relevant document. An MRR of 1.0 means the most relevant document is always first. An MRR below 0.5 means the relevant document is consistently third or lower. This is especially important for generators that give greater weight to earlier context.
- Context precision: Context precision complements recall by asking what fraction of the k retrieved chunks are relevant. High recall with low precision floods generation with noise. Some models handle that well; others become confused when most of the context window does not relate to the question.
To create ground-truth pairs, collect 200 to 400 queries with known-good answers. For each query, manually mark the document chunks in the corpus containing the information needed for the answer. This part of RAG evaluation cannot be fully automated. The resulting set remains useful whenever you change chunking, the embedding model, or retrieval parameters.
Answer faithfulness: checking that responses stay grounded
Faithfulness measures whether every generated claim can be traced to retrieved context. A score near 1.0 indicates that generation stayed within the retrieved information. A score below 0.80 indicates that a meaningful portion of the response contains unsupported claims.
Two practical approaches are available:
- NLI-based scoring: Break the response into atomic factual claims, then use a natural language inference model to check each claim against the context documents. The RAGAS library implements this pattern with reasonable defaults. It scales well and needs no LLM calls during scoring, but its results depend on the quality of claim decomposition.
- LLM judge: Give a capable language model the response and retrieved context, asking it to identify unsupported claims. This handles complex reasoning chains more flexibly, but costs more and depends on the wording of the judge prompt.
Chunking strategy can distort faithfulness scores. If a key sentence crosses a chunk boundary and only one half is retrieved, a judge may label a grounded claim unsupported because the complete evidence is missing. Before treating the issue as a generator problem, confirm that the relevant content is present in the retrieved chunks. Without response-level instrumentation, retrieval gaps can look like generation failures.
Answer relevance: did the question get answered?
Answer relevance asks whether the response addresses the user's actual question, regardless of grounding. A response may be fully faithful to retrieved context and still miss the point when the retriever returns related rather than directly relevant documents. Generation can faithfully combine those documents into an answer that does not address the original question.
A useful reference-free method generates multiple questions that the response could plausibly answer, then compares those synthetic questions semantically with the original question. High similarity indicates alignment with the original intent. Used in frameworks such as RAGAS, this method needs no reference answers, making it practical for production evaluation where answers cannot be maintained for every possible query.
Another option is an LLM judge given only the question and response, asked whether the response directly addresses the request. This is simpler, but depends on the judge's understanding of what a direct answer looks like in the domain. In highly specialized domains, the NLI-free synthetic-question method tends to be more reliable.
Wiring evaluators into CI
A one-time evaluation is only a snapshot. The important task is finding regressions as the pipeline changes, including re-indexing documents, changing chunking parameters, replacing embedding models, or updating prompts. Run the evaluation suite automatically for those changes.
Organize the suite into three independent evaluators with one combined report. Each evaluator should return a score and failing examples. Set a threshold for each metric. Reasonable starting points are recall@5 above 0.80, faithfulness above 0.85, and relevance above 0.78. Calibrate these values against actual data rather than treating them as ground truth. Any evaluator below threshold for a change should require review before merge.
Keeping CI runs fast
A full run over 400 query-document pairs with LLM judge scoring may take 15 to 20 minutes, which is too slow for every commit. Two practical measures help:
- Maintain a smoke-test subset of 50 examples covering the most common failure modes. It should exercise all three evaluation dimensions and finish in under 2 minutes. Run it for every pull request.
- Run the full suite nightly or when a change directly affects the retriever.
Cache embedding computations and NLI scores for the test set. Query-document relevance scores remain unchanged when the document index is unchanged, so recompute them only after document changes or an embedding-model update. Every commit needs generation-side faithfulness and relevance evaluation against fixed context, not a complete end-to-end retrieval rebuild. This provides continuous coverage without making CI unusably slow.