A RAG system can post a respectable end-to-end score on your eval set and still be broken in a way that score cannot see. The pipeline has two independent stages -- a retriever that selects context and a generator that writes an answer from that context -- and a single pass/fail number averages over both. When the score drops, that average tells you nothing about which stage to open first: whether the retriever handed the generator the wrong documents, or handed it the right documents and the generator wrote the wrong answer anyway. Debugging the wrong stage wastes a cycle and, worse, can leave the actual defect live in production.
This is an exercise lesson, and everything below is buildable: a small labelled corpus of cricket facts, a retriever you can inspect, the four standard retrieval metrics -- recall@k, precision@k, MRR, nDCG -- run over it, a groundedness checker that verifies generated claims against the context that was actually retrieved, and a harness that combines both into a single diagnosis per query. Every number quoted in the prose below is copied from a script's printed output, not estimated -- rerun any of the four code blocks and you will get the same figures.
The organizing idea is a four-way split: good retrieval with a good answer, good retrieval with a bad answer, bad retrieval with a bad answer, and one quadrant that looks harmless and is not -- bad retrieval with a right answer. That last case means the model answered from what it memorized during pretraining rather than from the documents it was given, and it will keep looking correct on your eval set right up until the underlying fact changes and nothing in the pipeline notices.