liveBook · Manning

Metrics-driven development

As explained in Retrieval Augmented Generation, The Foundational Ideas

Metrics-Driven Development is an iterative development approach that uses evaluation metric scores to guide system changes.

Metrics-Driven Development (MDD) turns RAG development from subjective demo testing into data-driven iteration: metric scores form the feedback loop for every system change. section 14.0

Use the evaluation triad to locate the failing part of the system:

  • Context relevance identifies missing content, missed top-ranked results, and information absent from context. section 14.0
  • Faithfulness identifies factual inconsistency and hallucination in the response. section 14.0
  • Answer relevance identifies information not extracted, incorrect specificity, and incomplete answers. section 14.0

Evaluation loop

When scores drop, use the triad and its failure-point mapping to decide whether to change retrieval, generation, or both, then evaluate the change again. section 14.0

How Retrieval Augmented Generation, The Foundational Ideas covers it

How deep it goes
Maps three evaluation dimensions to retrieval and generation failure points, but does not detail implementation beyond structured evaluation prompts.
Examples
Uses context relevance, faithfulness, and answer relevance to diagnose missing content, hallucination, extraction errors, specificity, and incompleteness.
What it assumes
Assumes a RAG system producing queries, contexts, and responses; deliberately avoids requiring human-annotated ground truth.
What you can do after
Apply it via RAGAS to evaluate query-context-response triples and select retrieval or generation changes from metric results.
How it connects
Connects RAG evaluation with retrieval, generation, hallucination, and failure-point diagnosis.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Retrieval Augmented Generation, The Foundational Ideas. The section links point to the parts of the book it draws on.