liveBook · Manning

Component-wise evaluation

As explained in Retrieval Augmented Generation, The Foundational Ideas

Component-wise evaluation maps separate evaluation metrics to the system components responsible for the measured behavior.

Component-wise evaluation assigns separate metrics to the RAG components responsible for observed behavior, turning vague demo impressions into targeted diagnosis. section 14.0

The evaluation triad separates three questions:

  • Faithfulness asks whether the answer is grounded in the retrieved context and primarily evaluates generation. section 14.0
  • Answer relevance asks whether the response addresses the query and evaluates generation. section 14.0
  • Context relevance asks whether retrieval returned focused, useful context and evaluates the retriever. section 14.0

From score to repair

A low context-relevance score directs attention to missing content, ranking failures, or absent context; a low faithfulness score points toward factual inconsistency in generation. Scores therefore guide retrieval or generation changes instead of merely labeling the whole system as good or bad. section 14.0

How Retrieval Augmented Generation, The Foundational Ideas covers it

How deep it goes
Explains a three-metric decomposition, component mapping, reference-free judging, and failure-point diagnosis.
Examples
Uses the faithfulness, answer-relevance, and context-relevance triad with mapped retrieval and generation failure points.
What you can do after
Diagnose RAG weaknesses from query, context, and response scores without human-annotated ground truth.
How it connects
Connects retrieval, generation, faithfulness, answer relevance, context relevance, hallucination, and RAGAS.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Retrieval Augmented Generation, The Foundational Ideas. The section links point to the parts of the book it draws on.