liveBook · Manning

Intent-aware metrics

As explained in Retrieval Augmented Generation, The Foundational Ideas

An intent-aware metric evaluates retrieval by rewarding coverage of distinct facets or subtopics of an information need.

An intent-aware metric evaluates retrieval by rewarding coverage of distinct facets or subtopics of an information need, not merely repeated relevance. section 8.6

Use diversity-sensitive measures when a precision-only score can hide incomplete retrieval:

  • Recall@k measures how many possible relevant items appear in the top k. section 8.6
  • nDCG rewards relevant results appearing earlier and can reward multiple distinct relevant documents. section 8.6
  • α-nDCG explicitly rewards documents that cover different query facets or subtopics. section 8.6

Warm-jacket retrieval

A result containing three warm jackets may achieve higher Precision@3, yet a list containing warm, fleece, and other relevant jacket types can be more useful because it covers more of the information need. section 8.6

How Retrieval Augmented Generation, The Foundational Ideas covers it

How deep it goes
Defines facet-aware metrics and distinguishes Recall@k, nDCG, and α-nDCG, while deferring full RAG evaluation methodology.
Examples
Compares three warm-jacket results with a more varied list containing fleece and other relevant jacket types.
What you can do after
Evaluate retrieval by choosing metrics that expose missing facets hidden by precision-only scores.
How it connects
Connects retrieval evaluation with Recall@k, nDCG, precision, diversity, and query facets.

This explainer was generated by Manning's knowledge graph from the text of Retrieval Augmented Generation, The Foundational Ideas. The section links point to the parts of the book it draws on.