liveBook · Manning

Alpha-ndcg

As explained in Retrieval Augmented Generation, The Foundational Ideas

α-nDCG is an intent-aware ranking metric that rewards relevant coverage of multiple query facets while reducing credit for redundancy.

A retriever can return three relevant-looking results and still miss part of the user’s intent. For a “warm winter jacket” query, three similar warm-jacket results may score well on Precision@3, while a set that also includes a fleece jacket may be more useful despite that item’s marginal relevance. α-nDCG is an intent-aware ranking metric for evaluating coverage of different query facets rather than relevance alone. section 8.6

Ordinary NDCG rewards relevant results appearing earlier in a ranking. α-nDCG adds an explicit concern for whether results cover different subtopics, so repeated coverage does not substitute for breadth. That makes it a candidate when evaluating diverse results from RAG-Fusion. Precision@k remains useful for counting relevant results, but can favor a narrow set. section 8.6 section 14.1.1

Use α-nDCG to ask whether a ranking surfaces distinct relevant facets, not merely more hits on one facet. No α-nDCG calculation or ground-truth construction procedure is specified here; choosing the metric is distinct from implementing a full RAG evaluation. section 8.6

How Retrieval Augmented Generation, The Foundational Ideas covers it

How deep it goes
Defines the metric’s intent-aware role and contrasts it with precision and NDCG, but provides no α-nDCG formula or calculation.
Examples
Uses a warm-winter-jacket query where similar relevant hits may score well on precision while offering less useful variety.
What it assumes
Assumes familiarity with ranked retrieval and relevance; explains why precision alone can miss facet coverage.
What you can do after
Evaluate/understand it as a candidate metric when distinct query facets matter more than a narrow set of relevant hits.
How it connects
Connects α-nDCG to RAG-Fusion, Precision@k, Recall@k, and NDCG as alternative views of retrieval quality.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Retrieval Augmented Generation, The Foundational Ideas. The section links point to the parts of the book it draws on.