liveBook · Manning

Relevance

As explained in Building Reliable AI Systems

A measure of how directly an answer addresses the question or task posed.

Answer relevance measures how directly a generated answer addresses its original question. For RAG evaluation, one practical method embeds the question and answer with a pretrained model such as BERT, then calculates cosine similarity; the resulting score ranges from 0 to 1. section 3.6.3

Embedding similarity

If a question and answer have an embedding similarity score of 0.85, that value is the answer-relevance score. section 3.6.3

How Building Reliable AI Systems covers it

How deep it goes
Calculates question-answer embedding cosine similarity but does not detail model selection or implementation.
Examples
Uses a question-answer similarity score of 0.85 as an answer-relevance example.
What you can do after
Evaluate answer relevance by comparing question and answer embeddings with cosine similarity.
How it connects
Distinguishes answer relevance from retrieval precision, factual accuracy, and RAGAS answer relevancy.
Cover of Building Reliable AI Systems
liveBookmanning.com
Read Building Reliable AI Systems on liveBook Get the book at manning.com

Also explained in

Compare how 11 books explain relevance →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Building Reliable AI Systems. The section links point to the parts of the book it draws on.