Relevance
Relevance is the degree to which information, a result, or a response pertains to a query, task, purpose, or information need.
Elasticsearch in Action, Second Edition
Relevance is the degree to which a search result matches a query's intent or information need.
- How it is used
- Ranks full-text search results so documents most appropriate to a user query appear first.
- How deep it goes
- Defines scored ranking, explains BM25 factors and field-length normalization, and defers detailed alternatives to similarity algorithms.
- Examples
- Uses Java book-title searches, Stack Overflow results, retail searches, and a Java-score result figure.
- What you can do after
- Apply it via Elasticsearch full-text queries to retrieve and rank documents for text searches.
Read how Elasticsearch in Action, Second Edition explains relevance →
LLM Evaluation and Alignment, The Foundational Ideas
Answer relevance is a generation-quality metric measuring how directly and appropriately a generated answer addresses the question or information need.
- How it is used
- Evaluates whether a RAG response addresses the user’s question, including in reference-free production monitoring.
- How deep it goes
- Defines a three-step back-translation procedure using generated questions, embeddings, and average cosine similarity, with a worked partial-answer example.
- Examples
- Uses a financial research question about Q3 revenue drivers and Q4 risks to illustrate an accurate but incomplete answer.
- What you can do after
- Diagnose RAG response failures by comparing answer relevance with faithfulness and retrieval scores.
Read how LLM Evaluation and Alignment, The Foundational Ideas explains relevance →
MongoDB 8.0 in Action, Third Edition
A relevance score is a numeric value representing how closely a search result matches the search criteria.
- How it is used
- Ranks Atlas Search matches by how closely documents satisfy search criteria.
- How deep it goes
- Explains scoring factors and demonstrates score extraction, sorting, and summary statistics without deriving Lucene’s scoring formula.
- Examples
- Uses inspection searches with required text matches and a local fuzzy search that returns the three highest-scoring results.
- What you can do after
- Apply it via Atlas Search to display scores and order matching documents by relevance.
Read how MongoDB 8.0 in Action, Third Edition explains relevance →
Retrieval Augmented Generation, The Foundational Ideas
Answer relevance is a metric measuring how well a generated answer addresses the user's question.
- How it is used
- Diagnoses whether a RAG response answers the user's question, complementing retrieval-focused context relevance and grounding-focused faithfulness.
- How deep it goes
- Explains reverse-question generation, embedding similarity, and averaged scores, then works a numerical example without deriving embedding internals.
- Examples
- Works a mortgage reapplication query with similarities of 0.89, 0.82, and 0.91, yielding an answer-relevance score of 0.87.
- What you can do after
- Apply it via RAGAS to flag off-topic or wrongly specific answers, while checking faithfulness before trusting a relevant response.
Read how Retrieval Augmented Generation, The Foundational Ideas explains relevance →
AI-Powered Search
A relevance grade is an ordinal, categorical, or numeric assessment of how relevant a document is to a query.
- How it is used
- Uses query–document relevance grades as training labels and relevance scores to order search results.
- How deep it goes
- Explains manual binary judgments and click-derived probability grades, including prior adjustment for limited evidence.
- Examples
- Uses movie-query judgments and blue-ray click and examination counts to illustrate binary and probability grades.
- What you can do after
- Apply it via Python judgment and training-data utilities to label query–document pairs and generate click-derived grades.
AI for Everyday IT
Relevance is the quality of directly relating information or requirements to the task or use case being addressed.
- How it is used
- Guides AI prompts toward the information needed for a particular task or use case.
- How deep it goes
- Explains relevance through comparisons between broad requests and prompts with task-specific context and requirements.
- Examples
- Uses database selection, a small IT project, and job interview practice to illustrate relevance.
- What you can do after
- Apply it via an AI assistant by adding task context and requirements to prompts.
AI Model Evaluation
The degree to which retrieved, recommended, or generated results address the intended query, context, task, or user need.
- How it is used
- Uses relevance to evaluate ranked recommendations and generated answers against user interests or queries.
- How deep it goes
- Works through offline recommendation evaluation and a structured judge rubric, while cautioning that benchmarks may miss live product behavior.
- Examples
- Uses a genre-based movie recommender, interaction-based proxy labels, Precision@K, and a JSON-scoring judge prompt.
- What you can do after
- Evaluate recommendation relevance with offline ranking metrics and structured judge rubrics while recognizing proxy-label and benchmark limitations.
Build an Advanced RAG Application (From Scratch)
Answer relevancy measures how closely a generated answer addresses the original question, including through similarity to questions generated from the answer.
- How it is used
- Evaluates whether generated answers directly address user questions within a RAG generation pipeline.
- How deep it goes
- Defines the metric and its artificial-question similarity procedure, but does not provide implementation code or scoring details.
- Examples
- Generates artificial questions from an answer and compares them with the original question rather than relying on keyword matching.
- What you can do after
- Evaluate RAG answer targeting and compare system configurations using answer relevancy within the RAGAS metric suite.
Read how Build an Advanced RAG Application (From Scratch) explains relevance →
Effective Conversational AI
Answer relevance is the degree to which an answer directly addresses the user's question and context.
- How it is used
- Assesses whether a generated chatbot answer addresses the user’s particular question and context.
- How deep it goes
- Defines question–answer fit and contrasts it with faithfulness, but gives no scoring formula for answer relevancy.
- Examples
- Contrasts an order-status reply with unnecessary detail and generic restaurant advice with location-specific guidance.
- What you can do after
- Evaluate/understand whether a chatbot’s response answers the user’s actual question, separately from whether retrieved information supports it.
Generative AI in Action
Answer relevancy is a metric that measures how directly a generated answer addresses the question or information need.
- How it is used
- Uses relevance to assess whether LLM outputs satisfy the context and intent of their intended application.
- How deep it goes
- Defines a five-point rubric and comparison steps, but defers deeper metric mechanics and full framework implementation.
- Examples
- Compares two summaries of a Gates Foundation article and demonstrates G-Eval and DeepEval evaluation workflows.
- What you can do after
- Evaluate summaries and other LLM outputs with explicit criteria, scores, comparisons, and ongoing test monitoring.
The Art of AI Product Development
A metric measuring how directly a generated answer addresses the user's query or intent.
- How it is used
- Evaluates whether generated RAG responses satisfy the user’s query, requested content, and required depth.
- How deep it goes
- Defines relevance, contrasts it with groundedness, and illustrates evaluation criteria without specifying a formal scoring method.
- Examples
- Uses an AI-trends report, requested depth, user-specific transformations, and style or tone feedback as evaluation cases.
- What you can do after
- Evaluate/understand RAG responses with query-specific criteria and identify when prompting, retrieval, or generation needs improvement.
Read how The Art of AI Product Development explains relevance →
WordPress Power Toolkit: Harness AI to build next-level websites
Search relevance is the degree to which a search result corresponds to the meaning and subject matter of a user's query.
- How it is used
- Explains how search systems prioritize meaningful results and how website elements support a page’s central message.
- How deep it goes
- Defines relevance, connects it to semantics and latent semantic indexing, and applies it to a visual critique without developing the underlying mathematics.
- Examples
- Uses query interpretation, the meanings of “right,” semantically rich content, and a disconnected potato image on an Instant Pot homepage.
- What you can do after
- Evaluate search content and website elements for alignment with user intent, subject matter, message, placement, and context.
Read how WordPress Power Toolkit: Harness AI to build next-level websites explains relevance →
Designing AI Systems
A retrieval relevance distribution is the statistical distribution of relevance scores produced for retrieved results across queries.
- How it is used
- Monitors retrieval quality over time through aggregate metrics and operational dashboards.
- How deep it goes
- Explains histogram aggregation, dimensional labels, and the limits of averages, but stops short of defining the scoring algorithm.
- Examples
- Uses Data Service search metrics, relevance-score histograms, dashboard distributions, and trace-based automated scoring.
- What you can do after
- Diagnose retrieval degradation by comparing score distributions across workflows, indexes, and question categories.
Build an AI Agent (From Scratch)
Relevance is the degree to which an answer focuses on the central aspects of a user's question without unrelated content.
- How it is used
- Evaluates whether an agent answers the user’s core question directly instead of producing unrelated, evasive, or incomplete content.
- How deep it goes
- Defines binary PASS/FAIL criteria and embeds them in a rubric-based evaluator, without using partial credit for this item.
- Examples
- Uses user prompts, final answers, expected answers, evidence excerpts, and PASS/FAIL evaluation outputs as assessment artifacts.
- What you can do after
- Evaluate agent answers with a binary rubric and track answer relevancy rates alongside other quality indicators.
Read how Build an AI Agent (From Scratch) explains relevance →
Starting Data Analytics with Generative AI and Python
Relevance is the data-quality property of being pertinent to the specific business question or problem being addressed.
- How it is used
- Keeps text analysis tied to the specific business question and the portion of reality being examined.
- How deep it goes
- Defines relevance and supplies inspection questions, but stops short of prescribing a formal relevance score or algorithm.
- Examples
- Uses competitor reviews, outdated reviews, order-linked comments, general impressions, and duplicated complaints as checks.
- What you can do after
- Evaluate whether text data belongs to the business question before analyzing it.
Read how Starting Data Analytics with Generative AI and Python explains relevance →
Machines that Think
Relevance is the degree to which a response appropriately addresses the topic or requirements of an interaction.
- How it is used
- Distinguishes context-sensitive stakes in learning, dialogue quality, and token-to-token relationships in attention.
- How deep it goes
- Explains query-key dot products, softmax weights, and value contributions, while distinguishing internal token scores from response quality.
- Examples
- Uses dialogue quality and the attention comparison between “rises,” “sun,” and “the.”
- What you can do after
- Understand how relevance differs across human learning, language quality, and attention scores.
Building Reliable AI Systems
A measure of how directly an answer addresses the question or task posed.
- How it is used
- Evaluates how relevant generated answers are to their questions in RAG systems.
- How deep it goes
- Calculates question-answer embedding cosine similarity but does not detail model selection or implementation.
- Examples
- Uses a question-answer similarity score of 0.85 as an answer-relevance example.
- What you can do after
- Evaluate answer relevance by comparing question and answer embeddings with cosine similarity.
Also covered in
- Context EngineeringIn early access (MEAP)
- Natural Language Processing in Action, Second Edition
- Human-in-the-Loop Machine Learning: Active learning and annotation for human-centered AI
- Ensemble Methods for Machine Learning
- AI Recommender SystemsIn early access (MEAP)
- Enterprise RAGIn early access (MEAP)
- Software Testing with Generative AI
- Grokking AI Algorithms, Second Edition
- Domain-Specific Small Language Models
- Architecting for AutonomyIn early access (MEAP)
- Think Distributed Systems
What to know first
- language modelA language model is a computational model that represents patterns in linguistic sequences to assign probabilities, predict, generate, or evaluate language.
- retrieval-augmented generationRetrieval-augmented generation is a technique that retrieves relevant information from external sources and supplies it as context to a generative model when producing an output.
- hallucinationA hallucination is model-generated content that is false, fabricated, or unsupported by relevant evidence but presented as plausible or factual.
- similarity searchSimilarity search is a retrieval technique that finds or ranks items by their similarity to a query, often by comparing feature or vector representations under a similarity or distance measure.
Related concepts
- cosine similaritymeasured by · Cosine similarity measures the directional alignment of two nonzero vectors as the cosine of the angle between them, equal to their dot product divided by the product of their magnitudes.
- Ragaspart of · Ragas is an open-source framework and Python library for evaluating retrieval-augmented generation systems and applications.
- Azure AI Searchproduced by · Azure AI Search is Microsoft’s managed cloud search service for indexing, enriching, and retrieving content through keyword, semantic, vector, and hybrid search.
- BM25produced by · BM25 is a probabilistic information-retrieval ranking function that scores documents for a query using term frequency, inverse document frequency, and document-length normalization.
- boost factorused by · A numeric multiplier that increases the contribution of a field to a document's relevance score.
Questions about relevance
What is relevance?
Relevance is the degree to which information, a result, or a response pertains to a query, task, purpose, or information need.
Which Manning books explain relevance?
Elasticsearch in Action, Second Edition, LLM Evaluation and Alignment, The Foundational Ideas, MongoDB 8.0 in Action, Third Edition, Retrieval Augmented Generation, The Foundational Ideas, AI-Powered Search, AI for Everyday IT, AI Model Evaluation, Build an Advanced RAG Application (From Scratch) and 9 more.
What should I know before learning relevance?
Manning books that teach relevance usually introduce language model, retrieval-augmented generation, hallucination and similarity search first.
What is relevance part of?
Relevance is part of Ragas. Ragas is an open-source framework and Python library for evaluating retrieval-augmented generation systems and applications.
What is relevance also called?
Relevance is also called relevancy.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.
















