A customer asks where an order is. An answer about shipping policies may be accurate, but it does not resolve the question. Answer relevance is how directly a response addresses the user’s specific question and context. “Your order #12345 is in transit and expected by August 15th” is relevant because it gives the requested status without unnecessary detail. section 6.5.3
Relevance is distinct from faithfulness. Faithfulness asks whether an answer is supported by retrieved context; relevance asks whether it answers the question. A response needs both: correct source information is not enough if the chatbot uses it to answer the wrong question. section 6.5.3
From generic to context-aware
Asked about current restaurant guidelines in New York, an intent-based chatbot gives general dining advice and tells the user to check local regulations. A retrieval-augmented response instead uses current local guidance to address the location in the question. The improvement is not merely more detail; it is detail that matters to this user. section 6.2.2
Evaluate answer relevance with representative user questions, then compare responses when prompts, retrieval, indexing, or embeddings change. Keep retrieval quality separate from the final answer: finding a pertinent document does not by itself show that the generated response addresses the user. For sensitive questions, avoid filling gaps with unsupported advice; when the available resources do not yield a clear answer, hand the user to a specialist. section 6.5.4 section 6.2.2
Table 6.5 Critical aspects influencing generation metrics
|
Aspect
|
Summary
|
Example
|
|---|---|---|
|
Faithfulness
|
Evaluates the factual accuracy of generated output based on retrieved context
|
Ensuring chatbot answers are factually correct when responding to questions. E.g., “What are the consequences of breaching a contract?”
|
|
Answer relevancy
|
Assesses how relevant the generated answer is to the specific user question
|
“What is the status of my order?” The chatbot, after retrieving the relevant data, responds with, “Your order #12345 is currently in transit and expected to be delivered by August 15th.” This response is directly relevant to the user’s question, providing specific information about the order status without unnecessary details.
|
|
Fine-tuning
|
Improves generation by aligning the LLM with domain- specific data, enhancing accuracy and relevance
|
Fine-tuning an LLM for legal advice, ensuring generated responses are accurate and legally relevant
|
|
Prompt engineering
|
Crafting prompts to guide the LLM in generating more contextually appropriate and relevant responses
|
Using prompt engineering to ensure a healthcare chatbot provides clear, relevant medical suggestions
|
|
Model blending
|
Combining specialized models to enhance the quality of generation, balancing accuracy and fluency
|
Blending a retrieval-focused model with a language-focused model to generate accurate, fluent responses
|
|
Sensibleness and Specificity Average (SSA)
|
Measures response quality in open-domain chatbots, ensuring responses are sensible and specific
|
Assessing open-domain chatbot responses to ensure they make sense and are not overly vague
|
|
Faithfulness- Evaluator
|
Assesses whether the generated response avoids hallucinations by aligning with retrieved context
|
Using FaithfulnessEvaluator to ensure that a finance chatbot’s responses are grounded in retrieved documents
|
Compare the order-status example with the table’s separate faithfulness criterion to see why a directly useful answer needs both question fit and factual support. section 6.5.3
