Faithfulness measures whether a generated answer’s factual claims are supported by the retrieved context. It differs from answer relevancy: an answer can be supported by sources yet still miss the user’s question. section 6.5.3
Use faithfulness as one part of RAG evaluation:
- Check whether the answer aligns with retrieved documents; an LLM can provide an initial score, with human reviewers verifying accuracy and contextual relevance. section 6.5
- LlamaIndex’s FaithfulnessEvaluator assesses alignment with retrieved context to measure hallucination; RAGAS also offers faithfulness as a custom metric. section 6.5.3 section 6.5.4
A grounding check
For “What are the consequences of breaching a contract?”, check whether the chatbot’s answer is factually supported by the retrieved context. section 6.5.3
Table 6.5 Critical aspects influencing generation metrics
|
Aspect
|
Summary
|
Example
|
|---|---|---|
|
Faithfulness
|
Evaluates the factual accuracy of generated output based on retrieved context
|
Ensuring chatbot answers are factually correct when responding to questions. E.g., “What are the consequences of breaching a contract?”
|
|
Answer relevancy
|
Assesses how relevant the generated answer is to the specific user question
|
“What is the status of my order?” The chatbot, after retrieving the relevant data, responds with, “Your order #12345 is currently in transit and expected to be delivered by August 15th.” This response is directly relevant to the user’s question, providing specific information about the order status without unnecessary details.
|
|
Fine-tuning
|
Improves generation by aligning the LLM with domain- specific data, enhancing accuracy and relevance
|
Fine-tuning an LLM for legal advice, ensuring generated responses are accurate and legally relevant
|
|
Prompt engineering
|
Crafting prompts to guide the LLM in generating more contextually appropriate and relevant responses
|
Using prompt engineering to ensure a healthcare chatbot provides clear, relevant medical suggestions
|
|
Model blending
|
Combining specialized models to enhance the quality of generation, balancing accuracy and fluency
|
Blending a retrieval-focused model with a language-focused model to generate accurate, fluent responses
|
|
Sensibleness and Specificity Average (SSA)
|
Measures response quality in open-domain chatbots, ensuring responses are sensible and specific
|
Assessing open-domain chatbot responses to ensure they make sense and are not overly vague
|
|
Faithfulness- Evaluator
|
Assesses whether the generated response avoids hallucinations by aligning with retrieved context
|
Using FaithfulnessEvaluator to ensure that a finance chatbot’s responses are grounded in retrieved documents
|
Compare faithfulness with answer relevancy, which checks whether the response fits the user’s question. section 6.5.3
