Fidelity
Fidelity is the degree to which an output, model, or representation accurately reflects or is supported by its relevant source, target, evidence, or behavior.
LLM Evaluation and Alignment, The Foundational Ideas
Faithfulness is a metric representing how accurately generated content reflects factual evidence or supported source information.
- How it is used
- Measures hallucination by checking whether generated claims are supported, including against retrieved context in retrieval-augmented generation.
- How deep it goes
- Explains claim decomposition, evidence verification, and score calculation, with caveats about ambiguous claims and support.
- Examples
- Calculates a 0.50 score for a financial answer with three supported claims and three unsupported claims.
- What you can do after
- Evaluate generated answers by decomposing claims, checking evidence, and interpreting faithfulness alongside retrieval quality.
Read how LLM Evaluation and Alignment, The Foundational Ideas explains fidelity →
Retrieval Augmented Generation, The Foundational Ideas
Faithfulness measures whether a generated response is supported by the retrieved context rather than introducing unsupported claims.
- How it is used
- Diagnoses whether RAG generation introduces claims unsupported by retrieved passages, separately from retrieval quality and answer relevance.
- How deep it goes
- Works through atomic claim extraction, context verification, score calculation, per-claim debugging, and an LLM-judge implementation.
- Examples
- Uses a reapplication answer with one unsupported automatic-approval claim and compares grounded and fabricated compliance answers.
- What you can do after
- Diagnose/debug with it by inspecting unsupported claims and distinguishing generation drift from retrieval problems.
Read how Retrieval Augmented Generation, The Foundational Ideas explains fidelity →
Build an Advanced RAG Application (From Scratch)
Faithfulness measures whether the claims in a generated response are factually inferable from the retrieved context.
- How it is used
- Evaluates whether generated answers stay factually grounded in retrieved context within a RAG pipeline.
- How deep it goes
- Defines claim-level factual consistency and its purpose, but does not specify a scoring algorithm or implementation procedure.
- Examples
- Uses retrieved documents, generated responses, and their individual claims as evaluation artifacts.
- What you can do after
- Evaluate whether a RAG response is grounded and identify unsupported information that may indicate hallucination.
Read how Build an Advanced RAG Application (From Scratch) explains fidelity →
Building LLM Applications with DSPy
Groundedness measures how well the ideas in a summary are supported by the original document.
- How it is used
- Evaluates whether generated summaries are supported by their original documents.
- How deep it goes
- Develops a claim-based judge and its validation workflow, then combines groundedness with completeness; defers optimization of the combined program.
- Examples
- Tests a dialog summary against its source and a summary from another dialog, returning groundedness scores of 1.0 and 0.0.
- What you can do after
- Apply it via DSPy to evaluate summaries with a calibrated judge and combine groundedness with completeness.
Read how Building LLM Applications with DSPy explains fidelity →
Effective Conversational AI
Faithfulness is a generation metric measuring whether an answer is factually supported by the retrieved context.
- How it is used
- Measures whether RAG-generated answers are factually supported by retrieved context.
- How deep it goes
- Defines faithfulness, distinguishes it from relevancy, and names evaluators without detailing their scoring procedures.
- Examples
- Uses a question about contract-breach consequences to illustrate checking generated answers against retrieved context.
- What you can do after
- Evaluate whether generated answers align with retrieved passages, using LLM assessment alongside human review.
Essential GraphRAG
Faithfulness is a metric that measures whether a generated answer is supported by the retrieved context without introducing unsupported information.
- How it is used
- Evaluates whether generated RAG answers remain factually grounded in retrieved context.
- How deep it goes
- Decomposes answers into atomic statements and applies binary context-support judgments, stopping short of measuring complete answer correctness.
- Examples
- Uses atomic statement prompts, retrieved contexts, benchmark datasets, and comparative metric tables.
- What you can do after
- Evaluate generated answers with RAGAS using faithfulness, answer correctness, and context recall metrics.
Generative AI in Action
Groundedness is the degree to which an AI output conforms to the information in its sources rather than speculation or fabrication.
- How it is used
- Uses groundedness to assess whether generated responses are supported by factual sources.
- How deep it goes
- Demonstrates an API request and returned ungrounded text, but does not explain score calibration.
- Examples
- Uses a medical summarization request with patient information as the grounding source.
- What you can do after
- Apply groundedness detection through the Azure Content Safety endpoint using a response and grounding sources.
Evaluating AI Systems
A measure of whether a response is faithful to its source information or supporting evidence.
- How it is used
- Uses faithfulness to assess whether RAG answers are grounded in retrieved evidence.
- How deep it goes
- Defines the metric and outlines its inputs, scoring approaches, output, and limitations without specifying a single implementation.
- Examples
- Uses a generated RAG answer and its retrieved context as evaluation inputs.
- What you can do after
- Evaluate whether an answer is supported by retrieved context while recognizing that faithfulness alone does not establish completeness or usefulness.
The Art of AI Product Development
Groundedness is the degree to which generated content is supported by the information retrieved or provided as evidence.
- How it is used
- Evaluates whether RAG-generated content is supported by retrieved information.
- How deep it goes
- Defines groundedness and applies it as an evaluation criterion, without specifying a scoring scale.
- Examples
- Tests a report on AI trends in SaaS and contrasts grounding with relevance to a request for detailed analysis.
- What you can do after
- Evaluate RAG responses for source support using LLM-as-a-judge, while checking evaluation criteria closely.
Read how The Art of AI Product Development explains fidelity →
AI Model Evaluation
Groundedness is a metric measuring whether an output is supported by the information it is expected to rely on.
- How it is used
- Measures whether generated answers are supported by the retrieved context.
- How deep it goes
- Defines groundedness and gives a product-language interpretation, without detailing a calculation method.
- Examples
- Uses a RAG chatbot and explains improvement as more answers supported by retrieved context.
- What you can do after
- Evaluate whether retrieved context supports generated answers and interpret score changes in product-relevant terms.
LLM Customization and Fine-Tuning
A measure of whether a sentiment label matches the content of its sentence.
- How it is used
- Screens generated financial sentiment examples for agreement between each sentence and its assigned label.
- How deep it goes
- Defines a 1–5 label-match score and a pass threshold of 4 within a three-part gate, but does not assess teacher-model quality.
- Examples
- Scores whether a financial sentence’s sentiment label fits, alongside checks for realistic financial writing and plausible facts.
- What you can do after
- Apply it via an LLM-as-judge rubric to accept or reject candidate sentence-label examples.
Read how LLM Customization and Fine-Tuning explains fidelity →
Also covered in
- Bayesian Optimization in Action
- Robotics for ProgrammersIn early access (MEAP)
- Spec-Driven DevelopmentIn early access (MEAP)
- Ensemble Methods for Machine Learning
- Build a Text-to-Image Generator (from Scratch)
- Domain-Specific Small Language Models
- Digital Twins in Action
- Machines that ThinkIn early access (MEAP)
What to know first
- language modelA language model is a computational model that represents patterns in linguistic sequences to assign probabilities, predict, generate, or evaluate language.
- precisionPrecision is the proportion of items predicted, selected, or retrieved as positive that are correct or relevant.
- recallRecall is the proportion of all relevant or actual positive items that a system successfully retrieves or correctly identifies.
- hallucinationA hallucination is model-generated content that is false, fabricated, or unsupported by relevant evidence but presented as plausible or factual.
Related concepts
- Ragasused by · Ragas is an open-source framework and Python library for evaluating retrieval-augmented generation systems and applications.
- accuracycontrasts with · Accuracy is the degree to which a result agrees with a specified truth, reference, or criterion, commonly measured as the proportion of evaluated predictions that are correct.
- hallucinationmeasures · A hallucination is model-generated content that is false, fabricated, or unsupported by relevant evidence but presented as plausible or factual.
- DSPy programmeasured by · A DSPy program is an executable composition of one or more module instances that can be called with inputs declared by a signature.
Questions about fidelity
What is fidelity?
Fidelity is the degree to which an output, model, or representation accurately reflects or is supported by its relevant source, target, evidence, or behavior.
Which Manning books explain fidelity?
LLM Evaluation and Alignment, The Foundational Ideas, Retrieval Augmented Generation, The Foundational Ideas, Build an Advanced RAG Application (From Scratch), Building LLM Applications with DSPy, Effective Conversational AI, Essential GraphRAG, Generative AI in Action, Evaluating AI Systems and 3 more.
What should I know before learning fidelity?
Manning books that teach fidelity usually introduce language model, precision, recall and hallucination first.
What is the difference between fidelity and accuracy?
Fidelity is the degree to which an output, model, or representation accurately reflects or is supported by its relevant source, target, evidence, or behavior. Accuracy is the degree to which a result agrees with a specified truth, reference, or criterion, commonly measured as the proportion of evaluated predictions that are correct.
What is fidelity also called?
Fidelity is also called faithfulness.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.










