liveBook · Manning

Accuracy

As explained in LLM Evaluation and Alignment, The Foundational Ideas

Accuracy is the proportion of evaluated outputs that match the correct answer or ground truth.

Accuracy is the proportion of evaluated outputs that match the correct answer or ground truth: accuracy = correct outputs / evaluated outputs. For language-model claims, correctness depends on what evidence is accepted as authoritative, so evaluation criteria should make that choice explicit. section 5.1.2

A correct answer is not necessarily grounded in the supplied context. A summary may include a true claim that does not appear in the source document. Conversely, a claim can appear grounded while being factually wrong. Accuracy alone does not distinguish these cases. section 5.1.2 section 5.5

Figure 5.3 The LLM Reliability Quadrant. Contextual grounding (vertical axis) asks whether the output stays within the provided sources. Factual alignment (horizontal axis) asks if the output is correct according to verifiable evidence. The safest output is both grounded and aligned. The riskiest output is neither grounded nor aligned. The mixed-signal quadrants are operationally important because the output can look based on source but wrong, or look correct but not grounded.

The reliability quadrant separates correctness from grounding, including cases where only one holds. section 5.1.2

In a closed-domain task, a supplied document or database provides a reference for checking claims. In an open-domain task, there is no single grounding document; checks may instead use external evidence or consistency across generations. Consistency alone can miss a claim that is repeatedly generated but wrong. section 5.1.1 section 5.5

Checking a financial summary

For a summary of a financial report, check whether each claim traces back to the report. A claim absent from the report is ungrounded, even if it happens to be true. Record grounding separately from factual correctness rather than treating them as the same result. section 5.1.2

The choice of what to evaluate matters. Classifying stated claims does not by itself capture an omitted answer: missing relevant information is a recall or answerability failure, rather than necessarily a hallucination. section 5.1.2

How LLM Evaluation and Alignment, The Foundational Ideas covers it

How deep it goes
Distinguishes correctness from grounding and describes verification approaches, without specifying an accuracy calculation.
Examples
Uses financial-report summaries and document question answering to illustrate source-based verification.
What you can do after
Evaluate and understand whether claims are grounded and factually aligned, while recognizing that omitted answers require separate attention.
How it connects
Connects factual correctness with contextual grounding, hallucination detection, natural language inference, and recall.

Also explained in

Compare how 49 books explain accuracy →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of LLM Evaluation and Alignment, The Foundational Ideas. The section links point to the parts of the book it draws on.