liveBook · Manning

Accuracy

As explained in AI Model Evaluation

A measure of how often a model's outputs match desired or correct outcomes.

Accuracy measures how often an LLM judge’s verdicts agree with known-correct labels in a trusted reference dataset. section 10.4.1

Accuracy and stability measure different things: a judge may be right often but vary across reruns, or repeat the same incorrect verdict. section 10.4.1

Correctness is not consistency

A judge can reach correct verdicts most of the time yet give different outputs for the same input on different runs. A judge that repeats its verdict reliably may still always be wrong. section 10.4.1

How AI Model Evaluation covers it

How deep it goes
Distinguishes agreement with labels from run-to-run stability, without specifying a formula or calculation procedure.
Examples
Contrasts a judge that is often correct but noisy with one that is consistent yet wrong.
What it assumes
Assumes known-correct labels or judgments in a gold-standard dataset and does not explain how to create them.
What you can do after
Evaluate whether an LLM judge matches known-correct labels; do not treat that result as evidence of rerun stability.
How it connects
Connects accuracy with gold-standard labels, judge stability, and production validation.

Also explained in

Compare how 49 books explain accuracy →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of AI Model Evaluation. The section links point to the parts of the book it draws on.