liveBook · Manning

Exact-match accuracy

As explained in Build an AI Agent (From Scratch)

Exact-match evaluation scores a prediction as correct only when it matches the reference answer under specified normalization rules.

Exact-match accuracy marks a prediction correct only when it equals the reference after case-insensitive comparison and surrounding-whitespace removal. Missing predictions receive a failing score. section 2.4.2

Apply the metric with these evaluation rules:

  • Return false when the prediction is None, including error or refusal outcomes. section 2.4.2

GAIA answer validation

A prediction such as " Paris " matches the reference "paris" after stripping whitespace and lowercasing; a missing prediction remains incorrect. section 2.4.2

How Build an AI Agent (From Scratch) covers it

How deep it goes
Defines and implements case and whitespace normalization, but does not describe broader semantic or partial-credit matching.
Examples
Uses GAIA final answers, normalized strings, missing predictions, and refusal or error outcomes.
What it assumes
Assumes a prediction and reference answer, while teaching case and whitespace normalization explicitly.
What you can do after
Apply it via a Python validator that returns a Boolean correctness result for each prediction-reference pair.
How it connects
Connects answer validation with model evaluation, structured responses, refusals, and failure metadata.

Also explained in

Compare how 3 books explain exact-match accuracy →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Build an AI Agent (From Scratch). The section links point to the parts of the book it draws on.