liveBook · Manning

Exact-match accuracy

As explained in LLM Customization and Fine-Tuning

Exact match is a metric that assigns a binary score based on whether a generated response exactly matches a reference after normalization.

Exact-match accuracy measures how often generated responses match their reference answers exactly after whitespace normalization. Each response receives 1 for a match and 0 otherwise; the overall score is the average across examples. section 5.1.3 section 5.1.4

Exact match is strict: a helpful rephrasing can score 0. Use it alongside Token-F1, which rewards word overlap, rather than treating a low exact-match score alone as proof that an answer is unhelpful. Safety refusal rate measures a separate concern: whether the model refuses harmful prompts. section 5.1.4

For a base-model versus LoRA-adapter comparison, evaluate both on held-out examples that were not used earlier. Generate an answer for each example, compare it with the reference, and aggregate the binary scores. The evaluation can also report Token-F1 and safety refusal rate. section 5.1.3

## base
- Overall exact match: 0.0%
- Overall token-F1: 0.149
- Safety refusal rate: 100.0%
## adapter
- Overall exact match: 0.0%
- Overall token-F1: 0.154
- Safety refusal rate: 60.0%
## adapter (Improvement vs Base)
- Token-F1 delta: +0.005
- Safety refusal rate delta: -40.0%
- Per-category token-F1 (base to adapter):
  - networking: 0.169 to 0.170 (+0.001)
  - security: 0.199 to 0.185 (-0.014)
  - windows: 0.133 to 0.141 (+0.008)
  - linux: 0.129 to 0.144 (+0.014)
  - hardware: 0.152 to 0.188 (+0.037)
  - software: 0.122 to 0.111 (-0.011)

In this IT-support evaluation, both models scored 0% exact match on long, open-ended answers. The adapter’s safety refusal rate also fell from 100% to 60%, a separate warning that the exact-match score does not capture. section 5.1.4

How LLM Customization and Fine-Tuning covers it

How deep it goes
Defines binary scoring after whitespace normalization and averages per-example results, while leaving normalization implementation details unspecified.
Examples
Reports 0% exact match for both base and adapter on long, open-ended IT-support answers.
What you can do after
Evaluate and understand exact-match results without mistaking rephrasing for an exact reference match.
How it connects
Pairs exact match with Token-F1 and safety refusal rate to assess distinct aspects of model behavior.

Also explained in

Compare how 8 books explain exact-match accuracy →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of LLM Customization and Fine-Tuning. The section links point to the parts of the book it draws on.