liveBook · Manning

Exact-match accuracy

As explained in Rearchitecting LLMs

Exact match is a metric that counts outputs as correct only when they exactly match the reference under the specified comparison rules.

Exact-match accuracy reports the share of outputs accepted as correct under specified comparison rules. In a tool-call evaluation, those rules normalize case and whitespace, allow small coordinate differences, and accept reasonable supersets of expected text; the comparison functions themselves are not detailed. section 13.3

Fine-tuning comparison

Using exact match as the main metric, the original model scored 88.9% and the fine-tuned T specialist scored 96.3%. The remaining T failures involved incorrect arguments. section 13.3

Table 13.3 Pruning and distillation results for M→S
Model
(transformer blocks / expansion)
Valid JSON
Exact match
Tolerant match
No call
Wrong function
Wrong args
M
(25, 2304)
100.0%
92.6%
92.6%
0
0
4
Depth Pruning
(22, 2304)
100.0%
13.0%
20.4%
0
38
5
+ Width pruning
(22, 1792)
100.0%
22.2%
24.1%
0
33
8
S, post-KD
(22, 1792)
100.0%
88.9%
88.9%
0
0
6

Compare exact and tolerant match alongside valid JSON and failure types across pruning and distillation stages. section 13.5

How Rearchitecting LLMs covers it

How deep it goes
Defines an evaluation harness with normalized, tolerance-aware comparisons, but defers details of the comparison functions.
Examples
Compares original and fine-tuned models at 88.9% and 96.3%, and reports scores across pruning and distillation stages.
What it assumes
Assumes test examples with expected responses and a way to extract tool-call JSON; comparison-function details are deferred.
What you can do after
Evaluate tool-call models by scoring extracted responses against expected outputs under configured comparison rules.
How it connects
Links model fine-tuning, pruning, and knowledge distillation to tool-call evaluation.

Also explained in

Compare how 8 books explain exact-match accuracy →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Rearchitecting LLMs. The section links point to the parts of the book it draws on.