Exact-match accuracy marks an evaluation attempt correct only when the generated answer matches the required answer string or format. For a multiple-choice task, sampling a response and checking whether it is A, B, C, or D is exact match; log-likelihood scoring instead compares token probabilities without sampling. section 16.1.1
Use exact match with attention to the evaluation interface:
- Treat a single sampled attempt as exact match; aggregate multiple samples with majority voting, while pass@k serves the analogous role for coding evaluations. section 16.1.1
- Normalize or extract answers carefully when prompts require suffixes such as “The answer is:” or use regular expressions to find answer letters or strings. section 16.1.1
- Interpret a low score cautiously when rigid formatting, rather than the model’s underlying capability, creates the bottleneck. section 16.1.1
Multiple-choice checking
A prompt asks for the correct option and requires the response to end with “Therefore, the answer is (ANSWER_LETTER).” The evaluator searches the generated response for the final option and counts it correct only when it matches the reference answer. section 16.1.4
Answer the following multiple-choice question by giving the correct answer
letter in parentheses.
Provide CONCISE reasoning for the answer, and make sure to finish the
response with "Therefore, the answer is (ANSWER_LETTER)" where
(ANSWER_LETTER) is one of (A), (B), (C), (D), (E), etc.
Question: {question}
(A) {choice_A}
(B) {choice_B}
(C) ...
Answer the above question and REMEMBER to finish your response with the
exact phrase "Therefore, the answer is (ANSWER_LETTER)" where
(ANSWER_LETTER) is one of (A), (B), (C), (D), (E), etc.
Shows how concise chain-of-thought prompting constrains the output format so exact-match checking can extract a final answer. section 16.1.4
