Generation-based evaluation scores a model’s completed response instead of sampling a single answer token from a fixed representation. It better mirrors open-ended usage, especially when models produce chain-of-thought reasoning before their final answer. section 16.1.3 section 16.1.4
A typical evaluation flow combines:
- A task-specific prompt that requests reasoning and constrains the final answer format. section 16.1.4
- Nonzero-temperature generation when reasoning models benefit from sampling. section 16.1.4
- Exact-match checking against the required final answer, even when the response includes long-form reasoning. section 16.1.4
Multiple-choice generation
A prompt can require concise reasoning and an ending such as “Therefore, the answer is (C).” The evaluator then extracts or checks the final answer letter rather than accepting a single sampled token as the whole response. section 16.1.4
Answer the following multiple-choice question by giving the correct answer
letter in parentheses.
Provide CONCISE reasoning for the answer, and make sure to finish the
response with "Therefore, the answer is (ANSWER_LETTER)" where
(ANSWER_LETTER) is one of (A), (B), (C), (D), (E), etc.
Question: {question}
(A) {choice_A}
(B) {choice_B}
(C) ...
Answer the above question and REMEMBER to finish your response with the
exact phrase "Therefore, the answer is (ANSWER_LETTER)" where
(ANSWER_LETTER) is one of (A), (B), (C), (D), (E), etc.
Use the prompt template to enforce a predictable reasoning-and-answer format for generative multiple-choice evaluation. section 16.1.4
