Evaluation systematically measures a language model’s behavior against defined tasks, criteria, or benchmarks. Benchmarks are discrete question sets or environments targeting properties such as knowledge, mathematical reasoning, software engineering, safety, or chat quality. In post-training, evaluation supplies signals for tracking whether training makes a model more useful, but the chosen signals also shape optimization priorities. section 16.0 section 16.3
Evaluation regimes have shifted with model behavior. Early RLHF systems emphasized chat quality through LLM-as-a-judge benchmarks such as MT-Bench and AlpacaEval; later suites spanned multiple skills; current reasoning and tool-use evaluations include GPQA, SWE-Bench, and AIME. A benchmark must match the behavior being elicited: a prompting mismatch with training format can reduce measured performance nearly to zero. section 16.0 section 16.6
A useful evaluation pipeline treats scores as outputs of a configured measurement procedure:
- Define the target capability and select tasks that measure it, while separating training, development, and genuinely held-out test sets where possible. section 16.2 section 16.3
- Specify prompts, answer format, sampling parameters, and inference-time settings; these choices can change scores as much as or more than small model differences. section 16.2 section 16.6
- Check for contamination, since evaluation examples may leak into training through web scraping or user queries; n-gram overlap matching is used for decontamination. section 16.6
- Combine scalable AI feedback with human evaluation when needed: human comparisons can yield Elo rankings, while reward-model evaluation can measure agreement. section 16.3 section 12.4.1
Generative multiple-choice reasoning evaluation
Rather than sample one answer token, prompt a model to produce concise reasoning and end with an exact answer-letter phrase. Exact-match checking then extracts the final answer. This generation-based approach accommodates chain-of-thought reasoning, commonly uses nonzero temperature for best performance, and makes prompt wording and output formatting part of the evaluation specification. section 16.1.4
Answer the following multiple-choice question by giving the correct answer
letter in parentheses.
Provide CONCISE reasoning for the answer, and make sure to finish the
response with "Therefore, the answer is (ANSWER_LETTER)" where
(ANSWER_LETTER) is one of (A), (B), (C), (D), (E), etc.
Question: {question}
(A) {choice_A}
(B) {choice_B}
(C) ...
Answer the above question and REMEMBER to finish your response with the
exact phrase "Therefore, the answer is (ANSWER_LETTER)" where
(ANSWER_LETTER) is one of (A), (B), (C), (D), (E), etc.
The prompt template illustrates how required reasoning and an exact final-answer string make output extraction reproducible, while also making the prompt itself consequential. section 16.1.4
Cross-lab score comparisons deserve large error bars, not precise rankings. Internal prompts, sampling settings, hidden evaluation sets, inference-time scaling, and benchmark hillclimbing are often undisclosed. Even under a fixed setup, reasoning-era evaluations can vary; benchmark saturation further removes meaningful signal. Treat reported numbers as rough indicators unless the full procedure and data separation are known. section 16.2 section 16.6
