A benchmark is a standardized dataset, task, or evaluation procedure for comparing language-model capabilities or performance. It provides an objective starting point for model selection, release criteria, and optimization, but does not replace testing against real users, tasks, and datasets. section 5.4.2 section 5.4.1
Use benchmarks as part of a broader evaluation strategy:
- Inspect public benchmarks and choose proxies close to the target application. section 5.4.2
- Compare model results using task-specific metrics, such as exact match or character-level accuracy. section 5.4.2
- Add custom, outcome-oriented measures for requirements such as accuracy, readability, brand alignment, latency, governance, and cost. section 5.4.1 section 5.5
- Validate shortlisted models on real-world tasks and datasets because public leaderboards change and may not predict application performance. section 5.4.1
Content-generation model selection
A content-generation feature can use MMLU and HELM to inspect broad linguistic performance, ARC and SQuAD to examine factual question answering, and application-specific tests to assess the quality and speed of drafts and edits. section 5.4.2 section 5.4.1
Table 5.2 Common benchmarks for LLM evaluation
Browse a reference table of common language-model benchmarks, including MMLU, AGIEval, ARC, SQuAD, and QuAC, with their evaluated capabilities. section 5.4.2
