Sample size is the number of examples in an evaluation dataset. Too few examples can produce unstable conclusions; collecting more than the evaluation needs can add cost for little benefit. There is no universally correct size. section 4.3.2
Start with the evaluation goal. A small sample may reveal obvious flaws in a smoke test, but benchmark comparisons need enough data to reduce noise. Safety, risk, and fairness evaluations may need larger datasets to capture rare failures or enough examples from small subgroups. section 4.3.2
Task variability matters too. Open-ended dialogue, reasoning, and planning generally require more examples than narrower tasks. Diverse inputs and outputs, multidimensional rubrics, and subgroup analysis can increase the data needed for reliable estimates. section 4.3.2
Figure 4.7 Using metric convergence to assess evaluation dataset size. As more examples are added, metric estimates become increasingly stable, helping identify the point at which additional data yields diminishing returns.

Metric convergence helps identify when estimates stabilize and additional examples yield diminishing returns. section 4.3.2




