liveBook · Manning

Sample size

As explained in Evaluating AI Systems

Sample size is the number of observations or examples included in a study or evaluation.

Sample size is the number of examples in an evaluation dataset. Too few examples can produce unstable conclusions; collecting more than the evaluation needs can add cost for little benefit. There is no universally correct size. section 4.3.2

Start with the evaluation goal. A small sample may reveal obvious flaws in a smoke test, but benchmark comparisons need enough data to reduce noise. Safety, risk, and fairness evaluations may need larger datasets to capture rare failures or enough examples from small subgroups. section 4.3.2

Task variability matters too. Open-ended dialogue, reasoning, and planning generally require more examples than narrower tasks. Diverse inputs and outputs, multidimensional rubrics, and subgroup analysis can increase the data needed for reliable estimates. section 4.3.2

Figure 4.7 Using metric convergence to assess evaluation dataset size. As more examples are added, metric estimates become increasingly stable, helping identify the point at which additional data yields diminishing returns.

Metric convergence helps identify when estimates stabilize and additional examples yield diminishing returns. section 4.3.2

How Evaluating AI Systems covers it

How deep it goes
Explains sizing factors and metric convergence without prescribing a universal sample size formula.
Examples
Contrasts smoke tests with benchmark comparisons and rare-failure or subgroup evaluations.
What you can do after
Evaluate whether an evaluation dataset is large enough for its goals, task variability, and risk tolerance.
How it connects
Connects sample size to statistical confidence, benchmark comparisons, fairness analysis, and safety testing.

Also explained in

Compare how 5 books explain sample size →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Evaluating AI Systems. The section links point to the parts of the book it draws on.