Effective sample size
Effective sample size is the number of independent observations that would provide an equivalent amount of statistical information as a possibly weighted or correlated sample.
AI Model Evaluation
Effective sample size is the amount of statistically usable evidence remaining after observations are weighted or otherwise adjusted.
- How it is used
- Assesses how much usable evidence remains for a counterfactual policy comparison after weighting logged outcomes.
- How deep it goes
- Defines the metric and develops practical checks for fragile estimates, including weight concentration and sensitivity to analysis choices.
- Examples
- Compares weighted and unweighted rewards for movie recommendation policies and considers an improvement driven by heavily weighted sessions.
- What you can do after
- Diagnose fragile off-policy estimates by comparing effective sample size with logged volume and inspecting weight distributions.
Read how AI Model Evaluation explains effective sample size →
Machine Learning Algorithms in Depth
Effective sample size is a measure of the amount of independent information represented by a possibly weighted or correlated sample.
- How it is used
- Diagnoses whether importance-sampling weights provide enough effective information for reliable estimates.
- How deep it goes
- Derives effective sample size by equating weighted and unweighted-average variances, then applies it as a diagnostic.
- Examples
- Uses importance-sampler pseudocode and visualizations of sampling components and estimates.
- What you can do after
- Diagnose importance-sampling estimators by interpreting effective sample size and weight concentration.
Read how Machine Learning Algorithms in Depth explains effective sample size →
Also covered in
- Grokking BayesIn early access (MEAP)
Related concepts
- off-policy evaluationmeasures · Off-policy evaluation estimates the performance of a target policy using interaction data collected under a different behavior policy.
- importance samplingmeasures · Importance sampling is a statistical technique that estimates expectations under a target distribution by reweighting samples drawn from another distribution.
Questions about effective sample size
What is effective sample size?
Effective sample size is the number of independent observations that would provide an equivalent amount of statistical information as a possibly weighted or correlated sample.
Which Manning books explain effective sample size?
AI Model Evaluation and Machine Learning Algorithms in Depth.
What kinds of effective sample size are there?
Kinds of effective sample size include bulk effective sample size and tail effective sample size.
What is effective sample size a kind of?
Effective sample size is a kind of size. Size is the extent, quantity, or magnitude of an entity or one of its properties.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.

