liveBook · Manning

Effective sample size

Effective sample size is the number of independent observations that would provide an equivalent amount of statistical information as a possibly weighted or correlated sample.

How 2 Manning books explain it

AI Model Evaluation

Effective sample size is the amount of statistically usable evidence remaining after observations are weighted or otherwise adjusted.

How it is used
Assesses how much usable evidence remains for a counterfactual policy comparison after weighting logged outcomes.
How deep it goes
Defines the metric and develops practical checks for fragile estimates, including weight concentration and sensitivity to analysis choices.
Examples
Compares weighted and unweighted rewards for movie recommendation policies and considers an improvement driven by heavily weighted sessions.
What you can do after
Diagnose fragile off-policy estimates by comparing effective sample size with logged volume and inspecting weight distributions.

Read how AI Model Evaluation explains effective sample size →

Machine Learning Algorithms in Depth

Effective sample size is a measure of the amount of independent information represented by a possibly weighted or correlated sample.

How it is used
Diagnoses whether importance-sampling weights provide enough effective information for reliable estimates.
How deep it goes
Derives effective sample size by equating weighted and unweighted-average variances, then applies it as a diagnostic.
Examples
Uses importance-sampler pseudocode and visualizations of sampling components and estimates.
What you can do after
Diagnose importance-sampling estimators by interpreting effective sample size and weight concentration.

Read how Machine Learning Algorithms in Depth explains effective sample size →

Also covered in

Related concepts

Questions about effective sample size

What is effective sample size?

Effective sample size is the number of independent observations that would provide an equivalent amount of statistical information as a possibly weighted or correlated sample.

Which Manning books explain effective sample size?

AI Model Evaluation and Machine Learning Algorithms in Depth.

What kinds of effective sample size are there?

Kinds of effective sample size include bulk effective sample size and tail effective sample size.

What is effective sample size a kind of?

Effective sample size is a kind of size. Size is the extent, quantity, or magnitude of an entity or one of its properties.

These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.