Z-score test

As explained in Outlier Detection in Python

A z-score test is an outlier-detection procedure that flags values whose absolute z-scores exceed a selected cutoff.

A z-score test detects unusual numeric values by measuring each value’s distance from the mean in standard deviations. It flags values whose absolute z-score exceeds a chosen threshold. section 2.1.1

Apply the test by:

• Calculating the mean and standard deviation of the values. section 2.1.1
• Computing each value’s z-score as its number of standard deviations from the mean. section 2.1.1
• Flagging values whose absolute z-score exceeds a selected cutoff; 3.0 is common but conventional rather than universally correct. section 2.1.1

Example values

For values with mean 6.6 and standard deviation 10.04, the value 34 has a z-score of 2.73. A 3.0 cutoff flags none, although 34 may be considered unusual in a larger comparable dataset. section 2.1.1

Figure 2.2 A swarm plot with at least one outlier, point A. Depending on the presence of A, points B and C may or may not be flagged as outliers, as A can affect the mean and standard deviation.

A swarm plot illustrates how an extreme point can change the mean and standard deviation, affecting whether neighboring points are flagged. section 2.1.1

How Outlier Detection in Python covers it

How deep it goes

Explains calculation, thresholding, examples, and distribution effects, but stops short of a full implementation.

Examples

Uses a small value set, the value 34, ideal distributions, and a real-world abalone dataset column.

What it assumes

Assumes numeric values and familiarity with the mean and standard deviation while defining their role in detection.

What you can do after

Diagnose/debug with it by calculating standardized distances and selecting a defensible outlier threshold.

How it connects

Connects outlier detection to the mean, standard deviation, percentiles, and distribution shape.

liveBookmanning.com

Read Outlier Detection in Python on liveBook Get the book at manning.com

Related concepts

• z-scoreuses · A z-score is a signed standardized value expressing how many standard deviations an observation lies above or below a reference mean.
• maskingproduces · Masking is an outlier effect in which extreme observations cause other outliers to appear insufficiently unusual for detection.
• swampingproduces · Swamping is an outlier effect in which extreme observations cause otherwise typical observations to be incorrectly identified as outliers.
• thresholdconfigured by · A threshold is a boundary value against which a quantity is compared to determine whether a condition is met or an action or decision is triggered.

This explainer was generated by Manning's knowledge graph from the text of Outlier Detection in Python. The section links point to the parts of the book it draws on.