An outlier is an extreme value near an edge of an observed distribution. It deserves investigation, not automatic deletion: it may be a data-collection error, a unit mismatch created while combining tables, or a legitimate rare case that matters to the analysis. section 2.4.2 section 7.1.5
Start by inspecting descriptive statistics. Compare each numeric feature’s minimum and maximum with its mean, then use charts to see the distribution’s shape. Boxplots make unusually distant observations visible across many features; histograms help distinguish isolated extremes from a skewed distribution. section 2.4.2 section 7.1.5
Choose a response only after identifying why the value is extreme:
- Correct or remove values caused by errors, such as measures expressed in centimeters in one table and meters in another. section 2.4.2
- Keep legitimate rare observations when they reflect the question being analyzed; removing them solely because they are unusual can discard meaningful cases. section 2.4.2
- Cap a feature’s upper tail with winsorization when extreme predictor values need limiting, then inspect the revised boxplot. section 7.1.5
- For a regression target with problematic tails, use percentiles to define a central range, visualize the selected distribution, and filter predictors and target together. section 7.1.6
Checking an unusually large target value
A target with minimum 1,450, average 36,573, and maximum 1,306,500 signals a possible right-tail problem. Percentiles sharpen the diagnosis: when the 99th percentile remains far below the maximum, select a percentile-bounded range and plot its density before retaining that subset for modeling. section 7.1.6
Listing 7.9 Winsorizing extreme values
from scipy.stats.mstats import winsorize
lower_cut_percentile = 0.00 #1
upper_cut_percentile = 0.001 #2
X['minimum_nights'] = winsorize(X['minimum_nights'].values,
limits=(lower_cut_percentile,
upper_cut_percentile))
X[['minimum_nights']].boxplot()
This listing caps the highest 0.1% of minimum_nights values with scipy’s winsorize, preserving rows while reducing the influence of the most extreme upper-tail values. section 7.1.5
