Data cleaning prepares data for modeling by identifying and removing strong outliers, while also addressing encoding, scaling, null values, feature engineering, and feature selection. section 4.6
For outlier detection, cleaning is especially important when a detector supports only novelty detection and requires training data without outliers. A practical strategy runs several statistical or machine-learning detectors, removes only a small set of strongest findings, then may add those records back for scoring. section 4.6
Common cleaning choices include:
- Inspect null counts and fill selected numeric values, such as replacing missing strikeouts with the column median. section 4.6
- Remove extreme numeric rows by calculating quartiles and an interquartile-range threshold, then retaining rows below every column limit. section 4.6
- Treat synthetic inlier generation as a stronger form of cleaning when contamination is high, while checking whether its distribution represents real data. section 11.1.2
Listing 4.2 Removing extreme values from the data
limit_dict = {}
for col_name in df.columns:
if (df[col_name].dtype in ['int64', 'float64']):
q1 = df[col_name].quantile(0.25)
q3 = df[col_name].quantile(0.75)
iqr = q3 - q1
limit = q3 + (3.5*iqr)
limit_dict[col_name] = limit
cond = [True]*len(df)
for key, limit in limit_dict.items():
cond = cond & (df[key] <= limit)
print(len(df))
clean_df = df[cond]
print(len(clean_df))
Use the worked dataframe pattern to calculate per-column limits and construct a cleaned subset. section 4.6
