liveBook · Manning

Data cleaning

As explained in Software Engineering for Data Scientists

Data cleaning is the detection and correction or removal of inaccurate, inconsistent, incomplete, or otherwise unsuitable data.

Data cleaning minimizes issues that make datasets unsuitable for analysis or modeling, following exploration and validation in a typical data science workflow. section 1.2.7

Common cleaning tasks include:

Ad-click prediction

A dataset for predicting ad clicks may be imbalanced because most users never click an ad, making imbalance an explicit cleaning concern. section 1.2.7

Cleaning can occur during preprocessing before model training; at larger scales, processing and transformations may also require attention to memory and efficiency. section 1.2.3 section 1.4.2

Figure 1.4 A sample data science workflow. Data science workflows involve key steps like gathering data, exploring and cleaning the data, feature engineering, and model development. The model evaluation piece is also a highly important step, that we will cover in detail when we discuss model monitoring later in this book.
sample data science workflow

Use the workflow figure to place cleaning alongside data gathering, exploration, feature engineering, model development, and evaluation. section 1.2.7

How Software Engineering for Data Scientists covers it

How deep it goes
Defines four issue categories and treatments but stops short of implementation procedures or detailed scaling techniques.
Examples
Uses missing-value replacement, correlated-variable removal, skew transformation, and ad-click imbalance as concrete examples.
What it assumes
Builds on exploratory data analysis and data validation, while treating cleaning as a distinct preparation activity.
What you can do after
Apply it by minimizing dataset issues before analysis or model training and by recognizing scaling concerns in larger workflows.
How it connects
Ties data cleaning to exploratory data analysis, data validation, feature engineering, preprocessing, model training, and scaling.

Also explained in

Compare how 18 books explain data cleaning →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Software Engineering for Data Scientists. The section links point to the parts of the book it draws on.