Data cleaning checks and treats quality problems before data is used for analysis or modeling. Raw business data may contain duplicates, missing values, outliers, and variables that need further treatment. Cleaning belongs alongside data discovery and preprocessing: the quality of the raw data and the rigor of cleaning affect the reliability of the final analysis. section 11.4 section 1.2.3
Start with a dataset sanity check, then decide which treatments fit the business problem:
- Count records and inspect missing, duplicate, and junk values. Check whether a variable is too sparse or carries little information before retaining it. section 3.8
- Inspect outliers and whether seasonal, daily, and weekly patterns are adequately represented. Resolve incompatible date formats when loading data from different sources. section 1.2.3
- Identify categorical variables and any additional variables or treatments the intended model needs. section 11.4
Deciding whether a variable helps
In a customer dataset, a gender field populated for only 0.01% of customers may be too sparse to keep. A field with the same value for every customer adds no distinguishing information. Both decisions require checking the data and its business meaning, not simply dropping columns by habit. section 3.8
Text needs its own cleaning choices. Reviews, comments, and similar sources may contain URLs, punctuation, unwanted characters, and low-value words. Options include stop-word removal, frequency-based removal, customized word lists, and normalization through stemming or lemmatization. Removing words can reduce complexity, but context matters: “it is” can be a complete answer, so deleting those words could discard useful meaning. Python libraries such as nltk can support text parsing and preprocessing. section 7.6 section 7.20
