Preprocessing turns discovered, often dirty data into a form suitable for analysis or model training. It follows a clear business problem and data discovery, and precedes exploratory data analysis and modeling. Cleaning and preparation can consume substantial project time, so plan for them rather than treating them as a quick prelude to fitting a model. section 1.4 section 11.4
Inspect the data and choose treatments that fit the task:
- Find duplicates, missing values or NaNs, outliers, and junk entries. Check whether categorical variables need treatment for the intended algorithm. section 11.4 section 1.4
- Create additional variables where needed; preparation is not limited to deleting bad rows. section 11.4
- Check data quality, relevance to the business problem, and relationships among variables before committing to modeling. section 11.3
Retail transactions for association rules
A retailer may start with invoices containing dates, item lists, and amounts. Association-rule algorithms require a particular input format, so converting the transaction table is a major preparation task; SQL pivoting is one concept to investigate. For this use case, ignore transactions with NULL or blank values rather than assuming the same missing-value treatment used elsewhere. Clean junk characters from entries before building the final dataset. section 4.8
Table 4.20 Example of invoices generated in a real-world retail store
|
Invoice number
|
Date
|
Items
|
Amount
|
|---|---|---|---|
|
1001
|
01-Jun-21
|
Milk, Eggs, Cheese, Bread
|
$10
|
|
1002
|
01-Jun-21
|
Bread, Bananas, Apples, Butter
|
$15
|
|
1003
|
01-Jun-21
|
Butter, Carrots, Cheese, Eggs, Bread, Milk, Bananas
|
$19
|
|
1004
|
01-Jun-21
|
Milk
|
$1
|
|
1005
|
01-Jun-21
|
Bread
|
$0.80
|
The invoice table shows the starting transaction format whose item lists must be converted for association-rule learning. section 4.8
Text needs a different preparation path. Remove unwanted characters and stop words, normalize words through stemming or lemmatization, and tokenize text to create analysis-ready data. Some text datasets need customized cleaning; Python libraries such as nltk can support text parsing and preprocessing. section 7.5 section 7.20
For generative AI training, preparation can also include gathering, labeling, and structuring representative data. The choice of treatment should follow the intended model and business problem, not a universal cleaning rule. section 10.6 section 4.8
