Preprocessing turns raw data into data a model can use. Think of cooking stew: collecting ingredients is not enough; you wash and chop them before cooking. Likewise, after collecting data and before training, you clean and format it. The workflow is iterative: test results may send you back to improve the data. section 8.6
The useful steps depend on the kind of data:
- Tabular data: decide whether to remove or estimate missing values, then encode categories appropriately. One-hot encoding represents a category with a separate on-or-off feature; assigning numbers to unordered categories can imply an order that is not there. section 8.9
- Text for a large language model: select and collect suitable data, then clean and preprocess it. Filter unwanted languages, strip boilerplate such as website navigation, and remove near-duplicates that could encourage memorization. section 11.5 section 11.7
- Images: standardize size and aspect ratio, remove watermarks and duplicates, and filter corrupted or inappropriate files. Otherwise, visual noise can degrade training or appear in generated images. section 12.6
Preparing a diamond dataset
Suppose a row has no Origin value and several rows lack numeric measurements. One possible cleanup removes the row with missing Origin, fills missing measurements with mean values, and one-hot encodes Origin. Giving origins numbered labels instead could suggest that one location ranks above another. Whether to remove the row depends on how important Origin is to the question being asked. section 8.9
Table 8.6 The diamond dataset with one-hot-encoded values
|
Carat |
Cut: Fair |
Cut: Good |
Cut: Very Good |
Cut: Premium |
Cut: Ideal |
|
| 1 |
0.30 |
0 |
1 |
0 |
0 |
0 |
| 2 |
0.41 |
0 |
0 |
0 |
0 |
1 |
| 3 |
0.75 |
0 |
0 |
1 |
0 |
0 |
| 4 |
0.91 |
0 |
0 |
0 |
0 |
0 |
| 5 |
1.20 |
1 |
0 |
0 |
0 |
0 |
| 6 |
1.21 |
0 |
1 |
0 |
0 |
0 |
| 7 |
1.31 |
0 |
0 |
0 |
1 |
0 |
| 8 |
1.50 |
0 |
0 |
0 |
1 |
0 |
| 9 |
1.74 |
0 |
0 |
1 |
0 |
0 |
| 10 |
1.83 |
1 |
0 |
0 |
0 |
0 |
| 11 |
1.96 |
1 |
0 |
0 |
0 |
0 |
| 12 |
1.19 |
0 |
0 |
0 |
1 |
0 |
The diamond table shows how one-hot encoding turns a category into separate columns of zeroes and ones. section 8.9
The practical aim is not to make every dataset look alike. It is to remove misleading noise and give the model consistent, relevant inputs for the question it must answer. section 8.6 section 12.6
