Preprocessing turns raw data into a representation that learning algorithms can use. For text, it converts free-form reviews into structured features; for numeric data, it can place features on comparable scales. section 3.4.1 section 7.2
A practical preprocessing workflow can include:
- Clean text by removing special symbols, then tokenize it into words. section 3.4.1
- Use lemmatization to treat related word forms such as organize, organizes, and organizing as the same word. section 3.4.1
- Build a bag-of-words term-document matrix whose rows represent examples and columns represent word features. section 3.4.1
- Remove selected stop words or transform counts into TF-IDF features when the representation benefits from reduced or reweighted vocabulary. section 3.4.1
- Encode categorical values because most machine-learning algorithms require numeric inputs, and standardize numeric features when scale differences matter. section 8.0 section 7.2
From review text to features
The sentence “this is a terrible terrible movie” becomes a count vector over a vocabulary such as {this, is, a, brilliant, terrible, movie}. The count for “terrible” is 2, while “brilliant” is 0. A larger review corpus can then be filtered, transformed with TF-IDF, and saved for modeling. section 3.4.1
Standardizing AutoMPG
For AutoMPG regression, engine-related features are split into training and test data, fitted with StandardScaler on the training data, and transformed before learning. section 7.2
