Standardization puts numerical features on a comparable scale by centering each feature at mean 0 and rescaling it to standard deviation 1. This prevents columns such as engine weight, horsepower, and acceleration from entering a downstream regression algorithm at drastically different scales. section 7.2 section 8.1.4
Use the procedure as a small, disciplined preprocessing pipeline:
- Split the data into training and test sets before fitting any scaler. section 7.2 section 7.4.1
- Fit the standardization parameters on training features, then transform both training and test features with that fitted preprocessor. section 7.2 section 7.4.1
- Apply scaling primarily to continuous numerical columns; treat categorical columns with an appropriate categorical preprocessing method instead. section 8.1.4
AutoMPG regression
For AutoMPG, split the data, fit sklearn.preprocessing.StandardScaler on the training data, transform both partitions, and then separate the scaled features from the MPG target. The same pattern applies to the Bike Sharing data, where the feature matrix is standardized while the count-valued target remains the prediction label. section 7.2 section 7.4.1
Listing 7.2 Loading and preprocessing the AutoMPG data set
import pandas as pd
data = pd.read_csv('./data/ch07/autompg.csv') #1
labels = data.columns.get_loc('MPG')
features = np.setdiff1d(np.arange(0, len(data.columns), 1),
labels) #2
from sklearn.model_selection import train_test_split
trn, tst = train_test_split(data, test_size=0.2,
random_state=42) #3
from sklearn.preprocessing import StandardScaler
preprocessor = StandardScaler().fit(trn) #4
trn, tst = preprocessor.transform(trn), preprocessor.transform(tst)
Xtrn, ytrn = trn[:, features], trn[:, labels] #5
Xtst, ytst = tst[:, features], tst[:, labels]
Inspect the complete AutoMPG loading, splitting, fitting, transforming, and feature-target separation workflow. section 7.2
Listing 7.8 Preprocessing the Bike Sharing data set
labels = data.columns.get_loc('casual') #1
features = np.setdiff1d(np.arange(0, len(data.columns), 1),
labels) #2
from sklearn.model_selection import train_test_split
trn, tst = train_test_split(data, test_size=0.2, #3
random_state=42)
Xtrn, ytrn = trn.values[:, features], trn.values[:, labels]
Xtst, ytst = tst.values[:, features], tst.values[:, labels]
from sklearn.preprocessing import StandardScaler
preprocessor = StandardScaler().fit(Xtrn) #4
Xtrn, Xtst = preprocessor.transform(Xtrn), preprocessor.transform(Xtst)
Inspect the Bike Sharing workflow, including fitting StandardScaler on training features and transforming the held-out test features. section 7.4.1
