Suppose you want to know whether a classifier will recognize a toddler meltdown it has never seen. Cross-validation repeatedly holds back part of the dataset, trains on the rest, and checks predictions on the held-out data. It turns “does this model generalize?” into a practical experiment rather than a guess. section 8.5 section 8.6
A cross-validation run typically follows this pattern:
- Partition the data into folds, such as three folds, and optionally shuffle the rows. section 8.6
- Use each fold as the test data once, while using the remaining folds for training. section 8.5 section 8.6
- Fit the model on the training portion, predict the held-out classes, and compare predictions with the known labels. section 8.6
- Compare the results across folds instead of trusting one lucky train/test split. section 8.11
Three folds with scikit-learn
Create KFold(n_splits=3, shuffle=True), loop over its training and test indices, fit LogisticRegression on each training portion, and predict the held-out portion. The resulting fold evaluations show how sensitive the classifier’s performance is to the particular rows it trained on. section 8.6
import numpy as np
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import KFold
x = np.array([0.1, 0.5, 0.7, 1.2, 1.2, 1.5, 1.6, 1.9, 2.5, 2.5,
2.7, 3.2, 3.3, 3.7, 3.8, 4.1, 4.5, 4.5, 4.5, 5.0,
5.1, 5.6, 5.7, 6.2, 6.3]) \
.reshape(-1, 1) #1
y = np.array([0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 1.0,
0.0, 0.0, 1.0, 1.0, 0.0, 1.0, 1.0, 1.0, 1.0, 1.0,
1.0, 1.0, 1.0, 1.0, 1.0]) #2
kf = KFold(n_splits=3, shuffle=True) #3
accuracies = [] #3
for fold, (train_index, test_index) \
in enumerate(kf.split(x), 1): #3
X_train, X_test = x[train_index], x[test_index] #3
Y_train, Y_test = y[train_index], y[test_index] #3
model = LogisticRegression(penalty=None) #4
model.fit(X_train, Y_train) #4
predictions = model.predict(X_test) #5
print(f"Fold {fold} predictions: "
f"{predictions}") #6
accuracy = np.sum(predictions == Y_test) \
/ len(Y_test) #7
print(f"Fold {fold} accuracy: {accuracy:.4f}") #7
accuracies.append(accuracy) #7
average_accuracy = np.mean(accuracies) #7
print(f"Average accuracy: {average_accuracy:.4f}") #7
std_dev = np.std(accuracies) #8
print(f"Standard deviation: {std_dev:.4f}") #8
Inspect the compact Python pattern for creating folds, selecting training and test indices, fitting logistic regression, and generating predictions. section 8.6
