Concept drift occurs when the relationships between predictors and the target change over time, causing a previously effective model to underperform on new data. section 4.3
Cross-validation estimates future performance using current information, but changing economic, consumer, social, or political conditions can eventually invalidate those estimates. section 4.3
Treat concept drift as a model-lifecycle concern:
- Expect models to require repeated retraining rather than assuming one training cycle remains effective indefinitely. section 7.2.6
- Interpret strong cross-validation results as estimates based on available data, not guarantees about changed future conditions. section 4.3
An end-to-end model
An XGBoost pipeline can be trained and evaluated through cross-validation, then retrained in later iterations when predictor-target relationships no longer reflect reality. section 7.2.6
Listing 7.18 Training the model with full data
best_params = study.best_params
print(best_params)
xgb = XGBRegressor(**best_params) #1
model_pipeline = Pipeline([('processing', column_transform), ('xgb', xgb)])
cv_splits = cv.split(X, y=X["neighbourhood_more_than_30"]) #2
r2_scores = []
rmse_scores = []
mae_scores = []
for train_index, test_index in cv_splits: #3
X_train, X_test = X.iloc[train_index], X.iloc[test_index]
y_train, y_test = y.iloc[train_index], y.iloc[test_index]
model_pipeline.fit(X_train, y_train)
y_pred = model_pipeline.predict(X_test)
r2_scores.append(r2_score(y_test, y_pred))
rmse_scores.append(np.sqrt(mean_squared_error(y_test, y_pred)))
mae_scores.append(mean_absolute_error(y_test, y_pred))
print(f"Mean cv R-squared: {np.mean(r2_scores):.3f}")
print(f"Mean cv RMSE: {np.mean(rmse_scores):.3f}")
print(f"Mean cv MAE: {np.mean(mae_scores):.3f}")
model_pipeline.fit(X, y) #4
The training listing shows a complete pipeline, cross-validation splits, repeated fitting, predictions, and performance metrics. section 7.2.6
