Root mean squared error (RMSE) is the square root of the average squared difference between predicted and actual numerical values. It summarizes how far predictions typically fall from the truth, with lower values indicating smaller prediction errors. section 9.3.2
Use RMSE to evaluate predictive regression models in a way that supports practical decisions:
- Calculate it separately for training and testing predictions to compare performance on seen and unseen data. section 9.3.2
- Interpret its magnitude in the target variable’s terms; an RMSE just under one means predictions are, on average, just under one unit from the actual value in the demonstrated model. section 9.3.2
- Compare it with other metrics, such as R-squared, rather than treating it as the only measure of model performance. section 9.3.2 section 9.3.3
- Continue testing on a new validation set with forecasted inputs and monitor for declining performance before delivering predictions to stakeholders. section 9.3.2
Rat-sightings model
A scikit-learn regression model produced training RMSE of 0.84 and testing RMSE of 0.90, alongside R-squared values of 0.61 and 0.57. The small difference between the splits did not raise immediate concern, while the RMSE values indicated predictions were typically less than one rat sighting from the actual count. section 9.3.2
from sklearn.metrics import mean_squared_error #1
y_pred_train = model.predict(X_train)
y_pred = model.predict(X_test) #2
r2_train = model.score(X_train, y_train)
r2_test = model.score(X_test, y_test) #3
rmse_train = np.sqrt(mean_squared_error(y_train, y_pred_train))
rmse_test = np.sqrt(mean_squared_error(y_test, y_pred)) #4
print(f"Training Set RMSE: {rmse_train}")
print(f"Test Set RMSE: {rmse_test}")
print(f"Training Set R-squared: {r2_train}")
print(f"Test Set R-squared: {r2_test}")
The listing shows how to generate predictions, calculate training and testing RMSE with scikit-learn, and compare the results with R-squared. section 9.3.2
