An out-of-sample forecast is a model’s estimate for an observation that was not used to fit it. A fitted housing-price equation, for example, can take the square footage and bedroom count of a new house and estimate its selling price. Making useful predictions for new records—not merely describing the records used for fitting—is the aim of a predictive model. section 9.1.2 section 9.3.2
A forecast is only useful when its inputs are available before the outcome and the model applies to the setting where it will be used. A housing model fitted in one geographic region may not transfer to another; missing predictors such as renovation spending or commuting distance may also limit it. Accuracy must be tested against the decision stakeholders need to make. section 9.1.2 section 9.3.2
A practical route from fitted model to forecast is:
- Fit on training records, then predict outcomes for held-out test records. section 9.3.2
- Compare training and test performance using measures suited to the outcome and stakeholder need, such as RMSE and R-squared for the illustrated regression. section 9.3.2
- Choose a way to deliver useful predictions: an interactive tool, a batch of estimates, or live predictions that inform actions. section 9.4
from sklearn.metrics import mean_squared_error #1
y_pred_train = model.predict(X_train)
y_pred = model.predict(X_test) #2
r2_train = model.score(X_train, y_train)
r2_test = model.score(X_test, y_test) #3
rmse_train = np.sqrt(mean_squared_error(y_train, y_pred_train))
rmse_test = np.sqrt(mean_squared_error(y_test, y_pred)) #4
print(f"Training Set RMSE: {rmse_train}")
print(f"Test Set RMSE: {rmse_test}")
print(f"Training Set R-squared: {r2_train}")
print(f"Test Set R-squared: {r2_test}")
The listing makes separate training- and test-set predictions and calculates RMSE and R-squared for each, so performance on fitted records is not mistaken for performance on held-out records. section 9.3.2
Forecasting rat sightings
A model uses temperature, wind speed, and weekday to estimate rat sightings. Forecast weather can supply inputs before a future day, but forecast conditions differ from actual conditions and may add error. Testing on multiple samples of forecast-weather data helps assess that risk before the predictions are used for planning. section 9.3.2
