Overfitting occurs when a machine-learning model captures irrelevant variation in training features instead of the signal needed to generalize to new data. In financial prediction, abundant fundamentals, technical indicators, economic data, news, sentiment, and other inputs can obscure useful relationships with noise. section 8.1.2 section 8.1.4
Use these distinctions when evaluating a financial model:
- Selecting too few features can cause underfitting, where the algorithm misses strong correlations between features and labels. section 8.1.4
- Adding too many features increases overfitting risk because uninformative inputs can hide the factors that carry predictive signal. section 8.1.4
- A held-out test set provides a practical check: compare predictions with actual values and calculate validation RMSE. section 8.1.2
Next-day stock-price prediction
A random forest uses historical Open, High, Low, Close, and Volume data to predict the next day’s closing price. The data is split chronologically, with 80% used for training and 20% for testing. Actual and forecast prices can then be plotted together, while RMSE summarizes test-set error. Uneven differences between the two lines show why apparent predictive patterns require cautious validation. section 8.1.2
Listing 8.6 Training data for the random forest
X = df.drop(columns=["Target"])
y = df["Target"]
X_train, X_test, y_train, y_test = train_test_split(X, y, shuffle=False,
test_size=0.2)
rf_model = RandomForestRegressor(n_estimators=100, random_state=42)
rf_model.fit(X_train, y_train)
rf_preds = rf_model.predict(X_test)
rf_rmse = np.sqrt(mean_squared_error(y_test, rf_preds))
results = pd.DataFrame({
"Date": df.index[-len(y_test):],
"Actual": y_test.values,
"RandomForest": rf_preds,
}).set_index("Date")
Inspect the chronological train-test split, random-forest configuration, predictions, and RMSE calculation used for the stock-price example. section 8.1.2
