A market-direction model must do more than produce a high accuracy number: its calls must identify useful moves without triggering too many trades on flat days. Directional accuracy measures the share of correct up-or-down calls among cases where the model predicted a direction and the market actually moved. section 10.5.5 section 10.5.1
For labels up (1), down (−1), and neutral (0), keep only predictions with pred ≠ 0 and actual ≠ 0. Directional accuracy is the number of matching labels divided by the number of retained cases. Report how many cases remain: this filter excludes directional calls on neutral days, even though those calls could prompt unproductive trades. section 10.5.5
Listing 10.10 Trading-focused evaluation
from sklearn.metrics import classification_report
print("=== TRADING-FOCUSED EVALUATION ===")
# Calculate directional accuracy (Up/Down predictions only)
directional_accuracies = []
prediction_counts = {'up': 0, 'down': 0, 'neutral': 0}
for i in range(len(all_predictions)):
pred, actual = all_predictions[i], all_actuals[i]
# Count prediction types
if pred == 1:
prediction_counts['up'] += 1
elif pred == -1:
prediction_counts['down'] += 1
else:
prediction_counts['neutral'] += 1
# Focus on directional accuracy
directional_mask = [(p != 0 and a != 0) for p, a in zip(all_predictions, all_actuals)] #1
directional_preds = [p for p, mask in zip(all_predictions, directional_mask) if mask]
directional_actuals = [a for a, mask in zip(all_actuals, directional_mask) if mask]
if len(directional_preds) > 0:
directional_correct = sum(p == a for p, a in zip(directional_preds, directional_actuals))
directional_accuracy = directional_correct / len(directional_preds)
else:
directional_accuracy = 0
# Summary results
balanced_acc = balanced_accuracy_score(all_actuals, all_predictions) #2
print(f"Balanced Accuracy: {balanced_acc:.1%}")
print(f"Directional Accuracy: {directional_accuracy:.1%}")
print(f"Prediction Distribution:")
print(f" Up signals: {prediction_counts['up']}")
print(f" Down signals: {prediction_counts['down']}")
print(f" Neutral: {prediction_counts['neutral']}")
The trading-focused evaluation listing constructs the nonneutral prediction-and-outcome mask and counts predictions by class. section 10.5.5
Reading a 52.2% result
An out-of-sample directional accuracy of 52.2% means the model got slightly more than half of eligible directional calls right. It does not measure performance on neutral days or establish trading profit. Check the separate balanced accuracy result, 45.7% for the three-class task, before judging broader classification performance. section 10.5.5
Treat striking accuracy claims cautiously. Look-ahead bias, survivorship bias, selected market regimes, overfitting, and class imbalance can inflate results. An always-up prediction can reach roughly 52%–55% accuracy when the market rises that often; that unconditional figure is not directly comparable with accuracy filtered to actual directional moves. Use independent time-series tests with embargo periods, then assess neutral-day calls, execution costs, and risk limits before acting on a signal. section 10.5.1 section 10.7 section 10.5.5
