A fraud detector can approve every transaction and appear excellent on a dashboard. Classification accuracy is the share of evaluated cases predicted correctly, but that number is useful only alongside the class balance and the cost of mistakes. If 99% of transactions are legitimate, approving all of them yields 99% accuracy while detecting no fraud. section 8.1.1
Check what a reported accuracy figure actually measures:
- In fraud detection, compare it with precision and recall. Precision tracks how many fraud flags are correct; recall tracks how much actual fraud is caught. These expose customer friction and missed losses that overall accuracy can hide. section 8.1.1
- In market-direction prediction, compare it with a naive baseline. If the market rises on roughly 52%–55% of days, always predicting “up” can achieve about 55% accuracy without predictive intelligence. section 10.5.1
- Check the test conditions. Look-ahead bias, cherry-picked periods, overfitting, and excluded market regimes can inflate reported accuracy; out-of-sample results still leave transaction costs and slippage to assess. section 10.5.1 section 10.5.5
A conditional directional score
One trading evaluation reports 52.2% directional accuracy, but counts only cases where the model predicted up or down and the market actually moved. It therefore does not measure whether the model avoids calling trades on neutral days. The same results report 45.7% balanced accuracy for a three-class problem. State the evaluated subset before comparing either figure with a baseline or deciding whether a signal is tradable. section 10.5.5
Listing 10.10 Trading-focused evaluation
from sklearn.metrics import classification_report
print("=== TRADING-FOCUSED EVALUATION ===")
# Calculate directional accuracy (Up/Down predictions only)
directional_accuracies = []
prediction_counts = {'up': 0, 'down': 0, 'neutral': 0}
for i in range(len(all_predictions)):
pred, actual = all_predictions[i], all_actuals[i]
# Count prediction types
if pred == 1:
prediction_counts['up'] += 1
elif pred == -1:
prediction_counts['down'] += 1
else:
prediction_counts['neutral'] += 1
# Focus on directional accuracy
directional_mask = [(p != 0 and a != 0) for p, a in zip(all_predictions, all_actuals)] #1
directional_preds = [p for p, mask in zip(all_predictions, directional_mask) if mask]
directional_actuals = [a for a, mask in zip(all_actuals, directional_mask) if mask]
if len(directional_preds) > 0:
directional_correct = sum(p == a for p, a in zip(directional_preds, directional_actuals))
directional_accuracy = directional_correct / len(directional_preds)
else:
directional_accuracy = 0
# Summary results
balanced_acc = balanced_accuracy_score(all_actuals, all_predictions) #2
print(f"Balanced Accuracy: {balanced_acc:.1%}")
print(f"Directional Accuracy: {directional_accuracy:.1%}")
print(f"Prediction Distribution:")
print(f" Up signals: {prediction_counts['up']}")
print(f" Down signals: {prediction_counts['down']}")
print(f" Neutral: {prediction_counts['neutral']}")
The trading-focused evaluation code builds a mask requiring both prediction and outcome to be non-neutral. That filter defines the denominator of its directional-accuracy calculation. section 10.5.5
