Chapter 10: Building the quantitative engine: Market direction prediction with machine learning

Introduction

The introduction shifts the book’s focus from protecting capital against credit and fraud losses to pursuing market-beating investment returns.

• Investment objective and alpha: Explain how the investment project differs from the earlier credit and fraud work: it aims to grow capital, not only protect it, by seeking alpha, or market-beating returns.
  Why it matters: This objective gives the quantitative investment work its purpose without assuming its methods will succeed.

Key terms: alpha, market-beating returns, protect capital

10.1 Architecting our AI-powered investment strategy

The section places an AI-powered investment strategy in the evolution of quantitative finance and maps a seven-step investment workflow onto four architectural layers.

• Evolution of quantitative investing: Describe the shift from hypothesis-driven mathematical strategies to machine-learning strategies and then to a paradigm that combines price data, text, automated analysis, and greater explainability.
  Why it matters: This progression explains why the proposed strategy combines kinds of evidence rather than relying on price-based ML alone.
• Seven steps across four layers: Trace how the seven stages of the investment workflow fit into the data asset, modeling, strategy and monitoring, and application layers, including what each layer contributes.
  Why it matters: The layer map keeps predictions connected to an investment thesis, risk controls, and real-world execution.
• Accumulation of quant methods: Explain why newer quantitative methods add to rather than replace older ones, using the section's example of LLMs, factors, ML, and portfolio risk principles.
  Why it matters: It frames the hybrid strategy as a combination of complementary tools, not a claim that each new tool makes previous ones obsolete.

Key terms: Quant 1.0, Quant 2.0, Quant 3.0, Quant 4.0, Modern Portfolio Theory, factor models, seven-step process, data asset layer, modeling layer, strategy and monitoring layer

10.1.1 Architecting our strategy: Aligning the playbook with the four-layer framework

This section maps a seven-step investment workflow onto the book’s four-layer framework to give each part of the application a clear architectural role.

• Investment workflow layer mapping: Explain how the seven-step investment workflow maps to the four-layer framework and why the chronological workflow also needs a system architecture.
  Why it matters: The mapping turns a sequence of tasks into a coherent blueprint for an investment application.

Key terms: seven-step process, four-layer framework, data asset layer, modeling layer, strategy and monitoring layer, application layer

10.1.2 Defining our scope: A practical path to powerful results

This section sets a practical scope for the quantitative engine: daily and weekly predictions using widely available information rather than speed or exclusive data.

• Accessible quantitative strategy scope: Explain why the project focuses on daily and weekly prediction horizons and widely available data instead of high-frequency trading or proprietary datasets.
  Why it matters: These choices make the proposed strategy feasible without specialized trading infrastructure or exclusive data.

Key terms: High-Frequency Trading (HFT), proprietary alternative datasets, daily and weekly prediction horizons, widely available data sources

10.2 The data foundation: The lifeblood of a quant strategy

Defines a quantitative investment strategy and explains why the quality of its data matters more than model complexity.

• Quant Strategy Data Foundation: Explain what makes an investment strategy quantitative and why data quality matters more than a sophisticated model.
  Why it matters: It keeps attention on the data foundation rather than treating a better algorithm as the cure for poor inputs.

Key terms: quantitative strategy, quants, data quality, sourcing, cleaning, and engineering data

10.2.1 The universe of financial data: A practitioner’s reality check

This section explains the strengths and pitfalls of market, fundamental, and alternative data, then gives five criteria for choosing data sources for a quantitative strategy.

• Market data integrity risks: Explain why market prices need corporate-action adjustments and why historical datasets must include delisted companies.
  Why it matters: These errors can make a trading signal or backtest look convincing for the wrong reasons.
• Market data representations: Distinguish the chart and bar representations described here, including what information their different forms emphasize or omit.
  Why it matters: A useful signal may depend on information that a simpler representation leaves out.
• Fundamental data timing limits: Explain what fundamental data contributes, why reporting delays and revisions limit it, and how other indicators can complement it.
  Why it matters: A useful company-health measure is not necessarily available early enough to trade on by itself.
• Alternative data trade-offs: Describe how nontraditional data can provide early signals and why its cost, fading edge, and processing needs complicate its use.
  Why it matters: An early signal is valuable only if its usable edge warrants the expense of obtaining and preparing it.
• Five data selection criteria: Apply the section’s five criteria to explain what makes a data source worth using in a quantitative strategy.
  Why it matters: A plausible signal is not enough if the data is unreliable, late, unscalable, or too expensive to use.

Key terms: market data, adjusted prices, corporate actions, survivorship bias, delisted securities, candlestick charts, line graphs, time bars, tick bars, volume bars

10.2.2 Our strategic choice: Why ETFs as economic proxies?

The section explains why ETFs are useful signals for predicting market direction and warns that relationships between them can change under stress.

• ETFs as market proxies: Explain why the strategy uses ETFs as indicators of macroeconomic conditions and investor sentiment, with SPY as its primary indicator of US equity market direction.
  Why it matters: This explains why ETF prices are useful inputs to the market-direction strategy.
• Regime-dependent asset correlations: Explain why an observed relationship between ETFs should not be assumed to hold in every market regime, using the SPY–GLD example.
  Why it matters: A normally useful ETF relationship may be unreliable during the market stress when a strategy most needs it.

Key terms: ETFs, economic proxies, directional signals, SPY, GLD, regime-dependent, correlation, safe-haven role, extreme liquidity events

10.2.3 Building the ETF toolkit: Selecting economic barometers

The section explains why a deliberately selected set of ETFs can serve as economic and market barometers for a quantitative strategy.

• Research-backed ETF barometers: Explain why the strategy uses a deliberate set of ETFs as proxies for macroeconomic and financial factors. Describe how commodity ETFs and factor-driven ETFs contribute different kinds of information.
  Why it matters: The ETFs are chosen for the economic information they represent, not merely as a collection of tradable funds.
• Credit spread risk signal: Describe what the credit-focused ETFs HYG and LQD represent in this selection and what their spread differential indicates.
  Why it matters: The credit ETFs add a corporate-risk perspective to the broader set of economic barometers.

Key terms: economic barometers, proxy, commodity diversification, factor-driven ETFs, risk premiums, credit-focused ETFs, HYG, LQD, spread differential, credit risk barometer

10.3 Acquiring and preparing the ETF data

The section explains how to choose a useful historical window for ETF data, acquire it for prototyping, and check its quality before modeling.

• Accessible ETF data ingestion: Explain why an open-source Python library that provides public market data is suitable for acquiring historical ETF prices and volume during rapid prototyping and initial strategy validation.
  Why it matters: The data source should fit the stage of the strategy, rather than requiring institutional infrastructure from the outset.
• Historical lookback tradeoff: Explain why both short and excessively long ETF histories can weaken a strategy, and why the section favors a rolling roughly 10-year window.
  Why it matters: The historical window affects whether a promising backtest has faced relevant market conditions.
• Post-download data quality: Justify checking ETF data quality after download, and explain why the selected liquid, established ETFs typically show few missing observations.
  Why it matters: A clean-looking strategy is unreliable if its input data contains consequential errors.

Key terms: ETF universe, historical price and volume data, public market data, data ingestion pipeline, lookback window, market regimes, rolling 10-year window, stress-testing events, fair-weather strategies, data quality checks

10.3.1 Beyond simple APIs: Industrial-strength data architecture

The section explains why an institutional quantitative data pipeline must make data trustworthy, timely, and reusable, not merely handle more instruments than a yfinance prototype.

• Production Data Trust Requirements: Explain why an institutional quant pipeline needs more than greater scale: before modeling, it must establish where data came from, whether it was validated and enriched consistently, whether processing is timely, and whether results can be reproduced.
  Why it matters: A prediction is difficult to rely on if its inputs or results cannot be checked.
• Source Quality And Enrichment: Describe how diversified sources, automated quality gates, and controlled enrichment make financial data suitable for modeling. Include why LLM-produced enrichments must be logged and validated.
  Why it matters: These safeguards keep questionable inputs from becoming seemingly credible model signals.
• Scalable Research Pipeline Operations: Explain how orchestration, parallel processing, and analytical storage help an institutional research pipeline run on time and remain reusable and traceable.
  Why it matters: Research signals are less useful if processing finishes too late or its inputs cannot be retrieved and traced.
• Data Infrastructure As Advantage: Justify why a reliable data pipeline can be a competitive advantage in quantitative finance, while the simple yfinance approach remains appropriate for learning.
  Why it matters: It explains why investing in data operations can matter as much as choosing a predictive model.

Key terms: production quant system, trust, validated, reproduce, data source diversification, automated quality gates, enrichment before modeling, controlled data transformations, workflow orchestration, parallel processing

10.4 Feature engineering: Crafting predictive signals

This section frames feature engineering for predicting tomorrow’s SPY direction and explains why economically meaningful inputs matter more than model complexity.

• Target and predictor roles: Describe feature engineering in this use case as turning raw market data into usable signals by defining a realistic target—tomorrow’s SPY direction—and creating economically meaningful predictors from ETFs used as macroeconomic proxies.
  Why it matters: Separating the outcome from the inputs makes the strategy’s prediction task clear.
• Feature quality over model complexity: Explain why carefully engineered, economically meaningful features can be more valuable than a sophisticated algorithm fed weak raw inputs, using the section’s contrasting examples.
  Why it matters: A more elaborate model cannot be assumed to compensate for weak inputs.

Key terms: feature engineering, target variable, predictors, SPY, ETFs, noise-fitting, logistic regression, rolling 30-day volatility spread

10.4.1 Step 1: Defining the target variable (Y-label)

The section defines a next-day, three-class market-direction target using a conservative return threshold and explains the resulting label imbalance.

• Next-day prediction target: Explain that the target is tomorrow’s return direction, predicted using only today’s information, so future information does not leak into the prediction.
  Why it matters: Keeping the target in the future and the inputs in the present makes the prediction task credible.
• Three-class return thresholds: Describe how tomorrow’s percent return becomes a bullish, bearish, or neutral label under the ±1% rule, and explain why this conservative cutoff is used instead of treating every small movement as directional.
  Why it matters: The threshold determines which market moves the model is asked to recognize.
• Label balance and cutoff choice: Explain why the resulting labels are mostly neutral and why the ±1% threshold is a strategy-dependent choice rather than a universal rule.
  Why it matters: The class mix reflects the trading question chosen, not necessarily a mistake in the data.

Key terms: target variable, Y-label, next day’s return direction, look-ahead bias, R t +1, bullish move, bearish move, neutral move, ±1% cutoff, target distribution

10.4.2 Step 2: Engineering quant-grade predictive features

The section builds predictive features for 20 ETFs by combining asset-level technical indicators with cross-asset relationships, while distinguishing the educational pipeline from institutional-scale feature engineering.

• Foundational ETF technical features: Describe the first feature-engineering step: apply the same technical indicators across all 20 ETFs to capture each asset’s momentum, volatility, and volume dynamics.
  Why it matters: This base layer captures individual asset behavior consistently before the model considers how assets move together.
• Cross-asset relationship signals: Explain why the pipeline adds relationships between ETFs and describe the features used to represent them.
  Why it matters: These features give the model a way to use market relationships, not just the behavior of each ETF in isolation.
• Educational versus institutional scale: Distinguish this streamlined feature pipeline from the broader feature-engineering and selection work described for institutional quant teams.
  Why it matters: It keeps the worked example in perspective without mistaking its feature count for an institutional limit.

Key terms: technical indicators, ETF universe, momentum, volatility, volume, cross-asset relationships, rolling correlations, cross-asset spreads, alpha, dimensionality reduction

10.4.3 Feature engineering realities

The section explains practical checks that make engineered market signals more robust when relationships, conviction, and trading conditions change.

• Correlation breakdown risk: Explain why a historical cross-asset correlation cannot be assumed to hold when a hedge is most needed.
  Why it matters: A signal built on a stable-looking relationship may fail at the moment it matters most.
• Volume as signal validator: Describe how volume ratio helps distinguish a high-conviction breakout from a noisy price move.
  Why it matters: Volume can help keep a price movement that merely looks like a breakout from being treated as a strong signal.
• Neutral market classification: Justify a neutral class rather than forcing every market observation into an up-or-down prediction.
  Why it matters: A model should not manufacture a trading decision when the market offers no clear direction.
• Cross-asset signal opportunities: Explain why relationships between assets may offer more durable signals than simple momentum in one asset.
  Why it matters: Looking across assets can reveal relationships that are harder for the broader market to exploit away.
• Production feature quality checks: Describe the two practical checks the section raises for composite signals and correlated features.
  Why it matters: These checks help keep a composite balanced and avoid feeding a model redundant signals.

Key terms: correlation breakdown, SPY-TLT, flight to quality, liquidity stress, volume ratio, high-conviction moves, false breakouts, three-class classification, neutrality, forced binary predictions

10.4.4 From features to alpha: Next steps

This section connects engineered ETF features to the aim of finding durable trading signals and stresses consistent feature management for modeling.

• Persistent relationships and regime changes: Explain why a quantitative strategy should look beyond price movements to market relationships and regime changes that last long enough to generate sustainable alpha.
  Why it matters: It clarifies what the engineered signals are ultimately meant to help identify.
• Consistent engineered feature management: Describe why engineered ETF features should be versioned and stored systematically for both model training and live inference.
  Why it matters: Consistent features help connect the modeling work to its live use.

Key terms: market relationships, regime changes, sustainable alpha, engineered features, versioned, model training, live inference

10.5 Predictive modeling: Selecting and validating robust ML methods

The section introduces the challenge of choosing a financial prediction model and calls for a skeptical approach to model selection.

10.5.1 The accuracy myth: Why high predictions demand scrutiny

This section explains why reported market-direction accuracy needs scrutiny and why a small, reliable edge can matter more than a spectacular backtest result.

• Historical accuracy inflation: Explain how biased data, selective market periods, and overfitting can make market-direction accuracy look higher than a model could achieve in live trading.
  Why it matters: A striking backtest score is useful only if it reflects performance the model could plausibly repeat.
• Always-up accuracy baseline: Explain why directional accuracy must be compared with a naive prediction baseline in the SPY task.
  Why it matters: The always-up baseline prevents ordinary market tendency from being mistaken for a model’s skill.
• Paper trading limitations: Describe why an accurate-looking backtest may not translate into successful live trading.
  Why it matters: A prediction score is not the same as a tradeable result after real-world costs and constraints.
• Value of persistent edges: Justify why a consistent directional accuracy near 56% could be valuable despite being only slightly above the naive baseline.
  Why it matters: The practical aim is a repeatable, managed advantage, not an eye-catching accuracy number.

Key terms: survivorship bias, look-ahead bias, regime filtering, overfitting, operating environment, class imbalance, SPY prediction task, naive model, baseline, directional accuracy

10.5.2 Our model choice: Why random forest excels for financial signals

This section explains why random forest suits financial prediction and how it fits within broader institutional modeling strategies.

• Regime-Dependent Signal Relationships: Explain how tree-based models capture financial signals whose effects change with market conditions, such as momentum behaving differently when volatility rises.
  Why it matters: A useful financial signal may depend on the market environment rather than having one fixed effect.
• Bagging and Prediction Stability: Describe how random forest uses bagging to make predictions more stable when financial signals are faint and data is noisy.
  Why it matters: Stability matters when the genuine predictive signal is weak relative to market noise.
• Feature Importance for Risk Management: Explain how inspecting random forest feature importances can help investigate a change in the model’s predictions.
  Why it matters: A team needs a way to investigate changing model behavior before relying on its predictions.
• Institutional Model Combinations: Distinguish this section’s single-model random forest approach from institutional strategies that combine models or use different models in different market regimes.
  Why it matters: Choosing random forest for one engine does not imply that one static model is the limit of a professional strategy.

Key terms: non-linearity, interactions, regime-dependent interactions, feature space, bagging, bootstrap aggregating, low SNR, variance-reduction, decorrelated decision trees, feature importances

10.5.3 Time-series cross-validation: The foundation of intellectual honesty

This section explains why market-direction models need time-respecting validation and why class imbalance makes accuracy misleading.

• Time-Respecting Model Validation: Explain why randomly shuffled k-fold cross-validation is unsuitable for market-direction prediction and why validation must respect time order.
  Why it matters: A promising backtest is unreliable if its validation lets future information influence past predictions.
• Class Imbalance And Accuracy: Describe why examining the bullish, bearish, and neutral target-class distribution matters before choosing evaluation metrics.
  Why it matters: Checking the class mix prevents a high accuracy score from being mistaken for a useful trading signal.

Key terms: k-fold cross-validation, arrow of time, time-aware validation splits, class imbalance, bullish (1), bearish (–1), neutral (0), imbalance-aware evaluation metrics

10.5.4 Building a robust validation scheme

This section builds a time-ordered validation scheme for market predictions and adds leakage-aware separation between training and test data.

• Chronological expanding window splits: Describe how TimeSeriesSplit keeps each validation set later than its training set and uses an expanding training window.
  Why it matters: Time ordering is the starting point for a credible test of a market prediction model.
• Overlapping feature information leakage: Explain why TimeSeriesSplit alone can still give inflated performance estimates when features draw on multiple days of data.
  Why it matters: A model can appear more accurate if its validation periods share underlying information with training.
• Embargo and purging safeguards: Distinguish an embargo from purging, and describe how this section’s implemented embargo separates the training and test periods.
  Why it matters: These safeguards reduce the risk that shared information makes validation results look better than they are.

Key terms: TimeSeriesSplit, expanding window, validation set, information leakage, 20-day momentum, performance metrics, embargo, purging, create_embargoed_splits, temporal separation

10.5.5 Training the final model with optimized parameters

This section trains a conservatively configured random forest and interprets its out-of-sample predictions as a modest, conditional trading signal rather than a proven profitable strategy.

• Conservative random forest configuration: Explain why the final random forest uses conservative settings and how its depth, leaf-size, feature-sampling, and split-size limits guard against overfitting financial noise.
  Why it matters: Conservative settings make a faint financial signal less likely to be mistaken for patterns learned from noise.
• Conditional directional trading signal: Interpret the reported 52.2% directional accuracy alongside neutral predictions and incorrect calls on flat days, and explain why trade filtering could help.
  Why it matters: A small directional edge is only useful if the strategy can also identify days worth acting on.
• Out-of-sample versus trading profitability: Distinguish what the test-fold evaluation establishes from what it cannot establish about sustainable trading returns.
  Why it matters: An honest test of prediction is necessary, but trading costs and constraints can erase a small apparent edge.
• Market regime dependence: Explain why aggregate model performance may hide differences across market conditions and name conditions the section suggests checking.
  Why it matters: A signal that looks modest overall may behave differently under different market conditions.

Key terms: random forest, overfitting, tree depth, samples per leaf, feature sampling, minimum samples for splits, Directional Accuracy, neutral market days, volatility thresholds, trade filtering

10.6 From signal to strategy: Next steps

The section interprets the ETF model as a diagnosed but limited signal and explains why profitable investing requires more than predictive accuracy or a single model.

• Prediction Versus Profitability: Explain why the ETF model’s predictive accuracy does not by itself establish that a trading strategy will be profitable, citing payoff ratios and transaction costs.
  Why it matters: A model can predict direction better than chance without producing profitable trades.
• The Diagnosed ETF Signal: Describe what the disciplined validation process established about the ETF model’s strengths and weakness, rather than treating it as a perfect predictor.
  Why it matters: Knowing where a signal works and falls short is more useful than treating its score as a blanket endorsement.
• Diversified Signal Rationale: Justify using the limited ETF signal as one component of a diversified strategy rather than expecting one model to carry all the risk.
  Why it matters: A known weakness in one signal can be managed through diversification rather than mistaken for a reason to discard it.

Key terms: predictive accuracy, profitability, payoff ratio, transaction costs, statistical edge, economic one, time-series validation, embargo, evaluation metrics, diagnosed signal

10.7 Summary