11 Building the qualitative engine with news analysis and LLMs

In chapter 10, we meticulously engineered our quantitative engine. By transforming a universe of exchange-traded fund (ETF) price and volume data into predictive features, we trained a machine learning (ML) model to decipher the market’s numerical language. That engine listens to the rhythms of price, momentum, and correlation. But the market doesn’t just speak in numbers; it speaks in narratives, fears, and expectations. A central bank’s subtle shift in tone, a breakthrough technological announcement, or a sudden geopolitical flare-up are the qualitative events that numbers alone often fail to capture until it’s too late.

11.1 The new playbook: From market numbers to market narratives

11.1.1 The hard truth: A reality check on LLMs in investment research

11.1.2 Our philosophy: The LLM as a probabilistic analyst, not a deterministic oracle

11.1.3 The strategist’s hunt for alpha: Where qualitative signals shine

11.1.4 Designing the qualitative engine architecture

11.2 Implementing the news ingestion and processing pipeline

11.2.1 The data sourcing reality: Building our own information infrastructure

11.2.2 Our pragmatic toolkit: A four-source blend

11.2.3 Implementing the fetcher functions

11.3 Building the RAG feature pipeline: From raw news to retrieval-ready vectors

11.3.1 Data unification and cleaning

11.3.2 Chunking strategy: Choosing the right granularity

11.3.3 Text embedding with OpenAI

11.3.4 Vector database storage with ChromaDB

11.3.5 Verification: Trust, but verify the pipeline

11.4 Building the LLM Analyst: From retrieval to quantified signals

11.4.1 Retrieval strategy: A two-step process for context and precision

11.4.2 Structured prompt engineering: The art of instructing an analyst