A financial news feed must connect a reader’s interests with relevant articles, even when their words do not match. An embedding is a high-dimensional numerical vector that represents meaning: similar concepts occupy nearby positions in a semantic space. This makes similarity search possible beyond exact keywords or ticker symbols. section 11.3.3 section 13.0
Build the vectors from usable data, not raw identifiers alone:
- For news ingestion, clean the text, send it to an embedding model, and store each returned vector alongside its source record. section 11.3.3
- For asset discovery, enrich tickers with semantic tags and natural-language descriptions so searches can connect related ideas beyond exact matches. section 13.1.3
- For personalization, represent portfolios and behavior as user vectors. Distances to financial concepts can support segmentation, recommendations, and risk monitoring without a rule for every relationship. section 13.2
Vectorizing a changing news feed
Ingest an article, community post, or educational video; use an LLM to produce a summary and metadata such as sentiment, tags, and complexity; then embed the enriched summary and tags with text-embedding-3-small. Content arrives continuously and needs near-real-time vectorization, while slower-changing user context can be recomputed in batches. The two pipelines can then meet when the feed selects content. section 13.3.1 section 13.0
Listing 11.7 Batch embedding with the OpenAI API
from openai import OpenAI
from tqdm import tqdm
df = pd.read_parquet('news_ready_for_embedding.parquet')
client = OpenAI(api_key="your-api-key")
print("="*80)
print("STEP 2: TEXT EMBEDDING")
print("="*80)
print(f"Total texts: {len(df)}")
texts = df['text_for_embedding'].tolist()
embeddings = []
batch_size = 100
for i in tqdm(range(0, len(texts), batch_size), desc="Embedding"):
batch = texts[i:i+batch_size]
response = client.embeddings.create(
model="text-embedding-3-small",
input=batch
)
embeddings.extend([item.embedding for item in response.data])
df['embedding'] = embeddings
df.to_parquet('news_embedded.parquet')
print(f"\nSaved: news_embedded.parquet")
if embeddings:
print(f"Embedding dimension: {len(embeddings[0])}")
print(f"Sample embedding (first 5 dims): {embeddings[0][:5]}")
else:
print("No embeddings were generated (empty input).")
The Python listing reads cleaned news from Parquet, embeds batches of 100 through the OpenAI API, writes the vectors back to Parquet, and checks the returned vector dimension. Batching avoids thousands of latency-heavy individual requests. For production comparability, pin the embedding model version so later vectors remain consistent with earlier ones. section 11.3.3
A close match is a retrieval signal, not permission to recommend a product. In investment personalization, validate the feed through A/B testing and retain safeguards against presenting high-risk products to vulnerable users. section 13.0
