A financial news service needs to match fresh articles to investors’ interests without relying only on shared keywords. An embedding model converts text into a numerical vector: a position in a semantic space where related concepts are close. It can represent news content for discovery, user interests for personalization, or document chunks for retrieval. section 11.3.3 section 13.2.1 section 7.4.3
Choose and operate the model as part of a pipeline:
- Check domain understanding, vector dimensionality, speed, and cost. More dimensions may preserve finer distinctions but require more memory and computation; retrieval benchmarks such as MTEB can inform selection. section 13.2.1
- Prepare the input before vectorizing it. For market content, ingest raw text, use an LLM to produce a summary and tags, then embed that enriched text. section 13.3.1
- Batch large embedding jobs and persist their vectors. For ongoing feeds, vectorize new content near real time while recomputing slower-changing user context in batches. section 11.3.3 section 13.3.1
- Keep vectors comparable by managing model versions. In retrieval systems, use a consistent model for indexing and querying; protect sensitive documents before they enter the retrieval corpus. section 11.3.3 section 7.4.3
Embedding a news dataset
Load cleaned news text from Parquet, send it to OpenAI’s text-embedding-3-small in batches of 100, attach each returned vector to its row, and save the result as Parquet. Check the first vector’s length and sample values. Batching avoids thousands of latency-heavy individual API calls. section 11.3.3
Listing 11.7 Batch embedding with the OpenAI API
from openai import OpenAI
from tqdm import tqdm
df = pd.read_parquet('news_ready_for_embedding.parquet')
client = OpenAI(api_key="your-api-key")
print("="*80)
print("STEP 2: TEXT EMBEDDING")
print("="*80)
print(f"Total texts: {len(df)}")
texts = df['text_for_embedding'].tolist()
embeddings = []
batch_size = 100
for i in tqdm(range(0, len(texts), batch_size), desc="Embedding"):
batch = texts[i:i+batch_size]
response = client.embeddings.create(
model="text-embedding-3-small",
input=batch
)
embeddings.extend([item.embedding for item in response.data])
df['embedding'] = embeddings
df.to_parquet('news_embedded.parquet')
print(f"\nSaved: news_embedded.parquet")
if embeddings:
print(f"Embedding dimension: {len(embeddings[0])}")
print(f"Sample embedding (first 5 dims): {embeddings[0][:5]}")
else:
print("No embeddings were generated (empty input).")
The listing shows the batch API call, vector collection, Parquet output, and basic embedding checks. section 11.3.3
