Vectorization maps text to a numerical vector that captures aspects of its meaning. An embedding model can turn a question, paragraph, or document chunk into a point in a high-dimensional space. Texts with similar meanings should land near one another even when they use different words. For example, “How do I get reimbursed for therapy?” and “What’s the process for submitting a mental health claim?” may be close enough to find through semantic search. section 4.1 section 4.1.1
In retrieval-augmented generation (RAG), documents and incoming queries are vectorized with the same embedding model. Similarity search then selects nearby document chunks to supply as context for an answer. This makes vectorization a reliability concern, not just a data-conversion step: an insurance assistant can have the correct policy and still miss a physical-therapy answer if its embeddings fail to surface that policy. Closeness is a retrieval signal, not proof that a passage is current or correct. section 4.0 section 4.5.4
Design the vectorization step around the retrieval task:
- Choose a model for the language and domain of both documents and user queries. Commercial APIs offer a quick route; open-source models offer local operation and tuning; domain-specific models can improve in-domain retrieval but generalize less well outside it. section 4.1.2 section 4.2.4
- Fit chunks to the model’s input limit. Long inputs may be truncated; small chunks can lose context, while large ones can introduce noise. Test chunk size, overlap, and the number of chunks retrieved rather than assuming one setting works everywhere. section 4.2.4 section 3.6.7
- Budget for vector dimensions. Larger embeddings can capture more nuance but cost more to store and process. If the model supports dimensional truncation, measure the retrieval-quality trade-off before using it to reduce cost. section 4.2.4
- Keep source metadata alongside vectors so retrieval can filter by fields such as policy ID, effective date, and status. Measure the results with evaluation queries; vectorization quality matters only insofar as it helps retrieve the right context. section 4.5.4 section 3.6.7
Health-care policy search
Split policy documents into overlapping chunks, then encode their text with a medical-domain SentenceTransformer model. Create a Pinecone index whose dimension matches the model output and whose similarity metric is cosine. Upsert each vector with its chunk text and policy metadata. For a question such as “Is physical therapy covered?”, encode the question with the same model, query the index, and optionally filter results to active policies before passing retrieved excerpts to the answer generator. Domain alignment may improve retrieval precision, while status and date metadata help avoid returning outdated policy context. section 4.5.4 section 4.0
from sentence_transformers import SentenceTransformer
import numpy as np
model = SentenceTransformer('pritamdeka/S-PubMedBert-MS-MARCO') #1
chunk_texts = [chunk['content'] for chunk in all_chunks] #2
embeddings = model.encode(chunk_texts, show_progress_bar=True) #3
print(f"Generated embeddings with shape: {embeddings.shape}")
The encoding listing shows the operational boundary: collect chunk text, run the selected model, and inspect the resulting embedding array’s shape before configuring the vector index. section 4.5.4
