A search for “automobile maintenance” may miss a document that says “I bought a new car today”: the words do not overlap. An embedding represents text as a dense vector—a compact set of numerical coordinates in which values generally carry information. A model can place related words or passages near one another even when they use different vocabulary. Unlike a sparse keyword vector, which has mostly zero values and records the words present, a dense vector can support similarity search across different phrasing. section 2.1.3 section 2.1.5 section 3.1
Word2Vec learns word embeddings by predicting words from context. Its CBOW and Skip-gram approaches offer different efficiency and quality tradeoffs; negative sampling and hierarchical softmax helped make training on large corpora practical. The learned geometry can reflect semantic relationships, but it inherits gaps and biases in its training data. Vector arithmetic is suggestive, not a dependable test: in the familiar king–man+woman example, the original word can remain the nearest neighbor unless the input words are excluded from ranking. section 2.1.2 section 2.1.5
Keyword ranking versus vector ranking
BM25 gives “I bought a new car today” a score of 0.00 for “automobile maintenance,” while similarity between averaged word vectors gives it 0.54. That makes the document visible to the ranking, but a computer-repair sentence scores 0.55. Averaging word vectors loses word order, negation, and sentence structure; recovering a candidate is not the same as ranking it correctly. section 2.1.3
For sentence retrieval, the representation must fit the task. A BERT cross-encoder processes two texts together and returns a score for that pair; it does not yield independent sentence vectors that can be indexed once. Sentence-BERT instead produces embeddings for separate texts. Similarity between two comparable sentences is also different from open-domain question answering, where a short question must retrieve a longer answer passage. DPR uses separate query and passage encoders for that asymmetric retrieval task. section 2.3 section 3.1 section 4.1.1
In a RAG pipeline, document chunks are embedded and indexed ahead of time. At query time, the question is embedded, nearby chunks are retrieved, and a generator answers from those chunks. This separation makes retrieval fast, but the answer is bounded by what retrieval returns: a top-three search can omit a passage that plainly contains the answer. Dense similarity can also miss an exact part number. Combine embedding-based retrieval with keyword search when exact matches matter, and inspect retrieved passages rather than treating the vector index as an answer store. section 4.1.1 section 4.6 section 11.6
Listing 4.2 Turning a paper into a searchable index.
from langchain_community.document_loaders import PyPDFLoader
from langchain_text_splitters import RecursiveCharacterTextSplitter
from langchain_huggingface import HuggingFaceEmbeddings
from langchain_community.vectorstores import FAISS
PAPER = "https://arxiv.org/pdf/2005.11401"
splitter = RecursiveCharacterTextSplitter(
chunk_size=512, chunk_overlap=50
) #A
chunks = PyPDFLoader(PAPER).load_and_split(text_splitter=splitter)
embeddings = HuggingFaceEmbeddings( #B
model_name="sentence-transformers/all-mpnet-base-v2"
)
index = FAISS.from_documents(chunks, embeddings) #C
print(f"{len(chunks)} chunks indexed")The Python listing shows the offline side of a basic pipeline: split a PDF into chunks, embed them with a sentence-transformer model, and build a FAISS index. section 4.6







































































