liveBook · Manning

Embedding

As explained in Retrieval Augmented Generation, The Foundational Ideas

A dense vector is a vector whose coordinates are generally populated with informative numerical values.

A search for “automobile maintenance” may miss a document that says “I bought a new car today”: the words do not overlap. An embedding represents text as a dense vector—a compact set of numerical coordinates in which values generally carry information. A model can place related words or passages near one another even when they use different vocabulary. Unlike a sparse keyword vector, which has mostly zero values and records the words present, a dense vector can support similarity search across different phrasing. section 2.1.3 section 2.1.5 section 3.1

Word2Vec learns word embeddings by predicting words from context. Its CBOW and Skip-gram approaches offer different efficiency and quality tradeoffs; negative sampling and hierarchical softmax helped make training on large corpora practical. The learned geometry can reflect semantic relationships, but it inherits gaps and biases in its training data. Vector arithmetic is suggestive, not a dependable test: in the familiar king–man+woman example, the original word can remain the nearest neighbor unless the input words are excluded from ranking. section 2.1.2 section 2.1.5

Keyword ranking versus vector ranking

BM25 gives “I bought a new car today” a score of 0.00 for “automobile maintenance,” while similarity between averaged word vectors gives it 0.54. That makes the document visible to the ranking, but a computer-repair sentence scores 0.55. Averaging word vectors loses word order, negation, and sentence structure; recovering a candidate is not the same as ranking it correctly. section 2.1.3

For sentence retrieval, the representation must fit the task. A BERT cross-encoder processes two texts together and returns a score for that pair; it does not yield independent sentence vectors that can be indexed once. Sentence-BERT instead produces embeddings for separate texts. Similarity between two comparable sentences is also different from open-domain question answering, where a short question must retrieve a longer answer passage. DPR uses separate query and passage encoders for that asymmetric retrieval task. section 2.3 section 3.1 section 4.1.1

In a RAG pipeline, document chunks are embedded and indexed ahead of time. At query time, the question is embedded, nearby chunks are retrieved, and a generator answers from those chunks. This separation makes retrieval fast, but the answer is bounded by what retrieval returns: a top-three search can omit a passage that plainly contains the answer. Dense similarity can also miss an exact part number. Combine embedding-based retrieval with keyword search when exact matches matter, and inspect retrieved passages rather than treating the vector index as an answer store. section 4.1.1 section 4.6 section 11.6

Listing 4.2 Turning a paper into a searchable index.
from langchain_community.document_loaders import PyPDFLoader 
from langchain_text_splitters import RecursiveCharacterTextSplitter 
from langchain_huggingface import HuggingFaceEmbeddings 
from langchain_community.vectorstores import FAISS
PAPER = "https://arxiv.org/pdf/2005.11401" 
splitter = RecursiveCharacterTextSplitter(
  chunk_size=512, chunk_overlap=50
)  #A
chunks = PyPDFLoader(PAPER).load_and_split(text_splitter=splitter) 
embeddings = HuggingFaceEmbeddings(  #B
  model_name="sentence-transformers/all-mpnet-base-v2"
)
index = FAISS.from_documents(chunks, embeddings)  #C
print(f"{len(chunks)} chunks indexed")

The Python listing shows the offline side of a basic pipeline: split a PDF into chunks, embed them with a sentence-transformer model, and build a FAISS index. section 4.6

How Retrieval Augmented Generation, The Foundational Ideas covers it

How deep it goes
Defines dense vectors against sparse vectors, explains word and sentence encoding, and demonstrates indexing while exposing ranking and top-k failures.
Examples
Compares BM25 with averaged word vectors on automobile queries and indexes PDF chunks with sentence-transformer embeddings and FAISS.
What it assumes
Introduces sparse and dense vectors through a concrete contrast; uses Python libraries for the implementation example.
What you can do after
Apply it via HuggingFaceEmbeddings and FAISS to index document chunks, retrieve passages, and check whether the returned context supports an answer.
How it connects
Connects Word2Vec, Sentence-BERT, DPR, BM25, vector similarity search, and retrieval-augmented generation.

Also explained in

Compare how 72 books explain embedding →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Retrieval Augmented Generation, The Foundational Ideas. The section links point to the parts of the book it draws on.