liveBook · Manning

Embedding

As explained in Build an Advanced RAG Application (From Scratch)

An embedding is a numerical vector representation of an item that encodes meaningful features or relationships in a continuous space.

An embedding is a dense numerical vector that represents text—such as a token, sentence, document, or chunk—so its semantic and syntactic meaning can be processed computationally. Nearby vectors represent contextually related items, making embeddings useful when meaning matters more than exact word overlap. section 2.2 section 4.2

Figure 2.3 illustrates the vector representation of words. Although the numerical values may not be immediately interpretable, the visualization effectively showcases the proximity of words in their numeric representation, highlighting their close association based on contextual meaning.

Vector proximity visualizes the central intuition: numerical coordinates are not directly readable, but closeness can reflect contextual association. section 2.2

An encoder creates embeddings by transforming input text into vectors. For sentence-level retrieval, a Sentence Transformer produces one fixed-size vector per sentence, allowing vectors to be stored, indexed, and compared independently. SBERT uses siamese networks and triplet loss during training to emphasize semantic relationships among sentences. section 2.3.1 section 4.2

A semantic retrieval workflow uses embeddings as follows:

  • Choose the text unit to represent, such as hotel reviews or semantically coherent document chunks. section 4.1 section 6.4
  • Encode each unit with a selected encoder model, producing dense vectors that capture semantic meaning. section 4.2.1 section 6.4
  • Encode the incoming query with the same model and compare it with stored vectors using cosine similarity or Euclidean distance. section 3.5 section 4.6
  • Index vectors for fast retrieval; bi-encoders permit precomputation and storage, while FAISS or Qdrant can search vectors at scale. section 4.6 section 6.4

Hotel review retrieval

Represent each Paris hotel review as an embedding, then represent a query such as “Hotel with a view of the Eiffel Tower” in the same space. A similarity search ranks reviews whose vectors are closest to the query, returning Eiffel Tower-view reviews even when phrasing differs. Retrieved reviews can then supply context to a generation model. section 4.0 section 6.1

Embedding design is a systems trade-off. Open-source models can run locally or in cloud infrastructure with hardware under your control; API models shift computation to a provider but incur usage costs and reduce control. Higher-dimensional models may improve results at a fee. Long inputs also require care: a fixed-size vector must compress increasingly diverse content, while longer context sharply increases attention computation. section 4.2 section 4.2.1 section 6.3.2

How Build an Advanced RAG Application (From Scratch) covers it

How deep it goes
Explains vector meaning, sentence encoders, similarity ranking, indexing, and long-context constraints without deriving embedding-model training internals.
Examples
Uses hotel reviews and Eiffel Tower queries, plus chunked research-paper abstracts stored as 768-dimensional vectors.
What it assumes
Introduces embeddings from raw text and tokens while assuming enough Python familiarity to load models, encode data, and run indexes.
What you can do after
Build it from scratch by encoding text, indexing vectors, and retrieving semantically similar content for search or RAG.
How it connects
Connects embeddings to Transformer encoders, Sentence-BERT, cosine similarity, semantic search, FAISS, Qdrant, and retrieval-augmented generation.

Also explained in

Compare how 72 books explain embedding →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Build an Advanced RAG Application (From Scratch). The section links point to the parts of the book it draws on.