liveBook · Manning

Encoder

As explained in Build an Advanced RAG Application (From Scratch)

An encoder is a neural-network component that transforms an input sequence into a numerical representation containing information about its content and relationships.

An encoder transforms tokens or text sequences into dense numerical representations that preserve semantic and contextual relationships. In a Transformer, it forms one tower of a multi-layer encoder–decoder architecture; self-attention lets it relate positions across a sequence while processing them in parallel. section 2.2 section 2.5

Encoder-only models produce contextual embeddings rather than generated continuations. BERT learns bidirectional context by predicting masked words, making its representations useful for text classification, named entity recognition, sentiment analysis, and question answering. SBERT adapts this idea to whole-sentence similarity by producing embeddings that can be compared directly. section 2.3 section 2.3.1

For semantic retrieval, an encoder is the representation stage of a larger pipeline:

  • Convert documents, reviews, and queries into vectors that capture meaning beyond exact keyword overlap. section 2.3.3 section 4.2
  • Store and index document vectors, then compare a query vector with them using similarity or distance metrics. section 4.0
  • Use a bi-encoder when query and documents must be encoded separately for efficient similarity search; use a cross-encoder when joint query-document interaction justifies greater computation. section 4.0
  • Pass retrieved information to a decoder when natural-language recommendations or answers are needed in a RAG system. section 5.3.1

Encoding hotel reviews

A hotel finder can encode hotel descriptions and reviews alongside free-form requests such as “quiet hotels with ocean views and a spa.” Vector similarity supports matches based on concepts such as peacefulness even when the same keywords are absent. Retrieved hotel information can then support personalized generated recommendations. section 2.4 section 2.3.3

A practical sentence-embedding workflow loads a pretrained Sentence Transformers model, optionally moves it to CUDA, calls encode on a list of texts, and checks the resulting embedding shape. Model selection balances representation quality and multilingual support against cost, hardware control, scalability, and whether inference runs locally or through an API. section 4.2.1 section 4.2

Figure 4.2 An Overview of the various steps involved in creating Semantic Search from scratch for Hotel Finder in Paris: Travelle

Use the pipeline overview to place encoder-based embedding generation before distance-based retrieval in semantic search. section 4.0

How Build an Advanced RAG Application (From Scratch) covers it

How deep it goes
Explains Transformer roles, bidirectional masked-language training, sentence embeddings, model selection, and encoding calls; defers detailed retrieval implementation.
Examples
Uses masked-word prediction, climate-article retrieval, hotel-review search, and SentenceTransformer encoding of sample sentences.
What it assumes
Introduces tokens and embeddings from raw text before using pretrained transformer encoders.
What you can do after
Apply it via Sentence Transformers to encode text for vector-based semantic search.
How it connects
Connects encoders to Transformers, self-attention, BERT, SBERT, semantic search, embeddings, bi-encoders, cross-encoders, and Retrieval-Augmented Generation.

Also explained in

Compare how 28 books explain encoder →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Build an Advanced RAG Application (From Scratch). The section links point to the parts of the book it draws on.