liveBook · Manning

Mean pooling

As explained in Build an Advanced RAG Application (From Scratch)

Mean pooling forms a sequence representation by averaging the vectors of its elements.

Mean pooling converts token embeddings into one sentence embedding by averaging the token vectors. In semantic search, it bridges BERT’s sequence output and a fixed-length representation for similarity matching. section 3.4

A Sentence Transformer pipeline applies mean pooling as one stage:

  • Encode text with a BERT Transformer to produce token embeddings. section 3.4
  • Average the token embeddings with mean pooling, producing a 384-dimensional sentence vector in the shown model. section 3.4
  • L2-normalize the resulting vector before comparing sentence representations. section 3.4

Sentence Transformer output

With all-MiniLM-L6-v2, six example sentences encode to an array shaped (6, 384). The pooling component enables each variable-length sentence to become one comparable vector for cosine-based search. section 3.4

 >>> SentenceTransformer(
>>> (0): Transformer({'max_seq_length': 256, 'do_lower_case': False}) with Transformer model: BertModel 
>>> (1): Pooling({'word_embedding_dimension': 384, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False})
>>> (2): Normalize()
>>> )

Inspect the model architecture showing Transformer, mean-token pooling, and normalization as separate components. section 3.4

How Build an Advanced RAG Application (From Scratch) covers it

How deep it goes
Explains the aggregation role and pipeline placement, but does not derive token masking or pooling internals.
Examples
Uses six short sentences, a 384-dimensional embedding matrix, and cosine-ranked semantic-search results.
What it assumes
Assumes basic embeddings and similarity concepts while grounding the role of Transformer token outputs.
What you can do after
Apply it via Sentence Transformers to convert encoded token sequences into vectors usable for semantic retrieval.
How it connects
Connects BERT, sentence embeddings, L2 normalization, cosine similarity, and semantic search.

Also explained in

Compare how 2 books explain mean pooling →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Build an Advanced RAG Application (From Scratch). The section links point to the parts of the book it draws on.