liveBook · Manning

Pooling

As explained in LLM Evaluation and Alignment, The Foundational Ideas

A pooling layer compresses a sequence of token representations into a fixed-sized representation.

Pooling compresses a sequence of token embeddings into one fixed-sized representation. This makes sentence or passage representations comparable with operations such as cosine similarity. section 3.2.2

Mean pooling averages token vectors, giving each active token equal weight. Max pooling keeps the strongest value in each dimension, while [CLS] uses BERT’s sentence-level token. For sentence and passage similarity, mean pooling performs better than both alternatives. section 3.2.2

Implement mean pooling by following these steps:

  • Run the encoder and collect the last hidden state, which contains one contextual vector per token. section 3.2.2
  • Expand the attention mask and multiply it with the token embeddings so padding tokens contribute nothing. section 3.2.2
  • Sum the masked vectors across the token dimension, count active tokens, and divide the sum by that count. section 3.2.2

Sentence similarity

Encode “The committee approved the proposal yesterday after extensive debate” and “Yesterday, following lengthy discussions, the proposal received committee approval.” Mean-pool each sequence, then compare the resulting vectors with cosine similarity. section 3.2.2

Pooling also serves as a component inside learned evaluation metrics. COMET independently encodes source, candidate, and reference text, pools each sequence into a sentence embedding, and combines those representations for a task-specific prediction. section 3.3.3

Listing 3.4 Encoding text inputs into embeddings using mean pooling .
python
def get_embeddings(text: str):
    inputs = tokenizer(text, return_tensors=’pt’, truncation=True, padding=True)
    with torch.no_grad():
        Outputs = model(**inputs)
    
    token_embeddings = outputs.last_hidden_state                          #A
    attention_mask = encoded["attention_mask"]                            #B
 
    #C
    mask_expanded = attention_mask.unsqueeze(-1).expand(token_embeddings.size()).float()
    masked_embeddings = token_embeddings * mask_expanded
 
    #D
    sum_embeddings = masked_embeddings.sum(dim=1)
    token_counts = mask_expanded.sum(dim=1).clamp(min=1e-9)
 
    return = sum_embeddings / token_counts

Use the implementation listing to inspect attention-mask expansion, masked summation, token counting, and division. section 3.2.2

How LLM Evaluation and Alignment, The Foundational Ideas covers it

How deep it goes
Defines mean, max, and CLS pooling, then works through masking, summation, token counting, and normalization.
Examples
Uses paired committee-approval sentences, masked BERT embeddings, cosine similarity, and COMET source-candidate-reference representations.
What it assumes
Assumes tokenization and transformer outputs, while introducing pooling and cosine similarity through BERT examples.
What you can do after
Build fixed-sized text embeddings with mean pooling and compare them using cosine similarity.
How it connects
Connects token embeddings with attention masks, cosine similarity, sentence embeddings, retrieval-augmented generation, and COMET.

Also explained in

Compare how 9 books explain pooling →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of LLM Evaluation and Alignment, The Foundational Ideas. The section links point to the parts of the book it draws on.