Pooling compresses a sequence of token embeddings into one fixed-sized representation. This makes sentence or passage representations comparable with operations such as cosine similarity. section 3.2.2
Mean pooling averages token vectors, giving each active token equal weight. Max pooling keeps the strongest value in each dimension, while [CLS] uses BERT’s sentence-level token. For sentence and passage similarity, mean pooling performs better than both alternatives. section 3.2.2
Implement mean pooling by following these steps:
- Run the encoder and collect the last hidden state, which contains one contextual vector per token. section 3.2.2
- Expand the attention mask and multiply it with the token embeddings so padding tokens contribute nothing. section 3.2.2
- Sum the masked vectors across the token dimension, count active tokens, and divide the sum by that count. section 3.2.2
Sentence similarity
Encode “The committee approved the proposal yesterday after extensive debate” and “Yesterday, following lengthy discussions, the proposal received committee approval.” Mean-pool each sequence, then compare the resulting vectors with cosine similarity. section 3.2.2
Pooling also serves as a component inside learned evaluation metrics. COMET independently encodes source, candidate, and reference text, pools each sequence into a sentence embedding, and combines those representations for a task-specific prediction. section 3.3.3
Listing 3.4 Encoding text inputs into embeddings using mean pooling .
python
def get_embeddings(text: str):
inputs = tokenizer(text, return_tensors=’pt’, truncation=True, padding=True)
with torch.no_grad():
Outputs = model(**inputs)
token_embeddings = outputs.last_hidden_state #A
attention_mask = encoded["attention_mask"] #B
#C
mask_expanded = attention_mask.unsqueeze(-1).expand(token_embeddings.size()).float()
masked_embeddings = token_embeddings * mask_expanded
#D
sum_embeddings = masked_embeddings.sum(dim=1)
token_counts = mask_expanded.sum(dim=1).clamp(min=1e-9)
return = sum_embeddings / token_counts
Use the implementation listing to inspect attention-mask expansion, masked summation, token counting, and division. section 3.2.2








