An attention mask marks real sequence positions so padding tokens do not contribute to mean pooling. For antibody classification, it preserves the true average over amino-acid representations. section 12.3.4
Listing 12.2 Average Token Embeddings
def mean_pooling(self, token_embeddings, attention_mask):
expanded_mask = attention_mask.unsqueeze(-1).expand(
token_embeddings.shape
).float() #A
sum_embeddings = torch.sum(
token_embeddings * expanded_mask, dim=1
) #B
num_tokens = torch.clamp(
expanded_mask.sum(1), min=1e-9
) #C
return sum_embeddings / num_tokens #DThe PyTorch implementation expands the mask across embedding dimensions, multiplies token embeddings by it, sums valid positions, clamps the token count, and divides to produce the pooled representation. section 12.3.4
Masked mean pooling
A padded batch carries input IDs and an attention mask. Masked pooling computes a fixed-size sequence vector without allowing padding to dilute the average, supporting a classifier that outputs class logits. section 12.3.4


