liveBook · Manning

Attention_mask

As explained in Build AI Drug Discovery Pipelines

An attention mask is a tensor or rule that marks which sequence positions should contribute to attention or aggregation computations.

An attention mask marks real sequence positions so padding tokens do not contribute to mean pooling. For antibody classification, it preserves the true average over amino-acid representations. section 12.3.4

Listing 12.2 Average Token Embeddings
def mean_pooling(self, token_embeddings, attention_mask):
    expanded_mask = attention_mask.unsqueeze(-1).expand(
        token_embeddings.shape
    ).float()    #A
 
    sum_embeddings = torch.sum(
        token_embeddings * expanded_mask, dim=1
    )    #B
    num_tokens = torch.clamp(
        expanded_mask.sum(1), min=1e-9
    )    #C
    return sum_embeddings / num_tokens    #D

The PyTorch implementation expands the mask across embedding dimensions, multiplies token embeddings by it, sums valid positions, clamps the token count, and divides to produce the pooled representation. section 12.3.4

Masked mean pooling

A padded batch carries input IDs and an attention mask. Masked pooling computes a fixed-size sequence vector without allowing padding to dilute the average, supporting a classifier that outputs class logits. section 12.3.4

How Build AI Drug Discovery Pipelines covers it

How deep it goes
Implements expansion, masking, summation, denominator clamping, and division for mean pooling.
Examples
Demonstrates masked average token embeddings for variable-length antibody sequences.
What it assumes
Assumes tokenized, padded sequences and token embeddings before defining how valid positions contribute.
What you can do after
Apply it via PyTorch to compute pooled sequence representations without padding contamination.
How it connects
Connects padding tokens with mean pooling, token embeddings, and sequence classification.

Also explained in

Compare how 3 books explain attention_mask →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Build AI Drug Discovery Pipelines. The section links point to the parts of the book it draws on.