liveBook · Manning

Mean pooling

As explained in Build AI Drug Discovery Pipelines

Mean pooling forms one representation by averaging the representations of sequence positions.

Mean pooling forms one fixed-size representation by averaging the vectors produced for sequence or graph positions. In molecular graphs, it converts atom-level features into one molecule-level vector, regardless of atom count. In protein classifiers, it converts token-level representations into one sequence vector for classification. section 11.5.2 section 12.3.4

The operation is simple, but its implementation must preserve which positions contain real data:

  • Graph batches use a batch vector so the mean is computed separately for each molecule rather than across every node in the mini-batch. section 11.5.2
  • Sequence batches use an attention mask to exclude padding tokens, preventing meaningless values from diluting the average. section 12.3.4
  • Mean pooling averages all positions equally; unlike sum pooling it is automatically size-normalized, and unlike max pooling it is less sensitive to outliers. section 11.5.2 section 12.3.4

Masked sequence pooling

For token embeddings shaped by batch and sequence position, expand the attention mask, multiply it into the embeddings, sum across positions, and divide by the masked token count. Clamping the count prevents division by zero. The resulting fixed-size vector can feed a classifier for antibody sequences. section 12.3.4

x = global_mean_pool(x, mol_batch)

Use the compact PyTorch Geometric operation to convert node features into one representation per molecular graph. section 11.5.2

Listing 12.2 Average Token Embeddings
def mean_pooling(self, token_embeddings, attention_mask):
    expanded_mask = attention_mask.unsqueeze(-1).expand(
        token_embeddings.shape
    ).float()    #A
 
    sum_embeddings = torch.sum(
        token_embeddings * expanded_mask, dim=1
    )    #B
    num_tokens = torch.clamp(
        expanded_mask.sum(1), min=1e-9
    )    #C
    return sum_embeddings / num_tokens    #D

Inspect the complete masked PyTorch implementation for averaging variable-length sequence representations while ignoring padding. section 12.3.4

How Build AI Drug Discovery Pipelines covers it

How deep it goes
Explains aggregation choices, batching, masking, normalization, and zero-division protection, but does not derive pooling theory.
Examples
Demonstrates molecular graphs, batched atom features, padded protein sequences, and antibody classification.
What it assumes
Assumes basic vectors and neural-network representations while teaching graph nodes, sequence positions, batching, and attention masks.
What you can do after
Apply it via PyTorch or PyTorch Geometric to create fixed-size representations while excluding padded sequence positions.
How it connects
Connects graph neural networks, transformers, attention masks, max pooling, sum pooling, and classification heads.

Also explained in

Compare how 2 books explain mean pooling →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Build AI Drug Discovery Pipelines. The section links point to the parts of the book it draws on.