liveBook · Manning

Classification head

As explained in Build AI Drug Discovery Pipelines

A classification head is a neural-network component that maps an input representation to scores or probabilities for discrete classes.

A classification head converts one sequence representation into scores for discrete classes. In antibody classification, pooling first replaces per-position language-model outputs with a single vector, which the head then classifies. section 12.3.4

A transformer classifier typically combines:

  • Mean pooling over non-padding token representations, using the attention mask to exclude padding. section 12.3.4
  • A sequential head with linear projection, layer normalization, LeakyReLU activation, and final class logits. section 12.3.4
Listing 12.2 Average Token Embeddings
def mean_pooling(self, token_embeddings, attention_mask):
    expanded_mask = attention_mask.unsqueeze(-1).expand(
        token_embeddings.shape
    ).float()    #A
 
    sum_embeddings = torch.sum(
        token_embeddings * expanded_mask, dim=1
    )    #B
    num_tokens = torch.clamp(
        expanded_mask.sum(1), min=1e-9
    )    #C
    return sum_embeddings / num_tokens    #D

Inspect the mean-pooling implementation, including attention-mask expansion, masked summation, and zero-safe division. section 12.3.4

Antibody sequence classification

A transformer classifier distinguishes HIV-1 from SARS-CoV-2 antibody sequences, reaching 0.7609 test accuracy, 0.8053 AUC, and 0.7580 F1.

How Build AI Drug Discovery Pipelines covers it

How deep it goes
Explains pooling, masking, architecture, loss, training, and evaluation, but defers deeper classifier theory.
Examples
Uses HIV-1 and SARS-CoV-2 antibody sequences, masked mean pooling code, logits, confusion matrices, and confidence thresholds.
What it assumes
Assumes basic neural-network and sequence-model concepts while defining sequence pooling, padding masks, logits, and class probabilities operationally.
What you can do after
Build it from scratch with PyTorch by pooling sequence embeddings, defining a multilayer head, training with cross-entropy, and evaluating predictions.
How it connects
Connects classification heads to transformer encoders, mean pooling, attention masks, protein language models, and cross-entropy loss.

Also explained in

Compare how 3 books explain classification head →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Build AI Drug Discovery Pipelines. The section links point to the parts of the book it draws on.