A classification head converts one sequence representation into scores for discrete classes. In antibody classification, pooling first replaces per-position language-model outputs with a single vector, which the head then classifies. section 12.3.4
A transformer classifier typically combines:
- Mean pooling over non-padding token representations, using the attention mask to exclude padding. section 12.3.4
- A sequential head with linear projection, layer normalization, LeakyReLU activation, and final class logits. section 12.3.4
Listing 12.2 Average Token Embeddings
def mean_pooling(self, token_embeddings, attention_mask):
expanded_mask = attention_mask.unsqueeze(-1).expand(
token_embeddings.shape
).float() #A
sum_embeddings = torch.sum(
token_embeddings * expanded_mask, dim=1
) #B
num_tokens = torch.clamp(
expanded_mask.sum(1), min=1e-9
) #C
return sum_embeddings / num_tokens #DInspect the mean-pooling implementation, including attention-mask expansion, masked summation, and zero-safe division. section 12.3.4
Antibody sequence classification
A transformer classifier distinguishes HIV-1 from SARS-CoV-2 antibody sequences, reaching 0.7609 test accuracy, 0.8053 AUC, and 0.7580 F1.


