A transformer encoder transforms a protein sequence representation by stacking self-attention layers and feed-forward networks. In a protein language model, embeddings receive positional encoding, the encoder processes every sequence position in parallel, and a language-modeling head predicts vocabulary tokens. section 12.3.3
Protein-language-model dimensions
For batch size 4, padded sequence length 50, embedding dimension 128, and vocabulary size 25, the encoder preserves shape: (4, 50, 128) becomes (4, 50, 128). The output head produces (4, 50, 25), or one vocabulary distribution for every position. section 12.3.3
Listing 12.1 Protein Language Model using Transformer Encoder
class ProteinLanguageModel(nn.Module):
def __init__(
self, vocab_size,
padding_idx, embedding_dim,
num_layers, num_heads,
ffn_dim, dropout,
):
super().__init__()
self.embedding = nn.Embedding( #A
vocab_size, embedding_dim, padding_idx #A
) #A
self.pe = PositionalEncoding(embedding_dim) #B
self.encoder = Encoder( #C
num_layers=num_layers, #C
embedding_dim=embedding_dim, #C
num_heads=num_heads, #C
ffn_dim=ffn_dim, #C
dropout=dropout, #C
) #C
self.lm_head = nn.Linear( #D
embedding_dim, vocab_size #D
) #D
def forward(self, batch):
input_ids = batch["input_ids"]
attention_mask = batch["attention_mask"]
x = self.embedding(input_ids) #E
x = self.pe(x) #F
x = self.encoder(x, attention_mask) #G
logits = self.lm_head(x) #H
return logitsInspect the PyTorch model structure connecting token embeddings, positional encoding, an encoder stack, and the language-modeling head. section 12.3.3


