A positional embedding, or positional encoding, is a vector that gives a token’s location in a sequence. It has the same dimension as the token embedding and is added to it before the first encoder layer. section 12.3.2
Self-attention alone does not know sequence order: shuffling the input shuffles its outputs in the same way. Position information matters for proteins because the order of amino acid residues helps determine the fold. section 12.3.2
One approach uses fixed sine and cosine waves at different frequencies to produce a bounded, deterministic vector for each position. The changing frequencies act like a positional fingerprint. Other models learn the encodings instead. section 12.3.2
Figure 12.10 (Left) A discrete binary counter (black/white), showing how lower bits (towards the left on the x-axis) oscillate rapidly while higher bits (towards the right on the x-axis) oscillate slowly. (Right) The Transformer's sinusoidal encoding (red/blue), showing the exact same frequency gradient but in a continuous, differentiable form, using sine and cosine waves of different frequencies to assign a unique, differentiable vector fingerprint to every position in the sequence.

Compare the changing frequencies of sinusoidal encodings with the bits of a binary counter. section 12.3.2
Predicting a masked amino acid
A protein language model can receive MKTAYIA[MASK]QRQISF and predict the missing amino acid, K. Token embeddings and positional encodings provide the input that self-attention turns into contextual representations. Evaluate masked-token accuracy or pseudo-perplexity on held-out sequences, while checking for near-duplicates across splits. section 12.3.3
Listing 12.1 Protein Language Model using Transformer Encoder
class ProteinLanguageModel(nn.Module):
def __init__(
self, vocab_size,
padding_idx, embedding_dim,
num_layers, num_heads,
ffn_dim, dropout,
):
super().__init__()
self.embedding = nn.Embedding( #A
vocab_size, embedding_dim, padding_idx #A
) #A
self.pe = PositionalEncoding(embedding_dim) #B
self.encoder = Encoder( #C
num_layers=num_layers, #C
embedding_dim=embedding_dim, #C
num_heads=num_heads, #C
ffn_dim=ffn_dim, #C
dropout=dropout, #C
) #C
self.lm_head = nn.Linear( #D
embedding_dim, vocab_size #D
) #D
def forward(self, batch):
input_ids = batch["input_ids"]
attention_mask = batch["attention_mask"]
x = self.embedding(input_ids) #E
x = self.pe(x) #F
x = self.encoder(x, attention_mask) #G
logits = self.lm_head(x) #H
return logitsThe PyTorch model declares a token embedding, a positional encoding module, and a Transformer encoder. section 12.3.3



















