liveBook · Manning

Positional embedding

As explained in Build AI Drug Discovery Pipelines

A positional encoding is a vector representation of a token's location that is added to its content embedding to provide sequence-order information.

A positional embedding, or positional encoding, is a vector that gives a token’s location in a sequence. It has the same dimension as the token embedding and is added to it before the first encoder layer. section 12.3.2

Self-attention alone does not know sequence order: shuffling the input shuffles its outputs in the same way. Position information matters for proteins because the order of amino acid residues helps determine the fold. section 12.3.2

One approach uses fixed sine and cosine waves at different frequencies to produce a bounded, deterministic vector for each position. The changing frequencies act like a positional fingerprint. Other models learn the encodings instead. section 12.3.2

Figure 12.10 (Left) A discrete binary counter (black/white), showing how lower bits (towards the left on the x-axis) oscillate rapidly while higher bits (towards the right on the x-axis) oscillate slowly. (Right) The Transformer's sinusoidal encoding (red/blue), showing the exact same frequency gradient but in a continuous, differentiable form, using sine and cosine waves of different frequencies to assign a unique, differentiable vector fingerprint to every position in the sequence.

Compare the changing frequencies of sinusoidal encodings with the bits of a binary counter. section 12.3.2

Predicting a masked amino acid

A protein language model can receive MKTAYIA[MASK]QRQISF and predict the missing amino acid, K. Token embeddings and positional encodings provide the input that self-attention turns into contextual representations. Evaluate masked-token accuracy or pseudo-perplexity on held-out sequences, while checking for near-duplicates across splits. section 12.3.3

Listing 12.1 Protein Language Model using Transformer Encoder
class ProteinLanguageModel(nn.Module):
    def __init__(
        self, vocab_size,
        padding_idx, embedding_dim,
        num_layers, num_heads,
        ffn_dim, dropout,
    ):
        super().__init__()
 
        self.embedding = nn.Embedding(    #A
            vocab_size, embedding_dim, padding_idx    #A
        )    #A
 
        self.pe = PositionalEncoding(embedding_dim)    #B
 
        self.encoder = Encoder(    #C
            num_layers=num_layers,    #C
            embedding_dim=embedding_dim,    #C
            num_heads=num_heads,    #C
            ffn_dim=ffn_dim,    #C
            dropout=dropout,    #C
        )    #C
 
        self.lm_head = nn.Linear(    #D
            embedding_dim, vocab_size    #D
        )    #D
 
    def forward(self, batch):
        input_ids = batch["input_ids"]
        attention_mask = batch["attention_mask"]
        x = self.embedding(input_ids)    #E
        x = self.pe(x)    #F
        x = self.encoder(x, attention_mask)    #G
        logits = self.lm_head(x)    #H
        return logits

The PyTorch model declares a token embedding, a positional encoding module, and a Transformer encoder. section 12.3.3

How Build AI Drug Discovery Pipelines covers it

How deep it goes
Explains why order information is needed and describes fixed sinusoidal vectors, while noting that some encodings are learned.
Examples
Illustrates masked amino acid prediction in a protein sequence and declares a positional encoding module in a PyTorch model.
What you can do after
Evaluate/understand how positional information supports order-sensitive protein representations.
How it connects
Connects positional encoding to self-attention, token embeddings, Transformer encoders, and masked language modeling.

Also explained in

Compare how 20 books explain positional embedding →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Build AI Drug Discovery Pipelines. The section links point to the parts of the book it draws on.