liveBook · Manning

Transformer encoder

As explained in Build AI Drug Discovery Pipelines

A transformer encoder is a stack of transformer layers that transforms a sequence representation using attention and feed-forward computations.

A transformer encoder transforms a protein sequence representation by stacking self-attention layers and feed-forward networks. In a protein language model, embeddings receive positional encoding, the encoder processes every sequence position in parallel, and a language-modeling head predicts vocabulary tokens. section 12.3.3

Protein-language-model dimensions

For batch size 4, padded sequence length 50, embedding dimension 128, and vocabulary size 25, the encoder preserves shape: (4, 50, 128) becomes (4, 50, 128). The output head produces (4, 50, 25), or one vocabulary distribution for every position. section 12.3.3

Listing 12.1 Protein Language Model using Transformer Encoder
class ProteinLanguageModel(nn.Module):
    def __init__(
        self, vocab_size,
        padding_idx, embedding_dim,
        num_layers, num_heads,
        ffn_dim, dropout,
    ):
        super().__init__()
 
        self.embedding = nn.Embedding(    #A
            vocab_size, embedding_dim, padding_idx    #A
        )    #A
 
        self.pe = PositionalEncoding(embedding_dim)    #B
 
        self.encoder = Encoder(    #C
            num_layers=num_layers,    #C
            embedding_dim=embedding_dim,    #C
            num_heads=num_heads,    #C
            ffn_dim=ffn_dim,    #C
            dropout=dropout,    #C
        )    #C
 
        self.lm_head = nn.Linear(    #D
            embedding_dim, vocab_size    #D
        )    #D
 
    def forward(self, batch):
        input_ids = batch["input_ids"]
        attention_mask = batch["attention_mask"]
        x = self.embedding(input_ids)    #E
        x = self.pe(x)    #F
        x = self.encoder(x, attention_mask)    #G
        logits = self.lm_head(x)    #H
        return logits

Inspect the PyTorch model structure connecting token embeddings, positional encoding, an encoder stack, and the language-modeling head. section 12.3.3

How Build AI Drug Discovery Pipelines covers it

How deep it goes
Explains embeddings, positional encoding, stacked attention and feed-forward layers, tensor shapes, and parallel processing; helper internals are deferred.
Examples
Uses padded protein batches, amino-acid vocabulary predictions, concrete tensor dimensions, and attention heatmaps over a 37-residue sequence.
What you can do after
Build it from scratch as a PyTorch sequence model with embeddings, positional encoding, an encoder stack, and an output projection.
How it connects
Connects positional encoding, self-attention, feed-forward networks, embeddings, and language-modeling heads.

Also explained in

Compare how 3 books explain transformer encoder →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Build AI Drug Discovery Pipelines. The section links point to the parts of the book it draws on.