An encoder-only transformer stacks identical encoder layers to convert an input sequence into abstract, continuous vector representations. Unlike decoder-only transformers, it is designed to understand inputs rather than generate text token by token. BERT is a representative example. section 2.1.2
Trace an encoder-only transformer from input preparation to prediction:
- Represent the input as tokens, calculate token embeddings and positional encodings, and add them to form the encoder input. section 3.1.2
- Pass the sequence through repeated encoder blocks, each combining multi-head self-attention with a position-wise feed-forward network. section 3.3.4
- Apply layer normalization and residual connections around both sublayers; residual additions help address vanishing gradients in deep networks. section 3.3.4
- Use the resulting representations for a task-specific prediction, such as classifying an image from the representation of its class token. section 3.1.2
Vision Transformer classification
For CIFAR-10, divide each 32 × 32 color image into an 8 × 8 grid of 4 × 4 patches. Convert the 64 patches into 48-value embeddings, prepend a class token, add positional encodings, and process the resulting 65-element sequence with an encoder-only transformer. Compare the predicted class with the label using cross-entropy loss and update the parameters. section 3.1.2
Listing 3.5 Creating an encoder block
class Block(nn.Module):
def __init__(self, config):
super().__init__()
self.attention = MultiHeadAttention(config)
self.layernorm_1 = nn.LayerNorm(config.hidden_size)
self.mlp = MLP(config)
self.layernorm_2 = nn.LayerNorm(config.hidden_size)
def forward(self, x, output_attentions=False):
attention_output, attention_probs = \
self.attention(self.layernorm_1(x), #1
output_attentions=output_attentions)
x = x + attention_output #2
mlp_output = self.mlp(self.layernorm_2(x)) #3
x = x + mlp_output #4
if not output_attentions:
return (x, None)
else:
return (x, attention_probs)
Inspect the encoder block implementation, including normalized attention, residual additions, the MLP path, and optional attention outputs. section 3.3.4
Listing 3.6 Creating an encoder-only transformer
class Encoder(nn.Module):
def __init__(self, config):
super().__init__()
self.blocks = nn.ModuleList([])
for _ in range(config.num_hidden_layers):
block = Block(config)
self.blocks.append(block) #1
def forward(self, x, output_attentions=False):
all_attentions = []
for block in self.blocks:
x, attention_probs = block(x,
output_attentions=output_attentions) #2
if output_attentions:
all_attentions.append(attention_probs)
if not output_attentions:
return (x, None)
else:
return (x, all_attentions) #3
Inspect the encoder implementation, which constructs repeated blocks and optionally collects attention probabilities from every layer. section 3.3.4
