An encoder turns an input sequence into a representation that another component can use. In sequence-to-sequence translation, the decoder uses that representation to produce an output sentence. The encoder–decoder division remains in transformers, but the way the encoder processes its input changes substantially. section 1.1.1 section 2.3
For “I don’t speak French” → “Je ne parle pas français,” an RNN encoder reads the English words one at a time and updates its recurrent state. Its final context vector bridges the input and decoder. This sequential processing can struggle with long-distance relationships. A transformer encoder instead uses attention to relate parts of the input while processing the sequence concurrently. section 1.1.1 section 1.1.2
A transformer encoder is a stack of layers. In the illustrated architecture, six identical layers each apply:
- Multihead self-attention, which weighs relationships among positions in the input sequence. section 2.2.1
- A position-wise feed-forward network, which transforms the attention-processed representation. section 2.2.1
- Dropout, residual connections, and layer normalization around the sublayers. The residual path carries the sublayer input forward alongside its transformed output. section 2.2.1
Tracing one encoder layer
Start with input x. Apply multihead attention with x supplied as its queries, keys, and values; the simplified layer also accepts an optional mask. Add the attention output back to x, apply dropout and layer normalization, then pass the result through the feed-forward network. Repeat the dropout, residual-addition, and normalization pattern for that second sublayer. Positional encoding supplies information about token order, which attention alone does not express in this description. section 2.2.1 section 2.3
Listing 2.1 Simplified encoder layer example
class EncoderLayer(nn.Module):
def __init (self, d_model, nhead, dim_feedforward, dropout=0.1):
super(). init ()
self.self_attn = nn.MultiheadAttention
(d_model, nhead, dropout=dropout) #1
self.feed_forward = nn.Sequential( #2
nn.Linear(d_model, 2 * dim_feedforward),
RELU(input_size=dim_feedforward, output_size=d_model),
nn.Dropout(dropout)
)
self.norm1 = LayerNorm(d_model) #3
self.norm2 = LayerNorm(d_model)
self.dropout1 = nn.Dropout(dropout) #4
self.dropout2 = nn.Dropout(dropout)
def forward(self, x, mask=None
# self-attention layer #5
attn_output, _ = self.self_attn(x, x, x, attn_mask=mask)
x = x + self.dropout1(attn_output)
x = self.norm1(x)
# feed-forward layer #6
ff_output = self.feed_forward(x)
x = x + self.dropout2(ff_output)
x = self.norm2(x)
return x
The simplified encoder-layer listing shows where multihead attention, the feed-forward network, dropout, and two normalization steps fit in a PyTorch-style module. section 2.2.1
The same input-to-representation role extends beyond text. Vision encoders map pixels to features; audio encoders represent waveforms or spectrograms; text encoders represent token sequences. Multimodal systems then align these outputs in a shared embedding space. A converter can map nontext features into an LLM’s token space, while a perceiver can turn encoded features into multimodal tokens for deeper interaction at greater architectural complexity. section 6.2 section 6.3
