A feed-forward network (FFN) transforms fixed-length vectors into more abstract representations. In a transformer encoder layer, it follows multihead self-attention: attention processes relationships in the sequence, then the FFN transforms the resulting representation. section 2.2.1 section 2.2.4
A position-wise FFN applies the same transformation independently at each sequence position. Its first fully connected layer maps a vector to a hidden dimension; a nonlinear activation follows, and a second layer returns it to the input dimension. Keeping that output dimension makes the component easier to stack with other transformer layers. section 2.2.4
A small PyTorch FFN
This module uses a ReLU activation and dropout between its two linear layers. It transforms the last dimension of its input while preserving that dimension at the output. ```python import torch from torch import nn class PositionwiseFeedforward(nn.Module): def __init__(self, input_dim, hidden_dim, dropout=0.1): super().__init__() self.fc1 = nn.Linear(input_dim, hidden_dim) self.dropout = nn.Dropout(dropout) self.fc2 = nn.Linear(hidden_dim, input_dim) def forward(self, x): return self.fc2(self.dropout(torch.relu(self.fc1(x)))) ``` section 2.2.4
Listing 2.1 Simplified encoder layer example
class EncoderLayer(nn.Module):
def __init (self, d_model, nhead, dim_feedforward, dropout=0.1):
super(). init ()
self.self_attn = nn.MultiheadAttention
(d_model, nhead, dropout=dropout) #1
self.feed_forward = nn.Sequential( #2
nn.Linear(d_model, 2 * dim_feedforward),
RELU(input_size=dim_feedforward, output_size=d_model),
nn.Dropout(dropout)
)
self.norm1 = LayerNorm(d_model) #3
self.norm2 = LayerNorm(d_model)
self.dropout1 = nn.Dropout(dropout) #4
self.dropout2 = nn.Dropout(dropout)
def forward(self, x, mask=None
# self-attention layer #5
attn_output, _ = self.self_attn(x, x, x, attn_mask=mask)
x = x + self.dropout1(attn_output)
x = self.norm1(x)
# feed-forward layer #6
ff_output = self.feed_forward(x)
x = x + self.dropout2(ff_output)
x = self.norm2(x)
return x
The simplified encoder-layer listing places the FFN after self-attention and shows the surrounding residual, dropout, and normalization operations. section 2.2.1
