A residual connection adds a sublayer’s input to its transformed output, preserving input context while allowing information to flow directly through a deep network. In a transformer encoder, attention and feed-forward sublayers use this pattern before layer normalization. section 2.2.1
For an input x and sublayer function Sublayer, the output is LayerNorm(x + Sublayer(x)). The addition provides a path that can bypass part of the transformation, while dropout is applied before the residual result is finalized. section 2.2.1
A transformer encoder layer applies residual connections around its main components:
- Compute multi-head self-attention, apply dropout, add the original input, and normalize the result. section 2.2.1
- Process the result with a position-wise feed-forward network, apply dropout, add the sublayer input, and normalize again. section 2.2.1
- Stack the resulting layers so essential input details and relationships remain available throughout encoding. section 2.2.1
Listing 2.1 Simplified encoder layer example
class EncoderLayer(nn.Module):
def __init (self, d_model, nhead, dim_feedforward, dropout=0.1):
super(). init ()
self.self_attn = nn.MultiheadAttention
(d_model, nhead, dropout=dropout) #1
self.feed_forward = nn.Sequential( #2
nn.Linear(d_model, 2 * dim_feedforward),
RELU(input_size=dim_feedforward, output_size=d_model),
nn.Dropout(dropout)
)
self.norm1 = LayerNorm(d_model) #3
self.norm2 = LayerNorm(d_model)
self.dropout1 = nn.Dropout(dropout) #4
self.dropout2 = nn.Dropout(dropout)
def forward(self, x, mask=None
# self-attention layer #5
attn_output, _ = self.self_attn(x, x, x, attn_mask=mask)
x = x + self.dropout1(attn_output)
x = self.norm1(x)
# feed-forward layer #6
ff_output = self.feed_forward(x)
x = x + self.dropout2(ff_output)
x = self.norm2(x)
return x
Inspect a simplified encoder-layer implementation showing attention, feed-forward processing, dropout, addition, and layer normalization. section 2.2.1
