A feedforward network processes each input independently and keeps no state between inputs. In a Transformer, it gives each token representation its own dense nonlinear update after attention has exchanged information across the sequence. section 13.3.1 section 15.3.2
The two-layer Transformer feedforward block
With a hidden size of 256, expand each token representation to 2,048 features, apply a ReLU, then project it back to 256. The expansion supplies nonlinear expressive capacity without passing information between token positions. section 15.3.4 section 15.3.2
Use the feedforward network to separate two jobs inside a Transformer:
- Let attention pass information across sequence positions. section 15.3.2
- Let the feedforward network transform each individual token representation. section 15.3.2
- Pair the block with residual connections and layer normalization when assembling an encoder or decoder. section 15.3.2
Listing 15.14 A Transformer encoder block
class TransformerEncoder(keras.Layer): def __init__(self, hidden_dim, intermediate_dim, num_heads): super().__init__() key_dim = hidden_dim // num_heads self.self_attention = layers.MultiHeadAttention(num_heads, key_dim)#1 self.self_attention_layernorm = layers.LayerNormalization() #1 self.feed_forward_1 = layers.Dense(intermediate_dim, activation="relu") #2 self.feed_forward_2 = layers.Dense(hidden_dim) #2 self.feed_forward_layernorm = layers.LayerNormalization() def call(self, source, source_mask): residual = x = source #3 mask = source_mask[:, None, :] #3 x = self.self_attention(query=x, key=x, value=x, attention_mask=mask) #3 x = x + residual #3 x = self.self_attention_layernorm(x) #3 residual = x #4#4 x = self.feed_forward_1(x) #4 x = self.feed_forward_2(x) #4 x = x + residual #4 x = self.feed_forward_layernorm(x) #4 return x
Inspect the Keras encoder implementation to see the two Dense layers, activation, normalization, and residual structure together. section 15.3.2
