A feedforward network processes each input independently without retaining state. In a Transformer, its feedforward block expands each token representation, applies an activation, and projects it back to the model dimension. section 13.3.1 section 15.3.2
A Transformer feedforward block:
- Uses two dense layers with an activation between them. section 15.3.2
- Updates individual sequence positions independently; attention, rather than the feedforward block, passes information between positions. section 15.3.2 section 16.2.3
- Typically expands the hidden representation before nonlinearity and contracts it back afterward. section 15.3.4
Transformer translation model
With a hidden dimension of 256, a feedforward block can expand each token to 2,048 features, apply a nonlinearity, and return to 256 features. Encoder and decoder blocks use this transformation alongside attention, normalization, and residual connections. section 15.3.4 section 15.3.2
Listing 15.14 A Transformer encoder block
layer_transformer_encoder <- new_layer_class(
"TransformerEncoder",
initialize = function(hidden_dim, intermediate_dim, num_heads) {
super$initialize()
key_dim <- hidden_dim %/% num_heads
self$self_attention <- layer_multi_head_attention( #1
num_heads = num_heads,
key_dim = key_dim
)
self$self_attention_layernorm <- layer_layer_normalization() #1
self$feed_forward_1 <- layer_dense(, intermediate_dim, #2
activation = "relu") #2
self$feed_forward_2 <- layer_dense(, hidden_dim) #2
self$feed_forward_layernorm <- layer_layer_normalization() #2
},
call = function(source, source_mask) {
residual <- x <- source #3
mask <- source_mask@r[, newaxis, ] #3
x <- self$self_attention( #3
query = x, #3
key = x, #3
value = x, #3
attention_mask = mask #3
) #3
x <- x + residual #3
x <- x |> self$self_attention_layernorm() #3
residual <- x #4
x <- x |> #4
self$feed_forward_1() |> #4
self$feed_forward_2() #4
x <- x + residual #4
x <- x |> self$feed_forward_layernorm() #4
x
}
)
Inspect the R implementation of a Transformer encoder block, including its two feedforward dense layers and intermediate dimension. section 15.3.2
