A Transformer encoder layer transforms a sequence into a representation with the same shape, allowing multiple blocks to be stacked for progressively richer sentence representations. section 15.3.2
A practical encoder block
Feed the sequence into multi-head self-attention, mask padding tokens, add normalization and a feed-forward network, then preserve the hidden dimension at the output. Layer normalization works within each sequence, making it more suitable for sequence data than batch normalization. section 15.3.2
multi_head_attention(key = source, value = source, query = source)
The self-attention call uses the same source sequence as query, key, and value, making the attention explicitly self-attentive. section 15.3.2
Listing 15.14 A Transformer encoder block
layer_transformer_encoder <- new_layer_class(
"TransformerEncoder",
initialize = function(hidden_dim, intermediate_dim, num_heads) {
super$initialize()
key_dim <- hidden_dim %/% num_heads
self$self_attention <- layer_multi_head_attention( #1
num_heads = num_heads,
key_dim = key_dim
)
self$self_attention_layernorm <- layer_layer_normalization() #1
self$feed_forward_1 <- layer_dense(, intermediate_dim, #2
activation = "relu") #2
self$feed_forward_2 <- layer_dense(, hidden_dim) #2
self$feed_forward_layernorm <- layer_layer_normalization() #2
},
call = function(source, source_mask) {
residual <- x <- source #3
mask <- source_mask@r[, newaxis, ] #3
x <- self$self_attention( #3
query = x, #3
key = x, #3
value = x, #3
attention_mask = mask #3
) #3
x <- x + residual #3
x <- x |> self$self_attention_layernorm() #3
residual <- x #4
x <- x |> #4
self$feed_forward_1() |> #4
self$feed_forward_2() #4
x <- x + residual #4
x <- x |> self$feed_forward_layernorm() #4
x
}
)
The implementation listing shows the block’s main components: multi-head attention, layer normalization, two dense feed-forward layers, and residual state. section 15.3.2
