Layer normalization standardizes activations within each sequence independently, avoiding the cross-sample statistics used by BatchNormalization. This makes it suitable for sequence models such as Transformer encoders. section 15.3.2
Per-sequence normalization
For each sequence, compute the mean and variance across its final feature axis, then return (values − mean) / sqrt(variance). Implementations also add a small epsilon for numerical stability and may learn scale and offset parameters. section 15.3.2
Transformer encoder blocks place LayerNormalization alongside self-attention, feedforward layers, and residual connections. Their unchanged input and output shape allows multiple blocks to be stacked. section 15.3.2
Listing 15.14 A Transformer encoder block
layer_transformer_encoder <- new_layer_class(
"TransformerEncoder",
initialize = function(hidden_dim, intermediate_dim, num_heads) {
super$initialize()
key_dim <- hidden_dim %/% num_heads
self$self_attention <- layer_multi_head_attention( #1
num_heads = num_heads,
key_dim = key_dim
)
self$self_attention_layernorm <- layer_layer_normalization() #1
self$feed_forward_1 <- layer_dense(, intermediate_dim, #2
activation = "relu") #2
self$feed_forward_2 <- layer_dense(, hidden_dim) #2
self$feed_forward_layernorm <- layer_layer_normalization() #2
},
call = function(source, source_mask) {
residual <- x <- source #3
mask <- source_mask@r[, newaxis, ] #3
x <- self$self_attention( #3
query = x, #3
key = x, #3
value = x, #3
attention_mask = mask #3
) #3
x <- x + residual #3
x <- x |> self$self_attention_layernorm() #3
residual <- x #4
x <- x |> #4
self$feed_forward_1() |> #4
self$feed_forward_2() #4
x <- x + residual #4
x <- x |> self$feed_forward_layernorm() #4
x
}
)
Inspect the R implementation of a Transformer encoder block, including its LayerNormalization layers. section 15.3.2
