An encoder turns an input into an intermediate representation that another part of a model can use. In sequence-to-sequence translation, it processes the source sentence; a decoder uses that representation and the target words so far to predict the next word. section 15.2
Why read the whole sentence first?
Consider “I will bring the bag to you.” Its Spanish translation begins “Te,” which corresponds to “you” at the end of the English sentence. An RNN that predicts each translated word before seeing the rest of the source cannot make that first prediction reliably. An encoder RNN instead reads the entire source and passes a representation—such as its final state—to the decoder. section 15.2.2
Listing 15.11 Building a sequence-to-sequence encoder
embed_dim <- 256 hidden_dim <- 1024 source <- keras_input(shape = c(NA), dtype = "int32", name = "english") encoder_output <- source |> layer_embedding(vocab_size, embed_dim, mask_zero = TRUE) |> bidirectional(layer_gru(units = hidden_dim), merge_mode = "sum")
This R/Keras encoder embeds source tokens, masks padding tokens, and runs a bidirectional GRU. Its output represents the source sequence and can initialize the decoder GRU. section 15.2.2
During training, the decoder predicts the next target token from earlier target tokens and the encoded source. During inference, it starts with a seed token, feeds each prediction back in, and stops at an end token. A Transformer encoder takes another approach: it uses bidirectional attention to build a rich sequence representation. section 15.2 section 15.6
Encoding images instead of sentences
An image autoencoder encodes an image into a latent representation and decodes it to reconstruct the input. A variational autoencoder (VAE) changes what the encoder produces: rather than one fixed code, it outputs parameters for a distribution from which a latent point is sampled. A VAE for MNIST digits uses a convolutional encoder to produce two vectors, z_mean and z_log_var. section 17.1.2 section 17.1.3
Listing 17.1 VAE encoder network
latent_dim <- 2 #1
encoder_inputs <- keras_input(shape = c(28, 28, 1))
x <- encoder_inputs |>
layer_conv_2d(32, 3, activation = "relu",
strides = 2, padding = "same") |>
layer_conv_2d(64, 3, activation = "relu",
strides = 2, padding = "same") |>
layer_flatten() |>
layer_dense(16, activation = "relu")
z_mean <- x |> layer_dense(latent_dim, name="z_mean") #2
z_log_var <- x |> layer_dense(latent_dim, name="z_log_var")
encoder <- keras_model(encoder_inputs, list(z_mean, z_log_var),
name="encoder")
The R/Keras VAE encoder downsamples 28 × 28 digit images with strided convolutions, then branches into z_mean and z_log_var. Strides preserve information about where features occur, which matters when reconstructing an image. section 17.1.3
