Suppose you need to translate “I will bring the bag to you” into Spanish. The first translated word, “Te,” depends on “you” at the end of the English sentence. An encoder solves the read-before-writing problem: it processes the source sequence into a representation the decoder can use while generating the target sequence. section 15.2.2
In sequence-to-sequence learning, the encoder reads a fixed source sequence; the decoder predicts target tokens using that representation and the target tokens so far. During training, those earlier target tokens are available. During inference, decoding starts with a seed token and feeds each prediction back until a stop token appears. Translation and question answering both fit this pattern. section 15.2
Figure 15.1 Sequence-to-sequence learning: the source sequence is processed by the encoder and is then sent to the decoder. The decoder looks at the target sequence so far and predicts the target sequence offset by one step in the future. During inference, we generate one target token at a time and feed it back into the decoder.
Follow the two paths separately: the source goes through the encoder once, while the decoder advances through target tokens one prediction at a time. section 15.2
A GRU encoder for translation
A Keras encoder can embed English token IDs and pass them through a bidirectional GRU: source = keras.Input(shape=(None,), dtype="int32", name="english") x = layers.Embedding(vocab_size, embed_dim, mask_zero=True)(source) rnn_layer = layers.Bidirectional(layers.GRU(hidden_dim), merge_mode="sum") encoder_output = rnn_layer(x) Here, `encoder_output` represents the whole source sentence. The decoder GRU receives it as its initial state, rather than starting from zeros. Bidirectionality is useful because the encoder is not predicting source tokens: it can use information from both ends of the sentence. section 15.2.2
An encoder need not always reduce a sequence to one vector. A TransformerEncoder produces a context-aware, order-aware sequence of vectors. In a sequence-to-sequence Transformer, its output goes to a TransformerDecoder; for single-sequence classification, the encoder can instead feed a pooling layer and classifier. Transformer encoders use bidirectional attention, whereas Transformer decoders use causal attention for next-token prediction. section 20.1.6 section 15.6
The same encode-then-use pattern appears beyond translation:
- In an image autoencoder, an encoder maps an image into a latent code that a decoder uses to reconstruct it. A variational autoencoder instead outputs parameters of a latent-space distribution; a sampled point is decoded into an image. section 17.1.2
- In text-to-image diffusion, a pretrained text encoder turns prompts into embeddings that guide denoising. The encoder does not generate the image itself. section 17.3
Listing 17.1 VAE encoder network
import keras from keras import layers latent_dim = 2 #1 image_inputs = keras.Input(shape=(28, 28, 1)) x = layers.Conv2D(32, 3, activation="relu", strides=2, padding="same")( image_inputs ) x = layers.Conv2D(64, 3, activation="relu", strides=2, padding="same")(x) x = layers.Flatten()(x) x = layers.Dense(16, activation="relu")(x) z_mean = layers.Dense(latent_dim, name="z_mean")(x) #2 z_log_var = layers.Dense(latent_dim, name="z_log_var")(x) #2 encoder = keras.Model(image_inputs, [z_mean, z_log_var], name="encoder")
This MNIST VAE encoder uses strided convolutions to downsample a 28 × 28 image, then outputs `z_mean` and `z_log_var` for a two-dimensional latent space. section 17.1.3
For a prompt-conditioned image model, the encoded result can be richer than one vector: the demonstrated Stable Diffusion path passes both a sequence representation and a final vector for each of the positive and negative prompts. Those embeddings can also be interpolated before denoising to produce images between two prompt descriptions. section 17.3.1
Rule of thumb: identify what must be understood first and what consumes that understanding. For translation, encode the source before predicting target tokens. For reconstruction, encode the input before decoding it back. For text-guided diffusion, encode the prompt before denoising. section 15.2 section 17.3
