A recurrent neural network (RNN) processes a sequence one token at a time, updating a hidden state from prior states and new inputs. This creates a natural bias toward temporal dependencies. section 3.1.1
RNN-based sequence modeling involves several practical trade-offs:
- Sequential processing prevents parallel token computation during training, making large-corpus training inefficient. section 3.1.1
- Distant-token information becomes difficult to retain because memory decays across many steps. section 3.1.1
- In encoder-decoder translation, the encoder compresses the input into one fixed-size final hidden state for the decoder. section 3.1.1
Translation bottleneck
An encoder reads a sentence word by word and passes its final hidden state to a decoder, which generates the translated sequence one token at a time. Longer sentences can lose important context because earlier encoder states are inaccessible. section 3.1.1
Figure 3.1 Encoder-decoder RNN architecture for language translation. The encoder processes the input sequence sequentially, compressing all information into a single hidden state. The decoder then uses this bottleneck representation to generate the output. A key limitation is that the decoder cannot directly access earlier encoder states. It relies entirely on the compressed final hidden state, which can lead to information loss in longer sequences. Image from Sebastian Raschka’s book Build a Large Language Model (From Scratch) [8].

Inspect the encoder-decoder diagram to trace sequential input processing, hidden-state compression, and decoder generation. section 3.1.1

















