A hidden state is an intermediate vector that accumulates a sequence model’s representation of preceding inputs. Recurrent architectures update it incrementally from prior states and new tokens, enforcing temporal dependencies. section 3.1.1
In an encoder-decoder sequence model, the hidden state serves two related roles:
- The encoder updates a hidden state for each input token, with the final state acting as a fixed-size representation of the input. section 3.1.1
- The decoder maintains its own hidden states while generating the output sequence one token at a time. section 3.1.1
Translation bottleneck
An encoder reads a sentence word by word and compresses it into one final hidden state; the decoder generates the translation without directly accessing earlier encoder states, so long inputs can lose information. section 3.1.1
Figure 3.1 Encoder-decoder RNN architecture for language translation. The encoder processes the input sequence sequentially, compressing all information into a single hidden state. The decoder then uses this bottleneck representation to generate the output. A key limitation is that the decoder cannot directly access earlier encoder states. It relies entirely on the compressed final hidden state, which can lead to information loss in longer sequences. Image from Sebastian Raschka’s book Build a Large Language Model (From Scratch) [8].

Inspect the encoder-decoder diagram to trace sequential hidden-state updates and the fixed-size bottleneck between encoding and decoding. section 3.1.1



















































