liveBook · Manning

State

As explained in Building Specialized AI Systems

A hidden state is an intermediate vector that stores a sequence model's accumulated representation of preceding inputs.

A hidden state is an intermediate vector that accumulates a sequence model’s representation of preceding inputs. Recurrent architectures update it incrementally from prior states and new tokens, enforcing temporal dependencies. section 3.1.1

In an encoder-decoder sequence model, the hidden state serves two related roles:

  • The encoder updates a hidden state for each input token, with the final state acting as a fixed-size representation of the input. section 3.1.1
  • The decoder maintains its own hidden states while generating the output sequence one token at a time. section 3.1.1

Translation bottleneck

An encoder reads a sentence word by word and compresses it into one final hidden state; the decoder generates the translation without directly accessing earlier encoder states, so long inputs can lose information. section 3.1.1

Figure 3.1 Encoder-decoder RNN architecture for language translation. The encoder processes the input sequence sequentially, compressing all information into a single hidden state. The decoder then uses this bottleneck representation to generate the output. A key limitation is that the decoder cannot directly access earlier encoder states. It relies entirely on the compressed final hidden state, which can lead to information loss in longer sequences. Image from Sebastian Raschka’s book Build a Large Language Model (From Scratch) [8].

Inspect the encoder-decoder diagram to trace sequential hidden-state updates and the fixed-size bottleneck between encoding and decoding. section 3.1.1

How Building Specialized AI Systems covers it

How deep it goes
Explains token-level updates, encoder-decoder roles, and fixed-size compression, but does not derive the underlying neural computations.
Examples
Uses encoder-decoder language translation to show sequential encoding, decoder generation, and information loss for long sequences.
What you can do after
Evaluate how recurrent models preserve context and diagnose information loss caused by compressing long inputs into one final state.
How it connects
Connects hidden states to recurrent neural networks, LSTMs, GRUs, encoder-decoder architectures, and transformer limitations.

Also explained in

Compare how 52 books explain state →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Building Specialized AI Systems. The section links point to the parts of the book it draws on.