liveBook · Manning

Hidden state

As explained in Timeless Algorithms, The Foundational Ideas

A hidden state is an internal representation that a recurrent neural network carries forward to summarize information from earlier sequence elements.

A recurrent neural network combines each current input with an evolving hidden state: a fixed-size internal summary of earlier sequence elements. Later predictions can use this stored context, not just the current input. section 12.1.1

At each update, the network must retain, modify, or discard information. Long-distance relationships can weaken when relevant context fails to survive successive updates; for example, the subject “book” must remain connected to “was” across intervening clauses. State updates also depend on preceding states, so the sequence is processed in order. LSTMs and GRUs preserve information more effectively, but retain recurrent state propagation. Transformers instead use self-attention to model token relationships directly, with positional encodings preserving order. section 12.1.1 section 12.3.2

How Timeless Algorithms, The Foundational Ideas covers it

How deep it goes
Explains fixed-size state updates, information loss, and sequential computation at the architectural level.
Examples
Uses a sentence where “book” must remain linked to “was” across intervening clauses.
What you can do after
Evaluate how recurrent state updates can weaken distant context and constrain parallel sequence processing.
How it connects
Connects recurrent neural networks, LSTMs, GRUs, transformers, self-attention, and positional encodings.

Also explained in

Compare how 4 books explain hidden state →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Timeless Algorithms, The Foundational Ideas. The section links point to the parts of the book it draws on.