liveBook · Manning

Hidden state

As explained in Machines that Think

An internal recurrent-network representation that carries information from earlier sequence elements to later processing steps.

A hidden state is an internal representation that carries context through a recurrent neural network. As the network reads a sequence one element at a time, it combines the current input with the previous state to produce an updated state and an output. The state acts as a form of memory: it lets later steps reflect earlier inputs rather than treating each element independently. section 3.4.3

Figure 3.14 A recurrent neural network unrolled across time. Each cell processes one element of the input sequence xt by combining the current input with the network’s previous internal state ht-1. This combination is performed using the same set of learned weights at every step: Un weights the current input, Wn transforms the previous internal state, and Vn maps the resulting state to the output. The internal state is passed forward from one step to the next, allowing information to accumulate across the sequence. Unrolling the network makes this temporal flow explicit and shows how a single computation is reused repeatedly over time.

The unrolled network shows how the state moves between steps while the same learned weights are reused. section 3.4.3

This memory has limits. Repeated transformations can weaken or amplify signals, making long-range context difficult to retain. The state also compresses past information into one evolving representation: later inputs blend with earlier signals, which can become entangled and hard to recover. section 3.4.3 section 3.5.2

Carrying a subject through a sentence

In “The book on the table near the window was written by a famous author,” a recurrent model must carry information about the subject across several intervening phrases. That information may become diffuse before it is needed. section 3.5.2

The term also describes contextualized token representations in attention-based models. There, attention combines information from tokens to produce hidden states that later layers refine. Unlike a recurrent state carried along a fixed path, these representations are recomputed at each position through token interactions. section 4.1.3

LSTMs and GRUs extend recurrent memory through gating mechanisms. Attention offers a different approach by allowing models to retrieve relevant information from positions across a sequence. section 3.6

How Machines that Think covers it

How deep it goes
Explains state updates, shared weights and memory limitations, without specifying a full implementation.
Examples
Illustrates information loss across intervening phrases and contextualization of “rises” using information from “sun.”
What you can do after
Evaluate how a model carries or recomputes context and why recurrent memory can become difficult to use over long spans.
How it connects
Connects hidden state to recurrent neural networks, LSTMs, GRUs and self-attention.

Also explained in

Compare how 4 books explain hidden state →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Machines that Think. The section links point to the parts of the book it draws on.