liveBook · Manning

Recurrent neural network

As explained in Machines that Think

A recurrent architecture processes sequences by carrying information from earlier sequence elements into later computations.

A recurrent neural network (RNN) processes ordered data one step at a time, carrying context forward rather than treating each input as independent. This makes it suited to sequences such as language, music, and sensor readings. section 3.4.3

At each step, the network combines the current input with a hidden state shaped by earlier inputs, then updates that state and produces an output. The same learned weights are reused at every step, allowing patterns to be processed across a sequence without separate parameters for each position. section 3.4.3

Figure 3.14 A recurrent neural network unrolled across time. Each cell processes one element of the input sequence xt by combining the current input with the network’s previous internal state ht-1. This combination is performed using the same set of learned weights at every step: Un weights the current input, Wn transforms the previous internal state, and Vn maps the resulting state to the output. The internal state is passed forward from one step to the next, allowing information to accumulate across the sequence. Unrolling the network makes this temporal flow explicit and shows how a single computation is reused repeatedly over time.

The unrolled diagram makes the repeated computation visible: each step combines a new input with the previous hidden state, using the same weights. section 3.4.3

This carried state supports short-range context, including local syntax and next-word prediction. But it compresses earlier information into one evolving representation. Signals may fade or grow unstable, and earlier details can become entangled with later context, making long-range dependencies difficult to retain or reinterpret. section 3.4.3 section 3.5.2

A subject carried across a sentence

In “The book on the table near the window was written by a famous author,” the subject must remain available across intervening phrases. An RNN carries it forward in its hidden state, where it may become harder to isolate. section 3.5.2

Recurrent networks established a way to model ordered information, but language also requires relationships to be revised as new words arrive. Even gated variants such as LSTMs and GRUs can struggle with this limitation, motivating architectures that move beyond strictly sequential processing. section 3.4.7 section 3.5.1

How Machines that Think covers it

How deep it goes
Explains hidden-state updates and shared weights, then identifies long-range memory limits without detailing training procedures.
Examples
Uses words, music, sensor readings, next-word prediction, tagging, and a sentence whose subject spans intervening phrases.
What you can do after
Understand how a hidden state carries sequence context and why compression can weaken or entangle earlier information.
How it connects
Connects recurrent networks with dense networks, LSTMs, GRUs, convolutional networks, and language modeling.

Also explained in

Compare how 18 books explain recurrent neural network →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Machines that Think. The section links point to the parts of the book it draws on.