liveBook · Manning

Recurrent neural network

As explained in Timeless Algorithms, The Foundational Ideas

A neural network that processes sequential inputs using a state carried across sequence positions.

A recurrent neural network (RNN) processes ordered inputs one step at a time. At each step, it combines the current input with a hidden state carrying information from earlier steps. That state supplies context for later predictions. section 12.1.1

The hidden state has fixed size, so each update must retain, modify, or discard information. As sequences grow, distant relationships can weaken; for example, the sentence “The book that the professor who the students admired assigned was difficult to understand” requires preserving the link between “book” and “was.” Computation is sequential too: each state depends on the preceding one, limiting parallel training. section 12.1.1 section 12.0

Long short-term memory and gated recurrent units were developed to preserve information more effectively, but the basic state-propagation constraint remains. Transformers offer another approach: self-attention models relationships among tokens directly, with positional encodings preserving order. This architectural contrast does not make RNNs universally obsolete, and an RNN is distinct from Bellman’s recurrence relations for sequential optimization. section 12.1.1 section 12.3.2

How Timeless Algorithms, The Foundational Ideas covers it

How deep it goes
Explains hidden-state updates, fixed-size compression, long-range information loss, and sequential computation, while stopping short of implementation details.
Examples
Uses a nested-clause sentence to illustrate how a distant subject–verb relationship can be lost.
What you can do after
Evaluate recurrent models against transformers by tracing how context propagation affects distant dependencies and parallel training.
How it connects
Connects sequence modeling to hidden states, long-range dependencies, LSTM, GRU, transformers, self-attention, and positional encodings.

Also explained in

Compare how 18 books explain recurrent neural network →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Timeless Algorithms, The Foundational Ideas. The section links point to the parts of the book it draws on.