liveBook · Manning

Recurrent neural network

As explained in Building Specialized AI Systems

A recurrent neural network is a sequence model that processes inputs sequentially while carrying a hidden state between positions.

A recurrent neural network (RNN) processes a sequence one token at a time, updating a hidden state from prior states and new inputs. This creates a natural bias toward temporal dependencies. section 3.1.1

RNN-based sequence modeling involves several practical trade-offs:

  • Sequential processing prevents parallel token computation during training, making large-corpus training inefficient. section 3.1.1
  • Distant-token information becomes difficult to retain because memory decays across many steps. section 3.1.1
  • In encoder-decoder translation, the encoder compresses the input into one fixed-size final hidden state for the decoder. section 3.1.1

Translation bottleneck

An encoder reads a sentence word by word and passes its final hidden state to a decoder, which generates the translated sequence one token at a time. Longer sentences can lose important context because earlier encoder states are inaccessible. section 3.1.1

Figure 3.1 Encoder-decoder RNN architecture for language translation. The encoder processes the input sequence sequentially, compressing all information into a single hidden state. The decoder then uses this bottleneck representation to generate the output. A key limitation is that the decoder cannot directly access earlier encoder states. It relies entirely on the compressed final hidden state, which can lead to information loss in longer sequences. Image from Sebastian Raschka’s book Build a Large Language Model (From Scratch) [8].

Inspect the encoder-decoder diagram to trace sequential input processing, hidden-state compression, and decoder generation. section 3.1.1

How Building Specialized AI Systems covers it

How deep it goes
Explains hidden-state updates and encoder-decoder compression, then stops at limitations without detailing training algorithms or cell equations.
Examples
Uses language translation to demonstrate token-by-token encoding, fixed-size memory, and decoder generation.
What you can do after
Evaluate sequential sequence models against transformers using parallelism, long-range memory, and representation-bottleneck trade-offs.
How it connects
Connects RNNs with LSTMs, GRUs, encoder-decoder models, convolutional models, and transformers.

Also explained in

Compare how 18 books explain recurrent neural network →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Building Specialized AI Systems. The section links point to the parts of the book it draws on.