liveBook · Manning

Recurrent neural network

As explained in Build an Advanced RAG Application (From Scratch)

A recurrent neural network is a neural model that processes sequence elements iteratively while maintaining a state representing prior inputs.

A recurrent neural network processes sequence elements one at a time while maintaining a hidden state that carries information from earlier inputs. This sequential design supports sequence modeling but limits parallel processing and can weaken context over long sequences. section 2.1 section 2.5

A recurrent neural network operates through a repeated sequence of steps:

  • Reads one sequence element at a time rather than processing the entire sequence in parallel. section 2.1
  • Updates a hidden state that carries information from previous elements into later processing. section 2.1
  • May struggle with long-range dependencies and experience vanishing or exploding gradients during training. section 2.1

Processing a sentence

For “The quick brown fox jumps over the lazy dog,” an RNN processes each word sequentially and carries earlier information through its hidden state. Compared with a Transformer, it has less direct access to distant word relationships. section 2.1

Figure 2.1 The strength of the transformer model is in its ability to understand the significance and context of every word in a sentence, unlike RNNs which primarily assess the connection of a word with its immediate neighbor.

Compare recurrent processing with self-attention: the figure emphasizes that RNNs primarily assess connections with immediate neighbors, while Transformers evaluate broader word significance and context. section 2.1

How Build an Advanced RAG Application (From Scratch) covers it

How deep it goes
Explains sequential hidden-state processing and training limitations, but stops short of parameter updates, equations, or implementation details.
Examples
Uses the sentence “The quick brown fox jumps over the lazy dog” to contrast sequential recurrence with Transformer processing.
What you can do after
Evaluate/understand why sequential recurrent models struggle with parallelization, distant dependencies, and unstable gradients.
How it connects
Connects recurrent neural networks to GRUs, Transformers, self-attention, parallel processing, and long-range dependencies.

Also explained in

Compare how 18 books explain recurrent neural network →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Build an Advanced RAG Application (From Scratch). The section links point to the parts of the book it draws on.