Recurrence

As explained in Natural Language Processing in Action, Second Edition

A recurrence loop is a computation pattern that repeatedly feeds a step's state or output into the next step of sequence processing.

Recurrence repeatedly reuses a computation’s state or output while processing a sequence. In an RNN, each token updates a hidden state that is carried to the next token, allowing the model to use information from the preceding sequence rather than only a fixed local window. section 8.0 section 8.1.3

A recurrence loop works through a small set of practical steps:

• Read one token and combine its vector with the previous hidden state. section 8.1.3
• Transform the combined representation into a new hidden state and an output or prediction. section 8.1.3
• Pass the new hidden state into the same computation for the next token. section 8.1.3
• Provide an exit or full-stop token when generation must end; otherwise, the loop can continue indefinitely. section 8.1.3

Minimal recurrent computation

A minimal PyTorch RNN stores the hidden state separately, concatenates it with the current input, applies learned transformations, and returns both the prediction and updated state: `combined = torch.cat((x, hidden), axis=1)`, followed by `return y, hidden`. This implements recurrence without a recursive function call or PyTorch’s RNNBase class. section 8.1.3

Unrolling recurrence turns the loop into a sequential chain of computations. Unlike recurrence inside RNNs, LSTMs, and GRUs, autoregressive transformers recycle a final output into the next input without recurrent neurons; their internal layers can run in parallel. section 9.1 section 8.1.3

Figure 8.6 A neuron with recurrence

Use the recurrence-loop figure to visualize how a neuron passes its hidden state to its future self. section 8.1.3

Listing 8.1 Recurrence in PyTorch

>>> from torch import nn

>>> class RNN(nn.Module):
...
...     def __init__(self,
...             vocab_size, hidden_size, output_size):     #1
...         super().__init__()
...         self.W_c2h = nn.Linear(
...             vocab_size + hidden_size, hidden_size)     #2
...         self.W_c2y = nn.Linear(vocab_size + hidden_size, output_size)
...         self.softmax = nn.LogSoftmax(dim=1)
...
...     def forward(self, x, hidden):                      #3
...         combined = torch.cat((x, hidden), axis=1)   #4
...         hidden = self.W_c2h(combined)            #5
...         y = self.W_c2y(combined)         #6
...         y = self.softmax(y)
...         return y, hidden    #7

Use the PyTorch listing to inspect a from-scratch recurrence implementation with input-state combination, state updates, output prediction, and return values. section 8.1.3

How Natural Language Processing in Action, Second Edition covers it

How deep it goes

Explains recurrence through unrolling, hidden-state flow, termination, and a minimal PyTorch implementation, while contrasting transformer recursion.

Examples

Uses token-by-token RNN processing, a recurrence-loop diagram, and a minimal PyTorch RNN returning predictions and hidden states.

What you can do after

Build a minimal recurrent neural network in PyTorch that carries hidden state across tokens and produces sequential predictions.

How it connects

Connects recurrence to recurrent neural networks, hidden states, unrolling, LSTMs, GRUs, transformers, and autoregressive generation.

liveBookmanning.com

Read Natural Language Processing in Action, Second Edition on liveBook Get the book at manning.com

Also explained in

• 

  liveBookmanning.com
  Sutskever's ListRecurrence is an architectural mechanism that performs repeated state or representation updates across sequential or iterative computation steps.
• 

  liveBookmanning.com
  Advanced Algorithms and Data StructuresA recurrence relation is an equation that defines a sequence or function in terms of values for preceding or smaller instances.

Compare how 3 books explain recurrence →

Related concepts

• transformer architecturecontrasts with · A transformer architecture is a neural-network architecture that processes sequences or other structured inputs using attention mechanisms and related transformations.
• base caseconfigured by · A base case is a condition in a recursive or inductive computation that can be solved directly without further expansion.
• recurrent neural networkuses · A recurrent neural network is a neural network that processes sequences by updating an internal state across successive inputs, allowing earlier inputs to influence later computations.
• dynamic programmingused by · Dynamic programming is an algorithmic technique that solves problems by decomposing them into overlapping subproblems and storing or reusing their solutions.
• knapsack problemuses · The knapsack problem is an optimization problem that selects items subject to a capacity constraint to maximize their combined value.

This explainer was generated by Manning's knowledge graph from the text of Natural Language Processing in Action, Second Edition. The section links point to the parts of the book it draws on.