A recurrent neural network (RNN) processes a sequence one element at a time. After each token, it carries forward a hidden state: a vector shaped by what it has read so far. This makes an RNN useful when a prediction depends on earlier words or characters, rather than only a fixed nearby window. The input can also be a sequence of numerical measurements, such as weather or traffic data. section 8.1.3 section 8.1.1
Table 8.1 The many inputs and outputs of RNNs
|
Type
|
Description
|
Applications
|
|---|---|---|
|
One to many
|
One input tensor used to generate a sequence of output tensors
|
Generate chat messages, answer questions, describe images
|
|
Many to one
|
A sequence of input tensors gathered up into a single output tensor
|
Classify or tag text according to its language, intent, or other characteristics
|
|
Many to many
|
A sequence of input tensors used to generate a sequence of output tensors
|
Translate, tag, or anonymize the tokens within a sequence of tokens, answer questions, participate in a conversation
|
The input–output table distinguishes generating a sequence from one input, classifying a whole sequence, and producing an output for each input token. section 8.1
Think of the hidden state as a paint roller picking up traces of each letter it crosses. At each step, the RNN combines the current token’s vector with the preceding hidden state, then computes a new state and an output. Unrolling the loop shows a chain of steps that reuse the same weights. Unlike a feedforward network that transforms a supplied vector in one pass, an RNN must advance through the tokens in order. An output can be taken after the final token or at intermediate steps, as in token tagging. section 8.1.2 section 8.1.3 section 8.1.4
Listing 8.1 Recurrence in PyTorch
>>> from torch import nn >>> class RNN(nn.Module): ... ... def __init__(self, ... vocab_size, hidden_size, output_size): #1 ... super().__init__() ... self.W_c2h = nn.Linear( ... vocab_size + hidden_size, hidden_size) #2 ... self.W_c2y = nn.Linear(vocab_size + hidden_size, output_size) ... self.softmax = nn.LogSoftmax(dim=1) ... ... def forward(self, x, hidden): #3 ... combined = torch.cat((x, hidden), axis=1) #4 ... hidden = self.W_c2h(combined) #5 ... y = self.W_c2y(combined) #6 ... y = self.softmax(y) ... return y, hidden #7
The PyTorch listing shows the recurrent calculation directly: concatenate a token vector and hidden state, apply linear layers, and return both a prediction and the next hidden state. section 8.1.3
Classify a surname character by character
A character-level RNN can read a surname as one-hot character vectors and predict a category after the last character. In the PyTorch example, the model stores its character-to-index mapping and category order alongside its layers; changing either mapping when reusing the model would corrupt the labels. Training starts with a zero hidden state for each sample, runs the characters in order, calculates negative log-likelihood loss, and backpropagates. Inspecting predictions after successive prefixes makes the changing output—and hidden state—visible. section 8.2.1 section 8.2.2 section 8.2.3
Training is called backpropagation through time because an error at the output must travel backward through the unrolled token steps. Reused weights do not make those steps free: each depends on the preceding state, so training can be costly. Gradients may disappear or explode. Hyperparameters such as hidden size, layer count, learning rate, and dropout therefore need testing against validation results, not just training accuracy. section 8.3 section 8.7 section 8.4.4
Check what a prediction means
Surname categories are uncertain even before modeling: a surname can appear under several nationalities, and a person’s name does not establish their nationality. One training run reports about 57% training accuracy but 29% validation accuracy. Ranked model scores and prefix-by-prefix predictions help diagnose behavior; neither makes the classification a reliable judgment about a person. Consider discriminatory uses before deploying such a system. section 8.2 section 8.2.2 section 8.2.3
RNNs can also predict the next word and generate text repeatedly. A word-level example uses WikiText2, where masked <unk> tokens complicate training and generated output can be repetitive or incoherent. The ability to keep reading or generating is not a guarantee of accurate long-range understanding. For comparison, transformers do not recur inside their neurons and can run those inner computations in parallel. section 8.5 section 8.4 section 9.1
