A recurrent neural network reads a sequence one token at a time. Its hidden state is a numerical vector passed from one step to the next: a running representation of what the network has processed so far. Unlike a feedforward layer that transforms an input vector once, the same recurrent computation uses the previous state alongside each new token. section 8.1.3
Think of the state as a relay baton that changes hands—and changes contents—at every token. An RNN can produce an output after any step, which makes token-level predictions possible. The vector is not a readable transcript or a guarantee that every earlier detail survives; inspect its values and predictions to see what a particular model retains. section 8.1.4 section 8.2.3
In a small PyTorch RNN, state moves through the sequence as follows:
- Choose the hidden-vector size and initialize a state before a new text; an all-zero vector is one starting point. section 8.2.1 section 8.3.1
- For each token, concatenate its input vector with the current hidden vector. Compute a new hidden vector and an output, then pass the new vector to the next step. section 8.1.3
- During training, propagate prediction error backward through the token steps. Each step depends on the preceding state, so this work cannot be collapsed into one independent calculation per token. section 8.3
Listing 8.1 Recurrence in PyTorch
>>> from torch import nn >>> class RNN(nn.Module): ... ... def __init__(self, ... vocab_size, hidden_size, output_size): #1 ... super().__init__() ... self.W_c2h = nn.Linear( ... vocab_size + hidden_size, hidden_size) #2 ... self.W_c2y = nn.Linear(vocab_size + hidden_size, output_size) ... self.softmax = nn.LogSoftmax(dim=1) ... ... def forward(self, x, hidden): #3 ... combined = torch.cat((x, hidden), axis=1) #4 ... hidden = self.W_c2h(combined) #5 ... y = self.W_c2y(combined) #6 ... y = self.softmax(y) ... return y, hidden #7
The forward method shows the handoff directly: it accepts `x` and `hidden`, concatenates them, and returns both an output and the next `hidden`. The output in this implementation is calculated from the same concatenated input, not from the newly computed hidden vector. section 8.1.3
Watch a prediction change
For a character-by-character surname classifier, reset the hidden layer and run successively longer prefixes through the model. A `predict_hidden` helper exposes the 128-dimensional vector after each prefix. For “Khalid,” the displayed category changes as characters arrive; a table of the six hidden vectors lets you inspect the accompanying numerical changes. This is evidence of an evolving representation, not a plain-language explanation of any individual dimension. section 8.2.3
An LSTM distinguishes its hidden state, associated with recently processed tokens, from a cell state that carries longer-term information; gates regulate how those states change. In chatbot design, “state” can instead mean a node in a conversation graph, with conversation context used to decide what happens next. That design state is not the RNN’s hidden vector. section 8.4.3 section 12.3.2
