A recurrent neural network (RNN) processes ordered vectors while reusing the same parameters at every time step. It carries a hidden state forward, making sequence order part of the model rather than treating each item independently. This suits text, weather measurements, and other variable-length sequences. section 4.0 section 4.6
For input vectors x₁ through xₜ, the recurrent step combines the previous summary hₜ₋₁ with the current input xₜ to produce hₜ. In a simple RNN, concatenated input and state pass through a linear layer and tanh. Starting with h₀ as zeros, repeatedly applying this step produces one hidden state per time step. section 4.1.2 section 6.6.1
Figure 4.4 Diagram of the RNN process. We use t to represent the current point in time (the tth item in a sequence). The RNN takes in a single representation of all previous content ht − 1 and information about the newest item in the sequence xt. The RNN merges these into a new representation of everything seen so far ht, and the process repeats until we reach the end of the sequence.
Use the unrolled diagram to trace how each current vector and previous hidden state become the next sequence summary. section 4.1.2
A practical PyTorch sequence-classification pipeline is:
- Represent tokens or other sequence items as vectors; text commonly uses an embedding layer. section 4.2
- Pass a batch shaped (B, T, D) to nn.RNN, where B is batch size, T is sequence length, and D is features per step. section 4.2
- Extract the final hidden state hₜ, whose shape is (B, hidden_nodes), and feed it to nn.Linear for a many-to-one prediction. section 4.2.3
- For mixed-length batches, pad sequences and use packing so the RNN processes only valid elements. section 4.3.3
Classifying a name by language
A compact model can embed character IDs, run nn.RNN, extract LastTimeStep, and classify with nn.Linear. The demonstrated task predicts a name’s language, using inputs such as “frank” and producing probabilities across language labels. The same structure supports one label for any sequence length because the final hidden state has fixed size. section 4.2.3
RNNs are effective but difficult to train: unrolling creates many operations over the same state, allowing gradient noise to accumulate across long sequences. Smaller batches may help performance, while multiple recurrent layers and bidirectional processing can improve accuracy at added computational cost. LSTMs and GRUs address recurrent modeling more robustly; an LSTM adds a context state and can often replace nn.RNN with minimal code changes. section 4.3.3 section 4.4.1 section 6.6 section 6.6.3
Figure 4.8 Four high-level steps for making a prediction based on an input sequence of length T.We need to create a representation of the sequence as vectors, pass that representation to an RNN toproduce T hidden states, reduce to one hidden state, and then use a fully connected layer to makea prediction.
Use the pipeline figure to connect vectorization, recurrent hidden states, reduction to one state, and the final fully connected prediction. section 4.2
