A recurrent neural network (RNN) models ordered data by carrying information from earlier inputs into later computations. It is suited to time-series data, videos, natural language, and other sequential datasets where order matters. section 8.7 section 8.7.1
An RNN processes a sequence one time step at a time:
- Receives the current input xₜ together with the previous hidden state hₜ₋₁. section 8.7.1
- Updates the hidden state hₜ through a nonlinear function, preserving information from earlier inputs. section 8.7.1
- Produces an output at every time step or uses only the final output, depending on the task. section 8.7.1
- Reuses the same weights across time steps, reducing the number of parameters while applying one network throughout the sequence. section 8.7.1
The recurrent connection feeds each time step's output information into the next. This persistent state acts as memory and helps the network recognize temporal dependencies. A basic RNN can struggle with long-term dependencies because gradients may vanish or explode, limiting its ability to retain information from far back in a sequence. section 8.7.1
Where RNNs fit
Use sequence models for time-series data, video, natural-language processing, speech recognition, or other data whose order carries meaning. For longer dependencies, long short-term memory (LSTM) networks use input, forget, and output gates to regulate information flow; gated recurrent units provide another related option. section 8.7 section 8.7.1
Figure 8.14 The RNN process. RNNs have internal memory, which allows them to use information from the previous inputs to influence the current input and outputs.
Trace how internal memory carries previous inputs forward to influence current inputs and outputs. section 8.7.1
