A recurrent neural network (RNN) reads a sequence one step at a time while carrying an internal memory, or hidden state, from one step to the next. This lets it use earlier words when predicting what comes next. section 1.2
Predicting the weather
For “I left my umbrella at home, and now it’s…,” the RNN reads each part, updates its memory, and can use the accumulated context to predict “raining.” During training, repeated next-word prediction helps it learn grammar, common phrases, and relationships between words from data rather than hand-coded rules. section 1.2
Figure 1.2 Step-by-step processing of the sentence “I left my umbrella at home, and now it's…” by an RNN. At each step, the model reads one word (or group of words), updates its internal memory, and passes that memory forward to the next step. By step 3, the accumulated context that someone left an umbrella at home is sufficient for the model to predict a contextually appropriate continuation, such as "raining."

Follow the sentence word by word to see how the RNN updates its memory and carries context forward toward a prediction. section 1.2
The approach works well for nearby context but becomes fragile across long sequences: earlier details can fade as information passes through many intermediate steps. Attention addresses this bottleneck by focusing directly on relevant earlier words, leading toward transformer architectures. section 1.2.2 section 1.3

















