A recurrent neural network (RNN) processes a sequence one timestep at a time. Instead of flattening the whole sequence into one vector, it keeps a state containing information from earlier steps and updates that state as each new input arrives. This makes order part of the computation: reversing the same inputs can change what the network learns from them. section 13.3.1 section 13.3.6
Forecast tomorrow’s temperature
Suppose recent readings of temperature, pressure, humidity, and other weather measurements are available. The task is to predict the temperature 24 hours ahead. A recurrent layer can read the measurements in time order, carrying information forward as it processes the sequence. The example also provides a practical check: compare the trained model against a commonsense no-learning baseline before deciding that its extra complexity helps. section 13.2 section 13.5
The smallest useful RNN is a loop. Start with a zero state. At each timestep, combine the current input with the state to produce an output, then use that output as the next state. A toy implementation computes `output_t = tanh(W @ input_t + U @ state_t + b)`. One sequence remains one input sample even though the network processes it in steps; the state is reset between independent sequences. section 13.3.1
Listing 13.15 NumPy implementation of a simple RNN
import numpy as np timesteps = 100 #1 input_features = 32 #2 output_features = 64 #3 inputs = np.random.random((timesteps, input_features)) #4 state_t = np.zeros((output_features,)) #5 W = np.random.random((output_features, input_features)) #6 U = np.random.random((output_features, output_features)) #6 b = np.random.random((output_features,)) #6 successive_outputs = [] for input_t in inputs: #7 output_t = np.tanh(np.dot(W, input_t) + np.dot(U, state_t) + b) #8 successive_outputs.append(output_t) #9 state_t = output_t #10 final_output_sequence = np.concatenate(successive_outputs, axis=0) #11
The NumPy loop makes the two inputs to each step—the new vector and the carried state—visible in code. section 13.3.1
Choose the recurrent setup to match the job:
- For a straightforward sequence model, use a Keras LSTM or GRU layer. Both are recurrent layers; the GRU is used for a compact character-level language model that predicts a label at every position. section 13.5 section 15.1.1
- For more representational power, stack recurrent layers, but check whether the added computation improves validation results. Recurrent dropout is available through Keras recurrent-layer arguments to help fight overfitting. section 13.3.3 section 13.5
- For tasks where both directions of an available input sequence may help, try a bidirectional layer. Do not assume reversing a sequence is harmless: a reversed-order LSTM performed poorly in the weather experiment. section 13.3.6
RNNs also serve text tasks. A character-level language model carries information about preceding characters into each prediction. In sequence-to-sequence translation, an encoder reads the complete source before its representation initializes a decoder. Predicting target tokens directly as source tokens arrive can fail when an early target word depends on a late source word. section 15.1.1 section 15.2.2
Listing 15.12 Building a sequence-to-sequence decoder
target = keras.Input(shape=(None,), dtype="int32", name="spanish") x = layers.Embedding(vocab_size, embed_dim, mask_zero=True)(target) rnn_layer = layers.GRU(hidden_dim, return_sequences=True) x = rnn_layer(x, initial_state=encoder_output) x = layers.Dropout(0.5)(x) target_predictions = layers.Dense(vocab_size, activation="softmax")(x)#1 seq2seq_rnn = keras.Model([source, target], target_predictions)
The decoder listing shows the encoder output passed as a GRU’s `initial_state`, rather than starting the decoder from zeros. section 15.2.2
Use an RNN when timestep order matters, particularly when recent observations may matter more than distant ones. Still, test simpler models first. RNNs can struggle to recall information across long sequences; attention and Transformers offer another approach to passing information across a sequence. For forecasting, tune sequence length, sampling rate, layer size, and dropout empirically rather than expecting one configuration to fit every dataset. section 20.1.6 section 15.3 section 13.4
