A recurrent neural network (RNN) processes a sequence one step at a time. At each step, it combines the new input with an internal state carrying information from earlier steps. This makes it a natural choice when order matters, as in forecasting from recent weather measurements or predicting the next character in a sentence. section 13.3.1 section 13.2 section 15.1.1
Forecast tomorrow’s temperature
Suppose recent measurements of temperature, pressure, humidity, and other weather quantities are available, and the goal is to predict temperature 24 hours ahead. A small Keras model takes a sequence of measurements, passes it through an LSTM layer with 16 units, and uses a Dense layer to produce one prediction. It is trained with mean squared error and assessed with mean absolute error. On the Jena weather data, the saved model reaches a test MAE of about 2.52°C, beating a commonsense baseline. That comparison matters: a more elaborate model is useful only if it improves on a simple alternative. section 13.2 section 13.3 section 13.5
Listing 13.13 A simple LSTM-based model
inputs <- keras_input(shape = c(sequence_length, ncol_input_data))
outputs <- inputs |>
layer_lstm(16) |>
layer_dense(1)
model <- keras_model(inputs, outputs)
callbacks <- list(
callback_model_checkpoint("jena_lstm.keras", save_best_only = TRUE)
)
compile(model, optimizer = "adam", loss = "mse", metrics = "mae")history <- model |> fit(
train_dataset,
epochs = 10,
validation_data = val_dataset,
callbacks = callbacks
)model <- load_model("jena_lstm.keras")
sprintf("Test MAE: %.2f", evaluate(model, test_dataset)[["mae"]])
[1] "Test MAE: 2.52"
The R/Keras listing builds, trains, checkpoints, and evaluates the temperature-forecasting LSTM. section 13.3
Why not flatten the measurements into one vector? Flattening discards the explicit notion of time. A convolutional approach that treats segments alike and pools them can also lose order information. Recurrence instead preserves a step-by-step path through the sequence. Architecture is a practical choice about which structure in the data a model should respect—not a guarantee that every RNN will win. section 13.3 section 5.3.2
A simple RNN makes the moving parts visible:
- Start with a zero-valued state. For each timestep, combine its input with the current state to produce an output; use that output as the next state. Reset the state between independent sequences. section 13.3.1
- For a batch, Keras expects inputs shaped as batch size × timesteps × input features. A recurrent layer can return either the last output or an output at every timestep; set return_sequences = TRUE when a following recurrent layer needs the full sequence. section 13.3.2
- SimpleRNN illustrates the idea but struggles to learn long-term dependencies because of vanishing gradients. LSTM and GRU are the more useful Keras recurrent layers. Stacking layers adds representational power and expense. section 13.3.2 section 13.5
Listing 13.16 R implementation of a simple RNN
runif_array <- function(dim) array(runif(prod(dim)), dim)
timesteps <- 100 #1
input_features <- 32 #2
output_features <- 64 #3
inputs <- runif_array(c(timesteps, input_features)) #4
state_t <- array(0, dim = output_features) #5
W <- runif_array(c(output_features, input_features)) #6
U <- runif_array(c(output_features, output_features)) #6
b <- runif_array(c(output_features, 1)) #6
outputs <- array(0, dim = c(timesteps, output_features)) #6
for(ts in 1:timesteps) {
input_t <- inputs[ts, ] #7
output_t <- tanh( (W %*% input_t) + (U %*% state_t) + b ) #8
outputs[ts, ] <- state_t <- output_t #9
}
final_output_sequence <- outputs #10
This short R loop exposes the input, weights, state update, and output at each timestep before those details are packaged into a Keras layer. section 13.3.1
The output choice follows the task. A weather forecast needs one value for the whole input sequence, so the final recurrent output can feed a Dense prediction. A character-level language model instead predicts at every position: an Embedding feeds a GRU that returns the full sequence, followed by a softmax prediction for each character. section 13.3 section 15.1.1
Sequence-to-sequence translation reveals another constraint. Predicting each target word directly from the corresponding source position may fail when the needed source word comes later. An encoder RNN can first read the entire source sequence; its representation then initializes a decoder RNN that generates the translation. This connects recurrent state to both remembering earlier inputs and passing information between two sequence-processing networks. section 15.2.2
