A Gated Recurrent Unit (GRU) is a recurrent layer for sequences. It maintains a state as it processes timesteps, allowing later predictions to use information from earlier inputs. Think of it as a streamlined LSTM: GRU is simpler and cheaper, while LSTM is more powerful. Both are practical choices when the recent past matters more than the distant past. section 13.3.5 section 20.1.6
Forecasting temperature
To increase a temperature forecaster’s capacity, stack two 32-unit GRU layers. Set `return_sequences=True` on the first so the second receives an output at every timestep; let the second return only its final output for a single temperature prediction. With recurrent dropout in both layers and another Dropout layer before the Dense output, this setup reached a test MAE of 2.39 degrees—an 8.8% improvement over the baseline. The extra layer helped, but not dramatically. section 13.3.5
Listing 13.23 Training and evaluating a dropout-regularized, stacked GRU model
inputs = keras.Input(shape=(sequence_length, raw_data.shape[-1])) x = layers.GRU(32, recurrent_dropout=0.5, return_sequences=True)(inputs) x = layers.GRU(32, recurrent_dropout=0.5)(x) x = layers.Dropout(0.5)(x) outputs = layers.Dense(1)(x) model = keras.Model(inputs, outputs) callbacks = [ keras.callbacks.ModelCheckpoint( "jena_stacked_gru_dropout.keras", save_best_only=True ) ] model.compile(optimizer="adam", loss="mse", metrics=["mae"]) history = model.fit( train_dataset, epochs=50, validation_data=val_dataset, callbacks=callbacks, ) model = keras.models.load_model("jena_stacked_gru_dropout.keras") print(f"Test MAE: {model.evaluate(test_dataset)[1]:.2f}")
The forecasting listing shows the two GRU calls, the crucial `return_sequences=True`, regularization, and training and evaluation code together. section 13.3.5
For next-character prediction, the output requirement changes: the model needs a prediction at every character position. An Embedding feeds a single GRU with `return_sequences=True`, followed by Dropout and a Dense softmax over the character vocabulary. Recurrent state carries information from past characters into each prediction. section 15.1.1
Listing 15.4 Building a miniature language model
embedding_dim = 256 hidden_dim = 1024 inputs = layers.Input(shape=(sequence_length,), dtype="int", name="token_ids") x = layers.Embedding(vocabulary_size, embedding_dim)(inputs) x = layers.GRU(hidden_dim, return_sequences=True)(x) x = layers.Dropout(0.1)(x) outputs = layers.Dense(vocabulary_size, activation="softmax")(x) #1 model = keras.Model(inputs, outputs)
The miniature language model listing shows how Embedding, GRU, Dropout, and Dense produce a probability distribution at each timestep. section 15.1.1
Choose the GRU output to match the job:
- Return the full sequence when another recurrent layer follows or when every timestep needs a prediction. section 13.3.5 section 15.1.1
- Return the final output when one result must summarize the input sequence. section 13.3.5 section 20.1.6
- Use `return_state=True` when a generation loop must pass the state explicitly between calls. section 15.1.2
Generating one character at a time
Training processes 100-character sequences, with the GRU state handled implicitly. For generation, a one-character model accepts an input state and returns both a character prediction and the updated state; it can reuse the trained model’s weights. Feed a prompt one character at a time to establish the starting state. Then select a predicted character, feed it back with the updated state, and repeat. The demonstrated loop selects the most likely character each time. section 15.1.1 section 15.1.2
Listing 15.6 Modifying the language model for autoregressive inference
inputs = keras.Input(shape=(1,), dtype="int", name="token_ids") #1 input_state = keras.Input(shape=(hidden_dim,), name="state") x = layers.Embedding(vocabulary_size, embedding_dim)(inputs) x, output_state = layers.GRU(hidden_dim, return_state=True)( x, initial_state=input_state ) outputs = layers.Dense(vocabulary_size, activation="softmax")(x) generation_model = keras.Model( inputs=(inputs, input_state), outputs=(outputs, output_state), ) generation_model.set_weights(model.get_weights()) #2
The inference listing makes state an explicit model input and output with `return_state=True`, then copies weights from the trained model. section 15.1.2
The same state-passing idea supports sequence-to-sequence translation. A bidirectional GRU encoder represents the English input; its output initializes a GRU decoder that predicts Spanish tokens. The decoder must use only preceding target tokens—not a bidirectional view of future ones. For these recurrent models, use GRU when its simpler, cheaper design is a useful trade-off against LSTM. section 15.2.2 section 20.1.6
