An LSTM (long short-term memory) layer is a recurrent neural network layer for ordered sequences. It processes inputs timestep by timestep while carrying state forward. That makes it useful when order and the recent past matter: flattening a timeseries discards its time structure, while pooling can destroy order information. section 13.3 section 20.1.6
Start with temperature forecasting
Feed a sequence of weather measurements to `layers.LSTM(16)`, then use `layers.Dense(1)` to predict a temperature. On the Jena forecasting task, this simple model reached a validation MAE as low as 2.39 degrees and a test MAE of 2.55 degrees. It beat the commonsense baseline, though only narrowly. section 13.3
Listing 13.12 A simple LSTM-based model
inputs = keras.Input(shape=(sequence_length, raw_data.shape[-1])) x = layers.LSTM(16)(inputs) outputs = layers.Dense(1)(x) model = keras.Model(inputs, outputs) callbacks = [ keras.callbacks.ModelCheckpoint("jena_lstm.keras", save_best_only=True) ] model.compile(optimizer="adam", loss="mse", metrics=["mae"]) history = model.fit( train_dataset, epochs=10, validation_data=val_dataset, callbacks=callbacks, ) model = keras.models.load_model("jena_lstm.keras") print("Test MAE: {model.evaluate(test_dataset)[1]:.2f}")
The working forecasting model shows the sequence-shaped Keras input, LSTM-to-Dense path, checkpoint, and training setup. section 13.3
The reason to choose an LSTM over a basic `SimpleRNN` is long-range learning. A simple recurrent layer can theoretically retain earlier inputs, but in practice vanishing gradients make long-term dependencies hard to learn. The LSTM adds a carry track and gated updates; its carry update combines new information with retained information. This is a practical intuition, not a requirement to implement the gates yourself. section 13.3.2
Three Keras decisions matter immediately:
- Shape inputs as `(batch_size, timesteps, input_features)`. Set the timestep dimension to `None` for variable-length sequences; specify a length when it is known. section 13.3.2
- Use the default last-output mode when one prediction represents the whole sequence. Set `return_sequences=True` when the next recurrent layer needs an output at every timestep. section 13.3.2 section 20.1.6
- Watch validation curves for overfitting. Keras offers `dropout` for input units and `recurrent_dropout` for recurrent units; recurrent dropout uses a mask held constant across timesteps. section 13.3.4
For text classification, a bidirectional LSTM can use information from both directions of a token sequence. Token IDs first need vector representations. One-hot vectors work but can be costly: a movie-review example used 600-token inputs and a 30,000-token vocabulary, trained slowly, and reached about 84% test accuracy. An `Embedding` layer reduced the model from roughly 15 million to 2 million parameters without improving that result on its own. section 14.5.1 section 14.5.3
Listing 14.24 Building an LSTM sequence model with an Embedding layer
hidden_dim = 64 inputs = keras.Input(shape=(max_length,), dtype="int32") x = keras.layers.Embedding( input_dim=max_tokens, output_dim=hidden_dim, mask_zero=True, )(inputs) x = keras.layers.Bidirectional(keras.layers.LSTM(hidden_dim))(x) x = keras.layers.Dropout(0.5)(x) outputs = keras.layers.Dense(1, activation="sigmoid")(x) model = keras.Model(inputs, outputs, name="lstm_with_embedding") model.compile( optimizer="adam", loss="binary_crossentropy", metrics=["accuracy"], )
The classifier places a masked embedding before a bidirectional LSTM, followed by dropout and a sigmoid output. section 14.5.3
Padding needs attention: without masking, the final LSTM representation may reflect repeated `[PAD]` tokens rather than the last real token. `Embedding(mask_zero=True)` supplies a mask that the LSTM uses to return the last non-masked representation; the layer also accepts an explicit `mask` argument. Pretraining the embedding with CBOW raised the movie-review classifier’s test accuracy to about 89%, but the ordered-sequence model remained constrained by its training data. section 14.5.3 section 14.5.5
Rule of thumb: consider an LSTM when sequence order matters, then compare it with cheaper options. A GRU is a simpler, less expensive recurrent alternative; for the movie-review task, fast set-based models were also competitive. section 20.1.6 section 14.5.5
