LSTM

As explained in Learn Generative AI with PyTorch

A recurrent neural network variant with long short-term memory units, mentioned here as an example of a deep learning model that cannot process raw text directly.

A long short-term memory (LSTM) network is a recurrent neural network (RNN) for sequential data. Where a network that treats inputs independently loses their order, an LSTM uses information from earlier time steps when processing the current one. This makes it useful for predicting the next token in text. section 8.0

Standard RNNs can struggle to learn relationships across long sequences because gradients diminish during training. LSTM units add a cell state that can carry relevant information along the sequence, helping retain longer-term patterns. LSTMs still struggle with extremely long-range dependencies and process tokens sequentially, unlike Transformers, which can train across sequence positions in parallel. section 8.1.2 section 1.3

To build a next-token text generator:

• Tokenize the training text and assign each distinct token an integer index. An LSTM cannot take raw text as input. section 8.1.3
• Cut the index sequence into inputs of 100 tokens. Shift each input one position to form its target sequence; batch the resulting pairs for training. section 8.1.3 section 8.3.2
• Pass indexes through a 128-dimensional word embedding, then a PyTorch LSTM layer and a linear layer sized to the vocabulary. The resulting logits can be converted to next-token probabilities. section 8.4

Learning a writing style

A model trained on Anna Karenina learns next-token patterns from the novel. For a small illustration of the training target, the input “how are you” pairs with “are you today”: each position asks the model to predict the following token. To generate text after training, start with a prompt, append a predicted token, and repeat. Temperature and top-K sampling can vary the output. section 8.1.3 section 8.3.2 section 8.5

Figure 8.2 An example of how an LSTM model is trained. We first break down the training text into tokens and assign a unique integer to each token, creating a numerical representation of the text as a sequence of indexes. We then divide this long sequence into shorter sequences of equal length. These sequences form the features (the x variable) of our model. We then generate the output y by shifting the input sequence one token to the right. This setup enables the LSTM model to predict the next token based on previous tokens in the sequence.

The diagram traces the preparation pipeline from tokens to integer-index inputs and one-token-shifted targets. section 8.1.3

How Learn Generative AI with PyTorch covers it

How deep it goes

Explains cell-state memory and walks through data preparation, model layers, training, and generation without detailing cell gates.

Examples

Uses Anna Karenina, 100-token training sequences, a shifted-token example, and prompt-based generation.

What it assumes

Introduces tokenization and integer-index mapping while contrasting sequential models with networks that treat inputs independently.

What you can do after

Build it from scratch with PyTorch’s LSTM layer, then generate text by repeatedly predicting and appending tokens.

How it connects

Connects LSTMs to RNNs, vanishing gradients, tokenization, word embeddings, next-token prediction, and Transformers.

liveBookmanning.com

Read Learn Generative AI with PyTorch on liveBook Get the book at manning.com

Also explained in

• 

  liveBookmanning.com
  Deep Learning with Python, Third EditionA long short-term memory recurrent neural network architecture designed to capture long-range dependencies in sequences.
• 

  liveBookmanning.com
  Deep Learning with R, Third EditionLong short-term memory is a gated recurrent-neural-network architecture designed to preserve and control information across sequence timesteps.
• 

  liveBookmanning.com
  Inside Deep Learning: Math, Algorithms, ModelsA gated recurrent neural-network architecture designed to learn long-range dependencies.
• 

  liveBookmanning.com
  Sutskever's ListLong short-term memory is a recurrent neural network architecture with gated state updates for modeling long-range sequence dependencies.
• 

  liveBookmanning.com
  Natural Language Processing in Action, Second EditionLong short-term memory is a recurrent neural network architecture that uses gated state updates to preserve and control information over long sequences.
• 

  liveBookmanning.com
  LLMs in ProductionA long short-term memory network is a recurrent neural network architecture that uses memory cells and gates to preserve or discard information across sequences.
• 

  liveBookmanning.com
  Grokking Machine LearningA long short-term memory network is a recurrent neural network architecture with gated memory cells designed to learn long-range dependencies in sequences.

Compare how 8 books explain LSTM →

Related concepts

• recurrent neural networkkind of · A recurrent neural network is a neural network that processes sequences by updating an internal state across successive inputs, allowing earlier inputs to influence later computations.
• vanishing gradient problemaddresses · The vanishing-gradient problem is a neural-network training difficulty in which gradients shrink toward zero as they propagate backward, making updates to earlier layers or distant sequence steps ineffective.
• carry statehas part · A carry state is an LSTM’s persistent long-term memory representation, transported across sequence positions through selectively gated updates.
• GRUalternative to · A gated recurrent unit (GRU) is a recurrent neural-network architecture that uses learned gates to regulate information flow and update a hidden state across sequence steps.
• maskused by · Masking is the selective representation, concealment, inclusion, exclusion, or restriction of parts of data or a system according to specified criteria.
• embeddingused by · A numerical vector representation of an item or input that encodes features or relationships useful for computational tasks.

This explainer was generated by Manning's knowledge graph from the text of Learn Generative AI with PyTorch. The section links point to the parts of the book it draws on.