LSTM

As explained in Sutskever's List

Long short-term memory is a recurrent neural network architecture with gated state updates for modeling long-range sequence dependencies.

A sentence can separate the clue from the word it predicts: “I grew up in France … I speak fluent French.” A recurrent neural network (RNN) carries context forward in a hidden state, but learning a connection across many intervening words is difficult. During backpropagation through time, the learning signal from a distant prediction can become negligibly small. The long short-term memory network, or LSTM, was developed by Sepp Hochreiter and Jürgen Schmidhuber to address this long-range dependency problem. section 4.2

An LSTM is a specialized RNN with a cell state as well as a hidden state. Its learned forget, input, and output gates control what is retained, what updates the cell state, and what is exposed as output. This selective pathway makes earlier information more useful later in a sequence. It does not remove the need to train and regularize a recurrent model. section 4.2 section 4.3

Figure 4.2 A single LSTM cell. The inputs are the current vector X_t, the previous hidden state h_t–1, and the previous cell state C_t–1. There are three learned gates: the forget f_t, input i_t, and output O_t. They are all produced by sigmoid layers (“σ” blocks). A tanh layer forms a candidate update \(\tilde{C}\)_t. Pointwise multiplications (“×”) and an addition (“+”) update the cell state on the top “conveyor” pathway. Arrows indicate vector flow; the legend marks pointwise operations, concatenation, and copies. This is the standard LSTM, emphasizing how gating enables selective memory retention and exposure. Used with permission granted by the author (Chris Olah), though it is slightly modified from its original form.

Trace the previous cell state along the upper pathway. The forget and input gates shape its update; the output gate controls how much of the resulting state is exposed. section 4.2

LSTMs belong to the resurgence of recurrent sequence models in the mid-2010s. Chris Olah’s 2015 visual explanation made their gates easier to understand; Zaremba, Sutskever, and Vinyals’s 2014 work addressed a different obstacle, overfitting, with selective dropout tailored to LSTM architectures. The engineering lesson is to distinguish remembering distant context from training a model that generalizes. section 4.0 section 4.2 section 4.3

How Sutskever's List covers it

How deep it goes

Explains the vanishing-gradient motivation and illustrates cell-state updates through gates, without deriving their equations.

Examples

Uses a France-to-French prediction and a single-cell diagram to show distant context and gated state flow.

What it assumes

Recaps feed-forward networks, n-grams, hidden state, and backpropagation through time before examining the LSTM cell.

What you can do after

Evaluate/understand it as a gated approach to long-range sequence context, distinct from regularization against overfitting.

How it connects

Connects LSTMs to recurrent neural networks, backpropagation through time, vanishing gradients, and dropout.

liveBookmanning.com

Read Sutskever's List on liveBook Get the book at manning.com

Also explained in

• 

  liveBookmanning.com
  Deep Learning with Python, Third EditionA long short-term memory recurrent neural network architecture designed to capture long-range dependencies in sequences.
• 

  liveBookmanning.com
  Deep Learning with R, Third EditionLong short-term memory is a gated recurrent-neural-network architecture designed to preserve and control information across sequence timesteps.
• 

  liveBookmanning.com
  Inside Deep Learning: Math, Algorithms, ModelsA gated recurrent neural-network architecture designed to learn long-range dependencies.
• 

  liveBookmanning.com
  Learn Generative AI with PyTorchA recurrent neural network variant with long short-term memory units, mentioned here as an example of a deep learning model that cannot process raw text directly.
• 

  liveBookmanning.com
  Natural Language Processing in Action, Second EditionLong short-term memory is a recurrent neural network architecture that uses gated state updates to preserve and control information over long sequences.
• 

  liveBookmanning.com
  LLMs in ProductionA long short-term memory network is a recurrent neural network architecture that uses memory cells and gates to preserve or discard information across sequences.
• 

  liveBookmanning.com
  Grokking Machine LearningA long short-term memory network is a recurrent neural network architecture with gated memory cells designed to learn long-range dependencies in sequences.

Compare how 8 books explain LSTM →

Related concepts

• recurrent neural networkkind of · A recurrent neural network is a neural network that processes sequences by updating an internal state across successive inputs, allowing earlier inputs to influence later computations.
• vanishing gradient problemaddresses · The vanishing-gradient problem is a neural-network training difficulty in which gradients shrink toward zero as they propagate backward, making updates to earlier layers or distant sequence steps ineffective.
• carry statehas part · A carry state is an LSTM’s persistent long-term memory representation, transported across sequence positions through selectively gated updates.
• GRUalternative to · A gated recurrent unit (GRU) is a recurrent neural-network architecture that uses learned gates to regulate information flow and update a hidden state across sequence steps.
• maskused by · Masking is the selective representation, concealment, inclusion, exclusion, or restriction of parts of data or a system according to specified criteria.
• embeddingused by · A numerical vector representation of an item or input that encodes features or relationships useful for computational tasks.

This explainer was generated by Manning's knowledge graph from the text of Sutskever's List. The section links point to the parts of the book it draws on.