Forget gate

As explained in Sutskever's List

A forget gate is an LSTM gating component that controls how much of the previous cell state is retained.

The forget gate is the LSTM component that decides how much of the previous cell state to retain. It combines the current input with the previous hidden state and produces values between 0 and 1: values near 0 discard memory, while values near 1 preserve it. section 4.2

This gate operates on the cell-state conveyor belt, distinct from the hidden state, which changes more actively with each input. Pointwise scaling lets different memory dimensions be forgotten or retained independently rather than applying one uniform decision. section 4.2

Selective memory retention

If a forget-gate value is close to 0 for one component, that component's previous cell-state information is mostly erased; a value near 1 carries it forward. section 4.2

Figure 4.2 A single LSTM cell. The inputs are the current vector X_t, the previous hidden state h_t–1, and the previous cell state C_t–1. There are three learned gates: the forget f_t, input i_t, and output O_t. They are all produced by sigmoid layers (“σ” blocks). A tanh layer forms a candidate update \(\tilde{C}\)_t. Pointwise multiplications (“×”) and an addition (“+”) update the cell state on the top “conveyor” pathway. Arrows indicate vector flow; the legend marks pointwise operations, concatenation, and copies. This is the standard LSTM, emphasizing how gating enables selective memory retention and exposure. Used with permission granted by the author (Chris Olah), though it is slightly modified from its original form.

A standard LSTM-cell diagram shows the forget, input, and output gates, their sigmoid layers, and the pointwise operations that update the cell-state pathway. section 4.2

How Sutskever's List covers it

How deep it goes

Explains inputs, sigmoid-valued retention, and pointwise scaling without deriving the full LSTM equations.

Examples

Uses near-zero and near-one gate values to contrast forgetting with preserving previous cell-state information.

What it assumes

Assumes basic LSTM terminology while recapping cell state, hidden state, and the three-gate architecture.

What you can do after

Evaluate how an LSTM selectively preserves or suppresses components of its previous cell state.

How it connects

Connects the forget gate to LSTM, cell state, hidden state, input gate, output gate, and sigmoid activation.

liveBookmanning.com

Read Sutskever's List on liveBook Get the book at manning.com

Related concepts

• LSTMpart of · An LSTM is a recurrent neural network architecture with gated memory mechanisms that control information flow and help retain or update information across sequences.
• gatingcontrasts with · Gating is the selective regulation of information flow within a computational system, controlling what information is passed, retained, combined, or suppressed.
• reset gatecontrasts with · A reset gate is a GRU mechanism that controls how much of the previous hidden state contributes to computing a candidate state.

This explainer was generated by Manning's knowledge graph from the text of Sutskever's List. The section links point to the parts of the book it draws on.