Gating

As explained in Sutskever's List

Gating is a learned mechanism that controls how much newly computed information is combined with retained information.

Gating is a learned way to control how much information a model retains, adds, or exposes. In an LSTM, gates use sigmoid values between 0 and 1 to scale information element by element. section 4.2

An LSTM uses three gates to manage its long-term cell state and short-term output:

• The forget gate scales the previous cell state, controlling how much past information remains. section 4.2
• The input gate filters a candidate update before it is added to the cell state. section 4.2
• The output gate controls how much of the updated cell state becomes the current output. section 4.2

Gating also appears in relational recurrent memory networks (RMCs). After attention lets memory slots exchange information and a feed-forward network refines them, a gate blends the new information with the old memory. section 7.4

Comparing remembered vectors

In a task asking which vector is second farthest from a reference vector, an RMC can use memory slots to track the reference and accumulate ranking evidence. Gating controls how much each updated slot retains. section 7.4

Figure 4.2 A single LSTM cell. The inputs are the current vector X_t, the previous hidden state h_t–1, and the previous cell state C_t–1. There are three learned gates: the forget f_t, input i_t, and output O_t. They are all produced by sigmoid layers (“σ” blocks). A tanh layer forms a candidate update \(\tilde{C}\)_t. Pointwise multiplications (“×”) and an addition (“+”) update the cell state on the top “conveyor” pathway. Arrows indicate vector flow; the legend marks pointwise operations, concatenation, and copies. This is the standard LSTM, emphasizing how gating enables selective memory retention and exposure. Used with permission granted by the author (Chris Olah), though it is slightly modified from its original form.

Trace how the LSTM's three gates selectively update the cell-state pathway and shape the output. section 4.2

How Sutskever's List covers it

How deep it goes

Explains LSTM gate roles and elementwise filtering, then relates gating to RMC memory updates.

Examples

Uses an LSTM cell diagram and an RMC task comparing vectors by distance from a reference.

What you can do after

Understand how gates regulate memory retention, updating, and exposure in recurrent models.

How it connects

Connects gating to LSTMs, cell states, recurrent neural networks, and attention.

liveBookmanning.com

Read Sutskever's List on liveBook Get the book at manning.com

Also explained in

• 

  liveBookmanning.com
  Natural Language Processing in Action, Second EditionA logic gate in a gated recurrent network is a learned signal-control mechanism that regulates how information passes into or out of the recurrent state.
• 

  liveBookmanning.com
  LLMs in ProductionA gating mechanism selectively controls how information enters, remains in, or leaves a recurrent model state.
• 

  liveBookmanning.com
  Inside Deep Learning: Math, Algorithms, ModelsA mechanism that modulates information flow by multiplying signals by gate values, typically in the range from zero to one.

Compare how 4 books explain gating →

Related concepts

• thresholduses · A threshold is a boundary value against which a quantity is compared to determine whether a condition is met or an action or decision is triggered.
• GRUpart of · A gated recurrent unit (GRU) is a recurrent neural-network architecture that uses learned gates to regulate information flow and update a hidden state across sequence steps.
• LSTMused by · An LSTM is a recurrent neural network architecture with gated memory mechanisms that control information flow and help retain or update information across sequences.
• neuronproduced by · A neuron is a computational unit in a neural network that transforms inputs, typically using learned parameters and an activation function, to produce an output.
• mixture of expertsused by · A mixture-of-experts model is a neural-network architecture with specialized expert subnetworks and a routing mechanism that selects or combines experts for each input.

This explainer was generated by Manning's knowledge graph from the text of Sutskever's List. The section links point to the parts of the book it draws on.