liveBook · Manning

Gating

As explained in Machines that Think

Gating is the selective control of information flow into, out of, or within a computational state.

Gating controls how information flows through a computational state. In recurrent neural networks, trainable gates help preserve useful information across time while filtering out less relevant details. Their learned weights and activation functions produce values that regulate how much information passes through. section 3.4.4

Long Short-Term Memory networks (LSTMs) use separate gates to manage memory. A forget gate controls how much old memory remains; an input gate regulates new information; and an output gate controls how much updated memory becomes visible in the hidden state. Gated Recurrent Units (GRUs) use a simpler structure with fewer components. section 3.4.4

Figure 3.15 An LSTM neuron. At each time step, the cell receives the input xt, the previous hidden state ht-1, and the previous cell state Ct-1. The forget gate uses these signals to determine how much of the old memory to keep, while the input gate and the candidate update, propose and regulate the new information to add. Together, these components produce the updated cell state Ct, which carries long-term memory forward. The output gate then controls how much of this updated memory becomes visible by shaping the hidden state ht. This division between a protected memory pathway and an exposed hidden state allows LSTMs to maintain information across many time steps while still responding flexibly to new inputs.

The diagram separates the LSTM’s memory pathway from the hidden state exposed at each time step. section 3.4.4

Keeping a sentence’s subject

In “The book that I borrowed last week was fascinating,” a model must retain information about “the book” across the intervening words. An LSTM’s memory pathway can preserve that information while its gates filter less relevant details. section 3.4.4

Gating makes recurrent memory more selective, but it does not eliminate the limits of a single evolving state. LSTMs and GRUs process tokens sequentially, limiting parallelism, and earlier information is accessible only through what the state carries forward. Distant dependencies can become compressed, blended, or difficult to separate. section 3.5.3

How Machines that Think covers it

How deep it goes
Explains learned gate values and the roles of LSTM gates, and contrasts them with the simpler GRU structure.
Examples
Uses a sentence with intervening words between its subject and predicate to illustrate preserved context.
What you can do after
Evaluate gated recurrent memory by weighing selective retention against sequential processing and the limits of a compressed state.
How it connects
Connects gating to recurrent neural networks, hidden states, long-range dependencies, and attention mechanisms.

Also explained in

Compare how 6 books explain gating →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Machines that Think. The section links point to the parts of the book it draws on.