Gating gives recurrent networks a learned version of an if statement. Instead of letting early tokens fade from memory, gates regulate which signals enter, remain in, or leave the recurrent state. Backpropagation learns the relevant thresholds by adjusting weights and biases, producing signals near 0, 1, or intermediate values. section 8.4.2
Gates are useful because their placement creates deliberate internal memory structure rather than merely adding another activation function.
- GRUs add reset gates to block hidden-layer information that is no longer relevant and update gates to control what should matter to the current output. section 8.4.2
- LSTMs use four internal gates: forgetting, input or update, relevance or cell, and output. Their cell state stores longer-term patterns, while the hidden state represents more recently processed tokens. section 8.4.3
Gate calculations
A GRU computes reset and update signals with sigmoid functions from the current input and previous hidden state, then uses the reset signal when forming a candidate state with tanh. An LSTM extends this pattern with forgetting, input, cell, and output calculations before updating its cell and hidden states.
