An update gate controls how much of a recurrent unit’s previous hidden state remains relevant versus how much new information should influence the current output. In a GRU, it is a sigmoid-based gate whose values range from near 0 to near 1: low values suppress a contribution, while high values preserve or emphasize it. This gives the recurrent unit an intentional structure for handling long-term memory rather than relying on an ordinary fully connected layer to discover that structure. section 8.4.2
>>> r = sigmoid(W_i2r.mm(x) + b_i2r + W_h2r.mm(h) + b_h2r) #1
>>> z = sigmoid(W_i2z.mm(x) + b_i2z + W_h2z.mm(h) + b_h2z) #2
>>> n = tanh(W_i2n.mm(x) + b_i2n + r∗(W_h2n.mm(h) + b_h2n)) #3
The GRU equations compute the update gate z from the current input x and previous hidden state h, alongside reset and candidate-state calculations. The gate is rebuilt recurrently for each token using learned matrix multiplications, biases, and nonlinear activations. section 8.4.2
Read the gate alongside the other GRU components:
- The reset gate blocks hidden-state information that is no longer relevant to the current output. section 8.4.2
- The update gate determines which parts of the hidden layer should matter at the current time step. section 8.4.2
