A gated recurrent unit (GRU) extends an ordinary recurrent neural network for sequences where early information would otherwise fade. Its reset and update gates regulate which signals enter the hidden representation and how that representation changes over time. section 8.4.2
Think of the gates as learned if-statements inside the recurrent unit. Backpropagation adjusts weights and biases so important token signals can persist, without manually choosing thresholds. Unlike an ordinary fully connected layer, the gates are deliberately connected within the recurrent structure. section 8.4.2
A GRU computes its recurrent update through learned transformations:
- The reset gate combines the current input and previous hidden state, controlling how much previous information contributes to the candidate representation. section 8.4.2
- The update gate combines the same sources and controls the hidden-state update, helping preserve useful information across a longer text. section 8.4.2
- Sigmoid gate values provide learned signals between zero and one, while a tanh transformation produces the candidate recurrent representation. section 8.4.2
Using a GRU for language modeling
A PyTorch language model can select a GRU with `RNNModel('GRU', vocab=corpus.dictionary, num_layers=1)`, load trained parameters, and generate text from a prompt such as `The`. WikiText2 provides a practical benchmark, but masked `<unk>` tokens replace about 2.7% of tokens and can limit achievable accuracy. section 8.5
>>> r = sigmoid(W_i2r.mm(x) + b_i2r + W_h2r.mm(h) + b_h2r) #1
>>> z = sigmoid(W_i2z.mm(x) + b_i2z + W_h2z.mm(h) + b_h2z) #2
>>> n = tanh(W_i2n.mm(x) + b_i2n + r∗(W_h2n.mm(h) + b_h2n)) #3
Inspect the compact GRU gate calculations: reset `r`, update `z`, and candidate representation `n` are formed from input and hidden-state transformations. section 8.4.2
