Long short-term memory (LSTM) extends recurrent neural networks for text whose important information must survive across many tokens. Instead of relying on one repeatedly updated hidden representation, it learns which signals to retain, update, forget, and output. section 8.4.2 section 8.4.3
An LSTM recurrent unit uses four learned internal gates and two recurrent state tensors:
- The forgetting gate decides which hidden-layer information to discard. section 8.4.3
- The input or update gate selects information relevant to the current output. section 8.4.3
- The relevance or cell gate regulates candidate information that should enter the cell state. section 8.4.3
- The output gate controls what becomes the activation and next-token prediction. section 8.4.3
- The hidden state carries short-term information, while the cell state stores longer-term patterns for reuse at the next token. section 8.4.3
A gated state update
A compact implementation view computes sigmoid gates f, i, and o, a tanh candidate g, then updates c = f*c + i*g and h = o*tanh(c). Learned weights and biases determine how strongly each signal passes. section 8.4.3
r = sigmoid(W_i2r.mm(x) + b_i2r + W_h2r.mm(h) + b_h2r)
z = sigmoid(W_i2z.mm(x) + b_i2z + W_h2z.mm(h) + b_h2z)
n = tanh(W_i2n.mm(x) + b_i2n + r∗(W_h2n.mm(h) + b_h2n))
f = sigmoid(W_i2f.mm(x) + b_i2f + W_h2f.mm(h) + b_h2f) #1
i = sigmoid(W_i2i.mm(x) + b_i2i + W_h2i.mm(h) + b_h2i) #2
g = tanh(W_i2g.mm(x) + b_i2g + W_h2y.mm(h) + b_h2g) #3
o = sigmoid(W_i2o.mm(x) + b_i2o + W_h2o.mm(h) + b_h2o) #4
c = f*c + i*g #5
h = o*tanh(c)
Inspect the gate equations and the final cell-state and hidden-state updates directly. section 8.4.3
LSTMs support sequence tasks such as next-token prediction, where ordinary RNN memory fades as sequences grow longer. WikiText2 provides a concrete benchmark, although masked <unk> tokens introduce an avoidable accuracy limitation. section 8.4.2 section 8.5
