Working memory is a temporary store for information that remains available during a limited span of processing. In recurrent networks, it gives the model a structured place to retain recently processed tokens while longer-term meaning is stored separately. section 8.4 section 8.4.3
An LSTM represents working memory with its hidden state. The hidden state carries information about the most recently processed tokens, while the cell state carries longer-term information and is reused at the next token. section 8.4.3
LSTM gates regulate what working memory retains, updates, and exposes:
- The forgetting gate ignores parts of the hidden layer to make room for future tokens. section 8.4.3
- The input or update gate selects which parts of the hidden layer matter to the current output. section 8.4.3
- The relevance or cell gate blocks information no longer relevant to the current output. section 8.4.3
- The output gate selects what becomes the activation and hidden-state output used to predict the next token. section 8.4.3
LSTM state update
The recurrent update combines the previous cell state with candidate information: c = f*c + i*g, followed by h = o*tanh(c).
Figure 8.12 LSTMs add a forgetting gate and a cell output.
Inspect how the forgetting gate and cell output fit into the LSTM architecture. section 8.4.3
