A memory slot is one component of a neural memory structure, holding a distinct piece of information. Adam Santoro and colleagues at DeepMind used multiple slots in the relational memory core (RMC) to address a weakness of recurrent networks: storing a sequence does not necessarily let a model compare what it has stored. Rather than keep one monolithic hidden state, the RMC lets its slots communicate at every time step. section 7.4
That communication uses Transformer-style multihead dot-product attention. Each slot forms a query; slots supply keys and values. Query–key matches determine which values a slot reads, so it can draw on information held elsewhere in memory. The current input also enters the update. A small feed-forward network refines the result, and a gate controls what is retained for the next step. Multiple attention heads allow different relationships to be considered in parallel. section 7.4
Figure 7.6 (a) The RMC takes the previous memory matrix and the current input and applies a multihead dot-product attention (MHDPA) module labeled A, allowing memory slots to exchange information with one another and with the input. It then refines each slot with a small feed-forward network (MLP) and writes the result through a gate to produce the output for the step and the next memory state. (b) For each memory slot and the input, the model forms queries, keys, and values, enabling multiple “heads” to view relations from different perspectives. (c) Similarity between queries and keys yields normalized attention weights, which are used to compute weighted combinations of the values. The results update the memory matrix so that each slot integrates the most relevant information from others. The MLP and gate then finalize what is retained for the next step. Used with permission granted by the lead author (Adam Santoro).
The diagram follows information from the previous memory matrix and current input through attention, slot refinement, and the gate that produces the next memory state. section 7.4
Finding the second-farthest vector
Suppose vectors arrive one at a time and the model must identify which is second farthest from a reference vector. Merely retaining the vectors is insufficient: it must compare distances and track their rank. One slot might track the reference while another accumulates evidence about relative distances; attention lets their information interact as new vectors arrive. On this task, the RMC reached about 91% accuracy, while a standard LSTM and a differentiable neural computer remained below 30%. section 7.4 section 7.4.1
Memory slots therefore matter not just as places to put information, but as participants in repeated comparison. The RMC also performed strongly on short-program evaluation tasks involving operations such as addition, copying, and reversing. The engineering lesson is narrower than a claim to have solved reasoning: organizing memory so stored items can interact can help where storage and retrieval alone fall short. section 7.4 section 7.4.1
