A recurrent network can remember a sequence without comparing what it remembers. That gap motivated the relational memory core (RMC), introduced by Adam Santoro and colleagues at DeepMind in 2018. Instead of one hidden state, it keeps multiple memory slots and lets them interact at every time step. section 7.4 section 7.4.2
Each step combines the current input with the existing memory. Multihead dot-product attention lets slots select information from one another; a small feed-forward network refines the result, and a gate controls what carries forward. Different attention heads can capture different relationships. The aim is memory that compares and updates, rather than merely stores and retrieves. section 7.4
Figure 7.6 (a) The RMC takes the previous memory matrix and the current input and applies a multihead dot-product attention (MHDPA) module labeled A, allowing memory slots to exchange information with one another and with the input. It then refines each slot with a small feed-forward network (MLP) and writes the result through a gate to produce the output for the step and the next memory state. (b) For each memory slot and the input, the model forms queries, keys, and values, enabling multiple “heads” to view relations from different perspectives. (c) Similarity between queries and keys yields normalized attention weights, which are used to compute weighted combinations of the values. The results update the memory matrix so that each slot integrates the most relevant information from others. The MLP and gate then finalize what is retained for the next step. Used with permission granted by the lead author (Adam Santoro).
The diagram follows one recurrent step, from input and previous memory through attention, refinement and the gate that produces the next memory state. section 7.4
Which item was second farthest?
Suppose vectors arrive one at a time, followed by a question asking which was second farthest from a reference vector. Recall alone is insufficient: the model must compare distances and rank candidates. RMC slots can repeatedly exchange information as new vectors arrive. On this synthetic task, the RMC reached about 91% accuracy, while an LSTM and a differentiable neural computer stayed below 30%. section 7.4 section 7.4.1
Results also extended to short-program evaluation, memorization and language modeling. They were evidence that interacting memory slots could help across sequence tasks, not proof that one architecture had solved reasoning. section 7.4.1 section 7.0
Table 7.4 Accuracy on program evaluation and memorization tasks
|
Model
|
Add
|
Control
|
Program
|
Copy
|
Reverse
|
Double
|
|---|---|---|---|---|---|---|
|
LSTM
|
99.8
|
97.4
|
66.1
|
99.8
|
99.7
|
99.7
|
|
EntNet
|
98.4
|
98.0
|
73.4
|
91.8
|
100.0
|
62.3
|
|
DNC
|
99.4
|
83.8
|
69.5
|
100.0
|
100.0
|
100.0
|
|
RMC
|
99.9
|
99.6
|
79.0
|
100.0
|
100.0
|
99.8
|
Compare RMC accuracy with LSTM, EntNet and differentiable neural computer results on program evaluation and memorization tasks; the gains vary by task. section 7.4.1
The trade-off soon mattered. RMC appeared as research momentum shifted from recurrence toward Transformers. Transformer-XL and the Universal Transformer offered approaches that scaled better, while RMC's sequential computation made it less attractive for large-scale pretraining. Its enduring idea is narrower but useful: stored information can be made to interact before a model acts. section 7.4.2
