A model can recognize a red cube and a green sphere yet struggle to answer a question about how they relate. In 2017, Adam Santoro and colleagues at Google DeepMind proposed the relation network (RN) as a module for that gap: it compares pairs of objects or entities, then combines the results to answer a question. Unlike a symbolic system, it does not require a handcrafted set of relation rules. section 7.2
For a visual question, a CNN extracts features from an image and an LSTM encodes the question. The RN applies a learned pairwise function, gθ, to object pairs in the question’s context. It sums the resulting evidence, and a final network, fφ, maps that sum to an answer. The pair comparisons give the model a place to work with relationships that local visual features alone may not resolve. section 7.2
Following a chain of objects
Ask: “What size is the cylinder that is left of the brown metal thing that is left of the big sphere?” One object pair can supply evidence about the brown metal object and sphere; another can supply evidence about the cylinder and brown metal object. Aggregating those comparisons leads to the answer “small,” without writing an explicit symbolic program. section 7.2
Figure 7.5 An image from CLEVR is paired with a reasoning question. A CNN extracts object features, a long short-term memory encodes the question, and the RN compares every object pair in light of that question. The results are aggregated and passed to another network, which outputs the answer (here, “small”). Note: Letters were added to the original figure to aid comprehension in black-and-white printing. Used with permission granted by the lead author (Adam Santoro).
The diagram traces the image and question through feature extraction, pairwise comparison, aggregation, and the final answer. section 7.2
The empirical case extended beyond one image format. On CLEVR, the RN reached 95.5% accuracy, against about 68.5% for the strongest CNN–LSTM baseline reported there. With structured scene descriptions instead of images, it reached 96.4%, showing that the relational module could also use explicit object properties. On the text-based bAbI benchmark, it passed 18 of 20 reasoning tasks. These are strong benchmark results, not a claim that relational reasoning is solved in general. section 7.2.1 section 7.6
