The Read-Process-Write (RPW) encoder addresses the sensitivity to input order found in sequential models. It embeds set elements in shared memory, then uses an LSTM controller and attention to read and refine that memory into an order-invariant representation. section 5.6
Three sentence orderings
The examples compare “This is a sentence,” its reverse, and a fixed three-word reversal that disrupts natural word groupings. These orderings illustrate how arbitrary rearrangement can make patterns harder to learn. section 5.6
Figure 5.9 Read-Process-Write reads an input sequence into memory, processes it with an LSTM-based recurrent controller, and writes an output sequence. The Read module encodes the inputs, the Process module iteratively updates a hidden state while attending to memory, and the Write module generates outputs one token at a time, feeding its own predictions back as inputs. The arrows between the decoder’s outputs highlight the autoregressive loop, where each generated token influences the next. Used with permission granted by the lead author (Oriol Vinyals).
See how the encoder reads inputs into memory, iteratively processes them with attention, and writes outputs. section 5.6
