A permutation-invariant encoder produces the same representation when the elements of its input are reordered. In Read-Process-Write (RPW), this lets the model handle unordered inputs; a specially designed loss can learn a suitable output ordering instead of fixing one in advance. section 5.6
Learning an ordering for 5-grams
In an illustrative 5-gram experiment, learning an ordering reduced perplexity from about 280 with a poor fixed order to about 225 with a learned near-optimal order. section 5.6
