Order sensitivity is a model’s dependence on the arrangement of input elements. In sequence-to-sequence models, different orderings of the same set can count as different inputs. section 5.7
Reversing the source sentence
In sequence-to-sequence translation, reversing the source while keeping the target unchanged improved BLEU by nearly five points and lowered perplexity from 5.8 to 4.7. Reversal shortened the path from early source words to initial target predictions, making the optimization problem easier rather than making the model smarter. section 5.7
Figure 5.10 The reversal trick shortens the path from relevant source tokens to the decoder’s first predictions. In a vanilla RNN encoder-decoder without attention (a), the decoder starts from the encoder’s final state. Reversing the source (b) makes tokens that matter for early outputs appear later in the encoder’s processing, so their influence reaches the decoder more directly.
The diagram contrasts how reversing source tokens shortens their path to the decoder’s first predictions in an RNN encoder-decoder without attention. section 5.7
