Input reversal

As explained in Sutskever's List

Input reversal is the technique of presenting a source sequence to a sequence model in reverse element order.

Input reversal presents a source sequence in reverse order while leaving the target unchanged. In an RNN encoder-decoder, this can shorten the path from early source words to the decoder’s first predictions, making their alignment easier to learn. section 5.7

Shortening the path

In French-to-English translation, the first French word may correspond to the first English word. Reversing the source places that French word later in the encoder’s processing, closer to the decoder’s initial state. The reported results included nearly five more BLEU points and perplexity falling from 5.8 to 4.7. section 5.7

Figure 5.10 The reversal trick shortens the path from relevant source tokens to the decoder’s first predictions. In a vanilla RNN encoder-decoder without attention (a), the decoder starts from the encoder’s final state. Reversing the source (b) makes tokens that matter for early outputs appear later in the encoder’s processing, so their influence reaches the decoder more directly.

Compare how reversing the source shortens the route from relevant tokens to the decoder’s first predictions. section 5.7

How Sutskever's List covers it

How deep it goes

Explains how source order affects token paths and gradient flow, without detailing an implementation.

Examples

Uses French-to-English first-word alignment and reports BLEU and perplexity changes after reversal.

What you can do after

Apply source reversal to shorten the effective path between early source words and initial target predictions.

How it connects

Connects sequence-to-sequence translation with recurrent neural networks, encoder-decoder models, and gradient flow.

liveBookmanning.com

Read Sutskever's List on liveBook Get the book at manning.com

Related concepts

• perplexityproduces · Perplexity is a language-model metric, typically the exponential of average cross-entropy, that measures how well a model predicts a sequence, with lower values indicating better predictive fit.
• BLEU scoreproduces · BLEU is an automatic text-generation evaluation metric that compares candidate text with reference text using modified n-gram precision and a brevity penalty.
• bidirectional encoderalternative to · A bidirectional encoder represents each sequence position using contextual information from both preceding and following positions.
• attentionalternative to · Attention is a neural-network mechanism that computes input-dependent weights to combine information according to its relevance to a computation.
• positional embeddingalternative to · A positional embedding is a learned or fixed representation that encodes an element’s position or order in a sequence and is incorporated into its content representation.

This explainer was generated by Manning's knowledge graph from the text of Sutskever's List. The section links point to the parts of the book it draws on.