Recurrence gives sequence models an explicit inductive bias for order by making computation depend on sequential steps. In debates over Transformers, critics questioned whether positional encodings could replace this bias. section 5.1.2
Experiments complicated that objection. A four-layer Transformer trained on roughly 40,000 Penn Treebank Wall Street Journal sentences reached 91.3 F1 on constituency parsing, surpassing earlier RNN-based seq2seq models without task-specific architecture changes. On translation, Transformers also learned positional information better than recurrent models. section 5.1.1 section 5.1.2
Recurrence offers an elegant prior for sequence order, but these results suggest it is not required for strong sequence-task performance. Its costs can include added complexity, reduced parallelism, and weaker modeling of long-range dependencies. section 5.1.1 section 5.1.2
