Order property is the information carried by token arrangement rather than token presence or count. Shuffling a sentence may preserve some clues while creating serious ambiguity, so models must decide whether and how to represent order. section 14.3
From words to sequences
Compare “the quick brown fox jumped over the lazy dog” with “the slow brown badger.” Tokenization turns each sentence into an integer sequence, then padding makes shorter sequences rectangular for batch computation. A model can discard order with bag-of-words, process tokens sequentially with an RNN, or inject positional information while comparing locations in parallel with a Transformer. section 14.3 section 14.5
Use the order property as a practical model-selection question:
- Discard it when token counts or presence are sufficient; choose a bag-of-words representation. section 14.3
- Preserve it when local or long-range positional dependencies affect meaning; use an RNN, 1D ConvNet, or Transformer. section 14.3 section 14.5
- Do not assume pooling preserves useful order: max pooling and global average pooling can destroy recency information in forecasting. section 13.2.4
