liveBook · Manning

Order property

As explained in Building Specialized AI Systems

Positional information is a representation of token order supplied to a sequence model.

Positional information supplies token order to Transformer models, which do not naturally know whether a token comes first, second, or later. It is combined with token embeddings before the first Transformer block. section 3.1.1

Common approaches differ in adaptability and length behavior:

  • Fixed positional patterns use sine and cosine functions. section 3.1.1
  • Learned positional representations adapt during training but are limited by the sequence lengths seen during training. section 3.1.1
  • Rotary position embeddings (RoPE) encode positions through geometric rotations to represent relative order efficiently for longer texts. section 3.1.1
Figure 3.3 The input pipeline before the Transformer block including the positional embedding. Raw text is tokenized into subword units and mapped to integer IDs, which are then looked up in a learned embedding matrix to produce dense token vectors. Positional embeddings encoding each token's position in the sequence are added to the token embeddings, producing the final input matrix passed to the first Transformer block.

Follow the input pipeline from raw text and subword tokenization through token embeddings, positional embeddings, and the matrix entering the first Transformer block. section 3.1.1

How Building Specialized AI Systems covers it

How deep it goes
Contrasts fixed, learned, and rotary methods, stopping short of implementation formulas or training procedures.
Examples
Uses a raw-text pipeline that maps subword tokens to vectors, adds positional embeddings, and passes the combined matrix into a Transformer block.
What it assumes
Builds from tokenization and token embeddings before introducing positional representations.
What you can do after
Evaluate positional methods by adaptability, training-length limits, and relative-order handling for longer texts.
How it connects
Connects tokenization, token embeddings, self-attention, Transformer blocks, and rotary position embeddings.

Also explained in

Compare how 10 books explain Order property →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Building Specialized AI Systems. The section links point to the parts of the book it draws on.