Positional information records where each element occurs in a sequence. Positional encoding adds a vector P(t) for position t to the token embedding h_t, combining content with relative order so faster models can use sequence information without depending entirely on an RNN or CNN architecture. section 12.4 section 12.7
A practical implementation follows this sequence:
- Represent each token as an embedding h_t in R^D, producing h_1, h_2, ..., h_T. section 12.4
- Create a position vector for each location t; sinusoidal functions provide the position-dependent values, with different dimensions using different frequencies. section 12.4
- Add each position vector to the embedding at the same location, then pass the combined sequence to the remaining model. section 12.4
- Use convolutions when local order is sufficient, or add positional information to faster embedding-based alternatives when the model otherwise loses sequence order. section 12.3 section 12.7
Visualizing position frequencies
Plot sin(position), then compare it with sin(position/10). The slower oscillation distinguishes positions differently; extending this idea across embedding dimensions creates multiple positional signals. A six-dimensional NumPy example computes frequency scaling with exp and plots one curve per dimension. section 12.4
Figure 12.4 Average embedding approach that we started with, augmented with positional embeddings. Separate from the inputs, vector values encode information about our location within a larger sequence. This is added to the vectors representing the inputs, creating a mix of content and sequence information.
Shows positional vectors added to average-embedding inputs, making content and sequence location visible as one combined representation. section 12.4
