Positional embedding

As explained in Grokking AI Algorithms, Second Edition

Positional encoding is a method of representing an item's position in a sequence with numerical values that can be combined with its content representation.

Positional embedding represents where an item occurs in a sequence and combines that signal with the item’s content representation. This matters because self-attention processes tokens simultaneously: without position information, “The dog bit the man” and “The man bit the dog” look alike. section 11.15

A sinusoidal positional encoding assigns each position a vector of sine and cosine values. Every position receives a distinct pattern, while nearby positions receive similar patterns; the vector width can be adjusted to the model’s embedding width. section 11.15

Combining meaning and location

For a token at position 1, a four-value positional vector can be [0.8415, 0.5403, 0.0100, 1.0000]. Adding it element by element to the token embedding produces one vector containing both what the token means and where it occurs. For example, [-0.008, 0.005, -0.002, 0.009] becomes [0.8335, 0.5453, 0.0080, 1.0090]. section 11.16

A simple implementation follows this sequence:

• Look up each token’s embedding, add its positional row, and pass the resulting matrix into the Transformer. section 11.16

The same sinusoidal idea can represent a diffusion model’s timestep. Encoding a value such as t = 22 creates a vector that tells the model how much noise is present, so it can adjust denoising strength. section 12.10

Figure 11.29 An odometer analogy showing how sinusoidal positional encoding works

Use the odometer analogy to see how several changing numerical signals can jointly identify a sequence position. section 11.15

How Grokking AI Algorithms, Second Edition covers it

How deep it goes

Works through sinusoidal calculations, vector addition, tables, and NumPy code, while deferring learned downstream interpretation.

Examples

Uses reordered sentences, an odometer analogy, a 32-token batch, four-dimensional vectors, and timestep 22.

What it assumes

Assumes token embeddings and introduces how positional signals supply the missing order information for self-attention.

What you can do after

Apply it via NumPy by generating sinusoidal rows and adding them to token embeddings.

How it connects

Connects positional encoding to token embeddings, self-attention, Transformers, and timestep embeddings.

liveBookmanning.com

Read Grokking AI Algorithms, Second Edition on liveBook Get the book at manning.com

Also explained in

• 

  liveBookmanning.com
  Natural Language Processing in Action, Second EditionA positional embedding is a learned vector representing a token's position within an input sequence.
• 

  liveBookmanning.com
  Build a Text-to-Image Generator (from Scratch)A positional encoding is a representation added to sequence-element embeddings to convey their positions or ordering.
• 

  liveBookmanning.com
  Learn Generative AI with PyTorchA positional-encoding method that learns position vectors as parameters rather than using fixed formulas.
• 

  liveBookmanning.com
  Sutskever's ListA positional encoding is a representation added to sequence-element embeddings to convey their positions to a model that otherwise lacks inherent order information.
• 

  liveBookmanning.com
  Transformers in ActionPositional encoding is a representation that incorporates sequence-position information into neural-network inputs or hidden states.
• 

  liveBookmanning.com
  LLMs in ProductionPositional encoding is a method of adding sequence-position information to token representations in a transformer model.
• 

  liveBookmanning.com
  Deep Learning with Python, Third EditionA representation added to sequence elements to preserve order information for models that otherwise treat inputs as a set.
• 

  liveBookmanning.com
  Build a Large Language Model (From Scratch)A position-encoding method that assigns a distinct embedding to each specific sequence position.
• 

  liveBookmanning.com
  Deep Learning with PyTorch, Second EditionAn embedding that injects token-order information into transformer inputs because the architecture is not inherently sequential.
• 

  liveBookmanning.com
  Deep Learning with R, Third EditionA positional embedding is a representation that encodes an element's location in a sequence and is combined with its content representation.
• 

  liveBookmanning.com
  Machine Learning Algorithms in DepthA positional encoding is a representation added to token embeddings to convey a token's position in a sequence.

Compare how 12 books explain positional embedding →

Related concepts

• embeddinguses · A numerical vector representation of an item or input that encodes features or relationships useful for computational tasks.
• transformer architectureused by · A transformer architecture is a neural-network architecture that processes sequences or other structured inputs using attention mechanisms and related transformations.
• sine functionuses · The sine function is a periodic trigonometric function that maps a real number or angle to a value between −1 and 1.
• cosine functionuses · The cosine function is a periodic trigonometric function that maps a real number or angle to a value between −1 and 1.
• Rotary Position Embeddingalternative to · Rotary Position Embedding is a positional encoding method that represents token positions by applying position-dependent rotations to components of attention query and key vectors.

This explainer was generated by Manning's knowledge graph from the text of Grokking AI Algorithms, Second Edition. The section links point to the parts of the book it draws on.