Positional embedding

As explained in Learn Generative AI with PyTorch

A positional-encoding method that learns position vectors as parameters rather than using fixed formulas.

A positional embedding is a learned vector for a place in a sequence. A Transformer processes tokens in parallel, so token embeddings alone do not tell it their order. Adding a position vector to each token vector gives self-attention an input that carries both token and position information. section 11.1.2 section 9.1.2

The nearby alternative is fixed positional encoding: sine and cosine functions produce vectors that do not change during training. With learned positional embeddings, an embedding layer holds position vectors whose weights are refined during training. Both approaches add vectors with the same width as the token embeddings. section 10.2.2 section 11.1.2

Four-token GPT-2 prompt

For “this is a prompt,” GPT-2 assigns positions 0 through 3. In GPT-2XL, each position selects a learned 1,600-value vector from a layer covering up to 1,024 positions. The resulting 4 × 1,600 position matrix is added to the 4 × 1,600 token-embedding matrix. section 11.1.2

Figure 11.2 GPT-2 first represents each token in a sequence with a 50,276-value one-hot vector. The token representation of the sequence goes through a word embedding layer to compress it into an embedding with a dimension of 1,600. GPT-2 also represents each position in a sequence with a 1,024-value one-hot vector. The positional representation of the sequence goes through a positional encoding layer to compress it into an embedding also with a dimension of 1,600. The word embedding and positional encoding are added together to form the input embedding.

Follow the token and position paths separately, then see their equal-width vectors added to form the input embedding. section 11.1.2

In a PyTorch decoder-only Transformer:

• Create a position table with `nn.Embedding(config.block_size, config.n_embd)`: one available position per row, with the same vector width as the token embeddings. section 12.3.3 section 12.2.2
• Look up vectors for the sequence positions, add them to the token embeddings, and pass the sum into the decoder blocks. section 11.2.4
• Set the table dimensions for the task: one text generator uses 128 positions and 256-value vectors; a music Transformer uses 2,048 positions and 512-value vectors. section 12.3.3 section 14.3.2

How Learn Generative AI with PyTorch covers it

How deep it goes

Explains position vectors and their training, then shows embedding-layer dimensions and their placement in decoder-only models.

Examples

Uses a four-token GPT-2 prompt, a 128-position text generator, and a 2,048-position music Transformer.

What it assumes

Builds from token indexes and word embeddings; explains why parallel token processing needs separate position information.

What you can do after

Build it from scratch with a PyTorch position-embedding layer added to token embeddings before decoder blocks.

How it connects

Connects word embeddings, self-attention, decoder-only Transformers, and fixed sinusoidal positional encoding.

liveBookmanning.com

Read Learn Generative AI with PyTorch on liveBook Get the book at manning.com

Also explained in

• 

  liveBookmanning.com
  Natural Language Processing in Action, Second EditionA positional embedding is a learned vector representing a token's position within an input sequence.
• 

  liveBookmanning.com
  Build a Text-to-Image Generator (from Scratch)A positional encoding is a representation added to sequence-element embeddings to convey their positions or ordering.
• 

  liveBookmanning.com
  Sutskever's ListA positional encoding is a representation added to sequence-element embeddings to convey their positions to a model that otherwise lacks inherent order information.
• 

  liveBookmanning.com
  Transformers in ActionPositional encoding is a representation that incorporates sequence-position information into neural-network inputs or hidden states.
• 

  liveBookmanning.com
  LLMs in ProductionPositional encoding is a method of adding sequence-position information to token representations in a transformer model.
• 

  liveBookmanning.com
  Deep Learning with Python, Third EditionA representation added to sequence elements to preserve order information for models that otherwise treat inputs as a set.
• 

  liveBookmanning.com
  Build a Large Language Model (From Scratch)A position-encoding method that assigns a distinct embedding to each specific sequence position.
• 

  liveBookmanning.com
  Deep Learning with PyTorch, Second EditionAn embedding that injects token-order information into transformer inputs because the architecture is not inherently sequential.
• 

  liveBookmanning.com
  Grokking AI Algorithms, Second EditionPositional encoding is a method of representing an item's position in a sequence with numerical values that can be combined with its content representation.
• 

  liveBookmanning.com
  Deep Learning with R, Third EditionA positional embedding is a representation that encodes an element's location in a sequence and is combined with its content representation.
• 

  liveBookmanning.com
  Machine Learning Algorithms in DepthA positional encoding is a representation added to token embeddings to convey a token's position in a sequence.

Compare how 12 books explain positional embedding →

Related concepts

• embeddinguses · A numerical vector representation of an item or input that encodes features or relationships useful for computational tasks.
• transformer architectureused by · A transformer architecture is a neural-network architecture that processes sequences or other structured inputs using attention mechanisms and related transformations.
• sine functionuses · The sine function is a periodic trigonometric function that maps a real number or angle to a value between −1 and 1.
• cosine functionuses · The cosine function is a periodic trigonometric function that maps a real number or angle to a value between −1 and 1.
• Rotary Position Embeddingalternative to · Rotary Position Embedding is a positional encoding method that represents token positions by applying position-dependent rotations to components of attention query and key vectors.

This explainer was generated by Manning's knowledge graph from the text of Learn Generative AI with PyTorch. The section links point to the parts of the book it draws on.