Positional embedding

As explained in Machine Learning Algorithms in Depth

A positional encoding is a representation added to token embeddings to convey a token's position in a sequence.

Positional embeddings inject token-position information into transformers, whose attention mechanism otherwise needs explicit sequence-order information. They are added to input token embeddings in both encoder and decoder stacks. section 11.3

The construction follows three constraints:

• Match the embedding dimensions so positional and token representations can be summed. section 11.3
• Make an identical word’s representation differ when its sequence position changes. section 11.3
• Support order-aware processing alongside self-attention, multihead attention, residual connections, and normalization layers. section 11.3

Figure 11.10 Transformer architecture

The transformer architecture diagram locates positional encodings at the input bottoms of the encoder and decoder stacks. section 11.3

How Machine Learning Algorithms in Depth covers it

How deep it goes

Explains purpose, placement, dimensional compatibility, and summation, but does not specify an encoding formula or implementation.

Examples

Uses encoder-decoder transformer stacks and the same word appearing at different sentence positions as illustrations.

What it assumes

Assumes familiarity with token embeddings and transformer attention while leaving encoding construction unexplained.

What you can do after

Evaluate how transformers preserve sequence order when attention processes token representations.

How it connects

Connects token embeddings with self-attention, multihead attention, transformers, encoder-decoder architecture, and layer normalization.

liveBookmanning.com

Read Machine Learning Algorithms in Depth on liveBook Get the book at manning.com

Also explained in

• 

  liveBookmanning.com
  Natural Language Processing in Action, Second EditionA positional embedding is a learned vector representing a token's position within an input sequence.
• 

  liveBookmanning.com
  Build a Text-to-Image Generator (from Scratch)A positional encoding is a representation added to sequence-element embeddings to convey their positions or ordering.
• 

  liveBookmanning.com
  Learn Generative AI with PyTorchA positional-encoding method that learns position vectors as parameters rather than using fixed formulas.
• 

  liveBookmanning.com
  Sutskever's ListA positional encoding is a representation added to sequence-element embeddings to convey their positions to a model that otherwise lacks inherent order information.
• 

  liveBookmanning.com
  Transformers in ActionPositional encoding is a representation that incorporates sequence-position information into neural-network inputs or hidden states.
• 

  liveBookmanning.com
  LLMs in ProductionPositional encoding is a method of adding sequence-position information to token representations in a transformer model.
• 

  liveBookmanning.com
  Deep Learning with Python, Third EditionA representation added to sequence elements to preserve order information for models that otherwise treat inputs as a set.
• 

  liveBookmanning.com
  Build a Large Language Model (From Scratch)A position-encoding method that assigns a distinct embedding to each specific sequence position.
• 

  liveBookmanning.com
  Deep Learning with PyTorch, Second EditionAn embedding that injects token-order information into transformer inputs because the architecture is not inherently sequential.
• 

  liveBookmanning.com
  Grokking AI Algorithms, Second EditionPositional encoding is a method of representing an item's position in a sequence with numerical values that can be combined with its content representation.
• 

  liveBookmanning.com
  Deep Learning with R, Third EditionA positional embedding is a representation that encodes an element's location in a sequence and is combined with its content representation.

Compare how 12 books explain positional embedding →

Related concepts

• embeddinguses · A numerical vector representation of an item or input that encodes features or relationships useful for computational tasks.
• transformer architectureused by · A transformer architecture is a neural-network architecture that processes sequences or other structured inputs using attention mechanisms and related transformations.
• sine functionuses · The sine function is a periodic trigonometric function that maps a real number or angle to a value between −1 and 1.
• cosine functionuses · The cosine function is a periodic trigonometric function that maps a real number or angle to a value between −1 and 1.
• Rotary Position Embeddingalternative to · Rotary Position Embedding is a positional encoding method that represents token positions by applying position-dependent rotations to components of attention query and key vectors.

This explainer was generated by Manning's knowledge graph from the text of Machine Learning Algorithms in Depth. The section links point to the parts of the book it draws on.