Positional embedding

As explained in LLMs in Production

Positional encoding is a method of adding sequence-position information to token representations in a transformer model.

Token embeddings capture what a token means, but not where it appears. Positional embedding adds sequence-position information so a transformer can distinguish identical tokens in different locations while preserving useful semantic similarity. section 2.3.1 section 2.1.1

Separating similar words by position

In “The cat in the hat rapidly leapt above the red fox and the brown unmotivated dog,” “red” and “brown” are semantically similar but occur at positions 10 and 14. Encoding separates their representations, while sine and cosine transformations restore much of their similarity. section 2.3.1

The technique works alongside embeddings and self-attention, allowing encoder models such as BERT to manipulate token vectors without losing the semantic comparisons established during embedding. section 2.3.1

Figure 2.5 An encoder visualized. Encoders are the first half of the full transformer architecture and excel in natural language understanding tasks like classification or named entity recognition. Encoder models improve upon previous designs by not requiring priors or recurrence and using clever positional encoding and multihead attention to create a vector embedding of each token.

Use the encoder diagram to see positional encoding placed among token embeddings and multihead attention. section 2.3.1

How LLMs in Production covers it

How deep it goes

Explains positional separation, semantic-similarity effects, and sine-cosine recovery without deriving the encoding functions.

Examples

Uses the red-and-brown sentence example, token positions, cosine-similarity changes, and an encoder architecture diagram.

What it assumes

Assumes basic tokenization and vector embeddings while explaining how positional information supplements them.

What you can do after

Evaluate how transformer representations preserve meaning while distinguishing tokens by sequence position.

How it connects

Connects tokenization, embeddings, semantic embeddings, self-attention, encoder models, and BERT.

liveBookmanning.com

Read LLMs in Production on liveBook Get the book at manning.com

Also explained in

• 

  liveBookmanning.com
  Natural Language Processing in Action, Second EditionA positional embedding is a learned vector representing a token's position within an input sequence.
• 

  liveBookmanning.com
  Build a Text-to-Image Generator (from Scratch)A positional encoding is a representation added to sequence-element embeddings to convey their positions or ordering.
• 

  liveBookmanning.com
  Learn Generative AI with PyTorchA positional-encoding method that learns position vectors as parameters rather than using fixed formulas.
• 

  liveBookmanning.com
  Sutskever's ListA positional encoding is a representation added to sequence-element embeddings to convey their positions to a model that otherwise lacks inherent order information.
• 

  liveBookmanning.com
  Transformers in ActionPositional encoding is a representation that incorporates sequence-position information into neural-network inputs or hidden states.
• 

  liveBookmanning.com
  Deep Learning with Python, Third EditionA representation added to sequence elements to preserve order information for models that otherwise treat inputs as a set.
• 

  liveBookmanning.com
  Build a Large Language Model (From Scratch)A position-encoding method that assigns a distinct embedding to each specific sequence position.
• 

  liveBookmanning.com
  Deep Learning with PyTorch, Second EditionAn embedding that injects token-order information into transformer inputs because the architecture is not inherently sequential.
• 

  liveBookmanning.com
  Grokking AI Algorithms, Second EditionPositional encoding is a method of representing an item's position in a sequence with numerical values that can be combined with its content representation.
• 

  liveBookmanning.com
  Deep Learning with R, Third EditionA positional embedding is a representation that encodes an element's location in a sequence and is combined with its content representation.
• 

  liveBookmanning.com
  Machine Learning Algorithms in DepthA positional encoding is a representation added to token embeddings to convey a token's position in a sequence.

Compare how 12 books explain positional embedding →

Related concepts

• embeddinguses · A numerical vector representation of an item or input that encodes features or relationships useful for computational tasks.
• transformer architectureused by · A transformer architecture is a neural-network architecture that processes sequences or other structured inputs using attention mechanisms and related transformations.
• sine functionuses · The sine function is a periodic trigonometric function that maps a real number or angle to a value between −1 and 1.
• cosine functionuses · The cosine function is a periodic trigonometric function that maps a real number or angle to a value between −1 and 1.
• Rotary Position Embeddingalternative to · Rotary Position Embedding is a positional encoding method that represents token positions by applying position-dependent rotations to components of attention query and key vectors.

This explainer was generated by Manning's knowledge graph from the text of LLMs in Production. The section links point to the parts of the book it draws on.