Positional embedding
A positional embedding is a learned or fixed representation that encodes an element’s position or order in a sequence and is incorporated into its content representation.
Natural Language Processing in Action, Second Edition
A positional embedding is a learned vector representing a token's position within an input sequence.
- How it is used
- Supplies token locations to transformer inputs so attention can use word order without recurrence.
- How deep it goes
- Defines position-vector addition and explains fixed sinusoidal encoding with a PyTorch module; mentions learned encodings without detailing their training.
- Examples
- Uses a PyTorch sinusoidal module, source and target preparation in a translation transformer, and BERT’s three-part input representation.
- What you can do after
- Build it from scratch with PyTorch PositionalEncoding and apply it to source and target embeddings in a translation transformer.
Read how Natural Language Processing in Action, Second Edition explains positional embedding →
Build a Text-to-Image Generator (from Scratch)
A positional encoding is a representation added to sequence-element embeddings to convey their positions or ordering.
- How it is used
- Adds order information to transformer inputs for language translation and spatial position information to image-patch sequences.
- How deep it goes
- Defines fixed and learned vectors, works through matching tensor shapes, and implements their addition; stops short of deriving 3D geometry.
- Examples
- Uses a four-token German phrase, CIFAR-10 image patches with a class token, and PyTorch encoding classes.
- What you can do after
- Build it from scratch in PyTorch by adding fixed or learned position vectors to matching sequence embeddings.
Read how Build a Text-to-Image Generator (from Scratch) explains positional embedding →
Learn Generative AI with PyTorch
A positional-encoding method that learns position vectors as parameters rather than using fixed formulas.
- How it is used
- Adds sequence-order information to token embeddings before self-attention in text and music Transformers.
- How deep it goes
- Explains position vectors and their training, then shows embedding-layer dimensions and their placement in decoder-only models.
- Examples
- Uses a four-token GPT-2 prompt, a 128-position text generator, and a 2,048-position music Transformer.
- What you can do after
- Build it from scratch with a PyTorch position-embedding layer added to token embeddings before decoder blocks.
Read how Learn Generative AI with PyTorch explains positional embedding →
Sutskever's List
A positional encoding is a representation added to sequence-element embeddings to convey their positions to a model that otherwise lacks inherent order information.
- How it is used
- Supplies sequence-order information to Transformer self-attention without restoring recurrence or convolution.
- How deep it goes
- Explains placement, sinusoidal functions, and later variants, but leaves line-by-line implementation to an annotated Transformer reproduction.
- Examples
- Contrasts “cat fat” with “fat cat” and uses word-reordering tests to probe sensitivity to order.
- What you can do after
- Evaluate/understand why Transformers need position signals and why their presence alone does not guarantee sensitivity to word order.
Transformers in Action
Positional encoding is a representation that incorporates sequence-position information into neural-network inputs or hidden states.
- How it is used
- Supplies token order to transformers and spatial arrangement to flattened image-patch sequences.
- How deep it goes
- Explains embedding addition and works through a sinusoidal PyTorch implementation, without specifying a construction for visual positional embeddings.
- Examples
- Uses a PyTorch PositionalEncoding class and a vision-transformer diagram of patch, classification, and positional embeddings.
- What you can do after
- Build it from scratch with PyTorch and inspect how changing inputs affects positional-encoding plots.
Read how Transformers in Action explains positional embedding →
LLMs in Production
Positional encoding is a method of adding sequence-position information to token representations in a transformer model.
- How it is used
- Adds word-order information to token representations before transformer processing.
- How deep it goes
- Explains positional separation, semantic-similarity effects, and sine-cosine recovery without deriving the encoding functions.
- Examples
- Uses the red-and-brown sentence example, token positions, cosine-similarity changes, and an encoder architecture diagram.
- What you can do after
- Evaluate how transformer representations preserve meaning while distinguishing tokens by sequence position.
Deep Learning with Python, Third Edition
A representation added to sequence elements to preserve order information for models that otherwise treat inputs as a set.
- How it is used
- Supplies word-order information to attention-based Transformers used for translation and text generation.
- How deep it goes
- Works through learned position-vector lookup and addition in Keras, comparing raw indices and cyclic vectors without deriving the latter.
- Examples
- Uses a Keras PositionalEmbedding layer, an English-to-Spanish translation model, and a decoder-only text generator.
- What you can do after
- Build it from scratch as a Keras layer and apply it to source and target inputs in a Transformer translation model.
Read how Deep Learning with Python, Third Edition explains positional embedding →
Build a Large Language Model (From Scratch)
A position-encoding method that assigns a distinct embedding to each specific sequence position.
- How it is used
- Adds exact position information to token embeddings before the main LLM layers.
- How deep it goes
- Shows the definition and a tensor-shape walkthrough, but stops short of training details beyond learned embeddings.
- Examples
- Uses a four-token batch, embedding shapes, and a position-table addition example.
- What you can do after
- Apply it by adding learned position vectors to token embeddings in the input pipeline.
Read how Build a Large Language Model (From Scratch) explains positional embedding →
Deep Learning with PyTorch, Second Edition
An embedding that injects token-order information into transformer inputs because the architecture is not inherently sequential.
- How it is used
- Supplies position information to parallel-processed transformer tokens so their order remains available to the model.
- How deep it goes
- Defines learned and fixed sinusoidal variants and works through learned PyTorch addition, but omits the sinusoidal embedding implementation.
- Examples
- Shows a PyTorch decoder adding position vectors, image-patch location embeddings, and sinusoidal embeddings for diffusion timesteps and coordinates.
- What you can do after
- Build it from scratch using PyTorch nn.Embedding to add learned position vectors to token embeddings in a decoder.
Read how Deep Learning with PyTorch, Second Edition explains positional embedding →
Grokking AI Algorithms, Second Edition
Positional encoding is a method of representing an item's position in a sequence with numerical values that can be combined with its content representation.
- How it is used
- Explains sequence order to self-attention by injecting positional signals into token representations.
- How deep it goes
- Works through sinusoidal calculations, vector addition, tables, and NumPy code, while deferring learned downstream interpretation.
- Examples
- Uses reordered sentences, an odometer analogy, a 32-token batch, four-dimensional vectors, and timestep 22.
- What you can do after
- Apply it via NumPy by generating sinusoidal rows and adding them to token embeddings.
Read how Grokking AI Algorithms, Second Edition explains positional embedding →
Deep Learning with R, Third Edition
A positional embedding is a representation that encodes an element's location in a sequence and is combined with its content representation.
- How it is used
- Restores word-order information when Transformers process sequences with attention.
- How deep it goes
- Explains learned and cosine-based encodings, then implements learned vectors but stops short of mathematical derivation.
- Examples
- Uses an R/Keras learned layer and an English-to-Spanish translation Transformer as concrete implementation examples.
- What you can do after
- Build it from scratch by adding learned position vectors to token embeddings in Transformer encoder and decoder inputs.
Read how Deep Learning with R, Third Edition explains positional embedding →
Machine Learning Algorithms in Depth
A positional encoding is a representation added to token embeddings to convey a token's position in a sequence.
- How it is used
- Injects sequence-order information into token embeddings used by transformer encoder and decoder stacks.
- How deep it goes
- Explains purpose, placement, dimensional compatibility, and summation, but does not specify an encoding formula or implementation.
- Examples
- Uses encoder-decoder transformer stacks and the same word appearing at different sentence positions as illustrations.
- What you can do after
- Evaluate how transformers preserve sequence order when attention processes token representations.
Read how Machine Learning Algorithms in Depth explains positional embedding →
Related concepts
- embeddinguses · A numerical vector representation of an item or input that encodes features or relationships useful for computational tasks.
- transformer architectureused by · A transformer architecture is a neural-network architecture that processes sequences or other structured inputs using attention mechanisms and related transformations.
- sine functionuses · The sine function is a periodic trigonometric function that maps a real number or angle to a value between −1 and 1.
- cosine functionuses · The cosine function is a periodic trigonometric function that maps a real number or angle to a value between −1 and 1.
- Rotary Position Embeddingalternative to · Rotary Position Embedding is a positional encoding method that represents token positions by applying position-dependent rotations to components of attention query and key vectors.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.