Positional embedding
A positional embedding is a numerical representation associated with an element’s position in a sequence and incorporated into its representation to convey position or order.
Natural Language Processing in Action, Second Edition
A positional embedding is a learned vector representing a token's position within an input sequence.
- How it is used
- Adds word-order information to transformer input representations for translation and BERT pretraining.
- How deep it goes
- Defines learned and fixed positional encodings and implements a sinusoidal variant, without detailing a learned implementation.
- Examples
- Uses a PyTorch translation transformer and BERT’s summed input representations to illustrate position information.
- What you can do after
- Applies positional encoding via PyTorch to source and target embeddings in a translation transformer.
Read how Natural Language Processing in Action, Second Edition explains positional embedding →
Build a Text-to-Image Generator (from Scratch)
A positional encoding is a representation added to sequence-element embeddings to convey their positions or ordering.
- How it is used
- Adds order information to transformer inputs for language translation and spatial position information to image-patch sequences.
- How deep it goes
- Defines fixed and learned vectors, works through matching tensor shapes, and implements their addition; stops short of deriving 3D geometry.
- Examples
- Uses a four-token German phrase, CIFAR-10 image patches with a class token, and PyTorch encoding classes.
- What you can do after
- Build it from scratch in PyTorch by adding fixed or learned position vectors to matching sequence embeddings.
Read how Build a Text-to-Image Generator (from Scratch) explains positional embedding →
Learn Generative AI with PyTorch
A positional-encoding method that learns position vectors as parameters rather than using fixed formulas.
- How it is used
- Uses learned position vectors to supply sequence order to decoder-only Transformers for text and music generation.
- How deep it goes
- Explains trainable position vectors through one-hot projection, tensor dimensions, and embedding-layer construction, contrasting them with fixed sinusoidal encoding.
- Examples
- Works through GPT-2XL’s four-token prompt and a music Transformer with 2,048 positions and 512-dimensional embeddings.
- What you can do after
- Builds a learned positional embedding layer with PyTorch and combines its output with token embeddings in a Transformer.
Read how Learn Generative AI with PyTorch explains positional embedding →
Build AI Drug Discovery Pipelines
A positional encoding is a vector representation of a token's location that is added to its content embedding to provide sequence-order information.
- How it is used
- Adds sequence-location information to residue embeddings before encoder self-attention in protein language modeling.
- How deep it goes
- Explains why order information is needed and describes fixed sinusoidal vectors, while noting that some encodings are learned.
- Examples
- Illustrates masked amino acid prediction in a protein sequence and declares a positional encoding module in a PyTorch model.
- What you can do after
- Evaluate/understand how positional information supports order-sensitive protein representations.
Read how Build AI Drug Discovery Pipelines explains positional embedding →
Sutskever's List
A positional encoding is a representation added to sequence-element embeddings to convey their positions to a model that otherwise lacks inherent order information.
- How it is used
- Supplies sequence-order information to Transformer self-attention without restoring recurrence or convolution.
- How deep it goes
- Explains placement, sinusoidal functions, and later variants, but leaves line-by-line implementation to an annotated Transformer reproduction.
- Examples
- Contrasts “cat fat” with “fat cat” and uses word-reordering tests to probe sensitivity to order.
- What you can do after
- Evaluate/understand why Transformers need position signals and why their presence alone does not guarantee sensitivity to word order.
Build a DeepSeek Model (From Scratch)
Positional encoding is a technique that represents a token's sequence position and incorporates that information into its semantic representation.
- How it is used
- Introduces positional information to distinguish token occurrences and motivate positional awareness in attention.
- How deep it goes
- Develops integer, binary, and sinusoidal schemes through formulas, frequency patterns, and relative-position rotations, then contrasts additive encoding with RoPE.
- Examples
- Uses repeated instances of “dog,” an eight-bit encoding of position 200, and sinusoidal plots over 1024 positions.
- What you can do after
- Evaluate and understand positional encoding designs by comparing their magnitude, continuity, and integration trade-offs.
Read how Build a DeepSeek Model (From Scratch) explains positional embedding →
Build an Advanced RAG Application (From Scratch)
A representation added to sequence elements to encode their positions or order for a model that otherwise lacks recurrence.
- How it is used
- Supplies sequence-order information to parallel Transformer token processing.
- How deep it goes
- Defines positional encoding through its role after input embeddings, without specifying an encoding formula or implementation.
- Examples
- Illustrates autoregressive next-token prediction using embeddings, positional encoding, and multiple decoder blocks.
- What you can do after
- Evaluate/understand why Transformer models need explicit order information when processing tokens in parallel.
Read how Build an Advanced RAG Application (From Scratch) explains positional embedding →
Inside Deep Learning: Math, Algorithms, Models
A technique that injects token position information into embeddings using vectors derived from a position-dependent function.
- How it is used
- Adds sequence-order information to embeddings so models can use position without encoding it through recurrent or convolutional structure.
- How deep it goes
- Derives sinusoidal position vectors, implements a PyTorch module, and integrates it into classifiers; the implementation requires a preset maximum length.
- Examples
- Demonstrates sinusoidal curves, average- and attention-based classifiers, and transformer inputs.
- What you can do after
- Build it as a PyTorch module and insert it after an embedding layer or before transformer layers.
Read how Inside Deep Learning: Math, Algorithms, Models explains positional embedding →
Transformers in Action
Positional encoding is a representation that incorporates sequence-position information into neural-network inputs or hidden states.
- How it is used
- Provides token order for language transformers and spatial layout for flattened image patches.
- How deep it goes
- Explains sinusoidal encoding and its PyTorch implementation, while mentioning later positional improvements without detailing them.
- Examples
- Uses a sinusoidal encoding class and a vision transformer’s flattened image patches.
- What you can do after
- Build it from scratch with PyTorch and vary the inputs to explore the resulting plots.
Read how Transformers in Action explains positional embedding →
LLMs in Production
Positional encoding is a method of adding sequence-position information to token representations in a transformer model.
- How it is used
- Adds word-order information to token representations before transformer processing.
- How deep it goes
- Explains positional separation, semantic-similarity effects, and sine-cosine recovery without deriving the encoding functions.
- Examples
- Uses the red-and-brown sentence example, token positions, cosine-similarity changes, and an encoder architecture diagram.
- What you can do after
- Evaluate how transformer representations preserve meaning while distinguishing tokens by sequence position.
Fight Fraud with Machine Learning
Positional encoding is information added to token representations to indicate the positions or order of tokens in a sequence.
- How it is used
- Supplies token order to transformer inputs during the walkthrough of language processing and generation.
- How deep it goes
- Defines positional encoding and illustrates its role, without detailing a formula or implementation procedure.
- Examples
- Uses the token sequence “let us fight fraud” to show vectors and their ordered relationships.
- What you can do after
- Evaluate/understand how transformers retain token order after words become embedding vectors.
Read how Fight Fraud with Machine Learning explains positional embedding →
Deep Learning with Python, Third Edition
A representation added to sequence elements to preserve order information for models that otherwise treat inputs as a set.
- How it is used
- Supplies word-order information to attention-based translation and text-generation models.
- How deep it goes
- Develops a learned position-embedding layer and compares it with raw position coordinates and cyclic encodings.
- Examples
- Uses English-to-Spanish translation and a decoder-only text-generation model to illustrate positional embeddings.
- What you can do after
- Build a learned positional embedding layer in Keras and apply it to Transformer inputs.
Read how Deep Learning with Python, Third Edition explains positional embedding →
Build a Large Language Model (From Scratch)
A position-encoding method that assigns a distinct embedding to each specific sequence position.
- How it is used
- Adds exact position information to token embeddings before the main LLM layers.
- How deep it goes
- Shows the definition and a tensor-shape walkthrough, but stops short of training details beyond learned embeddings.
- Examples
- Uses a four-token batch, embedding shapes, and a position-table addition example.
- What you can do after
- Apply it by adding learned position vectors to token embeddings in the input pipeline.
Read how Build a Large Language Model (From Scratch) explains positional embedding →
Deep Learning with PyTorch, Second Edition
An embedding that injects token-order information into transformer inputs because the architecture is not inherently sequential.
- How it is used
- Introduces position information into parallel transformer inputs and adapts sinusoidal embeddings to diffusion-model timesteps and coordinates.
- How deep it goes
- Explains learned position lookup and addition in a decoder, while omitting the sinusoidal embedding implementation.
- Examples
- Uses a character-level decoder, image-patch classification, and a 2D noise-prediction network as examples.
- What you can do after
- Apply it via PyTorch's nn.Embedding to add learned positions to transformer inputs.
Read how Deep Learning with PyTorch, Second Edition explains positional embedding →
Applied Reinforcement Learning
A representation added to inputs to encode their positions or identities so attention models can use ordering or spatial relationships.
- How it is used
- Injects zone-position information into Transformer inputs so attention can learn spatial relationships.
- How deep it goes
- Describes sinusoidal functions that distinguish zones and support spatial learning, without specifying their formula.
- Examples
- Uses data-center cooling zones, contrasting zone 1 with zone 5 and describing zone adjacency.
- What you can do after
- Understand how sinusoidal zone signals let a Transformer learn spatial relationships alongside state similarity.
Read how Applied Reinforcement Learning explains positional embedding →
Transfer Learning for Natural Language Processing
A method of representing sequence position by adding position-dependent information to token representations.
- How it is used
- Adds sequence-position information to token embeddings so transformers can represent order without recurrent components.
- How deep it goes
- Explains adding equal-sized, position-dependent vectors and notes sine-and-cosine generation, without detailing its formulas.
- Examples
- Uses a BERT sentence pair about cells and cell phones to visualize input embeddings, positional encodings, and segment encodings.
- What you can do after
- Understand how adding position-dependent vectors gives transformer inputs sequence-order information.
Read how Transfer Learning for Natural Language Processing explains positional embedding →
Grokking AI Algorithms, Second Edition
Positional encoding is a method of representing an item's position in a sequence with numerical values that can be combined with its content representation.
- How it is used
- Explains sequence order to self-attention by injecting positional signals into token representations.
- How deep it goes
- Works through sinusoidal calculations, vector addition, tables, and NumPy code, while deferring learned downstream interpretation.
- Examples
- Uses reordered sentences, an odometer analogy, a 32-token batch, four-dimensional vectors, and timestep 22.
- What you can do after
- Apply it via NumPy by generating sinusoidal rows and adding them to token embeddings.
Read how Grokking AI Algorithms, Second Edition explains positional embedding →
Deep Learning with R, Third Edition
A positional embedding is a representation that encodes an element's location in a sequence and is combined with its content representation.
- How it is used
- Restores word-order information when Transformers process sequences with attention.
- How deep it goes
- Explains learned and cosine-based encodings, then implements learned vectors but stops short of mathematical derivation.
- Examples
- Uses an R/Keras learned layer and an English-to-Spanish translation Transformer as concrete implementation examples.
- What you can do after
- Build it from scratch by adding learned position vectors to token embeddings in Transformer encoder and decoder inputs.
Read how Deep Learning with R, Third Edition explains positional embedding →
Rearchitecting LLMs
A positional embedding is a learned or fixed vector representation that supplies sequence-position information to a neural language model.
- How it is used
- Encodes sequence positions so input embeddings carry both token and position information.
- How deep it goes
- Explains the wpe layer, vector dimensions, position range, and addition to token embeddings.
- Examples
- Shows position vectors for positions 0–1023 combined with token embeddings.
- What you can do after
- Understand how positional vectors combine with token embeddings to form model inputs.
Read how Rearchitecting LLMs explains positional embedding →
Machine Learning Algorithms in Depth
A positional encoding is a representation added to token embeddings to convey a token's position in a sequence.
- How it is used
- Injects sequence-order information into token embeddings used by transformer encoder and decoder stacks.
- How deep it goes
- Explains purpose, placement, dimensional compatibility, and summation, but does not specify an encoding formula or implementation.
- Examples
- Uses encoder-decoder transformer stacks and the same word appearing at different sentence positions as illustrations.
- What you can do after
- Evaluate how transformers preserve sequence order when attention processes token representations.
Read how Machine Learning Algorithms in Depth explains positional embedding →
Also covered in
- Knowledge Graphs and LLMs in Action
- How Large Language Models Work
- Time Series Forecasting Using Foundation Models
- Hugging Face in Action
- GPU Programming with TritonIn early access (MEAP)
- Machines that ThinkIn early access (MEAP)
What to know first
- transformer architectureA transformer is a neural-network architecture that uses attention-based interactions and position-wise transformations to model relationships among elements of sequences or other structured inputs.
- attentionAttention is a neural-network mechanism that computes input-dependent relevance weights to combine information from representations for a context-dependent computation.
- embeddingAn embedding is a numerical vector representation of an item in a feature space, encoding selected properties or relationships for computational tasks such as comparison or retrieval.
- recurrent neural networkA recurrent neural network is a neural network that processes ordered inputs while carrying and updating an internal state across successive steps.
Related concepts
- embeddinguses · An embedding is a numerical vector representation of an item in a feature space, encoding selected properties or relationships for computational tasks such as comparison or retrieval.
- transformer architectureused by · A transformer is a neural-network architecture that uses attention-based interactions and position-wise transformations to model relationships among elements of sequences or other structured inputs.
- sine functionuses · The sine function is a periodic trigonometric function that maps a real number or angle to a value between −1 and 1.
- cosine functionuses · The cosine function is a periodic trigonometric function that maps a real number or angle to a value between −1 and 1.
- Rotary Position Embeddingalternative to · Rotary position embedding is a positional encoding method that incorporates position information by applying position-dependent rotations to components of attention query and key representations.
- sinusoidal positional encodingincludes · Sinusoidal positional encoding represents sequence positions with fixed vectors whose components are sine and cosine functions of position at varying frequencies.
Questions about positional embedding
What is positional embedding?
A positional embedding is a numerical representation associated with an element’s position in a sequence and incorporated into its representation to convey position or order.
Which Manning books explain positional embedding?
Natural Language Processing in Action, Second Edition, Build a Text-to-Image Generator (from Scratch), Learn Generative AI with PyTorch, Build AI Drug Discovery Pipelines, Sutskever's List, Build a DeepSeek Model (From Scratch), Build an Advanced RAG Application (From Scratch), Inside Deep Learning: Math, Algorithms, Models and 12 more.
What should I know before learning positional embedding?
Manning books that teach positional embedding usually introduce transformer architecture, attention, embedding and recurrent neural network first.
What is the difference between positional embedding and Rotary Position Embedding?
A positional embedding is a numerical representation associated with an element’s position in a sequence and incorporated into its representation to convey position or order. Rotary position embedding is a positional encoding method that incorporates position information by applying position-dependent rotations to components of attention query and key representations.
What kinds of positional embedding are there?
Kinds of positional embedding include sinusoidal positional encoding.
What is positional embedding also called?
Positional embedding is also called positional encoding.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.



















