liveBook · Manning

Positional embedding

Also called positional encoding

A positional embedding is a numerical representation associated with an element’s position in a sequence and incorporated into its representation to convey position or order.

How 20 Manning books explain it

Natural Language Processing in Action, Second Edition

A positional embedding is a learned vector representing a token's position within an input sequence.

How it is used
Adds word-order information to transformer input representations for translation and BERT pretraining.
How deep it goes
Defines learned and fixed positional encodings and implements a sinusoidal variant, without detailing a learned implementation.
Examples
Uses a PyTorch translation transformer and BERT’s summed input representations to illustrate position information.
What you can do after
Applies positional encoding via PyTorch to source and target embeddings in a translation transformer.

Read how Natural Language Processing in Action, Second Edition explains positional embedding →

Build a Text-to-Image Generator (from Scratch)

A positional encoding is a representation added to sequence-element embeddings to convey their positions or ordering.

How it is used
Adds order information to transformer inputs for language translation and spatial position information to image-patch sequences.
How deep it goes
Defines fixed and learned vectors, works through matching tensor shapes, and implements their addition; stops short of deriving 3D geometry.
Examples
Uses a four-token German phrase, CIFAR-10 image patches with a class token, and PyTorch encoding classes.
What you can do after
Build it from scratch in PyTorch by adding fixed or learned position vectors to matching sequence embeddings.

Read how Build a Text-to-Image Generator (from Scratch) explains positional embedding →

Learn Generative AI with PyTorch

A positional-encoding method that learns position vectors as parameters rather than using fixed formulas.

How it is used
Uses learned position vectors to supply sequence order to decoder-only Transformers for text and music generation.
How deep it goes
Explains trainable position vectors through one-hot projection, tensor dimensions, and embedding-layer construction, contrasting them with fixed sinusoidal encoding.
Examples
Works through GPT-2XL’s four-token prompt and a music Transformer with 2,048 positions and 512-dimensional embeddings.
What you can do after
Builds a learned positional embedding layer with PyTorch and combines its output with token embeddings in a Transformer.

Read how Learn Generative AI with PyTorch explains positional embedding →

Build AI Drug Discovery Pipelines

A positional encoding is a vector representation of a token's location that is added to its content embedding to provide sequence-order information.

How it is used
Adds sequence-location information to residue embeddings before encoder self-attention in protein language modeling.
How deep it goes
Explains why order information is needed and describes fixed sinusoidal vectors, while noting that some encodings are learned.
Examples
Illustrates masked amino acid prediction in a protein sequence and declares a positional encoding module in a PyTorch model.
What you can do after
Evaluate/understand how positional information supports order-sensitive protein representations.

Read how Build AI Drug Discovery Pipelines explains positional embedding →

Sutskever's List

A positional encoding is a representation added to sequence-element embeddings to convey their positions to a model that otherwise lacks inherent order information.

How it is used
Supplies sequence-order information to Transformer self-attention without restoring recurrence or convolution.
How deep it goes
Explains placement, sinusoidal functions, and later variants, but leaves line-by-line implementation to an annotated Transformer reproduction.
Examples
Contrasts “cat fat” with “fat cat” and uses word-reordering tests to probe sensitivity to order.
What you can do after
Evaluate/understand why Transformers need position signals and why their presence alone does not guarantee sensitivity to word order.

Read how Sutskever's List explains positional embedding →

Build a DeepSeek Model (From Scratch)

Positional encoding is a technique that represents a token's sequence position and incorporates that information into its semantic representation.

How it is used
Introduces positional information to distinguish token occurrences and motivate positional awareness in attention.
How deep it goes
Develops integer, binary, and sinusoidal schemes through formulas, frequency patterns, and relative-position rotations, then contrasts additive encoding with RoPE.
Examples
Uses repeated instances of “dog,” an eight-bit encoding of position 200, and sinusoidal plots over 1024 positions.
What you can do after
Evaluate and understand positional encoding designs by comparing their magnitude, continuity, and integration trade-offs.

Read how Build a DeepSeek Model (From Scratch) explains positional embedding →

Build an Advanced RAG Application (From Scratch)

A representation added to sequence elements to encode their positions or order for a model that otherwise lacks recurrence.

How it is used
Supplies sequence-order information to parallel Transformer token processing.
How deep it goes
Defines positional encoding through its role after input embeddings, without specifying an encoding formula or implementation.
Examples
Illustrates autoregressive next-token prediction using embeddings, positional encoding, and multiple decoder blocks.
What you can do after
Evaluate/understand why Transformer models need explicit order information when processing tokens in parallel.

Read how Build an Advanced RAG Application (From Scratch) explains positional embedding →

Inside Deep Learning: Math, Algorithms, Models

A technique that injects token position information into embeddings using vectors derived from a position-dependent function.

How it is used
Adds sequence-order information to embeddings so models can use position without encoding it through recurrent or convolutional structure.
How deep it goes
Derives sinusoidal position vectors, implements a PyTorch module, and integrates it into classifiers; the implementation requires a preset maximum length.
Examples
Demonstrates sinusoidal curves, average- and attention-based classifiers, and transformer inputs.
What you can do after
Build it as a PyTorch module and insert it after an embedding layer or before transformer layers.

Read how Inside Deep Learning: Math, Algorithms, Models explains positional embedding →

Transformers in Action

Positional encoding is a representation that incorporates sequence-position information into neural-network inputs or hidden states.

How it is used
Provides token order for language transformers and spatial layout for flattened image patches.
How deep it goes
Explains sinusoidal encoding and its PyTorch implementation, while mentioning later positional improvements without detailing them.
Examples
Uses a sinusoidal encoding class and a vision transformer’s flattened image patches.
What you can do after
Build it from scratch with PyTorch and vary the inputs to explore the resulting plots.

Read how Transformers in Action explains positional embedding →

LLMs in Production

Positional encoding is a method of adding sequence-position information to token representations in a transformer model.

How it is used
Adds word-order information to token representations before transformer processing.
How deep it goes
Explains positional separation, semantic-similarity effects, and sine-cosine recovery without deriving the encoding functions.
Examples
Uses the red-and-brown sentence example, token positions, cosine-similarity changes, and an encoder architecture diagram.
What you can do after
Evaluate how transformer representations preserve meaning while distinguishing tokens by sequence position.

Read how LLMs in Production explains positional embedding →

Fight Fraud with Machine Learning

Positional encoding is information added to token representations to indicate the positions or order of tokens in a sequence.

How it is used
Supplies token order to transformer inputs during the walkthrough of language processing and generation.
How deep it goes
Defines positional encoding and illustrates its role, without detailing a formula or implementation procedure.
Examples
Uses the token sequence “let us fight fraud” to show vectors and their ordered relationships.
What you can do after
Evaluate/understand how transformers retain token order after words become embedding vectors.

Read how Fight Fraud with Machine Learning explains positional embedding →

Deep Learning with Python, Third Edition

A representation added to sequence elements to preserve order information for models that otherwise treat inputs as a set.

How it is used
Supplies word-order information to attention-based translation and text-generation models.
How deep it goes
Develops a learned position-embedding layer and compares it with raw position coordinates and cyclic encodings.
Examples
Uses English-to-Spanish translation and a decoder-only text-generation model to illustrate positional embeddings.
What you can do after
Build a learned positional embedding layer in Keras and apply it to Transformer inputs.

Read how Deep Learning with Python, Third Edition explains positional embedding →

Build a Large Language Model (From Scratch)

A position-encoding method that assigns a distinct embedding to each specific sequence position.

How it is used
Adds exact position information to token embeddings before the main LLM layers.
How deep it goes
Shows the definition and a tensor-shape walkthrough, but stops short of training details beyond learned embeddings.
Examples
Uses a four-token batch, embedding shapes, and a position-table addition example.
What you can do after
Apply it by adding learned position vectors to token embeddings in the input pipeline.

Read how Build a Large Language Model (From Scratch) explains positional embedding →

Deep Learning with PyTorch, Second Edition

An embedding that injects token-order information into transformer inputs because the architecture is not inherently sequential.

How it is used
Introduces position information into parallel transformer inputs and adapts sinusoidal embeddings to diffusion-model timesteps and coordinates.
How deep it goes
Explains learned position lookup and addition in a decoder, while omitting the sinusoidal embedding implementation.
Examples
Uses a character-level decoder, image-patch classification, and a 2D noise-prediction network as examples.
What you can do after
Apply it via PyTorch's nn.Embedding to add learned positions to transformer inputs.

Read how Deep Learning with PyTorch, Second Edition explains positional embedding →

Applied Reinforcement Learning

A representation added to inputs to encode their positions or identities so attention models can use ordering or spatial relationships.

How it is used
Injects zone-position information into Transformer inputs so attention can learn spatial relationships.
How deep it goes
Describes sinusoidal functions that distinguish zones and support spatial learning, without specifying their formula.
Examples
Uses data-center cooling zones, contrasting zone 1 with zone 5 and describing zone adjacency.
What you can do after
Understand how sinusoidal zone signals let a Transformer learn spatial relationships alongside state similarity.

Read how Applied Reinforcement Learning explains positional embedding →

Transfer Learning for Natural Language Processing

A method of representing sequence position by adding position-dependent information to token representations.

How it is used
Adds sequence-position information to token embeddings so transformers can represent order without recurrent components.
How deep it goes
Explains adding equal-sized, position-dependent vectors and notes sine-and-cosine generation, without detailing its formulas.
Examples
Uses a BERT sentence pair about cells and cell phones to visualize input embeddings, positional encodings, and segment encodings.
What you can do after
Understand how adding position-dependent vectors gives transformer inputs sequence-order information.

Read how Transfer Learning for Natural Language Processing explains positional embedding →

Grokking AI Algorithms, Second Edition

Positional encoding is a method of representing an item's position in a sequence with numerical values that can be combined with its content representation.

How it is used
Explains sequence order to self-attention by injecting positional signals into token representations.
How deep it goes
Works through sinusoidal calculations, vector addition, tables, and NumPy code, while deferring learned downstream interpretation.
Examples
Uses reordered sentences, an odometer analogy, a 32-token batch, four-dimensional vectors, and timestep 22.
What you can do after
Apply it via NumPy by generating sinusoidal rows and adding them to token embeddings.

Read how Grokking AI Algorithms, Second Edition explains positional embedding →

Deep Learning with R, Third Edition

A positional embedding is a representation that encodes an element's location in a sequence and is combined with its content representation.

How it is used
Restores word-order information when Transformers process sequences with attention.
How deep it goes
Explains learned and cosine-based encodings, then implements learned vectors but stops short of mathematical derivation.
Examples
Uses an R/Keras learned layer and an English-to-Spanish translation Transformer as concrete implementation examples.
What you can do after
Build it from scratch by adding learned position vectors to token embeddings in Transformer encoder and decoder inputs.

Read how Deep Learning with R, Third Edition explains positional embedding →

Rearchitecting LLMs

A positional embedding is a learned or fixed vector representation that supplies sequence-position information to a neural language model.

How it is used
Encodes sequence positions so input embeddings carry both token and position information.
How deep it goes
Explains the wpe layer, vector dimensions, position range, and addition to token embeddings.
Examples
Shows position vectors for positions 0–1023 combined with token embeddings.
What you can do after
Understand how positional vectors combine with token embeddings to form model inputs.

Read how Rearchitecting LLMs explains positional embedding →

Machine Learning Algorithms in Depth

A positional encoding is a representation added to token embeddings to convey a token's position in a sequence.

How it is used
Injects sequence-order information into token embeddings used by transformer encoder and decoder stacks.
How deep it goes
Explains purpose, placement, dimensional compatibility, and summation, but does not specify an encoding formula or implementation.
Examples
Uses encoder-decoder transformer stacks and the same word appearing at different sentence positions as illustrations.
What you can do after
Evaluate how transformers preserve sequence order when attention processes token representations.

Read how Machine Learning Algorithms in Depth explains positional embedding →

Also covered in

What to know first

Related concepts

Questions about positional embedding

What is positional embedding?

A positional embedding is a numerical representation associated with an element’s position in a sequence and incorporated into its representation to convey position or order.

Which Manning books explain positional embedding?

Natural Language Processing in Action, Second Edition, Build a Text-to-Image Generator (from Scratch), Learn Generative AI with PyTorch, Build AI Drug Discovery Pipelines, Sutskever's List, Build a DeepSeek Model (From Scratch), Build an Advanced RAG Application (From Scratch), Inside Deep Learning: Math, Algorithms, Models and 12 more.

What should I know before learning positional embedding?

Manning books that teach positional embedding usually introduce transformer architecture, attention, embedding and recurrent neural network first.

What is the difference between positional embedding and Rotary Position Embedding?

A positional embedding is a numerical representation associated with an element’s position in a sequence and incorporated into its representation to convey position or order. Rotary position embedding is a positional encoding method that incorporates position information by applying position-dependent rotations to components of attention query and key representations.

What kinds of positional embedding are there?

Kinds of positional embedding include sinusoidal positional encoding.

What is positional embedding also called?

Positional embedding is also called positional encoding.

These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.