liveBook · Manning

Positional embedding

As explained in Build an Advanced RAG Application (From Scratch)

A representation added to sequence elements to encode their positions or order for a model that otherwise lacks recurrence.

Positional embedding, also called positional encoding, supplies order information to token representations. A Transformer processes tokens in parallel, so token embeddings alone do not intrinsically indicate which word came first, next, or last. Adding a distinct positional code gives the model a map of the sequence while retaining parallel processing. section 2.2

The encoding follows the input embedding step: tokens first become dense vectors, then receive positional information. This lets the model distinguish the sequential arrangement of words, an essential part of language understanding that recurrence would otherwise provide through sequential processing. section 2.2

Within Transformer models, positional embeddings work alongside other components:

  • Self-attention relates words across a sequence, while positional encoding keeps track of their order. section 2.5
  • Multi-head attention provides broader views of token relationships; positional information preserves where those relationships occur in the sequence. section 2.5
  • Decoder models use token embeddings, positional encodings, decoder layers, masked self-attention, and feed-forward networks for sequential generation. section 5.1

Autoregressive next-token generation

A decoder-only language model processes preceding tokens to predict the next token. Positional encodings maintain the order of those earlier tokens as embeddings pass through decoder blocks, supporting causal generation with masked self-attention. section 5.1

Figure 5.1 Autoregressive Language Models predict the next word by processing a sequence of previous words. They use embedding, positional encoding, and multiple decoder blocks to generate a probability distribution for the next word.

The diagram places positional encoding between token embedding and decoder blocks in an autoregressive next-word prediction flow. section 5.1

How Build an Advanced RAG Application (From Scratch) covers it

How deep it goes
Defines positional encoding through its role after input embeddings, without specifying an encoding formula or implementation.
Examples
Illustrates autoregressive next-token prediction using embeddings, positional encoding, and multiple decoder blocks.
What it assumes
Introduces positional encoding from dense token embeddings and contrasts it with recurrent sequential processing.
What you can do after
Evaluate/understand why Transformer models need explicit order information when processing tokens in parallel.
How it connects
Connects positional encoding with token embeddings, self-attention, multi-head attention, encoder models, and decoder models.

Also explained in

Compare how 20 books explain positional embedding →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Build an Advanced RAG Application (From Scratch). The section links point to the parts of the book it draws on.