liveBook · Manning

Positional embedding

As explained in Rearchitecting LLMs

A positional embedding is a learned or fixed vector representation that supplies sequence-position information to a neural language model.

A positional embedding represents where a token occurs in a sequence. In DistilGPT2, the wpe layer supplies a 768-dimensional vector for each position from 0 to 1023; the model adds it to the token embedding to form an input embedding that carries both token and position information. section 3.1.1

Figure 3.1 Embedding creation process in the input block. The wte layer converts each text token into a Token Embedding based on its meaning. The wpe layer generates a Positional Embedding based on the token's position in the sequence. The combination of both produces a vector that contains both semantic and positional information.

Trace how token embeddings and positional embeddings combine into a single input representation. section 3.1.1

How Rearchitecting LLMs covers it

How deep it goes
Explains the wpe layer, vector dimensions, position range, and addition to token embeddings.
Examples
Shows position vectors for positions 0–1023 combined with token embeddings.
What you can do after
Understand how positional vectors combine with token embeddings to form model inputs.
How it connects
Connects token embeddings, input embeddings, and the context window.

Also explained in

Compare how 20 books explain positional embedding →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Rearchitecting LLMs. The section links point to the parts of the book it draws on.