liveBook · Manning

Positional embedding

As explained in Build a Large Language Model (From Scratch)

A position-encoding method that assigns a distinct embedding to each specific sequence position.

Absolute positional embedding assigns each sequence position its own learned embedding, then adds that vector to the token embedding so the model can see exact order. Self-attention is position-agnostic by itself, so this extra signal supplies location information. section 2.8

The treatment follows the input pipeline: tokens become token IDs, token IDs become token embeddings, and a matching positional table is added before the main LLM layers. GPT models use learned absolute positional embeddings rather than fixed positional encodings here. section 2.8

Embedding addition

For a sequence of length 4, a position embedding vector exists for position 0, 1, 2, and 3. Each vector has the same dimension as the token embedding, so the two can be added elementwise.

How Build a Large Language Model (From Scratch) covers it

How deep it goes
Shows the definition and a tensor-shape walkthrough, but stops short of training details beyond learned embeddings.
Examples
Uses a four-token batch, embedding shapes, and a position-table addition example.
What it assumes
Assumes tokenization, token IDs, and embedding layers, then introduces positional embeddings from zero.
What you can do after
Apply it by adding learned position vectors to token embeddings in the input pipeline.
How it connects
Connects to token embeddings, self-attention, relative positional embeddings, and GPT models.
Cover of Build a Large Language Model (From Scratch)
liveBookmanning.com
Read Build a Large Language Model (From Scratch) on liveBook Get the book at manning.com

Also explained in

Compare how 12 books explain positional embedding →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Build a Large Language Model (From Scratch). The section links point to the parts of the book it draws on.