liveBook · Manning

Relative positional embedding

As explained in Build a Large Language Model (From Scratch)

A position-encoding method that represents how far apart tokens are rather than their exact indices.

Represents token order by learning how far apart tokens are, rather than assigning each position a unique index embedding. It complements position-agnostic self-attention by injecting order information into input embeddings. section 2.8

Relative embeddings contrast with absolute positional embeddings: the model learns relationships in terms of distance, not exact slot number. That makes them useful for varying sequence lengths and generalization beyond training lengths. section 2.8

Shape walkthrough

A batch of token IDs with shape [8, 4] is embedded to [8, 4, 256], a positional table for four positions yields [4, 256], and adding them keeps the final input embeddings at [8, 4, 256].

How Build a Large Language Model (From Scratch) covers it

How deep it goes
Defines the idea and illustrates embedding addition with shapes, but stops short of any special relative-position algorithm.
Examples
Uses token-ID batches, embedding layers, positional tables, and shape traces through [8, 4, 256].
What it assumes
Assumes basic token IDs, embedding layers, tensor shapes, and the idea that self-attention alone is order-agnostic.
What you can do after
Apply it by adding learned position vectors to token embeddings before passing inputs to the main transformer layers.
How it connects
Connects self-attention, token embeddings, absolute positional embeddings, and input embeddings.
Cover of Build a Large Language Model (From Scratch)
liveBookmanning.com
Read Build a Large Language Model (From Scratch) on liveBook Get the book at manning.com

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Build a Large Language Model (From Scratch). The section links point to the parts of the book it draws on.