liveBook · Manning

Input embedding

As explained in Rearchitecting LLMs

An input embedding is a vector representation of a token that combines information about the token and its position before contextual processing.

An input embedding combines token information with position information in a vector before contextual processing. In GPT-2, wte maps token IDs to vectors; wpe provides positional embeddings. The resulting embeddings enter the Transformer blocks. section 3.1.1

GPT-2 token lookup

The wte table has 50,257 entries, each a 768-dimensional vector; it maps token IDs from 0 to 50,256 to token embeddings. section 3.1.1

How Rearchitecting LLMs covers it

How deep it goes
Explains token lookup dimensions and positional combination, but leaves the combination operation unspecified.
Examples
Uses GPT-2's 50,257-entry, 768-dimensional token embedding table and diagrams token-position combination.
What it assumes
Requires token IDs from an external tokenizer; explains their lookup but not tokenization itself.
What you can do after
Understand how token IDs and positions become vectors supplied to Transformer blocks.
How it connects
Connects tokenization, token embeddings, positional embeddings, dropout, and Transformer blocks.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Rearchitecting LLMs. The section links point to the parts of the book it draws on.