An input embedding combines token information with position information in a vector before contextual processing. In GPT-2, wte maps token IDs to vectors; wpe provides positional embeddings. The resulting embeddings enter the Transformer blocks. section 3.1.1
GPT-2 token lookup
The wte table has 50,257 entries, each a 768-dimensional vector; it maps token IDs from 0 to 50,256 to token embeddings. section 3.1.1
