A token embedding is the vector assigned to a token ID. In DistilGPT2, the wte layer maps tokenizer IDs to 768-dimensional vectors; positional embeddings are added afterward to form input embeddings. section 3.1.1
Figure 3.1 Embedding creation process in the input block. The wte layer converts each text token into a Token Embedding based on its meaning. The wpe layer generates a Positional Embedding based on the token's position in the sequence. The combination of both produces a vector that contains both semantic and positional information.
Shows token and positional embeddings combining to form an input embedding. section 3.1.1
