A positional embedding represents where a token occurs in a sequence. In DistilGPT2, the wpe layer supplies a 768-dimensional vector for each position from 0 to 1023; the model adds it to the token embedding to form an input embedding that carries both token and position information. section 3.1.1
Figure 3.1 Embedding creation process in the input block. The wte layer converts each text token into a Token Embedding based on its meaning. The wpe layer generates a Positional Embedding based on the token's position in the sequence. The combination of both produces a vector that contains both semantic and positional information.
Trace how token embeddings and positional embeddings combine into a single input representation. section 3.1.1



















