One-hot encoding turns a token’s numerical ID into a vector. For a vocabulary of N tokens, each vector has N entries: one entry is 1 at the token’s index, and all others are 0. section 2.5.1
A token’s vector
In a 10,000-token vocabulary, “dog” at index 42 has a vector of length 10,000 with a 1 at position 42 and zeros elsewhere. section 2.5.1
The representation makes tokens usable in mathematical operations, but it does not encode similarity: “dog” is no closer to “cat” than to “spoon.” Its vocabulary-sized vectors are also sparse, using space inefficiently. Dense vectors, including embeddings, offer a later alternative that can represent similarity. section 2.5.1









