A vector is a group of continuous numbers used to represent a token in a form a neural network can manipulate. Unlike a token’s fixed numeric identity, a vector can encode several aspects of meaning and relationships at once. section 3.0 section 3.2.1
Vectors matter because they give language models a workable representation:
- They use multiple numbers rather than one number, allowing a token to occupy a position in a multidimensional semantic space. section 3.2.1
- Nearby vectors can represent related meanings, while transformations between vectors can express recurring relationships. section 3.2.1
- A vector alone does not provide token order; positional information must be added so the transformer can distinguish different arrangements. section 3.2.1
- The model repeatedly converts tokens into numeric representations and later converts its numeric output back into tokens. section 3.0
Why one number is not enough
Suppose stock is assigned 5.2, capital 5.3, rare -5.2, and debt another opposing value. Treating opposites as simple negatives makes rare and debt appear related even though that relationship is nonsensical. Multiple dimensions provide more room to represent different meanings and relationships without forcing every similarity or opposition onto one line. section 3.2.1
Figure 3.3 If you use just one number to represent a token, you quickly encounter problems where similar/dissimilar words cannot be made to fit each other. Here we see how trying to represent simple synonym/antonym relationships quickly becomes nonsensical even with just a handful of words.
The figure makes the limitation of one-number representations visible by showing how synonym and antonym relationships quickly become inconsistent. section 3.2.1
