An embedding maps each word in a corpus to a vector in a predefined space. Unlike one-hot encoding, it uses word usage and context to place similarly used words near one another, capturing meaning and relationships with fewer dimensions. section 7.13
Embedding methods address key limitations of one-hot and bag-of-words representations:
- Reduce extremely high-dimensional, sparse representations. section 7.13
- Preserve relationships between words with similar usage or meaning. section 7.13
- Represent each word with a vector whose dimensions may number in the tens or hundreds rather than the vocabulary size. section 7.13
Word2Vec learns vectors from a large text corpus: CBOW predicts a target word from surrounding words, while skip-gram predicts surrounding words from the current word. GloVe combines matrix factorization with local-context learning and derives vectors from word co-occurrence. Word2Vec is prediction-based; GloVe is frequency-based. section 7.14
Relationships represented by vectors
Embedding geometry can reflect relationships such as man–woman paralleling king–queen, good–nice paralleling bad–awful, and UK–London paralleling Japan–Tokyo. section 7.13
Figure 7.16 Word embeddings can be used to represent the relationships between words. For example, there is a relation from “men” to “women” that is similar to “king” to “queen” as both “men-women” and “king-queen” represent the male-female gender relationship.
Use the relationship diagram to inspect how vector directions can express recurring semantic relationships between word pairs. section 7.13
