GloVe learns word vectors by directly optimizing global word co-occurrence statistics. It targets broadly shared word meanings, so the training corpus and intended users shape what the vectors represent. section 6.3.2 section 6.4
Instead of relying on neural-network backpropagation, GloVe counts co-occurrences in a square matrix and applies mature matrix-optimization methods related to singular value decomposition. This can reach useful representations efficiently, including with smaller corpora. section 6.4 section 6.4.1
Practical strengths include:
- Faster training and better RAM/CPU efficiency for larger documents. section 6.4.1
- More efficient use of data, helping models trained on smaller corpora. section 6.4.1
- Higher accuracy for the same amount of training, according to the stated comparison. section 6.4.1
Initializing a CNN embedding layer
A text-classification pipeline can load pretrained 50-dimensional GloVe vectors, align them with its vocabulary, assign zero vectors to missing tokens, and initialize a PyTorch nn.Embedding layer. Setting freeze=False allows the vectors to adapt to the target corpus. section 7.4.3
>>> import spacy
>>>
>>> nlp = spacy.load("en_core_web_sm")
>>> text = "This is an example sentence."
>>> doc = nlp(text)
>>>
>>> for token in doc:
... print(token.text, token.vector)
Inspect token vectors produced through spaCy, where the loaded language pipeline uses GloVe embeddings under the hood. section 6.4.1
