A word can tell you something about its neighbors, and its neighbors can tell you something about the word. Word2Vec uses that idea to learn dense word vectors from a large collection of text. It creates its own training examples from nearby words, without requiring dictionary definitions or human-labeled meanings. Unlike a document-wide bag of words, it pays attention to a small window around each word. section 6.2.4 section 6.3
Training pairs link a target word with words in its surrounding window. A shallow prediction model learns vectors that help predict these occurrences. Continuous bag of words (CBOW) and skip-gram are two arrangements of this task; both turn local patterns of usage into numerical representations. Vector size depends on how much information the training corpus needs to capture, with examples commonly using hundreds of dimensions. section 6.3 section 6.3.2 section 6.3.3
Figure 6.5 Skip-gram neural network architecture
The skip-gram architecture makes the nearby-word prediction task concrete. section 6.3.2
To build vectors for a particular domain:
- Gather substantial text that reflects how your users employ the words. Medical journals, for example, may serve medical language better than general news. section 6.3.2 section 6.3.5
- Split documents into sentences and tokenize them. Keeping sentence boundaries prevents training pairs from linking unrelated words across adjacent sentences. section 6.3.3 section 6.3.5
- Build a vocabulary, map tokens to IDs, and generate pairs with a sliding window. Feed those pairs to a prediction model, such as the PyTorch example with an embedding layer and a vocabulary-sized output layer. section 6.3.3
- Alternatively, train with gensim, setting vector dimensions, minimum word count, window size, worker count, and subsampling; then save the model for later use. section 6.3.5
from torch import nn
EMBED_DIM = 100 #1
class Word2Vec(nn.Module):
def __init__(self, vocab_size=len(vocab), embedding_size=EMBED_DIM):
super().__init__()
self.embed = nn.Embedding(vocab_size, embedding_size) #2
self.expand = nn.Linear(embedding_size, vocab_size, bias=False)
def forward(self, input):
hidden = self.embed(input) #3
logits = self.expand(hidden) #4
return logits
A compact PyTorch model shows where learned embeddings sit in the prediction network. section 6.3.3
Reasoning with word vectors
A pretrained vector model can look for a word near the result of adding and subtracting other word vectors. For a city-and-soccer-team analogy, the proposed calculation is Portland Timbers + Seattle − Portland; the nearest vocabulary vector may point toward Seattle Sounders. With gensim, `most_similar(positive=['king', 'woman'], negative=['man'])` returns `queen` first in the supplied example. These are nearest-neighbor results, not exact symbolic deductions. section 6.3.1 section 6.3.4
Choose the corpus and vocabulary with care. Early word embeddings tend to represent an average or popular meaning rather than every person’s or community’s meaning. A Google News model may miss rare or specialized uses; limiting the loaded vocabulary saves memory but can leave words in your documents without vectors. Domain training may help, provided you have enough relevant text. section 6.3.2 section 6.3.4 section 6.3.5
The same proximity-based approach can also apply to meaningful sequences of symbols beyond English words, including identifiers, when enough varied context is available. The practical test is whether nearby tokens carry useful information about one another. section 6.4.7
