Skip-gram trains word embeddings by using a context word to predict words nearby or skipped within the same sentence. It reverses the prediction direction of continuous bag of words (CBOW): CBOW predicts a target from context, while skip-gram predicts context from a target. Both create many small training examples from sliding windows. section 6.3.2
A practical skip-gram pipeline proceeds as follows:
- Collect domain text, such as medical journals, Wikipedia, or online news, because the corpus determines which word relationships the embeddings can learn. section 6.3.2
- Tokenize and normalize the text; a space-delimited corpus can use lowercasing and split-based tokenization. section 6.3.3
- Build a vocabulary and map each token to an integer so pairs can be encoded for the neural network. section 6.3.3
WikiText2 and a 10-token window
A runnable experiment loads 10,000 WikiText2 text rows, lowercases and splits each row, constructs a vocabulary, and generates pairs with WINDOW_WIDTH = 10. A Word2Vec model then maps vocabulary IDs into 100-dimensional vectors; the resulting dataset can be batched with DataLoader and trained on a CPU or available GPU.
