liveBook · Manning

Negative sampling

As explained in Retrieval Augmented Generation, The Foundational Ideas

Negative sampling trains a predictive model by contrasting observed positive examples with a sampled set of negative examples.

Negative sampling makes large-vocabulary training practical by replacing a full probability calculation with a small set of binary checks. For each observed context word, the model asks whether it is the true context or a randomly drawn negative example. section 2.1.2

The baseline problem is the output sum: a vocabulary of 692,000 words requires considering every word at each step. Hierarchical softmax reduces that work through a binary-tree walk, while negative sampling avoids requiring a probability distribution altogether. section 2.1.2

Negative sampling is useful when training must scale:

  • It asks only a handful of yes-or-no questions instead of evaluating the entire vocabulary. section 2.1.2
  • It helped Word2Vec train on datasets 100–1,000 times larger than earlier approaches, including more than 100 billion words in a day. section 2.1.5
  • It supports dense-vector training for large vocabularies, which later enabled embeddings at production retrieval scale. section 2.1.5

A single training decision

Given an observed context word, retain it as the positive example and draw a handful of other words as negatives. Train the model to distinguish the true context from those random alternatives rather than scoring all 692,000 vocabulary entries. section 2.1.2

How Retrieval Augmented Generation, The Foundational Ideas covers it

How deep it goes
Explains the full-vocabulary bottleneck and sampled binary alternative, but does not detail sampling distributions or optimization equations.
Examples
Uses a true context word versus randomly drawn words and contrasts millions of vocabulary evaluations with a handful of binary questions.
What it assumes
Assumes familiarity with vocabularies, context words, and dense vectors while contrasting negative sampling with hierarchical softmax.
What you can do after
Apply it to understand how Word2Vec scales training across millions or billions of words.
How it connects
Connects negative sampling to Word2Vec, hierarchical softmax, dense vectors, embeddings, and retrieval.

Also explained in

Compare how 4 books explain negative sampling →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Retrieval Augmented Generation, The Foundational Ideas. The section links point to the parts of the book it draws on.