A sparse vector stores one coordinate for each possible feature, while most coordinates remain zero or inactive. It resembles a dictionary: a document uses only a small fraction of the vocabulary. A dense vector instead compresses information into a smaller coordinate set in which every number carries information. section 2.1.5
Sparse representations shape traditional retrieval in several ways:
- One-hot encoding assigns each word a vocabulary-sized vector with one active coordinate, so words such as “car” and “automobile” have no mathematical relationship. section 2.1.1
- TF-IDF improves word counting by weighting terms according to frequency and corpus rarity, but still treats synonymous terms as unrelated. section 2.1.1
- Earlier neural models used expensive operations to compress sparse input into dense output; Word2Vec removed the nonlinear hidden layer and used negative sampling or hierarchical softmax for scale. section 2.1.5
A vocabulary-sized example
With a vocabulary containing “car,” “automobile,” and other words, “car” might be [0,0,1,0,0,...], while “automobile” might be [0,0,0,0,1,0,...]. Each vector is sparse because only one coordinate is active, but the representation does not encode their semantic similarity. section 2.1.1



