liveBook · Manning

Sparse vector

As explained in Retrieval Augmented Generation, The Foundational Ideas

A sparse vector is a vector in which most coordinates are zero or otherwise inactive.

A sparse vector stores one coordinate for each possible feature, while most coordinates remain zero or inactive. It resembles a dictionary: a document uses only a small fraction of the vocabulary. A dense vector instead compresses information into a smaller coordinate set in which every number carries information. section 2.1.5

Sparse representations shape traditional retrieval in several ways:

  • One-hot encoding assigns each word a vocabulary-sized vector with one active coordinate, so words such as “car” and “automobile” have no mathematical relationship. section 2.1.1
  • TF-IDF improves word counting by weighting terms according to frequency and corpus rarity, but still treats synonymous terms as unrelated. section 2.1.1
  • Earlier neural models used expensive operations to compress sparse input into dense output; Word2Vec removed the nonlinear hidden layer and used negative sampling or hierarchical softmax for scale. section 2.1.5

A vocabulary-sized example

With a vocabulary containing “car,” “automobile,” and other words, “car” might be [0,0,1,0,0,...], while “automobile” might be [0,0,0,0,1,0,...]. Each vector is sparse because only one coordinate is active, but the representation does not encode their semantic similarity. section 2.1.1

How Retrieval Augmented Generation, The Foundational Ideas covers it

How deep it goes
Defines sparsity through zero coordinates and contrasts it with dense vectors, without detailing storage or indexing internals.
Examples
Uses one-hot vectors for “car” and “automobile” and compares dictionary-like sparse coordinates with compressed dense coordinates.
What you can do after
Evaluate why sparse word representations support exact matching but fail to encode synonymy without semantic transformation.
How it connects
Connects sparse vectors to one-hot encoding, TF-IDF, BM25, dense vectors, Word2Vec, and semantic retrieval.

Also explained in

Compare how 4 books explain sparse vector →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Retrieval Augmented Generation, The Foundational Ideas. The section links point to the parts of the book it draws on.