liveBook · Manning

Contrastive learning

As explained in Retrieval Augmented Generation, The Foundational Ideas

Contrastive learning trains representations by bringing related examples closer and pushing unrelated examples farther apart in representation space.

Contrastive learning shapes representation space directly by pulling semantically related examples together and pushing unrelated examples apart. In a Siamese or two-tower encoder, the same network processes each sentence independently, producing embeddings that can be compared after encoding. section 2.3.1

This architecture separates offline indexing from online search. Sentence embeddings can be computed once, stored, and compared with a dot product, replacing repeated pairwise processing with linear preprocessing and fast similarity calculations. Pooling, often token-embedding averaging, compresses variable-length text into a fixed-size vector. section 2.3.1

Contrastive retrieval with HyDE

HyDE generates a hypothetical document, then sends it and every corpus document through the same unsupervised contrastively learned encoder. The resulting vectors share one space, allowing retrieval without human-labeled query-document pairs. The encoder's lossy bottleneck can suppress incorrect details while preserving topical relevance, but it cannot reliably correct a fundamentally wrong question framing. section 7.3

Figure 2.4 BERT vs Sentence-BERT: Architecture comparison.

Use the architecture comparison to trace the shift from jointly processed BERT sentence pairs to independently encoded Sentence-BERT embeddings. section 2.3.1

How Retrieval Augmented Generation, The Foundational Ideas covers it

How deep it goes
Explains pairwise geometry, Siamese encoders, pooling, and indexing benefits, but does not specify a mathematical loss formula.
Examples
Uses Sentence-BERT similarity search and Contriever-powered HyDE retrieval as concrete applications.
What you can do after
Apply it via Siamese encoders and FAISS to precompute embeddings and perform fast similarity search.
How it connects
Connects Siamese networks, bi-encoders, embeddings, pooling, dense retrieval, FAISS, and HyDE.

Also explained in

Compare how 3 books explain contrastive learning →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Retrieval Augmented Generation, The Foundational Ideas. The section links point to the parts of the book it draws on.