liveBook · Manning

Encoder-only transformer

As explained in LLM Evaluation and Alignment, The Foundational Ideas

An encoder-only Transformer is a Transformer model composed of bidirectional encoder layers for representing an entire input sequence.

An encoder-only Transformer represents a complete input sequence with bidirectional context. Unlike autoregressive decoders, it can use later words to interpret earlier ones, making it useful for comparing existing texts rather than generating new text. section 3.2.2

Semantic paraphrase comparison

The committee-approved proposal and the passage describing committee approval after lengthy discussion express similar meaning despite different wording.

How LLM Evaluation and Alignment, The Foundational Ideas covers it

How deep it goes
Explains bidirectional context and embedding extraction, then stops short of detailing Transformer layer internals.
Examples
Uses legal-context interpretation, paraphrased committee passages, [CLS] pooling, and mean pooling.
What it assumes
Assumes basic text processing and vector familiarity while demonstrating tokenization, pooling, and similarity from implementation examples.
What you can do after
Builds text embeddings with BERT and compares them using cosine similarity.
How it connects
Connects BERTScore, COMET, embeddings, cosine similarity, natural language understanding, and natural language inference.

Also explained in

Compare how 7 books explain encoder-only transformer →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of LLM Evaluation and Alignment, The Foundational Ideas. The section links point to the parts of the book it draws on.