liveBook · Manning

[CLS] token

As explained in Retrieval Augmented Generation, The Foundational Ideas

The [CLS] token is a special BERT input token whose final hidden state aggregates the input sequence for classification.

The [CLS] token is a special BERT token placed at the beginning of an input. Its final hidden state aggregates the sequence for a classification head, producing an output such as a similarity score or entailment class. section 2.3

Paired-text input

For two sentences, BERT receives `[CLS] The cat sat on the mat. [SEP] A feline rested on the rug. [SEP]`. Self-attention processes both sentences jointly, and the [CLS] output summarizes their relationship for prediction. section 2.3

Use the [CLS] representation with an important limitation:

  • Treat it as an aggregate for a classification decision, not automatically as a standalone sentence embedding. section 2.3
  • Recognize that a cross-encoder output describes the specific pair of texts and returns a score rather than an independent representation. section 2.3

How Retrieval Augmented Generation, The Foundational Ideas covers it

How deep it goes
Defines placement and aggregation, then contrasts classification use with sentence-embedding geometry.
Examples
Uses a cat-and-feline sentence pair packed between [CLS] and [SEP] tokens.
What it assumes
Assumes familiarity with BERT, transformer processing, self-attention, and classification tasks.
What you can do after
Evaluate/understand when a [CLS] output suits pair classification and when it should not serve as a sentence embedding.
How it connects
Connects [CLS] with BERT, cross-encoders, self-attention, classification heads, and sentence embeddings.

Also explained in

Compare how 3 books explain [CLS] token →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Retrieval Augmented Generation, The Foundational Ideas. The section links point to the parts of the book it draws on.