liveBook · Manning

[CLS] token

As explained in Build AI Drug Discovery Pipelines

A CLS token is a special Transformer input token whose contextual representation can summarize an entire sequence for classification.

A [CLS] token is a special Transformer input token prepended to a sequence. It represents no amino acid; through attention, its final hidden state can gather information from every position and summarize the sequence for classification. section 12.3.4

Use [CLS] pooling as one of several ways to convert per-position representations into one sequence vector:

  • Use the final [CLS] representation when the model has learned to collect sequence-wide information there. section 12.3.4
  • Use mean pooling to average real-token representations, masking padding so meaningless values do not dilute the result. section 12.3.4
  • Use max pooling to emphasize the strongest activation at each representation dimension. section 12.3.4

How Build AI Drug Discovery Pipelines covers it

How deep it goes
Defines attention-based summarization and contrasts mean and max pooling, but does not develop a [CLS]-based implementation.
Examples
Contrasts [CLS] pooling with masked mean pooling and max pooling for antibody-sequence classification.
What it assumes
Assumes familiarity with Transformer token representations and attention; establishes why classification needs one vector per sequence.
What you can do after
Evaluate sequence-pooling choices when adapting a per-position protein language model for classification.
How it connects
Connects Transformer attention, sequence classification, mean pooling, max pooling, attention masks, and classification heads.

Also explained in

Compare how 3 books explain [CLS] token →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Build AI Drug Discovery Pipelines. The section links point to the parts of the book it draws on.