liveBook · Manning

Language modeling head

As explained in Build AI Drug Discovery Pipelines

A language modeling head is a model component that projects contextual token representations to scores or probabilities over a token vocabulary.

A language modeling head converts each contextual token representation into vocabulary-sized prediction scores. In a protein language model, it maps transformer outputs to logits for every amino-acid position, supporting masked-token prediction. section 12.3.3

Tracing tensor dimensions

For a batch of 4 sequences padded to length 50, with 128-dimensional representations and a 25-token vocabulary, the head transforms (4, 50, 128) into (4, 50, 25). Each position therefore receives one logit for every amino acid or special token. section 12.3.3

How Build AI Drug Discovery Pipelines covers it

How deep it goes
Explains the projection and output dimensions, while deferring probability conversion and full helper implementations.
Examples
Uses a protein-language-model dimension trace and per-position amino-acid predictions with ranked top-three tokens.
What you can do after
Build a vocabulary projection with nn.Linear and interpret logits for every sequence position.
How it connects
Connects embedding layers, positional encoding, Transformer encoders, self-attention, and masked language modeling.

Also explained in

Compare how 2 books explain language modeling head →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Build AI Drug Discovery Pipelines. The section links point to the parts of the book it draws on.