liveBook · Manning

Contextual embedding

As explained in Timeless Algorithms, The Foundational Ideas

A contextual representation is a numerical encoding of a token or sequence that incorporates information from its surrounding context.

A contextualized representation encodes a prompt after its tokens have been processed through transformer layers. The final prompt representation supports prediction, but relevant information can come from different places: token relationships within the prompt, or associations learned in the model’s parameters. For a factual question, attention helps represent the question and title; the title–author association is distributed across model parameters. The final representation is converted into vocabulary logits, then softmax assigns probabilities to possible continuations. section 12.2.2

A literary question

For “Who wrote The Count of Monte Cristo?”, the final prompt representation is used to score possible continuations such as “Dumas.” That association comes from patterns learned during training, not from the prompt directly attending to “Dumas” as a context token. section 12.2.2

hfinal=[0.9,0.4]

An illustrative final representation used in the vocabulary-scoring example. section 12.2.2

How Timeless Algorithms, The Foundational Ideas covers it

How deep it goes
Explains contextualization through transformer layers and distinguishes prompt relationships from associations learned in model parameters.
Examples
Uses the question about The Count of Monte Cristo to illustrate a learned title–author association.
What you can do after
Understand how a prompt representation contributes to predicting an answer based on learned associations.
How it connects
Links contextualized representations to transformer layers, attention, feed-forward layers, logits, and softmax.

Also explained in

Compare how 7 books explain contextual embedding →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Timeless Algorithms, The Foundational Ideas. The section links point to the parts of the book it draws on.