Contextualized representation

As explained in The AI Pocket Book

Token representations that encode surrounding context before being used to produce output predictions.

A contextualized representation is a token representation updated with information from surrounding tokens. It gives the model a better representation of the prompt before prediction. section 1.20

From token to context

For a token such as “bark,” attention weighs nearby tokens by relevance. Information from relevant words shapes its embedding, while an irrelevant token such as “’s” contributes little. Feed-forward processing then produces an improved representation. The same process contextualizes every token in the prompt. section 1.20

The representation is built in stages:

• Attention compares tokens within the context window and determines their relative relevance. section 1.20
• Feed-forward processing updates the embeddings, and repeated transformer layers refine them further. section 1.20
• Projections over the resulting representations produce probabilities for every possible next token. section 1.21

Figure 1.10 The attention mechanism calculates the relative relevance of all tokens in the context window to contextualize or disambiguate the last token.

Use the attention diagram to see how tokens in the context window help contextualize or disambiguate another token. section 1.20

How The AI Pocket Book covers it

How deep it goes

Explains attention, feed-forward updates, multilayer refinement, and multiheaded attention without deriving the computations.

Examples

Uses “bark,” “’s,” attention relevance, and vocabulary-wide token probabilities as concrete examples.

What it assumes

Assumes familiarity with tokens and embeddings while explaining contextualization from the attention process onward.

What you can do after

Evaluate how an LLM turns context-sensitive token representations into next-token probabilities.

How it connects

Connects attention, feed-forward processing, transformer layers, embeddings, deep learning, and next-token prediction.

liveBookmanning.com

Read The AI Pocket Book on liveBook Get the book at manning.com

Also explained in

• 

  liveBookmanning.com
  Transformers in ActionA contextual embedding is a vector representation whose value reflects the surrounding context of the represented text element.

Compare how 2 books explain contextualized representation →

Related concepts

• BERTScoreused by · BERTScore is an automatic text-generation evaluation metric that compares candidate and reference texts using contextual representations of their tokens.
• embeddingkind of · A numerical vector representation of an item or input that encodes features or relationships useful for computational tasks.
• contextconfigured by · Context is the surrounding information, state, or circumstances that condition how an entity, event, or task is interpreted or how a system behaves.
• BERTuses · BERT is a pretrained Transformer encoder language model that uses bidirectional context to produce contextual representations of text.
• FAISSused by · FAISS is an open-source software library for efficient similarity search and clustering of dense vectors.

This explainer was generated by Manning's knowledge graph from the text of The AI Pocket Book. The section links point to the parts of the book it draws on.