Contextualized representation
A contextualized representation is a numerical encoding of an item whose content depends on its surrounding sequence or other context.
Transformers in Action
A contextual embedding is a vector representation whose value reflects the surrounding context of the represented text element.
- How it is used
- Supports context-sensitive token meaning and similarity-based classification, semantic search, and retrieval.
- How deep it goes
- Defines the contrast with static vectors and explains bidirectional pretraining, but stops short of deriving layer-level vector computations.
- Examples
- Uses contrasting meanings of “bank” and code that encodes sentences, queries, and documents for similarity comparison.
- What you can do after
- Apply it via SentenceTransformer to encode text and compare query–document vectors with cosine similarity.
Read how Transformers in Action explains contextualized representation →
The AI Pocket Book
Token representations that encode surrounding context before being used to produce output predictions.
- How it is used
- Uses contextualized representations as the intermediate input for next-token prediction.
- How deep it goes
- Explains attention, feed-forward updates, multilayer refinement, and multiheaded attention without deriving the computations.
- Examples
- Uses “bark,” “’s,” attention relevance, and vocabulary-wide token probabilities as concrete examples.
- What you can do after
- Evaluate how an LLM turns context-sensitive token representations into next-token probabilities.
Read how The AI Pocket Book explains contextualized representation →
Related concepts
- BERTScoreused by · BERTScore is an automatic text-generation evaluation metric that compares candidate and reference texts using contextual representations of their tokens.
- embeddingkind of · A numerical vector representation of an item or input that encodes features or relationships useful for computational tasks.
- contextconfigured by · Context is the surrounding information, state, or circumstances that condition how an entity, event, or task is interpreted or how a system behaves.
- BERTuses · BERT is a pretrained Transformer encoder language model that uses bidirectional context to produce contextual representations of text.
- FAISSused by · FAISS is an open-source software library for efficient similarity search and clustering of dense vectors.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.