Contextual embedding
A contextual embedding is a numerical representation of an item whose values reflect information from its surrounding context.
LLM Evaluation and Alignment, The Foundational Ideas
A contextual embedding is a vector representation of a token whose value reflects the token's surrounding linguistic context.
- How it is used
- Enables reference-based semantic evaluation when generated text paraphrases rather than copies a reference.
- How deep it goes
- Explains context-sensitive vectors, self-attention, geometric similarity, and greedy token matching, while distinguishing semantic similarity from overall quality.
- Examples
- Contrasts financial and river senses of “bank,” reversed biting roles, and matching “attorney” with “lawyer.”
- What you can do after
- Evaluate/understand how contextual embeddings support semantic matching and why similarity scores alone cannot establish output quality.
Read how LLM Evaluation and Alignment, The Foundational Ideas explains contextual embedding →
Transfer Learning for Natural Language Processing
A vector representation of a linguistic item whose values depend on its surrounding context.
- How it is used
- Uses context-dependent vectors to distinguish meanings of words in NLP.
- How deep it goes
- Explains context dependence and ELMo’s bidirectional LSTM and next-word training.
- Examples
- Uses “duck” and “fair” as homographs whose meanings vary with context.
- What you can do after
- Understand how context-dependent vectors help distinguish a word’s possible meanings.
Read how Transfer Learning for Natural Language Processing explains contextual embedding →
Transformers in Action
A contextual embedding is a vector representation whose value reflects the surrounding context of the represented text element.
- How it is used
- Supports classification, semantic matching, and retrieval with representations sensitive to surrounding text.
- How deep it goes
- Defines context-dependent vectors, contrasts static embeddings, and demonstrates similarity search without detailing token-level computations.
- Examples
- Illustrates financial and riverbank uses of “bank” and compares query and document embeddings.
- What you can do after
- Apply it via SentenceTransformers to encode queries and documents and compare their embeddings with cosine similarity.
Read how Transformers in Action explains contextual embedding →
Human-in-the-Loop Machine Learning: Active learning and annotation for human-centered AI
A representation of an item whose content reflects its surrounding context, such as neighboring words or sentences.
- How it is used
- Supports transfer learning and diversity sampling when annotation effort is limited.
- How deep it goes
- Explains representation reuse, span extraction, and pooling choices without detailing how contextual embeddings are computed.
- Examples
- Uses predicted text spans and a faster-to-annotate related task to illustrate representation use.
- What you can do after
- Apply contextual embeddings to represent predicted spans for diversity sampling and reuse model representations as features for a related task.
Timeless Algorithms, The Foundational Ideas
A contextual representation is a numerical encoding of a token or sequence that incorporates information from its surrounding context.
- How it is used
- Uses the final prompt representation to score possible vocabulary continuations.
- How deep it goes
- Explains contextualization through transformer layers and distinguishes prompt relationships from associations learned in model parameters.
- Examples
- Uses the question about The Count of Monte Cristo to illustrate a learned title–author association.
- What you can do after
- Understand how a prompt representation contributes to predicting an answer based on learned associations.
Read how Timeless Algorithms, The Foundational Ideas explains contextual embedding →
Build an Advanced RAG Application (From Scratch)
A contextual embedding is a numerical representation of a token whose value reflects the token's surrounding sequence context.
- How it is used
- Represents tokens with surrounding context for downstream natural-language processing tasks.
- How deep it goes
- Explains bidirectional context, masked language modeling, and sentence-level extension, but does not detail embedding dimensions or implementation code.
- Examples
- Uses a masked chef-and-meal sentence and downstream tasks including sentiment analysis, named entity recognition, and question answering.
- What you can do after
- Evaluate/understand how bidirectional context produces token representations and how SBERT adapts them to sentence similarity.
Read how Build an Advanced RAG Application (From Scratch) explains contextual embedding →
The AI Pocket Book
Token representations that encode surrounding context before being used to produce output predictions.
- How it is used
- Uses contextualized representations as the intermediate input for next-token prediction.
- How deep it goes
- Explains attention, feed-forward updates, multilayer refinement, and multiheaded attention without deriving the computations.
- Examples
- Uses “bark,” “’s,” attention relevance, and vocabulary-wide token probabilities as concrete examples.
- What you can do after
- Evaluate how an LLM turns context-sensitive token representations into next-token probabilities.
Also covered in
- Machines that ThinkIn early access (MEAP)
- Natural Language Processing in Action, Second Edition
What to know first
- BERTBERT (Bidirectional Encoder Representations from Transformers) is a pretrained Transformer encoder language model that produces contextual text representations using information from both directions.
- embeddingAn embedding is a numerical vector representation of an item in a feature space, encoding selected properties or relationships for computational tasks such as comparison or retrieval.
- language modelA language model is a computational model that represents patterns in linguistic sequences to assign probabilities, predict, generate, or evaluate language.
- Word2VecWord2Vec is a family of shallow neural models that learns vector representations of words from their contexts in text.
Related concepts
- BERTScoreused by · BERTScore is an automatic text-generation evaluation metric that compares candidate and reference texts using contextual representations of their tokens.
- embeddingkind of · An embedding is a numerical vector representation of an item in a feature space, encoding selected properties or relationships for computational tasks such as comparison or retrieval.
- static embeddingcontrasts with · A static embedding is a fixed numerical vector representation of a word or token that does not vary with its context.
- BERTuses · BERT (Bidirectional Encoder Representations from Transformers) is a pretrained Transformer encoder language model that produces contextual text representations using information from both directions.
- contextconfigured by · Context is the relevant information, state, or circumstances surrounding an input, decision, or activity that influence its interpretation or behavior.
- FAISSused by · FAISS is an open-source software library for efficient similarity search and clustering of dense vectors.
Questions about contextual embedding
What is contextual embedding?
A contextual embedding is a numerical representation of an item whose values reflect information from its surrounding context.
Which Manning books explain contextual embedding?
LLM Evaluation and Alignment, The Foundational Ideas, Transfer Learning for Natural Language Processing, Transformers in Action, Human-in-the-Loop Machine Learning: Active learning and annotation for human-centered AI, Timeless Algorithms, The Foundational Ideas, Build an Advanced RAG Application (From Scratch) and The AI Pocket Book.
What should I know before learning contextual embedding?
Manning books that teach contextual embedding usually introduce BERT, embedding, language model and Word2Vec first.
What is the difference between contextual embedding and static embedding?
A contextual embedding is a numerical representation of an item whose values reflect information from its surrounding context. A static embedding is a fixed numerical vector representation of a word or token that does not vary with its context.
What is contextual embedding a kind of?
Contextual embedding is a kind of embedding. An embedding is a numerical vector representation of an item in a feature space, encoding selected properties or relationships for computational tasks such as comparison or retrieval.
What is contextual embedding also called?
Contextual embedding is also called contextualized embeddings, Contextual representation or contextualized representation.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.






