liveBook · Manning

Contextual embedding

Also called contextualized embeddings, Contextual representation, contextualized representation

A contextual embedding is a numerical representation of an item whose values reflect information from its surrounding context.

How 7 Manning books explain it

LLM Evaluation and Alignment, The Foundational Ideas

A contextual embedding is a vector representation of a token whose value reflects the token's surrounding linguistic context.

How it is used
Enables reference-based semantic evaluation when generated text paraphrases rather than copies a reference.
How deep it goes
Explains context-sensitive vectors, self-attention, geometric similarity, and greedy token matching, while distinguishing semantic similarity from overall quality.
Examples
Contrasts financial and river senses of “bank,” reversed biting roles, and matching “attorney” with “lawyer.”
What you can do after
Evaluate/understand how contextual embeddings support semantic matching and why similarity scores alone cannot establish output quality.

Read how LLM Evaluation and Alignment, The Foundational Ideas explains contextual embedding →

Transfer Learning for Natural Language Processing

A vector representation of a linguistic item whose values depend on its surrounding context.

How it is used
Uses context-dependent vectors to distinguish meanings of words in NLP.
How deep it goes
Explains context dependence and ELMo’s bidirectional LSTM and next-word training.
Examples
Uses “duck” and “fair” as homographs whose meanings vary with context.
What you can do after
Understand how context-dependent vectors help distinguish a word’s possible meanings.

Read how Transfer Learning for Natural Language Processing explains contextual embedding →

Transformers in Action

A contextual embedding is a vector representation whose value reflects the surrounding context of the represented text element.

How it is used
Supports classification, semantic matching, and retrieval with representations sensitive to surrounding text.
How deep it goes
Defines context-dependent vectors, contrasts static embeddings, and demonstrates similarity search without detailing token-level computations.
Examples
Illustrates financial and riverbank uses of “bank” and compares query and document embeddings.
What you can do after
Apply it via SentenceTransformers to encode queries and documents and compare their embeddings with cosine similarity.

Read how Transformers in Action explains contextual embedding →

Human-in-the-Loop Machine Learning: Active learning and annotation for human-centered AI

A representation of an item whose content reflects its surrounding context, such as neighboring words or sentences.

How it is used
Supports transfer learning and diversity sampling when annotation effort is limited.
How deep it goes
Explains representation reuse, span extraction, and pooling choices without detailing how contextual embeddings are computed.
Examples
Uses predicted text spans and a faster-to-annotate related task to illustrate representation use.
What you can do after
Apply contextual embeddings to represent predicted spans for diversity sampling and reuse model representations as features for a related task.

Read how Human-in-the-Loop Machine Learning: Active learning and annotation for human-centered AI explains contextual embedding →

Timeless Algorithms, The Foundational Ideas

A contextual representation is a numerical encoding of a token or sequence that incorporates information from its surrounding context.

How it is used
Uses the final prompt representation to score possible vocabulary continuations.
How deep it goes
Explains contextualization through transformer layers and distinguishes prompt relationships from associations learned in model parameters.
Examples
Uses the question about The Count of Monte Cristo to illustrate a learned title–author association.
What you can do after
Understand how a prompt representation contributes to predicting an answer based on learned associations.

Read how Timeless Algorithms, The Foundational Ideas explains contextual embedding →

Build an Advanced RAG Application (From Scratch)

A contextual embedding is a numerical representation of a token whose value reflects the token's surrounding sequence context.

How it is used
Represents tokens with surrounding context for downstream natural-language processing tasks.
How deep it goes
Explains bidirectional context, masked language modeling, and sentence-level extension, but does not detail embedding dimensions or implementation code.
Examples
Uses a masked chef-and-meal sentence and downstream tasks including sentiment analysis, named entity recognition, and question answering.
What you can do after
Evaluate/understand how bidirectional context produces token representations and how SBERT adapts them to sentence similarity.

Read how Build an Advanced RAG Application (From Scratch) explains contextual embedding →

The AI Pocket Book

Token representations that encode surrounding context before being used to produce output predictions.

How it is used
Uses contextualized representations as the intermediate input for next-token prediction.
How deep it goes
Explains attention, feed-forward updates, multilayer refinement, and multiheaded attention without deriving the computations.
Examples
Uses “bark,” “’s,” attention relevance, and vocabulary-wide token probabilities as concrete examples.
What you can do after
Evaluate how an LLM turns context-sensitive token representations into next-token probabilities.

Read how The AI Pocket Book explains contextual embedding →

Also covered in

What to know first

Related concepts

Questions about contextual embedding

What is contextual embedding?

A contextual embedding is a numerical representation of an item whose values reflect information from its surrounding context.

Which Manning books explain contextual embedding?

LLM Evaluation and Alignment, The Foundational Ideas, Transfer Learning for Natural Language Processing, Transformers in Action, Human-in-the-Loop Machine Learning: Active learning and annotation for human-centered AI, Timeless Algorithms, The Foundational Ideas, Build an Advanced RAG Application (From Scratch) and The AI Pocket Book.

What should I know before learning contextual embedding?

Manning books that teach contextual embedding usually introduce BERT, embedding, language model and Word2Vec first.

What is the difference between contextual embedding and static embedding?

A contextual embedding is a numerical representation of an item whose values reflect information from its surrounding context. A static embedding is a fixed numerical vector representation of a word or token that does not vary with its context.

What is contextual embedding a kind of?

Contextual embedding is a kind of embedding. An embedding is a numerical vector representation of an item in a feature space, encoding selected properties or relationships for computational tasks such as comparison or retrieval.

What is contextual embedding also called?

Contextual embedding is also called contextualized embeddings, Contextual representation or contextualized representation.

These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.