Context vector
A context vector is a numerical vector that summarizes information from relevant input context for a model’s prediction or subsequent processing.
Build a DeepSeek Model (From Scratch)
A context vector is a numerical representation summarizing information relevant to a model's prediction at a particular sequence position.
- How it is used
- Uses the newest context vector to motivate efficient next-token inference and the key-value cache.
- How deep it goes
- Derives context matrices from attention weights and values, then traces the newest vector back to its query, keys, and values.
- Examples
- Works through “The next day is” and its generated token “bright” using explicit matrix dimensions.
- What you can do after
- Evaluate how attention constructs a context vector and why cached keys and values reduce repeated inference work.
Read how Build a DeepSeek Model (From Scratch) explains context vector →
Sutskever's List
A context vector is a weighted combination of source representations that summarizes the input information selected by an attention mechanism at a decoding step.
- How it is used
- Summarizes source information selected by attention at each decoding step in translation.
- How deep it goes
- Defines the context vector as a weighted source summary but does not derive or implement its computation.
- Examples
- Illustrates attention aligning “zone” with “Area” before revisiting earlier words in “European Economic Area.”
- What you can do after
- Understand how attention selects source annotations while generating each target word.
Build AI Drug Discovery Pipelines
A context vector is the weighted sum of value representations produced by an attention operation for a particular output step.
- How it is used
- Uses attention to give each prediction access to relevant information across a long protein sequence.
- How deep it goes
- Defines the weighted-sum result and relates it to encoder states, queries, keys, values, and attention weights.
- Examples
- Uses a 500-amino-acid protein and hidden states from positions 50, 200, and 400 to illustrate distributed sequence information.
- What you can do after
- Understand why output-specific weighted summaries preserve information that a single final hidden state can lose.
Read how Build AI Drug Discovery Pipelines explains context vector →
Also covered in
What to know first
- recurrent neural networkA recurrent neural network is a neural network that processes ordered inputs while carrying and updating an internal state across successive steps.
- softmaxSoftmax is a function that exponentiates each value in a score vector and normalizes the results to produce positive values summing to one, forming a probability distribution.
- LSTMA long short-term memory (LSTM) network is a recurrent neural-network architecture with gated memory cells that control information flow and model dependencies in sequential data.
- transformer architectureA transformer is a neural-network architecture that uses attention-based interactions and position-wise transformations to model relationships among elements of sequences or other structured inputs.
Related concepts
- attentionuses · Attention is a neural-network mechanism that computes input-dependent relevance weights to combine information from representations for a context-dependent computation.
- attention weightproduced by · An attention weight is a coefficient indicating how strongly an associated input or value contributes to an attention-based output or representation.
- decoding strategyused by · A decoding strategy is a procedure for converting an encoded or internal representation into a target output, including selecting tokens during sequence generation.
- encoderproduced by · An encoder is a component or process that transforms input into an encoded or internal representation for subsequent processing, transmission, or decoding.
- valueproduced by · In attention, a value is a projected vector representation of content that is combined with other values according to attention weights to produce an output.
- vectorkind of · A vector is an ordered collection of components representing an element of a vector space.
Questions about context vector
What is context vector?
A context vector is a numerical vector that summarizes information from relevant input context for a model’s prediction or subsequent processing.
Which Manning books explain context vector?
Build a DeepSeek Model (From Scratch), Sutskever's List and Build AI Drug Discovery Pipelines.
What should I know before learning context vector?
Manning books that teach context vector usually introduce recurrent neural network, softmax, LSTM and transformer architecture first.
What is context vector a kind of?
Context vector is a kind of vector. A vector is an ordered collection of components representing an element of a vector space.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.


