liveBook · Manning

Context vector

As explained in Build AI Drug Discovery Pipelines

A context vector is the weighted sum of value representations produced by an attention operation for a particular output step.

A context vector summarizes the information selected for one output step. In cross-attention, a decoder query compares with encoder states, produces attention weights, and combines their value representations into a weighted sum. section 12.3.1

For protein sequences, attention replaces the bottleneck of forcing hundreds of amino acids into one final recurrent hidden state. Each output can draw on the encoder states most relevant to its prediction. section 12.3.1

Figure 12.8 The mechanics of cross-attention. A decoder state acts as a query (qj) and compares itself against all encoder states (keys, K). These comparisons generate attention weights (α), which are used to compute a weighted sum of the encoder states (values, V), creating a context vector (cj) specific to the current prediction step.

Trace how decoder queries, encoder keys, attention weights, and values produce an output-specific context vector. section 12.3.1

How Build AI Drug Discovery Pipelines covers it

How deep it goes
Defines the weighted-sum result and relates it to encoder states, queries, keys, values, and attention weights.
Examples
Uses a 500-amino-acid protein and hidden states from positions 50, 200, and 400 to illustrate distributed sequence information.
What it assumes
Assumes familiarity with sequences and hidden states while introducing the roles of queries, keys, values, and attention weights.
What you can do after
Understand why output-specific weighted summaries preserve information that a single final hidden state can lose.
How it connects
Connects recurrent neural networks, hidden states, encoder-decoder architectures, attention, and cross-attention.

Also explained in

Compare how 3 books explain context vector →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Build AI Drug Discovery Pipelines. The section links point to the parts of the book it draws on.