liveBook · Manning

Context vector

As explained in Inside Deep Learning: Math, Algorithms, Models

A summary vector formed from input representations, here produced by averaging or attention over word embeddings.

A context vector is a fixed-size summary of item representations. In the attention network, hidden states h are combined into h_context before attention scores are computed. section 10.3

For padded variable-length inputs, average only valid items. Multiply h by the mask, sum across items, and divide by the valid-item count; add 1e-10 to avoid division by zero. section 10.3

How Inside Deep Learning: Math, Algorithms, Models covers it

How deep it goes
Defines masked averaging through PyTorch operations and stops short of deriving attention-score internals.
Examples
Uses padded variable-size MNIST digit sets and compares dot, general, and additive attention configurations.
What it assumes
Assumes familiarity with hidden representations, masks, tensor dimensions, and basic attention terminology.
What you can do after
Build it from scratch with masked sums, valid-item counts, and an epsilon-stabilized division in PyTorch.
How it connects
Links hidden states, padding masks, attention scores, attention mechanisms, and prediction networks.
Cover of Inside Deep Learning: Math, Algorithms, Models
liveBookmanning.com
Read Inside Deep Learning: Math, Algorithms, Models on liveBook Get the book at manning.com

This explainer was generated by Manning's knowledge graph from the text of Inside Deep Learning: Math, Algorithms, Models. The section links point to the parts of the book it draws on.