A value vector carries transformed input content into self-attention. Multiplying each input vector by W_value produces values, which attention weights combine into a context vector. section 3.4.1
Follow the value-vector flow:
- Project every input with W_value; for six inputs and d_out = 2, values has shape [6, 2]. section 3.4.1
- Compute attention weights from query–key scores, then multiply those weights by values to form the context vector. section 3.4.1
Single-query computation
For the second input, compute value_2 = x_2 @ W_value. After obtaining all values and the attention weights for x_2, calculate context_vec_2 = attn_weights_2 @ values, producing a two-dimensional context vector. section 3.4.1
Figure 3.14 In the first step of the self-attention mechanism with trainable weight matrices, we compute query (q), key (k), and value (v) vectors for input elements x. Similar to previous sections, we designate the second input, x(2), as the query input. The query vector q(2) is obtained via matrix multiplication between the input x(2) and the weight matrix Wq. Similarly, we obtain the key and value vectors via matrix multiplication involving the weight matrices Wk and Wv.
Trace how W_value transforms input elements into value vectors alongside the corresponding query and key projections. section 3.4.1
