liveBook · Manning

Value

As explained in Rearchitecting LLMs

A value vector is the attention representation combined according to attention weights to produce an attended output.

A value vector is an attention representation combined according to attention weights to produce an attended output. In grouped-query attention (GQA), heads within a group share value projections. section 3.2.1

(q_proj): Linear(in_features=2048, out_features=2048, 
                             ➥ bias=False)  #A
(k_proj): Linear(in_features=2048, out_features=512, bias=False)   
(v_proj): Linear(in_features=2048, out_features=512, bias=False)   
(o_proj): Linear(in_features=2048, out_features=2048, bias=False)  #B

The projection listing shows the value projection's reduced width: v_proj maps 2048 input features to 512 output features. section 3.2.1

How Rearchitecting LLMs covers it

How deep it goes
Defines weighted value aggregation and identifies GQA projection sharing, but does not detail the computation.
Examples
Uses Llama-3.2-1B projections, where v_proj maps 2048 input features to 512 output features.
What you can do after
Understand how GQA shares value projections across heads within a group.
How it connects
Connects values to attention, queries, keys, grouped-query attention, and multi-head attention.

Also explained in

Compare how 15 books explain value →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Rearchitecting LLMs. The section links point to the parts of the book it draws on.