liveBook · Manning

Content projection

As explained in Rearchitecting LLMs

A content projection is the linear transformation in a gated unit that produces the representation modulated by the gate.

In a gated linear unit (GLU), the content projection transforms a token representation into an information-rich intermediate representation. In Llama-3.2-1B, this is `up_proj`. A parallel `gate_proj` produces a control vector that passes through SiLU; element-wise multiplication uses it to modulate the content before `down_proj` contracts the result. Unlike the gate, the content projection carries the transformed information. It corresponds to the expansion step in a classic MLP, represented by `c_fc` in DistilGPT2. section 3.2.2

(mlp): LlamaMLP(
    (gate_proj): Linear(in_features=2048, out_features=8192, bias=False)
    (up_proj): Linear(in_features=2048, out_features=8192, bias=False)
    (down_proj): Linear(in_features=8192, out_features=2048, bias=False)
    (act_fn): SiLU()
)

The LlamaMLP listing shows the input and output dimensions of up_proj alongside the parallel gate_proj and contracting down_proj. section 3.2.2

How Rearchitecting LLMs covers it

How deep it goes
Explains the projection’s role in GLU processing alongside gating, element-wise combination, and contraction.
Examples
Uses the token “spring” in “Fresh water flows from the mountain spring daily” to illustrate contextual processing.
What you can do after
Understand how a GLU content representation is modulated by a parallel gate before contraction.
How it connects
Connects content projection to GLUs, gate projection, SiLU activation, and the down projection.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Rearchitecting LLMs. The section links point to the parts of the book it draws on.