liveBook · Manning

Linear layer

As explained in Machines that Think

A linear layer is a neural-network layer that applies a learned linear transformation to its inputs.

A linear layer applies a learned linear transformation to its input. In a language model, such transformations help turn internal representations into forms suited to different tasks, from attention to predicting the next token. section 4.1.8 section 4.1.3

After a sequence passes through the Transformer blocks, each position has a contextualized representation. A linear layer transforms these representations, and a following softmax assigns probabilities to tokens in the vocabulary. During training, the predicted probabilities are compared with the actual next token, and the model’s parameters are adjusted to reduce the difference. section 4.1.8

Learned projections also play a role inside attention. Separate matrices of trainable weights transform each token representation into a query, a key, and a value. These serve different purposes: queries and keys help determine relevance, while values carry information that can be shared. section 4.1.3

Figure 4.3 A single token representation is transformed by three learned projections into a query, key, and value, which serve different roles in attention. The query represents what the token looks for in context, the key what makes it potentially relevant to other tokens, and the value the information it can contribute.

The figure distinguishes the three learned projections applied to a token representation. section 4.1.3

Combining attention heads

In “The sun rises in the east,” one attention head may emphasize the relationship between “rises” and “sun,” while another emphasizes “east.” Each head uses its own learned projections. Their outputs are joined, then a final learned transformation blends them into a unified representation. section 4.1.4

A dense layer is described by its connectivity: every neuron in one layer connects to every neuron in the next. A linear transformation, by contrast, describes an operation on an input. Dense networks offer flexibility, but their connectivity alone does not build in spatial, sequential, or temporal structure; those patterns must instead be learned from data. section 3.4.1 section 4.1.3

How Machines that Think covers it

How deep it goes
Explains output prediction and attention projections at an overview level, without specifying matrix shapes or implementation details.
Examples
Illustrates learned projections with attention to “sun” and “east” in a sentence about sunrise.
What you can do after
Evaluate how learned transformations contribute to attention and next-token prediction.
How it connects
Connects learned transformations to self-attention, queries, keys, values, and softmax.

Also explained in

Compare how 17 books explain linear layer →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Machines that Think. The section links point to the parts of the book it draws on.