The feed-forward step updates token embeddings after attention has decided what matters. It lets information from relevant tokens rub off onto others, producing a more contextualized representation. section 1.20
A simple example: “dog” can make “bark” feel more animal-like, while an irrelevant token like “’s” contributes little. The point is not raw attention scores alone, but the embedding update they drive. section 1.20
This update is applied across all tokens in the prompt, and it can be repeated through multiple transformer layers. Each layer has its own learnable parameters, so later layers can specialize in different contextualization tasks. section 1.20
Figure 1.10 The attention mechanism calculates the relative relevance of all tokens in the context window to contextualize or disambiguate the last token.
Shows attention as the relevance signal that guides contextualization of the last token. section 1.20
