Feedforward network

As explained in Sutskever's List

A feed-forward network is a position-wise neural network that transforms each sequence representation independently, typically through linear layers and a nonlinear activation.

Attention lets a token draw information from other tokens. A feed-forward network (FFN) then transforms the resulting representation at that token’s position. Attention mixes information across positions; the FFN reshapes it separately at each position. section 5.1

Expand, activate, project

In the original Transformer, an FFN applies two linear transformations with a ReLU activation between them. A 512-dimensional token vector expands to 2048 dimensions, passes through ReLU, then returns to 512 dimensions. This gives the attended representation a nonlinear transformation without another round of token-to-token mixing. section 5.1

FFNs appear in both encoder and decoder layers, alongside attention. Residual connections and layer normalization help stabilize the repeated stack. The 2018 Universal Transformer kept the pairing of attention and FFNs but reused those layers across depth to refine representations over several steps. That design offered a conceptual alternative, though it was slower than the original Transformer without clear performance gains. section 5.1 section 5.1.2

Figure 5.1 A simplified schematic of the Transformer architecture. The model consists of an encoder (left) and a decoder (right), each comprising repeated stacks of sublayers, including multihead attention mechanisms, feed-forward neural networks, and postprocessing steps such as residual connections and layer normalization. Positional encodings (depicted as circled crosses) are added to the input embeddings to explicitly represent token order. The diagram emphasizes the cross-attention operation (dashed box labeled “Cross-attention”), in which the decoder’s queries (Q) interact with the encoder’s keys (K) and values (V). During cross-attention, each decoding step computes attention scores to determine the relevance of encoded input representations, producing a context vector that guides token generation. This selective alignment enables the decoder to integrate contextual information from the encoder, improving the accuracy and coherence of the output sequence.

The architecture diagram locates feed-forward networks within the repeated encoder and decoder layers, after their attention sublayers. section 5.1

For a code-level view, The Annotated Transformer includes FFNs in a commented, working PyTorch reproduction of the architecture. section 5.2

How Sutskever's List covers it

How deep it goes

Explains the two linear transformations, ReLU, and dimensional changes, but leaves code-level treatment to an annotated implementation.

Examples

Uses a 512-to-2048-to-512 vector transformation and a Transformer architecture diagram to show operation and placement.

What it assumes

Introduces the FFN through token vectors and named neural-network operations, with a concrete dimensional example rather than a derivation.

What you can do after

Evaluate/understand how an FFN nonlinearly transforms attended token representations and how depth-wise layer reuse changes the architecture.

How it connects

Connects feed-forward networks to attention, encoder and decoder layers, residual connections, layer normalization, and depth-wise recurrence.

liveBookmanning.com

Read Sutskever's List on liveBook Get the book at manning.com

Also explained in

• 

  liveBookmanning.com
  Learn Generative AI with PyTorch
• 

  liveBookmanning.com
  Optimization Algorithms: AI techniques for design, planning, and control problemsA feedforward neural network is a neural network in which information flows from inputs through successive layers to outputs without recurrent connections.
• 

  liveBookmanning.com
  Build a Text-to-Image Generator (from Scratch)A feed-forward network that applies the same transformation independently to each position in a sequence.
• 

  liveBookmanning.com
  Classic Computer Science Problems in Swift: Essential techniques for practicing programmersA feed-forward network is a neural network in which signals move through successive layers in a single direction without cycles.
• 

  liveBookmanning.com
  Deep Learning with R, Third EditionA feedforward block is a neural-network subnetwork that expands a representation, applies a nonlinearity, and projects it back to the model dimension.
• 

  liveBookmanning.com
  Grokking AI Algorithms, Second EditionA feed-forward layer transforms its inputs through learned parameters without recurrent state or feedback connections.
• 

  liveBookmanning.com
  Machine Learning Algorithms in DepthA feed-forward network is a neural network whose computation flows directly from inputs to outputs without recurrent state.
• 

  liveBookmanning.com
  Transformers in ActionA feed-forward network is a neural-network component that transforms inputs through successive fully connected layers without recurrent state.
• 

  liveBookmanning.com
  Deep Learning with Python, Third EditionThe Transformer submodule that applies a position-wise dense transformation and nonlinearity to each token representation.
• 

  liveBookmanning.com
  Build a Large Language Model (From Scratch)The transformer subcomponent that applies position-wise nonlinear transformations after attention.
• 

  liveBookmanning.com
  LLMs in ProductionA feed-forward layer is a neural-network layer that transforms its input through learned operations without recurrent state.
• 

  liveBookmanning.com
  Grokking Machine LearningA neural network composed of multiple layers of perceptron-like processing units.

Compare how 13 books explain feedforward network →

Related concepts

• transformer architecturepart of · A transformer architecture is a neural-network architecture that processes sequences or other structured inputs using attention mechanisms and related transformations.
• language modelkind of · A language model is a computational model that represents patterns in linguistic tokens to estimate sequence probabilities and predict or generate language from context.
• recurrent neural networkcontrasts with · A recurrent neural network is a neural network that processes sequences by updating an internal state across successive inputs, allowing earlier inputs to influence later computations.
• transformer blockpart of · A transformer block is a neural-network module that transforms sequence representations using attention and feed-forward computation.
• linear layerhas part · A linear layer is a neural-network layer that maps input features to outputs by a learned affine transformation, typically connecting each output to every input feature.
• mixture of expertsalternative to · A mixture-of-experts model is a neural-network architecture with specialized expert subnetworks and a routing mechanism that selects or combines experts for each input.

This explainer was generated by Manning's knowledge graph from the text of Sutskever's List. The section links point to the parts of the book it draws on.