Feedforward network
A feedforward network is a neural network or subnetwork in which information flows from inputs toward outputs through successive computations without recurrent connections or cycles.
Learn Generative AI with PyTorch
- How it is used
- Uses a position-wise feedforward sublayer to transform features after attention in Transformer encoders and decoders.
- How deep it goes
- Works through two linear projections, fourfold hidden expansion, and a GPT block with GELU, dropout, normalization, and residual addition.
- Examples
- Shows a 256-to-1,024-to-256 text block and a 512-to-2,048-to-512 music Transformer block.
- What you can do after
- Build it from scratch as a PyTorch decoder-block sublayer that expands, activates, and projects token representations.
Read how Learn Generative AI with PyTorch explains feedforward network →
Optimization Algorithms: AI techniques for design, planning, and control problems
A feedforward neural network is a neural network in which information flows from inputs through successive layers to outputs without recurrent connections.
- How it is used
- Maps function parameters to candidate points, predicts graph-solution probabilities, and supplies CartPole policy networks.
- How deep it goes
- Shows a three-layer PyTorch implementation and training setup, but uses pretrained models rather than building the TSP pipeline from scratch.
- Examples
- Uses point prediction on a sphere, traveling-salesman tours, and CartPole actor–critic and policy implementations.
- What you can do after
- Build it from scratch in PyTorch for sphere-point prediction, or apply it via Stable Baselines3’s `MlpPolicy` for CartPole.
Sutskever's List
A feed-forward network is a position-wise neural network that transforms each sequence representation independently, typically through linear layers and a nonlinear activation.
- How it is used
- Transforms each token’s attended representation separately within repeated Transformer encoder and decoder layers.
- How deep it goes
- Explains the two linear transformations, ReLU, and dimensional changes, but leaves code-level treatment to an annotated implementation.
- Examples
- Uses a 512-to-2048-to-512 vector transformation and a Transformer architecture diagram to show operation and placement.
- What you can do after
- Evaluate/understand how an FFN nonlinearly transforms attended token representations and how depth-wise layer reuse changes the architecture.
Build a Text-to-Image Generator (from Scratch)
A feed-forward network that applies the same transformation independently to each position in a sequence.
- How it is used
- Refines position representations after attention inside encoder and decoder layers.
- How deep it goes
- Explains position-wise processing, dimensions, GELU, normalization, residuals, and PyTorch construction, but not parameter training details.
- Examples
- Uses a 48-to-192-to-48 MLP and encoder blocks that stack attention, normalization, residual addition, and feedforward computation.
- What you can do after
- Build it from scratch with PyTorch using two linear layers, GELU, layer normalization, and residual additions.
Read how Build a Text-to-Image Generator (from Scratch) explains feedforward network →
Classic Computer Science Problems in Swift: Essential techniques for practicing programmers
A feed-forward network is a neural network in which signals move through successive layers in a single direction without cycles.
- How it is used
- Explains the standard neural-network structure used for forward computation and backpropagation training.
- How deep it goes
- Develops neurons, layers, activation, backpropagation, and weight updates, while deferring calculus derivations.
- Examples
- Uses neuron and layered-network figures plus pretrained image classification deployed through Core ML.
- What you can do after
- Operate a pretrained feed-forward model in an application through Apple’s Core ML framework.
Deep Learning with R, Third Edition
A feedforward block is a neural-network subnetwork that expands a representation, applies a nonlinearity, and projects it back to the model dimension.
- How it is used
- Updates each token representation independently inside Transformer encoder and decoder blocks after attention exchanges sequence information.
- How deep it goes
- Defines two dense layers, an intervening activation, and expansion followed by projection, without deriving the transformation mathematically.
- Examples
- Uses R Transformer encoder code and a translation model with 256 hidden features and a 2,048-feature intermediate layer.
- What you can do after
- Builds a Transformer feedforward block from two dense layers with an activation and a chosen intermediate dimension.
Read how Deep Learning with R, Third Edition explains feedforward network →
Grokking AI Algorithms, Second Edition
A feed-forward layer transforms its inputs through learned parameters without recurrent state or feedback connections.
- How it is used
- Processes self-attention features in Transformers and converts conditioning embeddings into guidance vectors for diffusion models.
- How deep it goes
- Traces projections, activation, bias, and residual addition, while stopping short of theoretical analysis.
- Examples
- Uses a 4-to-8-to-4 Transformer block, a text-embedding MLP, Mixture-of-Experts routing, and “a happy face” conditioning.
- What you can do after
- Understand how learned projections reshape features and apply the pattern in a simple Python/NumPy embedding implementation.
Read how Grokking AI Algorithms, Second Edition explains feedforward network →
Machine Learning Algorithms in Depth
A feed-forward network is a neural network whose computation flows directly from inputs to outputs without recurrent state.
- How it is used
- Uses an MLP as a direct input-to-output model for classification and regression.
- How deep it goes
- Derives layer operations and outlines forward and backward gradient flow, then moves to a Keras/TensorFlow implementation.
- Examples
- Uses an MLP architecture diagram and a Keras/TensorFlow MNIST classification implementation with training and validation metrics.
- What you can do after
- Apply it via Keras/TensorFlow to classify MNIST digits and assess loss and accuracy.
Read how Machine Learning Algorithms in Depth explains feedforward network →
Transformers in Action
A feed-forward network is a neural-network component that transforms inputs through successive fully connected layers without recurrent state.
- How it is used
- Transforms attention outputs within stacked transformer encoder layers.
- How deep it goes
- Explains position-wise processing and two-layer dimensions, then implements activation and dropout; stops short of training details.
- Examples
- Shows a PyTorch position-wise FFN and a simplified encoder layer that places it after attention.
- What you can do after
- Build it from scratch as a PyTorch module and place it after self-attention in an encoder layer.
Read how Transformers in Action explains feedforward network →
Deep Learning with Python, Third Edition
The Transformer submodule that applies a position-wise dense transformation and nonlinearity to each token representation.
- How it is used
- Restores nonlinear per-token transformation alongside attention’s cross-sequence information flow.
- How deep it goes
- Defines two dense projections with an activation between them, but does not derive their optimization or internal mechanics.
- Examples
- Uses a 256-dimensional embedding, expands to 2,048 features, applies ReLU, and projects back to the hidden size.
- What you can do after
- Build it from scratch with Keras Dense layers inside a Transformer encoder or decoder.
Read how Deep Learning with Python, Third Edition explains feedforward network →
Build a Large Language Model (From Scratch)
The transformer subcomponent that applies position-wise nonlinear transformations after attention.
- How it is used
- Transforms each token position independently after attention within a transformer block.
- How deep it goes
- Defines the two-layer GELU module, including fourfold expansion and shape-preserving contraction, without deriving optimization behavior formally.
- Examples
- Demonstrates a 2×3×768 tensor flowing through 768→3,072→768 dimensions and returning shape [2, 3, 768].
- What you can do after
- Build it from scratch in PyTorch using configurable Linear layers and a GELU activation.
Read how Build a Large Language Model (From Scratch) explains feedforward network →
LLMs in Production
A feed-forward layer is a neural-network layer that transforms its input through learned operations without recurrent state.
- How it is used
- Uses stacked learned transformations as a baseline neural architecture and a starting point for comparing more specialized language-model designs.
- How deep it goes
- Defines layered computation and demonstrates implementation and training, while deferring the deeper data-flow mechanisms that distinguish major architectures.
- Examples
- Demonstrates a configurable PyTorch multilayer perceptron and a small token-level text generator trained with cross-entropy.
- What you can do after
- Build it from scratch using configurable PyTorch layers, train it on token sequences, and assess generated text with loss or perplexity.
Grokking Machine Learning
A neural network composed of multiple layers of perceptron-like processing units.
- How it is used
- Frames multilayer perceptrons as alternatives for complicated image classification and as models for classification and regression.
- How deep it goes
- Defines networks as perceptron collections and previews layer architecture, activation functions, backpropagation, training problems, and remedies.
- Examples
- Uses dog-image recognition and MNIST digit recognition as classification examples, and names sentiment analysis and image classification applications.
- What you can do after
- Understand how composing linear classifiers can yield nonlinear decision boundaries and identify classification and regression uses.
Read how Grokking Machine Learning explains feedforward network →
Related concepts
- transformer architecturepart of · A transformer architecture is a neural-network architecture that processes sequences or other structured inputs using attention mechanisms and related transformations.
- language modelkind of · A language model is a computational model that represents patterns in linguistic tokens to estimate sequence probabilities and predict or generate language from context.
- recurrent neural networkcontrasts with · A recurrent neural network is a neural network that processes sequences by updating an internal state across successive inputs, allowing earlier inputs to influence later computations.
- transformer blockpart of · A transformer block is a neural-network module that transforms sequence representations using attention and feed-forward computation.
- linear layerhas part · A linear layer is a neural-network layer that maps input features to outputs by a learned affine transformation, typically connecting each output to every input feature.
- mixture of expertsalternative to · A mixture-of-experts model is a neural-network architecture with specialized expert subnetworks and a routing mechanism that selects or combines experts for each input.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.