Feedforward network
A feedforward network is a neural network whose computations proceed from inputs toward outputs through directed connections without recurrent cycles or feedback.
Applied Reinforcement Learning
A feedforward neural network is a neural network in which information flows from input to output without recurrent connections.
- How it is used
- Uses feedforward networks to score treatment actions in a DQN and transform token representations in transformer blocks.
- How deep it goes
- Works through a Q-network’s forward pass and weight updates, while describing the transformer feedforward component only at a high level.
- Examples
- Uses simulated radiotherapy treatment decisions and contextual token representations in medical questions as examples.
- What you can do after
- Build it from scratch as a simple Q-network with a forward pass and weight updates.
Read how Applied Reinforcement Learning explains feedforward network →
Build AI Drug Discovery Pipelines
A position-wise feed-forward network is a neural network applied independently to each sequence position, commonly using two linear transformations and a nonlinear activation.
- How it is used
- Uses position-wise feedforward networks to process protein sequence representations after self-attention, and feedforward models as molecular docking surrogates.
- How deep it goes
- Defines the position-wise FFN as two linear transformations around an activation, and implements a separate three-layer molecular surrogate.
- Examples
- Uses protein residue representations, molecular fingerprints for docking-score prediction, and flattened SMILES strings to illustrate feedforward networks.
- What you can do after
- Evaluate and understand the FFN’s role in a protein encoder and apply a feedforward surrogate using PyTorch.
Read how Build AI Drug Discovery Pipelines explains feedforward network →
Build an Advanced RAG Application (From Scratch)
A position-wise feed-forward network independently transforms the representation at each sequence position using a feed-forward neural network.
- How it is used
- Refines attention-processed sequence representations within encoder and decoder layers.
- How deep it goes
- Defines position-wise independent processing and its representation role, without specifying layer equations, activations, or implementation details.
- Examples
- Uses a story-reading analogy and a BERT cross-encoder whose feedforward classifier scores concatenated sentence pairs.
- What you can do after
- Evaluate/understand where feed-forward processing fits in Transformer representations and cross-encoder similarity scoring.
Read how Build an Advanced RAG Application (From Scratch) explains feedforward network →
Learn Generative AI with PyTorch
- How it is used
- Uses position-wise feedforward networks to transform token features within Transformer encoder and decoder layers.
- How deep it goes
- Explains two-layer dimensions and position-wise processing, then implements dropout, GELU, normalization, and residual connections in decoder blocks.
- Examples
- Illustrates feedforward layers in an English-to-French Transformer, GPT-2XL, a text generator, and a music Transformer.
- What you can do after
- Builds a position-wise feedforward sublayer and integrates it into a PyTorch Transformer decoder block.
Read how Learn Generative AI with PyTorch explains feedforward network →
Optimization Algorithms: AI techniques for design, planning, and control problems
A feedforward neural network is a neural network in which information flows from inputs through successive layers to outputs without recurrent connections.
- How it is used
- Uses MLPs for function optimization, graph-based routing predictions, and reinforcement learning policies.
- How deep it goes
- Shows a three-layer PyTorch implementation and its training context, but does not derive general feedforward-network internals.
- Examples
- Illustrates predictions on a sphere, traveling salesman solutions, and CartPole balancing.
- What you can do after
- Builds a PyTorch MLP for sphere predictions and applies an MLP policy using Stable Baselines3.
CUDA for LLMs
A transformer feed-forward network independently transforms each position's representation using linear layers and a nonlinear activation.
- How it is used
- Uses feedforward networks to transform individual positions after attention gathers information across tokens.
- How deep it goes
- Shows PyTorch initialization and forward execution, including expansion, GELU, projection, normalization, and residual integration.
- Examples
- Uses an eight-block transformer with 128-dimensional embeddings and a 512-channel feedforward expansion.
- What you can do after
- Apply it via PyTorch to construct a position-wise network inside a transformer block.
Sutskever's List
A feed-forward network is a position-wise neural network that transforms each sequence representation independently, typically through linear layers and a nonlinear activation.
- How it is used
- Transforms each token’s attended representation separately within repeated Transformer encoder and decoder layers.
- How deep it goes
- Explains the two linear transformations, ReLU, and dimensional changes, but leaves code-level treatment to an annotated implementation.
- Examples
- Uses a 512-to-2048-to-512 vector transformation and a Transformer architecture diagram to show operation and placement.
- What you can do after
- Evaluate/understand how an FFN nonlinearly transforms attended token representations and how depth-wise layer reuse changes the architecture.
Build a Text-to-Image Generator (from Scratch)
A feed-forward network that applies the same transformation independently to each position in a sequence.
- How it is used
- Refines position representations after attention inside encoder and decoder layers.
- How deep it goes
- Explains position-wise processing, dimensions, GELU, normalization, residuals, and PyTorch construction, but not parameter training details.
- Examples
- Uses a 48-to-192-to-48 MLP and encoder blocks that stack attention, normalization, residual addition, and feedforward computation.
- What you can do after
- Build it from scratch with PyTorch using two linear layers, GELU, layer normalization, and residual additions.
Read how Build a Text-to-Image Generator (from Scratch) explains feedforward network →
Classic Computer Science Problems in Swift: Essential techniques for practicing programmers
A feed-forward network is a neural network in which signals move through successive layers in a single direction without cycles.
- How it is used
- Explains the standard neural-network structure used for forward computation and backpropagation training.
- How deep it goes
- Develops neurons, layers, activation, backpropagation, and weight updates, while deferring calculus derivations.
- Examples
- Uses neuron and layered-network figures plus pretrained image classification deployed through Core ML.
- What you can do after
- Operate a pretrained feed-forward model in an application through Apple’s Core ML framework.
Deep Learning with R, Third Edition
A feedforward block is a neural-network subnetwork that expands a representation, applies a nonlinearity, and projects it back to the model dimension.
- How it is used
- Updates each token representation independently inside Transformer encoder and decoder blocks after attention exchanges sequence information.
- How deep it goes
- Defines two dense layers, an intervening activation, and expansion followed by projection, without deriving the transformation mathematically.
- Examples
- Uses R Transformer encoder code and a translation model with 256 hidden features and a 2,048-feature intermediate layer.
- What you can do after
- Builds a Transformer feedforward block from two dense layers with an activation and a chosen intermediate dimension.
Read how Deep Learning with R, Third Edition explains feedforward network →
Grokking AI Algorithms, Second Edition
A feed-forward layer transforms its inputs through learned parameters without recurrent state or feedback connections.
- How it is used
- Processes self-attention features in Transformers and converts conditioning embeddings into guidance vectors for diffusion models.
- How deep it goes
- Traces projections, activation, bias, and residual addition, while stopping short of theoretical analysis.
- Examples
- Uses a 4-to-8-to-4 Transformer block, a text-embedding MLP, Mixture-of-Experts routing, and “a happy face” conditioning.
- What you can do after
- Understand how learned projections reshape features and apply the pattern in a simple Python/NumPy embedding implementation.
Read how Grokking AI Algorithms, Second Edition explains feedforward network →
Build a DeepSeek Model (From Scratch)
A feed-forward neural network transforms its inputs through layers whose connections do not form recurrent cycles.
- How it is used
- Processes each token's contextualized representation inside every Transformer block.
- How deep it goes
- Explains two-layer expansion, nonlinear transformation, contraction, shape preservation, and dense computational cost; defers parameter-level implementation.
- Examples
- Uses the context matrix for “The next day is,” GPT-2 block repetition, and the standard expansion-contraction diagram.
- What you can do after
- Apply the expansion-contraction pattern and evaluate its capacity against its dense training and inference costs.
Read how Build a DeepSeek Model (From Scratch) explains feedforward network →
Machine Learning Algorithms in Depth
A feed-forward network is a neural network whose computation flows directly from inputs to outputs without recurrent state.
- How it is used
- Uses an MLP as a direct input-to-output model for classification and regression.
- How deep it goes
- Derives layer operations and outlines forward and backward gradient flow, then moves to a Keras/TensorFlow implementation.
- Examples
- Uses an MLP architecture diagram and a Keras/TensorFlow MNIST classification implementation with training and validation metrics.
- What you can do after
- Apply it via Keras/TensorFlow to classify MNIST digits and assess loss and accuracy.
Read how Machine Learning Algorithms in Depth explains feedforward network →
Transformers in Action
A feed-forward network is a neural-network component that transforms inputs through successive fully connected layers without recurrent state.
- How it is used
- Uses feedforward networks to transform attention outputs within transformer encoder layers.
- How deep it goes
- Explains position-wise projection, activation and dimensional restoration with an implementation, while leaving the complete transformer to other components.
- Examples
- Demonstrates a position-wise FFN with two linear layers, ReLU and dropout.
- What you can do after
- Build it from scratch as a position-wise FFN using PyTorch linear layers, ReLU and dropout.
Read how Transformers in Action explains feedforward network →
Deep Learning with Python, Third Edition
The Transformer submodule that applies a position-wise dense transformation and nonlinearity to each token representation.
- How it is used
- Restores nonlinear per-token transformation alongside attention’s cross-sequence information flow.
- How deep it goes
- Defines two dense projections with an activation between them, but does not derive their optimization or internal mechanics.
- Examples
- Uses a 256-dimensional embedding, expands to 2,048 features, applies ReLU, and projects back to the hidden size.
- What you can do after
- Build it from scratch with Keras Dense layers inside a Transformer encoder or decoder.
Read how Deep Learning with Python, Third Edition explains feedforward network →
Timeless Algorithms, The Foundational Ideas
A neural network component that applies learned transformations independently at each sequence position.
- How it is used
- Places feed-forward networks among transformer components used in next-token prediction from contextual representations.
- How deep it goes
- Names feed-forward networks as transformer components but stops before specifying their computation or behavior.
Read how Timeless Algorithms, The Foundational Ideas explains feedforward network →
Build a Large Language Model (From Scratch)
The transformer subcomponent that applies position-wise nonlinear transformations after attention.
- How it is used
- Transforms each token position independently after attention within a transformer block.
- How deep it goes
- Defines the two-layer GELU module, including fourfold expansion and shape-preserving contraction, without deriving optimization behavior formally.
- Examples
- Demonstrates a 2×3×768 tensor flowing through 768→3,072→768 dimensions and returning shape [2, 3, 768].
- What you can do after
- Build it from scratch in PyTorch using configurable Linear layers and a GELU activation.
Read how Build a Large Language Model (From Scratch) explains feedforward network →
LLMs in Production
A feed-forward layer is a neural-network layer that transforms its input through learned operations without recurrent state.
- How it is used
- Uses stacked learned transformations as a baseline neural architecture and a starting point for comparing more specialized language-model designs.
- How deep it goes
- Defines layered computation and demonstrates implementation and training, while deferring the deeper data-flow mechanisms that distinguish major architectures.
- Examples
- Demonstrates a configurable PyTorch multilayer perceptron and a small token-level text generator trained with cross-entropy.
- What you can do after
- Build it from scratch using configurable PyTorch layers, train it on token sequences, and assess generated text with loss or perplexity.
Grokking Machine Learning
A neural network composed of multiple layers of perceptron-like processing units.
- How it is used
- Frames multilayer perceptrons as alternatives for complicated image classification and as models for classification and regression.
- How deep it goes
- Defines networks as perceptron collections and previews layer architecture, activation functions, backpropagation, training problems, and remedies.
- Examples
- Uses dog-image recognition and MNIST digit recognition as classification examples, and names sentiment analysis and image classification applications.
- What you can do after
- Understand how composing linear classifiers can yield nonlinear decision boundaries and identify classification and regression uses.
Read how Grokking Machine Learning explains feedforward network →
Quantization and Fast Inference
A feed-forward network is a neural-network component that transforms representations through successive learned layers without recurrent connections.
- How it is used
- Prioritizes transformer FFN layers when identifying which weights need protection during quantization-aware training.
- How deep it goes
- Explains the up- and down-projections and compares their weight-quantization error, without detailing the full network computation.
- Examples
- Compares BERT-base FFN and attention projection sizes and reports their relative output MSE under INT8 and INT4 fake quantization.
- What you can do after
- Diagnose which transformer sublayers contribute most to weight-quantization error and prioritize them for higher precision.
Read how Quantization and Fast Inference explains feedforward network →
Also covered in
- Building Specialized AI SystemsIn early access (MEAP)
- Hugging Face in Action
- AI Agents in Action, Second Edition
- Deep Learning with PyTorch, Second Edition
What to know first
- language modelA language model is a computational model that represents patterns in linguistic sequences to assign probabilities, predict, generate, or evaluate language.
- transformer architectureA transformer is a neural-network architecture that uses attention-based interactions and position-wise transformations to model relationships among elements of sequences or other structured inputs.
- recurrent neural networkA recurrent neural network is a neural network that processes ordered inputs while carrying and updating an internal state across successive steps.
- linear layerA linear layer is a neural-network layer that applies a learned affine transformation to its input, typically using weights and a bias.
Related concepts
- transformer architecturepart of · A transformer is a neural-network architecture that uses attention-based interactions and position-wise transformations to model relationships among elements of sequences or other structured inputs.
- language modelkind of · A language model is a computational model that represents patterns in linguistic sequences to assign probabilities, predict, generate, or evaluate language.
- recurrent neural networkcontrasts with · A recurrent neural network is a neural network that processes ordered inputs while carrying and updating an internal state across successive steps.
- transformer blockpart of · A transformer block is a modular unit of a transformer neural network that applies attention and other transformations, commonly including feed-forward computation, to representations.
- mixture of expertsalternative to · A mixture-of-experts model is a neural-network architecture with specialized subnetworks and a routing mechanism that selects or combines experts for each input.
- linear layerhas part · A linear layer is a neural-network layer that applies a learned affine transformation to its input, typically using weights and a bias.
Questions about feedforward network
What is feedforward network?
A feedforward network is a neural network whose computations proceed from inputs toward outputs through directed connections without recurrent cycles or feedback.
Which Manning books explain feedforward network?
Applied Reinforcement Learning, Build AI Drug Discovery Pipelines, Build an Advanced RAG Application (From Scratch), Learn Generative AI with PyTorch, Optimization Algorithms: AI techniques for design, planning, and control problems, CUDA for LLMs, Sutskever's List, Build a Text-to-Image Generator (from Scratch) and 12 more.
What should I know before learning feedforward network?
Manning books that teach feedforward network usually introduce language model, transformer architecture, recurrent neural network and linear layer first.
What is the difference between feedforward network and recurrent neural network?
A feedforward network is a neural network whose computations proceed from inputs toward outputs through directed connections without recurrent cycles or feedback. A recurrent neural network is a neural network that processes ordered inputs while carrying and updating an internal state across successive steps.
What is feedforward network a kind of?
Feedforward network is a kind of language model. A language model is a computational model that represents patterns in linguistic sequences to assign probabilities, predict, generate, or evaluate language.
What is feedforward network part of?
Feedforward network is part of transformer architecture. A transformer is a neural-network architecture that uses attention-based interactions and position-wise transformations to model relationships among elements of sequences or other structured inputs.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.



















