liveBook · Manning

Feedforward network

Also called feed-forward network, feed-forward neural network, feedforward neural network, FFN

A feedforward network is a neural network whose computations proceed from inputs toward outputs through directed connections without recurrent cycles or feedback.

How 20 Manning books explain it

Applied Reinforcement Learning

A feedforward neural network is a neural network in which information flows from input to output without recurrent connections.

How it is used
Uses feedforward networks to score treatment actions in a DQN and transform token representations in transformer blocks.
How deep it goes
Works through a Q-network’s forward pass and weight updates, while describing the transformer feedforward component only at a high level.
Examples
Uses simulated radiotherapy treatment decisions and contextual token representations in medical questions as examples.
What you can do after
Build it from scratch as a simple Q-network with a forward pass and weight updates.

Read how Applied Reinforcement Learning explains feedforward network →

Build AI Drug Discovery Pipelines

A position-wise feed-forward network is a neural network applied independently to each sequence position, commonly using two linear transformations and a nonlinear activation.

How it is used
Uses position-wise feedforward networks to process protein sequence representations after self-attention, and feedforward models as molecular docking surrogates.
How deep it goes
Defines the position-wise FFN as two linear transformations around an activation, and implements a separate three-layer molecular surrogate.
Examples
Uses protein residue representations, molecular fingerprints for docking-score prediction, and flattened SMILES strings to illustrate feedforward networks.
What you can do after
Evaluate and understand the FFN’s role in a protein encoder and apply a feedforward surrogate using PyTorch.

Read how Build AI Drug Discovery Pipelines explains feedforward network →

Build an Advanced RAG Application (From Scratch)

A position-wise feed-forward network independently transforms the representation at each sequence position using a feed-forward neural network.

How it is used
Refines attention-processed sequence representations within encoder and decoder layers.
How deep it goes
Defines position-wise independent processing and its representation role, without specifying layer equations, activations, or implementation details.
Examples
Uses a story-reading analogy and a BERT cross-encoder whose feedforward classifier scores concatenated sentence pairs.
What you can do after
Evaluate/understand where feed-forward processing fits in Transformer representations and cross-encoder similarity scoring.

Read how Build an Advanced RAG Application (From Scratch) explains feedforward network →

Learn Generative AI with PyTorch

How it is used
Uses position-wise feedforward networks to transform token features within Transformer encoder and decoder layers.
How deep it goes
Explains two-layer dimensions and position-wise processing, then implements dropout, GELU, normalization, and residual connections in decoder blocks.
Examples
Illustrates feedforward layers in an English-to-French Transformer, GPT-2XL, a text generator, and a music Transformer.
What you can do after
Builds a position-wise feedforward sublayer and integrates it into a PyTorch Transformer decoder block.

Read how Learn Generative AI with PyTorch explains feedforward network →

Optimization Algorithms: AI techniques for design, planning, and control problems

A feedforward neural network is a neural network in which information flows from inputs through successive layers to outputs without recurrent connections.

How it is used
Uses MLPs for function optimization, graph-based routing predictions, and reinforcement learning policies.
How deep it goes
Shows a three-layer PyTorch implementation and its training context, but does not derive general feedforward-network internals.
Examples
Illustrates predictions on a sphere, traveling salesman solutions, and CartPole balancing.
What you can do after
Builds a PyTorch MLP for sphere predictions and applies an MLP policy using Stable Baselines3.

Read how Optimization Algorithms: AI techniques for design, planning, and control problems explains feedforward network →

CUDA for LLMs

A transformer feed-forward network independently transforms each position's representation using linear layers and a nonlinear activation.

How it is used
Uses feedforward networks to transform individual positions after attention gathers information across tokens.
How deep it goes
Shows PyTorch initialization and forward execution, including expansion, GELU, projection, normalization, and residual integration.
Examples
Uses an eight-block transformer with 128-dimensional embeddings and a 512-channel feedforward expansion.
What you can do after
Apply it via PyTorch to construct a position-wise network inside a transformer block.

Read how CUDA for LLMs explains feedforward network →

Sutskever's List

A feed-forward network is a position-wise neural network that transforms each sequence representation independently, typically through linear layers and a nonlinear activation.

How it is used
Transforms each token’s attended representation separately within repeated Transformer encoder and decoder layers.
How deep it goes
Explains the two linear transformations, ReLU, and dimensional changes, but leaves code-level treatment to an annotated implementation.
Examples
Uses a 512-to-2048-to-512 vector transformation and a Transformer architecture diagram to show operation and placement.
What you can do after
Evaluate/understand how an FFN nonlinearly transforms attended token representations and how depth-wise layer reuse changes the architecture.

Read how Sutskever's List explains feedforward network →

Build a Text-to-Image Generator (from Scratch)

A feed-forward network that applies the same transformation independently to each position in a sequence.

How it is used
Refines position representations after attention inside encoder and decoder layers.
How deep it goes
Explains position-wise processing, dimensions, GELU, normalization, residuals, and PyTorch construction, but not parameter training details.
Examples
Uses a 48-to-192-to-48 MLP and encoder blocks that stack attention, normalization, residual addition, and feedforward computation.
What you can do after
Build it from scratch with PyTorch using two linear layers, GELU, layer normalization, and residual additions.

Read how Build a Text-to-Image Generator (from Scratch) explains feedforward network →

Classic Computer Science Problems in Swift: Essential techniques for practicing programmers

A feed-forward network is a neural network in which signals move through successive layers in a single direction without cycles.

How it is used
Explains the standard neural-network structure used for forward computation and backpropagation training.
How deep it goes
Develops neurons, layers, activation, backpropagation, and weight updates, while deferring calculus derivations.
Examples
Uses neuron and layered-network figures plus pretrained image classification deployed through Core ML.
What you can do after
Operate a pretrained feed-forward model in an application through Apple’s Core ML framework.

Read how Classic Computer Science Problems in Swift: Essential techniques for practicing programmers explains feedforward network →

Deep Learning with R, Third Edition

A feedforward block is a neural-network subnetwork that expands a representation, applies a nonlinearity, and projects it back to the model dimension.

How it is used
Updates each token representation independently inside Transformer encoder and decoder blocks after attention exchanges sequence information.
How deep it goes
Defines two dense layers, an intervening activation, and expansion followed by projection, without deriving the transformation mathematically.
Examples
Uses R Transformer encoder code and a translation model with 256 hidden features and a 2,048-feature intermediate layer.
What you can do after
Builds a Transformer feedforward block from two dense layers with an activation and a chosen intermediate dimension.

Read how Deep Learning with R, Third Edition explains feedforward network →

Grokking AI Algorithms, Second Edition

A feed-forward layer transforms its inputs through learned parameters without recurrent state or feedback connections.

How it is used
Processes self-attention features in Transformers and converts conditioning embeddings into guidance vectors for diffusion models.
How deep it goes
Traces projections, activation, bias, and residual addition, while stopping short of theoretical analysis.
Examples
Uses a 4-to-8-to-4 Transformer block, a text-embedding MLP, Mixture-of-Experts routing, and “a happy face” conditioning.
What you can do after
Understand how learned projections reshape features and apply the pattern in a simple Python/NumPy embedding implementation.

Read how Grokking AI Algorithms, Second Edition explains feedforward network →

Build a DeepSeek Model (From Scratch)

A feed-forward neural network transforms its inputs through layers whose connections do not form recurrent cycles.

How it is used
Processes each token's contextualized representation inside every Transformer block.
How deep it goes
Explains two-layer expansion, nonlinear transformation, contraction, shape preservation, and dense computational cost; defers parameter-level implementation.
Examples
Uses the context matrix for “The next day is,” GPT-2 block repetition, and the standard expansion-contraction diagram.
What you can do after
Apply the expansion-contraction pattern and evaluate its capacity against its dense training and inference costs.

Read how Build a DeepSeek Model (From Scratch) explains feedforward network →

Machine Learning Algorithms in Depth

A feed-forward network is a neural network whose computation flows directly from inputs to outputs without recurrent state.

How it is used
Uses an MLP as a direct input-to-output model for classification and regression.
How deep it goes
Derives layer operations and outlines forward and backward gradient flow, then moves to a Keras/TensorFlow implementation.
Examples
Uses an MLP architecture diagram and a Keras/TensorFlow MNIST classification implementation with training and validation metrics.
What you can do after
Apply it via Keras/TensorFlow to classify MNIST digits and assess loss and accuracy.

Read how Machine Learning Algorithms in Depth explains feedforward network →

Transformers in Action

A feed-forward network is a neural-network component that transforms inputs through successive fully connected layers without recurrent state.

How it is used
Uses feedforward networks to transform attention outputs within transformer encoder layers.
How deep it goes
Explains position-wise projection, activation and dimensional restoration with an implementation, while leaving the complete transformer to other components.
Examples
Demonstrates a position-wise FFN with two linear layers, ReLU and dropout.
What you can do after
Build it from scratch as a position-wise FFN using PyTorch linear layers, ReLU and dropout.

Read how Transformers in Action explains feedforward network →

Deep Learning with Python, Third Edition

The Transformer submodule that applies a position-wise dense transformation and nonlinearity to each token representation.

How it is used
Restores nonlinear per-token transformation alongside attention’s cross-sequence information flow.
How deep it goes
Defines two dense projections with an activation between them, but does not derive their optimization or internal mechanics.
Examples
Uses a 256-dimensional embedding, expands to 2,048 features, applies ReLU, and projects back to the hidden size.
What you can do after
Build it from scratch with Keras Dense layers inside a Transformer encoder or decoder.

Read how Deep Learning with Python, Third Edition explains feedforward network →

Timeless Algorithms, The Foundational Ideas

A neural network component that applies learned transformations independently at each sequence position.

How it is used
Places feed-forward networks among transformer components used in next-token prediction from contextual representations.
How deep it goes
Names feed-forward networks as transformer components but stops before specifying their computation or behavior.

Read how Timeless Algorithms, The Foundational Ideas explains feedforward network →

Build a Large Language Model (From Scratch)

The transformer subcomponent that applies position-wise nonlinear transformations after attention.

How it is used
Transforms each token position independently after attention within a transformer block.
How deep it goes
Defines the two-layer GELU module, including fourfold expansion and shape-preserving contraction, without deriving optimization behavior formally.
Examples
Demonstrates a 2×3×768 tensor flowing through 768→3,072→768 dimensions and returning shape [2, 3, 768].
What you can do after
Build it from scratch in PyTorch using configurable Linear layers and a GELU activation.

Read how Build a Large Language Model (From Scratch) explains feedforward network →

LLMs in Production

A feed-forward layer is a neural-network layer that transforms its input through learned operations without recurrent state.

How it is used
Uses stacked learned transformations as a baseline neural architecture and a starting point for comparing more specialized language-model designs.
How deep it goes
Defines layered computation and demonstrates implementation and training, while deferring the deeper data-flow mechanisms that distinguish major architectures.
Examples
Demonstrates a configurable PyTorch multilayer perceptron and a small token-level text generator trained with cross-entropy.
What you can do after
Build it from scratch using configurable PyTorch layers, train it on token sequences, and assess generated text with loss or perplexity.

Read how LLMs in Production explains feedforward network →

Grokking Machine Learning

A neural network composed of multiple layers of perceptron-like processing units.

How it is used
Frames multilayer perceptrons as alternatives for complicated image classification and as models for classification and regression.
How deep it goes
Defines networks as perceptron collections and previews layer architecture, activation functions, backpropagation, training problems, and remedies.
Examples
Uses dog-image recognition and MNIST digit recognition as classification examples, and names sentiment analysis and image classification applications.
What you can do after
Understand how composing linear classifiers can yield nonlinear decision boundaries and identify classification and regression uses.

Read how Grokking Machine Learning explains feedforward network →

Quantization and Fast Inference

A feed-forward network is a neural-network component that transforms representations through successive learned layers without recurrent connections.

How it is used
Prioritizes transformer FFN layers when identifying which weights need protection during quantization-aware training.
How deep it goes
Explains the up- and down-projections and compares their weight-quantization error, without detailing the full network computation.
Examples
Compares BERT-base FFN and attention projection sizes and reports their relative output MSE under INT8 and INT4 fake quantization.
What you can do after
Diagnose which transformer sublayers contribute most to weight-quantization error and prioritize them for higher precision.

Read how Quantization and Fast Inference explains feedforward network →

Also covered in

What to know first

Related concepts

Questions about feedforward network

What is feedforward network?

A feedforward network is a neural network whose computations proceed from inputs toward outputs through directed connections without recurrent cycles or feedback.

Which Manning books explain feedforward network?

Applied Reinforcement Learning, Build AI Drug Discovery Pipelines, Build an Advanced RAG Application (From Scratch), Learn Generative AI with PyTorch, Optimization Algorithms: AI techniques for design, planning, and control problems, CUDA for LLMs, Sutskever's List, Build a Text-to-Image Generator (from Scratch) and 12 more.

What should I know before learning feedforward network?

Manning books that teach feedforward network usually introduce language model, transformer architecture, recurrent neural network and linear layer first.

What is the difference between feedforward network and recurrent neural network?

A feedforward network is a neural network whose computations proceed from inputs toward outputs through directed connections without recurrent cycles or feedback. A recurrent neural network is a neural network that processes ordered inputs while carrying and updating an internal state across successive steps.

What is feedforward network a kind of?

Feedforward network is a kind of language model. A language model is a computational model that represents patterns in linguistic sequences to assign probabilities, predict, generate, or evaluate language.

What is feedforward network part of?

Feedforward network is part of transformer architecture. A transformer is a neural-network architecture that uses attention-based interactions and position-wise transformations to model relationships among elements of sequences or other structured inputs.

These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.