liveBook · Manning

Feedforward network

As explained in Quantization and Fast Inference

A feed-forward network is a neural-network component that transforms representations through successive learned layers without recurrent connections.

In a transformer encoder block, the feed-forward network (FFN) consists of an up-projection and a down-projection. Their larger weight matrices make FFN layers a priority when diagnosing weight-quantization error. section 5.5

BERT weight quantization

On BERT-base, FFN matrices are 768×3072, compared with 768×768 attention projections. Under per-channel weight fake quantization, FFN sublayers produce 16.2× the aggregate output MSE at INT8 and 2.0× at INT4. section 5.5

Figure 5.15 Sublayer sensitivity to per-channel weight fake quantization on BERT-base-uncased. FFN sublayers dominate—16.2× more aggregate output MSE than all four attention projections at INT8, 2.0× at INT4. Dashed line separates attention (below) from FFN (above).

Compare FFN and attention sublayer sensitivity to per-channel weight fake quantization. section 5.5

How Quantization and Fast Inference covers it

How deep it goes
Explains the up- and down-projections and compares their weight-quantization error, without detailing the full network computation.
Examples
Compares BERT-base FFN and attention projection sizes and reports their relative output MSE under INT8 and INT4 fake quantization.
What you can do after
Diagnose which transformer sublayers contribute most to weight-quantization error and prioritize them for higher precision.
How it connects
Connects feed-forward networks with transformer encoders, attention layers, and quantization-aware training.

Also explained in

Compare how 20 books explain feedforward network →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Quantization and Fast Inference. The section links point to the parts of the book it draws on.