liveBook · Manning

Feedforward network

As explained in Build an Advanced RAG Application (From Scratch)

A position-wise feed-forward network independently transforms the representation at each sequence position using a feed-forward neural network.

A feed-forward network (FFN) is a Transformer subcomponent that further transforms representations after attention. In an encoder, a position-wise FFN works independently at every sequence position, refining the information already gathered about the sequence and helping capture complex, nuanced patterns. section 2.2

Attention and FFNs have distinct jobs. Multi-head self-attention relates positions across a sequence, including distant elements; the FFN then processes each position separately. Together they support both broad dependencies and detailed representation processing rather than treating attention as the entire layer computation. section 2.2

FFNs recur in Transformer-based systems as part of larger processing stacks:

  • Encoder layers use position-wise FFNs after attention to strengthen input representations used for semantic and syntactic understanding. section 2.2 section 2.5
  • Decoder layers include an FFN alongside masked self-attention and encoder-decoder attention while generating output sequences token by token. section 2.3.2 section 5.1
  • A BERT cross-encoder can pass contextualized embeddings to a feedforward neural-network classifier that produces a sentence-similarity score. section 4.3

Similarity scoring with a cross-encoder

To compare two sentences, concatenate them with a [SEP] token, process the combined sequence with BERT, and send its output embeddings to a feedforward classifier. The classifier returns a similarity score, typically from 0 to 1. This differs from a position-wise Transformer FFN: it is an output classification component for a paired-sentence task. section 4.3

Figure 5.1 Autoregressive Language Models predict the next word by processing a sequence of previous words. They use embedding, positional encoding, and multiple decoder blocks to generate a probability distribution for the next word.

Use the decoder-model diagram to locate feed-forward networks among embeddings, positional encodings, masked self-attention, and repeated decoder layers used for next-token prediction. section 5.1

How Build an Advanced RAG Application (From Scratch) covers it

How deep it goes
Defines position-wise independent processing and its representation role, without specifying layer equations, activations, or implementation details.
Examples
Uses a story-reading analogy and a BERT cross-encoder whose feedforward classifier scores concatenated sentence pairs.
What you can do after
Evaluate/understand where feed-forward processing fits in Transformer representations and cross-encoder similarity scoring.
How it connects
Connects feed-forward networks with multi-head self-attention, masked self-attention, encoder-decoder attention, token embeddings, and positional encodings.

Also explained in

Compare how 20 books explain feedforward network →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Build an Advanced RAG Application (From Scratch). The section links point to the parts of the book it draws on.