A feed-forward network (FFN) is a Transformer subcomponent that further transforms representations after attention. In an encoder, a position-wise FFN works independently at every sequence position, refining the information already gathered about the sequence and helping capture complex, nuanced patterns. section 2.2
Attention and FFNs have distinct jobs. Multi-head self-attention relates positions across a sequence, including distant elements; the FFN then processes each position separately. Together they support both broad dependencies and detailed representation processing rather than treating attention as the entire layer computation. section 2.2
FFNs recur in Transformer-based systems as part of larger processing stacks:
- Encoder layers use position-wise FFNs after attention to strengthen input representations used for semantic and syntactic understanding. section 2.2 section 2.5
- Decoder layers include an FFN alongside masked self-attention and encoder-decoder attention while generating output sequences token by token. section 2.3.2 section 5.1
- A BERT cross-encoder can pass contextualized embeddings to a feedforward neural-network classifier that produces a sentence-similarity score. section 4.3
Similarity scoring with a cross-encoder
To compare two sentences, concatenate them with a [SEP] token, process the combined sequence with BERT, and send its output embeddings to a feedforward classifier. The classifier returns a similarity score, typically from 0 to 1. This differs from a position-wise Transformer FFN: it is an output classification component for a paired-sentence task. section 4.3
Figure 5.1 Autoregressive Language Models predict the next word by processing a sequence of previous words. They use embedding, positional encoding, and multiple decoder blocks to generate a probability distribution for the next word.
Use the decoder-model diagram to locate feed-forward networks among embeddings, positional encodings, masked self-attention, and repeated decoder layers used for next-token prediction. section 5.1



















