A residual connection adds a layer’s input representation to its transformed output. In an encoder-only transformer’s attention layer, the operation is: Output = Input + Attention(Input). This lets the representation retain input information while incorporating context from other positions. section 12.3.2
Keeping residue information
For an alanine residue in a protein sequence, the input representation carries information about its identity. Attention adds context, such as whether the residue is part of a helix. Adding the input to the attention output helps preserve the original information rather than replacing it entirely. section 12.3.2
The addition also gives gradients a direct path backward through the network, helping reduce vanishing gradients when many layers are stacked. After the addition, layer normalization normalizes each token across its feature dimensions and applies a learned scale and shift. This “Add & Norm” pattern follows each major encoder component. section 12.3.2
Figure 12.9 The transformer architecture. While the original transformer includes both an encoder and decoder for translation tasks, protein language modeling primarily uses the encoder-only architecture (left). This stack processes the entire sequence in parallel to generate embeddings for every residue.

The encoder stack provides the architectural context for the repeated Add & Norm pattern used to produce protein-residue embeddings. section 12.3.2
Skip connections also appear in the U-Net used for protein contact prediction, where encoder and decoder features are combined at multiple scales. That use is distinct from the attention-layer addition described above. section 11.3.1 section 12.3.2

















