liveBook · Manning

Residual connection

As explained in Build AI Drug Discovery Pipelines

A residual connection adds a layer's input representation to that layer's transformed output.

A residual connection adds a layer’s input representation to its transformed output. In an encoder-only transformer’s attention layer, the operation is: Output = Input + Attention(Input). This lets the representation retain input information while incorporating context from other positions. section 12.3.2

Keeping residue information

For an alanine residue in a protein sequence, the input representation carries information about its identity. Attention adds context, such as whether the residue is part of a helix. Adding the input to the attention output helps preserve the original information rather than replacing it entirely. section 12.3.2

The addition also gives gradients a direct path backward through the network, helping reduce vanishing gradients when many layers are stacked. After the addition, layer normalization normalizes each token across its feature dimensions and applies a learned scale and shift. This “Add & Norm” pattern follows each major encoder component. section 12.3.2

Figure 12.9 The transformer architecture. While the original transformer includes both an encoder and decoder for translation tasks, protein language modeling primarily uses the encoder-only architecture (left). This stack processes the entire sequence in parallel to generate embeddings for every residue.

The encoder stack provides the architectural context for the repeated Add & Norm pattern used to produce protein-residue embeddings. section 12.3.2

Skip connections also appear in the U-Net used for protein contact prediction, where encoder and decoder features are combined at multiple scales. That use is distinct from the attention-layer addition described above. section 11.3.1 section 12.3.2

How Build AI Drug Discovery Pipelines covers it

How deep it goes
Explains input-output addition, gradient flow, and subsequent layer normalization without a code implementation.
Examples
Illustrates preservation of an alanine’s identity alongside helix context in a protein-residue representation.
What you can do after
Evaluate and understand how residual addition supports information retention and gradient flow in a transformer encoder.
How it connects
Connects residual connections with self-attention, layer normalization, vanishing gradients, and skip connections in U-Net.

Also explained in

Compare how 18 books explain residual connection →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Build AI Drug Discovery Pipelines. The section links point to the parts of the book it draws on.