A long-range dependency links sequence positions that are far apart in primary sequence but related by molecular structure or meaning. In molecular generation, distant SMILES tokens may jointly determine ring closure, functional-group relationships, and global valence validity. In proteins, residues separated by dozens or hundreds of positions can become neighbors in the folded structure. section 10.4.2 section 12.3
Long-range dependencies create two practical modeling challenges:
- Basic recurrent neural networks can lose the learning signal across many timesteps because backpropagated gradients become exponentially smaller, making distant relationships difficult to learn. section 10.4.2
- Protein folding requires relationships between distant residues, including beta-sheet alignment and disulfide bonds, while the final structure reflects global interactions across the sequence. section 12.3
- Global self-attention lets every sequence position attend to every other position, allowing models to learn relevant interactions without predefined local neighborhoods. section 12.3
Molecular and protein examples
In the SMILES string “c1ccccc1”, the opening and closing ring markers must correspond despite their separation. In “CC(C)CC(=O)NC1=CC=C(C=C1)O”, parentheses and ring markers must be remembered across multiple positions. In a protein, residues at positions 10 and 200 may interact after folding even though they are distant in sequence. section 10.4.2 section 12.3
Gated recurrent units address the recurrent version of the problem with reset and update gates. The reset gate controls how much previous state contributes to a candidate state; the update gate controls how much candidate information replaces existing memory. Transformers address it with global attention, and protein language models learn such relationships through masked sequence prediction. section 10.4.2 section 12.4
Figure 10.6 Internal architecture of a Gated Recurrent Unit (GRU) cell. The diagram illustrates the information flow through a GRU cell, showing the two key gates (reset gate rt and update gate zt) that control memory retention and updates. The reset gate determines how much previous hidden state information to retain when computing the candidate hidden state h̃t, while the update gate controls the blend between the previous hidden state ht-1 and the new candidate state to produce the final output ht. The mathematical equations on the right define the computations for each component, with σ representing the sigmoid activation function and ⊗ denoting elementwise multiplication.

Use the GRU-cell diagram to trace how reset and update gates retain, replace, and combine information across sequence positions. section 10.4.2





