A hidden state is a vector that acts as a recurrent neural network’s memory. At each position, the network combines the current input with the previous hidden state to produce an output and an updated state. The updated state carries information forward; the final state summarizes the sequence processed so far. section 10.4.1
Reading a molecule
A SMILES string is a sequence representation of a molecule. For acetic acid, CC(=O)O, an RNN processes the tokens in order: after the second C, it has information about two connected carbons; the following parentheses mark a branch, and =O indicates a double-bonded oxygen. The evolving hidden state gives later steps access to information from earlier tokens. section 10.4.1
Listing 10.7 One Step of RNN Computation
def rnn_step(x_t, h_prev, W_input, W_hidden, W_output, b_hidden, b_output):
"""
One step of RNN computation.
Args:
x_t: Current input (e.g., embedded token)
h_prev: Previous hidden state (the memory)
W_input: Weights for processing input
W_hidden: Weights for processing previous hidden state
W_output: Weights for generating output
b_hidden, b_output: Bias terms
Returns:
y_t: Output at current timestep
h_t: New hidden state
"""
h_t = torch.tanh(
W_input @ x_t + W_hidden @ h_prev + b_hidden
) #A
y_t = W_output @ h_t + b_output #B
return y_t, h_t #CThis PyTorch RNN step combines the current input and previous state using learned weights and a bias, applies tanh, and uses the resulting state to compute an output. section 10.4.1
A simple recurrent state can lose earlier information as it is repeatedly updated. A gated recurrent unit (GRU) manages this tradeoff with a reset gate, which controls how much of the previous state contributes to a candidate state, and an update gate, which blends the previous and candidate states. This helps preserve longer-range dependencies, but requires more computation than a simple RNN cell. section 10.4.2
Figure 10.6 Internal architecture of a Gated Recurrent Unit (GRU) cell. The diagram illustrates the information flow through a GRU cell, showing the two key gates (reset gate rt and update gate zt) that control memory retention and updates. The reset gate determines how much previous hidden state information to retain when computing the candidate hidden state h̃t, while the update gate controls the blend between the previous hidden state ht-1 and the new candidate state to produce the final output ht. The mathematical equations on the right define the computations for each component, with σ representing the sigmoid activation function and ⊗ denoting elementwise multiplication.

The GRU cell diagram traces how the reset and update gates shape the next hidden state. section 10.4.2
In an encoder-decoder model, the final hidden state can serve as a fixed-size summary passed to the decoder. This creates a bottleneck for long sequences: a protein with hundreds of amino acids must fit its relevant information into one vector, while repeated updates can overwrite earlier details. Keeping the hidden states from all positions gives the decoder another option. section 12.3.1
Figure 12.8 The mechanics of cross-attention. A decoder state acts as a query (qj) and compares itself against all encoder states (keys, K). These comparisons generate attention weights (α), which are used to compute a weighted sum of the encoder states (values, V), creating a context vector (cj) specific to the current prediction step.

Cross-attention lets a decoder state compare against encoder states and form a weighted context vector for the current prediction, rather than relying only on the final state. section 12.3.1



