liveBook · Manning

Hidden state

As explained in Build AI Drug Discovery Pipelines

A hidden state is a learned intermediate vector that summarizes sequence information available to a recurrent or sequential model at a particular position.

A hidden state is a vector that acts as a recurrent neural network’s memory. At each position, the network combines the current input with the previous hidden state to produce an output and an updated state. The updated state carries information forward; the final state summarizes the sequence processed so far. section 10.4.1

Reading a molecule

A SMILES string is a sequence representation of a molecule. For acetic acid, CC(=O)O, an RNN processes the tokens in order: after the second C, it has information about two connected carbons; the following parentheses mark a branch, and =O indicates a double-bonded oxygen. The evolving hidden state gives later steps access to information from earlier tokens. section 10.4.1

Listing 10.7 One Step of RNN Computation
def rnn_step(x_t, h_prev, W_input, W_hidden, W_output, b_hidden, b_output):
    """
    One step of RNN computation.
    Args:
        x_t: Current input (e.g., embedded token)
        h_prev: Previous hidden state (the memory)
        W_input: Weights for processing input
        W_hidden: Weights for processing previous hidden state
        W_output: Weights for generating output
        b_hidden, b_output: Bias terms
    
    Returns:
        y_t: Output at current timestep
        h_t: New hidden state
    """
    h_t = torch.tanh(
        W_input @ x_t + W_hidden @ h_prev + b_hidden
    )    #A
    y_t = W_output @ h_t + b_output    #B
    return y_t, h_t    #C

This PyTorch RNN step combines the current input and previous state using learned weights and a bias, applies tanh, and uses the resulting state to compute an output. section 10.4.1

A simple recurrent state can lose earlier information as it is repeatedly updated. A gated recurrent unit (GRU) manages this tradeoff with a reset gate, which controls how much of the previous state contributes to a candidate state, and an update gate, which blends the previous and candidate states. This helps preserve longer-range dependencies, but requires more computation than a simple RNN cell. section 10.4.2

Figure 10.6 Internal architecture of a Gated Recurrent Unit (GRU) cell. The diagram illustrates the information flow through a GRU cell, showing the two key gates (reset gate rt and update gate zt) that control memory retention and updates. The reset gate determines how much previous hidden state information to retain when computing the candidate hidden state h̃t, while the update gate controls the blend between the previous hidden state ht-1 and the new candidate state to produce the final output ht. The mathematical equations on the right define the computations for each component, with σ representing the sigmoid activation function and ⊗ denoting elementwise multiplication.

The GRU cell diagram traces how the reset and update gates shape the next hidden state. section 10.4.2

In an encoder-decoder model, the final hidden state can serve as a fixed-size summary passed to the decoder. This creates a bottleneck for long sequences: a protein with hundreds of amino acids must fit its relevant information into one vector, while repeated updates can overwrite earlier details. Keeping the hidden states from all positions gives the decoder another option. section 12.3.1

Figure 12.8 The mechanics of cross-attention. A decoder state acts as a query (qj) and compares itself against all encoder states (keys, K). These comparisons generate attention weights (α), which are used to compute a weighted sum of the encoder states (values, V), creating a context vector (cj) specific to the current prediction step.

Cross-attention lets a decoder state compare against encoder states and form a weighted context vector for the current prediction, rather than relying only on the final state. section 12.3.1

How Build AI Drug Discovery Pipelines covers it

How deep it goes
Works through a simple RNN update and GRU gating, then contrasts final-state compression with attention over encoder states.
Examples
Illustrates state updates with acetic acid SMILES and a 500-amino-acid protein sequence.
What you can do after
Builds a simple RNN step from scratch using PyTorch tensor operations and interprets its hidden-state update.
How it connects
Connects recurrent neural networks and gated recurrent units to encoder-decoder models, long-range dependencies, and cross-attention.

Also explained in

Compare how 4 books explain hidden state →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Build AI Drug Discovery Pipelines. The section links point to the parts of the book it draws on.