liveBook · Manning

Inverted dropout

As explained in Build AI Drug Discovery Pipelines

Inverted dropout scales retained activations during training so their expected magnitude matches the full activation signal at inference.

Inverted dropout keeps dropout’s training-time regularization while preserving activation scale. During training, it randomly zeros activations and multiplies survivors by 1/(1 − p), matching the expected full signal. section 8.2.2

At inference, dropout is disabled and every neuron passes its activation unchanged. The training-time scaling removes the need for inference-time weight adjustment. section 8.2.2

A practical training cycle is:

  • Choose a new random subset of hidden neurons to drop for each training iteration. section 8.2.2
  • Scale retained activations; with p = 0.2, multiply them by 1.25. section 8.2.2
  • Use model.eval() so nn.Dropout passes activations through unchanged during testing. section 8.2.2

Kinase-binder network

A PyTorch network applies nn.Dropout(0.2) after ReLU in its first two fully connected layers, helping reduce overfitting and encourage generalizable features. section 8.2.2

Listing 8.6 Kinase Binder Neural Network Architecture
class KinaseBinderNN(nn.Module):
    def __init__(self, input_size=2048):
        super(KinaseBinderNN, self).__init__()
        self.fc1 = nn.Linear(input_size, 512)
        self.fc2 = nn.Linear(512, 256)
        self.fc3 = nn.Linear(256, 1)
        self.relu = nn.ReLU()
        self.dropout = nn.Dropout(0.2)
        
    def forward(self, x):
        x = self.dropout(self.relu(self.fc1(x)))
        x = self.dropout(self.relu(self.fc2(x)))
        x = self.fc3(x)
        return x
 

Inspect the complete kinase-binder architecture and its dropout placement. section 8.2.2

How Build AI Drug Discovery Pipelines covers it

How deep it goes
Explains random dropping, expected-signal scaling, and PyTorch evaluation behavior, but stops short of deriving dropout mathematically.
Examples
Uses a kinase-binder network with nn.Dropout(0.2) and a separate illustration with a 0.5 dropout rate.
What it assumes
Assumes basic neural-network layers, activations, training, and inference; introduces dropout’s operational behavior in context.
What you can do after
Apply it via PyTorch by placing nn.Dropout between layers and switching to model.eval() for inference.
How it connects
Connects dropout with neural networks, overfitting, generalization, ReLU, inference, and ensemble methods.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Build AI Drug Discovery Pipelines. The section links point to the parts of the book it draws on.