Inverted dropout keeps dropout’s training-time regularization while preserving activation scale. During training, it randomly zeros activations and multiplies survivors by 1/(1 − p), matching the expected full signal. section 8.2.2
At inference, dropout is disabled and every neuron passes its activation unchanged. The training-time scaling removes the need for inference-time weight adjustment. section 8.2.2
A practical training cycle is:
- Choose a new random subset of hidden neurons to drop for each training iteration. section 8.2.2
- Scale retained activations; with p = 0.2, multiply them by 1.25. section 8.2.2
- Use model.eval() so nn.Dropout passes activations through unchanged during testing. section 8.2.2
Kinase-binder network
A PyTorch network applies nn.Dropout(0.2) after ReLU in its first two fully connected layers, helping reduce overfitting and encourage generalizable features. section 8.2.2
Listing 8.6 Kinase Binder Neural Network Architecture
class KinaseBinderNN(nn.Module):
def __init__(self, input_size=2048):
super(KinaseBinderNN, self).__init__()
self.fc1 = nn.Linear(input_size, 512)
self.fc2 = nn.Linear(512, 256)
self.fc3 = nn.Linear(256, 1)
self.relu = nn.ReLU()
self.dropout = nn.Dropout(0.2)
def forward(self, x):
x = self.dropout(self.relu(self.fc1(x)))
x = self.dropout(self.relu(self.fc2(x)))
x = self.fc3(x)
return x
Inspect the complete kinase-binder architecture and its dropout placement. section 8.2.2
