liveBook · Manning

Regularization

As explained in Build AI Drug Discovery Pipelines

Regularization is a modeling technique that constrains learning to reduce overreliance on particular features and improve performance on unseen data.

Regularization constrains learning to reduce overfitting: a model’s tendency to learn noise in its training data rather than patterns that generalize to unseen examples. In molecular prediction, this matters when a model must make reliable predictions for compounds it has never encountered. section 3.4.1 section 3.3.1

One approach adds a penalty for large coefficients to the cost function. Training then balances prediction error against coefficient size. The penalty strength λ controls this tradeoff: a higher value imposes a stronger constraint. section 3.4.1 section 3.5

Common coefficient penalties include:

  • L1: Penalizes the sum of absolute weights and can drive unimportant weights to zero, effectively selecting features. section 3.4.1
  • L2: Penalizes the sum of squared weights, encouraging smaller, more evenly distributed weights. section 3.4.1
  • Elastic net: Combines L1 and L2 penalties. section 3.4.1

Predicting hERG blockage

Molecular fingerprints represent structures as bits used as model features. Because one structure can set multiple bits, features can be correlated, making coefficients unstable and difficult to interpret. L1 regularization is especially relevant to these sparse features because it can set some weights to zero. section 3.4.1

Regularization is not limited to coefficient penalties. For a decision tree, limiting depth or requiring more samples at a leaf can reduce variance, though this can increase bias. Pruning instead removes splits that do not add value, using validation error to assess them. section 5.3.1

Predicting drug–target affinity

A graph neural network uses dropout after fully connected layers: during training, it randomly zeroes elements of feature vectors to discourage reliance on particular features. Its graph convolution layers also share parameters across neighborhoods, reducing the effective number of parameters. section 11.5.2 section 11.5.6

Listing 11.7 Process Molecular Graph through GCN Layers
def forward_mol(self, data_mol):
    mol_x, mol_edge_index, mol_batch = \
        data_mol.x, data_mol.edge_index, data_mol.batch
 
    x = self.relu(self.mol_conv1(mol_x, mol_edge_index))
    x = self.relu(self.mol_conv2(x, mol_edge_index))
    x = self.relu(self.mol_conv3(x, mol_edge_index))
 
    x = global_mean_pool(x, mol_batch)
 
    x = self.relu(self.mol_fc_g1(x))
    x = self.dropout(x)
    x = self.mol_fc_g2(x)
    x = self.dropout(x)
    return x

The molecular graph forward pass applies dropout after each fully connected layer. section 11.5.2

Choose regularization strength and other hyperparameters by evaluation rather than assuming more constraint is always better. Cross-validation uses training data for both training and evaluation; grid search and randomized search provide ways to explore hyperparameters. section 3.5

How Build AI Drug Discovery Pipelines covers it

How deep it goes
Explains coefficient penalties, tree constraints, dropout, and parameter sharing; defers full coefficient diagnostics and tuning configurations.
Examples
Illustrates regularization with hERG fingerprint coefficients, decision-tree pruning, and dropout in drug–target affinity prediction.
What you can do after
Evaluate regularization approaches and identify controls for coefficient penalties, decision trees, and neural networks.
How it connects
Connects regularization to overfitting, multicollinearity, feature selection, bias–variance tradeoffs, and cross-validation.

Also explained in

Compare how 17 books explain regularization →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Build AI Drug Discovery Pipelines. The section links point to the parts of the book it draws on.