liveBook · Manning

Weight decay

As explained in Build AI Drug Discovery Pipelines

L2 regularization penalizes the sum of squared model weights, shrinking them without generally setting them to zero.

Weight decay is L2 regularization: it adds a penalty based on the sum of squared model weights, encouraging smaller, more evenly distributed coefficients without generally making them zero. This can reduce overfitting and stabilize models when features are correlated, such as molecular fingerprints. section 3.4.1

Use weight decay by considering three practical points:

  • Tune the regularization parameter λ: larger values impose stronger constraints and produce simpler models. section 3.4.1
  • Balance underfitting and overfitting: excessive regularization raises training error, while insufficient regularization can produce high-variance models that generalize poorly. section 4.4.3
  • Compare training and validation performance across regularization values instead of treating a single model score as sufficient evidence. section 4.4.3

Molecular generation with VAE-CYC

A VAE-CYC molecular generator uses a conservative learning rate of 0.0001 with weight decay regularization, alongside dropout, word dropout, and free bits. Here, weight decay is one component of a broader strategy for producing useful molecular representations and generation behavior. section 10.4.5

Figure 3.16 Our model weights are exceedingly high given the model’s output space.

Inspect the weight-magnitude visualization to connect the regularization penalty with unusually large model weights. section 3.4.1

Figure 4.20 Validation curves for SVR and preprocessing hyperparameters.

Use the validation curves to examine how L2 regularization changes SVR complexity and the gap between training and test performance. section 4.4.3

How Build AI Drug Discovery Pipelines covers it

How deep it goes
Defines the squared-weight penalty and λ, then interprets validation curves but defers full diagnostics and hyperparameter-search details.
Examples
Uses molecular fingerprints, SVR validation curves, and VAE-CYC training with a 0.0001 learning rate.
What it assumes
Assumes model weights, training error, test error, and overfitting; defines L2 regularization and its contrast with L1.
What you can do after
Apply it via an SVR estimator by comparing validation performance across regularization strengths.
How it connects
Links regularization with multicollinearity, overfitting, validation curves, SVR, dropout, and variational autoencoders.

Also explained in

Compare how 11 books explain weight decay →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Build AI Drug Discovery Pipelines. The section links point to the parts of the book it draws on.