liveBook · Manning

Stochastic gradient descent

As explained in Build AI Drug Discovery Pipelines

Mini-batch gradient descent is an optimization method that computes each parameter update from a small subset of the training data.

Stochastic gradient descent (SGD) minimizes a model’s cost by updating parameters from randomly selected training instances or small mini-batches rather than the full dataset. This reduces memory demands, often speeds convergence, and can help optimization escape saddle points, but introduces noisier movement near a minimum. section 4.2.3 section 4.5

Practical training decisions shape whether SGD converges usefully:

  • Tune the learning rate so updates shrink as the cost approaches a minimum; overly large steps can overshoot. section 4.2.3
  • Combine an iteration limit, an acceptable cost threshold, and a marginal-improvement threshold to avoid premature termination. section 4.2.3
  • Use gradient clipping for steep cost-function drops and scale features when differing magnitudes slow convergence. section 4.2.3

Solubility regression with Scikit-Learn

A descriptor-based solubility pipeline combines median imputation, degree-two polynomial features, standardization, and SGDRegressor with L2 regularization, inverse-scaling learning rate, early stopping, and epsilon-insensitive loss. On held-out solubility data, the resulting model reached test RMSE 1.2783, versus 1.3787 for ESOL and 2.4585 for a y-scrambled baseline. section 4.3.3 section 4.5

Figure 4.11 Impact of learning step size on the ability for gradient descent to converge.

Visualizes how learning-step size affects whether gradient descent converges. section 4.2.3

Listing 4.9 Evaluate Best Performing Model against ESOL and y-Scrambled Baselines
from tqdm.notebook import trange
from sklearn.linear_model import SGDRegressor
engineered_features = list(RDKIT_DESCRIPTORS.keys())
X = pd.concat([X_train, X_val])
y = pd.concat([y_train, y_val])
 
sgd_regressor = make_pipeline(
  SimpleImputer(strategy="median"),
  PolynomialFeatures(degree=2),
  StandardScaler(),
  SGDRegressor(
      alpha=0.1, early_stopping=True, learning_rate='invscaling',
      loss="epsilon_insensitive", penalty='l2'
  ),
)
 
sgd_regressor.fit(X, y)
 
PandasTools.AddMoleculeColumnToFrame(test_df, 'Drug', 'Molecule')
test_df['fragment_count'] = test_df.Molecule.apply(
    num_fragments
)
test_df = test_df[test_df.fragment_count == 1]
for desc, func in RDKIT_DESCRIPTORS.items():
  test_df[desc] = test_df["Molecule"].apply(func)
X_test, y_test = test_df[engineered_features], test_df.Y
 
y_test_preds = sgd_regressor.predict(X_test)
test_rmse = np.sqrt(mean_squared_error(y_test, y_test_preds))
 
print(f"SGD Regressor (SVR) Test RMSE: {test_rmse:.4f}")
# Output -> SGD Regressor (SVR) Test RMSE: 1.2783
 
esol_test_pred_original = (
    0.16
    - 0.63 * test_df.logP
    - 0.0062 * test_df["Molecular Weight"]
    + 0.066 * test_df["Number of Rotatable Bonds"]
    - 0.74 * test_df["Aromatic Proportion"]
)
print(
  f"RMSE with original ESOL coefficients: "
  f"{np.sqrt(mean_squared_error(test_df.Y, esol_test_pred_original)):.4f}"
)
# Output -> RMSE with original ESOL coefficients: 1.3787
 
y_scrambled_rmse = []
num_iterations = 100
for i in trange(num_iterations):
  y_scrambled = y.sample(frac=1, random_state=None).to_numpy()
  sgd_regressor.fit(X, y_scrambled)
  y_pred = sgd_regressor.predict(X_test)
  y_scrambled_rmse.append(np.sqrt(mean_squared_error(y_test, y_pred)))
 
average_rmse = sum(y_scrambled_rmse) / num_iterations
print(
    f'Average RMSE over {num_iterations} y-scrambled iterations: '
    f'{average_rmse}'
)
# Output -> Average RMSE over 100 y-scrambled iterations: 2.4585

Provides a runnable SGDRegressor evaluation pipeline with preprocessing and baseline comparison. section 4.3.3

How Build AI Drug Discovery Pipelines covers it

How deep it goes
Explains sampling, convergence, stopping, clipping, and scaling, then demonstrates an SGDRegressor pipeline without developing optimization theory deeply.
Examples
Uses AqSolDB solubility prediction, molecular descriptors, SGDRegressor, loss curves, early stopping, and ESOL and y-scrambled baselines.
What you can do after
Apply it via Scikit-Learn’s SGDRegressor and assess convergence and generalization against solubility baselines.
How it connects
Connects gradient descent with batch gradient descent, mini-batch gradient descent, learning rate, gradient clipping, feature scaling, and linear models.

Also explained in

Compare how 17 books explain stochastic gradient descent →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Build AI Drug Discovery Pipelines. The section links point to the parts of the book it draws on.