Stochastic gradient descent (SGD) minimizes a model’s cost by updating parameters from randomly selected training instances or small mini-batches rather than the full dataset. This reduces memory demands, often speeds convergence, and can help optimization escape saddle points, but introduces noisier movement near a minimum. section 4.2.3 section 4.5
Practical training decisions shape whether SGD converges usefully:
- Tune the learning rate so updates shrink as the cost approaches a minimum; overly large steps can overshoot. section 4.2.3
- Combine an iteration limit, an acceptable cost threshold, and a marginal-improvement threshold to avoid premature termination. section 4.2.3
- Use gradient clipping for steep cost-function drops and scale features when differing magnitudes slow convergence. section 4.2.3
Solubility regression with Scikit-Learn
A descriptor-based solubility pipeline combines median imputation, degree-two polynomial features, standardization, and SGDRegressor with L2 regularization, inverse-scaling learning rate, early stopping, and epsilon-insensitive loss. On held-out solubility data, the resulting model reached test RMSE 1.2783, versus 1.3787 for ESOL and 2.4585 for a y-scrambled baseline. section 4.3.3 section 4.5
Figure 4.11 Impact of learning step size on the ability for gradient descent to converge.

Visualizes how learning-step size affects whether gradient descent converges. section 4.2.3
Listing 4.9 Evaluate Best Performing Model against ESOL and y-Scrambled Baselines
from tqdm.notebook import trange
from sklearn.linear_model import SGDRegressor
engineered_features = list(RDKIT_DESCRIPTORS.keys())
X = pd.concat([X_train, X_val])
y = pd.concat([y_train, y_val])
sgd_regressor = make_pipeline(
SimpleImputer(strategy="median"),
PolynomialFeatures(degree=2),
StandardScaler(),
SGDRegressor(
alpha=0.1, early_stopping=True, learning_rate='invscaling',
loss="epsilon_insensitive", penalty='l2'
),
)
sgd_regressor.fit(X, y)
PandasTools.AddMoleculeColumnToFrame(test_df, 'Drug', 'Molecule')
test_df['fragment_count'] = test_df.Molecule.apply(
num_fragments
)
test_df = test_df[test_df.fragment_count == 1]
for desc, func in RDKIT_DESCRIPTORS.items():
test_df[desc] = test_df["Molecule"].apply(func)
X_test, y_test = test_df[engineered_features], test_df.Y
y_test_preds = sgd_regressor.predict(X_test)
test_rmse = np.sqrt(mean_squared_error(y_test, y_test_preds))
print(f"SGD Regressor (SVR) Test RMSE: {test_rmse:.4f}")
# Output -> SGD Regressor (SVR) Test RMSE: 1.2783
esol_test_pred_original = (
0.16
- 0.63 * test_df.logP
- 0.0062 * test_df["Molecular Weight"]
+ 0.066 * test_df["Number of Rotatable Bonds"]
- 0.74 * test_df["Aromatic Proportion"]
)
print(
f"RMSE with original ESOL coefficients: "
f"{np.sqrt(mean_squared_error(test_df.Y, esol_test_pred_original)):.4f}"
)
# Output -> RMSE with original ESOL coefficients: 1.3787
y_scrambled_rmse = []
num_iterations = 100
for i in trange(num_iterations):
y_scrambled = y.sample(frac=1, random_state=None).to_numpy()
sgd_regressor.fit(X, y_scrambled)
y_pred = sgd_regressor.predict(X_test)
y_scrambled_rmse.append(np.sqrt(mean_squared_error(y_test, y_pred)))
average_rmse = sum(y_scrambled_rmse) / num_iterations
print(
f'Average RMSE over {num_iterations} y-scrambled iterations: '
f'{average_rmse}'
)
# Output -> Average RMSE over 100 y-scrambled iterations: 2.4585Provides a runnable SGDRegressor evaluation pipeline with preprocessing and baseline comparison. section 4.3.3
















