liveBook · Manning

Regularization parameter

As explained in Grokking Machine Learning, Second Edition

A regularization parameter is a hyperparameter that controls the strength of the regularization penalty and the trade-off between model fit and complexity.

A regularization parameter controls how strongly a model trades prediction performance for simplicity. It is usually written as λ (lambda): multiply the regularization penalty by λ, then add it to the regression error. A value of 0 removes the penalty; a larger value favors a simpler model, which may fit the data less closely. section 4.11

Error = Regression error + λ Regularization term

The objective combines ordinary regression error with a penalty scaled by λ. section 4.11

Think of λ as a simplicity dial. Turn it down to let the model chase the training points; turn it up to shrink its coefficients and resist overfitting. L1 regularization subtracts a fixed amount from coefficients and can drive them to zero, while L2 repeatedly multiplies them by a factor slightly below one and usually leaves them nonzero. section 4.13 section 10.13

Choosing a value

Try candidate values such as 10, 1, 0.1, and 0.01, then choose the value producing the best validation performance. In a degree-20 polynomial example, an unregularized model fits the training points chaotically, while L1 and L2 models with parameter 0.01 generalize better; the reported testing RMSEs are 0.153 for L1 and 0.104 for L2. section 4.11 section 4.14

The parameter appears in several regularization workflows:

  • Use λ = 0 to recover the model without regularization. section 4.11
  • Use Lasso for L1 regularization or Ridge for L2 regularization. section 4.13
  • In scikit-learn, provide the value as alpha when constructing Lasso or Ridge. section 4.14
  • Tune regularization parameters alongside learning rate, epochs, architecture, and dropout probability in neural networks. section 10.16

How Grokking Machine Learning, Second Edition covers it

How deep it goes
Explains the penalty objective, coefficient-shrinkage procedures, L1/L2 differences, validation choices, and scikit-learn configuration.
Examples
Uses coefficient-shrinkage sequences, candidate powers of ten, a degree-20 polynomial dataset, and Lasso/Ridge testing errors.
What it assumes
Assumes familiarity with model training and error while introducing hyperparameters, coefficient penalties, L1 regularization, and L2 regularization.
What you can do after
Apply it via scikit-learn by configuring Lasso or Ridge alpha values and selecting among candidates with validation performance.
How it connects
Connects regularization parameter with hyperparameters, overfitting, validation, Lasso regression, Ridge regression, polynomial regression, and dropout.

Also explained in

Compare how 4 books explain regularization parameter →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Grokking Machine Learning, Second Edition. The section links point to the parts of the book it draws on.