L1 regularization
L1 regularization is a technique that adds a penalty proportional to the sum of absolute model-parameter values, encouraging sparse parameter estimates.
Machine Learning for Tabular Data: XGBoost, Deep Learning, and AI
L1 regularization adds a penalty proportional to the sum of absolute model coefficients, encouraging sparse coefficient estimates.
- How it is used
- Selects useful predictors while reducing regression-model complexity and overfitting.
- How deep it goes
- Defines the absolute-coefficient penalty, explains sparsity and multicollinearity, and applies Lasso in a cross-validated pipeline.
- Examples
- Uses a polynomial tabular-regression pipeline with Lasso and cross-validation, plus bootstrapped stability selection with L1 logistic regression.
- What you can do after
- Apply it via scikit-learn Lasso or L1-penalized logistic regression, then evaluate sparse models with cross-validation.
Grokking Machine Learning
L1 regularization adds the L1 norm of a model's non-bias coefficients as a complexity penalty to its regression error.
- How it is used
- Reduces overfitting by penalizing model complexity and can zero out features when a dataset contains many.
- How deep it goes
- Defines the L1 penalty, explains coefficient shrinkage, and compares polynomial-model results without deriving the optimization procedure.
- Examples
- Compares regularized polynomial models using a 200-column dataset and contrasts their training and testing performance.
- What you can do after
- Apply L1 regularization through Turi Create’s linear_regression.create by setting l1_penalty.
Read how Grokking Machine Learning explains L1 regularization →
Ensemble Methods for Machine Learning
L1 regularization penalizes the sum of absolute parameter values to constrain a model and encourage sparse parameters.
- How it is used
- Controls linear-regression complexity while encouraging sparse weights and implicit feature selection.
- How deep it goes
- Defines the absolute-value penalty and demonstrates its coefficient effect, but does not derive optimization internals.
- Examples
- Uses synthetic noisy data with two highly correlated features and compares coefficients, MSE, and mean absolute deviation.
- What you can do after
- Apply it via scikit-learn's Lasso estimator and evaluate predictions with MSE and mean absolute error.
Read how Ensemble Methods for Machine Learning explains L1 regularization →
Deep Learning with PyTorch, Second Edition
A regularization method that adds the sum of absolute weight values to the objective, encouraging sparse parameters.
- How it is used
- Uses a weight penalty to stabilize generalization and reduce overfitting in neural-network training.
- How deep it goes
- Defines the penalty and its sparse-weight effect, but does not provide an L1 implementation or optimization derivation.
- What you can do after
- Evaluate L1 regularization as a sparse-weight alternative to L2 regularization for controlling overfitting.
Read how Deep Learning with PyTorch, Second Edition explains L1 regularization →
Machine Learning System Design
A regularization penalty based on the sum of absolute parameter values that promotes sparse model coefficients.
- How it is used
- Uses L1 regularization within Lasso regression for embedded feature selection.
- How deep it goes
- Explains coefficient elimination and method tradeoffs, but does not derive the penalty mathematically.
- Examples
- Illustrates selection with Lasso regression, where increasing regularization reduces coefficients to zero one by one.
- What you can do after
- Apply it via Lasso regression to prune features by identifying coefficients reduced to zero.
Read how Machine Learning System Design explains L1 regularization →
Deep Learning with Python, Third Edition
A regularization method that penalizes the absolute values of weights to encourage sparsity.
- How it is used
- Uses an added weight penalty to reduce overfitting during model development and validation-based tuning.
- How deep it goes
- Defines the absolute-value penalty and shows its Keras declaration, but does not work through L1 training behavior or results.
- Examples
- Uses Keras declarations for L1 and combined L1/L2 regularizers within a broader IMDb classification example.
- What you can do after
- Apply it via Keras by passing an L1 regularizer to a layer’s weight-regularizer argument.
Read how Deep Learning with Python, Third Edition explains L1 regularization →
Related concepts
- regularizationkind of · Regularization is a modeling technique that constrains or penalizes solutions during fitting or optimization to discourage undesirable complexity or behavior and improve generalization or estimation.
- l2 normuses · The L2 norm is the nonnegative Euclidean magnitude of a vector, equal to the square root of the sum of its components’ squares, and to absolute value for a scalar.
- feature selectionuses · Feature selection is the process of choosing a subset of input features for use in a modeling, analysis, or prediction task.
- ridge regressioncontrasts with · Ridge regression is linear regression that applies an L2 penalty to coefficient magnitudes, shrinking estimates toward zero to reduce overfitting.
- lassoused by · LASSO is a linear regression model that uses an L1 penalty to encourage some feature weights to become exactly zero.
- XGBClassifiercontrasts with · XGBClassifier is XGBoost’s scikit-learn-compatible estimator for classification using gradient-boosted decision trees.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.