liveBookmanning.com

What you will learn in Ensemble Methods for Machine Learning

by Gautam Kunapuli

Read it on liveBook Get the book at manning.com

Chapter by chapter

Concepts explained in this book

• gradient boostingGradient boosting is an ensemble-learning technique that sequentially adds models trained to correct the errors of prior models using gradient-based optimization.
• base learnerA base estimator is an individual predictive model whose output contributes to an ensemble or higher-level estimator.
• regularizationA regularization function is a function that assigns a penalty to model parameters or complexity as part of a regularized objective.
• decision treeA decision tree is a predictive model that recursively partitions data according to feature-based decisions, producing predictions at the resulting leaves.
• Newton boostingNewton boosting is a boosting approach that uses first- and second-order loss information to construct successive learners.
• local explanationLocal explainability, or local interpretability, is the ability to explain why a model produced a particular prediction for an individual input.
• linear regressionLinear regression is a model that predicts a target as a linear combination of input variables.
• diversityEnsemble diversity is the degree to which the component models in an ensemble make different predictions or capture different aspects of the data.
• gradient descentGradient descent is an iterative optimization method that updates parameters in the direction opposite the loss gradient.
• weak learnerA weak learner is a simple predictive model that performs only slightly better than random guessing.
• ensembleAn ensemble is a predictive model formed by combining the outputs of multiple component models.
• meta-learnerA meta-estimator is a higher-level estimator whose output is learned from or constructed by combining other estimators.
• surrogate modelA surrogate model is an interpretable approximation trained to reproduce the behavior of a more complex model in a specified region or representation.
• AdaBoostAdaBoost is an adaptive boosting technique that combines sequentially trained weak learners while increasing emphasis on previously misclassified examples.
• glass-box modelA glass-box method is a modeling approach whose internal structure and component contributions can be directly inspected and interpreted.
• cross-validationK-fold cross-validation partitions data into k folds and repeatedly trains and evaluates a model using one fold for validation and the others for training.
• XGBoostXGBoost is an open-source machine-learning library implementing scalable gradient-boosted decision-tree algorithms.
• LightGBMLightGBM is an open-source gradient-boosting library designed for efficient tree-based machine learning.
• importance scoreFeature importance is a score representing the contribution or relevance of an input feature to a model's predictions.
• heterogeneous ensembleA heterogeneous parallel ensemble is an ensemble whose independently trained component models use different model types or learning algorithms.
• model aggregationModel averaging is an aggregation method that combines predictions by averaging outputs from multiple models.
• count dataA count label is a target variable representing a discrete nonnegative integer count of events or items.
• random forestRandomForestRegressor is a scikit-learn estimator that predicts continuous targets by aggregating randomized decision-tree regressors.
• Poisson distributionA Poisson distribution is a discrete probability distribution for the number of events occurring in a fixed interval under a constant-rate event process.
• newton-raphson methodNewton's method is an iterative optimization method that uses first- and second-derivative information to choose updates toward a local minimum.
• generalized linear modelA generalized linear model is a statistical model that combines a linear predictor with a link function and a probability distribution from the exponential family.
• early stoppingEarly stopping is a boosting technique that terminates training after enough base estimators have been added to achieve acceptable validation performance.
• baggingBagging is an ensemble technique that trains multiple models independently and combines their predictions.
• interpretabilityModel interpretability, or explainability, is the degree to which people can understand and explain a model's behavior and predictions.
• SHAPSHAP is a model-agnostic explainability technique that attributes an individual model prediction to its input features using Shapley values.
• link functionA link function connects the expected value of a response distribution to a linear predictor in a generalized linear model.
• Poisson regressionPoisson regression is a generalized linear regression method that models count-valued targets with a Poisson response distribution and typically a log link.
• learning ratelearning_rate is a boosting hyperparameter that scales the contribution of each successive weak learner and controls the rate at which ensemble complexity grows.
• model complexityThe degree of structural or functional richness a model can express, often associated with the number of adjustable or branching components.
• mean target encodingGreedy target encoding uses target-derived statistics computed from training data to encode categorical values, potentially introducing target leakage when the encoded examples contribute to those statistics.
• residual errorA residual is the discrepancy between an observed target and a model prediction, or the corresponding per-example loss gradient in gradient boosting.
• Hessian matrixA Hessian is the square matrix of second-order partial derivatives of a scalar-valued function.
• LeaveOneOutEncoderLeave-one-out target encoding computes each example's categorical target statistic after excluding that example from the statistic.
• Explainable Boosted MachineAn explainable boosting machine is an additive ensemble model that combines boosted shallow-tree components for individual features and selected feature pairs.
• black boxA black-box model is a predictive model whose internal structure or decision process is difficult to understand directly, often requiring specialized explanation methods.
• data leakageData leakage is the unintended use of information during model training or preprocessing that would not be available when making the intended predictions.
• Shapley valueA Shapley value is a cooperative-game-theoretic attribution that assigns each player its average marginal contribution across possible coalitions.
• preprocessingPreprocessing is the transformation of raw data into a representation suitable for machine-learning algorithms.
• bootstrap samplingA bootstrap sample is a dataset replicate formed by sampling observations with replacement from an original dataset.
• Tweedie distributionA Tweedie distribution is a family of exponential-dispersion distributions that, for 1<p<2, represents compound Poisson-gamma behavior with a mass at zero and positive continuous values.
• extremely randomized treesExtra Trees is an ensemble model of highly randomized decision trees that randomizes both candidate splitting features and splitting thresholds.
• homogeneous ensembleA homogeneous parallel ensemble uses base learners of the same type trained independently with the same base-learning algorithm.
• decision stumpA decision stump is a decision tree of depth one containing one decision node and two leaf nodes.
• log-link functionA log-link function relates the logarithm of an expected response to a linear predictor, ensuring a positive response after inversion.
• gradientA gradient is the vector of partial derivatives of a scalar function with respect to its input variables.
• high-cardinality categorical featureA high-cardinality categorical feature is a categorical input variable with a large number of distinct categories.
• additivityAdditivity is the property that attributions for a sum of value functions equal the sum of the separate attributions.
• TreeSHAPTreeSHAP is an efficient SHAP algorithm that exploits the structure of decision trees to compute feature Shapley values.
• overfittingOverfitting is a modeling failure in which a model fits training data too closely and does not necessarily generalize well to new data.
• second derivativeA second derivative is the derivative of a function's first derivative and describes its local curvature with respect to a variable.
• target leakingTarget leakage is the inadvertent use of information derived from a prediction target in model inputs or training computations in a way that would not be available at prediction time.
• line searchLine search is an optimization technique that selects a step length by evaluating an objective along a specified search direction.
• target statisticsA target statistic is a numerical summary of target values associated with a category, typically used to replace that category during encoding.
• pseudo-Huber lossPseudo-Huber loss is a smooth approximation to Huber loss that transitions continuously between quadratic and approximately absolute-error behavior.
• meta-featureA meta-feature is a feature supplied to a higher-level estimator and derived from the output of a base estimator.
• leaf nodeA leaf node is a terminal node of a decision tree that stores the prediction returned for an input reaching it.
• base-learning algorithmA base-learning algorithm is an algorithm used to train an individual base estimator within an ensemble.
• LIMELIME is a model-agnostic explanation technique that approximates individual predictions with an interpretable model fitted near the input being explained.
• CatBoostCatBoost is an open-source gradient-boosting library designed especially to handle categorical features.
• label encodingOrdinal encoding maps each category of a categorical variable to a single numeric value, optionally preserving an inherent category order.
• ordered target statisticsOrdered target statistics are categorical encodings computed from target values of preceding examples in a randomly permuted sequence, preventing an example from influencing its own encoding.
• feature interactionA pairwise feature interaction is a joint effect in which a model's output depends on a combination of two features rather than either feature independently.
• sequential ensemble methodA sequential ensemble is an ensemble whose base models are trained dependently in successive stages to reduce previous mistakes.
• parallel ensemble methodA parallel ensemble is an ensemble whose component base models are trained independently of one another.
• coalitionA coalition is a subset of cooperating players whose collective value is evaluated in a cooperative game.
• coalition vectorA coalition vector is a binary vector indicating which features belong to a selected coalition.
• feature attributionFeature attribution is a technique for assigning portions of a model output to individual input features.
• KernelSHAPKernelSHAP is a SHAP variant that estimates feature Shapley values by fitting a weighted linear surrogate model with the Shapley kernel.
• Shapley kernelThe Shapley kernel is a weighting function that assigns coalition samples weights based on coalition size according to the Shapley computation.
• weight decayL2 regularization penalizes the squared magnitude of model parameters or other model complexity measures in an optimization objective.
• standardizationStandardization rescales numerical features so that each feature has a specified mean and dispersion, commonly zero mean and unit standard deviation.
• LogitBoostLogitBoost is a boosting algorithm that constructs an ensemble by optimizing logistic loss, typically through probability-based classification updates.
• ridge regressionRidge regression is linear regression fitted with an L2 penalty on the coefficients to reduce model complexity.
• permutation feature importancePermutation feature importance estimates a feature's contribution by measuring the change in model performance after that feature's values are randomly permuted.
• categorical dataA categorical variable is an input variable whose values belong to a finite set of discrete categories.
• step lengthA step length is a scalar coefficient that determines the contribution of an update or weak estimator in an iterative model.
• Huber lossHuber loss is a piecewise regression loss that is quadratic for small errors and linear for errors beyond a threshold.
• categorical encodingCategorical encoding is the transformation of categorical feature values into numerical representations suitable for machine-learning models.
• balanced accuracyBalanced accuracy is the average of recall values computed separately for each class.
• partial dependence plotA partial dependence function is the average model prediction obtained by fixing selected feature values and integrating or averaging over the remaining features.
• dirty_catdirty_cat is an open-source Python library providing encoders for noisy and high-cardinality categorical string features.
• modeMajority voting is an aggregation method that selects the class prediction receiving the most individual model votes.
• strong learnerA strong learner is a sufficiently complex predictive model with good performance on its task.
• gradient-based one-side samplingGradient-based one-sided sampling (GOSS) is a gradient-boosting sampling technique that retains examples with large gradients while subsampling and reweighting examples with small gradients.
• hold-out encoding setA hold-out encoding set is a subset of training data reserved exclusively for computing target encodings for other data subsets.
• gamma distributionA gamma distribution is a continuous probability distribution on positive values commonly used to model waiting times or right-skewed magnitudes.
• weighted residualA weighted residual is a per-example prediction discrepancy scaled by a weight that represents the example's contribution to an optimization update.
• exponential lossExponential loss is a classification loss that increases exponentially with the margin by which an example is misclassified.
• hold-out target encodingHold-out target encoding computes category encodings from a separate subset of data excluded from the examples being encoded.
• global explanationA global interpretability method explains a model's overall behavior, features, and decision-making patterns across a broad range of examples.
• local surrogate methodA local surrogate model approximates a black-box model's behavior in the neighborhood of a particular example.
• blendingBlending is an ensemble technique that trains a higher-level learner using predictions generated on a single held-out validation set rather than multiple cross-validation folds.
• ordered boostingOrdered boosting is a boosting approach that uses ordered data processing to reduce prediction-shift bias during training.
• generalized additive modelA generalized additive model is a model whose prediction is formed by adding separate learned functions of input features, optionally including specified interactions.
• class imbalanceClass imbalance is a data-distribution condition in which the classes contain substantially different numbers of examples.
• entropyEntropy is an information-theoretic measure of uncertainty or impurity in a probability distribution.
• elastic netElastic net regression is a linear regression method that combines L1 and L2 regularization penalties.
• lassoLASSO is a linear regression model that uses an L1 penalty to encourage some feature weights to become exactly zero.
• ordinary least squaresOrdinary least squares is a linear-model fitting method that minimizes the sum of squared residuals.
• Hessian weightingA Hessian-weighted gradient is a loss gradient scaled using second-order curvature information from the Hessian.
• fit-complexity tradeoffThe fit-complexity tradeoff is the tension between matching training data closely and limiting model complexity to preserve generalization.
• normal distributionThe normal distribution, also called the Gaussian distribution, is a continuous probability distribution with a symmetric bell-shaped density determined by a mean and variance.
• continuous-valued targetA continuous-valued target is a prediction target whose possible values lie on a continuous numerical scale.
• weak regressorA weak regressor is a regression model with limited predictive strength that can serve as a component in a boosting ensemble.
• hyperparameterA hyperparameter is a configuration value selected before or outside the fitting of a machine-learning model.
• multiclass classificationMulticlass classification is a supervised prediction task whose target has more than two categorical classes.
• multicollinearityMulticollinearity is a condition in which input features are strongly dependent or correlated with one another.
• L1 regularizationL1 regularization penalizes the sum of absolute parameter values to constrain a model and encourage sparse parameters.
• histogram-based gradient boostingHistogram-based gradient boosting is a gradient-boosting technique that discretizes feature values into bins to accelerate split finding and reduce memory use.
• Poisson-gamma distributionA compound Poisson-gamma distribution combines a Poisson count of events with gamma-distributed positive event magnitudes.
• fidelity-interpretability tradeoffThe fidelity-interpretability tradeoff is the tension between accurately approximating a model and keeping the explanatory model simple enough to understand.
• out-of-bag errorAn out-of-bag score is an ensemble performance estimate computed from predictions on examples omitted from the corresponding component training samples.
• validation errorValidation error is a model's prediction error measured on a validation set that was not used to fit the model.
• varianceVariance is a measure of the expected squared deviation of a random variable from its mean.
• Branin functionThe Branin function is a two-variable mathematical test function with multiple local minima used to study optimization algorithms.
• sparse modelA sparse linear model is a linear model with many coefficients constrained or estimated to be zero.
• StratifiedKFoldStratifiedKFold is a scikit-learn cross-validation splitter that creates folds while preserving class proportions across the folds.
• one-hot encodingOne-hot vectorization represents each category of a categorical variable as a separate binary indicator component.
• boostingBoosting is an ensemble technique that combines weak learners trained sequentially so that later learners improve on earlier mistakes.
• coefficient of determinationThe coefficient of determination, or R² score, is a regression metric measuring the proportion of target variance explained by a model relative to a constant-mean baseline.
• logit link functionThe logit link function maps a probability to its log-odds, ln(p/(1-p)).
• logitLog-odds are the natural logarithm of the odds of an event.
• logistic regressionLogistic regression is a statistical classification model that estimates class probabilities by applying a logistic function to a linear combination of input features.
• out-of-bag observationAn out-of-bag sample is the subset of original training examples omitted from a particular bootstrap sample.
• regularization parameterA regularization parameter controls the strength of the complexity penalty relative to the data-fitting loss.
• nominal dataA nominal feature is a categorical feature whose categories have no inherent ordering or quantitative relationship.
• ordinal dataAn ordinal feature is a categorical input variable whose categories have an inherent ordering.
• convergenceConvergence is the approach of an iterative procedure toward a stable solution or stopping condition.
• cardinalityCardinality is the number of unique categories or values represented by a categorical feature.
• character-based string similarityCharacter-based string similarity compares strings according to character-level operations such as insertion, deletion, and substitution.
• GapEncoderGapEncoder is a dirty_cat transformer that represents categories using frequently co-occurring substring combinations.
• MinHashEncoderMinHashEncoder is a dirty_cat transformer that represents categorical strings using hashing applied to their substrings.
• n-gramAn n-gram is a contiguous sequence of n successive tokens, such as characters or words, extracted from a string.
• SimilarityEncoderSimilarityEncoder is a dirty_cat transformer that encodes categorical strings according to their character or n-gram similarities.
• string similarityA string similarity metric is a function that assigns a numeric score indicating how similar two strings are.
• token-based string similarityToken-based string similarity compares strings using tokenized components such as substrings or words.
• multiclass logistic lossSoftmax loss is the multiclass negative log-likelihood loss applied to predicted class probabilities.
• exclusive feature bundlingFeature bundling is a tree-learning optimization that groups compatible features to reduce the effective number of features processed.
• weighted quantile sketchA weighted quantile sketch is an approximate data-structuring algorithm for estimating quantile-based split points while accounting for example weights.
• ColumnTransformerColumnTransformer is a scikit-learn transformer that applies different transformations to specified subsets of input columns and combines their outputs.
• model-agnostic methodA model-agnostic explainability method explains a trained model or its predictions without relying on the model's internal architecture.
• recallRecall is the fraction of actual positive examples that are correctly predicted as positive.
• ExtraTreesExtraTreesClassifier is the scikit-learn estimator class that trains an extremely randomized tree ensemble for classification.
• BaggingClassifierBaggingClassifier is a scikit-learn estimator class that trains an ensemble of base estimators on sampled subsets of data and aggregates their predictions.
• random forest treeA randomized decision tree is a decision tree whose split construction evaluates a randomly selected subset of features rather than all available features.
• squared exponential kernelAn exponential smoothing kernel assigns similarity weights that decrease exponentially with the squared distance between two inputs.
• meta-learningMeta-learning is a machine-learning approach that learns how to select, combine, or improve other learning procedures.
• grid searchGrid search is a model-selection method that evaluates combinations from a specified set of parameter values to identify a performant configuration.
• oddsOdds are the ratio of the probability of an event to the probability of its complement.
• supervised encodingSupervised encoding is a categorical-feature encoding technique that uses target values when deriving category representations.
• marginalizationMarginalization is the computation of a distribution or expectation for selected variables by summing or integrating over the others.
• HistGradientBoostingClassifierHistogramGradientBoostingClassifier is a scikit-learn classifier that accelerates gradient-boosted decision-tree training by using histogram-based feature processing.
• category_encoderscategory_encoders is a Python library that provides encoders for converting categorical variables into numerical representations compatible with scikit-learn workflows.
• train-test leakageTrain-to-test target leakage is the unintended transfer of target information from training-related data into an encoding or evaluation process for test examples.
• absolute errorAbsolute error measures prediction error as the absolute difference between a target and its prediction.
• negative log-likelihoodNegative log-likelihood is the negative logarithm of the likelihood assigned by a probabilistic model to observed data.
• TweedieRegressorTweedieRegressor is a scikit-learn estimator that fits generalized linear regression with a Tweedie distribution and log link.
• point massA point mass is a positive probability concentrated at one exact value rather than spread over an interval.
• linear stackingLinear stacking is stacking in which the second-level estimator is a linear model.
• information retrievalInformation retrieval is the field concerned with finding and ranking relevant information items in response to queries.
• negative gradientA negative gradient is the vector opposite to a function's gradient and indicates a local direction of decrease.
• support vector machineA support vector machine is a supervised learning model that finds a maximum-margin separating function, optionally using kernels to represent nonlinear decision boundaries.
• log transformationA log transformation maps a nonnegative value y to log(1+y), avoiding the undefined logarithm of zero while compressing large values.
• decision ruleA human-readable if-then rule corresponding to a path from the root of a decision tree to a leaf.
• focal lossFocal loss is a classification loss that down-weights well-classified examples and emphasizes poorly classified examples, especially under class imbalance.
• saddle pointA saddle point is a stationary point where a function curves upward in some directions and downward in others, so it is not a local minimizer.
• randomizationRandomization is the deliberate introduction of stochastic variation into data sampling, feature selection, or model training.
• random patchesRandom patches is an ensemble technique that trains models using independently sampled subsets of both training examples and input features.
• random subspacesRandom subspaces is an ensemble technique that trains component models using randomly selected subsets of input features.
• linear blendingLinear blending is an ensemble technique that combines model outputs using a weighted linear combination.
• reweightingReweighting is a technique that changes the relative importance assigned to training examples during iterative learning.
• neural networkA neural network is a parameterized computational model composed of connected units that transform inputs through learned weights and nonlinear functions.
• feature scalingFeature scaling transforms input variables so their numerical magnitudes are placed on comparable scales.
• tree ensembleA tree-based ensemble is an ensemble model whose component predictors are decision trees or tree-derived models.

These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.