CatBoost

As explained in Ensemble Methods for Machine Learning

CatBoost is an open-source gradient-boosting library designed especially to handle categorical features.

CatBoost is an open-source gradient-boosting library built for datasets containing categorical features such as demographic groups, states, or high-cardinality ZIP codes. It combines categorical encoding and ensemble training instead of requiring a separate encode-then-model pipeline. section 8.0 section 8.3.4

Its central design choices address common problems in categorical boosting:

• Computes ordered target statistics by randomly permuting training examples and using only preceding examples to encode each categorical value. section 8.2.1
• Uses ordered boosting to compute gradients from preceding examples, reducing prediction shift caused by repeatedly reusing training data. section 8.2.1
• Creates categorical combinations and encodes them with target statistics, supporting complex categorical relationships. section 8.2.1
• Uses oblivious decision trees as base estimators, which can lead to faster training. section 8.2

Training and tuning a classifier

A CatBoostClassifier can receive categorical column names directly, train on a train/test split, and predict without separate categorical preprocessing. Cross-validation can search depth, iterations, and learning_rate; early stopping can monitor an evaluation Pool and stop when validation performance begins to overfit. On the Australian Credit Approval dataset, a simple example reported 83.333% accuracy, while a tuned or early-stopped run reported 82.609%. section 8.2.3

Figure 8.8 Ordered target statistics first permutes the examples into a random sequence, using only the previous examples in the ordered sequence to compute the target statistics.

Trace the permutation-based calculation of ordered target statistics step by step. section 8.2.1

Figure 8.9 Ordered boosting also permutes the examples into a random sequence and uses only the previous examples in the ordered sequence to compute the gradients (residuals). Shown here is how the residuals are computed at iteration 4 (using estimator M4 for example x₂), at iteration 5 (using estimator M5 for example x₆), and so on.

Follow how ordered boosting uses preceding examples to compute residuals during successive iterations. section 8.2.1

How Ensemble Methods for Machine Learning covers it

How deep it goes

Explains ordered statistics, ordered boosting, categorical combinations, tree structure, tuning, and early stopping with code examples.

Examples

Uses Australian Credit Approval data, categorical column lists, ordered permutations, grid search, randomized search, and validation curves.

What it assumes

Assumes supervised-learning basics and distinguishes continuous from categorical features while motivating numeric encoding requirements.

What you can do after

Operate/configure CatBoost classifiers with categorical columns, cross-validation, randomized search, learning rates, and early stopping.

How it connects

Connects categorical features with target encoding, gradient boosting, target leakage, prediction shift, oblivious decision trees, cross-validation, and early stopping.

liveBookmanning.com

Read Ensemble Methods for Machine Learning on liveBook Get the book at manning.com

Related concepts

• ordered target statisticsuses · Ordered target statistics are categorical encodings computed from target values of preceding examples in a randomly permuted sequence, preventing an example from influencing its own encoding.
• cross-validationuses · Cross-validation is a resampling technique that evaluates a model by repeatedly fitting it on some portions of available data and assessing it on held-out portions.
• early stoppinguses · Early stopping is a technique that terminates an iterative process before its planned completion when a specified condition indicates that further work is unnecessary or unlikely to improve the objective.
• categorical datauses · Categorical data consists of values representing membership in a discrete set of named or ordered categories rather than measured numerical magnitudes.
• ordered boostinguses · Ordered boosting is a boosting approach that uses ordered data processing to reduce prediction-shift bias during training.

This explainer was generated by Manning's knowledge graph from the text of Ensemble Methods for Machine Learning. The section links point to the parts of the book it draws on.