Surrogate model

As explained in Ensemble Methods for Machine Learning

A surrogate model is an interpretable approximation trained to reproduce the behavior of a more complex model in a specified region or representation.

A surrogate model is an interpretable glass-box model trained to mimic a complex black-box model. Rather than learning the original target, it learns the black box’s predictions, so its structure can be inspected as an explanation of the black box’s behavior. A decision tree or generalized linear model can serve as the surrogate. section 9.3.3 section 9.6

The central question is fidelity: does the surrogate make predictions sufficiently similar to the black box? Training therefore balances fidelity against interpretability. A larger decision tree may reproduce more of the ensemble’s behavior, but its added depth and leaf nodes make it harder to understand; a smaller tree is clearer but may omit important behavior. section 9.3.3 section 9.6

A global surrogate explains behavior across a problem space:

• Choose surrogate examples with the same distribution as the examples used for the black box; use the original training data when available or an alternate representative sample. section 9.3.3
• Run the black box on those examples and use its predictions as labels for the surrogate training set. section 9.3.3
• Fit an interpretable model, then assess whether its predictions match the black box closely enough before using it as an explanation. section 9.3.3
• Inspect the fitted glass box—for example, its decision paths or feature weights—to obtain a holistic account of the black box’s learned behavior. section 9.3.3 section 9.6

Figure 9.11 The procedure to train a global surrogate model from the predictions of a black-box model. Both models are trained on the same surrogate training examples. However, the surrogate model is trained on the predictions of the black-box model, so that it can learn to mimic its predictions. If the black box and surrogate make the same prediction, then the surrogate can be used to explain the black-box model’s prediction.

The diagram separates the original prediction task from the explanation task: both models see the same examples, but only the surrogate is trained to reproduce the black box’s outputs. section 9.3.3

Explaining an XGBoost classifier globally

For Bank Marketing predictions, train a decision-tree classifier on the original examples paired with XGBoost predictions. Restricting the tree to 14 leaf nodes makes it inspectable, while the resulting tree can reveal explanatory paths from its root to a leaf. The chosen size represents a practical compromise, not proof that the tree exactly captures the ensemble. section 9.3.3

A local surrogate answers a narrower question: why did the black box make this prediction for this example? LIME fits a linear model near the example of interest. It creates perturbed surrogate examples, gives closer examples greater weight, and fits the linear model to approximate local black-box behavior. Its coefficients express local feature contributions rather than a global description. section 9.4.1 section 9.6

Figure 9.14 LIME creates a surrogate training set of examples in the locality of the example whose prediction needs to be explained. These examples are further weighted by their distance. This is indicated by the sizes of the surrogate examples, with closer examples getting higher weights (and shown larger). A weighted loss function is used to fit a linear surrogate model, which provides local explanations.

Nearby perturbed examples receive higher weight, letting the linear surrogate prioritize fidelity around one prediction instead of across the full data distribution. section 9.4.1

SHAP likewise uses a local linear surrogate, but constructs examples through feature coalitions. A coalition vector marks which feature values come from the example being explained; absent features are replaced with values from another sampled example. This approximation avoids directly training a model for every possible feature coalition, which grows infeasibly with the number of features. For tree-based models, TreeSHAP computes Shapley values efficiently. section 9.4.2 section 9.6

Surrogates are explanations of a model, not necessarily explanations of the real-world process behind the model. Their usefulness depends on representative surrogate data and adequate fidelity. Global trees summarize overall behavior; LIME and SHAP focus on individual predictions. LIME and SHAP are designed for individual feature importance and do not fundamentally evaluate feature interactions. section 9.3.3 section 9.6

How Ensemble Methods for Machine Learning covers it

How deep it goes

Details surrogate-data construction, fidelity tradeoffs, decision trees, LIME weighting, and SHAP coalitions without deriving all optimization formulas.

Examples

Uses Bank Marketing XGBoost predictions, a 14-leaf decision-tree surrogate, and local explanations for test example 3104.

What it assumes

Assumes familiarity with black-box models, predictions, decision trees, linear models, and feature values.

What you can do after

Evaluate/understand whether an interpretable model faithfully approximates a black box globally or near one prediction.

How it connects

Connects black-box models with decision trees, generalized linear models, LIME, SHAP, Shapley values, and TreeSHAP.

liveBookmanning.com

Read Ensemble Methods for Machine Learning on liveBook Get the book at manning.com

Also explained in

• 

  liveBookmanning.com
  Experimentation for EngineersA surrogate function is a model that approximates a measured response as a function of parameter values and estimates the response at unmeasured values.
• 

  liveBookmanning.com
  Bayesian Optimization in ActionA surrogate model is a predictive model that approximates an objective function from observed evaluations.
• 

  liveBookmanning.com
  Outlier Detection in PythonA proxy model is an approximate model that imitates the behavior of another model or detector for analysis or explanation.
• 

  liveBookmanning.com
  Machine Learning for Tabular Data: XGBoost, Deep Learning, and AIA surrogate function is an approximate model of an objective function that predicts objective behavior across untested parameter settings.

Compare how 5 books explain surrogate model →

Related concepts

• Optunaused by · Optuna is an open-source Python framework for automated hyperparameter optimization through repeated trials that explore parameter configurations.
• Bayesian optimizationused by · Bayesian optimization is a sequential method for optimizing an expensive or unknown objective by using a probabilistic model to guide evaluations.
• interpretabilityused by · Interpretability is the degree to which people can understand a model’s behavior, outputs, predictions, or the factors that produce them.
• black boxuses · A black box is a system or component whose internal workings are treated as opaque while its inputs, outputs, or observable behavior are considered.
• Gaussian processproduced by · A Gaussian process is a probability distribution over functions in which every finite collection of function values has a joint Gaussian distribution.
• labelcontrasts with · A label is a value or annotation associated with an item that identifies its target category, outcome, or reference answer for learning or evaluation.

This explainer was generated by Manning's knowledge graph from the text of Ensemble Methods for Machine Learning. The section links point to the parts of the book it draws on.