Black box

As explained in Ensemble Methods for Machine Learning

A black-box model is a predictive model whose internal structure or decision process is difficult to understand directly, often requiring specialized explanation methods.

A black-box model produces predictions without offering a directly understandable account of how inputs became outputs. Its internal structure or parameters may be unavailable, or available but too complex to translate into an intuitive input–output relationship. Deep neural networks are common examples; random forests, AdaBoost, gradient boosting, and Newton boosting can also become black boxes because ensembling creates complex feature interactions. section 9.1.1

Black-box status is about interpretability, not predictive quality. Two models can perform similarly while differing sharply in how readily people can inspect them: a polynomial classifier may be opaque, while a decision tree can express its behavior as understandable decision rules. Glass-box models expose an intuitively comprehensible structure; black-box models generally need specialized explainers. section 9.0 section 9.1.1

Figure 9.1 With black-box machine-learning models, we can only use the input-output pairs to analyze and explain model behavior. The model internals in a black box are either unavailable or aren’t directly interpretable. With glass-box machine-learning models, in addition to input-output pairs, the model internals are also intuitively interpretable.

Compare the information available for explanation: black-box analysis works from input–output pairs, whereas glass-box analysis can also inspect interpretable internals. section 9.1.1

Explain a black box at two complementary scopes:

• Global explanation describes the model’s general decision-making behavior and broadly relevant factors across many examples. Model-agnostic global methods use inputs and outputs rather than internals, so they can apply to any machine-learning model. section 9.3
• Local explanation focuses on why the model made a particular prediction for one example, supporting a decision about what to do next. section 9.3

Explanation matters when a prediction must be acted on or trusted. In diabetes diagnosis, understanding a specific diagnosis can inform prescriptions, preventative care, or palliative care. More generally, explanation supports trust, model debugging, failure-case discovery, performance improvement, and detection of bias or data problems. section 9.0

Explaining an XGBoost model with a surrogate tree

A global surrogate turns an opaque model into an inspectable approximation. Collect examples representative of the original problem distribution, obtain the black box’s predictions for them, and train a glass-box model—such as a decision tree or GLM—to predict those outputs. If surrogate predictions closely match black-box predictions, inspect the surrogate to describe the black box’s behavior. Balance fidelity, meaning how well the surrogate mimics the black box, against interpretability: a more complex tree may match better but be harder to understand. For Bank Marketing, a 14-leaf decision tree is trained on predictions from an XGBoost ensemble. section 9.3.3 section 9.6

Figure 9.11 The procedure to train a global surrogate model from the predictions of a black-box model. Both models are trained on the same surrogate training examples. However, the surrogate model is trained on the predictions of the black-box model, so that it can learn to mimic its predictions. If the black box and surrogate make the same prediction, then the surrogate can be used to explain the black-box model’s prediction.

Follow the surrogate workflow: train the interpretable model on the black box’s predictions, then compare their outputs before using the surrogate as an explanation. section 9.3.3

How Ensemble Methods for Machine Learning covers it

How deep it goes

Defines opacity and contrasts glass boxes, then develops input–output explainers and surrogate-model fidelity–interpretability tradeoffs.

Examples

Contrasts diabetes polynomial and decision-tree classifiers, then approximates a Bank Marketing XGBoost ensemble with a 14-leaf surrogate tree.

What it assumes

Assumes familiarity with machine-learning models, predictions, features, decision trees, and ensemble methods.

What you can do after

Evaluate/understand opaque predictions using global or local explanations and assess surrogate fidelity against interpretability.

How it connects

Connects black boxes to glass-box models, ensemble methods, global explainability, local explainability, feature importance, partial dependence, and surrogate models.

liveBookmanning.com

Read Ensemble Methods for Machine Learning on liveBook Get the book at manning.com

Also explained in

• 

  liveBookmanning.com
  Get Programming: Learn to code with PythonA black box is an abstraction of a system whose internal steps are hidden while its inputs, outputs, and overall behavior remain visible.
• 

  liveBookmanning.com
  Inside AIA black-box algorithm is an algorithm whose internal operations and reasoning are not readily interpretable from its inputs and outputs.
• 

  liveBookmanning.com
  Grokking StatisticsA black-box model is a model whose internal decision process is not readily understood or inspected by its users.
• 

  liveBookmanning.com
  Outlier Detection in PythonA black-box model is a predictive model whose internal decision process is not readily understandable from its inputs and outputs.
• 

  liveBookmanning.com
  Experimentation for EngineersA function or system whose internal process is not known while its inputs and outputs are observable.

Compare how 6 books explain black box →

Related concepts

• surrogate modelused by · A surrogate model is an approximate model that estimates the behavior or outputs of a more complex, costly, or otherwise unavailable model, process, or function.
• interpretabilitycontrasts with · Interpretability is the degree to which people can understand a model’s behavior, outputs, predictions, or the factors that produce them.
• moduleequivalent to · A module is a distinct, bounded software unit that groups related code or functionality and provides an organizational or interaction boundary within a larger system.
• partial dependence plotused by · A partial dependence plot shows a model's average predicted response as selected features vary, averaging over the values of the remaining features.
• neural networkincludes · A neural network is a computational model of interconnected, parameterized units that learns transformations from inputs to outputs.

This explainer was generated by Manning's knowledge graph from the text of Ensemble Methods for Machine Learning. The section links point to the parts of the book it draws on.