liveBook · Manning

CalibratedClassifierCV

As explained in Build AI Drug Discovery Pipelines

CalibratedClassifierCV is a scikit-learn estimator that calibrates classifier probabilities using a selected calibration method and data-splitting procedure.

CalibratedClassifierCV adjusts classifier probability estimates so confidence better reflects observed outcomes. In CYP3A4 inhibition prediction, it supports Platt scaling with sigmoid regression or flexible isotonic regression. section 5.1.3

Use calibration carefully:

  • Fit the calibrator on validation data and assess it on untouched test data; reusing data produces optimistically biased results. section 5.1.3
  • Prefer holistic assessment with reliability diagrams and calibration metrics; isotonic regression may lower RMSCE while leaving probability bins empty. section 5.1.3

Calibrating a logistic regressor

Wrap a fitted logistic model in FrozenEstimator, fit CalibratedClassifierCV on validation fingerprints, and generate test probabilities with predict_proba using method='sigmoid' or method='isotonic'.

How Build AI Drug Discovery Pipelines covers it

How deep it goes
Compares sigmoid and isotonic calibration, data separation, reliability diagrams, and RMSCE, but does not detail estimator internals.
Examples
Uses logistic regression, CYP3A4 fingerprints, sigmoid and isotonic calibrators, held-out test probabilities, and reliability diagrams.
What it assumes
Assumes a fitted classifier, validation data, test data, labels, and probability predictions; defines calibration through practical comparisons.
What you can do after
Apply calibration with CalibratedClassifierCV and evaluate probabilities without overlapping fitting, calibration, and assessment data.
How it connects
Links logistic regression, Platt scaling, isotonic regression, reliability diagrams, RMSCE, validation sets, and test sets.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Build AI Drug Discovery Pipelines. The section links point to the parts of the book it draws on.