liveBook · Manning

Calibration data

As explained in Deep Learning with Python, Third Edition

Data whose distribution matches the real-world inputs a model will encounter after deployment.

Calibration data matches the real-world inputs a model will encounter after deployment. Without that match, strong test-set accuracy can collapse when users provide different inputs. section 6.1.2

Professional food photos versus phone snapshots

A dish-recognition app trained on polished food photographs scored above 90% on its test set, yet failed on mobile photos of random restaurant dishes. The training images were not representative of production data. Collecting images directly from the deployment environment would address the mismatch. section 6.1.2

Use these rules when collecting calibration data:

  • Collect data directly from the environment where the model will be used whenever possible. section 6.1.2
  • Match the expected users and input sources: classify new IMDB reviews with IMDB reviews, and tweets with tweets from similar users. section 6.1.2

How Deep Learning with Python, Third Edition covers it

How deep it goes
Explains distribution mismatch through a worked image example and recommends collecting data from the deployment environment.
Examples
Uses dish photographs, IMDB reviews, Yelp reviews, Twitter updates, and a historical sampling-bias example.
What you can do after
Apply it by collecting and annotating inputs from the expected deployment environment and user population.
How it connects
Connects training data, production data, test sets, sentiment classification, and sampling bias.

Also explained in

Compare how 6 books explain calibration data →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Deep Learning with Python, Third Edition. The section links point to the parts of the book it draws on.