Calibration data

As explained in Deep Learning with R, Third Edition

Representative data is data whose relevant properties and distribution resemble those of the environment and inputs encountered in production.

Calibration data is representative data whose relevant properties resemble the inputs and environment encountered in production. Models can make sense only of inputs similar to what they have seen. section 6.1.2

Collect calibration data with these practical rules:

• Collect data directly from the environment where the model will be used whenever possible. section 6.1.2
• Match the production population and input format rather than relying on a convenient but different dataset. section 6.1.2
• Treat high test-set accuracy as insufficient when the test data differs from real user inputs. section 6.1.2

Dish-recognition photos

A dish-recognition app trained on professional, well-lit social-network photographs achieved more than 90% test accuracy but failed on random restaurant photos from users’ smartphones. The training data was not representative of production data. section 6.1.2

“DEWEY DEFEATS TRUMAN”: a famous example of sampling bias

Use the sampling-bias example to connect representative-data requirements with misleading conclusions from poorly sampled data. section 6.1.2

How Deep Learning with R, Third Edition covers it

How deep it goes

Explains distribution mismatch through a concrete failure case and collection rule, without formal statistical criteria.

Examples

Uses dish photographs, IMDb reviews, Yelp reviews, and tweets to illustrate mismatched and matched data sources.

What you can do after

Evaluate whether collected data resembles production inputs and redirect collection toward the deployment environment.

How it connects

Connects training data, production data, test sets, data annotation, and sampling bias.

liveBookmanning.com

Read Deep Learning with R, Third Edition on liveBook Get the book at manning.com

Also explained in

• 

  liveBookmanning.com
  Deep Learning with Python, Third EditionData whose distribution matches the real-world inputs a model will encounter after deployment.

Compare how 2 books explain calibration data →

Related concepts

• production datacontrasts with · Production data is data generated or collected through actual operational processes or real-world activity, rather than solely through artificial or controlled experimental settings.

This explainer was generated by Manning's knowledge graph from the text of Deep Learning with R, Third Edition. The section links point to the parts of the book it draws on.