liveBook · Manning

Real data

As explained in Deep Learning with R, Third Edition

Production data is data generated by the real operating environment in which a deployed machine-learning model is used.

Production data is generated in the environment where a deployed model is used. Training data should represent this environment; otherwise, a model that performs well on tests may fail on real inputs. section 6.1.2

To keep data aligned with deployment:

  • Collect and annotate examples from the intended environment when possible. If that is not possible, understand and actively correct differences between training and production data. section 6.1.2
  • After deployment, monitor new data and model behavior, audit predictions against fresh annotations when possible, and keep collecting difficult examples for future training. section 6.3.4 section 6.3.3

When training images do not match deployment

A dish-recognition app trained on polished food-sharing photos may fail on casual mobile pictures taken in restaurants. The test score can be high while deployed predictions are often wrong. section 6.1.2

“DEWEY DEFEATS TRUMAN”: a famous example of sampling bias
sidebar figure

A visual example of sampling bias reinforces why collected data may not represent the population a model encounters. section 6.1.2

How Deep Learning with R, Third Edition covers it

How deep it goes
Connects representativeness to collection, deployment audits, and ongoing monitoring, then recommends continued annotation and difficult-example collection.
Examples
Contrasts polished food-sharing images with casual restaurant photos and recommends IMDb reviews for movie sentiment.
What you can do after
Apply it by collecting representative inputs, checking deployment data for mismatch, and auditing predictions against fresh annotations.
How it connects
Links production data to training data, data annotation, concept drift, model monitoring, and A/B testing.

Also explained in

Compare how 3 books explain real data →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Deep Learning with R, Third Edition. The section links point to the parts of the book it draws on.