liveBook · Manning

Synthetic data

As explained in AI Model Evaluation

Artificially generated data designed to represent properties or scenarios of real-world data.

Synthetic data is artificially generated data used to represent real-world properties or scenarios when historical data is limited, sensitive, imbalanced, or missing important edge cases. section 1.3.3

Use synthetic data to:

  • Test rare failure modes that may be underrepresented in historical logs. section 1.3.3
  • Simulate new product experiences before enough real user interactions exist. section 1.3.3
  • Evaluate LLM and agent behavior before production interaction data becomes available. section 1.3.3

Treat synthetic data as a coverage supplement, not an automatic replacement for real historical data. Validate it against realistic production patterns before relying on its results. section 1.3.3

How AI Model Evaluation covers it

How deep it goes
Defines practical uses and cautions against substitution, but stops short of describing generation or validation procedures.
Examples
Uses rare failure modes, new product experiences, and pre-production LLM or agent interactions as evaluation scenarios.
What you can do after
Evaluate models and product experiences across scenarios unavailable in existing real-world interaction data.
How it connects
Connects synthetic data to offline evaluation, historical data, holdout data, LLMs, and agent behavior.

Also explained in

Compare how 24 books explain synthetic data →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of AI Model Evaluation. The section links point to the parts of the book it draws on.