liveBook · Manning

Synthetic data

As explained in Evaluating AI Systems

Artificially generated data designed to represent selected characteristics or scenarios rather than collected directly from real-world events.

Synthetic evaluation data is generated by algorithms or models rather than collected directly from real-world interactions. It can supply controlled test cases when real data is scarce, sensitive, or inaccessible, and can expose rare or extreme situations that ordinary usage data may miss. section 4.2.5 section 4.1

Synthetic inputs are useful for early development and stress testing, but they may miss the ambiguity and diversity of real human behavior. They can oversimplify reality, inherit generator biases, or create an illusion of diversity through shallow variation. Do not use them as the sole evidence for deployment, fairness, or safety decisions. section 4.2.5 section 4.7

Two ways to generate evaluation cases are:

  • Rules and templates: Define patterns and constraints, then fill placeholders from controlled lists. Randomization can add variety, and explicit logic can often produce expected outputs. This approach supports reproducible tests and systematic variation. section 4.4.1
  • Large language models: Prompt a model to produce examples with specified topics, styles, and constraints. Examples and topic lists can guide the output toward broader coverage and away from repetitive patterns. section 4.4.2

Testing a medical advice chatbot

Use synthetic questions to probe ambiguous symptoms and dangerous follow-ups that may be rare in pilot logs. Combine them with benchmarks for an initial baseline, carefully sampled operational questions for realism, and clinician-authored cases for medical correctness and risk. section 4.2.6

Validate synthetic cases against real data wherever possible:

  • Compare characteristics such as length, structure, vocabulary, complexity, topic coverage, and relevant metadata. Large discrepancies may limit generalization to real use. section 4.4.4
  • Check whether synthetic cases capture difficulties observed in real failures, such as ambiguous symptom descriptions or poorly specified follow-ups. section 4.4.4
  • Compare system performance on synthetic and real cases. Much better synthetic results may indicate cases that are too clean or simple; much worse results may indicate implausible or overly adversarial prompts. section 4.4.4
Table 4.5 Examples of synthetic data generation templates and rules for different AI tasks and evaluation purposes.
Domain/Task
Purpose
Template/Rule
Example
Medical advice / symptom triage
Test clarity, safety, symptom reasoning, medication advice
Patient of AGE with SYMPTOM for DURATION asks about MEDICATION
“Patient of 72 with shortness of breath for two days asks about ibuprofen.”
RAG factual retrieval
Test grounding behavior
Context: “FACTUAL_STATEMENT.”
Query: “Is it true that PERTURBED_STATEMENT?”
Context: “Insulin is produced in the pancreas.”
Query: “Is insulin produced in the liver?
Logical reasoning / chain-of-thought evaluation
Test logical consistency, syllogistic reasoning, or rule application
“If A implies B and B implies C, does A imply C?”
“If all mammals breathe air and whales are mammals, do whales breathe air?”
Autonomous agent scenario testing
Validate planning and constraint satisfaction
Agent must reach GOAL while avoiding OBSTACLE under condition CONTEXT.
Agent must deliver package to Building B while avoiding restricted zone Z during rain.”

Compare template and rule examples across evaluation tasks, including medical questions and factual retrieval. section 4.4.1

How Evaluating AI Systems covers it

How deep it goes
Explains template-based and LLM-based generation, then gives checks for realism, failure-mode alignment, and difficulty against real data.
Examples
Uses medical chatbot questions, templated reasoning probes, and simulated obstacles to illustrate evaluation cases.
What you can do after
Apply it through templates or large language models to create controlled evaluation cases, and assess them against real examples.
How it connects
Connects synthetic data with benchmarks, operational data, stress testing, fairness evaluation, and safety evaluation.

Also explained in

Compare how 24 books explain synthetic data →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Evaluating AI Systems. The section links point to the parts of the book it draws on.