Synthetic evaluation data is generated by algorithms or models rather than collected directly from real-world interactions. It can supply controlled test cases when real data is scarce, sensitive, or inaccessible, and can expose rare or extreme situations that ordinary usage data may miss. section 4.2.5 section 4.1
Synthetic inputs are useful for early development and stress testing, but they may miss the ambiguity and diversity of real human behavior. They can oversimplify reality, inherit generator biases, or create an illusion of diversity through shallow variation. Do not use them as the sole evidence for deployment, fairness, or safety decisions. section 4.2.5 section 4.7
Two ways to generate evaluation cases are:
- Rules and templates: Define patterns and constraints, then fill placeholders from controlled lists. Randomization can add variety, and explicit logic can often produce expected outputs. This approach supports reproducible tests and systematic variation. section 4.4.1
- Large language models: Prompt a model to produce examples with specified topics, styles, and constraints. Examples and topic lists can guide the output toward broader coverage and away from repetitive patterns. section 4.4.2
Testing a medical advice chatbot
Use synthetic questions to probe ambiguous symptoms and dangerous follow-ups that may be rare in pilot logs. Combine them with benchmarks for an initial baseline, carefully sampled operational questions for realism, and clinician-authored cases for medical correctness and risk. section 4.2.6
Validate synthetic cases against real data wherever possible:
- Compare characteristics such as length, structure, vocabulary, complexity, topic coverage, and relevant metadata. Large discrepancies may limit generalization to real use. section 4.4.4
- Check whether synthetic cases capture difficulties observed in real failures, such as ambiguous symptom descriptions or poorly specified follow-ups. section 4.4.4
- Compare system performance on synthetic and real cases. Much better synthetic results may indicate cases that are too clean or simple; much worse results may indicate implausible or overly adversarial prompts. section 4.4.4
Table 4.5 Examples of synthetic data generation templates and rules for different AI tasks and evaluation purposes.
|
Domain/Task
|
Purpose
|
Template/Rule
|
Example
|
|
Medical advice / symptom triage
|
Test clarity, safety, symptom
reasoning, medication advice
|
Patient of AGE with SYMPTOM for DURATION asks about MEDICATION
|
“Patient of
72 with shortness of breath for two days asks about ibuprofen.”
|
|
RAG factual retrieval
|
Test
grounding behavior
|
Context: “FACTUAL_STATEMENT.”
Query: “Is
it true that PERTURBED_STATEMENT?”
|
Context:
“Insulin is produced in the pancreas.”
Query: “Is
insulin produced in the liver?
|
|
Logical
reasoning / chain-of-thought evaluation
|
Test
logical consistency, syllogistic reasoning, or rule application
|
“If A implies B and B implies C, does A imply C?”
|
“If all
mammals breathe air and whales are mammals, do whales breathe air?”
|
|
Autonomous
agent scenario testing
|
Validate
planning and constraint satisfaction
|
Agent must
reach GOAL while avoiding OBSTACLE under condition CONTEXT.
|
Agent must
deliver package to Building B while avoiding restricted zone Z during rain.”
|
Compare template and rule examples across evaluation tasks, including medical questions and factual retrieval. section 4.4.1























