Synthetic data consists of generated examples used to build or expand evaluation sets when real-world labeled data is scarce or incomplete. It can complement real examples and examples created by people, but generated data may be unrealistic or miss relevant cases. section 4.1
For LM-based classification, synthetic examples can be created in two ways:
- Generate examples from intent labels, including when a new project has no real data. section 4.1
- Generate alternative wordings from labeled examples while retaining each example’s intent label. section 4.1.5
- Review generated examples before use; spot-checking can catch unusable outputs. Generating several together can help keep them distinct, and duplicate checks can trigger regeneration. section 4.1.5
Listing 4.6 Generating synthetic data
import dspy
from typing import List
class CreateSynthSignature(dspy.Signature):
"""
Generate a list of 5 queries. Ensure all 5 are distinct from each other.
"""
intent: str = dspy.InputField()
example_questions: List[str] = dspy.OutputField()
lm = dspy.LM(model='gpt-4o-mini', cache=False, temperature=1.5)
dspy.configure(lm=lm)
generator = dspy.Predict(CreateSynthSignature)
synthetic_examples = []
for intent in unique_intents:
prediction = generator(intent=intent)
for example in prediction.example_questions:
synthetic_examples.append(
dspy.Example(message=example,
labels=unique_intents,
intent_label=intent
).with_inputs('message', 'labels')
)
A DSPy example generates five queries for each intent label, then packages them with labels for use in a dataset. section 4.1.5
Listing 4.7 Generating synthetic data
import dspy
lm = dspy.LM(model='gpt-4o-mini', cache=False, temperature=1.5)
dspy.configure(lm=lm)
generator = dspy.Predict("user_message -> alternative_wording")
synthetic_examples = []
for original_example in test_set:
for _ in range(2):
prediction = generator(user_message=original_example.message)
synthetic_examples.append(
dspy.Example(message=prediction.alternative_wording,
labels=unique_intents,
intent_label=original_example.intent_label
).with_inputs('message', 'labels')
)
A second DSPy example creates two alternative phrasings for each test example and keeps its intent label. section 4.1.5
