A development dataset tracks whether an application improves across iterations. Keep it separate from the validation data used to optimize prompts, so progress comparisons are less prone to overfitting the validation set. section 4.1.2 section 5.2
In a four-way workflow, each partition has a distinct role:
- Training examples provide demonstrations for prompts. section 4.1.2
- Validation examples help the optimizer choose prompts. section 4.1.2
- Development examples measure the baseline and each iteration. section 4.1.2
- Reserve the test set for a final estimate of performance. section 4.1.2
A practical split
One intent-classification setup uses 100 training examples, 300 validation examples, and 1,000 each for development and testing. These sizes support reasonably fast, lower-cost runs; a separate development set is not always used. section 5.2
Listing 5.2 Creating training, validation, and development examples
import numpy
import dspy
from datasets import load_dataset
ATIS_INTENT_MAPPING = { #1
'abbreviation': "Abbreviation and Fare Code
Meaning Inquiry",
'aircraft': "Aircraft Type Inquiry",
#'aircraft+flight+flight_no': "", #2
'airfare': "Airfare and Fees Questions",
#'airfare+flight_time': "",
'airline': "Airline Information Request",
#'airline+flight_no': "",
'airport': "Airport Information and Queries",
'capacity': "Aircraft Seating Capacity Inquiry",
'cheapest': "Cheapest Fare Inquiry",
'city': "Airport Location Inquiry",
'distance': "Airport Distance Inquiry",
'flight': "Flight Booking Request",
#'flight+airfare': "",
'flight_no': "Flight Number Inquiry",
'flight_time': "Time Inquiry",
'ground_fare': "Ground Transportation Cost Inquiry",
'ground_service': "Ground Transportation Inquiry",
#'ground_service+ground_fare': "Airport Ground Transportation and
Cost Query",
'meal': "Inquiry about In-flight Meals",
'quantity': "Flight Quantity Inquiry",
'restriction': "Flight Restriction Inquiry"
}
def create_examples_from_set(set_name, n=-1): #3
ds = load_dataset("tuetschek/atis")
ds.set_format(type='pandas')
df = ds[set_name][:]
df['intent'] = df['intent'].map(ATIS_INTENT_MAPPING)
df = df.dropna(subset='intent')
if n > 0:
df = df.sample(n=n, random_state=42)
examples = []
for index in df.index:
row = df.loc[index]
examples.append(
dspy.Example(message=row['text'],
intent_label=row['intent']).with_inputs('message')
)
return examples
train_val_set = create_examples_from_set('test') #4
np.random.shuffle(train_val_set)
train_set = train_val_set[:100]
val_set = train_val_set[100:400]
dev_test_set = create_examples_from_set('train') #5
np.random.shuffle(dev_test_set )
dev_set = dev_test_set [:1000]
test_set = dev_test_set[1000:2000]
print(len(train_set), len(val_set), len(dev_set), len(test_set)) #6
Browse the example code for creating training, validation, and development examples. section 5.2
