When categories are unevenly represented, a random train/test split can leave a rare category poorly covered in one subset. Stratification distributes examples according to their category labels, keeping the category mix in each subset similar to the full dataset. In a function-calling dataset, the function invoked by each example provides the label. section 13.2.1
Deduplicate queries before splitting to prevent the same query from appearing in both training and test data. Then pass one label per example to the split function’s `stratify` parameter. section 13.2.1 section 13.2.1
Function-calling data
After deduplication, 177 examples are split with a 30% test size and function labels for stratification. The resulting sets contain 123 training examples and 54 test examples. section 13.2.1
Listing 13.3 Stratified train/test split
RANDOM_STATE = 42
TEST_SIZE = 0.30
function_labels = [
json.loads(ex["answers"])[0]["name"]
for ex in domain_examples
]
train_examples, test_examples = train_test_split(
domain_examples, test_size=TEST_SIZE,
stratify=function_labels, random_state=RANDOM_STATE,
)
print(f"Train examples: {len(train_examples)}")
print(f"Test examples: {len(test_examples)}")See the train/test split code, including how function labels, test size, and random state are supplied. section 13.2.1
