liveBook · Manning

Stratified split

As explained in Rearchitecting LLMs

A stratified split divides data into subsets while preserving the representation of relevant categories in each subset.

When categories are unevenly represented, a random train/test split can leave a rare category poorly covered in one subset. Stratification distributes examples according to their category labels, keeping the category mix in each subset similar to the full dataset. In a function-calling dataset, the function invoked by each example provides the label. section 13.2.1

Deduplicate queries before splitting to prevent the same query from appearing in both training and test data. Then pass one label per example to the split function’s `stratify` parameter. section 13.2.1 section 13.2.1

Function-calling data

After deduplication, 177 examples are split with a 30% test size and function labels for stratification. The resulting sets contain 123 training examples and 54 test examples. section 13.2.1

Listing 13.3 Stratified train/test split
RANDOM_STATE = 42
TEST_SIZE = 0.30
function_labels = [
    json.loads(ex["answers"])[0]["name"] 
    for ex in domain_examples
]
 
train_examples, test_examples = train_test_split(
    domain_examples, test_size=TEST_SIZE, 
    stratify=function_labels, random_state=RANDOM_STATE,
)
 
print(f"Train examples: {len(train_examples)}")
print(f"Test examples: {len(test_examples)}")

See the train/test split code, including how function labels, test size, and random state are supplied. section 13.2.1

How Rearchitecting LLMs covers it

How deep it goes
Explains proportional label-based distribution and demonstrates a split, without detailing the algorithm’s internals.
Examples
Uses function-call labels to split 177 deduplicated examples into 123 training and 54 test records.
What it assumes
Assumes examples have category labels and deduplicates queries first to avoid overlap between training and test data.
What you can do after
Apply stratification by passing one function label per example to the split function’s `stratify` parameter.
How it connects
Connects stratification to train/test splitting and data leakage prevention.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Rearchitecting LLMs. The section links point to the parts of the book it draws on.