Ray is an open-source Python framework for scaling machine-learning applications across distributed computing resources. Its parallel processing layer supports distributed training and hyperparameter optimization without requiring prior distributed-systems expertise. section 8.0 section 8.2
A Ray-based tuning workflow separates data, training, and search:
- Ray Data converts Hugging Face dataset splits into Ray-native datasets for scalable processing and parallel preprocessing. section 8.2
- TorchTrainer distributes a training function across configured workers and GPUs, consuming training and evaluation dataset shards. section 8.2
- Tune searches learning rate, epochs, batch size, and weight decay while ASHA schedules trials using evaluation loss. section 8.2
- Checkpoints preserve the best result, which can be loaded into a Hugging Face model and pushed to the Hub. section 8.2
Listing 8.1 Using Ray data with Hugging Face
datasets = load_dataset("super_glue", "cb") #1
ray_datasets = { #2
"train": ray.data.from_items(hf_dataset["train"].to_list()),
"validation": ray.data.from_items(hf_dataset["validation"].to_list()),
"test": ray.data.from_items(hf_dataset["test"].to_list()),
}
Convert each Hugging Face SuperGLUE CommitmentBank split with ray.data.from_items before distributed preprocessing and training. section 8.2
Tuning a sequence-classification model
Load the CommitmentBank task, tokenize premise–hypothesis pairs, and attach labels. Define a training function that obtains Ray dataset shards and creates Torch batches. Instantiate TorchTrainer with worker and GPU scaling, then wrap it in Tuner with candidate hyperparameters and an ASHA scheduler. Select the trial with minimum evaluation loss, retrain using its configuration, fit the trainer, load the checkpoint, and optionally publish the model to the Hugging Face Hub. section 8.2
