Ray Tune
Ray Tune is a Ray library for distributed hyperparameter tuning and experiment management.
Designing Deep Learning Systems: A guide for software engineers
Ray Tune is a library for distributed hyperparameter tuning and experiment execution built on the Ray platform.
- How it is used
- Automates hyperparameter searches for model training on local machines, managed clusters, or multinode Ray deployments.
- How deep it goes
- Explains objective functions, search spaces, schedulers, and distributed execution, while stopping short of detailed algorithm internals.
- Examples
- Demonstrates convolutional-network training with learning rate and momentum searches, ASHA scheduling, and multinode cluster execution.
- What you can do after
- Build it from scratch by defining an objective function, search space, scheduler, and distributed Tune run.
Read how Designing Deep Learning Systems: A guide for software engineers explains Ray Tune →
Transformers in Action
Ray Tune is a Ray library for distributed hyperparameter tuning and experiment management.
- How it is used
- Orchestrates distributed hyperparameter searches and experiment management for model training.
- How deep it goes
- Implements dataset conversion, tokenization, distributed training, search scheduling, checkpoint selection, retraining, and model loading.
- Examples
- Uses SuperGLUE CommitmentBank classification with searched learning rates, epochs, batch sizes, and weight decay.
- What you can do after
- Apply it via Ray Tune to compare configurations, select the lowest-loss trial, retrain, restore, and publish a model.
Related concepts
- Rayuses · Ray is an open-source distributed-computing framework for scaling Python, artificial-intelligence, and machine-learning applications across processes, machines, and clusters.
- librarykind of · A software library is a reusable collection of code or components that other programs can incorporate or invoke to provide functionality.
- hyperparameter optimization servicekind of · A software service that coordinates automated searches for effective hyperparameter values by running and comparing multiple training trials.
- ASHA schedulerused by · The ASHA scheduler is an asynchronous successive-halving algorithm that allocates resources to promising hyperparameter trials while stopping underperforming trials early.
- resource isolationcontrasts with · Resource isolation separates or limits computational, storage, or communication resources so one workload cannot materially interfere with another.
- distributed programmingimplements · Distributed programming is the design of software whose computation is coordinated across multiple networked machines, processors, or processes.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.