ASHA is a scalable asynchronous successive-halving scheduler for hyperparameter optimization. It stops underperforming trials early and gives more time and resources to promising trials. section 5.4.3
Use ASHA within a Ray Tune workflow:
- Define an objective function that reads configuration values, trains a model, evaluates it, and reports a metric. section 5.4.3
- Define the search space for each hyperparameter and pass an ASHAScheduler to tune.run. section 5.4.3
- Increase num_samples to search more efficiently and support a larger search space. section 5.4.3
Listing 5.3 Getting started with Ray Tune
# Step 1: define objective_function
def objective_function(config):
model = ConvNet() #1
model.to(device)
optimizer = optim.SGD( #2
model.parameters(), lr=config["lr"], #2
momentum=config["momentum"]) #2
for i in range(10):
train(model, optimizer, train_loader) #3
acc = test(model, test_loader)
tune.report(mean_accuracy=acc) #4
# Step 2: define search space for each hyperparameter
search_space = {
"lr": tune.sample_from(lambda spec:
10**(-10 * np.random.rand())),
"momentum": tune.uniform(0.1, 0.9) #5
}
# Uncomment this to enable distributed execution
# `ray.init(address="auto")`
# Step 3: start the HPO execution
analysis = tune.run(
objective_function,
num_samples=20,
scheduler=ASHAScheduler(metric="mean_accuracy", mode="max"),
config=search_space)
# check HPO progress and result
# obtain a trial dataframe from all run trials
# of this `tune.run` call.
dfs = analysis.trial_dataframes
The reference listing shows a ConvNet objective, reported accuracy, learning-rate and momentum search spaces, and the Ray Tune execution call. section 5.4.3
