Model-free optimization

As explained in Designing Deep Learning Systems: A guide for software engineers

An optimization approach that makes no model-based assumptions about the relationship between evaluated configurations or the training code.

Model-free optimization is a hyperparameter optimization approach that makes no assumptions about training code and ignores correlation between HPO trials. It treats each trial configuration independently, making grid search and random search its common methods. section 5.2.2

Use it to select hyperparameters for training code without modeling how one trial’s result predicts another’s. It fits the broader automatic HPO workflow, where compute and an algorithm search for hyperparameters that minimize a predefined loss on a dataset. section 5.2.2 section 5.5

The two common approaches differ in how they produce trial configurations:

• Grid search defines a limited value set for every hyperparameter, forms their Cartesian product, and evaluates trials using those grid values. section 5.2.2
• Random search samples configurations randomly from the specified hyperparameter space until its search budget is exhausted. section 5.2.2
• Grid search grows exponentially as more hyperparameters or larger spaces are added, and can spend substantial compute on nonoptimal regions because every candidate set is treated equally. section 5.2.2

Learning rate and batch-size search

A grid search can combine learning-rate values {0.1, 0.005, 0.001} with batch sizes {10, 40, 100}, producing trials such as (0.1, 10), (0.1, 40), and (0.1, 100). A random search instead samples from ranges such as learning rate [0.001, 0.1] and batch size [10, 100] until the budget ends. section 5.2.2

Figure 5.4 Comparison of grid search and random search for minimizing a function with one important and one unimportant parameter. (Source: Figure 1.1. of “Hyperparameter Optimization,” by Matthias Feurer and Frank Hutter, in AutoML: Methods, Systems, Challenges, eds. Frank Hutter, Lars Kotthoff, and Joaquin Vanschoren; Springer, 2019. www.automl.org/wp-content/uploads/2019/05/AutoML_Book_Chapter1.pdf)

Compare how grid and random search cover a function with one important and one unimportant parameter. section 5.2.2

Model-free optimization is one of three broad HPO algorithm categories alongside Bayesian optimization and multifidelity optimization. No algorithm is universally best; choose according to the HPO task and its constraints. section 5.2.2 section 5.5

How Designing Deep Learning Systems: A guide for software engineers covers it

How deep it goes

Defines trial independence and outlines grid and random search with scaling trade-offs, while deferring mathematical reasoning.

Examples

Illustrates Cartesian-product trials and random sampling with learning-rate and batch-size search spaces.

What it assumes

Introduces model-free methods from basic HPO terminology while assuming a training code and tunable hyperparameters.

What you can do after

Apply it via grid or random search to generate HPO trial configurations within a defined search budget.

How it connects

Positions model-free optimization alongside hyperparameter optimization, grid search, random search, Bayesian optimization, and multifidelity optimization.

liveBookmanning.com

Read Designing Deep Learning Systems: A guide for software engineers on liveBook Get the book at manning.com

Related concepts

• grid searchincludes · Grid search is a search method that evaluates configurations formed from specified candidate values, typically every combination in their Cartesian product.
• random searchincludes · Random search is an optimization technique that explores a search space by evaluating candidate solutions or configurations selected through randomized choices.
• hyperparameter tuningkind of · Hyperparameter tuning is the process of selecting or adjusting configurable model or procedure settings and evaluating them against a chosen objective.
• model-free methodincludes · A model-free method is a reinforcement-learning approach that learns policies or value functions directly from experience without an explicit model of environment dynamics and rewards.

This explainer was generated by Manning's knowledge graph from the text of Designing Deep Learning Systems: A guide for software engineers. The section links point to the parts of the book it draws on.