A hypothesis space is the predefined set of representations or rules a learning algorithm can search. Imagine classifying points on a 2D plot: if the allowed operation is a change of coordinates, the hypothesis space consists of possible coordinate changes. Feedback guides the search toward a transformation that makes the classification easier. The algorithm does not invent operations outside the space it was given. section 1.4
In a neural network, the architecture sets the hypothesis space. Choosing layers, their configuration, and their connections determines the kinds of input-to-output functions the model can represent; training searches for useful weight values within that space. These choices encode assumptions about the problem. A single Dense layer without an activation, for example, assumes that a two-class problem can be separated by an affine transformation. section 3.6.3 section 9.0
Widening the space with an activation
For binary review classification, one Keras model is: model <- keras_model_sequential() |> layer_dense(16, activation = "relu") |> layer_dense(16, activation = "relu") |> layer_dense(1, activation = "sigmoid") Without activations such as relu, stacking Dense layers would still produce a linear operation; extra layers would not extend that hypothesis space. The relu activations allow richer representations, while the final sigmoid produces a score between 0 and 1. section 4.1.3
A useful space is not simply the largest one:
- Convolution layers express an assumption that relevant image patterns are translation invariant. Such architectural choices can make the search easier for gradient descent. section 9.0
- A good solution can exist in the space yet remain hard to find. In temperature forecasting, a simple commonsense heuristic was representable by a two-layer network, but training did not reliably beat the no-learning baseline. section 13.2.3
