An activation function transforms a layer’s affine output into a new representation. It supplies the nonlinearity that lets stacked Dense layers learn complex patterns instead of collapsing into one linear transformation. section 2.3.5 section 4.1.3
A three-layer classifier
A practical model uses relu in its intermediate layers and sigmoid at the output: model <- keras_model_sequential() |> layer_dense(16, activation = "relu") |> layer_dense(16, activation = "relu") |> layer_dense(1, activation = "sigmoid"). The intermediate relu layers learn richer representations; sigmoid converts the final value into a score between 0 and 1 that can represent positive-review probability. section 4.1.3
Choose an activation according to the representation or output the layer must provide:
- Use relu to zero out negative values and introduce nonlinearity in intermediate layers. section 4.1.3
- Use sigmoid when the final output should lie between 0 and 1 and be interpretable as a probability. section 4.1.3
- Recognize that alternatives such as prelu and elu exist, although relu is identified as the most popular choice. section 4.1.3
Without an activation, a Dense layer performs only a dot product followed by bias addition. Repeating such affine operations cannot expand the hypothesis space; inserting activations enables nonlinear geometric transformations and useful depth. section 2.3.5 section 4.1.3
Listing 4.4 Model definition
model <- keras_model_sequential() |> layer_dense(16, activation = "relu") |> layer_dense(16, activation = "relu") |> layer_dense(1, activation = "sigmoid")
Browse the complete sequential model definition showing relu in the hidden layers and sigmoid in the final layer. section 4.1.3
