Differentiable activation

As explained in Deep Learning with PyTorch, Second Edition

An activation function that is differentiable so gradients can be computed for learning.

A differentiable activation is a nonlinear function applied after a linear transformation, allowing neural-network parameters to be estimated with gradient descent. Stacking these building blocks helps networks approximate complicated input/output relationships without requiring a handcrafted function for each problem. section 6.1.6

Composing tanh neurons

Four neurons, A through D, use tanh activations with outputs ranging from −1 to 1. Different weights and biases shift or reshape their transitions. Adding and composing these neurons produces bumps, S-curves, and other nonlinear patterns that a single neuron cannot express. section 6.1.6

Figure 6.6 Composing multiple linear units and tanh activation functions to produce nonlinear outputs

Follow the figure to see how linear units and tanh activations combine into increasingly complex nonlinear outputs. section 6.1.6

How Deep Learning with PyTorch, Second Edition covers it

How deep it goes

Explains their role in nonlinear approximation and illustrates composition, but does not derive activation derivatives or training internals.

Examples

Uses tanh neurons A through D, weighted and biased curves, sums of neurons, and two-layer compositions.

What you can do after

Understand how composing differentiable activations lets neural networks represent increasingly complex nonlinear relationships.

How it connects

Connects differentiable activations to linear transformations, gradient descent, neural networks, tanh, and universal approximation.

liveBookmanning.com

Read Deep Learning with PyTorch, Second Edition on liveBook Get the book at manning.com

This explainer was generated by Manning's knowledge graph from the text of Deep Learning with PyTorch, Second Edition. The section links point to the parts of the book it draws on.