How deeply does Inside Deep Learning: Math, Algorithms, Models cover tanh?

Defines the output range, explains nonlinear placement and saturation, and demonstrates PyTorch usage without deriving tanh internals.

Tanh

As explained in Inside Deep Learning: Math, Algorithms, Models

A nonlinear activation function that maps real-valued inputs into the interval [-1,1].

tanh is a nonlinear activation function that maps every real-valued input into [-1, 1]. In a neural network, placing it between linear layers lets the model represent functions that a stack of linear operations cannot. Around zero, tanh is approximately linear; for larger positive or negative inputs, it approaches its range limits. section 2.2.3 section 6.1

Use tanh when a bounded, historically common nonlinearity fits the model:

Insert one activation between adjacent linear layers rather than after the final output layer. section 2.2.3
Use nn.Tanh() in PyTorch; a two-linear-layer network therefore needs one intermediate activation. section 2.2.3
Account for saturation: tanh and sigmoid can produce vanishing gradients, although bounded outputs are useful for LSTM gating. section 6.1 section 6.6.2

A tanh hidden layer in PyTorch

model = nn.Sequential( nn.Linear(1, 10), nn.Tanh(), nn.Linear(10, 1), ) section 2.2.3

activation_input = np.linspace(-2, 2, num=200) 
    tanh_activation = np.tanh(activation_input) 
    sigmoid_activation = np.exp(activation_input)/(np.exp(activation_input)+1) 
    sns.lineplot(x=activation_input, y=activation_input, 
     color=’black’, label="linear") 
    sns.lineplot(x=activation_input, y=tanh_activation, 
     color=’red’, label="tanh(x)") 
    ax = sns.lineplot(x=activation_input, y=sigmoid_activation, 
     color=’blue’, label="$\sigma(x)$)")

    ax.set_xlabel(’Input value x’)
    ax.set_ylabel(’Activation’)

[13]: Text(0, 0.5, 'Activation')

Compare tanh with a linear function and sigmoid by plotting their activation values across inputs from -2 to 2. section 2.2.3

How Inside Deep Learning: Math, Algorithms, Models covers it

Cover of Inside Deep Learning: Math, Algorithms, Models

liveBook manning.com

Read Inside Deep Learning: Math, Algorithms, Models on liveBook Get the book at manning.com

Also explained in

liveBook manning.com

Grokking Machine LearningA smooth S-shaped activation function that maps real-valued inputs to values between negative one and one.
liveBook manning.com

Optimization Algorithms: AI techniques for design, planning, and control problemsThe hyperbolic tangent is an activation function that maps real-valued inputs to the interval from -1 to 1.
liveBook manning.com

Math and Architectures of Deep LearningA hyperbolic tangent activation function used to introduce nonlinearity in neural networks.

Compare how 4 books explain tanh →

Related concepts

activation functionkind of · An activation function maps a neural-network unit’s or layer’s input to an output, often introducing nonlinearity into the network’s computation.
ReLUcontrasts with · ReLU, or rectified linear unit, is an activation function that maps an input x to max(0, x).
recurrent neural networkused by · A recurrent neural network is a neural network that processes sequences by updating an internal state across successive inputs, allowing earlier inputs to influence later computations.
sigmoidcontrasts with · A sigmoid is a smooth, bounded, S-shaped function that maps real-valued inputs between two limiting values, commonly zero and one.
additive scoreused by · An additive score is a learned attention compatibility function that combines representations to score a query-key pair.

This explainer was generated by Manning's knowledge graph from the text of Inside Deep Learning: Math, Algorithms, Models. The section links point to the parts of the book it draws on.