Tanh

As explained in Math and Architectures of Deep Learning

A hyperbolic tangent activation function used to introduce nonlinearity in neural networks.

The hyperbolic tangent, or tanh, is an activation function that adds nonlinearity to neural networks. Like sigmoid, it transforms each input nonlinearly, but its outputs range from −1 to 1 rather than 0 to 1, so the response is centered around zero. section 8.1.2

Its training advantage follows from the behavior of its gradient:

• Near x = 0, tanh has a larger derivative than sigmoid, producing stronger weight updates and potentially faster convergence. section 8.1.2
• This advantage applies mainly when inputs are centered around zero, as after standardization. section 8.1.2
• Without a nonlinear activation, stacking linear layers would still produce only a single linear transformation. section 11.1

tanh in LeNet

LeNet places a tanh activation layer after every feature map in its convolutional layers. These activations let the network model nonlinear combinations of image inputs; the original architecture used tanh, although ReLU and sigmoid are alternatives. section 11.1

Figure 8.4 Graph of a 1D tanh function.

Inspect the graph to connect tanh's centered output range with its activation-curve shape. section 8.1.2

Figure 8.5 Graph of the derivatives of 1D sigmoid and tanh functions

Compare the tanh and sigmoid derivative graphs to visualize their different gradient strengths near zero. section 8.1.2

How Math and Architectures of Deep Learning covers it

How deep it goes

Defines the output range and explains gradient behavior near zero, but stops short of implementation details or a derivative formula.

Examples

Uses derivative comparison graphs and LeNet's tanh layers after convolutional feature maps.

What you can do after

Evaluate/understand how centered outputs and stronger near-zero gradients make tanh useful as a neural-network activation.

How it connects

Connects tanh with sigmoid, ReLU, standardization, convolutional layers, feature maps, pooling, and softmax classification.

liveBookmanning.com

Read Math and Architectures of Deep Learning on liveBook Get the book at manning.com

Also explained in

• 

  liveBookmanning.com
  Grokking Machine LearningA smooth S-shaped activation function that maps real-valued inputs to values between negative one and one.
• 

  liveBookmanning.com
  Optimization Algorithms: AI techniques for design, planning, and control problemsThe hyperbolic tangent is an activation function that maps real-valued inputs to the interval from -1 to 1.
• 

  liveBookmanning.com
  Inside Deep Learning: Math, Algorithms, ModelsA nonlinear activation function that maps real-valued inputs into the interval [-1,1].

Compare how 4 books explain tanh →

Related concepts

• activation functionkind of · An activation function maps a neural-network unit’s or layer’s input to an output, often introducing nonlinearity into the network’s computation.
• ReLUcontrasts with · ReLU, or rectified linear unit, is an activation function that maps an input x to max(0, x).
• recurrent neural networkused by · A recurrent neural network is a neural network that processes sequences by updating an internal state across successive inputs, allowing earlier inputs to influence later computations.
• sigmoidcontrasts with · A sigmoid is a smooth, bounded, S-shaped function that maps real-valued inputs between two limiting values, commonly zero and one.
• additive scoreused by · An additive score is a learned attention compatibility function that combines representations to score a query-key pair.

This explainer was generated by Manning's knowledge graph from the text of Math and Architectures of Deep Learning. The section links point to the parts of the book it draws on.