Tanh
The hyperbolic tangent, or tanh, is a smooth nonlinear function that maps real-valued inputs to values between −1 and 1.
Grokking Machine Learning
A smooth S-shaped activation function that maps real-valued inputs to values between negative one and one.
- How it is used
- Uses tanh as a neural-network activation function that squishes values and helps construct nonlinear decision boundaries.
- How deep it goes
- Explains tanh’s output range, shape, sigmoid comparison, and vanishing-gradient limitation without deriving its formula.
- Examples
- Compares tanh visually with sigmoid and ReLU in a three-function activation plot.
- What you can do after
- Evaluate tanh against sigmoid and ReLU when selecting activation functions for neural-network training.
Optimization Algorithms: AI techniques for design, planning, and control problems
The hyperbolic tangent is an activation function that maps real-valued inputs to the interval from -1 to 1.
- How it is used
- Uses tanh as the hidden-layer activation in a neural network trained for classification with particle swarm optimization.
- How deep it goes
- Defines tanh and demonstrates its NumPy placement in forward propagation, but does not derive its mathematical formula or training gradients.
- Examples
- Demonstrates tanh on the hidden layer of a NumPy classifier using the Penguins dataset.
- What you can do after
- Apply it via NumPy when implementing a neural-network hidden layer and inspect its effect within forward propagation.
Inside Deep Learning: Math, Algorithms, Models
A nonlinear activation function that maps real-valued inputs into the interval [-1,1].
- How it is used
- Uses tanh between linear operations to add nonlinear modeling capacity and in LSTMs to support bounded gating computations.
- How deep it goes
- Defines the output range, explains nonlinear placement and saturation, and demonstrates PyTorch usage without deriving tanh internals.
- Examples
- Demonstrates activation curves, a two-layer PyTorch network, LSTM gates, and an additive-attention replacement exercise.
- What you can do after
- Apply it via PyTorch's nn.Tanh in hidden layers and recognize saturation-related gradient risks.
Read how Inside Deep Learning: Math, Algorithms, Models explains tanh →
Math and Architectures of Deep Learning
A hyperbolic tangent activation function used to introduce nonlinearity in neural networks.
- How it is used
- Introduces nonlinearity in neural networks and supports gradient-based training when inputs are centered around zero.
- How deep it goes
- Defines the output range and explains gradient behavior near zero, but stops short of implementation details or a derivative formula.
- Examples
- Uses derivative comparison graphs and LeNet's tanh layers after convolutional feature maps.
- What you can do after
- Evaluate/understand how centered outputs and stronger near-zero gradients make tanh useful as a neural-network activation.
Read how Math and Architectures of Deep Learning explains tanh →
Related concepts
- activation functionkind of · An activation function maps a neural-network unit’s or layer’s input to an output, often introducing nonlinearity into the network’s computation.
- ReLUcontrasts with · ReLU, or rectified linear unit, is an activation function that maps an input x to max(0, x).
- recurrent neural networkused by · A recurrent neural network is a neural network that processes sequences by updating an internal state across successive inputs, allowing earlier inputs to influence later computations.
- sigmoidcontrasts with · A sigmoid is a smooth, bounded, S-shaped function that maps real-valued inputs between two limiting values, commonly zero and one.
- additive scoreused by · An additive score is a learned attention compatibility function that combines representations to score a query-key pair.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.