Activation function
An activation function maps a neural-network unit’s or layer’s input to an output, often introducing nonlinearity into the network’s computation.
Grokking Machine Learning
A function that transforms a model's score into an output or prediction.
- How it is used
- Transforms a model’s weighted score into a prediction, probability-like score, or intermediate neural-network value.
- How deep it goes
- Defines threshold and continuous functions, compares their outputs, and connects them to neural-network nonlinearity and gradient problems.
- Examples
- Uses alien mood classification, word-count features, perceptron scores, sigmoid happiness ratings, and neural-network layers.
- What you can do after
- Apply a named activation function to turn weighted model scores into binary predictions, continuous scores, or nonlinear neural-network outputs.
Read how Grokking Machine Learning explains activation function →
Grokking AI Algorithms, Second Edition
An activation function is a nonlinear or linear mathematical function that transforms a neuron's weighted input before passing the result onward.
- How it is used
- Transforms weighted sums during forward propagation to produce neuron outputs and enable nonlinear patterns.
- How deep it goes
- Defines the operation, compares four functions, and traces its use across layers, including learning trade-offs but not a theoretical proof.
- Examples
- Uses a loan decision, a car-collision network, NumPy sigmoid code, and a worked ReLU token projection.
- What you can do after
- Apply it via NumPy to hidden and output layers, and evaluate basic function choices for binary decisions and nonlinear learning.
Read how Grokking AI Algorithms, Second Edition explains activation function →
Natural Language Processing in Action, Second Edition
An activation function is a nonlinear function applied to a neural-network signal to transform its value before further processing.
- How it is used
- Transforms neuron scores into decisions or likelihoods for word-based classifiers, convolutional text matching, and label prediction.
- How deep it goes
- Works through a weighted-sum example and threshold, then contrasts logistic, softmax, and sigmoid outputs without deriving those functions.
- Examples
- Uses a bag-of-words Python neuron, a sliding text kernel, and a classifier-to-tagger change.
- What you can do after
- Build it from scratch: threshold a weighted neuron score; choose logistic or sigmoid outputs when the prediction task calls for them.
Read how Natural Language Processing in Action, Second Edition explains activation function →
Classic Computer Science Problems in Swift: Essential techniques for practicing programmers
An activation function is a function applied to a neuron's output to introduce nonlinearity and constrain the output range in a neural network.
- How it is used
- Transforms each neuron's weighted input before its signal passes through a feed-forward network.
- How deep it goes
- Implements sigmoid and its derivative, then uses both in forward passes and backpropagation without deriving the calculus formulas.
- Examples
- Uses Swift sigmoid functions, neuron and layer implementations, and an exercise comparing alternative functions.
- What you can do after
- Build it from scratch in Swift and test how a replacement activation function changes network accuracy and training.
Inside Deep Learning: Math, Algorithms, Models
A nonlinear function applied in a neural network to transform layer outputs and increase representational power.
- How it is used
- Adds nonlinear capacity between neural-network layers so models can learn more complex functions.
- How deep it goes
- Defines sigmoid, tanh, ReLU, and Leaky ReLU with ranges, derivatives, plots, and PyTorch placement, while deferring broader properties.
- Examples
- Uses activation plots, a tanh regression network, deep fully connected and convolutional models, and LeakyReLU and PReLU exercises.
- What you can do after
- Build it from scratch by inserting PyTorch activations, plotting their curves, and comparing ReLU variants in trained models.
Read how Inside Deep Learning: Math, Algorithms, Models explains activation function →
Deep Learning with PyTorch, Second Edition
A nonlinear function applied within a neural network to transform signals so the whole model can represent more complex mappings than a purely linear model.
- How it is used
- Adds nonlinearity between neural-network layers or shapes final outputs for a task.
- How deep it goes
- Defines the neuron operation, compares common functions, and explains gradient saturation without prescribing a strict selection rule.
- Examples
- Uses a dog-likeness score, a temperature-conversion network, and a bird-versus-airplane classifier.
- What you can do after
- Apply it via PyTorch nn.Sequential by placing nn.Tanh between nn.Linear layers or after nn.Conv2d layers.
Read how Deep Learning with PyTorch, Second Edition explains activation function →
Data Without Labels
An activation function is a mathematical function applied to a neuron's combined input to determine its output, typically introducing nonlinearity.
- How it is used
- Determines neuron outputs as weighted information moves through network layers toward a prediction.
- How deep it goes
- Defines the function, explains the case for nonlinearity, and distinguishes common choices without implementing them in Python.
- Examples
- Illustrates the concept with a threshold perceptron, activation-function curves, and face-identification predictions.
- What you can do after
- Evaluate/understand it by relating sigmoid to binary output and TANH or ReLU to hidden-layer processing.
Optimization Algorithms: AI techniques for design, planning, and control problems
An activation function is a function applied to a neural-network unit’s pre-activation value to produce its output and introduce nonlinear behavior.
- How it is used
- Uses nonlinear unit outputs within neural-network forward passes and training optimization.
- How deep it goes
- Defines the role, lists major forms, and demonstrates tanh in code without deriving activation-function formulas.
- Examples
- Demonstrates tanh applied to hidden-layer values in a penguin-classification neural network.
- What you can do after
- Apply it via NumPy by inserting an activation into a neural-network forward pass.
Deep Learning with R, Third Edition
An activation function transforms a neural-network unit's pre-activation value into its output, often element by element.
- How it is used
- Deploys activations inside Dense layers to create nonlinear representations and probability-producing outputs for classification.
- How deep it goes
- Explains affine composition, relu and sigmoid behavior, and nonlinear necessity, while only naming alternatives such as prelu and elu.
- Examples
- Uses a three-layer Keras classifier, relu and sigmoid output expressions, and geometric figures contrasting affine and activated transformations.
- What you can do after
- Apply it via Keras by placing relu in intermediate layers and sigmoid in a binary-classification output layer.
Read how Deep Learning with R, Third Edition explains activation function →
Sutskever's List
An activation function transforms a neuron's weighted input before passing its output to subsequent layers.
- How it is used
- Shows how ReLU helped AlexNet train deeper networks by preserving gradient flow.
- How deep it goes
- Explains ReLU's input mapping, contrasts it with sigmoid and tanh, and connects gradient flow to convergence without formal derivation.
- Examples
- Compares ReLU, sigmoid, and tanh curves and reports a sixfold convergence gain over tanh in AlexNet.
- What you can do after
- Evaluate how ReLU's input mapping and gradient behavior differ from sigmoid and tanh.
Deep Learning with JAX
A nonlinear function applied inside a neural network to let stacked layers represent more than a single linear transformation.
- How it is used
- Enables multilayer neural networks to express nonlinear relationships during the forward pass.
- How deep it goes
- Defines layer computation, explains linear collapse, and demonstrates Swish in JAX without deriving activation formulas.
- Examples
- Demonstrates Swish inside a JAX prediction function for a ten-class neural network.
- What you can do after
- Apply activation functions through jax.nn.swish in a stateless JAX forward pass.
Read how Deep Learning with JAX explains activation function →
Graph Neural Networks in Action
An activation function is a nonlinear or otherwise specified function applied to a layer's transformed values.
- How it is used
- Uses nonlinear transformations inside graph message-passing layers to produce expressive node or graph representations.
- How deep it goes
- Defines the transformation and positions it within aggregation, transformation, and update steps, including a message-passing formula.
- Examples
- Uses ReLU, defined as the maximum of zero and the input, after aggregated messages are transformed.
- What you can do after
- Evaluate why nonlinear activation is required for expressive multilayer graph neural networks.
Read how Graph Neural Networks in Action explains activation function →
Machine Learning Algorithms in Depth
An activation function is a nonlinear function applied to a neural-network layer output to increase the model's expressive capacity.
- How it is used
- Introduces nonlinearity between multilayer perceptron layers to increase expressive power for classification and regression.
- How deep it goes
- Defines ReLU and identifies softmax and identity by output task, but defers detailed activation derivatives and comparisons.
- Examples
- Uses ReLU in hidden layers, softmax for classification, identity for regression, and an MLP architecture as concrete examples.
- What you can do after
- Evaluate activation choices for hidden layers and classification or regression output layers.
Read how Machine Learning Algorithms in Depth explains activation function →
Deep Learning with Python, Third Edition
A nonlinear function applied to a layer’s output to reshape it and increase model expressiveness.
- How it is used
- Shapes intermediate representations and constrains final predictions for classification or regression.
- How deep it goes
- Explains relu through tensor operations and output behavior, but defers formal principles for choosing layer architectures.
- Examples
- Uses a sentiment classifier with relu and sigmoid, plus scalar regression with a linear final layer.
- What you can do after
- Apply it via Keras by matching hidden-layer and output activations to the prediction task.
Read how Deep Learning with Python, Third Edition explains activation function →
Related concepts
- sigmoidincludes · A sigmoid is a smooth, bounded, S-shaped function that maps real-valued inputs between two limiting values, commonly zero and one.
- ReLUincludes · ReLU, or rectified linear unit, is an activation function that maps an input x to max(0, x).
- nonlinear mappingproduces · A nonlinear mapping is a function that does not preserve linear combinations of its inputs.
- step functionincludes · A step function is a piecewise-constant function that changes abruptly between discrete output values when its input crosses one or more specified thresholds.
- GELUincludes · GELU, or Gaussian error linear unit, is a smooth neural-network activation function that scales an input by the cumulative distribution function of the standard normal distribution at that input.
- Swishincludes · Swish, also called SiLU, is a smooth neural-network activation function defined as x multiplied by the sigmoid of βx.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.