Activation function
An activation function transforms a neural-network unit’s input into its output, often introducing nonlinearity.
Grokking Machine Learning
A function that transforms a model's score into an output or prediction.
- How it is used
- Transforms a model’s weighted score into a prediction, probability-like score, or intermediate neural-network value.
- How deep it goes
- Defines threshold and continuous functions, compares their outputs, and connects them to neural-network nonlinearity and gradient problems.
- Examples
- Uses alien mood classification, word-count features, perceptron scores, sigmoid happiness ratings, and neural-network layers.
- What you can do after
- Apply a named activation function to turn weighted model scores into binary predictions, continuous scores, or nonlinear neural-network outputs.
Read how Grokking Machine Learning explains activation function →
Build AI Drug Discovery Pipelines
An activation function is a function applied to a neuron's weighted input to produce its output, typically introducing nonlinearity.
- How it is used
- Uses activation functions to introduce nonlinearity in neural networks and motivate activation choice for stable training.
- How deep it goes
- Explains neuron transformations, decision boundaries, activation curves, and vanishing gradients, then implements ReLU in a neural network.
- Examples
- Compares sigmoid, tanh, ReLU, and step curves and uses ReLU in a kinase binder network.
- What you can do after
- Apply activation functions via PyTorch when constructing a neural network and evaluate their effects on gradient flow.
Read how Build AI Drug Discovery Pipelines explains activation function →
Grokking AI Algorithms, Second Edition
An activation function is a nonlinear or linear mathematical function that transforms a neuron's weighted input before passing the result onward.
- How it is used
- Uses activation functions to transform weighted sums in perceptrons, multilayer neural networks, and Transformer feed-forward layers.
- How deep it goes
- Explains common function shapes, learning trade-offs, forward-pass calculations, and the role of activation slopes in backpropagation.
- Examples
- Illustrates sigmoid forward propagation with car-collision predictions and traces ReLU through a Transformer projection.
- What you can do after
- Builds a sigmoid-based forward pass from scratch using NumPy.
Read how Grokking AI Algorithms, Second Edition explains activation function →
Grokking Machine Learning, Second Edition
An activation function is a function applied to a neural network unit's input to produce its output and introduce a nonlinear transformation.
- How it is used
- Turns perceptron scores into predictions and supplies nonlinearity between neural-network layers.
- How deep it goes
- Defines step, sigmoid, tanh, and ReLU, explains gradient effects, and implements layer choices in Keras.
- Examples
- Uses a step-function classifier, a weighted perceptron, a circular dataset, and Hyderabad housing-price regression.
- What you can do after
- Apply it via Keras by selecting ReLU and softmax layers for classification or omitting the final activation for regression.
Read how Grokking Machine Learning, Second Edition explains activation function →
Natural Language Processing in Action, Second Edition
An activation function is a nonlinear function applied to a neural-network signal to transform its value before further processing.
- How it is used
- Transforms neuron signals and CNN match scores into outputs suited to prediction tasks.
- How deep it goes
- Explains threshold and logistic activations through a weighted-sum example and distinguishes output activations; the neuron implementation covers only the feed-forward path.
- Examples
- Uses a four-word count vector and random weights to demonstrate a thresholded neuron.
- What you can do after
- Evaluate activation choices for binary decisions, continuous likelihoods, mutually exclusive categories, and multi-label outputs.
Read how Natural Language Processing in Action, Second Edition explains activation function →
Classic Computer Science Problems in Swift: Essential techniques for practicing programmers
An activation function is a function applied to a neuron's output to introduce nonlinearity and constrain the output range in a neural network.
- How it is used
- Transforms each neuron's weighted input before its signal passes through a feed-forward network.
- How deep it goes
- Implements sigmoid and its derivative, then uses both in forward passes and backpropagation without deriving the calculus formulas.
- Examples
- Uses Swift sigmoid functions, neuron and layer implementations, and an exercise comparing alternative functions.
- What you can do after
- Build it from scratch in Swift and test how a replacement activation function changes network accuracy and training.
Inside Deep Learning: Math, Algorithms, Models
A nonlinear function applied in a neural network to transform layer outputs and increase representational power.
- How it is used
- Adds nonlinear capacity between neural-network layers so models can learn more complex functions.
- How deep it goes
- Defines sigmoid, tanh, ReLU, and Leaky ReLU with ranges, derivatives, plots, and PyTorch placement, while deferring broader properties.
- Examples
- Uses activation plots, a tanh regression network, deep fully connected and convolutional models, and LeakyReLU and PReLU exercises.
- What you can do after
- Build it from scratch by inserting PyTorch activations, plotting their curves, and comparing ReLU variants in trained models.
Read how Inside Deep Learning: Math, Algorithms, Models explains activation function →
Deep Learning with PyTorch, Second Edition
A nonlinear function applied within a neural network to transform signals so the whole model can represent more complex mappings than a purely linear model.
- How it is used
- Introduces nonlinearity between linear operations and constrains outputs when a task calls for a particular range.
- How deep it goes
- Explains neuron-level composition, common activation shapes, differentiability, and saturation, with an intuitive account of function approximation.
- Examples
- Uses a dog-likeness scoring example and a sequential network with tanh between two linear layers.
- What you can do after
- Apply activation functions via PyTorch nn modules and consider output range and saturation when choosing one.
Read how Deep Learning with PyTorch, Second Edition explains activation function →
Data Without Labels
An activation function is a mathematical function applied to a neuron's combined input to determine its output, typically introducing nonlinearity.
- How it is used
- Determines neuron outputs as weighted information moves through network layers toward a prediction.
- How deep it goes
- Defines the function, explains the case for nonlinearity, and distinguishes common choices without implementing them in Python.
- Examples
- Illustrates the concept with a threshold perceptron, activation-function curves, and face-identification predictions.
- What you can do after
- Evaluate/understand it by relating sigmoid to binary output and TANH or ReLU to hidden-layer processing.
Timeless Algorithms, The Foundational Ideas
An activation function transforms a neural network unit’s combined inputs before the result is passed onward.
- How it is used
- Introduces nonlinearity into each unit’s forward-pass transformation to support learned representations and predictions.
- How deep it goes
- Works through weighted inputs and compares activation outputs, relating the choice to representations and gradients.
- Examples
- Compares sigmoid, tanh, and ReLU on two hidden-node pre-activations derived from an image-based braking scenario.
- What you can do after
- Evaluate how activation choices change internal representations and gradients during learning.
Read how Timeless Algorithms, The Foundational Ideas explains activation function →
Optimization Algorithms: AI techniques for design, planning, and control problems
An activation function is a function applied to a neural-network unit’s pre-activation value to produce its output and introduce nonlinear behavior.
- How it is used
- Uses nonlinear unit outputs within neural-network forward passes and training optimization.
- How deep it goes
- Defines the role, lists major forms, and demonstrates tanh in code without deriving activation-function formulas.
- Examples
- Demonstrates tanh applied to hidden-layer values in a penguin-classification neural network.
- What you can do after
- Apply it via NumPy by inserting an activation into a neural-network forward pass.
Fight Fraud with Machine Learning
An activation function is a function applied to a neural-network unit's weighted input to determine its output and introduce nonlinear behavior.
- How it is used
- Introduces nonlinearity as a core mechanism for deep neural networks used in fraud detection.
- How deep it goes
- Explains weighted inputs, layer collapse, expressive power, bounded outputs, computation, and vanishing gradients.
- Examples
- Compares the perceptron step function with Sigmoid, TanH, and ReLU through an activation-function figure.
- What you can do after
- Evaluate activation choices by relating bounded outputs, computation speed, nonlinearity, and vanishing-gradient behavior to model needs.
Read how Fight Fraud with Machine Learning explains activation function →
Machines that Think
An activation function is a mathematical function that transforms a computational unit's combined input signal, typically nonlinearly, into its response.
- How it is used
- Introduces nonlinearity into neural networks so they can model complex patterns and boundaries.
- How deep it goes
- Explains the neuron pipeline and compares activation functions by response and training limitations.
- Examples
- Compares sigmoid, tanh, ReLU, and GELU as ways to shape a neuron's response.
- What you can do after
- Evaluate/understand how activation functions affect signal flow and network expressiveness.
Deep Learning with R, Third Edition
An activation function transforms a neural-network unit's pre-activation value into its output, often element by element.
- How it is used
- Deploys activations inside Dense layers to create nonlinear representations and probability-producing outputs for classification.
- How deep it goes
- Explains affine composition, relu and sigmoid behavior, and nonlinear necessity, while only naming alternatives such as prelu and elu.
- Examples
- Uses a three-layer Keras classifier, relu and sigmoid output expressions, and geometric figures contrasting affine and activated transformations.
- What you can do after
- Apply it via Keras by placing relu in intermediate layers and sigmoid in a binary-classification output layer.
Read how Deep Learning with R, Third Edition explains activation function →
Sutskever's List
An activation function transforms a neuron's weighted input before passing its output to subsequent layers.
- How it is used
- Shows how ReLU helped AlexNet train deeper networks by preserving gradient flow.
- How deep it goes
- Explains ReLU's input mapping, contrasts it with sigmoid and tanh, and connects gradient flow to convergence without formal derivation.
- Examples
- Compares ReLU, sigmoid, and tanh curves and reports a sixfold convergence gain over tanh in AlexNet.
- What you can do after
- Evaluate how ReLU's input mapping and gradient behavior differ from sigmoid and tanh.
Deep Learning with JAX
A nonlinear function applied inside a neural network to let stacked layers represent more than a single linear transformation.
- How it is used
- Enables multilayer neural networks to express nonlinear relationships during the forward pass.
- How deep it goes
- Defines layer computation, explains linear collapse, and demonstrates Swish in JAX without deriving activation formulas.
- Examples
- Demonstrates Swish inside a JAX prediction function for a ten-class neural network.
- What you can do after
- Apply activation functions through jax.nn.swish in a stateless JAX forward pass.
Read how Deep Learning with JAX explains activation function →
Applied Reinforcement Learning
A function that transforms a neural network unit's pre-activation value, typically introducing nonlinearity.
- How it is used
- Introduces nonlinearity so deep reinforcement learning networks can model complex relationships.
- How deep it goes
- Defines the activation’s role after a weighted sum and bias, and explains why stacking layers requires nonlinearity.
- Examples
- Uses ReLU in hidden layers and linear outputs for Q-values as architectural examples.
- What you can do after
- Understand why neural networks use nonlinear activations and how hidden-layer activations differ from Q-value outputs.
Read how Applied Reinforcement Learning explains activation function →
Graph Neural Networks in Action
An activation function is a nonlinear or otherwise specified function applied to a layer's transformed values.
- How it is used
- Uses nonlinear transformations inside graph message-passing layers to produce expressive node or graph representations.
- How deep it goes
- Defines the transformation and positions it within aggregation, transformation, and update steps, including a message-passing formula.
- Examples
- Uses ReLU, defined as the maximum of zero and the input, after aggregated messages are transformed.
- What you can do after
- Evaluate why nonlinear activation is required for expressive multilayer graph neural networks.
Read how Graph Neural Networks in Action explains activation function →
Machine Learning Algorithms in Depth
An activation function is a nonlinear function applied to a neural-network layer output to increase the model's expressive capacity.
- How it is used
- Introduces nonlinearity between multilayer perceptron layers to increase expressive power for classification and regression.
- How deep it goes
- Defines ReLU and identifies softmax and identity by output task, but defers detailed activation derivatives and comparisons.
- Examples
- Uses ReLU in hidden layers, softmax for classification, identity for regression, and an MLP architecture as concrete examples.
- What you can do after
- Evaluate activation choices for hidden layers and classification or regression output layers.
Read how Machine Learning Algorithms in Depth explains activation function →
Deep Learning with Python, Third Edition
A nonlinear function applied to a layer’s output to reshape it and increase model expressiveness.
- How it is used
- Shapes intermediate representations and constrains final predictions for classification or regression.
- How deep it goes
- Explains relu through tensor operations and output behavior, but defers formal principles for choosing layer architectures.
- Examples
- Uses a sentiment classifier with relu and sigmoid, plus scalar regression with a linear final layer.
- What you can do after
- Apply it via Keras by matching hidden-layer and output activations to the prediction task.
Read how Deep Learning with Python, Third Edition explains activation function →
Also covered in
- Grokking BayesIn early access (MEAP)
- Local Image GenerationIn early access (MEAP)
- Rearchitecting LLMsIn early access (MEAP)
- Demand Forecasting Best Practices
- Evolutionary Deep Learning: Genetic algorithms and neural networks
- LLMs in Production
- Knowledge Graphs and LLMs in Action
What to know first
- linear regressionLinear regression is a statistical method that models a numeric response as a linear function of one or more predictor variables with fitted parameters.
- objective functionAn objective function is a mathematical function that assigns values to candidate solutions or parameter settings, which an optimization procedure seeks to minimize or maximize.
- sigmoidThe logistic sigmoid is a smooth, S-shaped function that maps real-valued inputs to values between zero and one.
- ReLUReLU is an activation function that maps an input x to the greater of x and zero.
Related concepts
- sigmoidincludes · The logistic sigmoid is a smooth, S-shaped function that maps real-valued inputs to values between zero and one.
- ReLUincludes · ReLU is an activation function that maps an input x to the greater of x and zero.
- step functionincludes · A step function is a piecewise-constant function that changes abruptly between discrete output values when its input crosses one or more specified thresholds.
- nonlinear mappingproduces · A function or transformation that does not preserve linear combinations of its inputs.
- GELUincludes · GELU is a smooth neural-network activation function that scales an input by the standard normal cumulative distribution function evaluated at that input.
- Swishincludes · Swish is a smooth activation function defined by f(x) = xσ(βx), where σ is the sigmoid function; SiLU is the special case with β equal to 1.
Questions about activation function
What is activation function?
An activation function transforms a neural-network unit’s input into its output, often introducing nonlinearity.
Which Manning books explain activation function?
Grokking Machine Learning, Build AI Drug Discovery Pipelines, Grokking AI Algorithms, Second Edition, Grokking Machine Learning, Second Edition, Natural Language Processing in Action, Second Edition, Classic Computer Science Problems in Swift: Essential techniques for practicing programmers, Inside Deep Learning: Math, Algorithms, Models, Deep Learning with PyTorch, Second Edition and 12 more.
What should I know before learning activation function?
Manning books that teach activation function usually introduce linear regression, objective function, sigmoid and ReLU first.
What kinds of activation function are there?
Kinds of activation function include sigmoid, ReLU, step function, GELU and Swish.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.



















