Swish
Swish is a smooth activation function defined by f(x) = xσ(βx), where σ is the sigmoid function; SiLU is the special case with β equal to 1.
Grokking AI Algorithms, Second Edition
SiLU is a smooth neural-network activation function defined by multiplying an input by its sigmoid.
- How it is used
- Uses SiLU inside generative image models, including convolutional layers and feed-forward networks for text embeddings.
- How deep it goes
- Defines the formula, compares behavior with ReLU, and demonstrates a Python implementation without developing activation-function theory.
- Examples
- Demonstrates SiLU in an eight-channel U-Net tensor and in a two-layer text-embedding projection.
- What you can do after
- Apply it via Python by computing sigmoid(x) and multiplying the result by x.
Read how Grokking AI Algorithms, Second Edition explains Swish →
GPU Programming with Triton
Swish is a smooth activation function defined as x multiplied by sigmoid(βx), with β controlling its shape.
- How it is used
- Applies Swish as the activation in a fused, element-wise SwiGLU computation.
- How deep it goes
- Defines Swish algebraically and distinguishes β = 1 from other values, without analyzing numerical behavior.
- Examples
- Uses gate and value projections in a fused SwiGLU kernel.
- What you can do after
- Understand when Swish and SiLU are interchangeable by checking whether β equals 1.
Rearchitecting LLMs
SiLU is a smooth activation function defined by multiplying an input by its sigmoid.
- How it is used
- Regulates gate signals in GLU MLPs before they combine with the up-projection.
- How deep it goes
- Defines the input-times-sigmoid activation and explains its role in GLU gating, without detailing its implementation.
- Examples
- Uses Llama-3.2-1B's 2048-to-8192 GLU expansion to show SiLU-gated outputs multiplied before down_proj.
- What you can do after
- Understand how SiLU gates one GLU projection before element-wise multiplication with its paired projection.
Deep Learning with JAX
A smooth activation function discovered by automatic search and used here as the example nonlinearity in the network.
- How it is used
- Uses Swish as the hidden-layer nonlinearity in a JAX neural-network forward pass.
- How deep it goes
- Explains why nonlinearities matter and demonstrates the JAX call, but does not derive Swish mathematically.
- Examples
- Demonstrates Swish inside a multilayer prediction function applied to a flattened 28×28×1 image.
- What you can do after
- Apply it via JAX's jax.nn.swish function in hidden layers of a parameterized neural network.
What to know first
- ReLUReLU is an activation function that maps an input x to the greater of x and zero.
- tensorA tensor is a multidimensional numerical array whose values are arranged along indexed dimensions and can be manipulated through mathematical computation.
- sigmoidThe logistic sigmoid is a smooth, S-shaped function that maps real-valued inputs to values between zero and one.
- parameterA parameter is a value or named input that specifies or influences the behavior, computation, or output of a function, model, or system.
Related concepts
- activation functionkind of · An activation function transforms a neural-network unit’s input into its output, often introducing nonlinearity.
- activationkind of · An activation is a numerical value or tensor produced by a neural-network unit, layer, or operation during a forward computation.
- gated linear unitpart of · A gated linear unit is a neural-network structure that combines a linear transformation with a gate derived from another transformation through element-wise interaction.
- ReLUalternative to · ReLU is an activation function that maps an input x to the greater of x and zero.
- sigmoiduses · The logistic sigmoid is a smooth, S-shaped function that maps real-valued inputs to values between zero and one.
Questions about Swish
What is Swish?
Swish is a smooth activation function defined by f(x) = xσ(βx), where σ is the sigmoid function; SiLU is the special case with β equal to 1.
Which Manning books explain Swish?
Grokking AI Algorithms, Second Edition, GPU Programming with Triton, Rearchitecting LLMs and Deep Learning with JAX.
What should I know before learning Swish?
Manning books that teach Swish usually introduce ReLU, tensor, sigmoid and parameter first.
What is the difference between Swish and ReLU?
Swish is a smooth activation function defined by f(x) = xσ(βx), where σ is the sigmoid function; SiLU is the special case with β equal to 1. ReLU is an activation function that maps an input x to the greater of x and zero.
What is Swish a kind of?
Swish is a kind of activation function. An activation function transforms a neural-network unit’s input into its output, often introducing nonlinearity.
What is Swish part of?
Swish is part of gated linear unit. A gated linear unit is a neural-network structure that combines a linear transformation with a gate derived from another transformation through element-wise interaction.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.



