liveBook · Manning

Swish

Also called SiLU

Swish is a smooth activation function defined by f(x) = xσ(βx), where σ is the sigmoid function; SiLU is the special case with β equal to 1.

How 4 Manning books explain it

Grokking AI Algorithms, Second Edition

SiLU is a smooth neural-network activation function defined by multiplying an input by its sigmoid.

How it is used
Uses SiLU inside generative image models, including convolutional layers and feed-forward networks for text embeddings.
How deep it goes
Defines the formula, compares behavior with ReLU, and demonstrates a Python implementation without developing activation-function theory.
Examples
Demonstrates SiLU in an eight-channel U-Net tensor and in a two-layer text-embedding projection.
What you can do after
Apply it via Python by computing sigmoid(x) and multiplying the result by x.

Read how Grokking AI Algorithms, Second Edition explains Swish →

GPU Programming with Triton

Swish is a smooth activation function defined as x multiplied by sigmoid(βx), with β controlling its shape.

How it is used
Applies Swish as the activation in a fused, element-wise SwiGLU computation.
How deep it goes
Defines Swish algebraically and distinguishes β = 1 from other values, without analyzing numerical behavior.
Examples
Uses gate and value projections in a fused SwiGLU kernel.
What you can do after
Understand when Swish and SiLU are interchangeable by checking whether β equals 1.

Read how GPU Programming with Triton explains Swish →

Rearchitecting LLMs

SiLU is a smooth activation function defined by multiplying an input by its sigmoid.

How it is used
Regulates gate signals in GLU MLPs before they combine with the up-projection.
How deep it goes
Defines the input-times-sigmoid activation and explains its role in GLU gating, without detailing its implementation.
Examples
Uses Llama-3.2-1B's 2048-to-8192 GLU expansion to show SiLU-gated outputs multiplied before down_proj.
What you can do after
Understand how SiLU gates one GLU projection before element-wise multiplication with its paired projection.

Read how Rearchitecting LLMs explains Swish →

Deep Learning with JAX

A smooth activation function discovered by automatic search and used here as the example nonlinearity in the network.

How it is used
Uses Swish as the hidden-layer nonlinearity in a JAX neural-network forward pass.
How deep it goes
Explains why nonlinearities matter and demonstrates the JAX call, but does not derive Swish mathematically.
Examples
Demonstrates Swish inside a multilayer prediction function applied to a flattened 28×28×1 image.
What you can do after
Apply it via JAX's jax.nn.swish function in hidden layers of a parameterized neural network.

Read how Deep Learning with JAX explains Swish →

What to know first

Related concepts

Questions about Swish

What is Swish?

Swish is a smooth activation function defined by f(x) = xσ(βx), where σ is the sigmoid function; SiLU is the special case with β equal to 1.

Which Manning books explain Swish?

Grokking AI Algorithms, Second Edition, GPU Programming with Triton, Rearchitecting LLMs and Deep Learning with JAX.

What should I know before learning Swish?

Manning books that teach Swish usually introduce ReLU, tensor, sigmoid and parameter first.

What is the difference between Swish and ReLU?

Swish is a smooth activation function defined by f(x) = xσ(βx), where σ is the sigmoid function; SiLU is the special case with β equal to 1. ReLU is an activation function that maps an input x to the greater of x and zero.

What is Swish a kind of?

Swish is a kind of activation function. An activation function transforms a neural-network unit’s input into its output, often introducing nonlinearity.

What is Swish part of?

Swish is part of gated linear unit. A gated linear unit is a neural-network structure that combines a linear transformation with a gate derived from another transformation through element-wise interaction.

These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.