Swish

As explained in Grokking AI Algorithms, Second Edition

SiLU is a smooth neural-network activation function defined by multiplying an input by its sigmoid.

Swish, also called SiLU, is a smooth neural-network activation function. It multiplies an input by that input’s sigmoid: x × sigmoid(x). Think of it as a flexible volume knob: positive signals pass through almost unchanged, while negative signals are softened instead of abruptly erased. section 12.17

Swish is useful because it:

• Acts nearly linearly for positive inputs, similarly to ReLU. section 12.17
• Keeps a smooth, slightly negative response rather than forcing every negative input to zero. section 12.17
• Can support better gradient flow and help prevent the “dying ReLU” problem. section 12.17

A small implementation

Define sigmoid(x) as 1 / (1 + exp(-x)), then return x * sigmoid(x). In a feed-forward network, apply this operation to each hidden-layer value. The same pattern appears in a two-layer network that transforms a text embedding into a specialized guidance vector for image generation. section 12.11

Figure 12.32 ReLU vs SiLU activation functions

Compare ReLU’s hard cutoff with SiLU’s smooth curve, then inspect the corresponding per-value calculations. section 12.17

How Grokking AI Algorithms, Second Edition covers it

How deep it goes

Defines the formula, compares behavior with ReLU, and demonstrates a Python implementation without developing activation-function theory.

Examples

Demonstrates SiLU in an eight-channel U-Net tensor and in a two-layer text-embedding projection.

What it assumes

Assumes basic neural-network terminology while explaining sigmoid, ReLU, and activation behavior directly.

What you can do after

Apply it via Python by computing sigmoid(x) and multiplying the result by x.

How it connects

Connects activation functions, ReLU, sigmoid, convolutional layers, feed-forward networks, text embeddings, and denoising.

liveBookmanning.com

Read Grokking AI Algorithms, Second Edition on liveBook Get the book at manning.com

Also explained in

• 

  liveBookmanning.com
  Deep Learning with JAXA smooth activation function discovered by automatic search and used here as the example nonlinearity in the network.

Compare how 2 books explain Swish →

Related concepts

• activation functionkind of · An activation function maps a neural-network unit’s or layer’s input to an output, often introducing nonlinearity into the network’s computation.
• ReLUalternative to · ReLU, or rectified linear unit, is an activation function that maps an input x to max(0, x).
• sigmoiduses · A sigmoid is a smooth, bounded, S-shaped function that maps real-valued inputs between two limiting values, commonly zero and one.
• activationkind of · An activation is a numerical signal or intermediate representation produced by a neural-network unit or layer during computation.

This explainer was generated by Manning's knowledge graph from the text of Grokking AI Algorithms, Second Edition. The section links point to the parts of the book it draws on.