Swish
Swish, also called SiLU, is a smooth neural-network activation function defined as x multiplied by the sigmoid of βx.
Grokking AI Algorithms, Second Edition
SiLU is a smooth neural-network activation function defined by multiplying an input by its sigmoid.
- How it is used
- Uses SiLU inside generative image models, including convolutional layers and feed-forward networks for text embeddings.
- How deep it goes
- Defines the formula, compares behavior with ReLU, and demonstrates a Python implementation without developing activation-function theory.
- Examples
- Demonstrates SiLU in an eight-channel U-Net tensor and in a two-layer text-embedding projection.
- What you can do after
- Apply it via Python by computing sigmoid(x) and multiplying the result by x.
Read how Grokking AI Algorithms, Second Edition explains Swish →
Deep Learning with JAX
A smooth activation function discovered by automatic search and used here as the example nonlinearity in the network.
- How it is used
- Uses Swish as the hidden-layer nonlinearity in a JAX neural-network forward pass.
- How deep it goes
- Explains why nonlinearities matter and demonstrates the JAX call, but does not derive Swish mathematically.
- Examples
- Demonstrates Swish inside a multilayer prediction function applied to a flattened 28×28×1 image.
- What you can do after
- Apply it via JAX's jax.nn.swish function in hidden layers of a parameterized neural network.
Related concepts
- activation functionkind of · An activation function maps a neural-network unit’s or layer’s input to an output, often introducing nonlinearity into the network’s computation.
- ReLUalternative to · ReLU, or rectified linear unit, is an activation function that maps an input x to max(0, x).
- sigmoiduses · A sigmoid is a smooth, bounded, S-shaped function that maps real-valued inputs between two limiting values, commonly zero and one.
- activationkind of · An activation is a numerical signal or intermediate representation produced by a neural-network unit or layer during computation.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.