How deeply does Deep Learning with JAX cover Swish?

Explains why nonlinearities matter and demonstrates the JAX call, but does not derive Swish mathematically.

Swish

A smooth activation function discovered by automatic search and used here as the example nonlinearity in the network.

Swish is a smooth activation function used to add nonlinearity to neural-network computations. Without an activation function, stacked linear layers collapse into a single linear model, limiting classification to linearly separable problems. In JAX, import it from jax.nn and apply it to each hidden layer's outputs. section 2.4.2

Using Swish in a forward pass

A prediction function computes each hidden layer with jnp.dot(w, activations) + b, then applies swish(outputs). The final layer produces logits without another shown activation. Calling predict(params, random_flattened_image) returns a ten-element output for the example image.

How Deep Learning with JAX covers it

liveBook manning.com

Read Deep Learning with JAX on liveBook Get the book at manning.com

Also explained in

liveBook manning.com

Grokking AI Algorithms, Second EditionSiLU is a smooth neural-network activation function defined by multiplying an input by its sigmoid.

Compare how 2 books explain Swish →

Related concepts

activation functionkind of · An activation function maps a neural-network unit’s or layer’s input to an output, often introducing nonlinearity into the network’s computation.
ReLUalternative to · ReLU, or rectified linear unit, is an activation function that maps an input x to max(0, x).
sigmoiduses · A sigmoid is a smooth, bounded, S-shaped function that maps real-valued inputs between two limiting values, commonly zero and one.
activationkind of · An activation is a numerical signal or intermediate representation produced by a neural-network unit or layer during computation.

This explainer was generated by Manning's knowledge graph from the text of Deep Learning with JAX. The section links point to the parts of the book it draws on.