liveBook · Manning

Exponential linear unit

As explained in Graph Neural Networks in Action

A nonlinear activation function that outputs a smooth negative-valued response for negative inputs and a linear response for positive inputs.

An exponential linear unit (ELU) is a nonlinear activation that produces negative-valued, smoothly changing responses for negative inputs and a linear response for positive inputs. Its smoother negative-side gradient and ability to preserve negative outputs can help smooth outputs across layers and prevent vanishing gradients. section 6.4.1

Use ELU inside multilayer perceptrons when a model needs nonlinear representations without restricting every activation output to nonnegative values. A spatiotemporal graph encoder places ELU activations throughout its message-processing MLPs and output network, alongside batch normalization. section 6.4.1

A practical encoder pipeline applies the activation within these stages:

  • Apply a final fully connected network with ELU activations before producing embeddings or predictions. section 6.4.1

How Graph Neural Networks in Action covers it

How deep it goes
Defines ELU behavior and motivation, then places it in encoder MLPs without deriving its formula or gradients.
Examples
Uses a 31-sensor, 50-timestep graph encoder with repeated MLP processing and a final embedding or prediction layer.
What it assumes
Assumes familiarity with ReLU, MLPs, batch normalization, and message passing in graph models.
What you can do after
Apply ELU within PyTorch-style MLP encoders that produce learned embeddings or predictions.
How it connects
Connects ELU to ReLU, multilayer perceptrons, batch normalization, message passing, and spatiotemporal graph neural networks.
Cover of Graph Neural Networks in Action
liveBookmanning.com
Read Graph Neural Networks in Action on liveBook Get the book at manning.com

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Graph Neural Networks in Action. The section links point to the parts of the book it draws on.