Activation function

As explained in Graph Neural Networks in Action

An activation function is a nonlinear or otherwise specified function applied to a layer's transformed values.

An activation function applies a nonlinear transformation to aggregated, transformed messages in a neural network layer. In graph message passing, it helps prevent stacked layers from collapsing into an overall linear model. section 3.4.2

A message-passing layer uses the activation function in this sequence:

• Aggregate node or edge information with an order-independent operator such as sum, average, or maximum. section 3.4.2
• Transform the aggregated messages with a neural network and activation function. section 3.4.2
• Combine the resulting embeddings with node or edge data and previous embeddings; apply pooling only when graph-level output requires it. section 3.4.2

ReLU

ReLU returns the maximum of zero and its input. Applied after message aggregation and transformation, it supplies the nonlinearity needed for a graph convolutional layer to model more than a linear relationship. section 3.4.2

Figure 3.15 Elements of our message-passing layer. Each message-passing layer consists of an aggregation, a transformation, and an update step.

Use the layer diagram to trace aggregation, transformation, and update operations around the activation function. section 3.4.2

How Graph Neural Networks in Action covers it

How deep it goes

Defines the transformation and positions it within aggregation, transformation, and update steps, including a message-passing formula.

Examples

Uses ReLU, defined as the maximum of zero and the input, after aggregated messages are transformed.

What you can do after

Evaluate why nonlinear activation is required for expressive multilayer graph neural networks.

How it connects

Links activation functions to message passing, aggregation operators, neural networks, graph convolutional networks, pooling, and node prediction.

liveBookmanning.com

Read Graph Neural Networks in Action on liveBook Get the book at manning.com

Also explained in

• 

  liveBookmanning.com
  Grokking Machine LearningA function that transforms a model's score into an output or prediction.
• 

  liveBookmanning.com
  Grokking AI Algorithms, Second EditionAn activation function is a nonlinear or linear mathematical function that transforms a neuron's weighted input before passing the result onward.
• 

  liveBookmanning.com
  Natural Language Processing in Action, Second EditionAn activation function is a nonlinear function applied to a neural-network signal to transform its value before further processing.
• 

  liveBookmanning.com
  Classic Computer Science Problems in Swift: Essential techniques for practicing programmersAn activation function is a function applied to a neuron's output to introduce nonlinearity and constrain the output range in a neural network.
• 

  liveBookmanning.com
  Inside Deep Learning: Math, Algorithms, ModelsA nonlinear function applied in a neural network to transform layer outputs and increase representational power.
• 

  liveBookmanning.com
  Deep Learning with PyTorch, Second EditionA nonlinear function applied within a neural network to transform signals so the whole model can represent more complex mappings than a purely linear model.
• 

  liveBookmanning.com
  Data Without LabelsAn activation function is a mathematical function applied to a neuron's combined input to determine its output, typically introducing nonlinearity.
• 

  liveBookmanning.com
  Optimization Algorithms: AI techniques for design, planning, and control problemsAn activation function is a function applied to a neural-network unit’s pre-activation value to produce its output and introduce nonlinear behavior.
• 

  liveBookmanning.com
  Deep Learning with R, Third EditionAn activation function transforms a neural-network unit's pre-activation value into its output, often element by element.
• 

  liveBookmanning.com
  Sutskever's ListAn activation function transforms a neuron's weighted input before passing its output to subsequent layers.
• 

  liveBookmanning.com
  Deep Learning with JAXA nonlinear function applied inside a neural network to let stacked layers represent more than a single linear transformation.
• 

  liveBookmanning.com
  Machine Learning Algorithms in DepthAn activation function is a nonlinear function applied to a neural-network layer output to increase the model's expressive capacity.
• 

  liveBookmanning.com
  Deep Learning with Python, Third EditionA nonlinear function applied to a layer’s output to reshape it and increase model expressiveness.

Compare how 14 books explain activation function →

Related concepts

• sigmoidincludes · A sigmoid is a smooth, bounded, S-shaped function that maps real-valued inputs between two limiting values, commonly zero and one.
• ReLUincludes · ReLU, or rectified linear unit, is an activation function that maps an input x to max(0, x).
• nonlinear mappingproduces · A nonlinear mapping is a function that does not preserve linear combinations of its inputs.
• step functionincludes · A step function is a piecewise-constant function that changes abruptly between discrete output values when its input crosses one or more specified thresholds.
• GELUincludes · GELU, or Gaussian error linear unit, is a smooth neural-network activation function that scales an input by the cumulative distribution function of the standard normal distribution at that input.
• Swishincludes · Swish, also called SiLU, is a smooth neural-network activation function defined as x multiplied by the sigmoid of βx.

This explainer was generated by Manning's knowledge graph from the text of Graph Neural Networks in Action. The section links point to the parts of the book it draws on.