liveBook · Manning

Bayesian neural network

As explained in Timeless Algorithms, The Foundational Ideas

A Bayesian neural network is a neural network whose weights and biases are represented by probability distributions, allowing predictions to account for parameter uncertainty.

A standard neural network uses fixed weights and biases: its predicted class probability can express class ambiguity, but not uncertainty about those parameter values. A Bayesian neural network instead represents weights and biases with probability distributions. A prior describes plausible parameter values before training; the data update those distributions through Bayes’ theorem. section 11.4.3 section 11.4.3

P(θ∣D)=P(D∣θ)P(θ)P(D)

The posterior over parameters combines the prior P(θ) with the likelihood P(D | θ), normalized by P(D). section 11.4.3

A pedestrian prediction

A fixed-parameter network’s 0.68 probability that a pedestrian is present describes its prediction under those parameter values; it does not say how certain the network is that those values are correct. section 11.4.3

How Timeless Algorithms, The Foundational Ideas covers it

How deep it goes
Defines parameter distributions and Bayesian updating, but stops short of detailing inference algorithms.
Examples
Uses a pedestrian-presence probability of 0.68 to distinguish class ambiguity from uncertainty about parameters.
What it assumes
Builds from fixed-parameter neural-network predictions and introduces parameter uncertainty through prior distributions.
What you can do after
Understand why a fixed-parameter prediction does not express uncertainty about parameter values and how data update a prior.
How it connects
Connects neural-network learning to Bayesian inference and links inference, information, regularization, and model evidence through MacKay.

Also explained in

Compare how 2 books explain bayesian neural network →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Timeless Algorithms, The Foundational Ideas. The section links point to the parts of the book it draws on.