A Bayesian neural network combines neural networks’ ability to model complex relationships with Bayesian uncertainty. Instead of fixing each weight and bias at one value, it represents them with probability distributions. Updating those distributions with data produces predictive distributions rather than single predictions. section 10.0
Imagine predicting a noisy measurement. Where observations are plentiful, predictions can be relatively certain; where data are sparse, the network should admit that several parameter settings remain plausible. This captures aleatoric uncertainty from data noise and epistemic uncertainty from limited knowledge about parameters. section 10.3 section 10.9
A Bayesian neural network typically follows this workflow:
- Place prior distributions on the network’s weights and biases. section 10.0 section 10.3
- Define the neural network’s computation and an observation model, such as a normal likelihood for noisy regression data. section 10.3
- Infer posterior distributions with an approximate method, commonly MCMC or variational inference. section 10.0 section 10.9
- Generate predictions by averaging uncertainty across plausible parameter values, producing distributions and credible intervals. section 10.3
Noisy regression with MCMC
A small network can use two hidden layers with 20 nodes each, standard normal priors, and a normal observation model. MCMC draws posterior samples for its parameters. Predictions interpolate the observed data, while 95% credible intervals widen outside the observed range, signaling greater uncertainty from missing data. MCMC can provide high-quality uncertainty estimates, but neural networks make sampling computationally expensive and require convergence checks. section 10.3 section 10.9
with pm.Model() as bnn:
w1 = pm.Normal("w1", 0, 1, shape=(1, 20)) #A
b1 = pm.Normal("b1", 0, 1, shape=(20,)) #B
w2 = pm.Normal("w2", 0, 1, shape=(20,)) #C
b2 = pm.Normal("b2", 0, 1) #D
Inspect how PyMC declares normally distributed priors for the network’s weights and biases before defining its likelihood and sampling posterior values. section 10.3

