A log probability is the logarithm of a probability assigned by a model. When the probability is assigned to observed data under model parameters θ, it is a log likelihood: log p(data | θ). Maximizing log likelihood is equivalent to minimizing its negative, called negative log likelihood (NLL). The same quantity can therefore serve as a training objective or as a score for how well an observation fits a trained model. section 5.4 section 11.1.1
In binary logistic regression, model parameters determine class probabilities. Training forms a log-likelihood objective for the labeled observations, changes its sign to obtain a loss, and differentiates that loss to update the parameters with stochastic gradient descent. Regularization can be added to control parameter size. This is an optimization use of log probability: seek parameters that assign greater likelihood to the observed labels. section 5.4
Log likelihood also supports a count-based derivation. For Bernoulli naive Bayes, word presence or absence supplies the observations. Maximizing a log-likelihood objective subject to class probabilities summing to one yields class-probability estimates from class counts; differentiating for the word parameters likewise yields ratios of counts. section 5.5
Scoring a time-series anomaly
Train a variational autoencoder on nonanomalous sensor signals. For each sensor, the model learns Gaussian likelihood parameters and reports log p(xᵢ | μᵢ, σᵢ) for an input xᵢ. Its objective maximizes log likelihood averaged over sensors while also shaping the latent space through KL divergence. At detection time, a drop in an input’s log likelihood can be thresholded to flag an anomaly. section 11.1.1
Figure 11.4 LSTM-VAE anomaly detection result
The anomaly-detection result illustrates the drop in log likelihood used to set a detection threshold. section 11.1.1
