Negative log-likelihood

As explained in Machine Learning Algorithms in Depth

A negative log likelihood is a loss equal to the negative logarithm of the likelihood assigned to observed outcomes by a probabilistic model.

Negative log-likelihood (NLL) measures how poorly a probabilistic model assigns probability to observed outcomes: it is the negative logarithm of their likelihood. Maximizing log likelihood is therefore equivalent to minimizing NLL. section 5.4 section 11.2.1

For binary logistic regression, NLL is the loss optimized from conditional class probabilities. Stochastic gradient descent updates parameters using its gradient; regularization can be added to control parameter size. section 5.4

The same objective extends from classification to learned probability densities:

• A mixture density network predicts Gaussian-mixture parameters, including means, covariances, and mixture proportions, and minimizes the resulting NLL. section 11.2.1
• The neural network enforces positive variances with exponential activations and mixture weights summing to one with softmax activations. section 11.2.1
• Under an isotropic Gaussian covariance assumption, the multivariate density can be rewritten as a product across dimensions before constructing the loss. section 11.2.1

Two implementations

A from-scratch SGD logistic-regression implementation computes a regularized objective and gradient, schedules the learning rate, and exposes fit and predict operations. A Keras/TensorFlow Gaussian mixture density network generates synthetic multivariate data, predicts constrained mixture parameters, computes NLL, and reports training and validation loss. section 5.4 section 11.2.1

Figure 11.7 A multioutput neural network used to learn Gaussian parameters

Shows the multi-output neural-network structure used to learn Gaussian mixture parameters for NLL training. section 11.2.1

How Machine Learning Algorithms in Depth covers it

How deep it goes

Derives gradients and implements SGD for logistic regression, then constructs constrained Gaussian-mixture loss, stopping short of broader likelihood theory.

Examples

Uses binary logistic regression with synthetic two-class data and a Gaussian mixture density network trained on synthetic multivariate data.

What it assumes

Assumes conditional probability models and parameterized distributions while deriving the logistic-regression gradient and Gaussian-mixture formulation.

What you can do after

Builds NLL objectives from scratch, optimizes logistic regression with SGD, and applies NLL in a Keras/TensorFlow mixture density network.

How it connects

Connects negative log-likelihood to logistic regression, softmax, stochastic gradient descent, regularization, mixture density networks, and Gaussian mixture models.

liveBookmanning.com

Read Machine Learning Algorithms in Depth on liveBook Get the book at manning.com

Also explained in

• 

  liveBookmanning.com
  Optimization Algorithms: AI techniques for design, planning, and control problemsNegative log likelihood is a loss equal to the negative logarithm of the probability assigned to the true label.
• 

  liveBookmanning.com
  Reinforcement Learning from Human FeedbackNegative log-likelihood is the negation of a model's log-likelihood, used as a loss for probabilistic predictions.
• 

  liveBookmanning.com
  Deep Learning with PyTorch, Second EditionA loss function that penalizes low predicted probability assigned to the correct class by taking the negative logarithm of that probability and summing over samples.
• 

  liveBookmanning.com
  Natural Language Processing in Action, Second EditionNegative log likelihood is the negative of the logarithm of a model-assigned likelihood, commonly minimized as a probabilistic prediction loss.
• 

  liveBookmanning.com
  Causal AIThe negative log-likelihood loss function is the negative logarithm of the likelihood and is minimized to estimate parameters.
• 

  liveBookmanning.com
  Ensemble Methods for Machine LearningNegative log-likelihood is the negative logarithm of the likelihood assigned by a probabilistic model to observed data.

Compare how 7 books explain negative log-likelihood →

Related concepts

• cross entropy lossequivalent to · Cross-entropy loss measures the expected negative log probability that a predicted distribution assigns to outcomes drawn from a target distribution.
• mean squared errorequivalent to · Mean squared error is the arithmetic mean of the squared differences between corresponding predicted and target numerical values.
• DPO lossused by · The DPO loss is a preference-training objective that optimizes a policy to assign higher relative likelihood to preferred responses than to rejected responses compared with a reference policy.
• NLLLossused by · NLLLoss is a PyTorch loss function that computes negative log-likelihood loss from log-probability outputs and target classes.
• mixture density networkused by · A mixture density network is a neural network that predicts the parameters of a conditional mixture probability distribution, optionally at each sequence step.
• variational lossy autoencodermeasures · A variational lossy autoencoder is an autoencoder variant that intentionally constrains its latent representation so reconstruction preserves selected information rather than every input detail.

This explainer was generated by Manning's knowledge graph from the text of Machine Learning Algorithms in Depth. The section links point to the parts of the book it draws on.