Negative log-likelihood

As explained in Ensemble Methods for Machine Learning

Negative log-likelihood is the negative logarithm of the likelihood assigned by a probabilistic model to observed data.

Negative log-likelihood turns a probabilistic model’s assigned likelihood into an optimization target. It connects the chosen residual distribution to the loss and therefore to the model’s behavior. section 7.3.1

Choose the distribution that matches the target and residual behavior:

• Gaussian residuals produce squared error, which heavily penalizes large errors and remains suitable for gradient and Newton boosting. section 7.3.1
• Laplacian residuals produce absolute error, reducing the influence of extreme errors but lacking the second derivative required for Newton boosting. section 7.3.1
• Huber loss combines squared-error behavior for small residuals with scaled absolute-error behavior beyond a chosen threshold. section 7.3.1

Comparing residual assumptions

For a continuous target, selecting Gaussian residuals emphasizes large mistakes through squared error; selecting Laplacian residuals treats extreme mistakes less aggressively through absolute error. section 7.3.1

Figure 7.10 Loss and log-likelihood functions for three different types of targets: continuous-valued (left), positive continuous-valued (center), and count-valued (right)

Compare loss and log-likelihood shapes for continuous, positive continuous, and count-valued targets. section 7.3.1

How Ensemble Methods for Machine Learning covers it

How deep it goes

Defines the optimization role and compares distribution-linked losses, stopping short of detailed derivations.

Examples

Uses Gaussian, Laplacian, and Huber losses for continuous-valued targets, alongside comparisons across target types.

What it assumes

Assumes regression, residuals, probability distributions, and derivatives while reinforcing how these determine loss functions.

What you can do after

Evaluate how distribution assumptions change loss penalties, differentiability, outlier sensitivity, and boosting suitability.

How it connects

Links negative log-likelihood to residual distributions, squared error, absolute error, Huber loss, gradient boosting, and Newton boosting.

liveBookmanning.com

Read Ensemble Methods for Machine Learning on liveBook Get the book at manning.com

Also explained in

• 

  liveBookmanning.com
  Optimization Algorithms: AI techniques for design, planning, and control problemsNegative log likelihood is a loss equal to the negative logarithm of the probability assigned to the true label.
• 

  liveBookmanning.com
  Reinforcement Learning from Human FeedbackNegative log-likelihood is the negation of a model's log-likelihood, used as a loss for probabilistic predictions.
• 

  liveBookmanning.com
  Deep Learning with PyTorch, Second EditionA loss function that penalizes low predicted probability assigned to the correct class by taking the negative logarithm of that probability and summing over samples.
• 

  liveBookmanning.com
  Natural Language Processing in Action, Second EditionNegative log likelihood is the negative of the logarithm of a model-assigned likelihood, commonly minimized as a probabilistic prediction loss.
• 

  liveBookmanning.com
  Causal AIThe negative log-likelihood loss function is the negative logarithm of the likelihood and is minimized to estimate parameters.
• 

  liveBookmanning.com
  Machine Learning Algorithms in DepthA negative log likelihood is a loss equal to the negative logarithm of the likelihood assigned to observed outcomes by a probabilistic model.

Compare how 7 books explain negative log-likelihood →

Related concepts

• cross entropy lossequivalent to · Cross-entropy loss measures the expected negative log probability that a predicted distribution assigns to outcomes drawn from a target distribution.
• mean squared errorequivalent to · Mean squared error is the arithmetic mean of the squared differences between corresponding predicted and target numerical values.
• DPO lossused by · The DPO loss is a preference-training objective that optimizes a policy to assign higher relative likelihood to preferred responses than to rejected responses compared with a reference policy.
• NLLLossused by · NLLLoss is a PyTorch loss function that computes negative log-likelihood loss from log-probability outputs and target classes.
• mixture density networkused by · A mixture density network is a neural network that predicts the parameters of a conditional mixture probability distribution, optionally at each sequence step.
• variational lossy autoencodermeasures · A variational lossy autoencoder is an autoencoder variant that intentionally constrains its latent representation so reconstruction preserves selected information rather than every input detail.

This explainer was generated by Manning's knowledge graph from the text of Ensemble Methods for Machine Learning. The section links point to the parts of the book it draws on.