Focal loss

As explained in Ensemble Methods for Machine Learning

Focal loss is a classification loss that down-weights well-classified examples and emphasizes poorly classified examples, especially under class imbalance.

Focal loss helps classification models handle severe class imbalance, such as detecting foreground objects among many background windows. It modifies cross-entropy so easy predictions contribute less and difficult predictions receive more attention. section 5.4.3

Its modulating factor is controlled by γ > 0. With γ = 0, focal loss becomes ordinary cross-entropy; increasing γ suppresses well-classified examples more strongly. section 5.4.3

Figure 5.21 The focal loss visualized for various values of γ. When γ = 0, the original cross-entropy loss is recovered. As γ increases, the part of the curve corresponding to well-classified examples becomes longer, reflecting the loss function’s focus on poor classification.

Compare how different γ values reshape the loss and shift emphasis toward poorly classified examples. section 5.4.3

A practical LightGBM implementation follows these steps:

• Convert raw predictions to probabilities with the sigmoid function and compute the focal-loss terms for labels 0 and 1. section 5.4.3
• Return the mean loss as an evaluation metric, then provide numerical first and second derivatives as the custom objective. section 5.4.3
• Train LightGBM with the custom objective, convert raw validation scores to probabilities, and threshold them at 0.5. section 5.4.3

How Ensemble Methods for Machine Learning covers it

How deep it goes

Defines the modulation mechanism, compares gamma values, and implements custom LightGBM metric and objective functions.

Examples

Uses dense object-detection windows, gamma-based loss curves, custom Python functions, and LightGBM validation predictions.

What it assumes

Assumes binary classification, cross-entropy, sigmoid probabilities, and familiarity with gradient boosting and LightGBM.

What you can do after

Apply it via LightGBM by defining focal-loss derivatives, training a classifier, and thresholding predicted probabilities.

How it connects

Connects cross-entropy loss, class imbalance, dense object detection, gradient boosting, and LightGBM.

liveBookmanning.com

Read Ensemble Methods for Machine Learning on liveBook Get the book at manning.com

Also explained in

• 

  liveBookmanning.com
  Machine Learning System DesignA classification loss that down-weights well-classified high-confidence examples so training emphasizes difficult examples.
• 

  liveBookmanning.com
  Math and Architectures of Deep LearningA modified classification loss that downweights easy examples and emphasizes hard examples, especially under class imbalance.

Compare how 3 books explain focal loss →

Related concepts

• cross entropy losskind of · Cross-entropy loss measures the expected negative log probability that a predicted distribution assigns to outcomes drawn from a target distribution.
• LightGBMused by · LightGBM is an open-source gradient-boosting framework for efficiently training tree-based machine-learning models.
• class imbalanceaddresses · Class imbalance is a condition in a classification dataset where classes have substantially unequal numbers of examples.
• fine-tuningused by · Fine-tuning is adapting a pretrained machine-learning model by training it further on targeted data or objectives for a task, domain, or behavior.
• object detectionaddresses · Object detection is a computer-vision task that identifies object instances in visual data, assigns them categories, and estimates their locations.

This explainer was generated by Manning's knowledge graph from the text of Ensemble Methods for Machine Learning. The section links point to the parts of the book it draws on.