liveBook · Manning

Soft target

Also called soft label

A soft target is a target representation that assigns probabilities or graded weights to multiple possible outputs rather than selecting a single categorical label.

How 3 Manning books explain it

Rearchitecting LLMs

A soft label is a probability-weighted target distribution that represents graded model predictions rather than a single discrete target.

How it is used
Transfers teacher output knowledge to a smaller student during distillation and recovery after structural pruning.
How deep it goes
Implements temperature-scaled KL divergence and gives configuration guidance for balancing it against hard-label and hidden-state losses.
Examples
Uses a three-logit softmax calculation and next-token vocabulary distributions to illustrate graded predictions.
What you can do after
Apply soft-target distillation via PyTorch’s softmax, log_softmax, and kl_div functions within a composite loss.

Read how Rearchitecting LLMs explains soft target →

Build a DeepSeek Model (From Scratch)

A soft target is a target distribution that assigns graded values or probabilities to multiple possible classes rather than selecting only one class.

How it is used
Uses teacher probability distributions to supervise a smaller student in knowledge distillation.
How deep it goes
Explains temperature-scaled target computation and combines soft and hard losses, then compares student training approaches.
Examples
Illustrates class probabilities with cat, dog, fox, and bird predictions, and cites DistilBERT’s compression of BERT.
What you can do after
Evaluate how softened teacher probabilities and hard labels contribute to student training and generalization.

Read how Build a DeepSeek Model (From Scratch) explains soft target →

Machine Learning System Design

A label representation that assigns weighted or probabilistic membership rather than a single categorical class.

How it is used
Uses weighted pixel membership when fine-grained segmentation makes hard foreground/background labels inadequate.
How deep it goes
Defines mixed class membership and specific weights, while deferring a detailed implementation recipe.
Examples
Uses Zoom hair artifacts, foreground/background pixel weights, and adapted manufacturing data as concrete examples.
What you can do after
Apply weighted labels to fine-grained segmentation problems and evaluate their effect through test error.

Read how Machine Learning System Design explains soft target →

Also covered in

What to know first

Related concepts

Questions about soft target

What is soft target?

A soft target is a target representation that assigns probabilities or graded weights to multiple possible outputs rather than selecting a single categorical label.

Which Manning books explain soft target?

Rearchitecting LLMs, Build a DeepSeek Model (From Scratch) and Machine Learning System Design.

What should I know before learning soft target?

Manning books that teach soft target usually introduce cross entropy loss, BERT, distillation and fine-tuning first.

What is the difference between soft target and label?

A soft target is a target representation that assigns probabilities or graded weights to multiple possible outputs rather than selecting a single categorical label. A label is a target value, category, or reference annotation associated with an example for learning, prediction, or evaluation.

What is soft target also called?

Soft target is also called soft label.

These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.