Soft target
A soft target is a target representation that assigns probabilities or graded weights to multiple possible outputs rather than selecting a single categorical label.
Rearchitecting LLMs
A soft label is a probability-weighted target distribution that represents graded model predictions rather than a single discrete target.
- How it is used
- Transfers teacher output knowledge to a smaller student during distillation and recovery after structural pruning.
- How deep it goes
- Implements temperature-scaled KL divergence and gives configuration guidance for balancing it against hard-label and hidden-state losses.
- Examples
- Uses a three-logit softmax calculation and next-token vocabulary distributions to illustrate graded predictions.
- What you can do after
- Apply soft-target distillation via PyTorch’s softmax, log_softmax, and kl_div functions within a composite loss.
Build a DeepSeek Model (From Scratch)
A soft target is a target distribution that assigns graded values or probabilities to multiple possible classes rather than selecting only one class.
- How it is used
- Uses teacher probability distributions to supervise a smaller student in knowledge distillation.
- How deep it goes
- Explains temperature-scaled target computation and combines soft and hard losses, then compares student training approaches.
- Examples
- Illustrates class probabilities with cat, dog, fox, and bird predictions, and cites DistilBERT’s compression of BERT.
- What you can do after
- Evaluate how softened teacher probabilities and hard labels contribute to student training and generalization.
Read how Build a DeepSeek Model (From Scratch) explains soft target →
Machine Learning System Design
A label representation that assigns weighted or probabilistic membership rather than a single categorical class.
- How it is used
- Uses weighted pixel membership when fine-grained segmentation makes hard foreground/background labels inadequate.
- How deep it goes
- Defines mixed class membership and specific weights, while deferring a detailed implementation recipe.
- Examples
- Uses Zoom hair artifacts, foreground/background pixel weights, and adapted manufacturing data as concrete examples.
- What you can do after
- Apply weighted labels to fine-grained segmentation problems and evaluate their effect through test error.
Read how Machine Learning System Design explains soft target →
Also covered in
What to know first
- cross entropy lossCross-entropy loss measures the expected negative log probability that a predicted distribution assigns to outcomes drawn from a target distribution.
- BERTBERT (Bidirectional Encoder Representations from Transformers) is a pretrained Transformer encoder language model that produces contextual text representations using information from both directions.
- distillationDistillation is a training technique in which a student model learns to approximate the outputs or behavior of a teacher model.
- fine-tuningFine-tuning is the continued training of a pretrained model on targeted data or objectives to adapt its parameters or behavior to a task, domain, or use.
Related concepts
- teacher modelproduced by · A teacher model is a model whose outputs, behavior, or learned representations provide training signals for another model.
- labelcontrasts with · A label is a target value, category, or reference annotation associated with an example for learning, prediction, or evaluation.
- distillation lossused by · Distillation loss is a training objective that measures how closely a student model’s predictions match outputs or other information provided by a teacher model.
- distillationused by · Distillation is a training technique in which a student model learns to approximate the outputs or behavior of a teacher model.
- student modeluses · A student model is a model trained with guidance from a teacher model to approximate its outputs, behavior, or learned capabilities.
- hard labelcontrasts with · A hard label is a discrete target that identifies a designated class or token without representing probabilities over alternatives.
Questions about soft target
What is soft target?
A soft target is a target representation that assigns probabilities or graded weights to multiple possible outputs rather than selecting a single categorical label.
Which Manning books explain soft target?
Rearchitecting LLMs, Build a DeepSeek Model (From Scratch) and Machine Learning System Design.
What should I know before learning soft target?
Manning books that teach soft target usually introduce cross entropy loss, BERT, distillation and fine-tuning first.
What is the difference between soft target and label?
A soft target is a target representation that assigns probabilities or graded weights to multiple possible outputs rather than selecting a single categorical label. A label is a target value, category, or reference annotation associated with an example for learning, prediction, or evaluation.
What is soft target also called?
Soft target is also called soft label.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.


