Active learning

As explained in Machine Learning System Design

Active learning is a machine-learning technique that selectively chooses informative or uncertain examples for labeling or inclusion in a dataset.

More data does not automatically improve a model: duplicate, overly similar, or extremely noisy samples may add little value. Active learning focuses dataset growth on examples most likely to teach the model something new. section 6.4

Use model behavior to prioritize examples for labeling or inclusion:

• Select samples where the model made mistakes, using human-in-the-loop signals when available. section 6.4
• Prioritize samples where the model has the lowest confidence. section 6.4
• Treat active learning as one option alongside sampling strategies for huge, imbalanced, duplicate-heavy datasets. section 6.4

PhotoStock search labels

For query–image relevance labels, begin with randomly selected pairs from historical queries and the photo library. Later, direct labelers toward pairs where the model is less confident or makes more errors, improving labeling efficiency. section 6.7.2

Figure 6.4 A schematic view of the main differences between aleatoric and epistemic uncertainties (source: “Explainable Uncertainty Quantifications for Deep Learning-Based Molecular Property Prediction” by ChuI Yang and YiPei Li, https://mng.bz/n0ZV)

Use the uncertainty schematic to frame why low-confidence examples can be valuable targets for review and labeling. section 6.4

How Machine Learning System Design covers it

How deep it goes

Defines error- and confidence-based selection, but defers detailed comparison of the many active-learning methods.

Examples

Uses PhotoStock query–image relevance pairs, model confidence, and model errors as concrete selection signals.

What it assumes

Assumes an existing model can expose errors or confidence and a human labeling process can review selected examples.

What you can do after

Apply active learning to prioritize human labeling and extend datasets with more informative examples.

How it connects

Connects active learning with dataset extension, human-in-the-loop labeling, sampling, uncertainty, and model errors.

liveBookmanning.com

Read Machine Learning System Design on liveBook Get the book at manning.com

Also explained in

• 

  liveBookmanning.com
  Machine Learning Algorithms in DepthActive learning is a supervised learning technique in which a learner selects unlabeled instances for labeling to improve performance with fewer labeled examples.
• 

  liveBookmanning.com
  AI-Powered SearchActive learning is a machine learning strategy in which the learner selects informative examples for labeling or feedback.

Compare how 3 books explain active learning →

Related concepts

• training datasetaddresses · A collection of examples or observations supplied to a learning procedure to fit or adapt a model or learn its patterns or parameters.
• acquisition functionuses · An acquisition function is a scoring rule that ranks candidate evaluations by their expected usefulness for selecting the next sample or experiment.

This explainer was generated by Manning's knowledge graph from the text of Machine Learning System Design. The section links point to the parts of the book it draws on.