liveBook · Manning

Data curation

Also called dataset curation

Data curation is the systematic selection, review, cleaning, organization, and maintenance of data for a particular purpose.

How 5 Manning books explain it

Introduction to Generative AI, Second Edition

Dataset curation is the process of selecting, organizing, documenting, reviewing, maintaining, and quality-controlling data for a particular purpose.

How it is used
Supports responsible AI by documenting datasets and reducing risks that curation introduces or amplifies bias.
How deep it goes
Describes documentation and review needs, transparency initiatives, and legal obligations without giving a detailed curation procedure.
Examples
Illustrates documentation with dataset cards, the Data Nutrition Project, and a search tool for the ROOTS multilingual corpus.
What you can do after
Evaluate and understand dataset documentation and its role in identifying limitations and bias risks.

Read how Introduction to Generative AI, Second Edition explains data curation →

Building Specialized AI Systems

The process of sourcing, formatting, cleaning, labeling, and curating data for model training.

How it is used
Supports fine-tuning by aligning training data with the specialized model's intended knowledge, style, and capabilities.
How deep it goes
Explains sourcing choices and data-quality criteria, but does not detail cleaning or labeling procedures.
Examples
Uses Common Crawl, the Pile, RefinedWeb, sentiment analysis, named entity recognition, and question answering as dataset or task examples.
What you can do after
Evaluate/understand whether a dataset is sufficiently relevant, comprehensive, filtered, and deduplicated for a fine-tuning task.

Read how Building Specialized AI Systems explains data curation →

Deep Learning with R, Third Edition

Data curation is the selection, cleaning, organization, and quality management of data for analysis or machine learning.

How it is used
Improves deep-learning generalization by making training data informative, low-noise, and densely representative.
How deep it goes
Connects curation to manifold coverage and interpolation, but stops short of a procedural curation workflow.
Examples
Uses MNIST with randomly shuffled labels to expose the consequences of corrupted training relationships.
What you can do after
Apply via data improvements that increase informative, accurate, low-noise, and dense coverage before changing the model.

Read how Deep Learning with R, Third Edition explains data curation →

Build AI Drug Discovery Pipelines

Data curation is the systematic review, correction, normalization, and organization of data to improve its quality and consistency.

How it is used
Uses exploratory checks to detect measurement and annotation problems before machine-learning modeling.
How deep it goes
Demonstrates schema inspection and distribution review, then stops at identifying possible unit errors rather than correcting records.
Examples
Uses the hERG blockers dataset and a simulated three-unit pIC50 shift to expose a possible unit inconsistency.
What you can do after
Diagnose potential annotation and unit problems with Python-based schema inspection and distribution plots.

Read how Build AI Drug Discovery Pipelines explains data curation →

LLM Customization and Fine-Tuning

The process of selecting, cleaning, organizing, and preparing data for adaptation or evaluation.

How it is used
Prioritizes curation over training when representative examples are insufficient.
How deep it goes
Sets a sequence of baseline, held-out evaluation, and curated examples, but does not detail curation operations.
Examples
Uses curated clinical examples and an illustrative cost breakdown spanning curation, preprocessing, training, and evaluation.
What you can do after
Build a held-out evaluation set and core task examples before choosing an adaptation technique.

Read how LLM Customization and Fine-Tuning explains data curation →

Also covered in

What to know first

Related concepts

Questions about data curation

What is data curation?

Data curation is the systematic selection, review, cleaning, organization, and maintenance of data for a particular purpose.

Which Manning books explain data curation?

Introduction to Generative AI, Second Edition, Building Specialized AI Systems, Deep Learning with R, Third Edition, Build AI Drug Discovery Pipelines and LLM Customization and Fine-Tuning.

What should I know before learning data curation?

Manning books that teach data curation usually introduce language model, bias, security and foundation model first.

What is the difference between data curation and model architecture?

Data curation is the systematic selection, review, cleaning, organization, and maintenance of data for a particular purpose. Model architecture is the structural arrangement of a computational model’s components, connections, and operations that determines how it processes inputs and produces outputs.

What kinds of data curation are there?

Kinds of data curation include rejection sampling.

What is data curation part of?

Data curation is part of pipeline. A pipeline is an ordered sequence of connected processing stages through which data or work flows, with each stage passing or transforming its output for subsequent stages.

These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.