liveBook · Manning

Data curation

As explained in Building Specialized AI Systems

The process of sourcing, formatting, cleaning, labeling, and curating data for model training.

Data curation prepares fine-tuning data so a specialized model develops the intended domain knowledge, style, and capabilities. It is a decisive stage: data quality, relevance, and comprehensiveness can determine whether fine-tuning succeeds. Fine-tuning sets are smaller than pretraining corpora, so they demand substantially higher quality and closer task fit. section 1.4.2

Choose data sources according to the target task and available resources:

  • Use publicly available datasets for broad coverage and established benchmarks, particularly when beginning fine-tuning. section 1.4.2
  • Prefer curated, task-specific datasets with quality filtering and deduplication for fine-tuning tasks such as sentiment analysis, named entity recognition, and question answering. section 1.4.2
  • Treat large web corpora such as Common Crawl and the Pile as better suited to pretraining than task-focused fine-tuning. section 1.4.2
Figure 1.7 Task-specific fine-tuning process. Pretraining creates a base model that is then further trained to perform multiple, more complex tasks, such as summarization, classification, and code or text generation.

Task-specific fine-tuning builds on a pretrained base model and can specialize it for summarization, classification, or code and text generation. section 1.4.2

How Building Specialized AI Systems covers it

How deep it goes
Explains sourcing choices and data-quality criteria, but does not detail cleaning or labeling procedures.
Examples
Uses Common Crawl, the Pile, RefinedWeb, sentiment analysis, named entity recognition, and question answering as dataset or task examples.
What it assumes
Assumes an appropriate foundation model has been selected before data preparation begins.
What you can do after
Evaluate/understand whether a dataset is sufficiently relevant, comprehensive, filtered, and deduplicated for a fine-tuning task.
How it connects
Connects data curation to fine-tuning, pretraining, foundation models, sentiment analysis, named entity recognition, and question answering.

Also explained in

Compare how 5 books explain data curation →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Building Specialized AI Systems. The section links point to the parts of the book it draws on.