liveBook · Manning

Data curation

As explained in LLM Customization and Fine-Tuning

The process of selecting, cleaning, organizing, and preparing data for adaptation or evaluation.

Data curation prepares representative, high-quality examples for evaluation and model adaptation. When there are not yet a few hundred suitable examples, curate data before choosing a training technique. section 1.7

Start with the data needed to make and measure the adaptation decision:

  • Build a held-out evaluation set and core task examples before selecting a technique; reuse the same evaluation set to judge each technique. section 1.7
  • Size the evaluation set for the decision: 50 examples can reveal a large gap, while resolving differences of a few points typically takes 200 to 500. section 1.7
  • Expect human effort in curation and evaluation to outweigh compute costs in the illustrated fine-tuning project. section 6.12.4

Clinical adaptation

A hospital team curates a few hundred clinical examples, measures a baseline with a self-hosted open model, and trains a LoRA adapter. section 1.7

Table 6.14 Training cost breakdown (illustrative)
Phase
Duration
Compute Cost
Human Cost
Total
Data curation and quality review
2 weeks
–
$8,000 (clinical reviewer time)
$8,000
Data formatting and preprocessing
3 days
$50 (CPU instances)
$2,000 (ML engineer time)
$2,050
Training (including failed experiments)
2 days
$400 (GPU hours)
$1,500 (ML engineer time)
$1,900
Evaluation and safety testing
1 week
$200 (inference costs)
$4,000 (clinical + ML engineer)
$4,200
Total
~3.5 weeks
$650
$15,500
$16,150

Compare illustrative staff time and compute costs across curation, preprocessing, training, and evaluation. section 6.12.4

How LLM Customization and Fine-Tuning covers it

How deep it goes
Sets a sequence of baseline, held-out evaluation, and curated examples, but does not detail curation operations.
Examples
Uses curated clinical examples and an illustrative cost breakdown spanning curation, preprocessing, training, and evaluation.
What you can do after
Build a held-out evaluation set and core task examples before choosing an adaptation technique.
How it connects
Connects shared task examples and evaluation to LoRA, supervised fine-tuning, distillation, preference tuning, and retrieval-augmented generation.

Also explained in

Compare how 5 books explain data curation →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of LLM Customization and Fine-Tuning. The section links point to the parts of the book it draws on.