liveBook · Manning

Calibration data

As explained in Rearchitecting LLMs

Calibration data is representative input data used to measure model behavior and guide data-dependent model compression decisions.

Calibration data is representative input used to measure model behavior and guide data-dependent compression. Importance scores are not fixed properties of model components: they depend on the inputs and task used during measurement. Static pruning avoids calibration but ignores how the model processes the target data. section 4.6 section 8.3.1

Use calibration data to guide different pruning decisions:

  • For depth pruning, capture block input and output activations with PyTorch hooks and use them to assess block importance. section 4.6
  • For width pruning, combine neuron activation norms with weight-magnitude scores. The resulting hybrid scores determine which neuron pairs to preserve. section 5.2.4
  • For attention removal, capture hidden states at the input and output boundaries of attention, including its residual, and score contribution using cosine similarity. section 8.3.1

Choose the calibration distribution for the model's intended work. A domain-specific set can steer pruning toward that domain; a general-purpose set should cover a range of relevant capabilities. In either case, validate the compressed model on the tasks that matter rather than treating calibration scores as final performance measures. section 5.2.5 section 8.3.1

Balanced calibration for attention scoring

A general-purpose attention-pruning workflow collects 400 samples from six Cosmopedia subsets. Stories and web samples supply narrative and general-domain text; WikiHow supplies procedural text; OpenStax and Stanford supply academic content. The subset weights reflect the capabilities probed by the evaluation benchmarks. section 8.3.1

Listing 8.5 Calibration dataset composition
from datasets import load_dataset, Dataset
from torch.utils.data import DataLoader
 
BATCH_SIZE = 4
MAX_LENGTH = 2048
CALIBRATION_SAMPLES = 400
 
dataset_name = "HuggingFaceTB/cosmopedia"
 
subsets = [
    ("stories",        0.300),
    ("web_samples_v2", 0.200),
    ("web_samples_v1", 0.150),
    ("wikihow",        0.150),
    ("openstax",       0.125),
    ("stanford",       0.075),
]
 
all_samples = []
for subset, weight in subsets:
    n_samples = int(CALIBRATION_SAMPLES * weight) #A
    subset_data = load_dataset(
            dataset_name, 
            subset, 
            split="train", 
            streaming=True)
    subset_samples = list(subset_data.take(n_samples))
    all_samples.extend(subset_samples)
    print(f"    Collected {len(subset_samples):,} samples")
 
calibration_dataset = Dataset.from_dict(
            {"text": [s["text"] for s in all_samples]}
            )
 

The listing shows the subset weights, streaming collection, sample count, and data-loading settings for the balanced calibration set. section 8.3.1

Domain-specific width pruning

Two models prune 20% of each MLP's intermediate neurons, using activation norms collected from WikiText or SMS data. Both have approximately 1.07B parameters, down from 1.24B. The Wiki-calibrated model has lower Wiki perplexity than the SMS-calibrated model, while the SMS-calibrated model has lower SMS perplexity. Both remain worse than the base model on both datasets, illustrating the capacity cost of pruning. section 5.2.4 section 5.2.5

Table 5.1 Perplexity results
Model
Parameters
Wiki
Perplexity
SMS
Perplexity
Base
1.24B
25.69
122.91
Wiki model
1.07B
(~87%)
36.28
182.54
SMS model
1.07B
(~87%)
48.65
165.54
Static 20% pruned
1.07B
(~87%)
52
217

Compare both specialized models with the base model and a weight-magnitude-only pruning baseline across the two datasets. section 5.2.5

How Rearchitecting LLMs covers it

How deep it goes
Works through calibration-set construction, activation capture, importance scoring, and evaluation of pruned models.
Examples
Uses a weighted Cosmopedia sample for attention scoring and WikiText and SMS data for domain-specific pruning.
What you can do after
Evaluate/understand how calibration choices affect pruning decisions and compare compressed models against baselines.
How it connects
Connects calibration data to depth pruning, width pruning, activation hooks, cosine similarity, and perplexity.

Also explained in

Compare how 6 books explain calibration data →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Rearchitecting LLMs. The section links point to the parts of the book it draws on.