Calibration data is representative input used to measure model behavior and guide data-dependent compression. Importance scores are not fixed properties of model components: they depend on the inputs and task used during measurement. Static pruning avoids calibration but ignores how the model processes the target data. section 4.6 section 8.3.1
Use calibration data to guide different pruning decisions:
- For depth pruning, capture block input and output activations with PyTorch hooks and use them to assess block importance. section 4.6
- For width pruning, combine neuron activation norms with weight-magnitude scores. The resulting hybrid scores determine which neuron pairs to preserve. section 5.2.4
- For attention removal, capture hidden states at the input and output boundaries of attention, including its residual, and score contribution using cosine similarity. section 8.3.1
Choose the calibration distribution for the model's intended work. A domain-specific set can steer pruning toward that domain; a general-purpose set should cover a range of relevant capabilities. In either case, validate the compressed model on the tasks that matter rather than treating calibration scores as final performance measures. section 5.2.5 section 8.3.1
Balanced calibration for attention scoring
A general-purpose attention-pruning workflow collects 400 samples from six Cosmopedia subsets. Stories and web samples supply narrative and general-domain text; WikiHow supplies procedural text; OpenStax and Stanford supply academic content. The subset weights reflect the capabilities probed by the evaluation benchmarks. section 8.3.1
Listing 8.5 Calibration dataset composition
from datasets import load_dataset, Dataset
from torch.utils.data import DataLoader
BATCH_SIZE = 4
MAX_LENGTH = 2048
CALIBRATION_SAMPLES = 400
dataset_name = "HuggingFaceTB/cosmopedia"
subsets = [
("stories", 0.300),
("web_samples_v2", 0.200),
("web_samples_v1", 0.150),
("wikihow", 0.150),
("openstax", 0.125),
("stanford", 0.075),
]
all_samples = []
for subset, weight in subsets:
n_samples = int(CALIBRATION_SAMPLES * weight) #A
subset_data = load_dataset(
dataset_name,
subset,
split="train",
streaming=True)
subset_samples = list(subset_data.take(n_samples))
all_samples.extend(subset_samples)
print(f" Collected {len(subset_samples):,} samples")
calibration_dataset = Dataset.from_dict(
{"text": [s["text"] for s in all_samples]}
)
The listing shows the subset weights, streaming collection, sample count, and data-loading settings for the balanced calibration set. section 8.3.1
Domain-specific width pruning
Two models prune 20% of each MLP's intermediate neurons, using activation norms collected from WikiText or SMS data. Both have approximately 1.07B parameters, down from 1.24B. The Wiki-calibrated model has lower Wiki perplexity than the SMS-calibrated model, while the SMS-calibrated model has lower SMS perplexity. Both remain worse than the base model on both datasets, illustrating the capacity cost of pruning. section 5.2.4 section 5.2.5
Table 5.1 Perplexity results
|
Model
|
Parameters
|
Wiki
Perplexity
|
SMS
Perplexity
|
|
Base
|
1.24B
|
25.69
|
122.91
|
|
Wiki model
|
1.07B
(~87%)
|
36.28
|
182.54
|
|
SMS model
|
1.07B
(~87%)
|
48.65
|
165.54
|
|
Static 20% pruned
|
1.07B
(~87%)
|
52
|
217
|
Compare both specialized models with the base model and a weight-magnitude-only pruning baseline across the two datasets. section 5.2.5





