Hybrid mode ranks neurons for MLP width pruning by combining each neuron’s weight information with activation statistics measured on calibration prompts. In OptiPFair, PPM is the supported selection method for this mode. section 13.4.1
Pruning a GLU MLP
For GLU-structured MLPs, pass PPM and a calibration dataloader to prune_model. Setting expansion_rate to 225 targets an intermediate width of 2304 for a hidden size of 1024. section 13.4.1
Listing 13.9 Calibration data
CALIBRATION_BATCH_SIZE = 8
MAX_CALIBRATION_LENGTH = 1152
calibration_texts = list(train_dataset["prompt"]) #A
lens = [len(tokenizer(t)["input_ids"])
for t in calibration_texts]
calibration_dataset = Dataset.from_dict({"text": calibration_texts})
calibration_dataset = calibration_dataset.map(
lambda ex: tokenizer(
ex["text"], truncation=True,
max_length=MAX_CALIBRATION_LENGTH,
return_tensors=None,
), #1
batched=True,
)
def collate_calibration(batch): #2
max_len = max(len(ex["input_ids"]) for ex in batch)
pad_id = tokenizer.pad_token_id
return {
"input_ids": torch.tensor([
ex["input_ids"] + [pad_id] * (max_len - len(ex["input_ids"]))
for ex in batch
]),
"attention_mask": torch.tensor([
ex["attention_mask"]
+ [0] * (max_len - len(ex["attention_mask"]))
for ex in batch
]), #3
}
calibration_dataloader = DataLoader(
calibration_dataset, batch_size=CALIBRATION_BATCH_SIZE,
collate_fn=collate_calibration,
)Review the calibration-data preparation used to supply prompts to the pruning workflow. section 13.4.1
