liveBook · Manning

Hybrid mode

As explained in Rearchitecting LLMs

Hybrid mode combines neuron weight information with activation statistics measured on calibration inputs when ranking neurons for pruning.

Hybrid mode ranks neurons for MLP width pruning by combining each neuron’s weight information with activation statistics measured on calibration prompts. In OptiPFair, PPM is the supported selection method for this mode. section 13.4.1

Pruning a GLU MLP

For GLU-structured MLPs, pass PPM and a calibration dataloader to prune_model. Setting expansion_rate to 225 targets an intermediate width of 2304 for a hidden size of 1024. section 13.4.1

Listing 13.9 Calibration data
CALIBRATION_BATCH_SIZE = 8
MAX_CALIBRATION_LENGTH = 1152
 
calibration_texts = list(train_dataset["prompt"]) #A
lens = [len(tokenizer(t)["input_ids"])
        for t in calibration_texts]
 
calibration_dataset = Dataset.from_dict({"text": calibration_texts})
calibration_dataset = calibration_dataset.map(
    lambda ex: tokenizer(
        ex["text"], truncation=True,
        max_length=MAX_CALIBRATION_LENGTH, 
        return_tensors=None,
    ), #1
    batched=True,
)
 
def collate_calibration(batch): #2
    max_len = max(len(ex["input_ids"]) for ex in batch)
    pad_id = tokenizer.pad_token_id
    return {
        "input_ids": torch.tensor([
            ex["input_ids"] + [pad_id] * (max_len - len(ex["input_ids"]))
            for ex in batch
        ]),
        "attention_mask": torch.tensor([
            ex["attention_mask"]
            + [0] * (max_len - len(ex["attention_mask"])) 
            for ex in batch
        ]), #3
    }
 
calibration_dataloader = DataLoader(
    calibration_dataset, batch_size=CALIBRATION_BATCH_SIZE,
    collate_fn=collate_calibration,
)

Review the calibration-data preparation used to supply prompts to the pruning workflow. section 13.4.1

How Rearchitecting LLMs covers it

How deep it goes
Combines neuron weights with activation statistics measured over calibration prompts.
Examples
Uses PPM to target a 2304-neuron intermediate width from a 1024-wide hidden state.
What it assumes
Requires calibration prompts supplied through a dataloader for activation measurements.
What you can do after
Apply it via OptiPFair by selecting PPM and supplying a calibration dataloader.
How it connects
Connects neuron pruning with PPM, calibration data, and activation statistics.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Rearchitecting LLMs. The section links point to the parts of the book it draws on.