liveBook · Manning

Taylor approximation

As explained in Rearchitecting LLMs

A Taylor approximation estimates the change in a model's loss caused by removing a component using a local approximation based on derivatives.

Taylor approximation estimates a model block’s importance by approximating how much the loss would increase if that block were removed. It requires one forward/backward cycle, making it more efficient than removing every block separately and measuring perplexity, but gradients add implementation and computational overhead compared with forward-only metrics. section 4.4

How Rearchitecting LLMs covers it

How deep it goes
Defines the loss-based estimate and its forward/backward cost, but does not derive the approximation or detail its gradient calculation.
Examples
Contrasts one forward/backward cycle with evaluating perplexity after temporarily removing each block from a 28-block model.
What you can do after
Evaluate block importance for pruning while weighing gradient overhead against repeated perplexity measurements.
How it connects
Connects depth pruning with loss, gradients, perplexity, and cosine similarity as alternative block-importance metrics.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Rearchitecting LLMs. The section links point to the parts of the book it draws on.