Taylor approximation estimates a model block’s importance by approximating how much the loss would increase if that block were removed. It requires one forward/backward cycle, making it more efficient than removing every block separately and measuring perplexity, but gradients add implementation and computational overhead compared with forward-only metrics. section 4.4
Taylor approximation
A Taylor approximation estimates the change in a model's loss caused by removing a component using a local approximation based on derivatives.
How Rearchitecting LLMs covers it
- How deep it goes
- Defines the loss-based estimate and its forward/backward cost, but does not derive the approximation or detail its gradient calculation.
- Examples
- Contrasts one forward/backward cycle with evaluating perplexity after temporarily removing each block from a 28-block model.
- What you can do after
- Evaluate block importance for pruning while weighing gradient overhead against repeated perplexity measurements.
- How it connects
- Connects depth pruning with loss, gradients, perplexity, and cosine similarity as alternative block-importance metrics.
Related concepts
- approximationkind of · Approximation is representing or computing a result that is sufficiently close to an exact result for an intended purpose, without requiring exact equality.
- perplexityalternative to · Perplexity is a language-model metric, typically the exponential of average cross-entropy, that measures how well a model predicts a sequence, with lower values indicating better predictive fit.
This explainer was generated by Manning's knowledge graph from the text of Rearchitecting LLMs. The section links point to the parts of the book it draws on.
