liveBook · Manning

Optimization

As explained in Rearchitecting LLMs

An optimization technique is a method for improving a model's efficiency or task performance through changes to its design, parameters, or configuration.

LLM optimization changes model design or behavior to improve efficiency or task performance. Architecture knowledge supports techniques intended to preserve performance or improve it in specific fields. section 1.4

The treatment points toward two kinds of work:

  • Study Transformer architecture to identify optimization methods that can apply across models. section 3.1
  • Identify model behaviors and eliminate neurons to change unwanted behavior without affecting performance. section 1.4

How Rearchitecting LLMs covers it

How deep it goes
Frames architecture-aware optimization as a research capability but gives no specific method or implementation workflow here.
Examples
Uses DistilGPT2 to study Transformer structure and neuron elimination to modify unwanted model behavior.
What you can do after
Build from scratch architecture-aware optimization techniques and implement current research methods.
How it connects
Connects Transformer architecture, model activations, neuron removal, and explainability.

Also explained in

Compare how 38 books explain optimization →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Rearchitecting LLMs. The section links point to the parts of the book it draws on.