liveBook · Manning

Sparse autoencoder

As explained in Rearchitecting LLMs

A sparse autoencoder is an autoencoder that reconstructs neural activations through a wider intermediate representation constrained to activate sparsely.

A sparse autoencoder (SAE) reconstructs activations from a model layer through a much wider intermediate space. A training penalty encourages only a few expanded dimensions, or features, to activate per input, aiming to separate mixed concepts into more interpretable features. section 10.4 section 11.5.1

This addresses superposition, in which an original neuron responds to multiple topics. Once a concept feature is identified, modifying or clamping its activation during inference can steer the model toward that concept. section 10.4 section 11.5.1

Training an SAE from scratch requires substantial data and compute. The described analysis instead studies activations in MLP expansion layers, which contain more neurons. section 10.4 section 11.5.1

Steering with a feature

Clamping the feature associated with the Golden Gate Bridge led Claude to mention the bridge in virtually any response, regardless of the question. section 11.5.1

How Rearchitecting LLMs covers it

How deep it goes
Defines sparse reconstruction and feature intervention, but does not provide training procedures or implementation details.
Examples
Uses feature isolation in a single-layer Transformer and Claude 3, plus Golden Gate Bridge steering.
What it assumes
Builds on transformer-layer and MLP activations while explaining sparse expansion and superposition.
What you can do after
Understand how sparse features can make concept-specific inference steering more precise.
How it connects
Connects superposition, MLP activations, dictionary learning, and inference-time feature steering.

Also explained in

Compare how 2 books explain sparse autoencoder →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Rearchitecting LLMs. The section links point to the parts of the book it draws on.