liveBook · Manning

Neuron

As explained in Rearchitecting LLMs

A neuron is a computational unit in a neural network that transforms inputs using learned parameters and a nonlinear activation.

In a GLU-based MLP, each intermediate neuron corresponds to a row in both gate_proj and up_proj. Width pruning removes paired neurons while retaining hidden_size at MLP boundaries. section 5.1.1 section 5.5

Neuron selection can use:

  • Static scoring: combine the weight ranges for each row in gate_proj and up_proj; lower scores mark removal candidates. section 5.1.1
  • Data-driven scoring: combine structural importance with activation norms measured during calibration. section 5.5
def compute_neuron_pair_importance(gate_weight, up_weight):
 
    gate_max_abs = torch.max(gate_weight, dim=1).values + ➥
torch.abs(torch.min(gate_weight, dim=1).values)
    up_max_abs = torch.max(up_weight, dim=1).values + ➥
torch.abs(torch.min(up_weight, dim=1).values)
 
    importance_scores = gate_max_abs + up_max_abs
 
    return importance_scores

Computes a score for each paired neuron by summing the gate and up projection weight ranges. section 5.1.1

Comparing weight ranges

A neuron with weights from -0.2 to 0.1 has a range score of 0.3; one spanning -0.5 to 0.4 scores 0.9 and is less likely to be removed. section 5.1.1

How Rearchitecting LLMs covers it

How deep it goes
Defines neurons as projection-matrix rows and details weight-range scoring alongside activation-calibrated selection.
Examples
Uses weight-range comparisons and contrasts WikiText- and SMS-calibrated neuron selection.
What it assumes
Uses GLU projection structure and PyTorch Linear weight layout while spelling out matrix dimensions and neuron rows.
What you can do after
Apply width pruning by scoring paired projections, selecting neurons, and rebuilding smaller Linear layers while preserving module boundaries.
How it connects
Connects neurons to GLU MLPs, gate_proj, up_proj, down_proj, activation norms, and width pruning.

Also explained in

Compare how 10 books explain neuron →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Rearchitecting LLMs. The section links point to the parts of the book it draws on.