What you will learn in Rearchitecting LLMs
Structural techniques for efficient models
Chapter by chapter
Chapter 1: Why rearchitecting LLMs matters
The introduction explains why broad, generic LLMs can be a poor fit for specialized tasks and presents structural rearchitecting as an alternative to relying only on prompts.
What you learn in chapter 1 →Chapter 2: An end-to-end rearchitecting project
Introduces a general-purpose project to make a base model smaller and faster while preserving its capabilities through structural optimization and knowledge recovery.
What you learn in chapter 2 →Chapter 3: A blueprint to modern transformers
The introduction identifies attention memory growth and MLP compute as distinct bottlenecks that motivate studying transformer components.
What you learn in chapter 3 →Chapter 4: Building smaller and faster LLMs with depth pruning
The introduction motivates task-aware transformer block selection by questioning whether a depth-pruning heuristic that works for general-purpose models also suits specific tasks.
What you learn in chapter 4 →Chapter 5: Shaping model architectures via width pruning
The chapter introduction contrasts coarse depth pruning with more precise neuron-level width pruning and identifies wide GLU-based MLPs as suitable targets.
What you learn in chapter 5 →Chapter 6: Knowledge recovery through distillation
The introduction explains why pruning makes knowledge recovery necessary and why recovery begins with the choice of what to remove.
What you learn in chapter 6 →Chapter 7: Model specialization
The chapter introduction explains how fine-tuning can specialize a model’s behavior, contrasts it with RAG, and motivates parameter-efficient fine-tuning.
What you learn in chapter 7 →Chapter 8: Attention optimization
Introduces attention as an inference bottleneck whose costs grow with context length, motivating the chapter’s optimization work.
What you learn in chapter 8 →Chapter 9: Dynamic routing with Mixture of Experts
The introduction presents MoE as a way to add specialized experts to a dense model and defines the two routing strategies the chapter will compare.
What you learn in chapter 9 →Chapter 10: Exploring the transformer black box
The introduction frames transformer activations as a way to inspect model behavior and briefly explains what activation heatmaps and the logit lens reveal.
What you learn in chapter 10 →Chapter 11: Precision pruning for bias
The introduction defines demographic bias, explains the basic idea of comparing prompts and adjusting a few neurons, and clarifies the limits of that technique.
What you learn in chapter 11 →Chapter 12: Case study — replacing one agent call with a specialized SLM in production
The chapter introduces a costly agent-loop bottleneck and the use of shadow testing to reduce the risk of replacing its frontier API with a specialized SLM.
What you learn in chapter 12 →Chapter 13: Capstone — cascading specialists into a model family for agents
This introduction motivates using simpler models for repetitive tool-routing decisions in LLM-based agents.
What you learn in chapter 13 →Concepts explained in this book
- distillationKnowledge distillation is a model-compression technique in which a smaller student model learns to reproduce useful behavior from a larger teacher model.
- width pruningWidth pruning is a structured pruning technique that removes selected neurons or channels to reduce a model's width.
- routerA request router is a component that directs incoming requests to an appropriate processing queue or model configuration.
- mixture of expertsA mixture-of-experts block is a neural-network component containing multiple expert subnetworks and a router that selects a subset of them for each token.
- multilayer perceptronA multilayer perceptron is a feed-forward neural network composed of successive learned linear transformations and nonlinear activations.
- expert networkAn expert is an independently parameterized neural-network component in a mixture-of-experts architecture that processes selected inputs.
- hookA model hook is a callback attached to a neural-network component that observes or modifies computation at a specified point during execution.
- attentionAttention is a neural-network mechanism that computes input-dependent weights to prioritize information during processing.
- key-value cacheA key-value cache stores previously computed attention key and value tensors to accelerate autoregressive inference.
- feature importance scoreAn importance score numerically estimates how important a model component is to its information or performance.
- depth pruningDepth pruning is a model-reduction technique that removes complete layers from a neural network, often targeting later layers.
- down projection layerA down projection is a linear transformation that reduces an intermediate representation back to the model's hidden dimension.
- transformer blockA transformer layer, or transformer block, is a processing unit that uses attention and feed-forward components to transform representations.
- benchmarkA benchmark is a standardized task or test suite used to measure and compare model performance.
- gated linear unitA gated linear unit is a neural-network structure that modulates one linear transformation with a learned gate derived from another transformation.
- gate projectionA gate projection produces the signal that controls another intermediate signal in a gated feed-forward block.
- data-driven methodData-driven pruning selects model components for removal using measurements from representative input data.
- baselineA quantitative baseline is an objective reference measurement established before interventions to evaluate their effects.
- cosine similarityCosine similarity measures the cosine of the angle between two vectors, reflecting their directional alignment independently of magnitude.
- QLoRAQLoRA is a parameter-efficient fine-tuning technique that quantizes a base model and trains low-rank adapters while preserving the quantized model weights.
- activation normAn activation norm is a numerical magnitude of a neural activation signal, used to quantify how strongly a unit responds to input data.
- knowledge recoveryKnowledge recovery is a model-rearchitecting phase that restores or preserves model capabilities after structural changes.
- inference performanceInference speed is the rate at which a trained model produces outputs from inputs.
- LAMBADALAMBADA is a language-modeling benchmark that evaluates whether a model can predict a target word from a textual passage using its preceding context.
- recoveryA recovery process is a post-restructuring procedure that restores or improves a model’s performance after substantial structural components have been removed.
- grouped-query attentionGrouped query attention is an attention mechanism in which multiple query heads share key and value heads.
- softmaxSoftmax converts a vector of scores into nonnegative probabilities that sum to one.
- Kullback–Leibler divergenceKullback–Leibler divergence measures how one probability distribution differs from a reference distribution.
- throughputThroughput is the amount of model work or output that a system completes per unit of time.
- up projectionAn up projection is a feed-forward linear transformation that expands a hidden representation into an intermediate dimension.
- intermediate sizeAn intermediate size is the dimensionality of a neural module's intermediate representation, such as the expansion dimension of an MLP.
- structural modificationAttention removal is a model modification that removes attention sub-layers from selected transformer blocks.
- soft targetA soft label is a probability-weighted target distribution that represents graded model predictions rather than a single discrete target.
- foundation modelA foundation model is a broadly pretrained model intended to support adaptation to many downstream tasks.
- protection heuristicA protection heuristic is a rule that excludes selected model components from pruning to reduce the risk of damaging model capabilities.
- training loopA training loop is an iterative procedure that computes model outputs, evaluates an error, and updates trainable parameters over batches and iterations.
- perplexityPerplexity is a language-modeling metric that measures how well a model predicts a sequence, with lower values indicating better predictive performance.
- calibration dataCalibration data is representative input data used to measure model behavior and guide data-dependent model compression decisions.
- hard routingHard routing is a routing behavior that assigns a token to selected experts without continuously weighting all candidate expert outputs.
- quantizationQuantization represents model parameters or intermediate states with lower-precision numerical formats to reduce storage and computation requirements.
- pruningA pruning strategy is a method for deciding which model components to remove during model compression.
- hidden stateA hidden state is an internal vector representation produced by a neural model layer for the information processed up to that layer.
- kv cache quantizationKV cache quantization reduces the numerical precision of stored attention key and value states to lower memory use during autoregressive inference.
- low-rank adaptationLow-rank adaptation (LoRA) is a parameter-efficient fine-tuning technique that trains low-rank updates while keeping the original model weights frozen.
- structured pruningStructured pruning is a model-compression technique that removes organized groups of parameters, such as neurons, channels, or layers.
- teacher modelA teacher model is a model whose outputs or behavior guide the training of a student model during knowledge distillation.
- activationAn activation is a value produced by a neural-model unit during computation.
- small language modelA small language model is a language model with comparatively limited parameters and computational requirements, typically optimized for efficient deployment.
- activation captureActivation capture records intermediate neural signals produced during model execution.
- Attention DropAttention Drop is a training-free technique that removes attention layers from a transformer model to reduce computational or memory requirements.
- routing weightA routing weight is a scalar produced by a mixture-of-experts router that controls an expert's contribution to the combined output.
- student modelA student model is a model trained to reproduce a teacher model's behavior during knowledge distillation.
- fine-tuningFine-tuning that adapts a model for a particular specialization or domain.
- QDoRAQDoRA is a quantized weight-decomposition low-rank adaptation technique that adds magnitude parameters to direction-based low-rank updates.
- expansion ratioAn expansion rate is the ratio of a module's intermediate dimension to the model's hidden-state dimension, expressed as a percentage or factor.
- long-range contextLong-range context is information occurring substantially earlier in a sequence that remains relevant to predicting a later token.
- compound lossA compound loss function combines multiple loss components, often using configurable weights.
- logit lossLogit loss is a distillation loss that measures disagreement between student and teacher logits or their derived output distributions.
- task lossA task loss is an optimization objective that measures disagreement between model predictions and hard labels for a specified task.
- custom modelA specialized model is a machine-learning model adapted for a particular domain, dataset, or set of tasks and assigned as an expert within a larger model.
- schema complianceSchema compliance accuracy is the proportion of model outputs that conform to a required structured output schema.
- attention pruningAttention pruning removes or bypasses attention components in a neural model to reduce resource requirements.
- pre-hookA pre-hook is a callback that receives a module's input arguments before the module executes.
- DoRADoRA is a parameter-efficient adaptation method that decomposes model weights into separate magnitude and direction components and updates them independently.
- model rearchitectingModel rearchitecting is the deliberate modification of a model's architecture or components to meet particular requirements or constraints.
- data-driven selectionData-driven selection is the choice of model components based on their measured behavior on task-relevant data.
- memory usageMemory consumption is the amount of computer memory required to store and execute a model or computation.
- neural network layerA model layer is a computational stage in a neural network that transforms incoming representations.
- freezeSelective freezing is a training technique that keeps chosen model parameters fixed while allowing other parameters to update.
- feature alignmentFeature alignment is a distillation technique that matches intermediate representations produced by corresponding blocks of teacher and student models.
- Gaussian noiseGaussian noise is a random perturbation whose values follow a normal probability distribution.
- language modeling headA language modeling head maps model representations to scores over a language model's vocabulary.
- Skew KLDSkew KL divergence is an asymmetric divergence that compares a probability distribution with a mixture of distributions, and can be used as a distillation objective.
- hard labelA hard label is a target representation that identifies the designated correct class or token without encoding alternative probabilities.
- large language modelA large language model is a neural language model trained on extensive text data to understand and generate language.
- NormalFloat44-bit NormalFloat quantization represents model weights with a four-bit NormalFloat numerical encoding tailored to the distribution of language-model weights.
- torch.topktorch.topk is a PyTorch operation that returns the largest or smallest k elements of a tensor together with their indices.
- register_forward_pre_hookregister_forward_pre_hook is a PyTorch method that registers a callback to run before a module's forward computation.
- logit lensThe logit lens is an interpretability technique that projects intermediate hidden states through a model's final normalization and prediction head to inspect layer-wise predictions.
- intermediate representationAn intermediate representation is a vector-valued activation produced inside a neural model between its input and output layers.
- attention headAn attention head is an independent attention mechanism that learns to focus on particular types of relationships among sequence elements.
- keyA key vector is an attention representation matched against queries to determine attention weights.
- multi-head attentionMulti-head attention is an attention mechanism that computes attention through multiple parallel representation subspaces and combines their outputs.
- queryA query vector is an attention representation used to determine how strongly a position should attend to available keys.
- hidden lossA hidden loss is a loss term that measures disagreement between intermediate representations produced by a teacher and a student model.
- speedup degradation ratioSDR, or Speedup Degradation Ratio, is the percentage of performance drop divided by the percentage of speedup gained from a model modification.
- activation differenceA signed activation difference is the stereotyped-prompt mean activation minus the majority-prompt mean activation for a neuron.
- static pruningStatic pruning selects model components for removal using fixed structural information or parameter statistics rather than runtime activation behavior.
- capture pointA capture point is a specified model location at which intermediate activations are recorded during computation.
- parameter countParameter count is the number of learned scalar parameters in a machine-learning model.
- chat templateA chat template is a formatting specification that structures messages and other content into a model's expected conversational input format.
- Llama-3.2-3BLlama-3.2-3B is a language model in Meta's Llama 3.2 family with approximately three billion parameters.
- cosine distanceCosine distance measures dissimilarity between vectors based on their directional alignment.
- cross entropy lossCross-entropy is a loss function that measures the discrepancy between a target probability distribution and a predicted probability distribution.
- token shiftingToken shifting aligns each input position with the subsequent token that a causal language model must predict.
- tokenizationTokenization is the process of converting text into discrete numerical units that a language model can process.
- register_forward_hookregister_forward_hook is a PyTorch method that registers a callback to run after a module's forward computation.
- hybrid importance scoreA hybrid importance score combines activation norms with a structural measure such as weight magnitude to assess neuron importance.
- inner productA dot product is a scalar operation between two equal-length vectors that measures their directional alignment.
- multi-query attentionMulti-query attention is an attention architecture in which multiple query heads share a single set of key and value vectors.
- qkv mechanismThe QKV mechanism computes query, key, and value vectors so attention can score relationships and combine contextual information.
- valueA value is a vector containing the semantic information that attention aggregates from tokens selected by query-key compatibility.
- l2 normThe L2 norm is the square root of the sum of squared vector components, measuring vector magnitude with greater emphasis on large components.
- logitA logit is an unnormalized numerical score produced by a model before conversion into probabilities.
- GELUGELU is a smooth neural-network activation function that weights inputs using the Gaussian error function.
- soft routingSoft routing is a mixture-of-experts routing strategy that combines the outputs of multiple experts using continuous routing weights.
- sparse upcyclingSparse Upcycling is a method for converting a pretrained dense model into a sparse mixture-of-experts model, commonly by duplicating feed-forward components as experts and further training the resulting architecture.
- speculative decodingSpeculative decoding is a generation technique in which a smaller model proposes tokens that a larger model verifies, preserving the larger model’s output while potentially accelerating generation.
- top-k gatingTop-k routing selects the k experts with the highest router scores for each token and normalizes their selected weights.
- BBQBBQ (Bias Benchmark for QA) is a question-answering benchmark dataset designed to measure social biases in language models.
- forward methodA forward method is a neural-module method that defines the computation performed when input data passes through the module.
- GPT-2GPT2Model is a GPT-2 transformer model implementation that organizes embeddings, transformer blocks, normalization, and output projection.
- gradient accumulationGradient accumulation is a training technique that sums gradients across multiple batches before updating model parameters.
- domain-specific dataA domain-specific dataset is a collection of examples representing the terminology, patterns, or tasks of a particular domain.
- custom architectureA custom architecture is a model structure implemented outside the standard architectures supported by a serving engine.
- adjacent block protectionAdjacent block protection is a pruning constraint that prevents or limits the removal of neighboring model blocks.
- generation collapseGeneration collapse is a failure mode in which autoregressive text generation rapidly becomes incoherent, repetitive, or structurally degenerate.
- magnitude-based selectionMagnitude-based selection chooses model components for pruning according to the magnitudes of their parameters or related scores.
- heat mapA heatmap is a visualization that represents numerical values through variations in color across an arranged set of elements.
- sparse autoencoderA sparse autoencoder is an autoencoder that reconstructs neural activations through a wider intermediate representation constrained to activate sparsely.
- cascaded routingCascaded routing is a model-serving strategy that tries a less costly model first and escalates requests that fail specified criteria to a larger model.
- layer normalizationLayer normalization standardizes neural-network activations within each representation to stabilize computation.
- specializationDomain specialization is the adaptation of a general model to perform effectively on a particular domain or task.
- structural optimizationStructural optimization improves a model by changing its architecture, components, or organization.
- hybrid modeHybrid mode combines neuron weight information with activation statistics measured on calibration inputs when ranking neurons for pruning.
- peak-to-peak magnitude pruningPeak-to-peak magnitude pruning ranks neurons using the peak-to-peak magnitude of their activations or weights.
- residual connectionA residual connection adds a module's input representation to its transformed output so information can bypass the transformation.
- block selection strategyA block selection strategy is a method for choosing which transformer blocks to retain or remove during depth pruning.
- prompt engineeringPrompt engineering is the deliberate design and adjustment of instructions provided to a model to influence its outputs.
- BBQ bias scoreA BBQ bias score measures the extent to which a model's answers align with stereotypes rather than contradict them.
- capability retentionCapability retention is the proportion of a model's evaluated performance preserved after a structural modification such as pruning.
- percentileA percentile is a value below which a specified percentage of observations fall in an ordered distribution.
- dynamic importance componentDynamic importance is a measure of a model component's significance derived from its observed activations on representative inputs.
- logit-based distillationLogit distillation is a knowledge-distillation method in which a student learns from a teacher model's output scores.
- labels-only distillationLabels-only distillation trains a student model using the teacher’s output labels without additional intermediate-feature or distribution-matching signals.
- cosine importanceCosine importance estimates a model component's importance using cosine similarity between related activation or representation vectors.
- paddingPadding is the addition of extra elements to a computational dimension so its size meets a hardware or algorithmic alignment requirement.
- time to first tokenTime to first token is the elapsed time from a generation request until the model produces its first output token.
- vLLMvLLM is an open-source library and inference engine for efficient large language model serving.
- positional embeddingA positional embedding is a learned or fixed vector representation that supplies sequence-position information to a neural language model.
- input embeddingAn input embedding is a vector representation of a token that combines information about the token and its position before contextual processing.
- token embeddingA token embedding is a vector representation encoding the identity or learned meaning of a token independently of its surrounding sequence context.
- dynamic inferenceDynamic inference is model execution that makes input-dependent runtime decisions about which computations, tokens, or submodels to use.
- transformer architectureA transformer is a neural-network architecture that processes representations using attention and position-wise feed-forward components.
- loss maskingLabel masking excludes selected token positions from contributing to a model's training loss while leaving the input available to the model.
- fairness pruningFairness pruning is a model-modification technique that removes selected neurons to change unwanted behavioral tendencies while preserving performance on desired capabilities.
- semantic adaptationSemantic adaptation is model adaptation that changes how a model interprets or generates meaning rather than only enforcing a fixed output form.
- hybrid importanceHybrid importance is a neuron-selection technique that multiplies normalized weight-based structural importance by activation magnitude measured on calibration data.
- structural scoreA structural score is a neuron's weight-based importance obtained by summing normalized contributions from its associated projection matrices.
- block influenceBlock Influence is a metric defined as one minus the cosine similarity between a model block's input and output representations.
- continued pretrainingContinued pretraining is additional unsupervised training of a pretrained model on text or other pretraining data.
- Taylor approximationA Taylor approximation estimates the change in a model's loss caused by removing a component using a local approximation based on derivatives.
- bfloat16bfloat16 is a 16-bit floating-point format with a dynamic range similar to float32 and reduced precision and storage requirements.
- prepare_model_for_kbit_trainingprepare_model_for_kbit_training is a PEFT function that prepares a quantized model for gradient-based parameter-efficient training.
- quantized modelA quantized model is a neural model whose parameters or computational values are represented using reduced-precision numerical formats.
- SFTConfigSFTConfig is a configuration class for supervised fine-tuning settings such as epochs, learning rate, scheduling, and numerical precision.
- SFTTrainerSFTTrainer is a training component that performs supervised fine-tuning of language models using a configured dataset and training procedure.
- expert routingSparse routing processes each token with only a subset of available expert modules rather than with every expert.
- stratified splitA stratified split divides data into subsets while preserving the representation of relevant categories in each subset.
- exact-match accuracyExact match is a metric that counts outputs as correct only when they exactly match the reference under the specified comparison rules.
- model familyA model family is a set of related models arranged or designed to provide different capabilities or resource requirements.
- instruction followingInstruction following is the ability of a model to carry out requested operations while satisfying specified constraints.
- neuron pairA neuron pair is the corresponding pair of intermediate MLP neurons represented across the gated projection paths.
- contrastive prompt pairA paired-prompt method compares prompts that differ in one controlled attribute to isolate its effect on model behavior.
- draft modelA draft model proposes candidate tokens for a target model to verify during speculative decoding.
- target modelA target model verifies candidate tokens proposed by a draft model during speculative decoding.
- adapterAn adapter is a trainable model component that modifies a pretrained model's behavior while leaving most of the original parameters unchanged.
- content projectionA content projection is the linear transformation in a gated unit that produces the representation modulated by the gate.
- element-wise multiplicationElement-wise multiplication is the operation of multiplying corresponding entries of two equally sized vectors or arrays.
- multiplicative gateA multiplicative gate is a learned vector that scales another representation element by element to control information flow.
- structured tool argument extractionStructured tool argument extraction is the example application's task of converting free-form user intent into validated tool names and typed parameters.
- specialization setThe specialization set is the example dataset collected to adapt and evaluate the model for structured tool argument extraction.
- coreference resolutionCoreference resolution is the task of determining which previously mentioned entity a referring expression, such as a pronoun, denotes.
- attention sinkAn attention sink is a sequence position that accumulates unusually large activations and serves as a reference for other tokens without carrying corresponding semantic information.
- beginning-of-sequence tokenA BOS token is a beginning-of-sequence marker automatically inserted at the start of a tokenized sequence.
- SwishSiLU is a smooth activation function defined by multiplying an input by its sigmoid.
- neuronA neuron is a computational unit in a neural network that transforms inputs using learned parameters and a nonlinear activation.
- pruned modelA pruned model is a model from which selected parameters or structural components have been removed.
- PagedAttentionPagedAttention is an attention-memory management technique that stores key-value cache blocks in pages to improve memory utilization during inference.
- confidence thresholdA confidence threshold is a cutoff applied to a model's token probability to decide whether its output is sufficiently reliable.
- FDDFDD is an advanced knowledge-distillation technique that transfers feature-distribution information from a teacher model to a student model.
- domain-guided pruningDomain-guided pruning selects model components for removal or preservation according to their relevance to a target domain.
- generalist modelA generalist model is a model designed for broad capabilities across many tasks and domains rather than a narrow specialization.
- open-source modelAn open-source model is a model distributed with source or parameter artifacts under terms that permit specified forms of inspection, modification, or reuse.
- proprietary modelA closed model is a model whose provider controls access to its implementation, weights, or execution interface.
- retrieval-augmented generationRetrieval-augmented generation is a technique that retrieves external information at generation time and provides it to a language model as context.
- vendor lock-inVendor lock-in is a condition in which dependence on a provider makes switching to an alternative prohibitively costly or difficult.
- dimension alignmentDimension alignment is the adjustment of tensor or matrix dimensions to values that match hardware processing constraints efficiently.
- energy consumptionEnergy consumption is the quantity of electrical energy used by a computation or hardware system.
- energy efficiencyEnergy efficiency is the amount of energy required to perform a computational workload or produce a unit of model output.
- Tensor CoreA Tensor Core is a specialized NVIDIA GPU execution unit optimized for high-throughput matrix arithmetic.
- autoregressive generationAutoregressive generation produces a sequence one token at a time, conditioning each new token on previously generated tokens.
- memory boundMemory-bound describes a workload whose performance is limited primarily by memory transfers rather than arithmetic processing.
- repetition loopA repetition loop is a generation failure mode in which a language model repeatedly produces the same text pattern instead of progressing normally.
- prune_modelprune_model is a function that removes selected structural components from a neural model to produce a pruned model.
- reliabilityReliability is the consistency with which a model produces correct or constraint-compliant outputs across inputs.
- rankRank is a LoRA configuration hyperparameter that determines the dimensionality and expressive capacity of a low-rank update.
- break-even request rateA break-even request rate is the request volume at which the costs of self-hosting and using an external API are equal.
- self-hostingSelf-hosting is operating a model on infrastructure controlled by the organization deploying it rather than relying on an external service.
- feature dynamics distillationFeature dynamics distillation is a knowledge-distillation technique that aligns the changes in intermediate representations across consecutive layers of student and teacher models.
- optimizationAn optimization technique is a method for improving a model's efficiency or task performance through changes to its design, parameters, or configuration.
- structural importance metricA structural importance metric measures a model component's significance based on its parameter structure or contribution to behavior.
- batchingRequest batching is the processing of multiple inference requests together as a single batch.
- latencyLatency is elapsed time measured for a language model request or for producing an individual output token.
- valueA value vector is the attention representation combined according to attention weights to produce an attended output.
- routingRouting is the process of directing requests to a selected model or service according to a system’s rules.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.
