Activation
An activation is a numerical value or tensor produced by a neural-network unit, layer, or operation during a forward computation.
Quantization and Fast Inference
A numerical output produced by a neural-network layer or operation and passed to subsequent computation.
- How it is used
- Distinguishes activations from weights and KV cache when choosing quantization targets and precision.
- How deep it goes
- Works through affine calibration, quantization, linear-layer zero-point correction, and output error; introduces granularity choices without resolving them.
- Examples
- Compares ReLU activation resolution and tests shifted inputs with and without zero-point correction.
- What you can do after
- Evaluate activation quantization using tensor ranges, integer-grid resolution, and output error against an FP32 baseline.
Read how Quantization and Fast Inference explains activation →
Rearchitecting LLMs
An activation is a value produced by a neural-model unit during computation.
- How it is used
- Uses activations to connect inference behavior with explainability and behavior-driven model optimization.
- How deep it goes
- Defines activations and develops capture, visualization, and cross-model validation workflows.
- Examples
- Uses a four-token Llama tensor, binned heatmaps, and demographic prompt pairs.
- What you can do after
- Diagnose model behavior with PyTorch hooks, activation heatmaps, and logit lens without training.
Human-in-the-Loop Machine Learning: Active learning and annotation for human-centered AI
A neural network activation is the output value of a unit or layer during a forward computation.
- How it is used
- Uses activations to find model-based outliers for diversity sampling.
- How deep it goes
- Explains layer selection, activation extraction, and rank-based comparison, including limitations of absolute values.
- Examples
- Illustrates ranked neuron values for ten items and a forward function that returns hidden activations and logits.
- What you can do after
- Apply it via PyTorch to expose layer outputs and rank low-activation candidates for outlier sampling.
Deep Learning with Python, Third Edition
The numeric value of a model activation.
- How it is used
- Uses activations to inspect how successive ConvNet layers transform images and to interpret learned filters.
- How deep it goes
- Defines activation tensors, visualizes individual channels, and optimizes filter responses, but does not survey every interpretation technique.
- Examples
- Uses a cat image, a sixth-channel edge visualization, complete activation grids, and optimized Xception filter patterns.
- What you can do after
- Diagnose/debug with it: extract, inspect, visualize, and optimize ConvNet activations to understand learned visual responses.
Read how Deep Learning with Python, Third Edition explains activation →
Natural Language Processing in Action, Second Edition
A hidden layer activation is the numerical output produced by a neural-network unit or layer for a particular input state.
- How it is used
- Uses activations to carry an RNN’s accumulated interpretation of sequential text into processing subsequent tokens.
- How deep it goes
- Explains storage, sequential overwriting, and compact representation, but does not detail the underlying neuron equations.
- Examples
- Uses token sequences, evolving text encodings, and a paint-smudge analogy to illustrate activation values.
- What you can do after
- Evaluate how an RNN activation changes as each token contributes to the sequence representation.
Read how Natural Language Processing in Action, Second Edition explains activation →
Deep Learning with R, Third Edition
An activation is the numerical output produced by a neural-network unit or layer for a given input.
- How it is used
- Uses activations to inspect how a convolutional network decomposes an input into learned features.
- How deep it goes
- Defines activations, extracts layer outputs, examines tensor shapes, and visualizes channels as feature maps.
- Examples
- Uses a held-out cat image, a 178 × 178 × 32 first-layer activation, individual channels, and complete activation grids.
- What you can do after
- Diagnose and interpret convolutional representations by extracting, inspecting, and plotting intermediate activations with Keras.
Read how Deep Learning with R, Third Edition explains activation →
Also covered in
- Build a DeepSeek Model (From Scratch)In early access (MEAP)
- Math and Architectures of Deep Learning
- LLMs in Production
- Build AI Drug Discovery PipelinesIn early access (MEAP)
What to know first
- adamAdam is an adaptive stochastic-gradient optimization algorithm that updates parameters using exponential moving estimates of the first and second moments of gradients.
- backpropagationBackpropagation is reverse-mode differentiation that propagates derivatives backward through a computational graph to compute gradients of an output or loss with respect to inputs and parameters.
- gradientA gradient is the vector of partial derivatives of a scalar function with respect to its variables, indicating its local rate and direction of change.
- language modelA language model is a computational model that represents patterns in linguistic sequences to assign probabilities, predict, generate, or evaluate language.
Related concepts
- AWQused by · AWQ is a post-training weight-quantization method that uses activation statistics to identify and protect salient weights, reducing model-size and precision while limiting quality loss.
- activation functionproduced by · An activation function transforms a neural-network unit’s input into its output, often introducing nonlinearity.
- l2 normmeasured by · The L2 norm of a vector is its Euclidean length, calculated as the square root of the sum of the squares of its components.
- hookproduced by · A hook is a software extension point that triggers a callback or action at a specified event, lifecycle point, or stage of execution.
- rank orderused by · A rank order is an arrangement of items according to their relative positions or assessed value, quality, importance, or preference.
- sparse autoencoderused by · A sparse autoencoder is an autoencoder trained to produce representations in which most encoded units are inactive or near zero.
Questions about activation
What is activation?
An activation is a numerical value or tensor produced by a neural-network unit, layer, or operation during a forward computation.
Which Manning books explain activation?
Quantization and Fast Inference, Rearchitecting LLMs, Human-in-the-Loop Machine Learning: Active learning and annotation for human-centered AI, Deep Learning with Python, Third Edition, Natural Language Processing in Action, Second Edition and Deep Learning with R, Third Edition.
What should I know before learning activation?
Manning books that teach activation usually introduce adam, backpropagation, gradient and language model first.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.





