Hidden state
An internal condition or representation of a system that is not directly observable but can influence its outputs or subsequent processing.
Build AI Drug Discovery Pipelines
A hidden state is a learned intermediate vector that summarizes sequence information available to a recurrent or sequential model at a particular position.
- How it is used
- Uses hidden states to carry sequence information through molecular generation and to motivate alternatives for long protein sequences.
- How deep it goes
- Works through a simple RNN update and GRU gating, then contrasts final-state compression with attention over encoder states.
- Examples
- Illustrates state updates with acetic acid SMILES and a 500-amino-acid protein sequence.
- What you can do after
- Builds a simple RNN step from scratch using PyTorch tensor operations and interprets its hidden-state update.
Read how Build AI Drug Discovery Pipelines explains hidden state →
Rearchitecting LLMs
A hidden state is an internal vector representation produced by a neural model layer for the information processed up to that layer.
- How it is used
- Uses intermediate representations for distillation, expert routing, and inspection of predictions across transformer layers.
- How deep it goes
- Explains hidden-state extraction, tensor shapes, layer alignment requirements, and logit-lens projection.
- Examples
- Uses a 14-block student and 18-block teacher for distillation, and a Barcelona prompt to inspect layer predictions.
- What you can do after
- Diagnose predictions with a logit lens and apply hidden-state alignment during knowledge distillation.
Machines that Think
An internal recurrent-network representation that carries information from earlier sequence elements to later processing steps.
- How it is used
- Explains how sequence models retain context and how attention-based models form contextualized token representations.
- How deep it goes
- Explains state updates, shared weights and memory limitations, without specifying a full implementation.
- Examples
- Illustrates information loss across intervening phrases and contextualization of “rises” using information from “sun.”
- What you can do after
- Evaluate how a model carries or recomputes context and why recurrent memory can become difficult to use over long spans.
Timeless Algorithms, The Foundational Ideas
A hidden state is an internal representation that a recurrent neural network carries forward to summarize information from earlier sequence elements.
- How it is used
- Summarizes earlier sequence inputs so recurrent predictions can depend on context beyond the current element.
- How deep it goes
- Explains fixed-size state updates, information loss, and sequential computation at the architectural level.
- Examples
- Uses a sentence where “book” must remain linked to “was” across intervening clauses.
- What you can do after
- Evaluate how recurrent state updates can weaken distant context and constrain parallel sequence processing.
Read how Timeless Algorithms, The Foundational Ideas explains hidden state →
Also covered in
- Graph Neural Networks in Action
- Understanding AIIn early access (MEAP)
- Data Without Labels
- Grokking BayesIn early access (MEAP)
What to know first
- recurrent neural networkA recurrent neural network is a neural network that processes ordered inputs while carrying and updating an internal state across successive steps.
- tokenA token is a discrete unit of text, such as a word, subword, punctuation mark, or symbol, produced by tokenization and processed by a language model.
- foundation modelA foundation model is a broadly trained machine-learning model whose learned capabilities can be reused or adapted for a range of downstream tasks.
- LSTMA long short-term memory (LSTM) network is a recurrent neural-network architecture with gated memory cells that control information flow and model dependencies in sequential data.
Related concepts
- recurrent neural networkused by · A recurrent neural network is a neural network that processes ordered inputs while carrying and updating an internal state across successive steps.
- ELMoused by · ELMo (Embeddings from Language Models) is a deep contextualized word-representation model that generates context-dependent embeddings from a bidirectional language model.
- logit lensused by · The logit lens is an interpretability technique that projects intermediate hidden states through a model's final normalization and prediction head to inspect layer-wise predictions.
- statekind of · State is information describing the condition or configuration of a system or computation at a particular time that may influence its behavior or subsequent transitions.
- distillationused by · Distillation is a training technique in which a student model learns to approximate the outputs or behavior of a teacher model.
- feature alignmentused by · Feature alignment is a distillation technique that matches intermediate representations produced by corresponding blocks of teacher and student models.
Questions about hidden state
What is hidden state?
An internal condition or representation of a system that is not directly observable but can influence its outputs or subsequent processing.
Which Manning books explain hidden state?
Build AI Drug Discovery Pipelines, Rearchitecting LLMs, Machines that Think and Timeless Algorithms, The Foundational Ideas.
What should I know before learning hidden state?
Manning books that teach hidden state usually introduce recurrent neural network, token, foundation model and LSTM first.
What is hidden state a kind of?
Hidden state is a kind of state. State is information describing the condition or configuration of a system or computation at a particular time that may influence its behavior or subsequent transitions.
What is hidden state also called?
Hidden state is also called internal state.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.



