Autoregressive model
An autoregressive model represents or predicts each value in a sequence based on preceding values in that sequence.
Build a Text-to-Image Generator (from Scratch)
A process that produces an ordered output sequence one element at a time, conditioning each element on previously generated elements and available conditioning information.
- How it is used
- Generates translations, image captions, and image tokens by repeatedly predicting the next token from prior output and encoded input.
- How deep it goes
- Traces start tokens, iterative prediction, stopping, and a 256-step decoder loop, including state updates and sampling settings.
- Examples
- Uses German-to-English translation, a skier caption, and intermediate images from a text-to-image prompt.
- What you can do after
- Apply it via a transformer decoder to generate output token by token and inspect saved stages of image generation.
Read how Build a Text-to-Image Generator (from Scratch) explains autoregressive model →
Statistics Every Programmer Needs
An autoregressive component models the current value of a time series as a linear function of specified previous values.
- How it is used
- Supports time-series forecasting by relating current observations to their own historical values.
- How deep it goes
- Explains lagged relationships, stationarity preparation, ACF/PACF selection, and AR(1), but defers full ARIMA estimation details.
- Examples
- Uses Apple daily closing prices, first-order differences, ACF/PACF plots, and an actual-versus-forecast comparison.
- What you can do after
- Builds an ARIMA forecasting workflow by preparing data, selecting an AR order, fitting the model, and forecasting held-out observations.
Read how Statistics Every Programmer Needs explains autoregressive model →
Build AI Drug Discovery Pipelines
Autoregressive generation produces a sequence one element at a time, conditioning each prediction on previously generated elements.
- How it is used
- Generates novel molecular SMILES strings from sampled latent vectors in a variational autoencoder pipeline.
- How deep it goes
- Explains token-by-token decoding, decoder directionality, teacher forcing, exposure bias, dropout, and sampling strategies.
- Examples
- Uses SMILES reconstruction and novel-molecule generation with greedy decoding or temperature sampling.
- What you can do after
- Build it from scratch with a unidirectional GRU decoder and generate molecules using greedy or temperature-based sampling.
Read how Build AI Drug Discovery Pipelines explains autoregressive model →
Build a DeepSeek Model (From Scratch)
Autoregressive generation is a text-generation process that predicts one token at a time and feeds each generated token back into the context for the next prediction.
- How it is used
- Motivates KV caching by exposing repeated work in token-by-token inference.
- How deep it goes
- Traces generation through a concrete loop, repeated attention matrices, and a per-layer, per-head caching workflow.
- Examples
- Traces “The next day” through successive predictions and demonstrates token generation with pretrained GPT-2.
- What you can do after
- Apply it via PyTorch and Transformers to generate tokens with pretrained GPT-2.
Read how Build a DeepSeek Model (From Scratch) explains autoregressive model →
Build an Advanced RAG Application (From Scratch)
An autoregressive model generates each successive output element conditioned on previously generated elements.
- How it is used
- Uses causal next-token generation to produce coherent responses and support retrieval-augmented generation.
- How deep it goes
- Explains token-level conditioning, sequence-probability factorization, decoder processing, and decoding choices without detailing training internals.
- Examples
- Uses next-word completion, decoder-only Transformer processing, and a hotel-search system that analyzes retrieved results.
- What you can do after
- Evaluate/understand how decoder models generate context-sensitive sequences and how decoding choices affect their outputs.
Read how Build an Advanced RAG Application (From Scratch) explains autoregressive model →
The Art of AI Product Development
An autoregressive language model generates each token by predicting it from the preceding context.
- How it is used
- Explains the basic language-modeling objective behind sequential text generation.
- How deep it goes
- Defines next-token prediction and its directional limitation, but stops short of implementation details.
- Examples
- Uses sentence completion, Mistral, ChatGPT, and context illustrations to contrast raw prediction with conversational behavior.
- What you can do after
- Evaluate how next-token objectives support generation while limiting context and conversational usefulness.
Read how The Art of AI Product Development explains autoregressive model →
Build a Robo-Advisor with Python (From Scratch)
A time-series model in which the current value depends linearly on one or more previous values and a random error term.
- How it is used
- Models persistent, mean-reverting inflation within retirement Monte Carlo simulations.
- How deep it goes
- Estimates one-lag parameters with regression and implements simulated paths, stopping at the AR(1) case.
- Examples
- Uses CPI-U data, lagged-inflation regression, residual volatility, and simulated thirty-year wealth paths.
- What you can do after
- Build an AR(1) inflation component and insert it into a retirement Monte Carlo simulation.
Read how Build a Robo-Advisor with Python (From Scratch) explains autoregressive model →
Introduction to Generative AI, Second Edition
Autoregressive generation produces a sequence by predicting each successive element from previously generated elements and available context.
- How it is used
- Generates coherent sequences by predicting successive tokens from preceding context.
- How deep it goes
- Explains the prediction task, training intuition, and bidirectional contrast without descending into implementation details.
- Examples
- Uses a honeymoon-completion prompt and identifies GPT-family models and PaLM as autoregressive examples.
- What you can do after
- Evaluate/understand why generative models use left-context next-token prediction.
Read how Introduction to Generative AI, Second Edition explains autoregressive model →
Also covered in
- Fight Fraud with Machine LearningIn early access (MEAP)
- How Large Language Models Work
- Time Series Forecasting Using Foundation Models
- Deep Learning with R, Third Edition
- Data Without Labels
- Causal AI
- Natural Language Processing in Action, Second Edition
- Build AI-Enhanced Web Apps
What to know first
- language modelA language model is a computational model that represents patterns in linguistic sequences to assign probabilities, predict, generate, or evaluate language.
- generationGeneration is the production of an output by a computational model from supplied input or context, often using patterns learned from data.
- attentionAttention is a neural-network mechanism that computes input-dependent relevance weights to combine information from representations for a context-dependent computation.
- transformer architectureA transformer is a neural-network architecture that uses attention-based interactions and position-wise transformations to model relationships among elements of sequences or other structured inputs.
Related concepts
- bidirectional language modelcontrasts with · A bidirectional language model predicts or represents tokens using contextual information from both preceding and following tokens in a sequence.
- GPT-4includes · GPT-4 is a multimodal large language model developed by OpenAI that processes text and image inputs and generates text.
- arimapart of · ARIMA is a statistical time-series model that combines autoregressive terms, differencing, and moving-average terms to represent temporal dependence and generate forecasts.
- decoding strategyincludes · A decoding strategy is a procedure for converting an encoded or internal representation into a target output, including selecting tokens during sequence generation.
- forward passuses · A forward pass is a computation that propagates inputs through a model or dependency structure in the forward direction to calculate outputs or intermediate values.
Questions about autoregressive model
What is autoregressive model?
An autoregressive model represents or predicts each value in a sequence based on preceding values in that sequence.
Which Manning books explain autoregressive model?
Build a Text-to-Image Generator (from Scratch), Statistics Every Programmer Needs, Build AI Drug Discovery Pipelines, Build a DeepSeek Model (From Scratch), Build an Advanced RAG Application (From Scratch), The Art of AI Product Development, Build a Robo-Advisor with Python (From Scratch) and Introduction to Generative AI, Second Edition.
What should I know before learning autoregressive model?
Manning books that teach autoregressive model usually introduce language model, generation, attention and transformer architecture first.
What is the difference between autoregressive model and bidirectional language model?
An autoregressive model represents or predicts each value in a sequence based on preceding values in that sequence. A bidirectional language model predicts or represents tokens using contextual information from both preceding and following tokens in a sequence.
What kinds of autoregressive model are there?
Kinds of autoregressive model include GPT-4 and decoding strategy.
What is autoregressive model part of?
Autoregressive model is part of arima. ARIMA is a statistical time-series model that combines autoregressive terms, differencing, and moving-average terms to represent temporal dependence and generate forecasts.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.







