Inference performance
Inference performance measures how quickly and efficiently a model processes inputs and produces outputs, including latency, throughput, and resource use.
Rearchitecting LLMs
Inference speed is the rate at which a trained model produces outputs from inputs.
- How it is used
- Evaluates operational efficiency alongside capability when comparing optimized models.
- How deep it goes
- Describes timing and throughput comparisons, explains pruning mechanisms, and defers the inference evaluation system.
- Examples
- Compares baseline and width-pruned model throughput on WikiText and SMS prompts.
- What you can do after
- Evaluate inference efficiency against a baseline while accounting for capability loss and measurement variability.
Read how Rearchitecting LLMs explains inference performance →
Time Series Forecasting Using Foundation Models
Inference time is the duration required for a model to generate predictions from its input.
- How it is used
- Guides forecasting-model selection when prediction latency and available hardware constrain deployment choices.
- How deep it goes
- Measures elapsed prediction time across validation windows and compares it with error, but stops short of a complete benchmark.
- Examples
- Compares Seasonal Naive, SARIMA, TimeGPT, Chronos, Moirai, TimesFM, and Time-LLM on daily website-visitor forecasts.
- What you can do after
- Evaluate forecasting models by balancing prediction error against measured inference time and available computing resources.
Read how Time Series Forecasting Using Foundation Models explains inference performance →
Also covered in
What to know first
- language modelA language model is a computational model that represents patterns in linguistic sequences to assign probabilities, predict, generate, or evaluate language.
- AI agentAn AI agent is a computational system that uses artificial intelligence to interpret information and select or perform actions toward goals while interacting with an environment.
- fine-tuningFine-tuning is the continued training of a pretrained model on targeted data or objectives to adapt its parameters or behavior to a task, domain, or use.
Related concepts
- delivery cadenceconfigures · Delivery cadence is the recurring frequency and timing pattern at which outputs, updates, or activities are delivered or carried out.
- performancepart of · Performance is a measure of how effectively a person, process, model, or system achieves its objectives, including qualities such as output quality, speed, and resource efficiency.
- language modelmeasures · A language model is a computational model that represents patterns in linguistic sequences to assign probabilities, predict, generate, or evaluate language.
- depth pruningproduced by · Depth pruning is a model-reduction technique that removes complete layers from a neural network, often targeting later layers.
- latencymeasured by · Latency is the elapsed time between initiating an operation or request and receiving its corresponding result or observable response.
Questions about inference performance
What is inference performance?
Inference performance measures how quickly and efficiently a model processes inputs and produces outputs, including latency, throughput, and resource use.
Which Manning books explain inference performance?
Rearchitecting LLMs and Time Series Forecasting Using Foundation Models.
What should I know before learning inference performance?
Manning books that teach inference performance usually introduce language model, AI agent and fine-tuning first.
What is inference performance part of?
Inference performance is part of performance. Performance is a measure of how effectively a person, process, model, or system achieves its objectives, including qualities such as output quality, speed, and resource efficiency.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.

