liveBook · Manning

Inference performance

Inference performance measures how quickly and efficiently a model processes inputs and produces outputs, including latency, throughput, and resource use.

How 2 Manning books explain it

Rearchitecting LLMs

Inference speed is the rate at which a trained model produces outputs from inputs.

How it is used
Evaluates operational efficiency alongside capability when comparing optimized models.
How deep it goes
Describes timing and throughput comparisons, explains pruning mechanisms, and defers the inference evaluation system.
Examples
Compares baseline and width-pruned model throughput on WikiText and SMS prompts.
What you can do after
Evaluate inference efficiency against a baseline while accounting for capability loss and measurement variability.

Read how Rearchitecting LLMs explains inference performance →

Time Series Forecasting Using Foundation Models

Inference time is the duration required for a model to generate predictions from its input.

How it is used
Guides forecasting-model selection when prediction latency and available hardware constrain deployment choices.
How deep it goes
Measures elapsed prediction time across validation windows and compares it with error, but stops short of a complete benchmark.
Examples
Compares Seasonal Naive, SARIMA, TimeGPT, Chronos, Moirai, TimesFM, and Time-LLM on daily website-visitor forecasts.
What you can do after
Evaluate forecasting models by balancing prediction error against measured inference time and available computing resources.

Read how Time Series Forecasting Using Foundation Models explains inference performance →

Also covered in

What to know first

Related concepts

Questions about inference performance

What is inference performance?

Inference performance measures how quickly and efficiently a model processes inputs and produces outputs, including latency, throughput, and resource use.

Which Manning books explain inference performance?

Rearchitecting LLMs and Time Series Forecasting Using Foundation Models.

What should I know before learning inference performance?

Manning books that teach inference performance usually introduce language model, AI agent and fine-tuning first.

What is inference performance part of?

Inference performance is part of performance. Performance is a measure of how effectively a person, process, model, or system achieves its objectives, including qualities such as output quality, speed, and resource efficiency.

These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.