liveBook · Manning

Inference engine

An inference engine is software that derives outputs or conclusions by executing a trained machine-learning model or evaluating facts and rules.

How 5 Manning books explain it

Inside AI

An inference engine is a program that evaluates known facts against rules to derive or modify facts and control an expert system's reasoning loop.

How it is used
Executes rule-based reasoning for expert systems and automates decisions or actions within business processes.
How deep it goes
Explains the evaluation loop, fact updates, stopping condition, and supporting rule-management components without implementation details.
Examples
Uses MYCIN’s infection diagnosis and an industrial emergency-pump rule to demonstrate fact-driven inference.
What you can do after
Operate/configure rule-based decision systems by understanding fact evaluation, rule triggering, and maintainable rule separation.

Read how Inside AI explains inference engine →

Machine Learning System Design

An inference engine is software that loads a machine-learning model and executes it to produce predictions from inputs.

How it is used
Deploys machine-learning models for prediction while balancing execution hardware, latency, cost, and operational constraints.
How deep it goes
Explains graph execution, hardware placement, optimization boundaries, and serverless cold starts without deriving engine internals.
Examples
Uses CoreML graph splitting, ONNX Runtime optimization, specialized LLM engines, and a quantified serverless cold-start scenario.
What you can do after
Evaluate inference engines against hardware placement, model characteristics, optimization needs, latency, scaling, and deployment cost.

Read how Machine Learning System Design explains inference engine →

Self-Hosting LLMs at Scale

An inference engine is software that loads a trained machine-learning model and computes outputs for input requests.

How it is used
Places the engine at the center of a self-hosted language-model serving platform.
How deep it goes
Defines engine responsibilities and sketches request flow, while deferring detailed vLLM internals.
Examples
Compares vLLM for concurrent users with llama.cpp, Ollama, and LM Studio for single-machine use.
What you can do after
Evaluate and understand engine choices against single-machine and concurrent-user workloads.

Read how Self-Hosting LLMs at Scale explains inference engine →

AI Governance

An inference engine is software that loads and serves trained machine-learning models to generate predictions or outputs.

How it is used
Assesses inference engines as trusted components in GenAI supply-chain risk.
How deep it goes
Names inference engines as supply-chain components but gives no engine-specific implementation details or controls.
What you can do after
Understand how a compromised inference engine could provide trusted access to data or networks without attacking the model.

Read how AI Governance explains inference engine →

Natural Language Processing in Action, Second Edition

An inference engine is a computational system that derives conclusions from facts and rules and can use them to support decisions.

How it is used
Supports chatbot decisions by combining extracted information with domain knowledge and logical extrapolation.
How deep it goes
Defines inference and places it within an NLP pipeline, but stops short of implementation details for the engine itself.
Examples
Uses chatbot pipelines, extracted statements, stored statement–response pairs, and edit-distance matching as concrete examples.
What you can do after
Evaluate how a chatbot can combine extracted information and domain knowledge to produce logically supported decisions.

Read how Natural Language Processing in Action, Second Edition explains inference engine →

Also covered in

What to know first

Related concepts

Questions about inference engine

What is inference engine?

An inference engine is software that derives outputs or conclusions by executing a trained machine-learning model or evaluating facts and rules.

Which Manning books explain inference engine?

Inside AI, Machine Learning System Design, Self-Hosting LLMs at Scale, AI Governance and Natural Language Processing in Action, Second Edition.

What should I know before learning inference engine?

Manning books that teach inference engine usually introduce language model first.

What kinds of inference engine are there?

Kinds of inference engine include vLLM, SGLang and TGI.

What is inference engine part of?

Inference engine is part of business rules management system. A business rules management system is software for authoring, managing, governing, deploying, and executing business rules separately from application code.

These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.