Inference framework

As explained in Machine Learning System Design

Software that supports model serving and inference through an engine plus associated conversion, optimization, and operational components.

An inference framework supports production model serving through an inference engine and related conversion, optimization, and operational components. Separating training and inference frameworks can preserve research flexibility while keeping long-lived production systems replaceable and maintainable. section 15.3.1

Choose the arrangement that fits the system’s priorities:

• Use separate frameworks when research tools such as Pandas, scikit-learn, or Keras provide flexibility but are not suited to production inference. section 15.3.1
• Use one framework when its production capabilities close the research-to-serving gap; Torch 2.0 compilation is one example. section 15.3.1
• Use ONNX as an interframework language: its ecosystem includes model conversion, optimization tools, and ONNX Runtime for inference across varied backends. section 15.3.1

Figure 15.3 A wide spectrum of tools with their focus on either research or production serving goals

Compare tools positioned toward research flexibility, production serving, or the space between them. section 15.3.1

How Machine Learning System Design covers it

How deep it goes

Explains framework separation, unified compilation, and ONNX conversion and runtime, but stops short of implementation procedures.

Examples

Uses Pandas, scikit-learn, Keras, Torch 2.0 compilation, ONNX, and ONNX Runtime as contrasting framework examples.

What it assumes

Assumes familiarity with model training and inference, while clarifying why their frameworks may be separated.

What you can do after

Evaluate framework architectures and convert models through ONNX for portable, optimized inference deployment.

How it connects

Connects training frameworks, model conversion, model optimization, ONNX, ONNX Runtime, and production serving.

liveBookmanning.com

Read Machine Learning System Design on liveBook Get the book at manning.com

Related concepts

• frameworkcontrasts with · A software framework is a reusable structure that provides components, conventions, services, or extension points for building applications or other software systems.
• inference enginehas part · An inference engine is software that derives outputs or conclusions from inputs by executing a trained model or evaluating facts and rules.

This explainer was generated by Manning's knowledge graph from the text of Machine Learning System Design. The section links point to the parts of the book it draws on.