liveBook · Manning

ML serving engine

As explained in Delivering Machine Learning Projects: From design to deployment

An ML serving engine is software specialized for executing machine learning models in an operational environment.

An ML serving engine is software specialized for executing machine learning models in operational environments. It manages model invocation and may optimize execution for processor instruction sets. section 9.4.2

Choose the execution surface alongside the data flow and model-hydration process. Treat pricing, latency, scale, modularity, user controls, and an exit path as production decisions rather than implementation details. section 9.4.2

Common execution choices include:

  • Use specialist engines such as OpenVINO or TensorFlow Serving for optimized model execution. section 9.4.2
  • Use database functions for modular, decoupled execution when application constraints are acceptable. section 9.4.2
  • Use application servers for low-scale, permissive-latency and permissive-cost workloads. section 9.4.2
  • Use Spark-like systems, Kubernetes, Kubeflow, or serverless platforms according to processing, scalability, cost, and modularity needs. section 9.4.2
Table 9.5 Using generic execution engines for ML models (view table figure)

Engine

Comments

In a database:

BigQuery ML, Redshift ML, Oracle OCI, Vertica PMML

Execution in database functions means accepting constraints on how the application runs, but this is a modular and decoupled approach.

Application servers:

NGINX, Apache Tomcat, Flask, Appserver

For generic low-scale and permissive latency/cost use cases. Consider because of the low barrier in terms of skills and setup requirements.

SPARK-like solutions:

Apache SPARK, Databricks, Dataproc

Good support for bulk and stream processing with permissive cost and latency requirements.

Kubernetes, Kubeflow

Flexible and scalable support for interactive applications with permissive cost requirements.

Serverless:

Lambda, Cloud Functions, Azure Functions, Cloud Run.

A flexible, modular, and decoupled approach that’s cost effective and performative in some applications but can become expensive in unpredictable ways.

Compare generic execution engines by deployment surface, workload scale, latency tolerance, cost, flexibility, and processing mode. section 9.4.2

How Delivering Machine Learning Projects: From design to deployment covers it

How deep it goes
Compares execution surfaces and optimization goals but stops short of implementation procedures or engine internals.
Examples
Uses OpenVINO, TensorFlow Serving, BigQuery ML, Apache Spark, Kubernetes, Kubeflow, and serverless platforms as deployment examples.
What it assumes
Assumes decisions about data flow, model hydration, execution surfaces, and required user controls.
What you can do after
Evaluate execution options against workload scale, latency, cost, flexibility, processing mode, and pricing risk.
How it connects
Connects model execution with inference services, serverless computing, data-flow processing, model hydration, and processor instruction sets.
Cover of Delivering Machine Learning Projects: From design to deployment
liveBookmanning.com
Read Delivering Machine Learning Projects: From design to deployment on liveBook Get the book at manning.com

This explainer was generated by Manning's knowledge graph from the text of Delivering Machine Learning Projects: From design to deployment. The section links point to the parts of the book it draws on.