Model optimization

As explained in Machine Learning System Design

Model optimization is the improvement of a trained model's execution efficiency or resource requirements while preserving acceptable predictive behavior.

Model optimization improves inference efficiency by matching operations to suitable hardware and simplifying the execution graph. A GPU may handle matrix multiplication while a CPU handles control flow or sparse data. section 15.3.1

Practical optimization choices include:

• Choose an inference framework according to the target device and serving constraints. section 15.3.1
• Apply separate preparation optimizers to remove training-only graph components and fuse operations. section 15.3.1
• Treat large-language-model inference separately when memory footprint and autoregressive generation limit parallelization. section 15.3.1

Mixed-device execution

CoreML can dynamically split an execution graph across the CPU, GPU, and Apple Neural Engine to maximize efficiency rather than assigning every operation to one device. section 15.3.1

Figure 15.3 A wide spectrum of tools with their focus on either research or production serving goals

Use the spectrum to compare tools oriented toward research workflows with those designed for production serving. section 15.3.1

How Machine Learning System Design covers it

How deep it goes

Explains hardware placement, graph trimming, operation fusion, and LLM constraints, but stops short of implementation tutorials.

Examples

Uses CoreML device splitting, ONNX Runtime graph optimization, and VLLM, TGI, and GGML as inference-tool examples.

What you can do after

Operate/configure inference tools by selecting frameworks and preparing graphs for target hardware.

How it connects

Connects inference engines with GPUs, CPUs, graph optimization, CoreML, ONNX Runtime, and large-language-model serving.

liveBookmanning.com

Read Machine Learning System Design on liveBook Get the book at manning.com

Also explained in

• 

  liveBookmanning.com
  Deep Learning with R, Third EditionModel optimization is the modification of a trained model to reduce computational, memory, or latency costs while preserving acceptable predictive performance.

Compare how 2 books explain model optimization →

Related concepts

• pruningincludes · Pruning is the selective removal or restriction of parts of a model, structure, or candidate set, typically to reduce complexity or computation while retaining useful results.
• deploymentconfigures · Deployment is the process of making software, models, or other computational systems available to operate in a target environment.
• quantizationincludes · Quantization maps numerical values to a finite set of discrete representations, often using reduced precision to lower model storage or computation requirements.
• distillationincludes · Knowledge distillation is a training technique in which a student model learns to reproduce or approximate the outputs or behavior of a teacher model.
• ONNXaddressed by · ONNX is an open standard for representing machine-learning models in a portable format that supports exchange and execution across frameworks and runtimes.

This explainer was generated by Manning's knowledge graph from the text of Machine Learning System Design. The section links point to the parts of the book it draws on.