liveBook · Manning

Deployment surface

As explained in Quantization and Fast Inference

A deployment surface is the target device and operating environment on which a model is delivered and executed.

A deployment surface—the target device and operating environment—sets which inference tier and runtime are viable. Choose the tier from the hardware and workload, not from the model alone. section 11.6

Match the deployment surface to the available execution path:

  • Phones use platform paths such as TFLite or LiteRT-LM, Core ML, and MLX. section 11.6
  • Small Linux boards commonly rely on LiteRT with XNNPACK on the ARM CPU; Coral or Hailo accelerators add their own operator-support limits. section 11.4
  • Fixed, throughput-bound installations can use a small-box tier and vendor SDK, trading flexibility for throughput and accepting calibration drift. section 11.6

An attached accelerator does not guarantee that the whole graph runs on it: unsupported operators can force CPU fallback and reduce throughput. Measure end-to-end latency, since preprocessing and postprocessing can outweigh inference. section 11.4 section 11.6

How Quantization and Fast Inference covers it

How deep it goes
Compares phone, SBC, and fixed-appliance paths through runtime availability, operator support, and deployment tradeoffs.
Examples
Uses consumer phones, Raspberry Pi and Rockchip boards, Coral and Hailo accelerators, and Jetson Orin Nano Super.
What you can do after
Evaluate a target device and route it to a suitable inference tier, runtime, and accelerator path.
How it connects
Connects model deployment to inference runtimes, operator support, quantization, and end-to-end latency.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Quantization and Fast Inference. The section links point to the parts of the book it draws on.