liveBook · Manning

Small-box tier

As explained in Quantization and Fast Inference

The small-box tier is the deployment class for sustained workloads on fixed installations with discrete inference accelerators.

The small-box tier targets throughput-bound, sustained workloads on fixed installations. Its deployment choice follows the installation’s power envelope and workload, not the model alone. section 11.4

Plan for vendor-specific runtimes and hardware limits:

  • Jetson uses TensorRT; Intel small boxes use OpenVINO IR with NNCF INT8 quantization; Coral uses the Edge TPU compiler; Hailo uses the Hailo Dataflow Compiler. section 11.4
  • Each compiler produces a non-portable artifact and has its own operator-support limits. Validate a conversion pipeline for each vendor. section 11.4
  • For a fixed 5–25 W installation with a Cortex-A78 host, use Coral or Hailo when the model fits the accelerator’s operator set; otherwise use the SBC CPU path. section 11.4

Calibration drift across vendors

Converting the same EfficientNet for Jetson, OpenVINO, Coral, and Hailo requires separate conversion scripts and runtime binaries. Vendor INT8 calibration can shift accuracy by 1–3 percentage points on the same source model, so calibrate and validate for each vendor. section 11.4 section 11.6

How Quantization and Fast Inference covers it

How deep it goes
Compares vendor compilers, operator limits, and per-vendor INT8 calibration drift without detailing conversion procedures.
Examples
Uses EfficientNet conversions across four vendors to illustrate accuracy shifts of 1–3 percentage points.
What you can do after
Apply vendor-specific conversion and calibration pipelines, then validate converted-model accuracy and performance.
How it connects
Connects tier selection to TensorRT, OpenVINO, INT8 quantization, Coral Edge TPU, and Hailo accelerators.

This explainer was generated by Manning's knowledge graph from the text of Quantization and Fast Inference. The section links point to the parts of the book it draws on.