liveBook · Manning

Tier selection

As explained in Quantization and Fast Inference

Tier selection is the choice of deployment class based on the constraints and capabilities of the target deployment surface.

Tier selection chooses the deployment surface before runtime and model-variant choices. The same model can target structurally different hardware, each with its own runtime, conversion script, and portability constraints. section 11.4

Distinguish tiers by their hardware and runtime constraints:

  • Phones bundle a CPU, GPU, and integrated NPU on one die. section 11.4
  • Single-board computers generally lack an integrated NPU, though RK3588 boards are an exception; attached accelerators are an option. section 11.4
  • Small-box appliances use a discrete inference accelerator and replace the runtime stack. section 11.4

How Quantization and Fast Inference covers it

How deep it goes
Distinguishes phone, SBC, and small-box hardware and runtime constraints, without detailing tier-specific conversion procedures.
Examples
Uses phones, Raspberry Pi and other SBCs, and small-box appliances to illustrate deployment surfaces.
What you can do after
Evaluate deployment surfaces before selecting a runtime or model variant.
How it connects
Connects deployment surfaces to runtimes, model variants, conversion scripts, and portability matrices.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Quantization and Fast Inference. The section links point to the parts of the book it draws on.