liveBook · Manning

Routing

As explained in Machine Learning System Design

Routing between models is a pattern that selects among models according to the characteristics or difficulty of each input.

Routing selects a model according to how difficult an input is: simpler queries can go to cheaper models, while harder ones use more advanced, costly models. It is used mainly for general LLMs and other natural-language tasks, including machine translation. section 15.2.2

A related pattern trades compute efficiency for a faster first response:

  • A fast, imperfect model responds first; a slower, more accurate model later replaces its output. This improves time to initial response, but does not reduce total compute. section 15.2.2

How Machine Learning System Design covers it

How deep it goes
Distinguishes difficulty-based selection from a two-model latency variation without detailing routing implementation.
Examples
Illustrates a fast, imperfect model followed by a slower, more accurate model.
What you can do after
Evaluate routing tradeoffs by distinguishing compute savings from faster time to initial response.
How it connects
Links model routing to mixture of experts, large language models, and machine translation.
Cover of Machine Learning System Design
liveBookmanning.com
Read Machine Learning System Design on liveBook Get the book at manning.com

Also explained in

Compare how 22 books explain routing →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Machine Learning System Design. The section links point to the parts of the book it draws on.