liveBook · Manning

Model selection

As explained in Designing AI Systems

Model selection is the process of choosing an AI model for a task based on criteria such as quality, cost, latency, or reliability.

Model selection turns production evidence into routing decisions. Quality, cost, latency, reliability, error rates, token usage, and cache behavior reveal which model or provider fits each workflow. section 3.8.3

Apply model selection through measurable operational criteria:

  • Route classification tasks to a lower-cost model when it delivers comparable quality. section 3.8.3
  • Adjust provider routing when rate limits or workflow-specific error rates make the primary choice unreliable. section 3.8.3
  • Configure embedding model choice at the index level so every vector remains comparable. section 5.3.6
  • Balance embedding dimensionality against retrieval quality, storage cost, and search latency. section 5.3.6

Evidence-based classification routing

If metrics show that Claude Sonnet matches GPT-4 quality for classification while costing 60% less, route classification requests to Claude and continue monitoring the result. section 3.8.3

Listing 5.12 Embedding generation through the Model Service
class EmbeddingGenerator:
    def __init__(self, model_client):    #A
        self.model_client = model_client
 
    def embed_chunks(
        self,
        chunks: List[Chunk],
        model: str = "text-embedding-3-small",
        batch_size: int = 100    #B
    ) -> List[List[float]]:
 
        all_embeddings = []
 
        for i in range(0, len(chunks), batch_size):
            batch = chunks[i:i + batch_size]
            texts = [chunk.text for chunk in batch]
 
            response = self.model_client.embed(    #C
                texts=texts,
                model=model
            )
            all_embeddings.extend(response.embeddings)
 
        return all_embeddings
 
    def embed_query(
        self, query: str, model: str
    ) -> List[float]:    #D
        response = self.model_client.embed(
            texts=[query],
            model=model
        )
        return response.embeddings[0]

Embedding generation exposes model choice, batching, and provider abstraction through a reusable Model Service interface. section 5.3.6

How Designing AI Systems covers it

How deep it goes
Explains operational selection and embedding dimensionality, including routing consequences, storage costs, search latency, and vector consistency.
Examples
Uses Claude Sonnet versus GPT-4 for classification and embedding models with configurable dimensions for retrieval indexes.
What it assumes
Assumes familiarity with embeddings and vector representations while establishing consistency requirements within an index.
What you can do after
Evaluate model performance with production metrics and route workloads or configure indexes according to measured tradeoffs.
How it connects
Connects model selection to observability, provider abstraction, routing, cost tracking, embeddings, vector indexes, and API gateways.

Also explained in

Compare how 12 books explain model selection →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Designing AI Systems. The section links point to the parts of the book it draws on.