Model selection
Model selection is the process of choosing among candidate models or configurations according to evaluation criteria and suitability for a task.
Delivering Machine Learning Projects: From design to deployment
Model selection is the process of choosing among candidate machine learning models based on evaluation results and project requirements.
- How it is used
- Chooses production models from evaluation evidence and system requirements through a decision stakeholders can review.
- How deep it goes
- Specifies selection tickets and comparison approaches without prescribing one method for every project.
- Examples
- Uses churn and demand prediction requirements across locales and product types to illustrate selection constraints.
- What you can do after
- Practice/perform documented model selection using evaluation evidence, then obtain customer sign-off.
Read how Delivering Machine Learning Projects: From design to deployment explains model selection →
Evolutionary Deep Learning: Genetic algorithms and neural networks
Model selection is the process of choosing a model’s architecture and training-related configuration for a problem.
- How it is used
- Automates comparison and configuration of candidate models within broader AutoML and evolutionary deep-learning workflows.
- How deep it goes
- Defines candidate-model and configuration choices, then identifies weights, optimization, and loss as tunable elements without detailed search procedures.
- Examples
- Uses model comparisons, ensembles, layer choices, initial weights, optimization mechanisms, loss measures, and an AutoML workflow figure.
- What you can do after
- Apply it via AutoML to compare candidate models and tune architectural and training-related choices for a problem.
Distributed Machine Learning Patterns
The process of choosing a machine learning model or configuration from a set of candidates according to an evaluation criterion.
- How it is used
- Promotes the top trained candidate to model serving in an end-to-end machine-learning workflow.
- How deep it goes
- Implements testing-data evaluation, highest-accuracy comparison, artifact copying, and Kubernetes Pod execution without detailing selection algorithms beyond accuracy.
- Examples
- Uses basic CNN, CNN with dropout, and CNN with batch normalization evaluated on Fashion-MNIST test data.
- What you can do after
- Operate/configure it by evaluating saved TensorFlow models, promoting the highest-accuracy artifact, and running selection as a Kubernetes workflow step.
Read how Distributed Machine Learning Patterns explains model selection →
Sutskever's List
The process of choosing among candidate models according to an evaluation criterion such as description length or predictive performance.
- How it is used
- Uses description length to choose models that capture data regularity without encoding noise.
- How deep it goes
- Explains the model-and-data code-length trade-off and its overfitting rationale, while deferring universal shortest-description guarantees.
- Examples
- Compares linear, high-degree, and intermediate polynomial curves using total description length.
- What you can do after
- Evaluate candidate models by comparing complexity costs with data-fit costs and identifying likely overfitting.
Machine Learning Platform Engineering
Model selection is the process of choosing among candidate machine learning models or architectures based on their suitability and measured performance.
- How it is used
- Guides production LLM deployments toward task-appropriate models instead of a single expensive default.
- How deep it goes
- Explains cost comparisons and tiered routing implementation, while stopping short of detailed evaluation methodology.
- Examples
- Uses DakkaBot query costs, three Gemini tiers, and password-reset, troubleshooting, and complex-analysis routing examples.
- What you can do after
- Operate a tiered LLM router that assigns queries to models according to complexity and cost-performance requirements.
Read how Machine Learning Platform Engineering explains model selection →
Demand Forecasting Best Practices
Model selection is the process of choosing a model structure or algorithm that is suitable for representing relationships in data and producing predictions.
- How it is used
- Guides supply-chain forecasting choices by matching models to data, objectives, metrics, and planning requirements.
- How deep it goes
- Extends from model comparison through data inspection, fitting, and error feedback, while deferring advanced model internals.
- Examples
- Uses demand, prices, promotions, holidays, products, regions, benchmarks, and competing forecasting approaches as evaluation cases.
- What you can do after
- Apply a comparative workflow to inspect data, test models, align metrics, fit candidates, and investigate forecast errors.
Read how Demand Forecasting Best Practices explains model selection →
Data Analysis with LLMs
Model selection is the process of choosing a model based on task requirements, performance, cost, data sensitivity, and deployment constraints.
- How it is used
- Choose a model that meets an analysis task’s quality needs while limiting costs, especially before processing large datasets.
- How deep it goes
- Compares model costs and sample performance, and recommends inspecting test cases without giving a general ranking procedure.
- Examples
- Tests sentiment labels on movie reviews, comparing GPT-3.5 Turbo with GPT-4 and inspecting a mixed-polarity review.
- What you can do after
- Evaluate model choices on a data sample and judge whether better handling of borderline cases warrants higher costs.
Financial AI in Practice
Model selection is the process of choosing among candidate predictive models according to performance and other requirements.
- How it is used
- Guides financial teams in choosing models for generative AI products and predictive strategies.
- How deep it goes
- Demonstrates a qualitative task-specific comparison and discusses cost, privacy, vendor lock-in, and infrastructure tradeoffs.
- Examples
- Compares summaries of financial posts and weighs commercial APIs against open-source models for an MVP and longer-term use.
- What you can do after
- Evaluate candidate models on financial tasks and weigh their performance against cost, privacy, and infrastructure requirements.
Read how Financial AI in Practice explains model selection →
Building Reliable AI Systems
The process of choosing an AI model for a particular application or task.
- How it is used
- Select a model for an application before tuning prompts or parameters, balancing cost, latency, capability, and available parameters.
- How deep it goes
- Frames selection as consequential and continuous, but gives no implementation details for model routing.
- Examples
- Uses model deprecation, cheaper candidates with comparable shadow-test quality, and capability gaps such as structured output, longer context, or vision.
- What you can do after
- Configure model names as replaceable parameters and evaluate candidates with shadow tests and golden datasets.
Read how Building Reliable AI Systems explains model selection →
Related concepts
- parameter estimationcontrasts with · Parameter estimation is the process of inferring unknown numerical model parameters from observed data and, where applicable, substantive assumptions.
- language modeluses · A language model is a computational model that represents patterns in linguistic tokens to estimate sequence probabilities and predict or generate language from context.
- cross-validationimplemented by · Cross-validation is a resampling technique that evaluates a model by repeatedly fitting it on some portions of available data and assessing it on held-out portions.
- fine-tuningpart of · Fine-tuning is adapting a pretrained machine-learning model by training it further on targeted data or objectives for a task, domain, or behavior.
- ensemblepart of · An ensemble is a technique that combines or considers outputs from multiple models or component methods collectively to produce or assess a joint result.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.