How deeply does Data Analysis with LLMs cover model selection?

Compares model costs and sample performance, and recommends inspecting test cases without giving a general ranking procedure.

Model selection

As explained in Data Analysis with LLMs

Model selection is the process of choosing a model based on task requirements, performance, cost, data sensitivity, and deployment constraints.

Model selection balances task performance against cost: choose the least expensive model that still handles the analysis well. Compare candidates on a data sample before processing a large dataset, since a small upfront investment may reduce total spend. section 9.0

Choosing a model for review labels

For movie-review sentiment labels, GPT-4 costs roughly 120 times more per token than GPT-3.5 Turbo. A mixed review that GPT-4 labels correctly but GPT-3.5 does not can help judge whether paying more to handle borderline cases is worthwhile. section 9.4

Label: neg; Ground truth: neg
Label: neg; Ground truth: neg
Label: neg; Ground truth: neg
Label: neg; Ground truth: neg
Label: pos; Ground truth: pos
Label: pos; Ground truth: neg
Label: pos; Ground truth: neg
Label: neg; Ground truth: neg
Label: pos; Ground truth: pos  #1
Label: neg; Ground truth: neg
Number of correct labels:    8  #2
Number of tokens used   :    2228  #3

A sample evaluation lists predicted and ground-truth labels, then reports the number of correct labels and tokens used. section 9.4

How Data Analysis with LLMs covers it

liveBook manning.com

Read Data Analysis with LLMs on liveBook Get the book at manning.com

Also explained in

liveBook manning.com

Delivering Machine Learning Projects: From design to deploymentModel selection is the process of choosing among candidate machine learning models based on evaluation results and project requirements.
liveBook manning.com

Evolutionary Deep Learning: Genetic algorithms and neural networksModel selection is the process of choosing a model’s architecture and training-related configuration for a problem.
liveBook manning.com

Distributed Machine Learning PatternsThe process of choosing a machine learning model or configuration from a set of candidates according to an evaluation criterion.
liveBook manning.com

Sutskever's ListThe process of choosing among candidate models according to an evaluation criterion such as description length or predictive performance.
liveBook manning.com

Machine Learning Platform EngineeringModel selection is the process of choosing among candidate machine learning models or architectures based on their suitability and measured performance.
liveBook manning.com

Demand Forecasting Best PracticesModel selection is the process of choosing a model structure or algorithm that is suitable for representing relationships in data and producing predictions.
liveBook manning.com

Financial AI in PracticeModel selection is the process of choosing among candidate predictive models according to performance and other requirements.
liveBook manning.com

Building Reliable AI SystemsThe process of choosing an AI model for a particular application or task.

Compare how 9 books explain model selection →

Related concepts

parameter estimationcontrasts with · Parameter estimation is the process of inferring unknown numerical model parameters from observed data and, where applicable, substantive assumptions.
language modeluses · A language model is a computational model that represents patterns in linguistic tokens to estimate sequence probabilities and predict or generate language from context.
cross-validationimplemented by · Cross-validation is a resampling technique that evaluates a model by repeatedly fitting it on some portions of available data and assessing it on held-out portions.
fine-tuningpart of · Fine-tuning is adapting a pretrained machine-learning model by training it further on targeted data or objectives for a task, domain, or behavior.
ensemblepart of · An ensemble is a technique that combines or considers outputs from multiple models or component methods collectively to produce or assess a joint result.

This explainer was generated by Manning's knowledge graph from the text of Data Analysis with LLMs. The section links point to the parts of the book it draws on.