Chapter 3: Selecting and evaluating AI tools

Introduction

The chapter introduces the AI-tool selection and performance-evaluation topics it will cover.

3.1 Proprietary vs. open source

This section contrasts how proprietary and open source AI can be accessed, then explains why open source models may still have undisclosed components and usage restrictions.

• Proprietary AI access: Explain what users can and cannot access in proprietary AI, and describe its two common routes: customer-facing apps and APIs.
  Why it matters: The route you use affects how you integrate the AI and how you are charged.
• Open source model access: Describe what an open source AI provider discloses and what options users have for customizing and running the model.
  Why it matters: Open source gives users deployment and customization options that a hidden proprietary model does not.
• Limits of model openness: Explain why access to an open source model’s final parameters does not necessarily let someone reproduce its training.
  Why it matters: A downloadable model is not necessarily a reproducible account of how it was made.
• Open source license restrictions: Explain why possession of an open source model does not guarantee unrestricted use, using the Llama examples.
  Why it matters: Checking the license matters even when you can download and run the model yourself.

Key terms: proprietary AI, customer-facing apps, APIs, tokens, open source AI, parameters, self-host, training datasets, reinforcement learning with human feedback, licenses

3.2 How to decide

This section compares when proprietary and open source AI are suitable, weighing ease of use, control, cost, customization, and output quality.

• Proprietary AI convenience: Explain why proprietary AI is suitable when a team wants a done-for-you solution without specialized technical knowledge.
  Why it matters: Ease of adoption can be a reason to choose a proprietary tool.
• Self-hosting control and trade-offs: Explain why an organization might self-host open source AI and what costs and risks the section associates with doing so.
  Why it matters: Control over code and data comes with operating costs and a need for caution about models you run.
• Open source customization: Explain why a team seeking to modify a model might choose open source AI.
  Why it matters: The ability to change a model can affect which kind of AI tool a team selects.
• Changing output quality gap: Describe how the section characterizes the output-quality difference between proprietary and open source AI.
  Why it matters: Past assumptions about output quality may not settle a current tool choice.

Key terms: proprietary AI, done-for-you solution, open source AI, self-host, transparency, governance, APIs, customization, fine-tuning, quality of outputs

3.3 Off-the-shelf vs. fine-tuning

This section contrasts customizing an unchanged AI model through prompts and context with fine-tuning a model, including fine-tuning’s overfitting risk and practical requirements.

• Off-the-shelf customization: Explain how prompts and added context can tailor an off-the-shelf model to a task without changing the model itself. Describe how examples in a prompt help clarify the desired output.
  Why it matters: You may be able to improve a model’s performance on your task without modifying or retraining it.
• Fine-tuning model parameters: Describe how fine-tuning uses task-specific training data to produce an altered model, and distinguish that from changing only the model’s prompt.
  Why it matters: Fine-tuning is a different commitment from supplying a better prompt because it creates a customized model.
• Overfitting and its controls: Explain the overfitting risk in fine-tuning and describe ways this section says to limit it, including restricting updates with LoRA.
  Why it matters: A fine-tuned model must work on cases beyond the examples used to customize it.
• Fine-tuning access and effort: Distinguish the fine-tuning access offered by open source and proprietary models, and state the additional expertise or resources fine-tuning may require.
  Why it matters: Access to a model and the effort required to customize it affect whether fine-tuning is practical.

Key terms: off-the-shelf models, prompt engineering, few-shot prompting, zero-shot prompting, RAG, fine-tuning, training data, internal parameters, overfitting, early stopping

3.4 How to decide

This section explains when to improve an off-the-shelf model with prompts and when fine-tuning may be the better choice.

• Prompt-first improvement strategy: Explain why prompt engineering is usually the first way to improve a model’s output, and when fine-tuning becomes worth considering.
  Why it matters: Trying prompts first may improve the output without fine-tuning a model.
• Small-model fine-tuning tradeoff: Explain why fine-tuning a smaller model may be preferable to prompting a larger model, particularly when self-hosting.
  Why it matters: The cheapest effective approach may depend on whether you must run the model yourself.

Key terms: prompt engineering, fine-tuning, context windows, self-host, small, fine-tuned model

3.5 Customer-facing AI apps vs. foundation models

Distinguishes customer-facing AI apps from the foundation models that power them and notes that developers can access some models through APIs.

• Apps and foundation models: Distinguish customer-facing AI apps from foundation models, using ChatGPT and GPT-4o to describe how an app can be powered by a model behind the scenes.
  Why it matters: This distinction makes it clearer whether you are considering a product people use or the model powering it.
• Developer access through APIs: Describe how a company may both offer a customer-facing app and provide API access to its underlying foundation model for developers building their own apps.
  Why it matters: The same company may offer both a ready-to-use app and a model that developers can use in their own apps.

Key terms: customer-facing AI apps, foundation models, ChatGPT, GPT-4o, APIs, underlying foundation models, software developers

3.6 How to decide

This section explains when to choose a customer-facing AI app and when to use a foundation model.

• Choosing apps or foundation models: Distinguish when to use a customer-facing AI app versus a foundation model: choose an app for help with a specific task, and a foundation model as a building block for an AI-based app of your own.
  Why it matters: The choice depends on whether you want help doing a task or want to build an app yourself.

Key terms: customer-facing apps, foundation models, building block

3.7 Model validation, selection, and testing

The section recommends benchmarking AI models through a well-designed assessment process and introduces the use of three datasets without yet explaining their roles.

• Benchmarks for model selection: Explain why a benchmark and a well-designed assessment process help when comparing and selecting AI models.
  Why it matters: A careful comparison makes model selection less likely to rest on an overly optimistic impression of performance.
• Three dataset protocol: State that the introduced evaluation protocol builds and evaluates AI models using three different datasets, without assigning roles to those datasets.
  Why it matters: The three-dataset structure is the starting point for the evaluation process described next.

Key terms: benchmark, assessment process, performance, datasets, three different collections of data

3.8 Training set

This section explains what a training set does, how training uses loss, and why training loss may not reflect performance on an intended task.

• Training set purpose: Describe a training set as examples used to build a model, and explain that a smaller training set can be used to fine-tune one.
  Why it matters: It identifies the data the model learns from, whether it is built from scratch or fine-tuned.
• Training loss and task performance: Explain that training adjusts model parameters to minimize loss on the training set, but that this loss may not directly measure performance on the task you intend to use the model for.
  Why it matters: A low training loss alone does not tell you whether a model is good at your intended task.
• Training data in evaluation: State when you might need to create a training set and why the data used for training may still matter when creating validation and test sets.
  Why it matters: Even if you do not train the model yourself, its training data can matter when you prepare to evaluate it.

Key terms: training set, fine-tune, model parameters, loss, training loss, validation set, test set

3.9 Validation set

This section explains how a validation set helps compare models and other choices, and why its data must be kept separate from training data.

• Validation-based model selection: Explain how validation-set performance is used to choose between models and to compare training, fine-tuning, or prompting choices.
  Why it matters: Validation results help you choose an approach based on observed task performance rather than guesswork.
• Business-aligned performance measure: Describe what kind of measure is usually used to calculate validation-set performance and how it may differ from the training loss function.
  Why it matters: A comparison is useful only if its measure reflects what you want the model to accomplish.
• Training data contamination: Explain why validation data must not overlap training data and how the section’s crossword example illustrates this risk for LLMs.
  Why it matters: Without genuinely unseen examples, a high validation score may give a misleading impression of how well the model will handle new problems.

Key terms: validation set, model selection, fine-tuning, prompt engineering, performance measures, business objective, loss function, training set, overfits, training data cut-off date

3.10 Test set

Explains why a model needs a final assessment on an independent test set and why that set must not influence model selection.

• Independent final performance check: Explain why selecting the best model on a validation set can give an overly optimistic impression, and describe how a separate test set provides a final check.
  Why it matters: A strong validation result alone may not reflect how well the chosen model performs more generally.
• Single-use test set boundary: Explain why the test set can be used only once and what to do if disappointing results lead to model changes or consideration of alternatives.
  Why it matters: Keeping test data out of modeling decisions preserves the value of the final assessment.

Key terms: validation set, test set, modeling decisions, model selection

3.11 Performance measures

3.12 Accuracy

Defines accuracy as a performance measure and shows how to calculate and use it for classification and other tasks.

• Accuracy calculation and uses: Explain that accuracy is the percentage of tasks performed correctly: divide the number of correct solutions by the total number of tasks in a validation or test set. Describe its use for classification and for other tasks, such as coding problems.
  Why it matters: Accuracy gives you a straightforward way to report how often a tool gets tasks right.

Key terms: accuracy, classification tasks, validation or test set

3.13 Precision and recall

This section explains precision and recall, why neither is sufficient alone, and how to evaluate retrieval systems against a business-specific minimum recall.

• Precision and recall meanings: Distinguish recall as the share of relevant instances identified from precision as the share of retrieved instances that are truly relevant.
  Why it matters: The two measures reveal different kinds of retrieval error.
• Retrieval measure tradeoff: Explain why retrieving nearly everything can produce high recall but low precision, while retrieving very little can produce high precision but low recall.
  Why it matters: A strong score on one measure can conceal poor results on the other.
• F-measure and its limits: Describe the F-measure as a harmonic mean combining precision and recall, and explain why the author cautions against treating it as a universally suitable measure.
  Why it matters: A combined score is useful only if its priorities fit the decision being made.
• Minimum recall evaluation approach: Describe how to set a business-required minimum recall, tune a relevance-score threshold to meet it, and then compare qualifying models by precision.
  Why it matters: This approach makes the performance comparison reflect what the business requires.

Key terms: recall, precision, relevant instances, false positive, tradeoff, false positives, F-measure, harmonic mean, minimum desirable level of recall, relevance score

3.14 Mean absolute error and root mean squared error

This section explains two ways to measure how far numerical AI predictions are from actual values: mean absolute error and root mean squared error.

• Mean absolute error calculation: Explain how to calculate mean absolute error for numerical predictions by averaging the absolute differences between predicted and known values.
  Why it matters: MAE gives you a straightforward measure of how far numerical predictions are from actual values.
• Root mean squared error: Describe how to calculate root mean squared error and why it gives larger deviations more weight than MAE.
  Why it matters: The choice between MAE and RMSE affects how strongly large prediction errors influence the reported result.

Key terms: mean absolute error, MAE, absolute difference, root mean squared error, RMSE, squaring, larger deviations

3.15 Summary