Chapter 4: When to use (and not to use) AI
The chapter introduction previews seven questions for deciding when to build an AI product, use conversational AI at work, or build an LLM wrapper.
After this chapter you will be able to
- AI product business case. Explain why asking whether there is a business case for AI is a good starting point when building an AI-based product: overlooking business considerations can lead to unsuccessful projects.
- Risks of AI-first products. Explain how starting with AI because of hype or funding pressure can lead teams to build products customers do not need, using the supermarket checkout example to illustrate the risk.
- Customer-first product validation. Describe the customer-centric alternative: learn what customers genuinely need, test an early minimal product with them, and choose AI only if it is the best solution to their problem.
- Hallucination tolerance by application. Explain why the acceptability of hallucinations depends on their consequences for users, and how that affects the decision to use AI.
- Safer task framing. Describe how framing a legal-contract tool as an aid for finding overlooked mistakes, rather than an automatic corrector, changes whether hallucinations may be acceptable.
- Predictive performance and explainability. Explain why a model with strong predictive performance may still be a poor fit for a client who needs to understand its outputs.
- Choosing an explainable approach. Describe how the client’s need for explainability can change the choice of model or even the formulation of the problem.
- Conversational AI as an assistant. Describe conversational AI as using a natural-language prompt to ask AI to help with a task, such as coding or creating a logo.
- Succinct task description. Explain why conversational AI is more likely to help when a task can be described briefly, and why a highly specific task may not save time.
- Easy output validation. Explain why ease of checking the output matters when using conversational AI, and describe how prior expertise or a quick test can make checking easier.
- Existing tools before AI. Explain why a developer should look for an existing library before asking AI to implement a common task, using the dropdown example to show when AI offers little advantage.
- Hidden standards of quality. Explain why a polished AI output may still do a task poorly, and why you should investigate what competent work requires before deciding whether AI is good enough.
- LLM wrapper division of work. Explain how an LLM wrapper turns an app task into a prompt, lets the LLM do the main work, and processes its output for the user. State that this can be a useful approach in some cases but is not always the best solution.
- Natural-language fit for LLMs. Explain why natural-language interaction can favor an LLM, and why a product without it should consider other tools before building an LLM wrapper.
What it builds on
- Hallucinations can kill a product (chapter 2) · The possibility that hallucinations can undermine a product.
- Living with hallucinations (chapter 2) · The need to live with hallucinations rather than assume they disappear.
- Performance measures (chapter 3) · The use of performance measures to evaluate models.
- What are hallucinations? (chapter 2) · The possibility of AI hallucinations
Section by section
Introduction
The chapter introduction previews seven questions for deciding when to build an AI product, use conversational AI at work, or build an LLM wrapper.
4.1 Building an AI-based product
The section argues that deciding whether to use AI in a product should start with a business question, not just technical ones.
- AI product business case: Explain why asking whether there is a business case for AI is a good starting point when building an AI-based product: overlooking business considerations can lead to unsuccessful projects.
Why it matters: A technically plausible use of AI still needs a business reason to be worth building.
Key terms: business case, AI-based product
4.2 Am I putting AI before the customer?
The section argues that commercial AI products should start with genuine customer needs, not with a desire to use AI.
- Risks of AI-first products: Explain how starting with AI because of hype or funding pressure can lead teams to build products customers do not need, using the supermarket checkout example to illustrate the risk.
Why it matters: Starting with the technology can obscure whether anyone wants the product it enables. - Customer-first product validation: Describe the customer-centric alternative: learn what customers genuinely need, test an early minimal product with them, and choose AI only if it is the best solution to their problem.
Why it matters: Checking customer needs before committing to AI gives a commercial product a better chance of solving a real problem.
Key terms: AI-first approach, Just Walk Out, customer needs, customer-centric approach, minimal product, prototype, mock-up, product validation
4.3 Are hallucinations okay?
This section explains how to decide whether an AI product can tolerate hallucinations and how changing the task’s role may make them more acceptable.
- Hallucination tolerance by application: Explain why the acceptability of hallucinations depends on their consequences for users, and how that affects the decision to use AI.
Why it matters: The cost of an AI mistake matters more than whether the system sometimes makes mistakes. - Safer task framing: Describe how framing a legal-contract tool as an aid for finding overlooked mistakes, rather than an automatic corrector, changes whether hallucinations may be acceptable.
Why it matters: Changing what a tool promises and how people use it can make its errors less consequential without eliminating them.
Key terms: hallucinations, hotel reviews, reformulate the task, automatically correct contracts, double-check the contract
4.4 Do I need to explain how the output is generated?
This section explains how a client’s need to understand predictions should influence the choice of AI model or approach.
- Predictive performance and explainability: Explain why a model with strong predictive performance may still be a poor fit for a client who needs to understand its outputs.
Why it matters: The best-scoring model is not necessarily the one a client can use or trust. - Choosing an explainable approach: Describe how the client’s need for explainability can change the choice of model or even the formulation of the problem.
Why it matters: Checking the client’s needs first can prevent you from building a model whose outputs they cannot use. - Limits of model explanations: Describe what explainable AI techniques can reveal about a complex model and what they cannot yet provide.
Why it matters: An explanation technique may help a client, but it should not be presented as a complete account of a complex model’s decision.
Key terms: predictive performance, black box, explainability, linear, predictive model, analytical or statistical study, explainable AI, XAI, SHAP
4.5 Using conversational AI as an assistant
The section defines conversational AI as an assistant you give tasks to through natural-language prompts, setting up later questions about when to use it.
- Conversational AI as an assistant: Describe conversational AI as using a natural-language prompt to ask AI to help with a task, such as coding or creating a logo.
Why it matters: This defines the kind of AI assistance the following decision questions address.
Key terms: conversational AI, natural-language prompt
4.6 Can I describe the task succinctly and validate the output easily?
This section offers two questions for deciding whether conversational AI will save time on a task: can you describe it briefly, and can you check the result easily?
- Succinct task description: Explain why conversational AI is more likely to help when a task can be described briefly, and why a highly specific task may not save time.
Why it matters: Prompting is less useful as a shortcut if explaining the task takes as much work as completing it. - Easy output validation: Explain why ease of checking the output matters when using conversational AI, and describe how prior expertise or a quick test can make checking easier.
Why it matters: A fast answer is less helpful if you cannot tell whether it is right without substantial work.
Key terms: good bang for your prompt, describe the task succinctly, step-by-step prompt, validate the output, hallucinates
4.7 Has anyone done it before?
The section argues that, before asking an AI assistant to write code for a common task, developers should check whether a proven library already does the job.
- Existing tools before AI: Explain why a developer should look for an existing library before asking AI to implement a common task, using the dropdown example to show when AI offers little advantage.
Why it matters: Checking what already exists can save work without sacrificing reliability. - Documentation over generated code: Describe how the sun-position example compares finding and reading a library’s documentation with relying on an AI-generated algorithm or library instructions.
Why it matters: Even when AI suggests a suitable library, its instructions for using it may be wrong.
Key terms: AI assistant, libraries and frameworks, Material UI, reinventing the wheel, sun position, documentation, hallucinated
4.8 What does an excellent job look like?
The section explains why deciding whether to use AI requires understanding what excellent work involves and how much quality the task actually needs.
- Hidden standards of quality: Explain why a polished AI output may still do a task poorly, and why you should investigate what competent work requires before deciding whether AI is good enough.
Why it matters: Knowing what good work looks like helps you judge whether an AI result is actually fit for your purpose. - Logo quality and requirements: Describe what a high-end logo designer contributes beyond producing a logo, and explain why a startup and a growing company might make different choices about AI and specialist design.
Why it matters: The logo example shows why the right choice depends on the quality the business needs, not merely on whether AI can produce a logo.
Key terms: competently, pacing, organization, logo design, competitors’ logos, distinctive look
4.9 Building LLM wrappers
The section explains how an LLM wrapper delegates a product’s core task to an LLM and cautions that this approach is not always the best choice.
- LLM wrapper division of work: Explain how an LLM wrapper turns an app task into a prompt, lets the LLM do the main work, and processes its output for the user. State that this can be a useful approach in some cases but is not always the best solution.
Why it matters: It helps you recognize what work your app is handing to an LLM before deciding whether that design fits.
Key terms: LLM wrappers, natural-language prompt
4.10 Will users interact with my product using natural language?
This section explains how to decide whether an LLM belongs in a product by checking whether users interact through natural language, using a travel-planning app to show when conventional software works better.
- Natural-language fit for LLMs: Explain why natural-language interaction can favor an LLM, and why a product without it should consider other tools before building an LLM wrapper.
Why it matters: The way users interact with a product helps determine whether an LLM is worth its drawbacks. - Travel planner alternative: Describe why the travel app’s LLM approach struggled and how the replacement produced an itinerary.
Why it matters: The example shows how a tool matched to structured data can solve a problem better than prompting an LLM.
Key terms: natural language, LLM wrapper, hallucinations, unstructured outputs, Google Places API, pathfinding algorithm, structured information, sightseeing itinerary
4.11 Summary
Where this is used later
- Smoke and mirrors (chapter 6) · Evaluating claims about what an AI-based product actually does
- Stringent validation (chapter 5) · Considers whether hallucinations are acceptable for a use case.
- Economics of AI (chapter 6) · The idea of products built around LLMs provides context for the thin AI-wrapper example.
Concepts explained in this chapter
- entrepreneurship mindsetAn approach to selecting AI use cases by starting from the customer problem and choosing the most appropriate solution rather than assuming AI is needed.
- AI-first approachA product-development strategy that starts by seeking uses for AI before verifying a customer need.
- antipatternA commonly repeated but harmful approach that looks plausible yet tends to produce bad outcomes.
- customer-centric approachA product-development strategy that begins with customer needs and works backward to the best solution.
- hallucinationA model output that is confidently or plausibly produced despite being unsupported or wrong relative to the intended real-world relationship.
- logo designThe process of creating a visual brand mark, including research, originality, and fit with a market context.
- AstralA Python library mentioned as an off-the-shelf way to calculate the sun position from time and location.
- GPT-4An OpenAI large language model used as an example tokenizer and reference for token counts.
- cookie-cutter taskA routine, standard task that is common enough that a prebuilt solution is likely to exist.
- documentationWritten reference material describing a model or system, including development details and supported information as required here.
- sun position calculationThe computation of the sun's apparent position in the sky for a given time and location.
- LLM wrapperSoftware that sits around an LLM and modifies inputs and outputs.
- machine learningA field that builds systems that learn patterns from data to perform tasks.
- customer problemA genuine problem or requirement that a product can satisfy for its intended users.
- minimal productA minimal product version intended to learn about customers with the least necessary implementation.
- product validationThe process of checking with prospective customers that a product idea actually addresses a real need.
- text-to-image modelA generative AI technique that produces images conditioned on textual descriptions.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.