Chapter 1: AI reliability: Building LLMs for the real world

Introduction

The introduction contrasts rapidly expanding LLM capabilities with their unreliable performance in production, then defines reliability and its three layers.

• Capability and production gap: Explain why impressive LLM capabilities do not by themselves make a successful production system, using the pilot failures and obstacles described here.
  Why it matters: A capable demo is not enough if the system cannot deliver dependable results for real users.
• Production reliability and its layers: Describe what reliability means in this book and distinguish the roles of reliable outputs, reliable agents, and reliable operations.
  Why it matters: The three layers show why dependable answers alone are not the whole of production reliability.

Key terms: LLMs, hallucinations, flaky outputs, brittle tools, reliable outputs, reliable agents, reliable operations, grounded, monitoring

1.1 From benchmarks to the real world

The section contrasts impressive benchmark results with weaker performance on unfamiliar tasks and explains why increasingly capable models still need production reliability engineering.

• Benchmark Capability Versus Reliability: Explain why strong benchmark scores do not establish production reliability, using the drop from SWE-bench Verified to SWE-bench Pro as the section’s example.
  Why it matters: A high benchmark score is evidence of capability, not a guarantee of dependable performance on new problems.
• New Capabilities New Failures: Describe how frontier models’ expanded capabilities create value while introducing failures beyond those of text-only generation.
  Why it matters: Reliability matters more as models take on work beyond generating text.

Key terms: SWE-bench Verified, SWE-bench Pro, reliability gap, production reliability, frontier models, external tools, complex workflows

1.2 The tangible impact of LLMs in the real world

This section introduces the real-world reach of LLMs by linking four application areas to distinct reliability risks.

• Industry-specific reliability risks: Describe the different reliability challenges the section associates with legal work, customer service, development, and agentic systems.
  Why it matters: The kind of failure to watch for depends on where an LLM is used.

Key terms: fabricated outputs, incorrect policy information, insecure code, unsafe actions

1.2.1 Legal industry

This section shows how legal LLM tools are being used at scale while explaining why their outputs require citation checks and human validation.

• Legal LLM adoption and benefits: Describe the legal tasks supported by Harvey AI and Legora and the benefits reported by surveyed Legora users.
  Why it matters: The reported benefits explain why legal teams are adopting these tools despite their risks.
• Legal output validation safeguards: Explain why legal AI outputs need validation and identify the safeguards the section calls for before an output reaches a court.
  Why it matters: A plausible-looking legal answer is not safe to use in court without checking its sources and citations.

Key terms: Harvey AI, Legora, nonbillable hours, fabricated case citation, legal RAG hallucinations, source grounding, automated citation verification, human review

1.2.2 Customer service

Customer-service assistants can resolve inquiries quickly and at scale, but their policy answers need reliable sources and human escalation.

• Customer service impact: Describe how the Klarna and Intercom examples illustrate the scale, speed, and cost benefits of LLM-powered customer service.
  Why it matters: These examples show why companies put virtual assistants in front of customers.
• Reliable policy responses: Explain why a fluent customer-service assistant can cause harm when it gives incorrect policy information, and state the section’s recommended safeguards.
  Why it matters: A convincing answer about a refund or warranty can become a real commitment to a customer.

Key terms: intelligent virtual assistants, resolution rate, policy response, retrieved, version-controlled policy document, escalate to humans

1.2.3 Programming and development

This section contrasts the rapid adoption and productivity gains of AI coding tools with the production defects that still make rigorous validation necessary.

• Coding assistant adoption and speed: Describe the widespread use of AI coding assistants and the task-speed gains developers report.
  Why it matters: Their widespread use and speed gains make the reliability of AI-generated code an immediate engineering concern.
• Productivity versus code reliability: Explain why faster, more capable AI coding tools do not remove the need for code review, automated testing, and security validation.
  Why it matters: Generating code faster is useful only if teams still catch defects before that code affects users.

Key terms: AI coding assistants, adopted, 30–55% faster, reliability gap, manual debugging, logic bugs, code review, automated testing, security validation

1.2.4 Enterprise AI

Enterprise AI is moving from text generation toward agents that act, creating opportunities for productivity while expanding what reliability must cover.

• Enterprise Agentic AI Adoption: Describe how agentic AI differs from simple text generation and use Agentforce as an example of enterprise adoption.
  Why it matters: The enterprise example shows why reliability matters for systems that do more than produce text.
• Persistent LLM Output Risks: Explain why enterprise progress does not remove the risks of biased, inaccurate, or hallucinated outputs, especially in high-stakes fields.
  Why it matters: Impressive adoption does not make an LLM’s answers safe to trust without scrutiny.
• Reliability Beyond Answer Quality: Explain why autonomous actions broaden reliability beyond the quality of an AI system’s answers.
  Why it matters: An agent can cause problems through its actions even when assessing its answers alone would miss them.

Key terms: agentic AI, Agentforce, AI agents, biased outputs, inaccurate outputs, hallucination, permissions, reversibility, tool robustness, failure containment

1.3 Understanding hallucinations and reliable AI

The section identifies hallucination as a fundamental, dangerous reliability challenge involving LLMs fabricating information.

• Hallucination as a reliability challenge: Explain that hallucination involves LLMs fabricating information and is a fundamental, dangerous challenge when building reliable systems.
  Why it matters: Recognizing fabricated information as a reliability problem gives you a reason to scrutinize LLM outputs.

Key terms: hallucination, reliable systems

1.3.1 When AI hallucinates convincingly

A legal-research example shows how convincing AI fabrications can be mistaken for facts and cause real-world harm.

• Convincing fabricated legal precedents: Explain how an AI research assistant could invent cases that appear credible, and why relying on them matters beyond the example.
  Why it matters: A credible-looking answer is not necessarily a factual one, especially when people use it in consequential work.

Key terms: precedents, fabrications, legal research

1.3.2 What is a hallucination?

Defines LLM hallucinations, distinguishes their forms, and explains why plausible but false outputs arise.

• Hallucinations and their forms: Explain what makes an LLM output a hallucination, why it can be harder to spot than a traditional software error, and how its forms include fabricated facts, invented citations, and contradictions of context or world knowledge.
  Why it matters: Recognizing a confident-looking wrong answer is the first step toward assessing an LLM’s reliability.
• Why plausible fabrications arise: Explain how learned patterns, probabilistic token generation, imperfect training data, and lack of built-in verification can produce authoritative-sounding fabricated content.
  Why it matters: A polished, authoritative style is not evidence that an LLM has verified its answer.

Key terms: hallucination, intrinsic hallucination, extrinsic hallucination, invented citations, pattern-matching systems, probabilistically, ground truth, training data

1.3.3 What is reliable AI?

Defines reliable AI as more than avoiding hallucinations: it must produce trustworthy results, operate safely over time, and balance competing production requirements.

• Dimensions of reliable AI: Describe the qualities of a reliable AI system beyond avoiding hallucinations, including consistent and grounded answers, graceful failure, fairness, efficiency, safe actions, and sustained quality.
  Why it matters: A system can avoid fabricated answers and still be unreliable in other ways that matter to users.
• Semantic correctness beyond uptime: Distinguish traditional software reliability metrics from the semantic correctness that AI reliability also requires.
  Why it matters: Operational health alone cannot tell you whether users are receiving correct answers.
• Deliberate production tradeoffs: Explain why production reliability requires deliberate choices among competing goals rather than maximizing every dimension at once.
  Why it matters: Production decisions must account for competing requirements, not just optimize one reliability measure.

Key terms: reliable AI, grounded, fails gracefully, data drifts, traditional software reliability, semantic correctness, uptime, tradeoffs, production reliability, latency

1.4 The AI reliability framework

Introduces a three-layer framework for organizing AI reliability challenges and techniques, with details left to later sections.

1.4.1 Layer 1: Reliable outputs

This section explains why trustworthy outputs are the first layer of AI reliability and introduces the roles of prompting, RAG, semantic retrieval, fine-tuning, and distillation.

• Trustworthy Outputs First: Explain why an AI system needs accurate, grounded answers before it can safely take actions or operate autonomously.
  Why it matters: An agent that acts on untrustworthy answers can turn an output error into a consequential action.
• Prompt Engineering Defense: Describe how prompt engineering improves output reliability by shaping how a model approaches a problem, without changing the rest of the system.
  Why it matters: Changing the prompt can improve reliability before making larger system changes.
• RAG Grounding And Risks: Explain how RAG grounds answers in retrieved documents rather than relying only on training knowledge, and name the retrieval problems that can undermine it.
  Why it matters: Grounding helps only when the retrieved context is useful and current.
• Meaning Based Retrieval: Describe why embeddings enable semantic retrieval even when a query and a relevant document use different words.
  Why it matters: Relevant source material may use different wording from the user’s question.
• Fine Tuning Versus RAG: Distinguish the knowledge supplied by RAG at inference time from the model changes made through fine-tuning.
  Why it matters: The two approaches address different needs: supplying current context and adapting the model itself.
• Knowledge Distillation Purpose: State what knowledge distillation produces and why the section connects it to production deployment.
  Why it matters: A smaller model that keeps larger-model performance can help make production deployment practical.

Key terms: reliable outputs, accurate, grounded answers, prompt engineering, hallucinations, RAG, knowledge base, retrieved content, semantic retrieval, embeddings, vector search

1.4.2 Layer 2: Reliable agents

This section explains why reliable agents need safeguards when they act, and introduces standardized tool integration and specialized agents as ways to support reliable action.

• Risks of Agent Actions: Explain why enabling an AI system to use tools, access APIs, and coordinate workflows raises the stakes beyond generating reliable answers, and why errors can compound across a workflow.
  Why it matters: Reliable answers alone are not enough when an AI system can change things in the world.
• Standardized Tool Integration: Explain how standardized tool integration supports reliability when agents connect to external systems, and identify MCP as the standard named here.
  Why it matters: Consistent handling across tools makes agent failures easier to contain and trace.
• Specialized Agent Coordination: Describe why a complex task might call for a multi-agent system and what such a system coordinates.
  Why it matters: Dividing complex work among specialists offers a reason to use more than one agent.

Key terms: reliable outputs, agents, errors cascade, standardized tool integration, Model Context Protocol (MCP), error handling, traceability, failure containment, multi-agent systems, specialized AI components

1.4.3 Layer 3: Reliable operations

This section explains why reliable operations require ongoing evaluation, AI-specific production monitoring, and attention to fairness and privacy.

• Reliability After Deployment: Explain why a system that works in testing still needs operational discipline after deployment, and describe the kinds of changes and failures that can undermine its reliability.
  Why it matters: A successful test does not guarantee that users will continue to get reliable results.
• Complementary Evaluation Methods: Describe the evaluation approaches named here and explain why production systems combine methods rather than treating them as interchangeable.
  Why it matters: Combining evaluation methods gives a broader picture of quality than relying on one score.
• AI Specific Monitoring: Distinguish production monitoring for AI systems from monitoring only conventional service performance.
  Why it matters: A service can be available and fast while still giving users poor or fabricated answers.
• Fairness and Privacy as Reliability: Explain why fairness and privacy count as dimensions of reliability even when an AI system scores well for accuracy.
  Why it matters: Accuracy alone cannot establish that a system is safe and trustworthy for its users.

Key terms: reliable operations, operational discipline, data drifts, reference-based metrics, ROUGE, BLEU, LLM-as-judge, red-teaming, human evaluation, production monitoring

1.5 The reliability toolbox

This section presents a simple-first way to choose reliability techniques while evaluating and monitoring the system from the start.

• Simple-first technique selection: Explain how to add reliability techniques according to the problem rather than using every technique at once: start with prompt engineering, then consider RAG for grounding, fine-tuning for deeper domain expertise, and agent capabilities when actions are needed.
  Why it matters: Choosing a technique based on the problem avoids adding unnecessary complexity.
• Evaluation and monitoring from day one: Explain why evaluation and monitoring should be built in from the beginning rather than added after other reliability techniques.
  Why it matters: Early measurement helps reveal whether reliability work is helping and exposes problems before they persist.

Key terms: reliability toolbox, prompt engineering, RAG, fine-tuning, agent capabilities, evaluation, monitoring

1.6 Why reliable AI systems matter now

This section explains why reliable AI must be engineered across the whole production system, not judged by model performance alone.

• Whole-system reliability risks: Explain why reliability becomes more important as AI enters consequential settings and why checking the model alone is insufficient.
  Why it matters: A capable model cannot make an application dependable if other parts of the production system can fail.
• Production outcomes and public trust: Distinguish benchmark or testing success from the reliability users and organizations need in the real world.
  Why it matters: A system that looks good in testing can still cost an organization money and lose users’ trust when deployed.
• System-wide regulatory demands: Describe why regulation adds urgency to reliability engineering across the AI pipeline.
  Why it matters: Regulatory expectations give teams another reason to examine the entire deployed system rather than only their choice of model.

Key terms: production AI system, prompts, retrieval pipelines, agents, monitoring, production, benchmarks, user trust, model capability, system reliability

1.7 Requirements for following along

This section states the tools and background needed to follow the book’s projects and explains why most examples use API calls.

• Project setup and background: State the tools and prior knowledge needed to work through the examples, and explain that most examples use API calls instead of local model hosting to keep setup simple and costs low.
  Why it matters: These requirements help you prepare for the projects without assuming you need to host a model yourself.

Key terms: Python 3, pip, OpenAI API key, API calls, local model hosting

1.8 Summary