Chapter 12: Designing LLM-powered systems

Introduction

The chapter introduces LLMOps as an extension of existing MLOps infrastructure and previews building an observable RAG application before hardening it for production.

• MLOps foundation and LLM extensions: Explain that LLM applications retain the reliable infrastructure, deployment, and monitoring practices of MLOps while adding components for LLM-specific needs, rather than requiring a platform rebuild.
  Why it matters: The chapter builds on the ML platform already developed in the book instead of starting from scratch.
• Distinct LLM operational challenges: Describe the LLM-specific challenges named in the introduction: nondeterministic outputs complicate conventional testing, multistep reasoning calls for new debugging approaches, and prompts and safety need deliberate attention beyond model accuracy.
  Why it matters: These differences explain why familiar engineering practices need to evolve for LLM applications.
• DakkaBot chapter roadmap: Describe DakkaBot as a RAG application for querying company documentation and distinguish this chapter’s aim of building an observable system from the next chapter’s focus on production testing, safety, and governance.
  Why it matters: DakkaBot provides the concrete case study through which the book develops its LLM architecture.

Key terms: MLOps, LLMOps, Kubernetes (K8s), CI/CD, document ingestion pipelines, vector databases, nondeterministic outputs, multistep reasoning chains, prompt engineering, safety concerns

12.1 LLMOps: New challenges, familiar principles

LLM applications change how ML systems are built, but still depend on the infrastructure and operational practices established in earlier chapters.

• LLM change and platform continuity: Explain that LLMs change how we build ML applications without making existing platform work obsolete: LLM-powered applications still need Kubernetes infrastructure, systematic deployment through CI/CD, and continuous monitoring.
  Why it matters: LLMOps builds on the platform practices you have already learned rather than replacing them.

Key terms: LLMs, LLM-powered applications, K8s clusters, monitoring pipelines, CI/CD systems

12.1.1 What makes LLM applications different

This section explains why LLM applications need different testing, token management, observability, prompt practices, and cost controls from traditional ML systems.

• Variable outputs and evaluation: Explain why the same prompt can produce different valid responses and why exact-output assertions are insufficient. Describe evaluating semantic correctness, factual accuracy, and adherence to guidelines instead.
  Why it matters: Reliable LLM tests must judge the quality of a response, not whether it matches one expected string.
• Sampling controls and use cases: Distinguish temperature, Top-K, and Top-P as ways to control token selection and output variability. Explain why consistent factual retrieval calls for less randomness while creative or conversational uses may benefit from controlled variation.
  Why it matters: Sampling settings help match the model’s consistency or variety to the application.
• Context and response token budgets: Explain the difference between the input context window and the output token limit, and describe how retrieved chunks, prompts, and queries consume a RAG system’s token budget. State what happens when input or output limits are reached.
  Why it matters: A RAG system must leave enough room for a useful response without exceeding model limits or paying for unnecessary context.
• Multistep workflow observability: Contrast a single traditional model prediction with an LLM application that retrieves documents, assembles context, generates a response, and may validate it. Explain why debugging requires visibility into intermediate inputs and outputs.
  Why it matters: Seeing each step helps locate failures that a final answer alone cannot explain.
• Prompts as managed code: Explain how structured instructions, examples, and context in prompts shape LLM performance, and why prompts should be versioned, tested, and improved systematically rather than treated as casual chat text.
  Why it matters: Changes to a prompt can change application behavior much like changes to code.
• Usage based LLM costs: Contrast traditional provisioned-infrastructure costs with LLM charges that vary with input and output token counts. Explain why tracking usage and improving prompt efficiency help manage widely varying query costs.
  Why it matters: Token-aware cost tracking reveals which requests and prompts are expensive.

Key terms: nondeterministic outputs, semantic correctness, factual accuracy, adherence to guidelines, temperature, Top-K sampling, Top-P, nucleus sampling, next token, token limits

12.1.2 Extending our ML platform for LLMs

This section shows how an existing ML platform can support LLM applications by retaining its operational foundation and adding LLM-specific artifacts, infrastructure, monitoring, and safeguards.

• Shared ML platform foundation: Explain that traditional models and LLM applications can run on the same container and Kubernetes infrastructure, using existing deployment automation, scaling, system monitoring, and access controls.
  Why it matters: LLMOps extends a proven platform rather than requiring a rebuild.
• Expanded LLM artifact lineage: Describe the richer set of artifacts and settings an LLM application must track and version, and explain why tracing an output to its data, model, and prompt helps debugging.
  Why it matters: A prompt or retrieval change can alter behavior as significantly as a model change.
• Vector and prompt infrastructure: Distinguish the roles of vector databases and prompt management systems as additions to the existing platform: vector databases support embedding-based semantic search, while prompt systems version, test, deploy, and roll back prompts.
  Why it matters: These components support retrieval and controlled prompt changes without replacing the platform.
• LLM routing and caching: Explain how LLM gateways and caching differ from ordinary load balancing and exact-match caching, including provider-aware routing, semantic response reuse, token-based limits, and provider-specific fallback.
  Why it matters: Production routing must account for differences in provider APIs, costs, limits, and failures.
• LLM-specific observability: Describe how specialized LLM monitoring complements infrastructure metrics and ML experiment tracking by exposing prompt traces, token usage and cost, conversation or multistep behavior, response quality, and safety alerts.
  Why it matters: Healthy servers alone do not show whether generated answers are useful, safe, or affordable.
• LLM security threats: Describe LLM-specific threats, particularly malicious instructions in user input or external data, and connect them to input validation, output filtering, and continued response monitoring alongside existing access controls.
  Why it matters: Standard API security does not by itself prevent a model from following hostile instructions or producing unsafe output.

Key terms: containers, Kubernetes, CI/CD, Horizontal Pod Auto-scaler, prompt artifacts, configuration parameters, retrieval components, evaluation artifacts, lineage, MLflow

12.1.3 Essential tools for LLM applications

This section introduces the embedding, retrieval, orchestration, observability, and security tools used to build LLM applications beyond a simple RAG prototype.

• Semantic role of embeddings: Explain how embedding models turn text into numerical vectors so a RAG system can find documents with similar meaning even when their wording differs.
  Why it matters: Semantic search lets the system retrieve useful context that keyword matching might miss.
• FAISS retrieval workflow: Trace how document chunks and a user query are embedded, how FAISS finds similar chunks, and how those chunks become context for LLM generation. Distinguish FAISS’s similarity-search role from an ordinary database’s exact-match queries.
  Why it matters: Retrieval gives the LLM relevant information from a knowledge base before it answers.
• LangChain component chains: Describe how LangChain connects reusable LLM application components into sequential workflows and why this reduces the integration work of a manual RAG pipeline. Explain that standardized interfaces make models and vector stores easier to swap.
  Why it matters: Reusable components make it easier to prototype and maintain a multistep RAG application.
• Sequential versus graph workflows: Distinguish LangChain’s linear chains from LangGraph’s graph-based workflows, using branching decisions or validation-and-refinement loops as examples. State when the section suggests reconsidering LangChain abstractions.
  Why it matters: The workflow’s shape determines whether a simple chain or more flexible orchestration is appropriate.
• Langfuse observability needs: Describe LLM-specific information Langfuse can trace or monitor in a RAG application, and identify its prompt-management capability.
  Why it matters: Observing retrieved context, cost, and response timing helps diagnose the behavior users actually experience.
• Promptfoo security testing: Explain why LLM applications need security testing and describe Promptfoo’s role in finding vulnerabilities in both the model and the surrounding application, including through jailbreak prompts.
  Why it matters: Testing the whole application helps uncover unwanted responses or behavior before users encounter them.

Key terms: embeddings, embedding models, semantic search, vector space, FAISS, vector store, document chunks, similarity search, LangChain, chains

12.2 Building DataKrypt’s DakkaBot: A simple RAG architecture

Introduces DataKrypt’s onboarding problem and DakkaBot, a developer assistant intended to help new engineers find answers in company documentation.

• DakkaBot onboarding motivation: Explain why DataKrypt is building DakkaBot: rapid growth has made its systems and procedures harder for new engineers to learn, and abundant documentation has become difficult to search for relevant answers.
  Why it matters: This problem explains the purpose of the developer assistant built in the following sections.

Key terms: DataKrypt, DakkaBot, developer assistant, technical guides, API documentation, runbooks

12.2.1 What you’ll build

This subsection previews the end-to-end DakkaBot build and explains why observability and cost tracking matter beyond the first working chatbot.

• DakkaBot build roadmap: Describe the planned build: ingest documents in various formats into a vector database, choose an LLM and embedding model, orchestrate them together, and add a user interface for colleagues.
  Why it matters: These pieces establish what the rest of the chapter will assemble into a usable RAG chatbot.
• RAG pipeline observability: Explain why the initial chatbot is only a starting point: LLM answers can differ for the same question, so observability is added to show what happens at each stage of the RAG pipeline.
  Why it matters: Tracing the pipeline helps make variable LLM behavior understandable rather than opaque.
• Token cost monitoring: Explain that commercial LLMs charge by tokens, making production use potentially expensive, and that the chapter will track spending in real time and optimize costs without sacrificing response quality.
  Why it matters: A usable production chatbot must be economically manageable as well as functional.

Key terms: vector database, LLM, embedding model, user-friendly interface, nondeterministic, observability, RAG pipeline, token-based pricing, real-time spending, response quality

12.2.2 Beyond single API calls: Designing for composability

This section explains why complex LLM applications need composable steps and how RAG addresses the limits of single API calls.

• Single-call limitations: Explain why one large LLM call is a poor fit for complex enterprise tasks: context windows are finite, long multipurpose prompts lose focus, a misunderstanding can undermine the whole response, and repeated large prompts waste tokens.
  Why it matters: These limitations explain why the application needs more than a bigger prompt.
• Composable step responsibilities: Describe how breaking a complex task into focused steps helps maintain context, isolate failures for targeted recovery, and reuse cached intermediate results across requests.
  Why it matters: Focused components make a complex application more reliable and less wasteful than an all-or-nothing call.
• Selective RAG retrieval: Explain that RAG separates information retrieval from response generation, selecting query-relevant document chunks rather than placing an entire collection in the prompt; this improves focus and reduces repeated context-processing costs.
  Why it matters: Selective retrieval is the section’s central way to make large document collections usable within LLM limits.
• Fresh and grounded answers: Explain how RAG can supply post-training, changing, or private information without retraining the model, and how grounding an answer in retrieved documents enables source attribution and reduces hallucinations.
  Why it matters: Retrieval lets an LLM answer from information its training did not contain while giving readers a basis for checking its claims.

Key terms: context window, tokens, reasoning quality degradation, error propagation, cost inefficiency, composable system, multistep systems, focused responsibility, targeted recovery, cache intermediate results

12.2.3 Google’s Gemini LLM and embeddings

This section selects Gemini models for DakkaBot’s embeddings and response generation, then outlines API setup and production safeguards.

• Gemini model roles: Distinguish how DakkaBot uses text-embedding-004 to turn documents and queries into vectors and gemini-2.5-flash to generate responses. Explain that the generation model is chosen for speed and cost while maintaining quality.
  Why it matters: The two models perform different jobs in the RAG application.
• Gemini access and evaluation: Describe creating an API key in Google AI Studio, checking access to both model types, and reviewing quotas before production use. Explain that AI Studio also lets developers test models interactively.
  Why it matters: Access and quota checks help prevent integration surprises.
• Hosted API production safeguards: Explain why hosted Gemini calls need exponential-backoff retries when throttled, including when platform-wide demand causes rate limits. Describe tracking tokens for both embeddings and generation and considering OpenAI or Claude as fallbacks when high availability is required.
  Why it matters: Retries, usage monitoring, and fallback options make a hosted-model application more robust in production.

Key terms: text-embedding-004, gemini-2.5-flash, embeddings, vector representations, Google AI Studio, API key, model access, rate limits, exponential backoff, throttling

12.2.4 The retrieval component

This section builds DakkaBot’s retrieval component by turning documentation into a persistent FAISS index for semantic search and explaining the choices that affect retrieval quality.

• Task-specific embedding wrapper: Explain how GeminiEmbeddings gives the application one interface to Google’s embedding API while using RETRIEVAL_DOCUMENT for indexed content and RETRIEVAL_QUERY for user searches. Describe why the separate task types can improve retrieval and why isolating the API makes providers easier to change.
  Why it matters: The embedding setup determines how documents and questions are represented for search.
• Retrieval quality tradeoffs: Explain why finding relevant chunks is necessary for reliable RAG answers, and describe how chunk size, embedding-model fit, similarity metrics, and retrieval count or thresholds trade off context, precision, coverage, and noise.
  Why it matters: A capable LLM cannot compensate for missing or irrelevant retrieved information.
• Document loading and chunking: Trace how the pipeline recursively loads Markdown documentation into Document objects with content and source metadata, then splits large documents at natural boundaries with overlap to preserve context.
  Why it matters: Searchable chunks need both coherent text and metadata that can identify their sources.
• FAISS indexing pipeline: Trace the complete path from loaded and split documentation through document embeddings to a FAISS vector index, then explain how a query embedding finds semantically nearby chunks even without matching keywords.
  Why it matters: This pipeline makes the documentation searchable quickly enough to support DakkaBot’s question answering.
• Persistent index restoration: Explain why the pipeline saves and reloads the FAISS index and associated metadata instead of rebuilding them at startup, and state the configuration-matching and deserialization cautions when loading it.
  Why it matters: Persistence avoids repeated embedding time and API expense while making index restoration a deliberate operational choice.
• Production index update strategies: Describe how selective updates, versioned indexes, and blue-green deployments can keep production retrieval current while reducing rebuild work, rollback risk, or downtime.
  Why it matters: Stale or unavailable retrieval results directly affect user trust.

Key terms: GeminiEmbeddings, text-embedding-004, RETRIEVAL_DOCUMENT, RETRIEVAL_QUERY, chunking strategy, similarity metrics, cosine similarity, k, similarity thresholds, DirectoryLoader

12.2.5 The augmentation component

The section shows how to turn retrieved document chunks into context for an LLM, while safely reconnecting to the persistent index used to retrieve them.

• Retrieved context assembly: Explain how augmentation turns retrieved document chunks and a user query into a prompt the LLM can use. Describe concatenating chunks into one context string as a minimal approach, while noting that production systems need more careful formatting, deduplication, and organization.
  Why it matters: Relevant search results help only when they are assembled into usable context for generation.
• Persistent index reconnection: Trace the preparation for query processing: load the prebuilt vector index with the same embedding model used to build it, then expose similarity search through a standardized retriever interface. Explain that changes in embedding dimensions, normalization, or model version can cause loading or retrieval failures.
  Why it matters: A consistent index and retriever let the application obtain the chunks that augmentation needs.
• FAISS deserialization risk: Explain why loading a persisted FAISS index with dangerous deserialization enabled requires caution: its Python pickle-based format can execute arbitrary code if index files are tampered with. State the production precautions of securing index files and considering integrity checks.
  Why it matters: Protecting persisted index files prevents a retrieval convenience from becoming a code-execution risk.

Key terms: augmentation, document chunks, context assembly, prompt, prebuilt vector index, embedding model, retriever interface, similarity search, dangerous deserialization, FAISS

12.2.6 The generation component

This section shows how DakkaBot configures Gemini and uses a structured prompt to turn retrieved documentation into a grounded answer, then places generation within the complete RAG flow.

• Consistent Gemini configuration: Explain why the generation component uses Gemini with zero temperature and the gemini-2.5-flash model, and what the Gemini message-format setting does.
  Why it matters: These settings make responses more predictable while keeping an interactive assistant responsive.
• Grounded response prompt: Describe how the prompt template combines retrieved context with the user’s question and instructs the LLM to answer as a DataKrypt assistant using the provided documentation.
  Why it matters: Clear prompt boundaries help keep the answer tied to retrieved facts rather than unsupported general knowledge.
• End to end RAG flow: Trace the section’s minimal RAG example from a user query through semantic retrieval and context assembly to prompt-guided answer generation, and distinguish it from a production-ready system.
  Why it matters: The full sequence shows how an LLM can answer using specific, up-to-date documentation.

Key terms: zero temperature, gemini-2.5-flash, convert_system_message_to_human, prompt template, Context, Question, Answer, hallucination, RAG, semantic retrieval

12.3 Giving DakkaBot a UI

This section turns DakkaBot’s RAG pipeline into a reusable core and a responsive Chainlit chat interface with sources, streaming, validation, and graceful error handling.

• Chainlit interface advantages: Explain why Chainlit is used to give colleagues a chat interface for querying company documentation instead of exposing only an API or building a custom frontend.
  Why it matters: The interface makes DakkaBot’s internal knowledge base practical for colleagues to use.
• Reusable DakkaBotCore architecture: Describe how extracting DakkaBotCore separates RAG logic from its interfaces, and how asynchronous lazy initialization, readiness checks, and synchronous singleton wrappers support different uses.
  Why it matters: One core can serve a chat UI, other services, and tests without duplicating expensive RAG setup.
• Structured answers and attribution: Explain how enhanced query processing returns an answer with sources and error information, and how formatted context and a more specific system prompt shape the generated answer.
  Why it matters: Structured results let the UI show evidence and useful failures rather than an unverifiable answer or a generic error.
• Progressive streaming response flow: Trace how process_query_stream uses an async generator to deliver retrieved sources first, then generated tokens, and finally a completion signal or useful error information.
  Why it matters: Progressive delivery makes the bot feel responsive while keeping the UI informed throughout retrieval and generation.
• Token budget constraints: Explain how input and output token limits affect RAG context size, response completeness, latency, and cost.
  Why it matters: A responsive chat UI still needs requests sized so retrieval and generation can complete reliably.
• Chainlit session and message handling: Trace how Chainlit’s startup and message handlers initialize the shared bot, orient users, validate input, show progress, stream results, and handle failures.
  Why it matters: These handlers connect the reusable RAG core to a usable, responsive chat experience.

Key terms: Chainlit, chat interface, conversation history, DakkaBotCore, separation of concerns, lazy initialization, initialized flag, singleton pattern, asyncio.run(), process_query

12.4 Observability for LLM applications

This section explains why diagnosing poor LLM answers requires tracing information through retrieval, augmentation, and generation rather than relying only on conventional system metrics.

• Limits of traditional observability: Explain that CPU, memory, and request-latency metrics can reveal that an LLM application has a problem but cannot explain why it produces poor answers; breakpoints and stack traces likewise do not show how information shaped a response.
  Why it matters: Healthy system metrics do not necessarily mean the answers are useful or correct.
• Tracing the LLM information flow: Describe the visibility needed across retrieval, augmentation, and generation: which documents were retrieved and influenced the response, how context was assembled and interpreted, and where hallucinations or irrelevant answers arose.
  Why it matters: Tracing the information flow helps locate the stage behind a poor answer.

Key terms: traditional observability, CPU, memory, request latency, breakpoints, stack traces, LLM observability, reasoning chain, retrieval, augmentation

12.4.1 Set up Langfuse via Docker

This subsection explains how to run Langfuse locally with Docker and prepare it to observe DakkaBot.

• Local Langfuse setup sequence: Describe what the Docker deployment starts and the initial steps needed to prepare Langfuse for DakkaBot: open the dashboard, create an admin account and organization and project, configure team access, and generate API keys.
  Why it matters: These steps turn a running Langfuse instance into a project DakkaBot can use.
• Local deployment fit: Explain why the open-source Docker deployment is useful locally and where the section points readers for enterprise deployments.
  Why it matters: The deployment choice affects data control and how Langfuse fits existing infrastructure.

Key terms: Langfuse stack, web interface, database, worker processes, organization, project, open source, tracing, monitoring, Docker deployment

12.4.2 Integrating Langfuse with DakkaBot

This subsection explains how adding a Langfuse callback handler gives DakkaBot’s RAG pipeline automatic observability.

• Langfuse callback integration: Explain that, once Langfuse is running, adding a callback handler to DakkaBot’s RAG pipeline enables automatic tracking without extensive new code.
  Why it matters: The callback handler is the connection that makes the pipeline’s activity visible in Langfuse.
• Automatically captured pipeline activity: Describe the activity Langfuse tracks: LLM prompts, responses, tokens, and latency; retrieval embeddings and document metadata; chain steps and timing; errors with context; and estimated per-query API costs.
  Why it matters: These records show what happened across a RAG query, where it failed or slowed down, and what it may cost.

Key terms: Langfuse, callback handler, RAG pipeline, LLM calls, query embeddings, retrieved document metadata, chain execution, error traces, cost tracking

12.4.3 Enhanced observability in DakkaBotCore

This section instruments DakkaBotCore with Langfuse and shows how dashboard metrics and individual traces make its RAG pipeline measurable and debuggable.

• Automatic LangChain tracing: Explain how initializing a Langfuse callback handler in DakkaBotCore automatically traces retrieval, LLM generation, and other LangChain operations without adding tracing code throughout the application.
  Why it matters: A single integration point provides broad visibility without disrupting DakkaBot’s core functionality.
• Business context in traces: Describe how custom metadata adds query analysis, retrieval success, context utilization, and user-session information to the prompts, responses, token usage, and latency Langfuse captures automatically. Explain how that context helps teams improve retrieval and detect performance problems.
  Why it matters: Technical measurements become more useful when teams can relate them to the queries, documents, and sessions behind them.
• Dashboard performance and cost views: Describe how the Langfuse dashboard combines traces over time, model and token costs, and latency measurements for end-to-end requests, generation, and individual spans. Explain how these views help identify usage patterns, bottlenecks, and the cost implications of model choice.
  Why it matters: Aggregate views help teams balance user experience, operational health, and spending.
• Individual interaction trace detail: Trace what the detailed Langfuse view reveals about one DakkaBot request: its latency, input and output tokens, cost, and the conversation flow containing the system prompt, documentation context, and user question. Explain how seeing the supplied context and resulting interaction helps debug incorrect responses or unexpected behavior.
  Why it matters: An individual trace lets developers investigate what the model was actually given, rather than relying only on aggregate charts.

Key terms: Langfuse, DakkaBotCore, callback handler, LangChain, traces, custom metadata, query analysis, document retrieval success rates, context utilization patterns, user session information

12.4.4 Beyond traditional metrics

This section explains why LLM applications need quality-aware metrics, multistep traces, and token monitoring alongside traditional infrastructure metrics.

• Limits of traditional metrics: Explain why accuracy and latency alone cannot describe LLM application quality: responses can be partly correct, inconsistent, unsafe, or unhelpful, while perceived responsiveness differs from total response time.
  Why it matters: These limits determine what an LLM application needs to measure beyond conventional system health.
• Multistep RAG tracing: Trace a RAG query through retrieval, context assembly or augmentation, and generation, and explain how per-stage spans help locate failures and inspect timing and token use.
  Why it matters: A final bad answer does not reveal which stage caused the problem.
• Token use and cost monitoring: Describe how input/output token ratios, useful context utilization, cost by query type, and token waste reveal efficiency and cost problems that ordinary compute metrics miss.
  Why it matters: Token use varies with the question and context, so it affects both production cost and performance.
• Extended LLM system architecture: Explain that LLMOps extends rather than replaces Kubernetes, monitoring, and CI/CD, while a composite RAG system adds embeddings, retrieval, generation, interfaces, and specialized tools that need coordinated observability.
  Why it matters: The added components make RAG flexible but create more places for failures and debugging work.

Key terms: accuracy, latency, time-to-first-token, hallucination, user-perceived responsiveness, reasoning trace, span tracking, retrieval, augmentation, generation

12.5 Summary

The summary recaps how RAG separation, LLM-specific validation, observability, and workflow tools shape reliable LLM applications.