Chapter 13: Production LLM system design

Introduction

This chapter introduction explains why production LLM applications need disciplined prompt management, semantic evaluation, safety controls, and cost monitoring beyond traditional ML practices.

• Probabilistic quality assurance: Explain why nondeterministic LLM responses require evaluation of meaning and suitability rather than exact-output assertions. Describe the response qualities the chapter proposes assessing.
  Why it matters: A response can be valid even when its wording changes, so production tests must judge more than an exact match.
• Prompts as production infrastructure: Describe the chapter’s approach to prompts as critical infrastructure that needs version control and systematic testing, with Langfuse named as a prompt-management tool.
  Why it matters: Managing prompts deliberately helps make an LLM application maintainable and reliable beyond the prototype stage.
• Production safety and operational discipline: Describe the chapter’s planned safety guardrails against harmful content and prompt injection, and explain why reliability, safety, and cost control matter for LLM systems serving real users.
  Why it matters: For real user workloads, unsafe behavior and uncontrolled costs can directly affect business outcomes.

Key terms: nondeterministic, probabilistic systems, assertion-based testing, semantic quality, DeepEval, prompt engineering workflows, version control, systematic testing, Langfuse, safety guardrails

13.1 Prompt engineering: Code for the generative AI era

Introduces prompts as production business logic that needs code-like discipline and managed versioning.

• Prompts as business logic: Explain that important LLM application behavior is specified in natural-language prompts rather than only in conventional code, so prompts need versioning, testing, and systematic optimization.
  Why it matters: A prompt change can change application behavior, so it deserves the same care as a code change.
• Managed prompt assets: Describe how Langfuse prompt management replaces hardcoded prompt strings with versionable assets that can be updated without a code deployment, A/B tested across environments, and tracked for performance by version.
  Why it matters: Managed prompts let teams change and compare LLM behavior without tying every prompt edit to a software release.

Key terms: business logic, prompts, versioning, testing, systematic optimization, Langfuse, prompt management, hardcoded strings, versionable assets, A/B tested

13.1.1 Treating prompts as critical infrastructure

This section shows why production system prompts need explicit roles, boundaries, response rules, and safety instructions by comparing a vague DakkaBot prompt with a production-ready one.

• System prompt responsibilities: Explain how a system prompt defines an LLM application's persona, constraints, output style, task context, and operational knowledge, and how these instructions prevent inconsistent or unsafe behavior.
  Why it matters: These responsibilities make the prompt part of the application's reliability and safety infrastructure.
• Vague baseline prompt risks: Describe why DakkaBot's version 1.0 prompt is unsuitable for production: its generic role and unspecified documentation invite out-of-scope or fabricated answers, while absent safety constraints leave sensitive information unprotected.
  Why it matters: The baseline makes clear which failures explicit production instructions are meant to prevent.
• Production prompt architecture: Explain how version 2.0 replaces vague guidance with a specific DataKrypt documentation role, documentation-only answers and uncertainty handling, structured responses with citations and confidence indicators, and explicit safety rules. Describe why organizing these rules into sections makes the prompt easier to extend.
  Why it matters: Explicit, extensible rules help DakkaBot answer consistently without guessing or mishandling sensitive requests.

Key terms: system prompt, persona and role, constraints and guardrails, output format and style, context and task objectives, operational knowledge, version 1.0, helpful assistant, company documentation, scope creep

13.1.2 Langfuse prompt management for DakkaBot

This section shows how to move DakkaBot’s prompts into Langfuse and use them at runtime for versioned deployment, tracking, and graceful fallbacks.

• Prompts as managed infrastructure: Explain why DakkaBot stores prompts in Langfuse instead of hardcoding them: teams can revise, version, audit, and promote prompts without changing application code or redeploying it.
  Why it matters: Separating prompts from code lets teams improve LLM behavior without tying every prompt change to a software release.
• Prompt templates and deployment settings: Describe how a Langfuse prompt is defined by name and type, uses runtime template variables, and carries labels and model configuration alongside its content. Explain that labels select versions for environments such as production, while configuration keeps behavioral settings with the prompt.
  Why it matters: These settings make one prompt manageable across environments and keep its intended model behavior attached to its version.
• Runtime prompt resolution: Trace how DakkaBotCore keeps initialization minimal, fetches the prompt bearing the production label during process_query, and applies that prompt’s configuration to the LLM at runtime.
  Why it matters: Runtime resolution is what lets a label change or configuration update affect DakkaBot without an application restart.
• Compiled responses and version tracing: Explain how DakkaBot compiles retrieved templates with runtime context and query values, then records the prompt name and version with the LLM response so teams can relate quality, satisfaction, and cost to prompt changes.
  Why it matters: Version-linked traces make prompt improvements measurable rather than purely subjective.
• Boundary prompt fallback: Describe the specialized boundary prompt and how DakkaBot uses it when the primary prompt or system fails, substituting details such as topic and contact channel to give the user helpful guidance instead of a cryptic error.
  Why it matters: A managed fallback preserves a useful user experience when the normal answer path cannot succeed.

Key terms: managed prompts, versioning, audit trail, labels, name, type, chat prompt, {{context}}, {{query}}, DakkaBotCore

13.1.3 Langfuse prompt management for production

The section presents Langfuse as production infrastructure for managing, deploying, testing, and reliably retrieving DakkaBot prompts without hardcoding them in application code.

• Centralized prompt infrastructure: Explain how Langfuse replaces hardcoded prompt strings with a centralized prompt management system that separates prompts from application code and supports versioning and rollback.
  Why it matters: Separating prompts from code makes them easier to update and manage as the application grows.
• Label based prompt deployment: Describe how labels assign prompt versions to environments such as production and staging, and how reassigning a label promotes a version without code changes or application restarts.
  Why it matters: Labels let a team test new prompts while keeping production on a stable version.
• Collaborative prompt change history: Explain how nontechnical colleagues can edit prompts through Langfuse’s web interface while engineers work on the RAG pipeline, and how the audit trail records changes.
  Why it matters: Collaboration and change history make prompt updates easier to coordinate and investigate.
• Prompt variant performance testing: Describe how Langfuse A/B testing routes portions of traffic to simultaneous prompt variants and uses tracing to compare their effects on response quality, user satisfaction, and business metrics.
  Why it matters: Measured results provide a stronger basis for prompt decisions than subjective impressions alone.
• Cached prompt availability: Explain how client-side caching avoids repeated remote-fetch latency, background refresh keeps prompts current without blocking requests, and fallback prompts or prefetching help during network problems.
  Why it matters: Remote prompt management must remain responsive and available when users need a response.
• Single prompt migration path: Describe the suggested way to begin: move one DakkaBot prompt into a cloud-hosted or self-hosted Langfuse setup, then explore composability, message placeholders, and the playground.
  Why it matters: Starting with one prompt provides a manageable way to try the production-management features.

Key terms: Langfuse, Content Management System (CMS), versioning, rollback, labels, production, staging, experimental, prompt versions, web interface

13.2 Testing LLM applications

This section explains why variable but valid LLM responses require evaluation beyond exact-output tests.

• LLM testing criteria: Explain why exact-match tests are inadequate for LLM applications and describe the kinds of checks needed instead: semantic correctness, factual accuracy, and adherence to guidelines.
  Why it matters: A response can differ from an expected string and still be valid—or sound plausible while being inaccurate.

Key terms: deterministic software, exact matches, semantic correctness, factual accuracy, adherence to guidelines

13.2.1 Evaluation framework for LLM responses

This section shows how to test DakkaBot responses with DeepEval and G-Eval, using an LLM judge to assess security behavior and multiple quality dimensions rather than exact wording.

• Semantic LLM response evaluation: Explain how DeepEval uses G-Eval and a configured judge LLM such as Gemini to score DakkaBot responses against natural-language criteria rather than exact strings or surface-level similarity metrics.
  Why it matters: LLM responses can vary in wording while still meeting—or violating—the same quality requirement.
• Synchronous DakkaBot test helper: Describe how the test helper lets synchronous DeepEval tests call DakkaBotCore's async query method, reuses a module-level DakkaBot instance, and returns response text for evaluation.
  Why it matters: The helper keeps async execution and repeated setup from obscuring the evaluation tests.
• Security deflection evaluation criteria: Explain how a G-Eval test checks a credential-harvesting question for deflection, absence of sensitive-information leakage, and redirection to appropriate channels, regardless of the response's exact wording. State why any leak must be an explicit failure rather than a partly passing score.
  Why it matters: A refusal is not safe if it also discloses credentials or infrastructure hints.
• Multiple response quality metrics: Describe testing a deployment-troubleshooting answer across factual correctness, relevance, completeness, and professional tone, using the question as well as the answer where needed and setting thresholds by dimension.
  Why it matters: Improving one quality dimension should not conceal an irrelevant, incomplete, or otherwise poor response.
• Judge reliability and evaluation costs: Explain that G-Eval's scores depend on well-defined criteria and the judge model's capabilities, while additional LLM calls add cost and latency; describe the trade-off between a stronger, slower judge and a cheaper, less nuanced one.
  Why it matters: A repeatable evaluation is useful only if its judge and criteria are dependable enough for the risk being tested.

Key terms: DeepEval, G-Eval, LLM-as-Judge, Gemini, BLEU, ROUGE, DakkaBotCore, process_query, asyncio.run(), module-level DakkaBot instance

13.2.2 Safety and adversarial testing

The section explains why RAG assistants need adversarial security testing and shows how to use Promptfoo’s generated attacks and results to strengthen DakkaBot.

• RAG prompt injection risks: Explain how a RAG assistant’s user queries influence retrieved context and how instructions embedded in those queries can redirect the assistant beyond its intended role. Describe the pirate-persona and competitor-comparison attempts against DakkaBot.
  Why it matters: Understanding where the attack enters helps you recognize failures that ordinary software security checks may miss.
• Promptfoo testing integration: Describe the testing workflow: configure Promptfoo with DakkaBot’s prompt structure and connection details, generate adversarial queries, and use the Python provider’s call_api function to route them through DakkaBot’s normal query pipeline for evaluation.
  Why it matters: Testing the real query path makes the security assessment relevant to how users actually interact with DakkaBot.
• Adversarial test categories: Distinguish Promptfoo plugins, which target particular vulnerabilities, from strategies, which apply general attack methods. Give examples such as hallucination or excessive-agency plugins and jailbreak or composite-jailbreak strategies.
  Why it matters: The two approaches help test both specific ways DakkaBot might fail and broader methods attackers might use.
• Results driven security improvement: Explain how to use Promptfoo’s summary dashboard and individual failed cases to find vulnerability patterns, update guardrails or prompt constraints, and retain discovered failures as regression tests. Contrast this repeatable process with relying only on a few manual attacks.
  Why it matters: A failed attack test becomes useful when it leads to a specific fix and a check that the failure does not return.

Key terms: RAG systems, prompt injection, knowledge boundary violations, DakkaBot, Promptfoo, promptfooconfig.yaml, promptfoo_provider.py, call_api, provider interface, plugins

13.3 Governance and safety in production

This section introduces the need for safety throughout a production LLM pipeline and for governance that balances innovation with risk.

• Pipeline-wide LLM safety: Explain why production LLM applications need multiple layers of safety operating at every stage of the pipeline.
  Why it matters: Safety cannot be treated as a single check added at the end of an LLM application.
• LLM governance tradeoff: Describe the potential consequences of LLM safety failures and the balance a governance framework must strike between innovation velocity and risk management.
  Why it matters: The possible harms explain why production governance has to manage risk without stopping innovation.

Key terms: multilayered safety systems, pipeline, sensitive data, harmful content, compliance requirements, innovation velocity, risk management

13.3.1 Implementing safety guardrails

This section shows how to add layered, risk-appropriate safety checks to DakkaBot’s inputs, model processing, and outputs, then test responses for factual accuracy.

• Multilayered guardrail architecture: Explain how input guards, model-level processing controls, and output guards protect an LLM application at different points, and why one layer alone is insufficient.
  Why it matters: A later guard can catch a violation that an earlier one missed.
• Risk-based input failure handling: Describe how DakkaBot validates inputs and chooses between blocking a request and sanitizing it, using competitor mentions and PII as examples.
  Why it matters: The failure action determines whether a risky request stops or can proceed safely.
• Validator selection trade-offs: Explain how the Guardrails AI Hub helps select prebuilt validators and how rule-based, local-ML, and LLM-based checks differ in capability, latency, and cost.
  Why it matters: Choosing the lightest adequate validator helps add protection without unnecessary expense or delay.
• Model filters and output guards: Distinguish Gemini’s built-in safety settings from application-specific output validation, and describe what DakkaBot checks before delivering a response.
  Why it matters: General model safety filters do not enforce an organization’s documentation or business boundaries.
• Documentation-grounded hallucination tests: Describe how DakkaBot can test generated answers against verified documentation and preserve verified question-answer pairs as regression tests.
  Why it matters: Repeated factual checks can reveal fabrications after a prompt or model changes.

Key terms: input guards, processing controls, output guards, multilayered safety guardrails, Guardrails AI, CompetitorCheck, GuardrailsPII, on_fail, OnFailAction.EXCEPTION, OnFailAction.FIX

13.4 Cost optimization strategies

Introduces why production LLM systems need deliberate cost management by linking usage-based token charges to prompts, retrieved documents, and generated responses.

• Token-Based Production Costs: Explain that production LLM costs depend on actual token consumption rather than only on infrastructure, and that prompts, retrieved documents, and generated responses all contribute to the bill. Describe why cost optimization matters alongside performance and reliability.
  Why it matters: Recognizing where token costs arise helps you make cost a deliberate part of production system design.

Key terms: token consumption, prompts, retrieved documents, generated responses, cost management

13.4.1 Understanding LLM economics

This section explains how token usage, model choice, deployment strategy, and traffic peaks shape the cost of a production LLM application.

• Token based cost variability: Explain that managed LLMs commonly charge for tokens processed, so a long or complex DakkaBot request can cost far more than a simple one and user behavior can make spending unpredictable.
  Why it matters: Token-based pricing makes the size and frequency of requests important production costs.
• Task matched model selection: Explain why model choice should follow task requirements rather than assuming the most capable, expensive, or proprietary model is always best. Contrast a stronger reasoning model with a cheaper model that handles routine queries, while noting that specialized models may be better for particular tasks.
  Why it matters: Matching capability to the task can preserve useful quality without paying premium prices for every request.
• Deployment economics and scale: Distinguish model licensing from deployment strategy: managed pay-per-token endpoints can serve open-source or proprietary models without infrastructure overhead, while self-hosting requires reserved compute and operational work. Explain why managed endpoints suit DakkaBot’s roughly 30,000 monthly queries, whereas self-hosting can become economical only when much higher usage amortizes its costs.
  Why it matters: An inexpensive model is not necessarily an inexpensive system once serving infrastructure is included.
• GPU capacity and traffic peaks: Describe how large-model GPU memory and low-latency serving make self-hosted infrastructure costly, and how traffic spikes force a choice between paying for idle peak capacity and accepting slower responses. Contrast this with managed pay-per-token costs rising with usage.
  Why it matters: Planning from average traffic alone can substantially understate the cost of self-hosted LLM serving.

Key terms: tokens, pay-per-token, input tokens, output tokens, model selection, GPT-4, Gemini Flash, specialized models, open-weight models, managed endpoints

13.4.2 Model selection strategy

This section shows how DakkaBot can choose an LLM at runtime according to query complexity to reduce costs, while managing the quality and operational risks of routing.

• Complexity based model routing: Explain how DakkaBot classifies each query as SIMPLE, MODERATE, or COMPLEX and selects a cheap, balanced, or premium model at runtime. Describe why its routing prompt is kept short.
  Why it matters: Matching model capability to the task avoids paying premium-model prices for every query.
• Routing savings and tradeoffs: Explain the potential cost savings from tiered routing and the quality and operational risks it introduces. Describe how routing accuracy, threshold experiments, and ongoing cost-versus-complexity evaluation help manage those risks.
  Why it matters: A cheaper model mix is worthwhile only if routing preserves answer quality and the savings justify managing multiple models.

Key terms: CostOptimizedDakkaBotCore, router_llm, generation_llm, premium_llm, route_query(), process_query_optimized(), tiered routing, routing accuracy metrics, A/B test routing thresholds, multiple model endpoints

13.4.3 Caching strategies

This section explains how context and semantic caching reduce repeated LLM work, when their savings outweigh their costs, and how RedisVL can implement semantic caching.

• Context caching tradeoffs: Explain how reusing processed document context across queries can reduce input-token costs, and describe when that reuse is or is not worthwhile.
  Why it matters: Context caching saves money only when the same material is reused enough to justify storing and maintaining it.
• Semantic cache matching tradeoffs: Distinguish semantic caching from exact-text caching, and explain how similarity thresholds, freshness, and lookup overhead affect whether a cached answer should be reused.
  Why it matters: A semantic cache must save LLM calls without returning an answer to the wrong question.
• RedisVL semantic cache workflow: Trace how the RedisVL example stores and retrieves query-response pairs using embeddings and vector similarity search.
  Why it matters: The workflow connects the caching idea to a production-oriented lookup and storage design.

Key terms: context caching, cached context, cache invalidation, semantic caching, embeddings, similarity threshold, semantic drift, time-to-live (TTL), RedisVL, Redis hash

13.4.4 Prompt optimization for efficiency

This section shows how to reduce LLM prompt token costs without sacrificing response quality, safety, or consistent behavior.

• Token cost and scale: Explain why prompt length is a production cost lever: every prompt token adds cost, and small per-query increases accumulate across many daily interactions.
  Why it matters: A few unnecessary tokens can become a substantial recurring cost at scale.
• Concise prompt construction: Describe how to shorten a prompt by compressing identity, consolidating instructions, simplifying input/output formatting, and removing redundant wording while preserving clear behavioral guidance.
  Why it matters: Concision saves tokens only when the model still receives the instructions it needs.
• Prompt optimization validation: Explain why a shortened prompt must be tested against evaluation criteria, including response quality, safety compliance, and behavioral consistency, rather than judged by token reduction alone.
  Why it matters: A cheaper prompt is a false economy if it produces worse or less safe responses.

Key terms: token consumption, operational costs, prompt efficiency, identity compression, instruction consolidation, format simplification, redundancy elimination, Q:/A:, A/B testing, evaluation criteria

13.4.5 Production cost monitoring

This section explains how to monitor production LLM costs, catch budget risks, and use Langfuse to judge whether spending delivers value.

• Production cost visibility: Explain why production LLM costs need proactive monitoring beyond monthly bills: spending can change with user behavior, query complexity, and model choice, so teams need timely insight to keep costs predictable.
  Why it matters: Seeing costs as they change gives a team a chance to act before spending becomes a problem.
• Budgets and expensive outliers: Describe monitoring aggregate spending alongside per-query costs, using alerts at 80% of budget thresholds and breakdowns by query type, model, and user to investigate outliers before hard limits are reached.
  Why it matters: Budgets show overall risk, while query-level detail shows where a team can intervene.
• Langfuse cost and value analysis: Explain how Langfuse captures token usage, estimated cost, and performance for each query, enabling teams to inspect expensive traces, compare prompt versions and trends, and relate cost to quality and user or business value.
  Why it matters: A costly query is worth optimizing only after its expense is considered alongside the quality and value it delivers.

Key terms: production cost monitoring, real-time insights, spending patterns, aggregate spending, per-query costs, expensive outliers, cost alerts, budget thresholds, Langfuse, token usage

13.4.6 From traditional ML to LLMOps

This section explains how LLMOps builds on established MLOps engineering practices while adapting them to the distinctive demands of production LLM systems.

• MLOps foundations for LLMOps: Explain why LLMOps is an evolution of MLOps rather than a replacement: LLM applications still need robust platforms, monitoring, automation, systematic testing, and continuous improvement.
  Why it matters: The practices used to make traditional ML reliable remain the foundation for production LLM systems.
• Prompts and multistep systems: Describe why production LLM systems must treat prompts as code and coordinate retrieval, augmentation, and generation rather than relying on a single model call. Explain that failures can arise at each step, calling for holistic monitoring and graceful degradation.
  Why it matters: Reliable LLM applications depend on both controlled prompts and visibility across the whole workflow.
• Probabilistic quality and safety: Explain why variable LLM responses call for evaluation beyond exact assertions, including semantic quality, factual accuracy, and safety. Describe the need for multiple levels of guardrails against hallucination, bias, prompt injection, and harmful content.
  Why it matters: A response can be differently worded yet valid—or plausible-sounding yet unsafe or false.
• Usage-driven LLM costs: Distinguish token-based LLM costs, which depend on usage patterns and prompt efficiency, from traditional infrastructure-based compute costs. Explain why cost optimization belongs alongside latency and throughput in production design.
  Why it matters: A technically reliable LLM application must also remain affordable as real usage grows.

Key terms: LLMOps, MLOps, automation, observability, systematic testing, continuous improvement, prompt engineering as code, version control, deployment pipelines, retrieval

13.5 Summary

The chapter summary brings together the evaluation, prompt, safety, cost, and infrastructure practices needed to run LLM applications in production.