Chapter 8: Applications and Agents: Building an interactive experience

Introduction

The chapter introduces the shift from building and deploying LLMs to creating user-facing applications, including edge applications and agents.

• Delightful LLM application experience: Explain that the chapter brings together training, deployment, and prompting to build an application using a deployed LLM service, and that making it useful and delightful for users takes more than creating a simple interface.
  Why it matters: A working LLM application is only the starting point for a good user experience.
• Edge application setting: Describe edge devices as another place an LLM application may run, including devices without a GPU.
  Why it matters: Where an application runs changes the setting in which it must work.
• Agent roles and multistep problems: Distinguish the chapter’s proposed LLM agents from applications that answer a single request: agents can fulfill a role and solve multistep problems.
  Why it matters: This distinction explains why the chapter moves beyond a conventional request-and-response application.

Key terms: LLM service, delightful experience, application, edge devices, GPU, LLM agents, multistep problems, role, request

8.1 Building an application with RAG

Defines an LLM application as the user-facing client and explains why a simple chat interface needs additional features to become a good user experience.

• Application and service boundary: Distinguish the LLM application—the frontend client users interact with—from the LLM service it calls, and give examples such as a web app, phone app, CLI, SDK, or VS Code extension.
  Why it matters: This distinction keeps the focus on the user-facing application rather than the service behind it.
• Basic chatbox simplicity: Describe the basic chatbox as an input field, send button, and conversation text field, and explain why a bot participant removes some complications of a chat between people.
  Why it matters: A simple chatbox is easy to build, so the harder work is making the experience good.
• Polished application features: Explain that prompt composition and experimentation are a starting point, while a better LLM application also uses features such as streaming responses, chat history, and behind-the-scenes handling of prompts and responses.
  Why it matters: Those finishing touches can improve results and user satisfaction beyond what a basic chatbox provides.

Key terms: LLM application, frontend, LLM Service, client, chatbox, input field, send button, text field, timeout error, streaming responses

8.1.1 Streaming on the frontend

This section shows how a minimal browser app displays an LLM response as it arrives, and why client-side streaming improves the chat experience.

• Client-side streaming flow: Trace how the browser sends a prompt, receives the server’s streaming response as a JavaScript ReadableStream, and iterates over the results to show generated text to the user.
  Why it matters: Server-side streaming only becomes visible to users when the frontend displays the arriving text.
• Streaming user experience: Explain why showing text during generation feels more responsive than waiting for the complete answer and lets users judge where the output is going before it finishes.
  Why it matters: The benefit of streaming is not just technical: immediate visible progress can keep users engaged and help them decide whether to reprompt.
• Minimal browser demo structure: Describe the deliberately simple HTML, CSS, and vanilla JavaScript app: a prompt form that does not refresh the page, a chat-message container, and JavaScript that adds messages and handles the stream.
  Why it matters: A plain HTML-file demo isolates the streaming behavior without requiring a frontend framework.

Key terms: StreamingResponse, ReadableStream, sendToServer, client side, streaming on the client side, instant feedback, stop generation, reprompt, HTML, CSS

8.1.2 Keeping a history

The section adds conversation history to an interactive LLM app and explains how to manage that history when it becomes too long.

• Conversation history as context: Explain why sending only the latest user message leaves an LLM without the context needed for follow-up questions, and how including earlier user prompts and LLM responses helps it respond.
  Why it matters: Follow-up questions often depend on things said earlier in the conversation.
• Streamlit chat history workflow: Describe how the example Streamlit app accepts chat input, displays conversation messages, stores chat history in session_state, and uses that history to craft the next prompt while streaming responses.
  Why it matters: These pieces turn the history idea into an interactive application.
• Long history tradeoffs: Explain why growing chat history must be shortened, and contrast dropping older messages with using an LLM-generated summary as context.
  Why it matters: Keeping every turn can exceed the model’s input limit and slow generation, while shortening history risks losing useful details.
• Relevant message retrieval: Describe the section’s alternative of embedding chat messages and searching for relevant earlier messages to include as prompt context.
  Why it matters: Selecting relevant past messages offers another way to use a limited context budget.

Key terms: chat history, prompts, responses, context, Streamlit, chat_input, chat_message, session_state, chat_history, streaming

8.1.3 Chatbot interaction features

The section explains interaction features that make LLM chatbots more usable and shows how Gradio can provide a streaming chat interface with several controls built in.

• Fallbacks and orderly submissions: Explain how a fallback response handles a failed request and why Submit should be disabled while a response is in progress.
  Why it matters: These safeguards keep the conversation understandable when requests fail or take time.
• Midstream response control: Explain what Stop does during a streamed response and why it improves the experience and can reduce cost.
  Why it matters: Stopping an unhelpful answer saves the user time and avoids generating unnecessary tokens.
• Retry and history deletion: Distinguish retrying the last query from deleting parts of chat history, and explain how each helps prevent an unfavorable answer from shaping later responses.
  Why it matters: Earlier answers become context for later prompts, so users need ways to correct the conversation's direction.
• User feedback for improvement: Describe what a feedback form collects, how teams can use the results, and why training feedback needs cleaning first.
  Why it matters: Feedback can reveal failures and guide improvements even when a team does not train its own model.
• Gradio chat interface setup: Describe how the section uses Gradio's ChatInterface to build a chatbot and obtain streaming and interaction controls without implementing the whole interface manually.
  Why it matters: A UI library can supply common chatbot controls while the developer concentrates on the model request.

Key terms: fallback response, Submit button, chat history, Stop button, output per token, Retry button, Delete button, temperature, soft delete, feedback form

8.1.4 Token counting

Token counting helps an LLM application warn users before prompts exceed limits and adjust how much retrieved context it adds.

• Prompt limits and early feedback: Explain why an application should count submitted tokens and give users feedback before they submit a prompt that exceeds the model’s token limit.
  Why it matters: Early feedback prevents users from spending time writing a query that the model cannot accept.
• Token-aware retrieval context: Describe how a Q&A bot can use the length of a user’s question to adjust RAG retrieval: return more search results for a short question, but limit a long question so there is room to append context.
  Why it matters: The user’s question and retrieved material must share the available prompt space.
• Tokenizer-matched token counters: Describe counting tokens by encoding a string and counting the resulting tokens, using fast OpenAI-oriented tiktoken where appropriate. Explain why its count may be inaccurate for another model’s tokenizer and why a different tokenizer, such as SentencePiece, may call for a custom counter.
  Why it matters: A fast count is useful only if its accuracy is suitable for the model whose limit the application must enforce.

Key terms: submitted tokens, token limits, prompts, Q&A bot, search results, retrieval-augmented generation (RAG), context, tiktoken, BPE tokenizer, OpenAI’s encoders

8.1.5 RAG applied

This section contrasts frontend and backend RAG, then shows how to use an existing vector store to augment queries in an LLM application.

• Frontend and backend RAG tradeoffs: Distinguish frontend from backend RAG: backend setup gives developers more control over context use, a consistent experience for users, and more protection for vector-database data, though prompt injection remains a risk; frontend setup lets an application add business context to a generic LLM at runtime.
  Why it matters: Where RAG runs affects both the application experience and who can access its data.
• Runtime business context: Explain that frontend RAG can give a generic LLM business context or dataset content at runtime without finetuning, adding personality and functionality to an application as well as helping accuracy and reducing hallucinations.
  Why it matters: Runtime context can make an available model useful for a particular business without training it on that business’s dataset.
• Vector store query augmentation: Describe the section’s frontend example: it accesses the previously set-up vector store and uses LangChain with OpenAI and Pinecone for query augmentation; once the LLM and vector-store connections are defined, the application can start making queries.
  Why it matters: The example shows how little connection setup is needed to put an existing RAG system to use in an application.

Key terms: RAG, frontend, backend, vector database, prompt injection, business context, finetune, run time, hallucinations, query augmentation

8.2 Edge applications

This section explains why running an LLM inside an edge application is challenging and demonstrates a llama.cpp workflow for obtaining and running a quantized model locally.

• Edge deployment constraints: Explain why an application might run an LLM on the user's device instead of calling an API, and describe the resulting model-size and local-runtime constraints.
  Why it matters: These constraints determine which models and runtime tools are practical for an edge application.
• GGUF model preparation: Describe llama.cpp's edge-deployment flow: obtain a model in GGUF format, move it to the device, install bindings for the application's language, and run it. Distinguish converting a model yourself from downloading one that is already converted.
  Why it matters: The GGUF model and language bindings are the pieces the section uses to make local inference possible.
• Quantized model demonstration: Trace the section's example of downloading an already converted WizardCoder-7B GGUF model, selecting its small 2-bit version, and loading and running it with llama-cpp-python. Explain the size-versus-output-quality tradeoff of that quantization.
  Why it matters: The example makes the edge workflow concrete while showing why the smallest model is not necessarily the best-quality choice.

Key terms: edge device, local environment, accelerator, GPU, llama.cpp, GGUF, GGML, bindings, convert script, WizardCoder-7B

8.3 LLM agents

This section explains how LLM agents combine a model, memory, and tools to attempt multistep tasks, then demonstrates how to build them and why they remain unreliable.

• Agent components and purpose: Explain that an agent uses an LLM within a supporting system to accomplish multistep tasks, rather than merely answer multistep questions. Describe its three components: the LLM, memory of earlier steps, and external tools that let it retrieve information or take actions.
  Why it matters: The agent’s capabilities come largely from the system around the model, not a fundamentally different kind of model.
• Agent memory design choices: Distinguish a memory buffer, memory summarization, and structured memory storage by how each supplies prior information to the LLM. Explain why the section recommends structured storage despite its setup difficulty.
  Why it matters: An agent needs useful information from earlier steps without exhausting its context or losing important details.
• Tool based agent setup: Trace the examples from LangChain search tools through locally run Python and CSV agents to the custom agent. Describe how the custom agent is given tools and memory, then a system prompt explaining tool use, with few-shot examples and instructions before it runs.
  Why it matters: The examples show how tools and prompting are assembled into an agent that can attempt tasks.
• Agent reliability and performance limits: Explain why the section treats current agents as promising but unreliable, using the flawed Python and CSV outputs or the custom agent’s near miss as evidence. Describe why chaining tasks magnifies LLM flakiness and note the reported latency cost of LangChain’s wrapper.
  Why it matters: An agent’s ability to attempt a task does not mean it completes that task correctly or quickly.

Key terms: LLM agents, memory, external data retrieval tools, memory buffer, memory summarization, structured memory storage, context limits, lost in the middle, LangChain, Duck Duck Go search tool

8.4 Summary

The chapter summary identifies the features that make LLM applications more usable, then highlights edge deployment and the promise and difficulty of agents.