6 Communication strategies: Disclaimers, feedback, and prompt tuning

 

This chapter covers

  • Writing a clear disclaimer and user guide
  • Collecting user feedback from the front‑end application
  • Using that feedback to improve agent prompts
  • Responding constructively to user feedback

They say, “No plan survives contact with the enemy,” and no program’s first draft survives contact with real users. It was quite a shock to me, after spending 8 months building a RAG chatbot, when my first test users arrived. They expected it to know things it could not possibly know, like which product is the most popular in a list of products. They expected it to know things that are not in the product’s description, and are not written down anywhere. It seems like these people expected the chatbot to be able to read their minds! It was then that I realized: I needed to set their expectations up-front. AI is hyped a lot, and not everybody understands it well. Some people, understandably so, have expectations that differ from reality. If the people who will trust your system to help them do their work do not understand what the chatbot can and cannot do, or if they feel that their concerns go unheard, the technical excellence of your code will not matter. Communication bridges the gap between the system’s capabilities and the user’s expectations, ensuring that both evolve together instead of pulling in opposite directions.

6.1 Disclaimer message and user guide

6.2 Introduction to user feedback

6.3 Building a user feedback system part 1: CosmosDB

6.4 Building a user feedback system part 2: Function App

6.5 Why not direct? Why not use our web app?

6.6 App setup

6.7 Code

6.8 Deploying the function app

6.9 Environment variables

6.10 Trying out our feedback loop

6.11 Ask a question and give negative feedback

6.12 Check the logs and diagnose the problem

6.13 Update the prompt and/or your code

6.14 Redeploy and try it again

6.15 Email the user

6.16 A fully fleshed-out writer prompt

6.17 Responding (nicely) to dumb user feedback

6.18 Summary