Evaluation turns a RAG chatbot’s expected behavior into testable question-and-answer pairs. Rather than building for an undefined promise such as “answer company questions,” define the questions stakeholders need answered and the answers they consider correct. The resulting evals set measurable scope, expose missing requirements, and give developers a concrete target before implementation begins. section 2.0 section 2.1 section 2.2
An eval is a test case: a user question paired with an expected answer. Gather roughly 10–20 likely questions from customers, business stakeholders, or subject-matter experts, then expand the set as new supported query types emerge. Each addition is an explicit decision about what the system must handle, not an implicit expectation discovered after release. section 2.0 section 2.1 section 2.2
Use eval-driven development as an iterative delivery loop:
- Define question-and-answer pairs before coding, starting with the simplest case. section 2.3
- Identify the data needed to answer each question and implement the RAG behavior. section 2.1 section 2.5
- Run each question, compare the generated answer with the expected answer, and revise prompts or code when it fails. section 2.3
- Retest the full suite after every change to catch regressions in previously working answers. section 2.4
- Include at least one eval for every data source so failures such as an unavailable product database become visible. section 2.4.2
Checking a product fact
For the question “What is Product XYZ’s weight?”, set “Product XYZ weighs 2.5 pounds” as the expected answer. Submit the question to the chatbot and compare its response with that expectation. A mismatch directs investigation toward the prompt or code, followed by another test run. section 2.3
Manual checking can work for a small set, but automated evals provide a safer production loop. A test can send every eval question through the RAG application, use an LLM to judge whether each generated answer conveys the expected information, and assert a pass result. Run that suite on changes so prompt adjustments, missing data, and unclear-query behavior do not silently degrade other answers. section 2.4 section 2.4.3
Figure 2.4 The automated eval system for RAG development. Evals compare chatbot answers to expected outputs and automate testing via GitHub Actions, ensuring high-quality performance.
This implementation view connects expected answers, chatbot output, automated comparison, and GitHub Actions into one repeatable evaluation system. section 2.4.3
Operationally, keep evaluation inputs secure and executable in continuous integration. The illustrated setup loads an OpenAI API key from environment variables rather than source code, runs Python tests with pytest, and supplies the key through repository secrets in a GitHub Actions workflow. section 2.4.3




























