liveBook · Manning

Evaluation

As explained in Enterprise RAG

Evaluation is the systematic assessment of a system's behavior, quality, or performance against defined criteria.

Evaluation turns a RAG chatbot’s expected behavior into testable question-and-answer pairs. Rather than building for an undefined promise such as “answer company questions,” define the questions stakeholders need answered and the answers they consider correct. The resulting evals set measurable scope, expose missing requirements, and give developers a concrete target before implementation begins. section 2.0 section 2.1 section 2.2

An eval is a test case: a user question paired with an expected answer. Gather roughly 10–20 likely questions from customers, business stakeholders, or subject-matter experts, then expand the set as new supported query types emerge. Each addition is an explicit decision about what the system must handle, not an implicit expectation discovered after release. section 2.0 section 2.1 section 2.2

Use eval-driven development as an iterative delivery loop:

  • Define question-and-answer pairs before coding, starting with the simplest case. section 2.3
  • Identify the data needed to answer each question and implement the RAG behavior. section 2.1 section 2.5
  • Run each question, compare the generated answer with the expected answer, and revise prompts or code when it fails. section 2.3
  • Retest the full suite after every change to catch regressions in previously working answers. section 2.4
  • Include at least one eval for every data source so failures such as an unavailable product database become visible. section 2.4.2

Checking a product fact

For the question “What is Product XYZ’s weight?”, set “Product XYZ weighs 2.5 pounds” as the expected answer. Submit the question to the chatbot and compare its response with that expectation. A mismatch directs investigation toward the prompt or code, followed by another test run. section 2.3

Manual checking can work for a small set, but automated evals provide a safer production loop. A test can send every eval question through the RAG application, use an LLM to judge whether each generated answer conveys the expected information, and assert a pass result. Run that suite on changes so prompt adjustments, missing data, and unclear-query behavior do not silently degrade other answers. section 2.4 section 2.4.3

Figure 2.4 The automated eval system for RAG development. Evals compare chatbot answers to expected outputs and automate testing via GitHub Actions, ensuring high-quality performance.

This implementation view connects expected answers, chatbot output, automated comparison, and GitHub Actions into one repeatable evaluation system. section 2.4.3

Operationally, keep evaluation inputs secure and executable in continuous integration. The illustrated setup loads an OpenAI API key from environment variables rather than source code, runs Python tests with pytest, and supplies the key through repository secrets in a GitHub Actions workflow. section 2.4.3

How Enterprise RAG covers it

How deep it goes
Details an end-to-end workflow from eval design through LLM-based comparisons, pytest assertions, and CI automation.
Examples
Uses product facts, customer order history, and product-database availability as evaluation cases.
What it assumes
Assumes Python and Git for implementation while teaching eval design and automated-test setup from zero.
What you can do after
Build it from scratch as automated RAG tests that compare generated answers with expected answers and detect regressions.
How it connects
Connects evaluation to Retrieval Augmented Generation, test-driven development, prompts, CI/CD, GitHub Actions, pytest, and large language models.

Also explained in

Compare how 29 books explain evaluation →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Enterprise RAG. The section links point to the parts of the book it draws on.