Chapter 2: Evaluating generated responses

Introduction

This introduction explains why ordinary assertions and mocked responses cannot fully test generative AI behavior, then introduces Spring AI evaluators as a way to judge generated answers.

• Limits of conventional AI tests: Explain why repeated LLM calls cannot reliably be tested with exact-match assertions, and distinguish what a mocked response tests from what it leaves untested.
  Why it matters: It clarifies why a passing test with a mocked API response does not establish that a real generated answer is acceptable.
• Evaluator inputs and judgment: Describe what a Spring AI evaluator receives and what it decides, including the typical role of an LLM in making that judgment.
  Why it matters: Evaluators offer a way to assess an answer without requiring its wording to match a fixed expected string.

Key terms: nondeterministic, assert equals, WireMock, evaluator, user text, response content, ChatClient

2.1 Ensuring relevant answers

This section defines answer relevance and shows how to test it with Spring AI’s RelevancyEvaluator.

• Answer relevance versus correctness: Distinguish an answer that addresses the user’s question from one that is factually correct but off-topic. Explain that passing a relevance check does not establish factual accuracy.
  Why it matters: A relevance test checks whether the answer fits the question, not whether its claims are true.
• RelevancyEvaluator test flow: Trace how the integration test obtains an answer, submits the question and answer to RelevancyEvaluator, and checks its verdict.
  Why it matters: This test automatically checks whether the service’s generated answer fits the question it received.

Key terms: relevant, factual accuracy, BoardGameService, EvaluationRequest, RelevancyEvaluator, EvaluationResponse, isPass()

2.2 Testing for factual accuracy

This section explains why relevant answers still need fact-checking and how a Spring AI test can flag generated answers judged incorrect.

• Relevance versus factual accuracy: Distinguish an answer that addresses a question from one that answers it correctly, using the sky-and-blueberry-jam example.
  Why it matters: A response can be on topic without being safe to present as a correct answer.
• Fact-checking test behavior: Describe how the section uses FactCheckingEvaluator in a test and what happens when it judges a generated answer incorrect.
  Why it matters: It connects a factual judgment to a test failure a developer can investigate.
• Tests during prompt changes: Explain why the section recommends keeping relevancy and factual-accuracy tests while changing prompts.
  Why it matters: Prompt changes should not silently undermine the quality of generated answers.

Key terms: RelevancyEvaluator, FactCheckingEvaluator, factual accuracy, EvaluationResponse, isPass(), evaluateFactualAccuracy(), evaluateRelevancy(), prompt

2.3 Applying self-evaluation at runtime

This section shows how to check answer relevance at runtime, retry failed answers, and recover when retries are exhausted.

• Runtime answer evaluation: Explain why passing evaluator-based tests does not guarantee relevant or correct answers in production, and why evaluators can also be used at runtime.
  Why it matters: A passing test is not a guarantee about every answer users will receive.
• Relevance retry and recovery: Trace how the service evaluates an answer, retries after a relevance failure, and returns a fallback after retries are exhausted.
  Why it matters: A failed relevance check leads to another attempt rather than immediately returning the bad answer.
• Retry limit tradeoffs: Describe the default retry limit, how maxAttempts changes it, and why the section recommends keeping it low.
  Why it matters: More chances to obtain a relevant answer also mean more LLM requests.
• Custom evaluator option: State how to address evaluation needs not covered by Spring AI’s supplied evaluators.
  Why it matters: The built-in checks are not the only evaluation options available to an application.

Key terms: evaluators, runtime, nondeterministic, RelevancyEvaluator, isPass(), AnswerNotRelevantException, @Retryable, @Recover, maxAttempts, tokens

2.4 Summary