RelevancyEvaluator is a Spring AI evaluator that asks a language model whether an answer addresses the submitted question. It distinguishes an on-topic answer from a factually correct but unrelated response. section 2.1
Use the evaluator in two complementary ways:
- Test a service response by creating an EvaluationRequest from the original question and returned answer, then checking EvaluationResponse.isPass(). section 2.1
- Guard runtime responses by throwing an exception when evaluation fails and retrying the request with Spring Retry. section 2.3
Board-game question answering
A Spring Boot integration test injects BoardGameService and ChatClient.Builder, evaluates the service's answer, and asserts that the result passes. A runtime service can retry failed answers, but should keep maxAttempts low because retries increase model calls, token costs, and possible rate limiting. section 2.1 section 2.3
Listing 2.1 Testing that the response from an LLM is relevant to the question
package com.example.boardgamebuddy;
import org.assertj.core.api.Assertions;
import org.junit.jupiter.api.BeforeEach;
import org.junit.jupiter.api.Test;
import org.springframework.ai.chat.client.ChatClient;
import org.springframework.ai.chat.evaluation.RelevancyEvaluator;
import org.springframework.ai.evaluation.EvaluationRequest;
import org.springframework.ai.evaluation.EvaluationResponse;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.boot.test.context.SpringBootTest;
@SpringBootTest
public class SpringAiBoardGameServiceTests {
@Autowired
private BoardGameService boardGameService; #1
@Autowired
private ChatClient.Builder chatClientBuilder; #2
private RelevancyEvaluator relevancyEvaluator;
@BeforeEach
public void setup() {
this.relevancyEvaluator = new RelevancyEvaluator(chatClientBuilder);
}
@Test
public void evaluateRelevancy() {
String userText = "Why is the sky blue?";
Question question = new Question(userText);
Answer answer = boardGameService.askQuestion(question); #3
EvaluationRequest evaluationRequest = new EvaluationRequest(
userText, answer.answer());
EvaluationResponse response = relevancyEvaluator
.evaluate(evaluationRequest); #4
Assertions.assertThat(response.isPass()) #5
.withFailMessage("""
========================================
The answer "%s"
is not considered relevant to the question
"%s".
========================================
""", answer.answer(), userText)
.isTrue();
}
}
Shows the Java integration-test setup, dependency injection, evaluation request, and pass/fail assertion. section 2.1
