Adversarial prompting deliberately crafts user input to make a generative AI application ignore instructions, reveal prompt contents, bypass safeguards, or perform harmful actions. Common forms include prompt injection, prompt leaking, and jailbreaking. section 10.3
Mitigate adversarial prompting with layered controls:
- Restrict access to documents and tools so unauthorized users cannot reach protected capabilities indirectly through the LLM. section 10.0
- Apply Spring AI's SafeGuardAdvisor to block prompts containing configured sensitive terms and return a controlled failure response. section 10.3.1
- Intercept requests or responses with custom advisors that inspect suspected prompt leaks and replace unsafe results. section 10.3.2
- Treat safeguards as incomplete defenses: monitor their effectiveness and adjust them as new prompting vulnerabilities appear. section 10.3
Blocking a prompt leak
A custom CanaryWordAdvisor can detect an attempt such as “Ignore all previous instructions” followed by a request for the full system prompt. Registering it with ChatClient.Builder.defaultAdvisors() lets the application return “Detected attempt to leak system prompt message.” instead of the model's answer, and a test can verify that behavior. section 10.3.2
Listing 10.3 A custom advisor to prevent prompt leaks
package com.example.canarywordadvisor;
import org.springframework.ai.chat.client.ChatClientRequest;
import org.springframework.ai.chat.client.ChatClientResponse;
import org.springframework.ai.chat.client.advisor.api.CallAdvisor;
import org.springframework.ai.chat.client.advisor.api.CallAdvisorChain;
import org.springframework.ai.chat.messages.AssistantMessage;
import org.springframework.ai.chat.model.ChatResponse;
import org.springframework.ai.chat.model.Generation;
import java.util.List;
import java.util.UUID;
public class CanaryWordAdvisor implements CallAdvisor {
private static final String DEFAULT_CANARY_FOUND_MESSAGE =
"Canary word detected!";
private final String canaryWordFoundMessage;
public CanaryWordAdvisor(String canaryWordFoundMessage) {
this.canaryWordFoundMessage = canaryWordFoundMessage;
}
@Override
public ChatClientResponse adviseCall(
ChatClientRequest chatClientRequest,
CallAdvisorChain chain) {
var canaryWord = generateCanaryWord();
var originalSystemMessage = chatClientRequest.prompt()
.getSystemMessage().getText();
var newSystemMessage = String.format("%s (%s)",
originalSystemMessage, canaryWord); #1
var advisedRequest = chatClientRequest.mutate().prompt(
chatClientRequest.prompt()
.augmentSystemMessage(newSystemMessage))
.build(); #2
var chatClientResponse = chain.nextCall(advisedRequest); #3
if (chatClientResponse.chatResponse()
.getResult()
.getOutput()
.getText()
.contains(canaryWord)) { #4
return createFailureResponse(advisedRequest);
}
return chatClientResponse;
}
private ChatClientResponse createFailureResponse(
ChatClientRequest advisedRequest) {
return new ChatClientResponse(
ChatResponse.builder()
.generations(
List.of(
new Generation(
new AssistantMessage(canaryWordFoundMessage))))
.build(),
advisedRequest.context());
}
public static CanaryWordAdvisor.Builder builder() {
return new CanaryWordAdvisor.Builder();
}
private String generateCanaryWord() {
return UUID.randomUUID().toString();
}
@Override
public String getName() {
return this.getClass().getSimpleName();
}
@Override
public int getOrder() {
return 0;
}
public static class Builder { #5
private String canaryWordFoundMessage = DEFAULT_CANARY_FOUND_MESSAGE;
public Builder canaryWordFoundMessage(String canaryWordFoundMessage) {
this.canaryWordFoundMessage = canaryWordFoundMessage;
return this;
}
public CanaryWordAdvisor build() {
return new CanaryWordAdvisor(canaryWordFoundMessage);
}
}
}
Inspect the Java implementation of a custom advisor that detects canary words in model responses and substitutes a configured warning. section 10.3.2
