Chapter 3: Submitting prompts for generation

Introduction

The chapter introduction explains why more advanced Spring AI applications may need richer prompt and response handling than simple strings.

• Limits of string prompts: Explain why directly submitting a String-based prompt may become insufficient as an application's generative AI requirements grow more advanced.
  Why it matters: This explains the need to move beyond the chapter 1 prompt approach.
• Response metadata and usage: Describe how a generated result can contain useful metadata beyond its text, including usage data that helps gauge billing impact.
  Why it matters: The generated text alone may not tell you what a request costs.
• Incremental response delivery: Distinguish a streamed response from one returned all at once.
  Why it matters: A client need not wait for the entire generated response before receiving any of it.

Key terms: String-based prompt, ChatClient, metadata, usage data, billing, stream

3.1 Working with prompt templates

The section introduces prompt templates and prepares a board-game question-answering example with game context and request validation.

• Prompt template structure: Explain how a prompt template combines static guiding text with placeholders filled by invocation-specific model data, and why this is preferable to assembling prompts with String concatenation.
  Why it matters: This explains what the template will replace in the existing prompt-building code.
• Game title context: Explain why the example adds a game title to both the Question and Answer records.
  Why it matters: The game title gives both the prompt and the returned answer an explicit subject.
• Request validation and errors: Describe how the example requires nonblank Question fields, triggers validation for controller requests, and presents validation failures as standard API errors.
  Why it matters: A missing game title can be rejected clearly instead of leaving the application without needed context.

Key terms: prompt templates, placeholders, model data, String concatenation, Question, Answer, gameTitle, @NotBlank, @Valid, Problem Details

3.1.1 Defining a prompt template

Shows how to replace prompt-string concatenation with a parameterized template and improve the prompt’s instructions.

• Parameterized prompt template: Explain how the prompt template uses {game} and {question} placeholders for variable data instead of building the prompt through String concatenation.
  Why it matters: A single template makes the wording easier to find and change.
• Template values in ChatClient: Trace how askQuestion() supplies the template and its parameter values through user(), then sends the completed prompt and obtains the answer text.
  Why it matters: Supplying both the template and its values is what lets the application submit a complete prompt.
• Instructions within the template: Describe the example change that gives the LLM more context about its role and expected answers, including what to do when it does not know an answer.
  Why it matters: A template is also a convenient place to refine the instructions that shape an answer.

Key terms: prompt template, StringTemplate, placeholders, String concatenation, askQuestion(), user(), Consumer<UserSpec>, text(), param(), call()

3.1.2 Importing the template as a resource

The section moves a prompt template into a classpath resource and explains why adding context to prompts can be a practical response to fabricated answers.

• External prompt template resource: Describe how to move the prompt template into a file under src/main/resources and inject it into SpringAiBoardGameService as a Resource without changing how askQuestion() calls text().
  Why it matters: You can revise the prompt independently of the service code while preserving the application's behavior.
• Hallucinations despite prompt instructions: Explain what the Burger Battle example shows about answers to questions on unfamiliar games, even when the prompt tells the LLM to say it does not know.
  Why it matters: A plausible answer is not necessarily a correct one, particularly for a game the model may not know.
• Just-in-time context tradeoff: Justify why adding context to a prompt is presented as simpler for a newly released game than training or fine-tuning a model.
  Why it matters: Prompt context offers a more immediate way to help answer questions about games absent from the model's training.

Key terms: StringTemplate, Resource, @Value, classpath:, text(), hallucinations, Burger Battle, additional context, training, fine-tuning

3.2 Stuffing the prompt with context

The section shows how to add game rules to a prompt as context and explains the token costs and limits of doing so.

• Open-book prompt context: Explain how stuffing a prompt lets an LLM answer a question about a subject it was not trained on by supplying relevant text alongside the question.
  Why it matters: This explains why adding Burger Battle’s rules can improve an answer about a game the model has not learned.
• Game rules prompt injection: Describe how the example loads game rules from a resource and passes them into a prompt template for the user question.
  Why it matters: The rules must reach the prompt as text before the model can use them to answer.
• Context token tradeoffs: Explain why adding context increases prompt tokens and what consequences the section gives for that increase.
  Why it matters: Supplying useful context has a practical cost, so sending ever-larger documents is not always viable.

Key terms: stuffing the prompt, context, gameRules, GameRulesService, getRulesFor(), Resource, {rules}, param(), input tokens, context window

3.3 Assigning prompt roles

This section explains prompt message roles and shows how to separate application instructions from a user’s question in a Spring AI request.

• Prompt message roles: Distinguish user, system, assistant, and tool messages by the kind of content each carries, and describe what Spring AI does when an API does not support the system role.
  Why it matters: The roles clarify whose content each message represents, even though APIs differ in which roles they accept.
• Separating instructions from questions: Explain why the board-game prompt is split into a system message for response instructions and a user message for the question, rather than sent as one user message.
  Why it matters: Separating the instructions from the question gives each part the role that matches its purpose.
• Spring AI role assignment: Describe how the example changes its template and ChatClient message specifications to send separate system instructions and a user question.
  Why it matters: These changes turn the conceptual role split into two distinct messages in the request.
• Request inspection with Logbook: Describe how the section configures Logbook to inspect Spring AI HTTP requests and responses, including its logging level and stated limitation.
  Why it matters: Inspecting raw traffic lets you verify that Spring AI sent separate role-labeled messages.

Key terms: User, System, Assistant, Tool, message roles, system message, user message, prompt roles, systemPromptTemplate.st, questionPromptTemplate.st

3.4 Influencing response generation

Introduces three ways Spring AI can influence response generation and delivery: generation options, output conversion, and streaming.

• Ways to influence responses: Describe how generation options, output conversion, and streaming affect a response at the introductory level explained here.
  Why it matters: These are three different ways to shape what an application receives from a model.

Key terms: generation options, output conversion, streaming, next token, Java object

3.4.1 Specifying chat options

This section explains how Spring AI chat options are set and how temperature, Top-P, and Top-K influence token selection during response generation.

• Chat option settings and precedence: Explain how to set chat options through API-specific application properties, ChatClient defaultOptions(), or prompt-level options(), and which setting takes precedence.
  Why it matters: This lets you choose whether an option applies broadly or to a particular prompt.
• Probabilistic token selection: Describe how a response is built token by token using candidate probabilities and weighted random selection.
  Why it matters: This explains why repeated submissions of the same prompt can produce different responses.
• Temperature and response variability: Explain how temperature changes candidate probabilities and the variability of generated responses.
  Why it matters: Temperature gives you a way to adjust how varied the model’s choices are.
• Top-P probability cutoff: Explain how Top-P limits candidate tokens by cumulative probability and what happens after the cutoff.
  Why it matters: Top-P restricts which possible tokens can be chosen without making the choice among those remaining deterministic.
• Top-K count cutoff: Distinguish Top-K’s count-based cutoff from Top-P’s probability-sum cutoff, and state its OpenAI limitation.
  Why it matters: The distinction helps you choose a supported cutoff option for the API you use.

Key terms: application.properties, defaultOptions(), ChatOptions, options(), token, candidate tokens, probability, random selection, temperature, probabilities

3.4.2 Formatting response output

This section shows how Spring AI converts LLM responses into Java objects and lists by adding formatting instructions to prompts, while noting that conversion depends on the model following those instructions.

• Java object output conversion: Explain how calling entity(Answer.class) instead of content() lets ChatClient return an Answer object: Spring AI adds formatting instructions based on the record, then parses the JSON response into that type.
  Why it matters: You can receive a typed result without writing the extract-and-instantiate code yourself.
• Model compliance and conversion failures: Explain why requesting formatted output does not guarantee an object: an LLM may ignore the formatting instructions, and binding its non-JSON response can throw a JsonParseException.
  Why it matters: A typed return request is not a guarantee that every model will produce a parseable result.
• Generic list output conversion: Describe how to request a List<String> from entity() using a new ParameterizedTypeReference<List<String>>, and why that type reference is needed instead of passing List or List<String>.
  Why it matters: Keeping the element type lets the client request a typed list rather than handling a list-shaped answer as raw text.

Key terms: ChatClient, content(), entity(), Answer.class, output conversion, formatting instructions, JsonParseException, non-JSON response, List<String>, ParameterizedTypeReference

3.4.3 Streaming the response

This section explains why streaming improves chat responses, how to stream with Spring AI, how response MIME types affect clients, and why streaming does not pair well with output conversion.

• Streaming user experience: Explain why showing a response as it is generated improves the experience of a chat application, especially for lengthy answers.
  Why it matters: The visible progress is the main reason to choose streaming for a chat interface.
• ChatClient streaming changes: Describe the service and controller changes used here to return a streamed response, and explain what the returned Flux contains.
  Why it matters: Both the service and its controller must pass along the stream for callers to receive pieces as they arrive.
• Stream response MIME types: Distinguish the effects of text/event-stream, application/json, and application/ndjson on the streamed response, including the HTTPie behavior described here.
  Why it matters: The chosen response type affects both whether the client receives a stream and how it handles each piece.
• Streaming output conversion limit: Explain why this section advises against combining streamed responses with output conversion.
  Why it matters: This prevents expecting structured output conversion to work on each arriving fragment.

Key terms: streaming the response, ChatClient, stream(), call(), Flux<String>, Project Reactor, AskController, @PostMapping, text/event-stream, application/json

3.5 Working with response metadata

This section shows how to retrieve and log token usage metadata alongside a generated answer, and notes that availability depends on the AI service.

• Token usage and cost: Explain why token usage metadata is useful: prompts and generated responses are broken into tokens, and token counts help determine the cost of an LLM interaction.
  Why it matters: Token counts help you understand what an interaction with a paid AI service may cost.
• Retrieving metadata with responseEntity: Trace how the example obtains usage metadata while still returning an Answer: responseEntity() supplies the bound Answer and a ChatResponse, whose metadata is passed to logUsage().
  Why it matters: Keeping the ChatResponse alongside the answer lets the application log usage without changing the answer it returns.
• AI service usage reporting: Distinguish services that the section says provide usage data from examples that do not, and explain why a log entry cannot be assumed for every provider.
  Why it matters: A missing usage count may reflect the provider rather than a problem with the logging approach.

Key terms: usage statistics, tokens, token usage, responseEntity(), ResponseEntity<ChatResponse, Answer>, ChatResponse, getMetadata(), logUsage(), usage metrics, Ollama

3.6 Summary