Chapter 8: Generating with voice and pictures
The introduction explains why moving beyond text-only interaction would make Board Game Buddy more natural for people to use.
After this chapter you will be able to
- Rationale for richer communication. Explain why the chapter considers voice and images for Board Game Buddy: people communicate through speech, text, and images, while the application has so far accepted text questions and returned text answers.
- Two directions of voice interaction. Explain how generative AI can let a Spring application listen to users by turning audio into text and speak to users by turning text into spoken audio. Describe why voice can be a natural alternative to typing or tapping.
- Voice service transcription. Trace how the voice service turns an uploaded audio Resource into transcribed text.
- Spoken question request flow. Trace how the /audioAsk endpoint converts a submitted recording into an answer.
- Speech model conversion flow. Trace how OpenAiVoiceService turns answer text into an audio Resource using SpeechModel.
- Spoken answer HTTP response. Describe how the controller returns a BoardGameService answer as audio and how a client requests that response.
- Configurable speech generation options. Explain when to use a SpeechPrompt and OpenAiAudioSpeechOptions instead of passing a String directly to SpeechModel.call(), and name the options shown.
- Audio question prompt construction. Trace how askQuestion() puts a spoken question into the user message sent to the audio model.
- Direct audio answer handling. Trace how askQuestion() requests and extracts a spoken answer, and how the controller returns it without calling the separate voice service.
- Image-grounded game answers. Explain how an image can provide visual context for a question about a game, while the existing RAG and conversation advisors also supply game rules and chat history.
- Image in user message. Trace how the service adds a submitted image and question to the prompt, including the image resource, its MIME type, and the user message's text() and media() methods.
- Prompt and image options. Describe what the image service supplies when requesting an image and how Spring AI simplifies that request.
- URL and byte response formats. Distinguish the two image response formats and trace how the service turns each result into something a caller can use.
- Burger art controller flow. Trace how the burger-art controller builds an image prompt and chooses between returning a URL and returning PNG bytes.
- Image style and quality. Explain how natural style and hd quality affect the generated image in the section’s examples.
- Image option builder choices. Distinguish ImageOptionsBuilder from OpenAiImageOptions.builder() when specifying image options.
- Configuration property defaults. Explain what happens when image options are set in application.properties instead of specified through a builder.
What it builds on
- Configuring OpenAI models (chapter 1) · Configuring OpenAI models and an API key
- Understanding RAG (chapter 4) · The use of RAG to provide document text as context.
- Adding conversational memory (chapter 5) · The use of conversational memory to provide chat history.
- Updating the service (chapter 4) · The Board Game Buddy service used to answer questions about game content
Section by section
Introduction
The introduction explains why moving beyond text-only interaction would make Board Game Buddy more natural for people to use.
- Rationale for richer communication: Explain why the chapter considers voice and images for Board Game Buddy: people communicate through speech, text, and images, while the application has so far accepted text questions and returned text answers.
Why it matters: It gives a reason for adding voice and image interactions rather than treating them as features for their own sake.
Key terms: Board Game Buddy, text-based interaction, voice-based, images
8.1 Working with voice
Introduces voice as a natural way to interact with Spring applications and identifies the two directions of speech generation.
- Two directions of voice interaction: Explain how generative AI can let a Spring application listen to users by turning audio into text and speak to users by turning text into spoken audio. Describe why voice can be a natural alternative to typing or tapping.
Why it matters: These two directions explain what voice adds to an application’s user experience.
Key terms: audio file, textual representation, spoken audio, voice interaction
8.1.1 Transcribing speech
This section adds speech transcription to Board Game Buddy so a spoken question can be submitted to its existing question-answering service.
- Transcription model setup: Explain which Spring AI models support speech transcription here and how OpenAI transcription can coexist with another configured AI provider.
Why it matters: This explains why the application can use OpenAI for audio without switching its other AI features to OpenAI. - Voice service transcription: Trace how the voice service turns an uploaded audio Resource into transcribed text.
Why it matters: This is the conversion that lets the existing application work with a spoken question as text. - Spoken question request flow: Trace how the /audioAsk endpoint converts a submitted recording into an answer.
Why it matters: The new endpoint connects voice input to Board Game Buddy's existing answer flow.
Key terms: TranscriptionModel, OpenAiAudioTranscriptionModel, AzureOpenAiAudioTranscriptionModel, autoconfiguration, VoiceService, transcribe(), Resource, call(), OpenAiVoiceService, /audioAsk
8.1.2 Generating speech from text
This section implements text-to-speech in a Spring AI voice service, returns spoken answers through an HTTP endpoint, and shows how to customize speech generation.
- Speech model conversion flow: Trace how OpenAiVoiceService turns answer text into an audio Resource using SpeechModel.
Why it matters: This is the conversion that lets the service return spoken answers instead of text alone. - Spoken answer HTTP response: Describe how the controller returns a BoardGameService answer as audio and how a client requests that response.
Why it matters: The endpoint connects generated answers to a client that requests an audio response. - Configurable speech generation options: Explain when to use a SpeechPrompt and OpenAiAudioSpeechOptions instead of passing a String directly to SpeechModel.call(), and name the options shown.
Why it matters: The prompt form lets an application customize its spoken output without changing the basic conversion flow.
Key terms: SpeechModel, textToSpeech(), call(), ByteArrayResource, Resource, audioAskAudioResponse(), BoardGameService, Answer, @PostMapping, produces
8.1.3 Applying audio input and output directly
This section shows how Board Game Buddy can send a spoken question directly to an audio-enabled chat model and return its spoken answer without separate transcription or speech-generation steps.
- Direct audio model choice: Explain why an audio-enabled chat model can replace separate transcription and speech-generation steps, and describe the capability-versus-cost choice between the two models named in the section.
Why it matters: The model choice explains why the application no longer needs separate services for the two audio conversions. - Audio question prompt construction: Trace how askQuestion() puts a spoken question into the user message sent to the audio model.
Why it matters: This is how the application submits the recording itself rather than a transcript. - Direct audio answer handling: Trace how askQuestion() requests and extracts a spoken answer, and how the controller returns it without calling the separate voice service.
Why it matters: The response path completes the direct-audio flow without converting a textual answer to speech afterward.
Key terms: GPT-4o Audio, GPT-4o mini Audio, audio-enabled models, AudioQuestion, Resource, Media, media(), text(), options(), ALLOY
8.2 Asking questions about images
Shows how Board Game Buddy accepts an uploaded image and uses it alongside a question, game rules, and chat history to answer questions about a game.
- Image-grounded game answers: Explain how an image can provide visual context for a question about a game, while the existing RAG and conversation advisors also supply game rules and chat history.
Why it matters: It explains why an answer can draw on both the photographed game state and information the application already has. - Image in user message: Trace how the service adds a submitted image and question to the prompt, including the image resource, its MIME type, and the user message's text() and media() methods.
Why it matters: This is the step that makes the uploaded picture available as context in the prompt. - Multipart vision endpoint setup: Describe how the new controller handler receives an image upload, passes it to the service with the game title and question, and accommodates photos larger than Spring's default multipart file-size limit.
Why it matters: The endpoint must pass along the uploaded photo and accept files large enough for typical photos.
Key terms: vision, RAG, ChatClient, chat history, askQuestion(), Resource, MimeType, text(), media(), visionAsk()
8.3 Generating images
Shows how to configure Spring AI image generation, return generated images as URLs or bytes, and use it in a burger-art controller.
- Image model provider selection: Explain how an application obtains an ImageModel and selects a provider for image generation.
Why it matters: Provider selection lets you use a different service for images without changing the rest of the application’s model choices. - Prompt and image options: Describe what the image service supplies when requesting an image and how Spring AI simplifies that request.
Why it matters: This is the basic path from an application’s description to a generated image. - URL and byte response formats: Distinguish the two image response formats and trace how the service turns each result into something a caller can use.
Why it matters: The chosen format determines whether the caller receives a link to fetch or image data it can save directly. - Burger art controller flow: Trace how the burger-art controller builds an image prompt and chooses between returning a URL and returning PNG bytes.
Why it matters: The example connects application-specific information to an image prompt and lets clients choose how to receive the result.
Key terms: ImageModel, starter dependency, spring.ai.model.image, autoconfiguration, ImageService, ImageOptions, instructions, width, height, url
8.3.1 Specifying image options
This section explains how to choose image-generation options with builders or configuration properties, including OpenAI and Azure OpenAI variants.
- Image style and quality: Explain how natural style and hd quality affect the generated image in the section’s examples.
Why it matters: These options let you steer an image toward the look and level of detail you want. - Image option builder choices: Distinguish ImageOptionsBuilder from OpenAiImageOptions.builder() when specifying image options.
Why it matters: The builder choice determines whether OpenAI-specific options are available alongside the shared ones. - Configuration property defaults: Explain what happens when image options are set in application.properties instead of specified through a builder.
Why it matters: Configuration properties keep recurring image settings out of each Java builder call. - Azure image option variant: Describe how AzureOpenAiImageOptions differs from OpenAiImageOptions and how its property names differ.
Why it matters: Azure users need the deployment option and the Azure-specific property prefix.
Key terms: style(), vivid, natural, hd quality, ImageOptionsBuilder, OpenAiImageOptions, builder(), configuration properties, application.properties, defaults
8.4 Summary
Concepts explained in this chapter
- ImageOptionsImageOptions is a Spring AI options object that specifies parameters controlling image generation, such as dimensions and response format.
- media()media() is a Spring AI user-message specification method that attaches media content and its MIME type to a prompt.
- image generationImage generation is the production of images by an AI model from textual or other input specifications.
- ImageModelImageModel is a Spring AI interface for generating images through provider-specific implementations.
- OpenAiAudioTranscriptionModelOpenAiAudioTranscriptionModel is a Spring AI component that converts spoken audio into text using OpenAI services.
- TranscriptionModelTranscriptionModel is a Spring AI interface for converting audio resources into text.
- image inputImage input is the inclusion of visual data in a model request so the model can interpret and answer questions about it.
- SpeechModelSpeechModel is a Spring AI model interface that converts text into synthesized speech audio.
- OpenAiAudioSpeechOptionsOpenAiAudioSpeechOptions is a configuration object for selecting OpenAI text-to-speech voice, model, and audio format.
- SpeechPromptSpeechPrompt is a Spring AI request object containing text to synthesize and optional speech-generation settings.
- SpeechResponseSpeechResponse is a Spring AI response object containing synthesized audio bytes and related response information.
- GPT-4o-miniGPT-4o mini Audio is an OpenAI audio-capable model variant optimized for lower-cost multimodal generation.
- Spring AISpring AI is a Java library that provides abstractions and integrations for incorporating generative AI capabilities into Spring applications.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.