A voice agent is an AI agent that communicates with users through spoken input and output. In a browser, real-time interaction depends on low latency, lightweight tools, and responsive client-side JavaScript. section 8.1.1
A practical voice-agent design separates four concerns:
- Conversation: accept spoken requests and respond vocally through a real-time agent running in the web application. section 8.1.1
- Tools: connect spoken requests to capabilities such as image generation while keeping the conversation interactive. section 8.1.3
- Deployment: use ephemeral client keys for demonstrations, but mint them through authenticated backend services in production. section 8.1.1
Voice-controlled image generation
A browser-hosted voice agent can receive a spoken request to create an image, call a generate_image tool, and let the user continue talking while a backend image agent performs the slower task. The tool acts as a proxy rather than blocking the conversation. section 8.1.3
Listing 8.1 01_embedded_agent_speech.html (agent only)
<footer> #1
<div>Get a <strong>client ephemeral key</strong> (valid briefly):</div>
<pre class="kbd"># PowerShell (no jq needed)
(Invoke-RestMethod -Uri "https://api.openai.com/v1/realtime/sessions" -Method POST `
-Headers @{ Authorization = "Bearer $env:OPENAI_API_KEY" } `
-ContentType "application/json" `
-Body '{"model":"gpt-4o-realtime-preview-2025-06-03"}').client_secret.value</pre>
<div>In production, mint the key on your backend and pass it to the browser after auth.</div>
</footer> #1
<!-- #2
Option: Bash (commented out). Uncomment below if you prefer to use curl + jq to obtain the ephemeral key.
<div>Get a <strong>client ephemeral key</strong> using Bash:</div>
<pre class="kbd"># Bash (curl + jq)
curl -s -X POST "https://api.openai.com/v1/realtime/sessions" \
-H "Authorization: Bearer $OPENAI_API_KEY" \
-H "Content-Type: application/json" \
-d '{"model":"gpt-4o-realtime-preview-2025-06-03"}' | jq -r '.client_secret.value'
</pre>
--> #2
const agent = new RealtimeAgent({ #3
name: 'Assistant',
instructions: 'You are a helpful, concise voice assistant. Speak naturally and keep replies brief unless asked for detail.',
}); #4
Inspect the embedded browser example for ephemeral-key handling, real-time session setup, and the client-side voice-agent scaffold. section 8.1.1
