Chapter 9: Observing AI operations
The introduction explains why observing application metrics and tracing data helps reveal the health and behavior of a Spring AI application.
After this chapter you will be able to
- Application health through observability. Explain how observing an application's metrics, along with tracing, can provide insight into its health, behavior, and inner workings.
- Actuator metrics setup. Explain how to make AI metrics available in Board Game Buddy: add the Actuator starter and expose the metrics endpoint alongside the health endpoint. State that Spring AI then measures token usage and operation timing without additional instrumentation.
- AI metrics endpoint contents. Describe how a request to `/ask` produces AI activity to inspect at `/actuator/metrics`, and identify the kinds of measurements listed there: vector store, generative AI API, token usage, advisor, and ChatClient metrics.
- Active and aggregate metrics. Distinguish metrics ending in `.active` from those without that suffix, including what each says about the application and why active values may be zero during development.
- Vector store metric measurements. Describe how to request a metric from the Actuator metrics endpoint and interpret the count and timing measurements for vector store operations.
- Vector operation tag filtering. Explain how availableTags identifies kinds of vector store activity and how a tag parameter narrows the reported measurements to one operation type.
- AI operation metric queries. Describe how to request gen_ai.client.operation statistics from the Actuator metrics endpoint and use tags to narrow the results to embedding operations.
- Operation names and embedding requests. Explain how the gen_ai.operation.name tag distinguishes chat from embedding operations and why embedding requests occur before the question goes to the LLM.
- Token usage and cost. Explain why token usage matters for estimating an application's AI costs, and why the count should be broken down by token type and model or operation.
- Token type aggregation pitfall. Explain how the gen_ai.token.type tag distinguishes input, output, and total counts, and why an unfiltered gen_ai.client.token.usage value can double the actual total.
- ChatClient operation metrics. Explain what the spring.ai.chat.client metric reports and how its stream tag distinguishes streaming from nonstreaming operations.
- Advisor metrics and filtering limits. Describe what the spring.ai.advisor metric measures, what BEFORE, AFTER, and AROUND mean, and why its tags do not usefully distinguish the built-in advisors by name.
- Prometheus metrics collection setup. Explain how the application makes Spring AI metrics available to Prometheus and how Prometheus collects them.
- Metric views and tag filtering. Describe how to find and view a metric in Prometheus, then narrow the displayed data by a tag.
- Grafana and Prometheus connection. Explain how Grafana uses the existing Prometheus metrics for Board Game Buddy, and describe how to connect the two services.
- AI metrics dashboard panels. Describe how to create a Grafana dashboard panel from a Prometheus metric and filter the displayed operations.
- Tracing data path. Explain how Board Game Buddy sends traces to Jaeger: Micrometer Tracing is bridged to OpenTelemetry, an OTLP exporter is declared as a Spring bean, and its destination is configured as a property.
- Request trace structure. Describe how to locate an /ask request in Jaeger and read its nested spans to see the major operations involved in answering a question.
- Distinct Qdrant operations. Distinguish the two Qdrant queries and two adds in a single question's trace, using span placement and query-filter tags as evidence.
What it builds on
- Implementing RAG with an advisor (chapter 4) · The use of an advisor to retrieve documents for an answer.
- Searching for similar documents (chapter 4) · Searching for similar documents supplies context for querying the vector store.
- Streaming the response (chapter 3) · The distinction between streaming and nonstreaming responses.
- Storing chat memory in a vector store (chapter 5) · The use of a vector store for chat memory.
Section by section
Introduction
The introduction explains why observing application metrics and tracing data helps reveal the health and behavior of a Spring AI application.
- Application health through observability: Explain how observing an application's metrics, along with tracing, can provide insight into its health, behavior, and inner workings.
Why it matters: Observations can help you see how your application is doing rather than guessing from its outward behavior.
Key terms: observability, metrics, tracing
9.1 Enabling Actuator metrics
This section shows how to expose Spring AI metrics through Spring Boot Actuator and interpret the metrics available after an application request.
- Actuator metrics setup: Explain how to make AI metrics available in Board Game Buddy: add the Actuator starter and expose the metrics endpoint alongside the health endpoint. State that Spring AI then measures token usage and operation timing without additional instrumentation.
Why it matters: These two configuration changes make the application’s AI activity observable without adding measurement code. - AI metrics endpoint contents: Describe how a request to `/ask` produces AI activity to inspect at `/actuator/metrics`, and identify the kinds of measurements listed there: vector store, generative AI API, token usage, advisor, and ChatClient metrics.
Why it matters: The metric names help you find measurements for the AI components involved in handling a request. - Active and aggregate metrics: Distinguish metrics ending in `.active` from those without that suffix, including what each says about the application and why active values may be zero during development.
Why it matters: This distinction prevents you from mistaking a quiet application right now for one that has done no AI work since startup.
Key terms: Actuator starter, health endpoint, metrics endpoint, application.properties, `/ask`, `/actuator/metrics`, vector store, generative AI APIs, token usage, Spring AI advisors
9.1.1 Inspecting vector store operations
This section shows how to read Actuator metrics for vector store operations and filter them by operation type.
- Vector store metric measurements: Describe how to request a metric from the Actuator metrics endpoint and interpret the count and timing measurements for vector store operations.
Why it matters: These measurements let you see how often the application uses the vector store and how long those operations take. - Vector operation tag filtering: Explain how availableTags identifies kinds of vector store activity and how a tag parameter narrows the reported measurements to one operation type.
Why it matters: Filtering separates the performance of add operations from query operations instead of combining them in one measurement.
Key terms: metrics endpoint, db.vector.client.operation, measurements, COUNT, TOTAL_TIME, MAX, availableTags, tag, db.operation.name, add
9.1.2 Examining AI model interaction
This section shows how Actuator metrics reveal the timing, types, and models involved in AI API operations, and how tags narrow the statistics to particular operations.
- AI operation metric queries: Describe how to request gen_ai.client.operation statistics from the Actuator metrics endpoint and use tags to narrow the results to embedding operations.
Why it matters: Filtering the metric helps you see which AI operations account for the reported activity and time. - Operation names and embedding requests: Explain how the gen_ai.operation.name tag distinguishes chat from embedding operations and why embedding requests occur before the question goes to the LLM.
Why it matters: The distinction explains why asking one question can produce both embedding and chat API operations. - Requested and returned model tags: Distinguish gen_ai.request.model from gen_ai.response.model using the section’s example of a requested model and a more specific returned version.
Why it matters: Comparing the tags lets you tell what model was requested from what the API reported using.
Key terms: gen_ai.client.operation, Actuator’s metrics endpoint, availableTags, embedding operations, gen_ai.operation.name, chat operations, vector store, gen_ai.request.model, gen_ai.response.model, gpt-4o-mini
9.1.3 Counting token usage
This subsection shows how to use Actuator token-usage metrics to estimate AI costs while avoiding a misleading unfiltered total.
- Token usage and cost: Explain why token usage matters for estimating an application's AI costs, and why the count should be broken down by token type and model or operation.
Why it matters: Breaking down usage helps you estimate spending and spot opportunities to conserve tokens. - Token type aggregation pitfall: Explain how the gen_ai.token.type tag distinguishes input, output, and total counts, and why an unfiltered gen_ai.client.token.usage value can double the actual total.
Why it matters: Filtering by token type prevents you from mistaking a doubled metric value for actual usage. - Model-specific token estimates: Describe how refining token-usage requests by token type and model helps estimate spending using the applicable per-token prices.
Why it matters: Model-specific input and output counts make a token-based bill estimate more meaningful.
Key terms: tokens, prompt tokens, generation tokens, gen_ai.client.token.usage, gen_ai.token.type, input, output, total, GPT-4o-mini, input tokens
9.1.4 Observing ChatClient operations
This section shows how to read ChatClient and advisor metrics, explains the limits of their filtering tags, and motivates viewing metrics through an observability tool.
- ChatClient operation metrics: Explain what the spring.ai.chat.client metric reports and how its stream tag distinguishes streaming from nonstreaming operations.
Why it matters: The stream tag lets you separate two kinds of ChatClient traffic rather than relying on the overall count. - Advisor metrics and filtering limits: Describe what the spring.ai.advisor metric measures, what BEFORE, AFTER, and AROUND mean, and why its tags do not usefully distinguish the built-in advisors by name.
Why it matters: Knowing the tag limits prevents you from expecting advisor metrics to show which specific advisor ran. - Observability tools and Micrometer: Justify using an observability tool rather than manually viewing metrics JSON, and describe Micrometer’s role in making Actuator metrics available to observability systems.
Why it matters: A visual tool makes operation metrics easier to read and lets you see how they change over time.
Key terms: spring.ai.chat.client, spring.ai.chat.client.stream, streaming, spring.ai.advisor, BEFORE, AFTER, AROUND, spring.ai.advisor.type, observability tool, Micrometer
9.2 Viewing metrics in Prometheus
This section shows how to expose Spring AI metrics to Prometheus and inspect them as time-series data.
- Prometheus metrics collection setup: Explain how the application makes Spring AI metrics available to Prometheus and how Prometheus collects them.
Why it matters: Prometheus cannot display the application's AI metrics until it can collect them. - Metric views and tag filtering: Describe how to find and view a metric in Prometheus, then narrow the displayed data by a tag.
Why it matters: A graph and a tag filter make it easier to inspect a particular AI operation over time. - Prometheus and Grafana roles: Distinguish Prometheus's metric collection and viewing role from Grafana's dashboard role.
Why it matters: This distinction explains why a dashboard can be added without changing how the application exposes metrics.
Key terms: Prometheus, Micrometer, Spring Boot Actuator, /actuator/prometheus, scrape, expression, Table, Graph, tag, gen_ai_client_operation_seconds_max
9.3 Creating AI dashboards
This section shows how to connect Grafana to the existing Prometheus metrics and create a dashboard for Spring AI operations.
- Grafana and Prometheus connection: Explain how Grafana uses the existing Prometheus metrics for Board Game Buddy, and describe how to connect the two services.
Why it matters: This connection lets you visualize the metrics already being collected without modifying Board Game Buddy. - AI metrics dashboard panels: Describe how to create a Grafana dashboard panel from a Prometheus metric and filter the displayed operations.
Why it matters: Panels turn the collected AI-operation metrics into graphs you can inspect and narrow to relevant operations.
Key terms: Grafana, Prometheus data source, Docker Compose, Prometheus server URL, dashboard, panel, gen_ai_client_operations_seconds_max, Label Filters, gen_ai_operation_name, chat
9.4 Tracing AI operations
This section configures Micrometer tracing for Board Game Buddy and uses Jaeger spans and tags to explain what happens during a question request.
- Tracing data path: Explain how Board Game Buddy sends traces to Jaeger: Micrometer Tracing is bridged to OpenTelemetry, an OTLP exporter is declared as a Spring bean, and its destination is configured as a property.
Why it matters: This explains how traces get from the Spring application into a tool where you can inspect them. - Development trace sampling: Explain why the section raises trace sampling to 100% during development and why a lower rate is preferable in production.
Why it matters: Sampling explains why a development request might not appear in Jaeger even when tracing works. - Request trace structure: Describe how to locate an /ask request in Jaeger and read its nested spans to see the major operations involved in answering a question.
Why it matters: The span hierarchy connects one user request to the application components and operations it triggers. - Distinct Qdrant operations: Distinguish the two Qdrant queries and two adds in a single question's trace, using span placement and query-filter tags as evidence.
Why it matters: The trace resolves why one question produces multiple vector-store queries and adds without treating them as duplicate work.
Key terms: Micrometer Tracing, OpenTelemetry, OTLP exporter, Jaeger, management.tracing.sampling.probability, sampling probability, trace, span, ChatClient, VectorStoreChatMemoryAdvisor
9.5 Summary
Concepts explained in this chapter
- PrometheusPrometheus is an open-source monitoring system that collects and stores time-series metrics and supports querying and alerting.
- metric endpointA metrics endpoint is an HTTP endpoint that exposes named application measurements and their associated statistics and tags.
- Spring Boot ActuatorSpring Boot Actuator is a Spring Boot module that provides production-oriented monitoring and management endpoints for an application.
- token countToken usage is the measured quantity of input and output tokens consumed during a language-model interaction.
- JaegerJaeger is an open-source distributed-tracing system for collecting, storing, and visualizing trace data.
- spring.ai.advisor metricThe spring.ai.advisor metric records counts and durations of operations performed through Spring AI advisors.
- spring.ai.chat.client metricThe spring.ai.chat.client metric records counts and durations of operations performed through ChatClient.
- vector store operation metricThe vector store operation metric records counts and durations of operations performed against vector stores.
- GrafanaGrafana is an open-source platform for querying, visualizing, and alerting on data from monitoring and observability systems.
- Micrometer TracingMicrometer Tracing is a Spring-agnostic tracing library that provides instrumentation and tracing integrations for applications.
- OTLP exporterAn OTLP exporter is a component that sends telemetry data using the OpenTelemetry Protocol.
- OpenTelemetryOpenTelemetry is an open-source observability framework for generating, collecting, and exporting telemetry such as traces, metrics, and logs.
- spanA span is a timed record of one operation or component segment within a distributed trace.
- Prometheus endpointA Prometheus endpoint is an HTTP endpoint that exposes application metrics in a format that Prometheus can scrape.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.