Arize Phoenix

As explained in AI Agents in Action, Second Edition

Arize Phoenix is an open-source observability platform for tracing, evaluating, and monitoring generative AI and machine-learning applications.

Arize Phoenix is an open-source observability platform for tracing and evaluating agent systems. It captures workflows, model responses, tools, sessions, latency, token usage, costs, and metadata so complex agent behavior becomes inspectable rather than guesswork. section 7.4 section 8.3.6

Use Phoenix to inspect an agent across several operational layers:

• Trace each agent turn and tool call, then drill into workflows, agents, and raw LLM responses. section 7.4.1
• Filter activity with session, user, turn, intent, and other metadata; correlate traces with identifiers such as session_id and tool_call_id. section 8.3.6
• Monitor operational measures such as latency, token usage, tool success, errors, and cost, alongside quality and product measures when instrumented. section 7.4 section 8.3.6
• Turn captured spans into datasets, run evaluators, and feed results back into prompts, tools, and policies. section 7.4.3

Smallest tracing setup

Run Phoenix locally in Docker, register a project through its OpenTelemetry integration, clear default trace processors, and wrap the agent workflow in a trace. The Projects dashboard then exposes traces, sessions, metrics, configuration, latency, token usage, and raw model responses. section 7.4.1

Figure 7.10 Viewing the dataset and clicking Run Experiment shows the generated code. You can copy the code into a Python file to create an experiment or evaluation.

Use the generated experiment code as a bridge from selected response spans to a repeatable evaluator run. section 7.4.3

How AI Agents in Action, Second Edition covers it

How deep it goes

Explains deployment, OpenTelemetry connection, metadata, dashboards, datasets, and experiments, but defers detailed evaluator implementation.

Examples

Uses a local Docker deployment, a traced haiku-generating agent, metadata filters, selected span datasets, and generated evaluation code.

What it assumes

Assumes an instrumentable agent platform and basic tracing concepts; teaches local deployment, registration, trace wrapping, and metadata capture.

What you can do after

Diagnose/debug agent workflows, inspect costs and latency, filter correlated activity, and evaluate responses through trace-derived experiments.

How it connects

Connects OpenTelemetry, agent observability, evaluation, datasets, metrics, structured logging, and the OpenAI Agents SDK.

liveBookmanning.com

Read AI Agents in Action, Second Edition on liveBook Get the book at manning.com

Related concepts

• OpenTelemetryuses · OpenTelemetry is an open, vendor-neutral observability standard and framework for instrumenting software and generating, collecting, and exporting telemetry such as traces, metrics, and logs.
• Langfusealternative to · Langfuse is an open-source observability platform for tracing, monitoring, analyzing, and evaluating applications built with large language models.
• labeluses · A label is a value or annotation associated with an item that identifies its target category, outcome, or reference answer for learning or evaluation.
• observabilityimplements · Observability is the ability to infer a system’s internal state and behavior from its externally available outputs, such as logs, metrics, and traces.
• experimentationuses · Experimentation is the deliberate testing of an idea, intervention, or alternative by observing outcomes to gather evidence or compare effects.
• trajectoryuses · A trajectory is an ordered sequence of states, actions, events, or operations that records or describes progression through successive steps or over time.

This explainer was generated by Manning's knowledge graph from the text of AI Agents in Action, Second Edition. The section links point to the parts of the book it draws on.