liveBook · Manning

Observability

As explained in Context Engineering

Observability is the ability to understand a system's internal behavior through signals such as logs, traces, and metrics.

Observability is the ability to infer a system’s internal state and behavior from runtime telemetry. In an LLM or agent system, it helps explain how context and deployment state shaped a response—not just whether the response met a quality bar. section 8.3.1 section 8.3

Monitoring detects known problems through predefined signals and thresholds; observability supplies evidence for investigating why they occurred. Traditional telemetry combines three signals: section 8.3.1

  • Logs record discrete events at particular times, such as a database connection timeout. section 8.3.1
  • Metrics show numerical patterns over time, such as latency or memory utilization. section 8.3.1
  • Traces connect spans across components, showing a request’s path and where time was spent. section 8.3.1

A successful infrastructure trace alone cannot explain why an assembled prompt misled a model. Context-aware observability also preserves selected context, retrieved chunks, prompt versions, tool arguments, model parameters, and evaluation scores. Active telemetry makes runtime evidence available for feedback rather than relying only on retrospective human debugging. section 8.3.1 section 8.3.4

Evaluation and observability form a feedback loop: evaluation identifies a failure, traces help reveal its cause, and improved context management or orchestration is validated before release. Failures can also become new evaluation cases. section 8.3 section 8.4.5

Tracing a tool-using agent

A Langfuse callback can capture an agent’s execution flow and send it to a server. Depending on configuration and policy, the trace may include model and tool calls, retrieved material, selected context, token counts, latency, and the final response. Inspect these together when a response fails despite successful tool calls. section 8.4.4

Listing 8.5 Python example for observability and tracing with Langfuse
import uuid
from langchain.agents import create_agent
from langchain_core.tools import tool
from langchain_openai import ChatOpenAI
from langfuse import get_client
from langfuse.langchain import CallbackHandler

LANGUAGE_FACTS = {
    "python": "Python was created by Guido van Rossum and first "
              "released in 1991.",
}


@tool
def lookup_language(name: str) -> str: #1
    """Look up who created a programming language and when it was
    first released."""
    return LANGUAGE_FACTS.get(name.lower(), f"No record for {name}.")


langfuse_client = get_client() #2
langfuse_callback_handler = CallbackHandler()

llm = ChatOpenAI(model="gpt-4o", temperature=0) #3
agent = create_agent(
    model=llm,
    tools=[lookup_language],
    system_prompt="Use the available tools to answer questions
    about programming languages.",
)

trace_id = str(uuid.uuid4()) #4
question = "Who created the Python programming language, and in
which year was it first released?"
response = agent.invoke( #5
    {"messages": [{"role": "user", "content": question}]},
    config={"callbacks": [langfuse_callback_handler],
            "run_id": trace_id},
)
print(response["messages"][-1].content)

langfuse_client.flush() #6
trace_url = langfuse_client.get_trace_url(trace_id=trace_id)
print(f"View the trace in Langfuse: {trace_url}")

Inspect the Python setup for a tool-using agent and its Langfuse callback handler. section 8.4.4

OpenTelemetry and its GenAI semantic conventions provide a foundation for standardized model-operation telemetry. AI-specific platforms such as Langfuse, LangSmith, and Arize Phoenix capture richer payload context. Correlate traces with prompt versions, retrieval indexes, model parameters, and evaluation scores; redact sensitive data and enforce role-based access before export. section 8.3.4

For coding agents, extend tracing beyond the session. Connect context and command logs with repository changes, CI results, review metadata, deployment records, and incidents. This helps identify context patterns associated with reliable changes or hidden rework; vendor telemetry alone may not provide the complete picture. section 11.4.2

How Context Engineering covers it

How deep it goes
Defines telemetry pillars, contrasts monitoring with observability, and gives instrumentation and operational guidance for AI systems.
Examples
Demonstrates a Langfuse-traced tool-using agent and a LangSmith chain that traces context selection.
What you can do after
Diagnose context-related failures using Langfuse traces and correlate runtime evidence with evaluation results.
How it connects
Connects observability with evaluation, context engineering, monitoring, and agent orchestration.

Also explained in

Compare how 25 books explain observability →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Context Engineering. The section links point to the parts of the book it draws on.