liveBook · Manning

Observability

As explained in Vibe Engineering

Automated observability is the machine-driven collection and analysis of signals that make a system's behavior, failures, and quality degradation detectable.

An unattended agent can fail quietly: errors accumulate, outputs drift, and a broken build or security incident may be the first visible symptom. Observability collects and analyzes signals that make those failures visible early enough to investigate and intervene. It does not make an agent smarter or its output inherently trustworthy. section 5.3.4 section 5.0

Removing continuous human supervision also removes a feedback mechanism. Replace it with checks and signals, while retaining human review where it can catch systematic drift. section 5.3.4

A practical control set combines detection with a response:

  • Validate outputs against expectations and flag anomalous agent behavior. section 5.3.4
  • Keep audit logs for post-hoc debugging and stop pipelines with circuit breakers when quality degrades. section 5.3.4
  • Review sampled outputs regularly to catch systematic drift that automated checks may miss. section 5.3.4

With multiple agents, a failure can cross several artifact handoffs. A testing agent’s problem may originate in an ambiguous design document or an underspecified prompt upstream. Track agents and their artifacts together; a single conversation log is no longer enough to reconstruct the cause. section 5.6.2 section 5.6.7

Pipeline: payment-retry-feature
Status: Running (7m 23s)
[✓] Architect      (2m 14s, 12k tokens)  → design.md
[✓] Implementer-1  (3m 02s, 28k tokens)  → resilience.py
[◐] Implementer-2  (4m 11s, running)     → gateway changes
[ ] Tester-1       (waiting)
[ ] Tester-2       (waiting)  
[ ] Documenter     (waiting)
 
Total tokens: 47k ($0.42)
Estimated remaining: 35k tokens, 5 minutes

The pipeline view shows agent status, duration, token consumption, and artifacts. These signals expose stalls, waiting work, and potential cost or throughput bottlenecks. section 5.6.2

Use the signals to distinguish a transient failure, where a retry might help, from a systematic one, where retries may only consume more tokens. Compare stage durations too: fast parallel implementation offers little benefit if every agent waits behind a slower planning or review stage. section 5.6.2

For deployed code, logs record events, traces show the path of a particular request, and metrics summarize behavior over time. Parse latency percentiles, error rate, and input-size distribution can reveal changes after deployment. A p99 latency spike points to slow cases; a higher error rate points to cases that now fail. section 8.7.2

Listing 8.24 Logging example
def parse_markup(text: str) -> str:
    start_time = time.time()
    
    if len(text) > 10000:
        logger.warning(f"Large input: {len(text)} chars")
    
    try:
        result = _do_parse(text)
    except Exception as e:
        logger.error(f"Parse failed: {e}", extra={'input_sample': text[:100]})
        raise
    
    duration = time.time() - start_time
    if duration > 0.1:
        logger.warning(f"Slow parse: {duration:.2f}s for {len(text)} chars")
    
    return result

The parser logs unusually large inputs, exceptions, and slow parses. Those events give investigators concrete starting points when generated code behaves unexpectedly. section 8.7.2

Treat production requests as continuous evaluation, not proof that unobserved cases are safe. Invest in instrumentation before scaling orchestration, and keep early workflows conservative and well understood. Observability is one part of the operational foundation alongside rollback and incident response. section 8.7.2 section 5.6.7 section 5.7

How Vibe Engineering covers it

How deep it goes
Explains operational checks, agent dashboards, and runtime signals, with a parser logging example rather than a complete observability implementation.
Examples
Uses a multi-agent pipeline dashboard and a parser that logs large inputs, exceptions, and slow requests.
What you can do after
Diagnose/debug agent stalls, cross-agent failures, bottlenecks, and production regressions using operational signals.
How it connects
Connects observability to anomaly detection, audit logs, circuit breakers, tracing, metrics, rollback, and incident response.

Also explained in

Compare how 25 books explain observability →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Vibe Engineering. The section links point to the parts of the book it draws on.