liveBook · Manning

Performance

As explained in AI Model Evaluation

A performance metric is a quantitative measure of how well a model or system achieves its intended task or objective.

Performance metrics measure whether an AI system delivers predictions quickly, reliably, and affordably under realistic product conditions—not just whether its predictions are good. section 4.2

Measure the system across layers:

  • Frontend load time and backend response latency. section 4.2
  • Model inference, infrastructure use, and orchestration for LLM or agent systems. section 4.2
  • Reliability, cost, and user-facing outcomes alongside latency; improvements can trade speed against prediction quality, throughput, or cost. section 4.2.4 section 4.5

Prioritize delays users can feel, such as slow recommendations or search results. Compare actual and perceived latency: hidden delays may not affect behavior, while visible ones can reduce engagement and product outcomes. section 4.1.1

Listing 4.1 mimic shadow traffic
# Create our models - production is faster, shadow is slower but theoretically better
production_model = RecommendationModel("Current Production Model", avg_latency_ms=150)  # 150ms avg
shadow_model = RecommendationModel("New Candidate Model", avg_latency_ms=220)  # 220ms avg
 
# Create router
router = ShadowTrafficRouter(production_model, shadow_model)
 
# Simulate user traffic (500 requests)
print("Simulating user traffic with shadow evaluation...")
for i in range(500):
    user_id = f"user_{i % 100}"  # 100 unique users making multiple requests
    context = {"page": "homepage", "device": np.random.choice(["mobile", "desktop", "tv"])}
    
    # Route request through both systems
    recommendations = router.route_request(user_id, context)
    
    # Simulate varying traffic patterns with small delays between requests
    time.sleep(np.random.exponential(0.02))
    
    # Show progress
    if (i+1) % 100 == 0:
        print(f"Processed {i+1} requests")

A shadow-traffic simulation routes requests to production and candidate recommendation systems, then compares latency distributions against performance thresholds. section 4.3.4

How AI Model Evaluation covers it

How deep it goes
Groups measurements across frontend, backend, model serving, infrastructure, and orchestration, then combines them to locate bottlenecks and assess trade-offs.
Examples
Uses recommendation latency, shadow-traffic comparisons, layered dashboards, and alert thresholds to make performance measurable.
What you can do after
Evaluate candidate systems by comparing latency distributions and checking measurements against predefined performance thresholds.
How it connects
Connects latency, load time, inference time, resource utilization, model quality, reliability, and product outcomes.

Also explained in

Compare how 55 books explain performance →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of AI Model Evaluation. The section links point to the parts of the book it draws on.