Performance metrics measure whether an AI system delivers predictions quickly, reliably, and affordably under realistic product conditions—not just whether its predictions are good. section 4.2
Measure the system across layers:
- Frontend load time and backend response latency. section 4.2
- Model inference, infrastructure use, and orchestration for LLM or agent systems. section 4.2
- Reliability, cost, and user-facing outcomes alongside latency; improvements can trade speed against prediction quality, throughput, or cost. section 4.2.4 section 4.5
Prioritize delays users can feel, such as slow recommendations or search results. Compare actual and perceived latency: hidden delays may not affect behavior, while visible ones can reduce engagement and product outcomes. section 4.1.1
Listing 4.1 mimic shadow traffic
# Create our models - production is faster, shadow is slower but theoretically better
production_model = RecommendationModel("Current Production Model", avg_latency_ms=150) # 150ms avg
shadow_model = RecommendationModel("New Candidate Model", avg_latency_ms=220) # 220ms avg
# Create router
router = ShadowTrafficRouter(production_model, shadow_model)
# Simulate user traffic (500 requests)
print("Simulating user traffic with shadow evaluation...")
for i in range(500):
user_id = f"user_{i % 100}" # 100 unique users making multiple requests
context = {"page": "homepage", "device": np.random.choice(["mobile", "desktop", "tv"])}
# Route request through both systems
recommendations = router.route_request(user_id, context)
# Simulate varying traffic patterns with small delays between requests
time.sleep(np.random.exponential(0.02))
# Show progress
if (i+1) % 100 == 0:
print(f"Processed {i+1} requests")
A shadow-traffic simulation routes requests to production and candidate recommendation systems, then compares latency distributions against performance thresholds. section 4.3.4






















































