An evaluation pipeline turns RAG improvement into a repeatable cycle: choose metrics tied to business objectives, test configurations systematically, analyze results, and refine the system. The goal is an upward performance trend rather than a single supposedly best setup. section 6.5.4
Useful evaluation priorities depend on what the application must achieve:
- Customer-facing accuracy emphasizes response accuracy and relevance. section 6.5.4
- Operational systems emphasize latency and throughput, while scalability adds queries per second and resource utilization. section 6.5.4
- Cost-sensitive deployments track CPU and memory usage alongside performance. section 6.5.4
Comparing PharmaBot configurations
Three configurations show the trade-off clearly: recall rises from 0.85 to 0.88 and F1 from 0.80 to 0.83, while average latency rises from 50 to 60 milliseconds and throughput falls from 20 to 16 queries per second. Evaluation therefore supports deliberate tuning rather than optimizing one metric in isolation.
