Performance asks whether a GenAI application meets its objectives in practice. For a production service, that includes latency, throughput, resource use, and the quality of its answers. A model can produce useful responses yet still disappoint users if it responds too slowly or fails under load. section 11.5.1 section 9.3.2
Measure the dimensions that matter to the application rather than relying on one score:
- Operational performance: monitor latency, throughput, token use, and resource utilization. Test how these change under different loads. section 11.5.1
- Output performance: assess relevance, coherence, accuracy, and fluency against the intended task. Compare an adapted model with its pre-fine-tuning version or another suitable baseline. section 9.3.2
- Risk-related performance: define harmful outputs for the use case, measure their frequency and severity, and repeat the assessment after mitigations. section 13.3.2
Testing a production assistant
For an enterprise question-answering assistant, use Locust or Apache JMeter to simulate concurrent users and measure latency and throughput under stress. Monitor the deployed service with Prometheus and Grafana, then evaluate answer relevance and accuracy on representative questions. If responses are slow, trim unnecessary prompt text or cache frequently requested, rarely changing information; check that shorter responses or other optimizations do not compromise answer quality. section 11.5.1 section 9.3.2
Keep evaluation tied to business needs. Use clear criteria, automated metrics, manual review, and user feedback; log production behavior so performance can be assessed over time. Benchmarks help compare models, but no single standardized evaluation establishes how good a model is in every context. For task-specific testing, Azure AI Studio can run ready-made or custom evaluations from a test dataset; its AI-assisted metrics require a supported deployed model and compute runtime. section 9.3.2 section 12.0 section 12.3.7
Treat performance as an ongoing deployment concern, alongside reliability and security. Validate data quality, monitor data sources and model behavior, and retest after changes. Token management, caching, load balancing, and auto-scaling can improve responsiveness or capacity, but their effects should be measured against cost and output quality. section 11.1 section 11.5.1
