Deployment makes a model available for inference as a usable service. A fine-tuned model appears alongside the base model in an Azure tenant or OpenAI subscription. section 9.4
Production deployment turns a proof of concept into an operational system. It requires decisions about scalability, latency, reliability, security, monitoring, integration, and cost rather than model availability alone. section 11.0
Choose the deployment path and supporting infrastructure according to model and business requirements:
- Use cloud deployment for scalability, varied compute, and managed services, while assessing ongoing costs, vendor lock-in, and data privacy. section 11.2
- Use on-premise deployment for greater control, performance optimization, and data security, accepting higher upfront investment and slower scaling. section 11.2
- Use hybrid deployment to keep sensitive data on-premise while obtaining cloud scalability, with added management complexity. section 11.2
- Support the service with containers, serverless functions, API gateways, load balancing, redundancy, monitoring, and automated testing. section 11.2
Fine-tuned model inference
After fine-tuning, deploy the resulting model so it appears as another selectable model in Azure or an OpenAI subscription. Compare it visually with the base model in the OpenAI playground, while recognizing that the comparison is not an efficient evaluation method. section 9.4
Figure 9.8 Deploying fine-tuned model for inference
Use the deployment diagram to connect fine-tuned model availability with the inference path. section 9.4
Before production, assess computational resources and data quality; test model reliability; plan redundancy, failover, and disaster recovery; apply encryption and access controls; manage cost; integrate existing workflows; and retain human oversight. section 11.7
Track inference metrics such as time to first token, time per output token, latency, and throughput. Manage quotas, rate limits, queuing, and usage with API management and monitoring tools. section 11.7
