Amazon EC2 provides resizable virtual servers for deploying inference workloads. It suits services that need sustained capacity, low latency, AWS integration, and the option to scale workers. section 15.5.2
A practical EC2 deployment separates workload needs:
- Run batch inference as a script on EC2, reading jobs from a queue and writing results to Amazon S3 and a database. section 15.5.2
- Place stateless real-time inference workers behind an AWS Application Load Balancer, retaining guaranteed capacity while scaling additional instances when needed. section 15.5.2
- Consider EC2 Spot Instances to reduce cost when terminated workers can be replaced safely; monitor CPU, RAM, and GPU usage. section 15.5.2
