Autoscaling changes active compute capacity as deployments demand it. Node auto-provisioning can add nodes instead of keeping a fixed pool, so payment follows used capacity; resource limits cap the available CPUs, memory, and GPUs. section 6.2.1
Scaling down is not always reversible in LLM services. Shutting down may take seconds, but redeployment can take minutes or hours, making an ill-timed reduction risky before traffic surges and complicating bursty workloads, rolling updates, and bug recovery. section 3.2.2
Capping accelerator capacity
A node auto-provisioning configuration can allow up to 40 T4 GPUs, 16 A100 GPUs, and 8 A100 80GB GPUs, while also enabling automatic repair and upgrades. section 6.2.1
Listing 6.8 Example NAP config file
resourceLimits:
- resourceType: 'cpu'
minimum: 10
maximum: 100
- resourceType: 'memory'
maximum: 1000
- resourceType: 'nvidia-tesla-t4'
maximum: 40
- resourceType: 'nvidia-tesla-a100'
maximum: 16
- resourceType: 'nvidia-a100-80gb'
maximum: 8
management:
autoRepair: true
autoUpgrade: true
shieldedInstanceConfig:
enableSecureBoot: true
enableIntegrityMonitoring: true
diskSizeGb: 100
Inspect a concrete resource-limit configuration for CPU, memory, and GPU-backed nodes. section 6.2.1
