The break-even request rate is the hourly request volume at which self-hosting costs the same as paying for an API. In the stated model, divide the GPU’s hourly cost by the API cost per request; above that rate, the GPU cost per request is lower. section 12.9
Illustrative production scenario
At $2.50 per GPU hour and $0.004 per API request, the break-even rate is 625 requests per hour. At 2,950 requests per hour, the estimated self-hosted cost is about $0.001015 per request, including API costs for 4.2% of requests escalated by the router. section 12.9
Listing 12.8 The break-even rate and the cost per request
def breakeven_requests_per_hour(gpu_cost_per_hour, api_cost_per_request):
return gpu_cost_per_hour / api_cost_per_request #1
def slm_cost_per_request(gpu_cost_per_hour, requests_per_hour,
api_cost, escalation):
fixed = gpu_cost_per_hour / requests_per_hour #2
return fixed + escalation * api_cost #3
api_cost = 0.004
be = breakeven_requests_per_hour(gpu_cost_per_hour=2.50,
api_cost_per_request=api_cost)
slm = slm_cost_per_request(
gpu_cost_per_hour=2.50, requests_per_hour=2950,
api_cost=api_cost, escalation=0.042)
saving = 1 - slm / api_cost
print(f"break-even at {be:.0f} requests/hour")
print(f"SLM cost per request ${slm:.6f} ({saving:.1%} below the API)")Use the Python listing to calculate the break-even rate and estimate self-hosted cost per request, including escalations. section 12.9
The threshold depends on utilization: compare actual request volume with the break-even rate. Self-hosting is not automatically cheaper, and its cost estimate should include requests that pay for both the SLM attempt and API escalation. section 12.9 section 12.10
