liveBook · Manning

Break-even request rate

As explained in Rearchitecting LLMs

A break-even request rate is the request volume at which the costs of self-hosting and using an external API are equal.

The break-even request rate is the hourly request volume at which self-hosting costs the same as paying for an API. In the stated model, divide the GPU’s hourly cost by the API cost per request; above that rate, the GPU cost per request is lower. section 12.9

Illustrative production scenario

At $2.50 per GPU hour and $0.004 per API request, the break-even rate is 625 requests per hour. At 2,950 requests per hour, the estimated self-hosted cost is about $0.001015 per request, including API costs for 4.2% of requests escalated by the router. section 12.9

Listing 12.8 The break-even rate and the cost per request
def breakeven_requests_per_hour(gpu_cost_per_hour, api_cost_per_request):
    return gpu_cost_per_hour / api_cost_per_request  #1
 
def slm_cost_per_request(gpu_cost_per_hour, requests_per_hour,
                         api_cost, escalation):
    fixed = gpu_cost_per_hour / requests_per_hour  #2
    return fixed + escalation * api_cost  #3
 
api_cost = 0.004
be = breakeven_requests_per_hour(gpu_cost_per_hour=2.50,
                                 api_cost_per_request=api_cost)
slm = slm_cost_per_request(
    gpu_cost_per_hour=2.50, requests_per_hour=2950,
    api_cost=api_cost, escalation=0.042)
saving = 1 - slm / api_cost
print(f"break-even at {be:.0f} requests/hour")
print(f"SLM cost per request ${slm:.6f} ({saving:.1%} below the API)")

Use the Python listing to calculate the break-even rate and estimate self-hosted cost per request, including escalations. section 12.9

The threshold depends on utilization: compare actual request volume with the break-even rate. Self-hosting is not automatically cheaper, and its cost estimate should include requests that pay for both the SLM attempt and API escalation. section 12.9 section 12.10

How Rearchitecting LLMs covers it

How deep it goes
Defines the threshold formula and works through cost estimates, but does not model variable GPU costs.
Examples
Calculates a 625-request hourly threshold from a $2.50 GPU hour and a $0.004 API request.
What it assumes
Requires hourly GPU cost and per-request API cost; illustrates the calculation with plain Python.
What you can do after
Evaluate whether observed request volume exceeds the modeled self-hosting threshold.
How it connects
Connects self-hosting economics with API pricing, GPU utilization, and escalation routing.

This explainer was generated by Manning's knowledge graph from the text of Rearchitecting LLMs. The section links point to the parts of the book it draws on.