liveBook · Manning

Self-hosting

As explained in Rearchitecting LLMs

Self-hosting is operating a model on infrastructure controlled by the organization deploying it rather than relying on an external service.

Self-hosting runs a specialized small language model on organization-controlled infrastructure instead of sending every request to an external API. Its central trade-off is fixed GPU cost versus per-request API pricing: whether it saves money depends on request volume. section 12.9

Calculate the break-even request rate and include API escalations in the hosted path’s cost: escalated requests incur both the SLM attempt and the API charge. Then compare latency and task quality, and verify deployment constraints on the real inference engine. section 12.9 section 12.10

Illustrative cost comparison

With a $2.50-per-hour GPU, a $0.004-per-request API, and 2,950 requests per hour, break-even is 625 requests per hour. At that volume, a 4.2% escalation rate yields about $0.001 per request, 74.6% below the API cost. section 12.9

Listing 12.8 The break-even rate and the cost per request
def breakeven_requests_per_hour(gpu_cost_per_hour, api_cost_per_request):
    return gpu_cost_per_hour / api_cost_per_request  #1
 
def slm_cost_per_request(gpu_cost_per_hour, requests_per_hour,
                         api_cost, escalation):
    fixed = gpu_cost_per_hour / requests_per_hour  #2
    return fixed + escalation * api_cost  #3
 
api_cost = 0.004
be = breakeven_requests_per_hour(gpu_cost_per_hour=2.50,
                                 api_cost_per_request=api_cost)
slm = slm_cost_per_request(
    gpu_cost_per_hour=2.50, requests_per_hour=2950,
    api_cost=api_cost, escalation=0.042)
saving = 1 - slm / api_cost
print(f"break-even at {be:.0f} requests/hour")
print(f"SLM cost per request ${slm:.6f} ({saving:.1%} below the API)")

Use the listing to calculate break-even volume and self-hosted cost per request, including escalations. section 12.9

How Rearchitecting LLMs covers it

How deep it goes
Calculates break-even and per-request costs, then calls for latency and task-quality measurements.
Examples
Uses an illustrative 2,950-requests-per-hour deployment and hypothetical latency comparisons between an SLM and a frontier API.
What it assumes
Requires GPU hourly cost, API price per request, request volume, and escalation rate as calculation inputs.
What you can do after
Evaluate whether request volume and escalation costs make self-hosting cheaper, then compare latency and task quality.
How it connects
Connects specialized small language models with API escalation, confidence-gated routing, structured outputs, and inference-engine constraints.

Also explained in

Compare how 9 books explain self-hosting →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Rearchitecting LLMs. The section links point to the parts of the book it draws on.