liveBook · Manning

Self-hosting

As explained in LLM Customization and Fine-Tuning

A deployment arrangement in which systems run within an organization's own infrastructure rather than an external provider's environment.

Self-hosting means downloading model weights and serving them on infrastructure an organization controls. That infrastructure can be an on-premises GPU cluster, a cloud VM with GPUs, or a dedicated inference service. The organization controls the model, serving stack, and data flow, but its team must keep the API available. section 2.4.2

The decision is not just whether a GPU is cheaper than an API call. It is whether the required privacy, customization, and request volume justify owning the infrastructure and its maintenance. section 2.4.2 section 6.11.4

Evaluate the deployment along four axes:

  • Cost and volume: API inference charges rise with requests; a GPU is paid for by the hour whether busy or idle. Compare expected traffic with GPU capacity, including redundancy and maintenance. section 1.3 section 2.4.2
  • Privacy: Restricted data may rule out a third-party-managed API. Choose the simplest platform that meets the actual privacy requirement; some managed platforms offer tenant isolation and contractual guarantees. section 2.4.2 section 6.11.4
  • Control: Self-hosting supports choices that managed APIs may not expose, including custom tokenization, partial weight updates, multi-LoRA serving, and custom decoding. section 2.4.2
  • Operations: Budget for model upgrades, security patches, GPU health monitoring, scaling, library upgrades, driver compatibility, and capacity planning. section 2.4.2 section 6.11.4

Compare one million inference requests

For calls with 500 input and 500 output tokens, the late-2026 estimate is $1,000–$3,000 per month on a fast API tier, $7,000–$10,000 on a mainstream tier, and $15,000–$30,000 on a frontier tier. One enterprise GPU running continuously costs roughly $1,500–$4,400 per month and can serve a 4B–8B model at several requests per second. A single GPU therefore loses on cost against the fast tier in this comparison, but wins against the mainstream and frontier tiers; a redundant pair roughly breaks even against mainstream pricing. Prompt caching and batch discounts can improve API economics, while adaptation and maintenance add costs to self-hosting. section 1.3

An offline or on-premises requirement is a hosting decision, not automatically a fine-tuning requirement. Self-host an open-weight model and add fine-tuning only if prompting on it plateaus. section 4.9.1

For supervised fine-tuning, infrastructure choices include cloud GPU VMs and on-premises GPU servers. Above 7B parameters, multi-GPU training can use Accelerate with DeepSpeed or FSDP; experiment tracking options include Weights & Biases, MLflow, and TensorBoard. section 6.11.3

How LLM Customization and Fine-Tuning covers it

How deep it goes
Explains hosting choices, cost comparisons, and operational responsibilities; identifies training tools without repeating the trainer setup.
Examples
Compares GPU and API costs for one million requests and lists cloud GPU VMs and on-premises servers as training infrastructure.
What you can do after
Evaluate self-hosting against managed platforms using privacy, customization, volume, and maintenance requirements.
How it connects
Connects self-hosting to data privacy, model adaptation, supervised fine-tuning, prompting, and multi-GPU training.

Also explained in

Compare how 9 books explain self-hosting →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of LLM Customization and Fine-Tuning. The section links point to the parts of the book it draws on.