liveBook · Manning

Cost

As explained in LLM Customization and Fine-Tuning

The money required to train, operate, or maintain a model.

Cost is the money and staff effort required to train, operate, and maintain a model. At production scale, even a successful API deployment can become too expensive: one million requests a day can cost an illustrative $210,000–$300,000 per month before caching and batch discounts. Adaptation may reduce that bill, but it also transfers maintenance work to the team. section 1.4 section 6.12.4

Compare options over the expected life and volume of the deployment:

  • For in-context learning, count input tokens in the instructions, examples, and query, plus output tokens. The example bank is paid for on every request. section 4.7.1
  • For a self-hosted fine-tuned model, spread training and hosting costs across the requests served. Fine-tuning requires more upfront spending but can have a lower per-request cost. section 4.7.1
  • Add data collection, annotation, evaluation, engineering, and ongoing infrastructure maintenance. GPU spend alone can substantially understate the investment. section 6.12.4 section 7.6.3
  • Project request volume before committing to self-hosting and revisit it after the first month. Fixed capacity can be economical at high volume but wasteful when usage is low or variable. section 4.9.3

More examples in a prompt increase token costs roughly linearly, while accuracy may plateau. A fixed example bank can benefit from cached-input discounts; selecting different examples for each query reduces the shared prefix. Batch pricing may also help non-interactive workloads. Check current provider rates before comparing approaches. section 4.3.2

Table 4.2 Cost framing at different shot counts (illustrative)
Shots
Prompt tokens (illustrative)
Inference cost rank
Accuracy plateau range
1
~250
Low
60 to 70%
8
~1,800
Low
78 to 84%
50
~10,200
Medium
86 to 90%
100
~20,200
High
88 to 92%
200
~40,200
very high
89 to 93%

Compare illustrative prompt sizes, cost ranks, and accuracy ranges across shot counts. section 4.3.2

Human effort can dominate training

An illustrative 4B fine-tuning project totals $16,150 over roughly 3.5 weeks: $650 for compute and $15,500 for staff time. Data curation and evaluation account for much of the human effort, so reducing GPU time alone would leave most of the project cost intact. section 6.12.4

Table 6.14 Training cost breakdown (illustrative)
Phase
Duration
Compute Cost
Human Cost
Total
Data curation and quality review
2 weeks
–
$8,000 (clinical reviewer time)
$8,000
Data formatting and preprocessing
3 days
$50 (CPU instances)
$2,000 (ML engineer time)
$2,050
Training (including failed experiments)
2 days
$400 (GPU hours)
$1,500 (ML engineer time)
$1,900
Evaluation and safety testing
1 week
$200 (inference costs)
$4,000 (clinical + ML engineer)
$4,200
Total
~3.5 weeks
$650
$15,500
$16,150

Inspect the phase-by-phase split between compute and staff costs. section 6.12.4

Platform choice changes the cost structure. Managed fine-tuning APIs charge per training token; self-hosting charges for GPU time and requires the team to maintain infrastructure, drivers, libraries, and capacity. Start with the simplest platform that meets privacy requirements, and consider self-hosting when control or scale justifies the overhead. section 6.11.4

Distillation break-even

An illustrative distillation project costs $1,000–$2,000 upfront and saves more than $1,000 per month in API costs, with break-even in one to three months. The estimate excludes engineering time and infrastructure maintenance. Recalculate using expected token budgets, current API prices, GPU provisioning, and the quality the student retains. section 7.6.3

Table 7.9 12-month cost comparison: API teacher vs. distilled student
Cost component
API teacher
Distilled student
Upfront investment
$0
$1,500 (data gen + training)
Month 1 inference
$2,000
$300
Month 2 to 12 inference
$22,000
$3,300
Quality retention
100% (teacher)
~85% of teacher

Compare illustrative 12-month API and distilled-student costs alongside retained quality. section 7.6.3

How LLM Customization and Fine-Tuning covers it

How deep it goes
Breaks costs into request-level, upfront, and ongoing components, with illustrative break-even calculations and exclusions.
Examples
Compares shot counts, a 4B fine-tuning project, and a 12-month distilled-student deployment.
What you can do after
Evaluate adaptation and hosting options by estimating volume-sensitive costs, staff effort, break-even, and retained quality.
How it connects
Connects cost to in-context learning, fine-tuning, knowledge distillation, caching, and data privacy.

Also explained in

Compare how 17 books explain cost →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of LLM Customization and Fine-Tuning. The section links point to the parts of the book it draws on.