Cost is the money and staff effort required to train, operate, and maintain a model. At production scale, even a successful API deployment can become too expensive: one million requests a day can cost an illustrative $210,000–$300,000 per month before caching and batch discounts. Adaptation may reduce that bill, but it also transfers maintenance work to the team. section 1.4 section 6.12.4
Compare options over the expected life and volume of the deployment:
- For in-context learning, count input tokens in the instructions, examples, and query, plus output tokens. The example bank is paid for on every request. section 4.7.1
- For a self-hosted fine-tuned model, spread training and hosting costs across the requests served. Fine-tuning requires more upfront spending but can have a lower per-request cost. section 4.7.1
- Add data collection, annotation, evaluation, engineering, and ongoing infrastructure maintenance. GPU spend alone can substantially understate the investment. section 6.12.4 section 7.6.3
- Project request volume before committing to self-hosting and revisit it after the first month. Fixed capacity can be economical at high volume but wasteful when usage is low or variable. section 4.9.3
More examples in a prompt increase token costs roughly linearly, while accuracy may plateau. A fixed example bank can benefit from cached-input discounts; selecting different examples for each query reduces the shared prefix. Batch pricing may also help non-interactive workloads. Check current provider rates before comparing approaches. section 4.3.2
Table 4.2 Cost framing at different shot counts (illustrative)
|
Shots
|
Prompt tokens (illustrative)
|
Inference cost rank
|
Accuracy plateau range
|
|
1
|
~250
|
Low
|
60 to 70%
|
|
8
|
~1,800
|
Low
|
78 to 84%
|
|
50
|
~10,200
|
Medium
|
86 to 90%
|
|
100
|
~20,200
|
High
|
88 to 92%
|
|
200
|
~40,200
|
very high
|
89 to 93%
|
Compare illustrative prompt sizes, cost ranks, and accuracy ranges across shot counts. section 4.3.2
Human effort can dominate training
An illustrative 4B fine-tuning project totals $16,150 over roughly 3.5 weeks: $650 for compute and $15,500 for staff time. Data curation and evaluation account for much of the human effort, so reducing GPU time alone would leave most of the project cost intact. section 6.12.4
Table 6.14 Training cost breakdown (illustrative)
|
Phase
|
Duration
|
Compute Cost
|
Human Cost
|
Total
|
|
Data curation and quality review
|
2 weeks
|
–
|
$8,000 (clinical reviewer time)
|
$8,000
|
|
Data formatting and preprocessing
|
3 days
|
$50 (CPU instances)
|
$2,000 (ML engineer time)
|
$2,050
|
|
Training (including failed experiments)
|
2 days
|
$400 (GPU hours)
|
$1,500 (ML engineer time)
|
$1,900
|
|
Evaluation and safety testing
|
1 week
|
$200 (inference costs)
|
$4,000 (clinical + ML engineer)
|
$4,200
|
|
Total
|
~3.5 weeks
|
$650
|
$15,500
|
$16,150
|
Inspect the phase-by-phase split between compute and staff costs. section 6.12.4
Platform choice changes the cost structure. Managed fine-tuning APIs charge per training token; self-hosting charges for GPU time and requires the team to maintain infrastructure, drivers, libraries, and capacity. Start with the simplest platform that meets privacy requirements, and consider self-hosting when control or scale justifies the overhead. section 6.11.4
Distillation break-even
An illustrative distillation project costs $1,000–$2,000 upfront and saves more than $1,000 per month in API costs, with break-even in one to three months. The estimate excludes engineering time and infrastructure maintenance. Recalculate using expected token budgets, current API prices, GPU provisioning, and the quality the student retains. section 7.6.3
Table 7.9 12-month cost comparison: API teacher vs. distilled student
|
Cost component
|
API teacher
|
Distilled student
|
|
Upfront investment
|
$0
|
$1,500 (data gen + training)
|
|
Month 1 inference
|
$2,000
|
$300
|
|
Month 2 to 12 inference
|
$22,000
|
$3,300
|
|
Quality retention
|
100% (teacher)
|
~85% of teacher
|
Compare illustrative 12-month API and distilled-student costs alongside retained quality. section 7.6.3
















