A break-even point is where the total costs of two alternatives become equal. It helps determine when a higher-fixed-cost approach is offset by lower costs per request. The crossing depends on the assumptions and expected request volume. section 4.7.2 section 4.7.1
For in-context learning (ICL), request costs include the prompt and examples on every call. Fine-tuning adds training and hosting costs, but can reduce per-request costs with a shorter prompt. At 100 ticket-classification requests per day, ICL can remain cheaper; at 100,000 per day, fine-tuning can cross over quickly. Retrieval-augmented generation (RAG) adds retrieval infrastructure and retrieved context, so it has its own cost profile. section 4.7.2
Distillation payback
An illustrative distillation project costing $1,000–$2,000 upfront may save $1,000 or more monthly in API costs, reaching break-even in roughly one to three months. This estimate excludes engineering time and ongoing training-infrastructure costs; new GPU provisioning adds costs, while falling API prices may reduce projected savings. section 7.6.3
Table 7.9 12-month cost comparison: API teacher vs. distilled student
|
Cost component
|
API teacher
|
Distilled student
|
|
Upfront investment
|
$0
|
$1,500 (data gen + training)
|
|
Month 1 inference
|
$2,000
|
$300
|
|
Month 2 to 12 inference
|
$22,000
|
$3,300
|
|
Quality retention
|
100% (teacher)
|
~85% of teacher
|
Compare the illustrative 12-month API-teacher and distilled-student costs, including upfront investment and inference. section 7.6.3
