liveBook · Manning

Break-even point

As explained in LLM Customization and Fine-Tuning

The point at which two alternatives have equal total cost under the given assumptions.

A break-even point is where the total costs of two alternatives become equal. It helps determine when a higher-fixed-cost approach is offset by lower costs per request. The crossing depends on the assumptions and expected request volume. section 4.7.2 section 4.7.1

For in-context learning (ICL), request costs include the prompt and examples on every call. Fine-tuning adds training and hosting costs, but can reduce per-request costs with a shorter prompt. At 100 ticket-classification requests per day, ICL can remain cheaper; at 100,000 per day, fine-tuning can cross over quickly. Retrieval-augmented generation (RAG) adds retrieval infrastructure and retrieved context, so it has its own cost profile. section 4.7.2

Distillation payback

An illustrative distillation project costing $1,000–$2,000 upfront may save $1,000 or more monthly in API costs, reaching break-even in roughly one to three months. This estimate excludes engineering time and ongoing training-infrastructure costs; new GPU provisioning adds costs, while falling API prices may reduce projected savings. section 7.6.3

Table 7.9 12-month cost comparison: API teacher vs. distilled student
Cost component
API teacher
Distilled student
Upfront investment
$0
$1,500 (data gen + training)
Month 1 inference
$2,000
$300
Month 2 to 12 inference
$22,000
$3,300
Quality retention
100% (teacher)
~85% of teacher

Compare the illustrative 12-month API-teacher and distilled-student costs, including upfront investment and inference. section 7.6.3

How LLM Customization and Fine-Tuning covers it

How deep it goes
Defines the crossing through fixed and per-request costs, then applies volume and flags engineering, infrastructure, and pricing omissions.
Examples
Uses ticket-classifier volumes of 100 versus 100,000 daily requests and a distillation investment with one-to-three-month payback.
What you can do after
Evaluate which approach is cheaper at an expected request volume by comparing setup, ongoing, and per-request costs.
How it connects
Connects in-context learning, fine-tuning, retrieval-augmented generation, and knowledge distillation through their cost structures.

This explainer was generated by Manning's knowledge graph from the text of LLM Customization and Fine-Tuning. The section links point to the parts of the book it draws on.