Cost is the expense of running a trained model to produce outputs. In LLM applications, inference cost usually rises with model size and the number of tokens processed or generated, making model choice an operational decision rather than an accuracy-only decision. section 1.5
Manage cost by balancing it against two closely related model-selection concerns:
- Accuracy: larger models are often more accurate but cost more and respond more slowly. section 1.5
- Speed: smaller models reduce latency and expense, but may sacrifice accuracy. section 1.5
- Workflow design: assign different models to different tasks instead of using the largest model everywhere. section 1.7
- Knowledge access: use RAG for dynamic runtime context when reducing retraining needs and costs matters; use fine-tuning for embedded domain knowledge and specialized efficiency. section 1.7
Task-specific model routing
A workflow can use GPT-5-nano for fast summarization, GPT-5-mini for answer synthesis, and full GPT-5 for complex query routing. This concentrates expensive inference where higher accuracy is justified. section 1.7
