Pay-per-token pricing charges for language-model usage according to processed input and output tokens. Unlike provisioned compute, costs vary with prompt context, response length, and query complexity, making prompt efficiency an operational concern. section 12.1.1
Production decisions typically involve these trade-offs:
- Managed endpoints provide usage-based billing, no infrastructure overhead, linear scaling costs, and straightforward multi-endpoint setup. section 13.4.1
- Self-hosting provides greater control but adds GPUs, memory, load balancing, monitoring, DevOps, and fixed infrastructure costs. section 13.4.1
- Match model capability to task requirements instead of defaulting to the most capable and expensive model. section 13.4.1
DakkaBot cost comparison
A 2,000-token input and 500-token output costs approximately $0.035 with GPT-4, $0.003 with Gemini Flash, or $0.0005 with a hosted open-source model, excluding infrastructure where applicable. At 1,000 daily queries, direct model costs are about $35, $3, and $0.50 respectively. section 13.4.1
Figure 12.5 Build-time setup (top) ingests documents into a vector database and configures the LLM. Query-time flow (middle) handles user questions through embedding, retrieval, augmentation, and generation. Implementation requires three lines of LangChain code.
Trace where token usage arises in a RAG application: document ingestion and configuration occur at build time, while embedding, retrieval, augmentation, and generation occur for each query. section 12.2.1
