Cost optimization systematically reduces computational and operational expense while preserving required performance and quality. In production LLM systems, token consumption makes every prompt, retrieved document, and generated response a billable choice, so cost management belongs alongside reliability and performance engineering. section 13.4
Apply cost optimization through complementary production practices:
- Compare pay-per-token APIs with reserved, self-hosted compute. Cloud APIs suit workloads below roughly 100,000 queries per month; reserved infrastructure becomes more attractive beyond approximately 500,000 queries, depending on model size and requirements. section 13.4.1
- Route simple classification and routing tasks to smaller models, use balanced models for standard generation, and reserve premium models for complex reasoning. Tiered routing can reduce costs by 60% to 80%, but requires monitoring, versioning, and threshold tuning. section 13.4.2
- Track aggregate spending, per-query costs, token usage, query types, models, users, and quality metrics. Set automated alerts at 80% of budget thresholds to investigate expensive outliers before hard limits are reached. section 13.4.5
- Combine tiered routing with semantic caching for repeated questions and prompt optimization that reduces token consumption. section 13.5
Tiered model selection
A cost-conscious documentation assistant uses Gemini 2.5 Flash Lite to classify queries as SIMPLE, MODERATE, or COMPLEX; Gemini 2.5 Flash handles ordinary answers; Gemini 2.5 Pro handles complex queries. The classifier prompt stays deliberately minimal to avoid wasting tokens. section 13.4.2
Listing 13.14 Cost-optimized DakkaBot with tiered model selection
class CostOptimizedDakkaBotCore:
def __init__(self):
self.router_llm = ChatGoogleGenerativeAI(
model="gemini-2.5-flash-lite", #1
temperature=0.0
)
self.generation_llm = ChatGoogleGenerativeAI(
model="gemini-2.5-flash", #2
temperature=0.0
)
self.premium_llm = ChatGoogleGenerativeAI(
model="gemini-2.5-pro", #3
temperature=0.0
)
async def route_query(self, query: str) -> str:
"""Use cheap model to classify query complexity"""
routing_prompt = f"""
Classify this query complexity: {query}
Options: SIMPLE, MODERATE, COMPLEX
SIMPLE: Basic factual questions, single-step procedures
MODERATE: Multi-step processes, troubleshooting
COMPLEX: Analysis, planning, integration across systems
Classification:""" #4
result = self.router_llm.invoke(routing_prompt) #5
return result.content.strip()
async def process_query_optimized(self, query: str) -> Dict[str, Any]:
complexity = await self.route_query(query) #6
if complexity == "SIMPLE":
llm = self.router_llm #7
elif complexity == "MODERATE":
llm = self.generation_llm
else:
llm = self.premium_llm
Inspect the runnable CostOptimizedDakkaBotCore configuration, including the three model clients and the query-routing method. section 13.4.2
Optimization is an operational feedback loop rather than a one-time configuration. Langfuse can capture token usage, estimated costs, performance, prompt versions, and quality correlations, exposing whether expensive queries deliver proportional value. section 13.4.5
