LLMOps is the operational practice of managing large language models and the applications built with them. It extends MLOps to cover prompts, fine-tuning, deployment, monitoring, and governance. The goal is to move beyond a working proof of concept to a reliable, secure production service. section 11.6 section 11.0
A production plan should address:
- Serving and scale: choose infrastructure that fits the model and workload, then plan for load balancing, auto-scaling, latency, quotas, and rate limits. section 10.2.2 section 11.4
- Visibility and improvement: log, trace, and monitor application behavior; track performance and feed findings back into updates. section 11.7 section 11.6
- Controls: protect data and models with encryption and access controls, meet applicable compliance requirements, and account for bias, hallucination, and prompt injection. section 10.2.2 section 11.6
Hosting an LLM for internal customers
An enterprise hosting an LLM in its data center must select an appropriate model, manage the workflow and GPU infrastructure, and monitor performance. Load balancing and auto-scaling help it serve multiple internal customers; encryption, access controls, and compliance checks protect sensitive data. This requires continuing investment and technical expertise. section 10.2.2
Table 11.4 Differences between MLOps and LLMOps
|
Area
|
Traditional MLOps
|
LLMOps
|
|---|---|---|
|
Target audience
|
ML engineers, data scientists
|
Application developers, ML engineering, and data scientists
|
|
Components
|
Model, data, inference environments, features
|
LLMs, prompts, tokens, generations, APIs, embeddings, vector databases
|
|
Metrics
|
Accuracy (F1 score, precision, recall, etc.)
|
Quality (similarity), groundedness (accuracy), cost (tokens), latency, evaluations (Perplexity, BLEU, ROUGE, etc.)
|
|
Models
|
Typically built from scratch
|
Typically, prebuilt with inference via an API and multiple versions in production simultaneously
|
|
Ethical concerns
|
Bias in training data
|
Misuse and generation of harmful, fake, and biased output
|
Compare LLMOps with traditional MLOps across components, metrics, models, and ethical concerns. Prompts, tokens, generated outputs, groundedness, and token cost make the operational scope more specific than model accuracy alone. section 11.6
