A deployed generative AI application needs continuous monitoring to reveal whether it remains reliable, performant, and responsible as demand and use change. Monitoring measures application health; logging records diagnostic details, while tracing follows requests across services. Together, these are parts of observability. section 1.9 section 11.4.7
Choose signals that help the team detect problems and decide what to change:
- Measure response times, throughput, error rates, and resource utilization to assess health and guide scaling. section 11.4.7
- Log requests, responses, errors, prompts, and model outputs to investigate failures and understand behavior; trace requests to locate bottlenecks across multiple services. section 11.4.7 section 10.2.2
- Create anomaly alerts and plan for downtime and API limits. Pair continuous measurement with performance tests under simulated load. section 10.2.2 section 11.5.1
Instrumenting a chat application
A chat application can time each completion request, count tokens, and record the resulting metrics in MLflow. Prometheus can collect operational metrics, while a dashboard makes trends visible. Under a load test, response times and throughput help reveal whether the service needs optimization or more capacity. section 11.4.7 section 11.5.1
Listing 11.4 MLflow observability example
import prometheus_client as prom
import mlflow
...
# Set OpenAI API key
API_KEY = os.getenv("OPENAI_API_BOOK_KEY")
MODEL = "gpt-3.5-turbo"
MLFLOW_URI = "http://localhost:5000"
...
# Initialize OpenAI client
client = OpenAI(api_key=API_KEY)
# Set MLflow tracking URI
mlflow.set_tracking_uri(MLFLOW_URI)
mlflow.set_experiment("GenAI_book")
def generate_text(conversation, max_tokens=100)->str:
start_time = time.time()
response = client.chat.completions.create(
model=MODEL,
messages=conversation,
)
latency = time.time() - start_time
message_response = response.choices[0].message.content
# Count tokens in the prompt, and the completion
prompt_tokens = count_tokens(conversation[-1]['content'])
conversation_tokens = count_tokens(str(conversation))
completion_tokens = count_tokens(message_response)
# Log metrics using MLflow
with mlflow.start_run():
mlflow.log_metrics({
"request_count": 1,
"request_latency": latency,
"prompt_tokens": prompt_tokens,
"completion_tokens": completion_tokens,
"conversation_tokens": conversation_tokens
})
mlflow.log_params({
"model": MODEL,
"temperature": TEMPERATURE,
"top_p": TOP_P,
"frequency_penalty": FREQUENCY_PENALTY,
"presence_penalty": PRESENCE_PENALTY
})
return message_response
if __name__ == "__main__":
conversation = [{"role": "system", "content":
"You are a helpful assistant."}]
while True:
user_input = input(f"You: ")
conversation.append({"role": "user", "content": user_input})
output = generate_text(conversation, 256)
print_ai_output(output)
conversation.append({"role": "assistant", "content": output})
The MLflow example shows a chat-completion call instrumented with elapsed time and prompt, conversation, and completion token counts. section 11.4.7
Monitoring belongs in the deployment plan, not just the launch checklist. Connect logs and alerts to a response plan, and use observed performance to guide caching, token management, load balancing, or auto-scaling. Because prompts and outputs may contain sensitive enterprise data, plan their collection alongside privacy, access controls, and data minimization. section 10.2.1 section 11.5.1 section 10.2.2
