When an LLM application moves into production, a successful API response is not enough to show that the whole system is working well. Observability combines monitoring, logging, and tracing so teams can understand application behavior, find problems, and make informed scaling and optimization decisions. section 11.4.7 section 11.0
Use three complementary views:
- Monitor response times, throughput, error rates, and resource utilization to assess application health. section 11.4.7
- Log requests and responses, including prompts and model outputs, to investigate behavior and debug failures. section 11.4.7
- Trace a request across models and services to locate bottlenecks in a complex application. section 11.4.7
Instrument a chat-completion call
A Python chat example sends requests to an OpenAI client and measures the elapsed time around each completion. It also counts prompt, conversation, and completion tokens. MLflow provides experiment tracking for examining the resulting metrics and prompt-and-response details. The example is a starting point for instrumentation, not a complete production monitoring setup. section 11.4.7
Listing 11.4 MLflow observability example
import prometheus_client as prom
import mlflow
...
# Set OpenAI API key
API_KEY = os.getenv("OPENAI_API_BOOK_KEY")
MODEL = "gpt-3.5-turbo"
MLFLOW_URI = "http://localhost:5000"
...
# Initialize OpenAI client
client = OpenAI(api_key=API_KEY)
# Set MLflow tracking URI
mlflow.set_tracking_uri(MLFLOW_URI)
mlflow.set_experiment("GenAI_book")
def generate_text(conversation, max_tokens=100)->str:
start_time = time.time()
response = client.chat.completions.create(
model=MODEL,
messages=conversation,
)
latency = time.time() - start_time
message_response = response.choices[0].message.content
# Count tokens in the prompt, and the completion
prompt_tokens = count_tokens(conversation[-1]['content'])
conversation_tokens = count_tokens(str(conversation))
completion_tokens = count_tokens(message_response)
# Log metrics using MLflow
with mlflow.start_run():
mlflow.log_metrics({
"request_count": 1,
"request_latency": latency,
"prompt_tokens": prompt_tokens,
"completion_tokens": completion_tokens,
"conversation_tokens": conversation_tokens
})
mlflow.log_params({
"model": MODEL,
"temperature": TEMPERATURE,
"top_p": TOP_P,
"frequency_penalty": FREQUENCY_PENALTY,
"presence_penalty": PRESENCE_PENALTY
})
return message_response
if __name__ == "__main__":
conversation = [{"role": "system", "content":
"You are a helpful assistant."}]
while True:
user_input = input(f"You: ")
conversation.append({"role": "user", "content": user_input})
output = generate_text(conversation, 256)
print_ai_output(output)
conversation.append({"role": "assistant", "content": output})
The listing shows where the chat call is timed and token counts are collected, making the measurement points concrete. section 11.4.7
Choose tooling to match the question: MLflow supports tracking and monitoring, while Traceloop and Prompt flow are also presented as observability options. Treat captured prompts and outputs as a security and compliance consideration; production deployment calls for access controls and encryption alongside observability. section 11.4.7 section 11.7
