Degradation is performance decay: a model keeps operating but produces poorer outputs and no longer meets quality expectations. Unlike an outage, it can remain silent until quality monitoring detects it. Language models are especially exposed because language, vocabulary, and word meanings continually change. section 6.2.5
Treat degradation as an ML monitoring problem alongside ordinary system monitoring. Track incoming prompts and generated text, then evaluate signals such as readability, complexity, toxicity, and similarity to known prompt-injection attacks. Data-drift detection is harder for unstructured text than for numerical data, so production evaluation practices remain necessary. section 6.2.5
Monitoring text with whylogs and LangKit
A rolling whylogs logger can collect text profiles every five minutes and use LangKit metrics to monitor prompts or generated responses. The resulting profiles help reveal when an LLM service begins failing quietly. section 6.2.5
Listing 6.11 Using whylogs and LangKit to monitor text data
import os
import pandas as pd
import whylogs as why
from langkit import llm_metrics
from datasets import load_dataset
OUTPUT_DIR = "logs"
class LoggingApp:
def __init__(self):
"""
Sets up a logger that collects profiles and writes them
locally every 5 minutes. By setting the schema with langkit
we get useful metrics for LLMs.
"""
self.logger = why.logger(
mode="rolling",
interval=5,
when="M",
base_name="profile_",
schema=llm_metrics.init(),
)
self.logger.append_writer("local", base_dir=OUTPUT_DIR)
def close(self):
self.logger.close()
def consume(self, text):
self.logger.log(text)
def driver(app):
"""Driver function to run the app manually"""
data = load_dataset(
"shahules786/OA-cornell-movies-dialog",
split="train",
streaming=True,
)
data = iter(data)
for text in data:
app.consume(text)
if __name__ == "__main__":
app = LoggingApp() #1
driver(app)
app.close()
pd.set_option("display.max_columns", None) #2
all_files = [ #3
f for f in os.listdir(OUTPUT_DIR) if f.startswith("profile_")
]
path = os.path.join(OUTPUT_DIR, all_files[0])
result_view = why.read(path).view()
print(result_view.to_pandas().head())
Inspect the Python listing for a rolling logger that records LLM-oriented text metrics and writes profiles locally. section 6.2.5
