Perplexity measures a language model’s uncertainty when predicting the next token in a sequence. Lower scores indicate greater confidence and prediction accuracy, making perplexity useful for evaluating fine-tuned models. section 9.3.2
Use perplexity as one part of a broader evaluation strategy:
- Compare performance before and after fine-tuning to assess adaptation to a new text style or domain. section 9.3.2
- Pair the score with task-specific and qualitative measures because perplexity does not cover relevance, coherence, factual accuracy, or task completion. section 9.3.2
Table 9.1 Fine-tuning evaluation criteria
|
Evaluation criteria
|
Description
|
|---|---|
|
Relevance
|
Gauges how well the model’s responses or outputs align with the context and intent of the input. This is especially crucial in applications such as chatbots, where providing contextually appropriate responses is key to user satisfaction. Relevance is often assessed by examining whether the model can stay on topic and provide information or responses directly applicable to the queries or tasks.
|
|
Coherence
|
Refers to the logical consistency of the model’s outputs. A fine-tuned model should generate contextually relevant, logically sound, and coherent text. This means the responses should follow a logical structure and narrative flow, making sense in the conversation or text context. Coherence is vital for maintaining user engagement and ensuring the model’s outputs are understandable and meaningful.
|
|
Accuracy
|
This particularly comes into play when the model is used for tasks involving factual information, such as educational tools, informational bots, or any application where providing correct information is critical. Accuracy is measured by how well the model’s responses align with factual correctness and objective truth.
|
|
Language fluency
|
Pertains to the grammatical and syntactical correctness of the model’s outputs. Even if a model is highly relevant, coherent, and accurate, poor language fluency can significantly detract from the user’s experience. This includes proper grammar, punctuation, and style, ensuring the text generated is correct and reads naturally to the end user.
|
Use the evaluation criteria table to place perplexity alongside broader measures such as relevance, coherence, and accuracy. section 9.3.2
