liveBook · Manning

Qualitative assessment

As explained in LLM Customization and Fine-Tuning

Assessment of model outputs by examining their characteristics rather than relying only on numerical scores.

Qualitative assessment examines model outputs directly instead of relying only on numerical scores. Human reviewers or LLM judges can assess qualities such as helpfulness, tone, and risk sensitivity—criteria that a single metric cannot capture. Reviewing results by relevant segments can also reveal weaknesses hidden by aggregate scores. section 5.11

Comparing VPN instructions

For a prompt about connecting a Mac to a company VPN, the base model gives generic advice, while the teacher model provides specific numbered steps. The distilled student keeps the structure but may omit details. Comparing these outputs reveals differences in actionability and specificity that Token-F1 alone misses. That gap may matter more for exact procedures than for general troubleshooting. section 7.7.1

Table 5.9 Responsible AI risks and controls in fine-tuning
Area
Key Risk
What to Watch For
Typical Mitigation
Privacy and Compliance
Data leakage, regulatory violations
PII in training data, unclear data ownership
Anonymization, consent checks, and data residency controls
Fairness and Bias
Discriminatory or uneven outputs
Skewed examples, demographic imbalance
Dataset rebalancing, bias tests, targeted evaluations
Safety
Over-compliance, harmful responses
Loss of refusal behavior after fine-tuning
Safety prompt sets, refusal examples, and comparative testing
Evaluation
False confidence in model quality
Overreliance on a single metric
Mixed metrics, human review, segmented analysis
Transparency
Misuse or misinterpretation
Unclear limitations or intended use
Model cards, documentation, usage boundaries
Monitoring
Degradation over time
New failure modes, user complaints
Logging, feedback loops, iterative LoRA updates

Use the risk-and-control table to place human review alongside mixed metrics and segmented analysis when evaluating fine-tuned models. section 5.11

How LLM Customization and Fine-Tuning covers it

How deep it goes
Examines responses for helpfulness, tone, risk sensitivity, and specificity without prescribing a formal scoring rubric.
Examples
Contrasts base, teacher, and distilled-student answers to a Mac company-VPN prompt.
What you can do after
Evaluate responses for structure, specificity, and actionability to judge whether a student model suits the task.
How it connects
Connects model evaluation with human review, LLM judging, quantitative metrics, and knowledge distillation.

Also explained in

Compare how 2 books explain qualitative assessment →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of LLM Customization and Fine-Tuning. The section links point to the parts of the book it draws on.