Qualitative assessment
Qualitative assessment evaluates a system or its outputs through descriptive judgments of their characteristics rather than primarily through numerical measurements.
The Art of AI Product Development
Qualitative evaluation assesses a system through descriptive observations, judgments, and feedback rather than primarily numerical measurements.
- How it is used
- Uses early user feedback to establish a search-system baseline before teams optimize retrieval or other system components.
- How deep it goes
- Progresses from manual vibe checks to user studies and structured judgments, while deferring advanced metrics until data and optimization mature.
- Examples
- Uses queries about advanced-plan feedback and API integration case studies, plus live retrieval of customer feedback for case studies.
- What you can do after
- Evaluate search usefulness, identify broad improvement areas, and establish a baseline for future changes.
Read how The Art of AI Product Development explains qualitative assessment →
LLM Customization and Fine-Tuning
Assessment of model outputs by examining their characteristics rather than relying only on numerical scores.
- How it is used
- Evaluates model outputs alongside quantitative metrics when ground truth alone cannot capture quality, safety, or fairness.
- How deep it goes
- Examines responses for helpfulness, tone, risk sensitivity, and specificity without prescribing a formal scoring rubric.
- Examples
- Contrasts base, teacher, and distilled-student answers to a Mac company-VPN prompt.
- What you can do after
- Evaluate responses for structure, specificity, and actionability to judge whether a student model suits the task.
Read how LLM Customization and Fine-Tuning explains qualitative assessment →
Also covered in
- AI Model EvaluationIn early access (MEAP)
What to know first
- accuracyAccuracy is the degree to which a result agrees with a specified truth, reference, or criterion, commonly measured as the proportion of evaluated predictions that are correct.
Related concepts
- manual evaluationimplemented by · Manual evaluation is the assessment of data, outputs, or behavior through direct human examination against relevant criteria.
- LLM-as-a-judgeimplemented by · LLM-as-a-judge is an evaluation technique in which a language model assesses or compares outputs according to specified criteria.
- quantitative evaluationcontrasts with · Quantitative evaluation is the measurement and analysis of numerical system behavior or outcomes against defined inputs, criteria, or desired results.
- token-F1contrasts with · An overlap-based evaluation metric for text generation that measures partial word overlap between generated and reference responses.
- deterministic metriccontrasts with · A deterministic metric is a measure that evaluates performance using fixed rules or assumptions about predictable outcomes, rather than probabilistic or interpretive judgments.
Questions about qualitative assessment
What is qualitative assessment?
Qualitative assessment evaluates a system or its outputs through descriptive judgments of their characteristics rather than primarily through numerical measurements.
Which Manning books explain qualitative assessment?
The Art of AI Product Development and LLM Customization and Fine-Tuning.
What should I know before learning qualitative assessment?
Manning books that teach qualitative assessment usually introduce accuracy first.
What is the difference between qualitative assessment and quantitative evaluation?
Qualitative assessment evaluates a system or its outputs through descriptive judgments of their characteristics rather than primarily through numerical measurements. Quantitative evaluation is the measurement and analysis of numerical system behavior or outcomes against defined inputs, criteria, or desired results.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.

