Semantic evaluation
Semantic evaluation assesses whether content conveys accurate, appropriate, or intended meaning rather than relying solely on wording or surface similarity.
Building LLM Applications with DSPy
A technique for evaluating whether texts express similar meanings without requiring similar wording.
- How it is used
- Compares generated summaries with ground-truth summaries when matching words cannot reliably establish matching meaning.
- How deep it goes
- Defines meaning-based comparison and identifies embeddings and LLM judges, while deferring judge-prompt implementation.
- Examples
- Uses paired plausible summaries of literary works to show why a document may have multiple valid summaries.
- What you can do after
- Evaluate/understand when meaning-based comparison is needed and which criteria an LLM judge should prioritize.
Read how Building LLM Applications with DSPy explains semantic evaluation →
Machine Learning Platform Engineering
Semantic correctness is the degree to which generated content conveys the intended meaning accurately.
- How it is used
- Evaluates LLM responses when nondeterministic outputs make exact-string assertions inadequate.
- How deep it goes
- Defines semantic, factual, and guideline-based assessment but stops short of specifying detailed scoring procedures.
- Examples
- Uses a RAG documentation assistant and DeepEval to illustrate evaluation of variable but potentially valid responses.
- What you can do after
- Apply it via DeepEval to assess LLM responses beyond exact string matching.
Read how Machine Learning Platform Engineering explains semantic evaluation →
Related concepts
- exact-match accuracyalternative to · Exact-match accuracy evaluates outputs by whether they exactly equal their references under specified comparison rules.
- lexical comparisoncontrasts with · A lexical comparison evaluates text similarity by comparing words or n-grams between texts.
- DeepEvalused by · DeepEval is an open-source Python framework for testing and evaluating large language models and applications built with them.
- evaluationmeasured by · Evaluation is the systematic assessment of a system, result, or approach against relevant criteria, evidence, or objectives.
- embeddinguses · A numerical vector representation of an item or input that encodes features or relationships useful for computational tasks.
- f-scorecontrasts with · An F-score is a classification or evaluation metric combining precision and recall via their weighted harmonic mean; F1 is the equal-weight case.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.