Semantic evaluation
Semantic evaluation assesses whether content conveys accurate, appropriate, or intended meaning rather than relying solely on wording or surface similarity.
Building LLM Applications with DSPy
A technique for evaluating whether texts express similar meanings without requiring similar wording.
- How it is used
- Evaluates generated summaries against ground-truth summaries when word overlap is insufficient.
- How deep it goes
- Defines meaning-based comparison and contrasts embeddings with LLM judges, without specifying an embedding calculation or judge implementation.
- Examples
- Uses pairs of plausible summaries of the same document to illustrate differing wording and emphases.
- What you can do after
- Evaluate/understand when semantic comparison is useful and which summary characteristics need explicit evaluation criteria.
Read how Building LLM Applications with DSPy explains semantic evaluation →
Machine Learning Platform Engineering
Semantic correctness is the degree to which generated content conveys the intended meaning accurately.
- How it is used
- Evaluates LLM responses when nondeterministic outputs make exact-string assertions inadequate.
- How deep it goes
- Defines semantic, factual, and guideline-based assessment but stops short of specifying detailed scoring procedures.
- Examples
- Uses a RAG documentation assistant and DeepEval to illustrate evaluation of variable but potentially valid responses.
- What you can do after
- Apply it via DeepEval to assess LLM responses beyond exact string matching.
Read how Machine Learning Platform Engineering explains semantic evaluation →
Also covered in
- Building Reliable AI Systems
- Retrieval Augmented Generation, The Foundational IdeasIn early access (MEAP)
What to know first
- retrieval-augmented generationRetrieval-augmented generation is a technique that retrieves relevant information from external sources and supplies it as context to a generative model when producing an output.
- embeddingAn embedding is a numerical vector representation of an item in a feature space, encoding selected properties or relationships for computational tasks such as comparison or retrieval.
- vector databaseA vector database is a data-management system that stores and indexes numerical vector representations and supports retrieval through similarity or distance searches.
Related concepts
- DeepEvalused by · DeepEval is an open-source Python framework for testing and evaluating large language models and applications built with them.
- evaluationmeasured by · Evaluation is the systematic assessment of a system, model, process, or outcome against relevant criteria or objectives using evidence.
- exact-match accuracyalternative to · Exact-match accuracy is the proportion of predictions or outputs that equal their reference answers under specified comparison rules.
- lexical comparisoncontrasts with · A lexical comparison evaluates text similarity by comparing words or n-grams between texts.
- embeddinguses · An embedding is a numerical vector representation of an item in a feature space, encoding selected properties or relationships for computational tasks such as comparison or retrieval.
- f-scorecontrasts with · An F-score is a classification metric that combines precision and recall through a weighted harmonic mean; F1 weights them equally.
Questions about semantic evaluation
What is semantic evaluation?
Semantic evaluation assesses whether content conveys accurate, appropriate, or intended meaning rather than relying solely on wording or surface similarity.
Which Manning books explain semantic evaluation?
Building LLM Applications with DSPy and Machine Learning Platform Engineering.
What should I know before learning semantic evaluation?
Manning books that teach semantic evaluation usually introduce retrieval-augmented generation, embedding and vector database first.
What is the difference between semantic evaluation and exact-match accuracy?
Semantic evaluation assesses whether content conveys accurate, appropriate, or intended meaning rather than relying solely on wording or surface similarity. Exact-match accuracy is the proportion of predictions or outputs that equal their reference answers under specified comparison rules.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.

