liveBook · Manning

Semantic evaluation

Semantic evaluation assesses whether content conveys accurate, appropriate, or intended meaning rather than relying solely on wording or surface similarity.

How 2 Manning books explain it

Building LLM Applications with DSPy

A technique for evaluating whether texts express similar meanings without requiring similar wording.

How it is used
Evaluates generated summaries against ground-truth summaries when word overlap is insufficient.
How deep it goes
Defines meaning-based comparison and contrasts embeddings with LLM judges, without specifying an embedding calculation or judge implementation.
Examples
Uses pairs of plausible summaries of the same document to illustrate differing wording and emphases.
What you can do after
Evaluate/understand when semantic comparison is useful and which summary characteristics need explicit evaluation criteria.

Read how Building LLM Applications with DSPy explains semantic evaluation →

Machine Learning Platform Engineering

Semantic correctness is the degree to which generated content conveys the intended meaning accurately.

How it is used
Evaluates LLM responses when nondeterministic outputs make exact-string assertions inadequate.
How deep it goes
Defines semantic, factual, and guideline-based assessment but stops short of specifying detailed scoring procedures.
Examples
Uses a RAG documentation assistant and DeepEval to illustrate evaluation of variable but potentially valid responses.
What you can do after
Apply it via DeepEval to assess LLM responses beyond exact string matching.

Read how Machine Learning Platform Engineering explains semantic evaluation →

Also covered in

What to know first

Related concepts

Questions about semantic evaluation

What is semantic evaluation?

Semantic evaluation assesses whether content conveys accurate, appropriate, or intended meaning rather than relying solely on wording or surface similarity.

Which Manning books explain semantic evaluation?

Building LLM Applications with DSPy and Machine Learning Platform Engineering.

What should I know before learning semantic evaluation?

Manning books that teach semantic evaluation usually introduce retrieval-augmented generation, embedding and vector database first.

What is the difference between semantic evaluation and exact-match accuracy?

Semantic evaluation assesses whether content conveys accurate, appropriate, or intended meaning rather than relying solely on wording or surface similarity. Exact-match accuracy is the proportion of predictions or outputs that equal their reference answers under specified comparison rules.

These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.