liveBook · Manning

DeepEval

DeepEval is an open-source Python framework for testing and evaluating large language models and applications built with them.

How 2 Manning books explain it

Machine Learning Platform Engineering

DeepEval is an open-source evaluation framework for testing and measuring the quality of large language model applications.

How it is used
Provides semantic quality testing for production LLM applications whose valid responses vary between runs.
How deep it goes
Builds evaluator configuration, async test adaptation, custom criteria, thresholds, and multi-metric cases, without detailing framework internals.
Examples
Uses security-boundary deflection tests and combined correctness, relevance, and completeness metrics.
What you can do after
Apply it via G-Eval and built-in metrics to automate semantic, policy, correctness, relevance, and completeness checks.

Read how Machine Learning Platform Engineering explains DeepEval →

Retrieval Augmented Generation, The Foundational Ideas

DeepEval is a software framework for evaluating large language model applications and generating evaluation data.

How it is used
Evaluates RAG and other LLM applications through reusable LLM-judged metrics and document-grounded synthetic test data.
How deep it goes
Provides implementation-ready metric and data-generation workflows, but leaves numeric-response parsing and production test-set design to the user.
Examples
Uses faithfulness scoring, configurable question evolution, cross-document questions, unanswerable cases, and captured successful interactions.
What you can do after
Apply it via G-Eval metrics and the Synthesizer to build broader RAG test suites and detect regressions.

Read how Retrieval Augmented Generation, The Foundational Ideas explains DeepEval →

Also covered in

What to know first

Related concepts

Questions about DeepEval

What is DeepEval?

DeepEval is an open-source Python framework for testing and evaluating large language models and applications built with them.

Which Manning books explain DeepEval?

Machine Learning Platform Engineering and Retrieval Augmented Generation, The Foundational Ideas.

What should I know before learning DeepEval?

Manning books that teach DeepEval usually introduce BLEU score, generative modeling, language model and ROUGE first.

What is the difference between DeepEval and Ragas?

DeepEval is an open-source Python framework for testing and evaluating large language models and applications built with them. Ragas is an open-source framework and Python library for evaluating retrieval-augmented generation systems and applications.

What is DeepEval a kind of?

DeepEval is a kind of evaluation framework. An evaluation framework is software that provides methods or infrastructure for assessing a system's behavior or performance.

What are the parts of DeepEval?

DeepEval includes synthesizer agent.

These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.