DeepEval
DeepEval is an open-source Python framework for testing and evaluating large language models and applications built with them.
Machine Learning Platform Engineering
DeepEval is an open-source evaluation framework for testing and measuring the quality of large language model applications.
- How it is used
- Provides semantic quality testing for production LLM applications whose valid responses vary between runs.
- How deep it goes
- Builds evaluator configuration, async test adaptation, custom criteria, thresholds, and multi-metric cases, without detailing framework internals.
- Examples
- Uses security-boundary deflection tests and combined correctness, relevance, and completeness metrics.
- What you can do after
- Apply it via G-Eval and built-in metrics to automate semantic, policy, correctness, relevance, and completeness checks.
Read how Machine Learning Platform Engineering explains DeepEval →
Retrieval Augmented Generation, The Foundational Ideas
DeepEval is a software framework for evaluating large language model applications and generating evaluation data.
- How it is used
- Evaluates RAG and other LLM applications through reusable LLM-judged metrics and document-grounded synthetic test data.
- How deep it goes
- Provides implementation-ready metric and data-generation workflows, but leaves numeric-response parsing and production test-set design to the user.
- Examples
- Uses faithfulness scoring, configurable question evolution, cross-document questions, unanswerable cases, and captured successful interactions.
- What you can do after
- Apply it via G-Eval metrics and the Synthesizer to build broader RAG test suites and detect regressions.
Read how Retrieval Augmented Generation, The Foundational Ideas explains DeepEval →
Also covered in
- Generative AI in Action
- Context EngineeringIn early access (MEAP)
What to know first
- BLEU scoreBLEU is an automatic text-generation evaluation metric that compares candidate text with reference text using modified n-gram precision and a brevity penalty.
- generative modelingGenerative modeling is a machine-learning approach that learns patterns or a data distribution to generate new samples resembling the data.
- language modelA language model is a computational model that represents patterns in linguistic sequences to assign probabilities, predict, generate, or evaluate language.
- ROUGEROUGE is a family of automatic text-generation evaluation metrics that compare generated text with reference text using lexical overlap, often emphasizing recall.
Related concepts
- Ragascontrasts with · Ragas is an open-source framework and Python library for evaluating retrieval-augmented generation systems and applications.
- semantic evaluationuses · Semantic evaluation assesses whether content conveys accurate, appropriate, or intended meaning rather than relying solely on wording or surface similarity.
- synthesizer agenthas part · A synthesizer agent combines information or outputs from multiple sources or agents into a coherent composite result.
- evaluationimplements · Evaluation is the systematic assessment of a system, model, process, or outcome against relevant criteria or objectives using evidence.
Questions about DeepEval
What is DeepEval?
DeepEval is an open-source Python framework for testing and evaluating large language models and applications built with them.
Which Manning books explain DeepEval?
Machine Learning Platform Engineering and Retrieval Augmented Generation, The Foundational Ideas.
What should I know before learning DeepEval?
Manning books that teach DeepEval usually introduce BLEU score, generative modeling, language model and ROUGE first.
What is the difference between DeepEval and Ragas?
DeepEval is an open-source Python framework for testing and evaluating large language models and applications built with them. Ragas is an open-source framework and Python library for evaluating retrieval-augmented generation systems and applications.
What is DeepEval a kind of?
DeepEval is a kind of evaluation framework. An evaluation framework is software that provides methods or infrastructure for assessing a system's behavior or performance.
What are the parts of DeepEval?
DeepEval includes synthesizer agent.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.

