MMLU
MMLU is a benchmark of questions spanning diverse academic and professional subjects, used to evaluate language models’ broad knowledge and reasoning.
LLMs in Production
MMLU is a benchmark suite that evaluates broad language understanding, world knowledge, and problem-solving across subjects and difficulty levels.
- How it is used
- Evaluates broad language understanding, world knowledge, and problem-solving across subjects and difficulty levels.
- How deep it goes
- Tests applied knowledge through elementary-to-professional questions, extending beyond surface understanding without detailing question-by-question scoring.
- Examples
- Uses questions about history, mathematics, morality, and law, plus a DeepEval evaluation wrapper for a Transformers model.
- What you can do after
- Evaluate a language model’s cross-subject knowledge and reasoning with DeepEval’s MMLU benchmark.
Introduction to Generative AI, Second Edition
MMLU is a benchmark that evaluates knowledge and reasoning across multiple academic and professional subjects.
- How it is used
- Evaluates language-model knowledge and reasoning across subjects while revealing disparities between languages.
- How deep it goes
- Defines the benchmark and illustrates its language disparity through accuracy figures, but does not detail task construction or scoring procedures.
- Examples
- Uses GPT-4 results above 85% for English tasks and about 62% for Telugu tasks.
- What you can do after
- Evaluate model performance while identifying language-specific gaps hidden by aggregate scores.
Read how Introduction to Generative AI, Second Edition explains MMLU →
Also covered in
- The Art of AI Product Development
- Generative AI in Action
- AI-Powered Search
- Building Specialized AI SystemsIn early access (MEAP)
What to know first
- language modelA language model is a computational model that represents patterns in linguistic sequences to assign probabilities, predict, generate, or evaluate language.
- BLEU scoreBLEU is an automatic text-generation evaluation metric that compares candidate text with reference text using modified n-gram precision and a brevity penalty.
- accuracyAccuracy is the degree to which a result agrees with a specified truth, reference, or criterion, commonly measured as the proportion of evaluated predictions that are correct.
- ROUGEROUGE is a family of automatic text-generation evaluation metrics that compare generated text with reference text using lexical overlap, often emphasizing recall.
Related concepts
- benchmarkkind of · A benchmark is a defined, repeatable task, workload, test, dataset, or reference standard used to measure or compare systems or methods under specified conditions.
- accuracymeasures · Accuracy is the degree to which a result agrees with a specified truth, reference, or criterion, commonly measured as the proportion of evaluated predictions that are correct.
- language modeluses · A language model is a computational model that represents patterns in linguistic sequences to assign probabilities, predict, generate, or evaluate language.
- Llamameasured by · Llama is a family of large language models developed by Meta.
- GLUE Benchmarkcontrasts with · The GLUE Benchmark is a suite of natural-language-understanding tasks for evaluating and comparing models across diverse language tasks.
Questions about MMLU
What is MMLU?
MMLU is a benchmark of questions spanning diverse academic and professional subjects, used to evaluate language models’ broad knowledge and reasoning.
Which Manning books explain MMLU?
LLMs in Production and Introduction to Generative AI, Second Edition.
What should I know before learning MMLU?
Manning books that teach MMLU usually introduce language model, BLEU score, accuracy and ROUGE first.
What is the difference between MMLU and MMMU?
MMLU is a benchmark of questions spanning diverse academic and professional subjects, used to evaluate language models’ broad knowledge and reasoning. MMMU is a multimodal benchmark dataset of college-level questions across multiple disciplines, designed to evaluate multimodal understanding.
What is MMLU a kind of?
MMLU is a kind of benchmark. A benchmark is a defined, repeatable task, workload, test, dataset, or reference standard used to measure or compare systems or methods under specified conditions.
What is MMLU also called?
MMLU is also called Massive Multitask Language Understanding.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.

