liveBook · Manning

MMLU

Also called Massive Multitask Language Understanding

MMLU is a benchmark of questions spanning diverse academic and professional subjects, used to evaluate language models’ broad knowledge and reasoning.

How 2 Manning books explain it

LLMs in Production

MMLU is a benchmark suite that evaluates broad language understanding, world knowledge, and problem-solving across subjects and difficulty levels.

How it is used
Evaluates broad language understanding, world knowledge, and problem-solving across subjects and difficulty levels.
How deep it goes
Tests applied knowledge through elementary-to-professional questions, extending beyond surface understanding without detailing question-by-question scoring.
Examples
Uses questions about history, mathematics, morality, and law, plus a DeepEval evaluation wrapper for a Transformers model.
What you can do after
Evaluate a language model’s cross-subject knowledge and reasoning with DeepEval’s MMLU benchmark.

Read how LLMs in Production explains MMLU →

Introduction to Generative AI, Second Edition

MMLU is a benchmark that evaluates knowledge and reasoning across multiple academic and professional subjects.

How it is used
Evaluates language-model knowledge and reasoning across subjects while revealing disparities between languages.
How deep it goes
Defines the benchmark and illustrates its language disparity through accuracy figures, but does not detail task construction or scoring procedures.
Examples
Uses GPT-4 results above 85% for English tasks and about 62% for Telugu tasks.
What you can do after
Evaluate model performance while identifying language-specific gaps hidden by aggregate scores.

Read how Introduction to Generative AI, Second Edition explains MMLU →

Also covered in

What to know first

Related concepts

Questions about MMLU

What is MMLU?

MMLU is a benchmark of questions spanning diverse academic and professional subjects, used to evaluate language models’ broad knowledge and reasoning.

Which Manning books explain MMLU?

LLMs in Production and Introduction to Generative AI, Second Edition.

What should I know before learning MMLU?

Manning books that teach MMLU usually introduce language model, BLEU score, accuracy and ROUGE first.

What is the difference between MMLU and MMMU?

MMLU is a benchmark of questions spanning diverse academic and professional subjects, used to evaluate language models’ broad knowledge and reasoning. MMMU is a multimodal benchmark dataset of college-level questions across multiple disciplines, designed to evaluate multimodal understanding.

What is MMLU a kind of?

MMLU is a kind of benchmark. A benchmark is a defined, repeatable task, workload, test, dataset, or reference standard used to measure or compare systems or methods under specified conditions.

What is MMLU also called?

MMLU is also called Massive Multitask Language Understanding.

These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.