MMLU
MMLU is a benchmark of questions spanning diverse academic and professional subjects, used to evaluate language models’ broad knowledge and reasoning.
LLMs in Production
MMLU is a benchmark suite that evaluates broad language understanding, world knowledge, and problem-solving across subjects and difficulty levels.
- How it is used
- Evaluates broad language understanding, world knowledge, and problem-solving across subjects and difficulty levels.
- How deep it goes
- Tests applied knowledge through elementary-to-professional questions, extending beyond surface understanding without detailing question-by-question scoring.
- Examples
- Uses questions about history, mathematics, morality, and law, plus a DeepEval evaluation wrapper for a Transformers model.
- What you can do after
- Evaluate a language model’s cross-subject knowledge and reasoning with DeepEval’s MMLU benchmark.
Introduction to Generative AI, Second Edition
MMLU is a benchmark that evaluates knowledge and reasoning across multiple academic and professional subjects.
- How it is used
- Evaluates language-model knowledge and reasoning across subjects while revealing disparities between languages.
- How deep it goes
- Defines the benchmark and illustrates its language disparity through accuracy figures, but does not detail task construction or scoring procedures.
- Examples
- Uses GPT-4 results above 85% for English tasks and about 62% for Telugu tasks.
- What you can do after
- Evaluate model performance while identifying language-specific gaps hidden by aggregate scores.
Read how Introduction to Generative AI, Second Edition explains MMLU →
Related concepts
- benchmarkkind of · A benchmark is a defined task, dataset, workload, procedure, or reference measure used to assess or compare performance or capability under specified criteria or conditions.
- Llamameasured by · Llama is a family of autoregressive large language models developed by Meta and released in multiple versions and sizes.
- language modeluses · A language model is a computational model that represents patterns in linguistic tokens to estimate sequence probabilities and predict or generate language from context.
- accuracymeasures · Accuracy is the degree to which an output, prediction, or measurement agrees with a correct or reference outcome under a specified criterion.
- GLUE Benchmarkcontrasts with · The GLUE Benchmark is a standardized suite of natural-language-understanding tasks for evaluating language models across multiple linguistic domains.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.