appendix-b

Appendix B. Capabilities evaluation with lm-evaluation-harness

 

You've performed transformations on models, removing layers, applying selective pruning and knowledge distillation, among others. These modifications need to be evaluated to determine if the resulting model meets expectations.

Evaluating LLMs requires standardized datasets, reproducible metrics, and a consistent way to measure capabilities ranging from common sense reasoning to factual knowledge.

EleutherAI’s lm-evaluation-harness (lm_eval, https://github.com/EleutherAI/lm-evaluation-harness) has become the de facto standard for evaluating language models. When you read a pruning paper that reports "HellaSwag accuracy: 0.583", they're using lm_eval. The framework offers over 200 standardized tasks with pre-processed datasets, consistent splits, and metrics calculated the same way as in the official benchmarks.

Using lm_eval instead of implementing your own evaluation avoids the errors that wreck reproducibility. A slightly different data split, a different way of calculating normalized accuracy, or inconsistent padding handling can make your numbers difficult to compare with the rest of the community. It also supports few-shot evaluation natively, so you can reproduce the exact conditions of the paper you want to compare with. And you don’t have to maintain evaluation code as new benchmarks appear, because the community adds them to lm_eval.

B.1 Evaluating models in memory

B.2 Example