LAMBADA evaluates whether a language model can predict the exact final word of a passage using its broader narrative context. The passages are designed so that the last sentence alone is insufficient to determine the answer. section 2.2 section 8.4.2
Unlike multiple-choice benchmarks, LAMBADA does not ask the model to select among supplied candidates. It can therefore reveal degradation in precise word prediction that remains less visible on closed-set tasks. section 2.2 section 8.4.2
benchmark_tasks = ['arc_easy', 'winogrande','hellaswag', 'lambada_openai']
baseline_results = model_evaluation(
model,
tokenizer,
benchmark_tasks, limit=None
)
Evaluate LAMBADA alongside three other benchmarks using the lm-eval task name lambada_openai. The helper wraps simple_evaluate and returns scores for comparison; limit=None runs without a sample limit. section 2.2
Baseline and pruning comparison
A Gemma-3-270M baseline scored 0.43 on lambada_openai. In a separate Llama-3.2-3B attention-pruning experiment, LAMBADA dropped 9.2% with three layers removed and 19.1% with six removed, while four other benchmarks changed by less than 2% at six layers. Removing 15 layers caused scores to collapse across tasks. section 2.2 section 8.4.2
Use LAMBADA with complementary task and generation checks rather than as a complete measure of model quality. In one pruning experiment, short outputs remained coherent even though sustained generation showed factual inconsistencies. section 8.5.2

