liveBook · Manning

LAMBADA

As explained in Rearchitecting LLMs

LAMBADA is a language-modeling benchmark that evaluates whether a model can predict a target word from a textual passage using its preceding context.

LAMBADA evaluates whether a language model can predict the exact final word of a passage using its broader narrative context. The passages are designed so that the last sentence alone is insufficient to determine the answer. section 2.2 section 8.4.2

Unlike multiple-choice benchmarks, LAMBADA does not ask the model to select among supplied candidates. It can therefore reveal degradation in precise word prediction that remains less visible on closed-set tasks. section 2.2 section 8.4.2

benchmark_tasks = ['arc_easy', 'winogrande','hellaswag', 'lambada_openai']
 
baseline_results = model_evaluation(
    model, 
    tokenizer, 
    benchmark_tasks, limit=None
)

Evaluate LAMBADA alongside three other benchmarks using the lm-eval task name lambada_openai. The helper wraps simple_evaluate and returns scores for comparison; limit=None runs without a sample limit. section 2.2

Baseline and pruning comparison

A Gemma-3-270M baseline scored 0.43 on lambada_openai. In a separate Llama-3.2-3B attention-pruning experiment, LAMBADA dropped 9.2% with three layers removed and 19.1% with six removed, while four other benchmarks changed by less than 2% at six layers. Removing 15 layers caused scores to collapse across tasks. section 2.2 section 8.4.2

Use LAMBADA with complementary task and generation checks rather than as a complete measure of model quality. In one pruning experiment, short outputs remained coherent even though sustained generation showed factual inconsistencies. section 8.5.2

How Rearchitecting LLMs covers it

How deep it goes
Defines the exact last-word task and compares pruning results, without detailing dataset construction.
Examples
Reports a Gemma-3-270M baseline and Llama-3.2-3B attention-pruning scores.
What you can do after
Apply it via lm-eval to compare model benchmark scores before and after optimization.
How it connects
Connects language modeling with attention pruning, knowledge distillation, and multiple-choice evaluation.

Also explained in

Compare how 2 books explain LAMBADA →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Rearchitecting LLMs. The section links point to the parts of the book it draws on.