liveBook · Manning

LAMBADA

As explained in Domain-Specific Small Language Models

A narrative text dataset used here to evaluate word-prediction and model accuracy after quantization.

LAMBADA is a collection of narrative passages used here to evaluate text understanding with a word-prediction task, especially before and after quantization. section 9.2

The evaluation uses the first 1,000 validation samples and compares FP16, naive W8A8, and SmoothQuant W8A8 model accuracy on the same prompt set. section 9.2

Typical use in this workflow:

  • Load a held-out validation slice from the dataset. section 9.2
  • Score the original model for a baseline accuracy. section 9.2
  • Repeat the same evaluation after quantization or smoothing. section 9.2

Accuracy comparison on the same sample set

A 2.7B OPT model keeps comparable accuracy on 1,000 LAMBADA samples after SmoothQuant W8A8 quantization, showing the dataset as a regression check for model quality. section 9.2

How Domain-Specific Small Language Models covers it

How deep it goes
Defines the dataset and its evaluation role, then shows accuracy comparisons without deeper corpus or task analysis.
Examples
Uses 1,000 validation samples and FP16, naive W8A8, and SmoothQuant W8A8 accuracy figures.
What it assumes
Assumes a language model and basic evaluation tooling; it introduces the dataset and task from the ground up.
What you can do after
Apply it to compare baseline and quantized model accuracy on narrative word-prediction tasks.
How it connects
Connects to quantization, model accuracy, word prediction, and text understanding.
Cover of Domain-Specific Small Language Models
liveBookmanning.com
Read Domain-Specific Small Language Models on liveBook Get the book at manning.com

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Domain-Specific Small Language Models. The section links point to the parts of the book it draws on.