How deeply does Domain-Specific Small Language Models cover perplexity?

Defines word, byte, and bits-per-byte measures with one concrete benchmark, but does not derive the metric mathematically.

Perplexity

As explained in Domain-Specific Small Language Models

A measure of how well a language model predicts tokens in sequence, with lower values generally indicating better predictive fit.

Perplexity measures how well a language model predicts a sequence. Lower values indicate less predictive uncertainty and generally better fit for the evaluated data. section 4.1.4 section 8.5

Evaluation workflows use related metrics for different prediction units:

Word perplexity measures prediction over words. section 4.1.4
Byte perplexity measures prediction of the next byte from preceding bytes. section 4.1.4
Bits per byte reports the average number of bits needed to predict the next symbol. section 4.1.4
For generated proteins, lower perplexity supports higher-quality output, but biological validation still requires domain-specific tools. section 8.5

Language-model evaluation

An LM-Eval wikitext run for GPT-Neo 2.7B reports word perplexity 13.9382, byte perplexity 1.6367, and bits per byte 0.7108. section 4.1.4

hf-causal (pretrained=EleutherAI/gpt-neo-2.7B,dtype=float16), limit: None, provide_description: False, num_fewshot: 0, batch_size: None
|  Task  |Version|    Metric     | Value |   |Stderr|
|--------|------:|---------------|------:|---|------|
|wikitext|      1|word_perplexity|13.9382|   |      |
|        |       |byte_perplexity| 1.6367|   |      |
|        |       |bits_per_byte  | 0.7108|   |      |

Inspect a tabular LM-Eval result showing perplexity alongside related prediction metrics. section 4.1.4

How Domain-Specific Small Language Models covers it

Cover of Domain-Specific Small Language Models

liveBook manning.com

Read Domain-Specific Small Language Models on liveBook Get the book at manning.com

Also explained in

liveBook manning.com

Build a Large Language Model (From Scratch)An evaluation metric defined as the exponential of cross entropy loss, indicating how uncertain the model is about the next token.
liveBook manning.com

Sutskever's ListPerplexity is a language-model metric representing the effective number of plausible next tokens, with lower values indicating greater predictive certainty.
liveBook manning.com

Transformers in ActionPerplexity is a language-modeling metric equal to the exponential of average cross-entropy loss, measuring how well a model predicts a target token distribution.
liveBook manning.com

Generative AI in ActionPerplexity is a language-model metric that quantifies uncertainty in predicting the next token of a sequence.

Compare how 5 books explain perplexity →

Related concepts

input reversalproduced by · Input reversal is the technique of presenting a source sequence to a sequence model in reverse element order.
language modelmeasures · A language model is a computational model that represents patterns in linguistic tokens to estimate sequence probabilities and predict or generate language from context.
fluencymeasures · Fluency is the degree to which language production is smooth, coherent, grammatical, natural, and readable.
compressioncontrasts with · Compression is the transformation of information into a more compact representation to reduce storage or transmission requirements while retaining information needed for recovery or use.
quantizationcontrasts with · Quantization maps numerical values to a finite set of discrete representations, often using reduced precision to lower model storage or computation requirements.
cross entropy lossuses · Cross-entropy loss measures the expected negative log probability that a predicted distribution assigns to outcomes drawn from a target distribution.

This explainer was generated by Manning's knowledge graph from the text of Domain-Specific Small Language Models. The section links point to the parts of the book it draws on.