liveBook · Manning

Causal language model

As explained in Retrieval Augmented Generation, The Foundational Ideas

A causal language model predicts each token from preceding tokens and is trained for autoregressive sequence modeling.

A causal language model predicts next-token probabilities from preceding tokens. In prompt compression, it acts as a proxy scorer: comparing probabilities with and without the question helps identify context worth preserving. section 13.2.4

Use a causal language model when:

  • Contrastive perplexity must judge document tokens in relation to a query. section 13.2.4
  • A small proxy model can run before generation to reduce context length and token cost. section 13.2.4
  • The additional scoring latency is justified by documents that exceed the generator's effective context window. section 13.2.4

Proxy scorer in LLMLingua

Listing 13.1 configures Qwen/Qwen2.5-0.5B as the causal proxy on a laptop CPU, then compresses documents with the question attached. The alternative LLMLingua-2 path uses a BERT-style token classifier and does not see the question. section 13.2.4

How Retrieval Augmented Generation, The Foundational Ideas covers it

How deep it goes
Explains next-token probability scoring and its compression role, but does not develop model architecture or training internals.
Examples
Uses Qwen/Qwen2.5-0.5B inside PromptCompressor to score and compress retrieved documents.
What it assumes
Assumes familiarity with tokens, queries, retrieved documents, and next-token probabilities.
What you can do after
Apply it via PromptCompressor to perform query-conditioned document compression.
How it connects
Connects causal language models to contrastive perplexity, prompt compression, retrieval-augmented generation, and token classification.

Also explained in

Compare how 5 books explain causal language model →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Retrieval Augmented Generation, The Foundational Ideas. The section links point to the parts of the book it draws on.