liveBook · Manning

LLMLingua-2

As explained in Retrieval Augmented Generation, The Foundational Ideas

LLMLingua-2 is a prompt-compression system that uses a distilled token classifier to remove less important tokens.

LLMLingua-2 compresses prompts with a BERT-style token classifier distilled from GPT-4's compression decisions. It removes less important tokens without seeing the question, producing faster, tidier text than question-conditioned compression, but potentially preserving less query-specific context. section 13.2.4

Choose between the two algorithms exposed by the same compressor interface:

  • Set use_llmlingua2=True for the faster classifier-based approach; LongLLMLingua parameters are ignored. section 13.2.4
  • Set use_llmlingua2=False for contrastive perplexity, which uses a causal language model and the question to score tokens. section 13.2.4

Compressing contract context

A LLMLingua-2 configuration compresses contract articles at rate=0.5. The resulting text retains limitation-of-liability terms but may become terse enough to affect interpretation, so compare its answer with the uncompressed context.

How Retrieval Augmented Generation, The Foundational Ideas covers it

How deep it goes
Explains the classifier distinction, configuration switch, and tradeoff, but does not detail classifier training or token-scoring internals.
Examples
Uses compressed contract clauses and a PromptCompressor configuration to illustrate retained legal context and answer risk.
What you can do after
Operate/configure LLMLingua-2 through PromptCompressor and compare its compressed output with uncompressed context.
How it connects
Connects prompt compression with token pruning, causal language models, contrastive perplexity, and retrieval-augmented generation.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Retrieval Augmented Generation, The Foundational Ideas. The section links point to the parts of the book it draws on.