liveBook · Manning

LLMLingua

As explained in Retrieval Augmented Generation, The Foundational Ideas

LLMLingua is a prompt-compression framework and software library for reducing prompt length while preserving task-relevant information.

LLMLingua compresses retrieved prompts before generation, reducing token cost and context pressure while preserving task-relevant information. A small proxy model scores and prunes text; a separate target generator answers from the compressed prompt. The proxy never generates, and the target never receives raw documents. section 13.2.4

Choose the pipeline according to the required signal:

  • LongLLMLingua uses contrastive perplexity, attaches the question during scoring, and supports context reordering and dynamic compression. section 13.2.4
  • LLMLingua-2 uses a BERT-style token classifier distilled from GPT-4 decisions; it is faster and tidier but does not see the question. section 13.2.4
  • Avoid the extra proxy-model latency when retrieved documents are already short and well targeted. section 13.2.4

Compression can change the answer

A contract example shrinks 347 tokens to 212, or 1.6x. The compressed text drops wording and alters key clauses; one query therefore changes from correctly excluding indemnification obligations from a liability cap to incorrectly including them. section 13.2.4

Listing 13.1 LLMLingua.
from llmlingua import PromptCompressor
from typing import List, Dict, Any
 
def compress_with_longllmlingua(
    query: str,
    documents: List[str],
    rate: float = 0.5
) -> Dict[str, Any]:
    """Compress documents with contrastive perplexity."""
 
    compressor = PromptCompressor(  #A
        model_name="Qwen/Qwen2.5-0.5B",
        use_llmlingua2=False,
        device_map="cpu"
    )
 
    compressed_output = compressor.compress_prompt(  #B
        documents,
        question=query,
        rate=rate,
        condition_compare=True,
        condition_in_question="after",
        reorder_context="sort",
        dynamic_context_compression_ratio=0.4,
        rank_method="longllmlingua"
    )
 
    return compressed_output

A Python listing shows PromptCompressor configuration, question-conditioned compression, context reordering, and result analysis. section 13.2.4

How Retrieval Augmented Generation, The Foundational Ideas covers it

How deep it goes
Explains the two-model architecture, algorithm choice, parameters, metrics, and failure tradeoffs, but not model internals.
Examples
Uses Python listings, contract clauses, token counts, answer comparisons, and benchmark results.
What it assumes
Assumes retrieved documents, a query, and separate proxy and target models; teaches the compressor pipeline from configuration onward.
What you can do after
Apply it via the llmlingua library and evaluate token reduction, compression ratio, latency, and answer preservation.
How it connects
Connects prompt compression with retrieval-augmented generation, context windows, lost-in-the-middle effects, and LlamaIndex post-processors.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Retrieval Augmented Generation, The Foundational Ideas. The section links point to the parts of the book it draws on.