liveBook · Manning

Dynamic token budgeting

As explained in Retrieval Augmented Generation, The Foundational Ideas

Dynamic token budgeting allocates different compression budgets to items according to their estimated importance or rank.

Dynamic token budgeting assigns stronger pruning to lower-ranked documents, preserving more tokens for context estimated to matter most. In a retrieval-generation pipeline, it follows coarse document ranking and precedes token pruning and context reordering. section 13.2.4

A practical compression pipeline combines four stages:

  • Rank documents against the query and discard entire low-value documents. section 13.2.4
  • Allocate different compression budgets by document rank, using more aggressive pruning for lower-ranked items. section 13.2.4
  • Prune tokens, then reorder surviving context so important material reaches the prompt edges. section 13.2.4

Contract-question compression

Compressing four contract articles reduced 347 tokens to 212, a 1.6x ratio. The resulting text retained the indemnification and liability-cap clauses but became visibly damaged; alternate compressed outputs produced opposite answers, showing that budget selection requires task-level evaluation. section 13.2.4

Listing 13.1 LLMLingua.
from llmlingua import PromptCompressor
from typing import List, Dict, Any
 
def compress_with_longllmlingua(
    query: str,
    documents: List[str],
    rate: float = 0.5
) -> Dict[str, Any]:
    """Compress documents with contrastive perplexity."""
 
    compressor = PromptCompressor(  #A
        model_name="Qwen/Qwen2.5-0.5B",
        use_llmlingua2=False,
        device_map="cpu"
    )
 
    compressed_output = compressor.compress_prompt(  #B
        documents,
        question=query,
        rate=rate,
        condition_compare=True,
        condition_in_question="after",
        reorder_context="sort",
        dynamic_context_compression_ratio=0.4,
        rank_method="longllmlingua"
    )
 
    return compressed_output

The listing shows dynamic budgeting configured through LLMLingua's dynamic_context_compression_ratio parameter. section 13.2.4

How Retrieval Augmented Generation, The Foundational Ideas covers it

How deep it goes
Implements rank-sensitive allocation within a four-stage pipeline but leaves ratio selection to task-specific evaluation.
Examples
Uses contract articles, a 347-to-212-token reduction, and conflicting answers to expose compression benefits and risks.
What it assumes
Requires a causal language model for contrastive perplexity and a query-document compression pipeline.
What you can do after
Apply via LLMLingua by configuring a dynamic compression ratio and inspecting token counts and compressed prompts.
How it connects
Ties coarse document ranking to token pruning, context reordering, and contrastive perplexity.

This explainer was generated by Manning's knowledge graph from the text of Retrieval Augmented Generation, The Foundational Ideas. The section links point to the parts of the book it draws on.