liveBook · Manning

Query-side fine-tuning

As explained in Retrieval Augmented Generation, The Foundational Ideas

Query-side fine-tuning adapts the query-processing component of a retrieval system without changing document representations in the corpus index.

A retrieval system can find a passage that repeats a question’s words while missing one that answers it. Query-side fine-tuning addresses this ranking failure without rebuilding the corpus index: freeze the document encoder, keep its indexed embeddings fixed, and train only the query encoder to place queries nearer useful documents. section 6.4.1 section 6.8

For answer generation, the reader can supply the training signal. Perplexity Distillation favors documents that make the correct answer easier for the reader to generate, rather than requiring document-relevance labels. This shifts retrieval toward usefulness to the answer, not just similarity to the question. section 6.5 section 6.8

Choose an update strategy according to data volume and indexing cost:

  • Full index update re-embeds the corpus after retriever changes. It offers the highest potential accuracy but incurs substantial update overhead. section 6.4.1
  • Re-ranking retrieves a broad candidate set from the existing index, then re-embeds and ranks those candidates with the updated model. It depends on the initial search finding the best documents. section 6.4.1
  • Query-side fine-tuning avoids historical index refreshes. In a 64-example comparison, it outperformed full fine-tuning; with larger datasets, its peak accuracy may be lower. section 6.4.1

Adapting an enterprise knowledge search

An analyst asks why operating margins declined. A baseline search ranks a passage reporting the margin above one explaining its causes. After training on 64 examples, the retriever gives more weight to causal commentary. When another quarter’s reports arrive, the team embeds and adds the new documents, then fine-tunes the query side with a small new example set—without re-embedding the historical corpus. section 6.7

Table 6.5 Update strategies
Strategy
Mechanism
Computational Cost
Use Case
Key Trade-off
Full Index Update
Re-embed the entire corpus periodically.
Very High
Full-dataset fine-tuning; when accuracy is paramount.
Highest potential accuracy at the cost of downtime/overhead.
Re-ranking
Retrieve N≫k with stale index; re-rank N live.
Low
Balancing freshness and efficiency; production systems.
Assumes the best documents are in the initial N candidates.
Query-Side Fine-tuning
Freeze document encoder; only train query encoder.
Very Low
Few-shot learning (e.g., <1000 examples); rapid adaptation.
Potentially lower peak accuracy with large datasets.

The comparison puts index-refresh cost, candidate re-ranking, and query-only training alongside their intended uses and limitations. section 6.4.1

How Retrieval Augmented Generation, The Foundational Ideas covers it

How deep it goes
Defines which encoder is frozen and compares update costs and few-shot accuracy, without specifying query-encoder training steps.
Examples
Contrasts a 64-shot strategy comparison with enterprise-report retrieval that favors causal commentary over matching statistics.
What it assumes
Assumes a retrieval system with separate query and document encoders, an indexed corpus, and a reader that evaluates candidate documents.
What you can do after
Apply it to adapt few-shot retrieval without re-embedding historical documents, and evaluate the accuracy trade-off against fuller updates.
How it connects
Links query encoders and document encoders to vector indexes, reader feedback, Perplexity Distillation, and re-ranking.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Retrieval Augmented Generation, The Foundational Ideas. The section links point to the parts of the book it draws on.