liveBook · Manning

Ablation study

As explained in Retrieval Augmented Generation, The Foundational Ideas

An ablation study is an experiment that removes or disables system components to measure their individual contributions.

An ablation study removes or disables system components, then compares performance to determine which parts contribute to the result. Reliable comparisons use structurally similar models and an unseen test set. section 3.2.1 section 4.5.4

A useful ablation checks:

  • Whether removing a feature causes a meaningful performance drop rather than a minor, potentially noisy gain or loss. section 3.2.1
  • Whether accuracy changes justify added latency, memory, or index size. section 3.2.1
  • Whether individual components improve evidence synthesis, retrieval coverage, or answer focus. section 4.5.4

RAG component removal

A RAG system scored 44.5% with its full configuration. Removing marginalization reduced performance to 41.5%; using pretrained models without fine-tuning reduced it to 37.2%; setting K=1 instead of K=5 reduced it to 39.8%. These comparisons attribute measurable contributions to the disabled components. section 4.5.4

How Retrieval Augmented Generation, The Foundational Ideas covers it

How deep it goes
Explains controlled comparisons, unseen test sets, efficiency checks, and concrete RAG component removals, but does not detail statistical procedures.
Examples
Uses RAG variants with marginalization disabled, fine-tuning removed, K reduced from 5 to 1, and RAG token replaced by RAG sequence.
What it assumes
Assumes a baseline model with separable components and a test set, while emphasizing comparisons between structurally similar systems.
What you can do after
Evaluate model components by comparing controlled variants on an unseen test set while weighing accuracy against latency and resource costs.
How it connects
Connects ablation studies with RAG, retrieval, fine-tuning, marginalization, model evaluation, latency, and memory.

Also explained in

Compare how 5 books explain ablation study →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Retrieval Augmented Generation, The Foundational Ideas. The section links point to the parts of the book it draws on.