liveBook · Manning

Ablation study

As explained in AI Model Evaluation

An ablation study evaluates the effect of removing or varying selected inputs, features, or components while holding other conditions constant.

An ablation study tests the effect of removing or varying selected inputs, features, or components while holding other conditions constant. This helps isolate which change may be responsible for a difference in outcomes. section 3.1.1

Testing age effects in movie recommendations

Give synthetic users identical viewing behavior, vary only their age, and compare the genre distributions of their recommendations. In one diagnostic, Indie recommendations rose from 8.81% for ages 18–24 to 14.30% for ages 55+. Because preferences were held constant, the difference warrants investigation of how age affects exposure; it does not by itself establish bias among real users. section 3.1.1

Listing 3.1 Analyzing Distribution of Genres by Age Group Cohorts
# Calculate the distribution of genres within each age group's recommendations
genre_diversity = analysis_df.groupby(['age_group', 'genre']).size().reset_index(name='count')
genre_pivot = genre_diversity.pivot(index='age_group', columns='genre', values='count').fillna(0)
genre_pivot = genre_pivot.div(genre_pivot.sum(axis=1), axis=0)  # Normalize by row to get proportions
 
genre_pivot
 

The listing groups recommendations by age and genre, then normalizes each age group's counts into proportions for comparison. section 3.1.1

How AI Model Evaluation covers it

How deep it goes
Works through varying age while holding viewing behavior constant, then compares normalized recommendation distributions.
Examples
Compares genre proportions across synthetic age cohorts, including differences in Action and Indie recommendations.
What you can do after
Evaluate how changing a demographic feature affects recommendation outputs when viewing behavior is held constant.
How it connects
Connects ablation to recommendation bias, subgroup analysis, and genre exposure.

Also explained in

Compare how 5 books explain ablation study →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of AI Model Evaluation. The section links point to the parts of the book it draws on.