liveBook · Manning

Counterfactual question

As explained in Hidden Influences

A question about what outcome would have occurred under an alternative action or condition that was not observed.

If a recommendation system had shown someone different content, how might they have responded? That is a counterfactual question: it asks what would have happened under an alternative condition that was never observed. section 4.4

Past interactions cannot answer this fully. People generally interact only with items they were shown, so historical data cannot reveal whether they would have clicked unseen content, spent more time on a platform, or encountered a new interest. Comparing a new recommender’s predictions with past behavior offers only partial evidence and reflects what the previous model chose to show. section 4.4

Figure 4.3 Generally, users interact only with the items presented to them. Very likely, there were many other items that users would have interacted with if they had been shown to them. Because the algorithm is learning via past interactions, it’s possible that certain topics (like the Books topic here) were never presented to the users, thus completely missing one of their interests.

The figure illustrates how items never shown to users can conceal interests and leave possible interactions unobserved. section 4.4

How Hidden Influences covers it

How deep it goes
Defines the question and explains its limits in historical comparisons, but does not quantify unobserved responses.
Examples
Uses unseen recommended content, possible new interests, platform time, and ad clicks as examples of unobserved behavior.
What you can do after
Evaluate why historical interactions provide only partial evidence about a new recommender’s quality.
How it connects
Connects recommender systems with historical evaluation and A/B testing.

This explainer was generated by Manning's knowledge graph from the text of Hidden Influences. The section links point to the parts of the book it draws on.