A click model turns search behavior into estimates of result relevance. Rather than treating every click as a reliable judgment, it uses records of the query, the ranked results shown, and which results were clicked. Those estimates can become implicit relevance judgments for training or retraining a learning-to-rank (LTR) model. section 11.1.3 section 11.5
A simple starting point is click-through rate (CTR): for each document returned for a query, divide its clicks by the number of sessions in which it appeared. This produces a useful first ranking of observed interest, but not a dependable ranking of relevance. Users tend to click results near the top, and a document seen only a few times can acquire a misleadingly high rate. section 11.1.3 section 11.5
From sessions to judgments
For the query “transformers dark of the moon,” a session records ranked product results and a click on a Blu-ray disc at rank 7. Aggregating sessions gives that disc a CTR of 0.0824. Another query, “dryer,” produces an apparently unrelated movie among results ranked by CTR—a prompt to inspect the signal before using it as a relevance judgment. Probabilistic judgments can also supply pairwise LTR examples: if one result has grade 0.80 and another 0.20, the higher-graded result is the preferred member of the pair. section 11.1.3 section 11.1.2
Listing 11.4 Computing CTR
def calculate_ctr(sessions):
click_counts = sessions.groupby("doc_id")["clicked"].sum()
sess_counts = sessions.groupby("doc_id")["sess_id"].nunique()
ctrs = click_counts / sess_counts
return ctrs.sort_values(ascending=False)
query = "transformers dark of the moon"
sessions = get_sessions(query, index=False)
click_through_rates = calculate_ctr(sessions)
print_series_data(click_through_rates, column="CTR")
The pandas calculation groups session rows by document, counts clicks and distinct sessions, then divides to obtain CTR. section 11.1.3
For automated LTR, improve on raw CTR by accounting for whether a result or position was likely examined and by treating estimates from sparse clicks cautiously; a beta-distribution prior is one way to update confidence gradually. Test the resulting ranking with users. If an LTR model performs well before deployment but fails an A/B test, revisit its click-derived judgments: presentation, sales, snippets, or post-click experiences may influence behavior without cleanly measuring search relevance. section 11.5 section 12.2.4
