Click-through rate (CTR) measures the fraction of shown ads that receive a click: sum of binary click outcomes divided by the number of ads shown. Each outcome is 1 for a click and 0 otherwise, so CTR is the mean of the click-indicator vector. In an ad-serving system, a click model estimates click probabilities across the inventory and the ad selector serves the highest-scoring ad. CTR is therefore both a production quality measure for the click model and a business metric: daily revenue equals cost per click × CTR × visitor count. section 3.1 section 3.3.1
Figure 3.1 An ad-serving system. The ad selector pulls ads from the ad inventory, then asks the click model to estimate the probability that the user will click on each ad. The ad selector sends the ad with the highest estimated click probability to the user. The page tells the server whether the user clicked on the ad, shown here as a variable clicked taking a value of either 0 (not clicked) or 1 (clicked). The rest of the web page is delivered to the user by a web server, not shown here.
The ad-serving flow connects predicted click probabilities, ad selection, and the logged binary outcome from which CTR is computed. section 3.1
Comparing two click models
Model A is simulated at CTR 0.0050 and model B at 0.0070. A conventional A/B test randomly assigns each exposure to A or B with equal probability, logs the clicks separately, and compares their mean outcomes. During one simulated run, the combined CTR ends near 0.0058; across many runs its mean is 0.0060, the average of the two models' CTRs. Equal allocation supports comparison, but exposes many users to the weaker model while the test runs. section 3.1.1
CTR is an aggregate measurement, not a collection of values with its own standard deviation. Its uncertainty can instead be summarized with the standard error of the binary click observations. For ten outcomes containing four clicks, CTR is 4/10 = 0.4 and the reported standard error is about 0.16. Bootstrap resampling produces many plausible CTR estimates from one observed click vector; their spread estimates uncertainty and can estimate the probability that each model is best. section 3.3.1
When CTR is the live objective, selection policy changes the trade-off between measurement and immediate performance:
- Use equal randomization for an A/B-test baseline; it does not adapt to observed CTR and its combined performance tends toward the average of the alternatives. section 3.1.1
- Use epsilon-greedy to select the model with higher realized CTR with probability 1 − epsilon, while randomly exploring with probability epsilon. Epsilon controls this bias toward the current leader. section 3.1.2
- Use Thompson sampling to select arms in proportion to their estimated probability of having the highest CTR, while retaining enough samples per arm to resolve practically significant differences. section 3.3.3
A small CTR difference may be unmeasurable at the available sample size. With n binary observations, changing one outcome changes CTR by 1/n; to make that resolution no larger than a practical-significance threshold PS, use at least 1/PS observations per arm. This discretization limit is separate from natural sampling variability represented by standard error. Bootstrap precision also trades computation against accuracy: more bootstrap means are more exact but slower. section 3.3.3
Treat a higher observed CTR as evidence rather than certainty. Even when one arm's expected CTR is 0.0070 versus another's 0.0050, measurement uncertainty means the data do not establish the better arm with 100% certainty. Adaptive policies improve live CTR by exploiting favorable results, but still reserve exploration or probability matching to keep learning. section 3.1.2 section 3.3.1
