liveBook · Manning

Prior probability

As explained in AI-Powered Search

A prior distribution is a probability distribution representing an initial belief about an unknown quantity before incorporating case-specific observations.

A search result clicked once in one examination has a raw click rate of 1.0, but that single observation is weak evidence of relevance. A prior probability represents an initial belief about an unknown quantity before incorporating observations for a particular case. A beta prior can keep a low-confidence relevance grade from being treated as certain. section 11.3.2

Choose a prior relevance grade and a prior weight. For a beta distribution, set prior_a = grade × weight and prior_b = (1 − grade) × weight. Update prior_a with clicks and prior_b with examines minus clicks. The posterior mean, posterior_a / (posterior_a + posterior_b), gives the adjusted relevance grade. A larger prior weight makes observations move the grade more slowly. section 11.3.2

Adjusting a sparse click rate

With a prior grade of 0.3 and weight of 100, the beta parameters are 30 and 70. A result with one click in one examination updates them to 31 and 70, giving an adjusted grade of about 0.307 rather than the raw grade of 1.0. A result with 14 clicks in 34 examinations receives an adjusted grade of about 0.328. section 11.3.2

Listing 11.13 Computing posterior beta distribution
def calculate_sdbn(sessions, prior_grade=0.3, prior_weight=100):
  sessions = calculate_prior(sessions, prior_grade, prior_weight)
  sessions["posterior_a"] = (sessions["prior_a"] +  #1
                             sessions["clicked"])  #1
 sessions["posterior_b"] = (sessions["prior_b"] +    #2
    sessions["examined"] - sessions["clicked"])  #2
  sessions["beta_grade"] = (sessions["posterior_a"] /  #3
    (sessions["posterior_a"] + sessions["posterior_b"]))  #3
  return sessions.sort_values("beta_grade", ascending=False)

query = "blue ray"
sessions = get_sample_sessions(query)
bluray_sdbn_data = calculate_sdbn(sessions)
print(bluray_sdbn_data)

The Python calculation updates the beta parameters from clicks and examines, then sorts results by their adjusted grades. section 11.3.2

Confidence weighting preserves sparse examples that a hard examination threshold might discard. It does not eliminate every source of bias: users can act only on results the search system presents, so the resulting feedback still reflects the current ranking. section 11.3.2 section 11.4

How AI-Powered Search covers it

How deep it goes
Works through beta parameters, posterior updates, and adjusted relevance grades in Python.
Examples
Uses blue ray click and examination counts to compare raw and beta-adjusted grades.
What you can do after
Apply it in Python to adjust click-model relevance grades before training a ranking model.
How it connects
Connects beta distributions with click models, confidence bias, relevance judgments, and learning to rank.

Also explained in

Compare how 12 books explain prior probability →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of AI-Powered Search. The section links point to the parts of the book it draws on.