Replay evaluation

As explained in Machine Learning System Design

An offline recommendation-evaluation procedure that replays historical interactions and retains cases where a candidate recommendation matches the historically selected item.

Replay evaluation provides an offline way to test recommendation algorithms when production experiments are costly or risky. It replays historical interactions and keeps cases where a candidate recommendation matches the item historically selected. This is useful because recommendation logs reveal feedback for shown items, but not for alternatives that were never shown. section 12.1.3

Use replay evaluation to:

• Evaluate candidate recommendations against historical user contexts and selected items before exposing them to users. section 12.1.3
• Prefer a more sophisticated and computationally expensive offline procedure when recommendation decisions carry high stakes. section 12.1.3

Advertising recommendations

An algorithm receives a user's recent history, chooses an ad banner, and seeks to improve CTR, CVR, or eCPM. Replay evaluation uses logged recommendation events to simulate this decision process offline. section 12.1.3

Figure 12.2 Unbiased estimator (offline validation): simulation of events, example 1

Compare the simulated event sequence with the historical interaction stream to see how replay-based offline validation is illustrated. section 12.1.3

How Machine Learning System Design covers it

How deep it goes

Defines replay as a simulation procedure and motivates its use, without developing the underlying estimator mathematically.

Examples

Uses news, ads, and product recommendations, including ad banners evaluated with CTR, CVR, and eCPM.

What it assumes

Assumes familiarity with logged impressions, user context, and partly labeled feedback in recommendation systems.

What you can do after

Evaluate candidate recommendation algorithms offline against logged user contexts and historically selected items.

How it connects

Links offline evaluation to contextual bandits, reinforcement learning, recommendation systems, and simulated environments.

liveBookmanning.com

Read Machine Learning System Design on liveBook Get the book at manning.com

Related concepts

• evaluationkind of · Evaluation is the systematic assessment of a system, result, or approach against relevant criteria, evidence, or objectives.
• historical datauses · Historical data is information recorded about past events, observations, outcomes, behaviors, or states and retained for later access or analysis.
• event replayuses · Event replay is the reprocessing of previously recorded events or inputs, typically in their original order, to reconstruct state or reproduce or evaluate behavior.

This explainer was generated by Manning's knowledge graph from the text of Machine Learning System Design. The section links point to the parts of the book it draws on.