Noise is corruption or unwanted variation in observed feedback that obscures a customer’s underlying evaluation. In A/B testing and product recommendation, external conditions can make ratings unreliable, so a platform should not treat a few responses as conclusive evidence of preference. section 10.0
Customer feedback may be distorted by factors unrelated to the product itself:
- The number of advertisements shown during an online streaming experience. section 10.0
- The quality of package delivery. section 10.0
- The customer’s general mood while consuming the product. section 10.0
Why more feedback may be necessary
A customer’s low product rating may reflect poor delivery rather than dissatisfaction with the product. Because querying customers can intrude on their experience, collecting enough evidence is useful but costly. section 10.0
Figure 10.1 Examples of user’s preference elicitation in production recommendation. On the left, the user is asked to rate a recommended product. On the right, the user is asked to pick the product they like better. The latter situation helps better elicit the user’s preference.
Compare numerical product ratings with pairwise choices; choosing between products can better elicit a customer’s preference. section 10.0
